Blogs

Private Docker Registry w/Nginx Proxy for Stats Collection

Dan Tehranian, Staff Engineer –

In using Docker at Virtual Instruments, we had to setup an internal Docker Registry in order to share our custom Docker images. Setting up a basic internal Docker Registry is trivially easy, but there was one issue we hit in production-izing this app.

In VI’s R&D infrastructure we run all of our web apps through proxies like Nginx for the following benefits:

  • We can run the app itself via an unprivileged user on an unprivileged port (>1024) and proxy HTTP traffic from port 80 on that host to the unprivileged app’s port. This means that our app does not need to run with root permissions in order to receive requests from port 80.
  • What follows from the above is the convenience that humans don’t have to remember the magical TCP port number for each of the numerous internal web apps that we have (ex, 5000 for Docker, 8080 for Jenkins, etc). A user can just put the DNS name of the server in their browser and go.
  • We can configure the “HttpStubStatus” module into our nginx proxy in order to provide a URL that displays realtime stats about our web server’s traffic. Ex: Number of HTTP Connections or HTTP Requests/second. We can then use collectd’s nginx plugin to collect that data and send it to our monitoring systems for analysis & visualization. The end result is something like this:

Nginx_connections_and_requests

The snag that we hit is that with our tried-and-true basic nginx proxy configuration, “docker push” commands from our clients were failing without any useful error messages. Googling around, I figured out that when serving a Docker Registry via nginx there is special nginx configuration required to enable chunked encoding of messages.

After manually making the above configuration changes to our nginx proxy, our test “docker push”es were finally completing successfully. The last step was to put all of this configuration into Puppet, our configuration management system.

In case anyone would benefit from this, I have provided the meat of the Puppet configuration for a private Docker Registry w/nginx proxy serving web server stats to collected here:

  # Create an internal docker registry w/an nginx proxy in front of it
  # directing requests from port 80. Also add a "/nginx_status" URL so that
  # we can monitor web server metrics via collectd.

  include docker

  $docker_dir = '/srv/docker'  # where to store images

  file { $docker_dir:
    ensure => directory,
    owner  => 'root',
    group  => 'root',
  }

  # see docs on docker's registry app at:
  #   https://github.com/dotcloud/docker-registry/blob/master/README.md

  docker::image { 'registry': }

  docker::run { 'registry':
    image           => 'registry',
    ports           => ['5000:5000',],
    volumes         => ["${docker_dir}:/tmp/registry"],
    use_name        => true,
    env             => ['SETTINGS_FLAVOR=local',],
    restart_service => true,
    privileged      => false,
    require         => File[$docker_dir],
  }

  # the following nginx vhost proxy and location are configured per:
  #   https://github.com/docker/docker-registry/blob/0.7.3/contrib/nginx.conf
  #
  # "nginx-extras" is the Ubuntu 12.04 nginx package with extra modules like
  # "chunkin" compiled in
  class { 'nginx':
    confd_purge   => true,
    vhost_purge   => true,
    manage_repo   => false,
    package_name  => 'nginx-extras',
  }
  nginx::resource::upstream { 'docker_registry_app':
    members => ['localhost:5000',],
  }
  nginx::resource::vhost { 'docker_registry_app':
    server_name          => ['vi-docker.lab.vi.local'],
    proxy                => 'http://docker_registry_app',
    listen_options       => 'default_server',
    client_max_body_size => 0,  # allow for unlimited image upload size
    proxy_set_header     => [
      'Host $http_host',
      'X-Real-IP $remote_addr',
    ],
    vhost_cfg_append     => {
      chunkin    => 'on',
      error_page => '411 = @my_411_error',
    },
  }
  nginx::resource::location { 'my_411_error':
    vhost               => 'docker_registry_app',
    location            => '@my_411_error',
    location_custom_cfg => {
      # HACK: the value for chunkin_resume key needs to be non-empty string
      'chunkin_resume' => ' ',
    },
  }

  # nginx_status URL for collectd to pull stats from. see:
  #   https://collectd.org/wiki/index.php/Plugin:nginx
  nginx::resource::location { 'nginx_status':
    ensure              => present,
    location            => '/nginx_status',
    stub_status         => true,
    vhost               => 'docker_registry_app',
    location_cfg_append => {
      'access_log' => 'off',
    },
  }

  # collectd configuration for nginx
  class { '::collectd':
    purge        => true,
    recurse      => true,
    purge_config => true,
  }
  class { 'collectd::plugin::nginx':
    url   => 'http://vi-docker.lab.vi.local/nginx_status',
  }