Search the Community

As a critical part of Docker’s transition into sustainability, we’ve been gradually rolling out limits on docker pulls to the heaviest users of Docker Hub. As we near the end of the implementation of the rate limits, we thought we’d share some of the facts and figures behind our effort. Our goal is to ensure that Docker becomes sustainable for the long term, while continuing to offer developers 100% free tools to build, share, and run their applications. We announced this plan in August with an effective date of November 1. We also shared that “roughly 30% of all downloads on Hub come from only 1% of our anonymous users,” illustrated in this chart: This shows the dramatic impact that a very small percentage of anonymous, free users have on all of Docker Hub. That excessive usage by just 1%–2% of our users results not only in an unsustainable model for Docker but also slows performance for the other 98%–99% of the 11.3 million developers, CI services, and other platforms using Docker Hub every month. Those developers rely upon us to save and share their own container images, as well as to pull images from Docker Verified Publishers and our own trusted library of Docker Official Images, amounting to more than 13.6 billion pulls per month. Based on our goal of ensuring the vast majority of developers can remain productive, we designed limits of 100 or 200 pulls in a 6-hour window for anonymous and authenticated free users, respectively. In the context of a developer’s daily workflow, 100 pulls in 6 hours amounts to a docker pull every 3.6 minutes on average, for 6 consecutive hours. We considered this more than adequate for an individual developer, while other use cases involving high pull rates such as CI/CD pipelines or production container platforms can decrease their usage or subscribe to a paid plan. Over the course of a month, a single anonymous developer can (with the help of automation) make up to 12,000 docker pulls. By authenticating, that number increases to 24,000 docker pulls for free. As Docker container images vary in size from a few MB to well above 1 GB, focusing on pulls rather than size provides predictability to developers. They can pull images as they’re building applications, without worrying about their size but rather about their value. Based on these limits, we expected only 1.5% of daily unique IP addresses to be included — roughly 40,000 IPs in total, out of more than 2 million IPs that pull from Docker Hub every day. The other 98.5% of IPs using Docker Hub can carry on unaffected — or more likely, receive improved performance as the heaviest users decreased. As November 1st approached, we created a rollout plan that provided additional advance notice and decreased impact — even to developers we haven’t been able to reach through our emails or blog posts. We’ve put a few things in place to ease the transition for anyone affected: Providing a grace period after November 1 prior to full enforcement for all usage, so only a small fraction of the heaviest users were limited early in our rollout; Progressive rollout of enforcement across the affected population, to provide for additional opportunities for communications to reach Docker developers, and to minimize any inadvertent impact; and Temporary time windows of full enforcement, to raise awareness of unknown reliance upon Docker Hub and to reach developers without Docker IDs who we could not otherwise. On Wednesday, November 18, we expect to complete our progressive rollout to the new limits of 100 pulls and 200 pulls per 6-hour window for anonymous and authenticated free users, respectively. At that point, anyone who has not yet encountered the limits can reasonably conclude that their current usage of docker pulls is in that 98.5% of unaffected Docker Hub users. As we’ve progressed down this path toward creating a sustainable Docker, we’ve heard multiple times from developers that the temporary full-enforcement windows were valuable. They surfaced unknown reliance upon Docker Hub, as well as areas where our paying customers had not yet authenticated their usage. We’ve also worked with customers to identify problems that were unknowingly causing some of the massive downloads, like runaway processes downloading once every 3 seconds. Alongside this, we’ve created additional paid offerings to support large enterprises, ISVs, and service providers with needs like IP whitelisting or namespace whitelisting. We greatly appreciate the trust placed in Docker by the entire software community, and we look forward to helping you continue to build the great applications of the future! The post Rate Limiting by the Numbers appeared first on Docker Blog. View the full article

The gradual enforcement of the Docker Hub progressive rate limiting enforcement on container image pulls for anonymous and free users began Monday, November 2nd. The next three hour enforcement window on Wednesday, November 4th from 9am to 12 noon Pacific time. During this window, the eventual final limit of 100 container pull requests per six hours for unauthenticated users and 200 for free users with Docker IDs will be enforced. After that window, the limit will rise to 2,500 container pull requests per six hours. As we implement this policy, we are looking at the core technologies, platforms and tools used in app pipelines to ensure a transition that supports developers across their entire development lifecycle. We have been working with leading cloud platforms, CI/CD providers and other ISVs to ensure their customers and end users who use Docker have uninterrupted access to Docker Hub images. Among these partners are the major cloud hosting providers, CI/CD vendors such as CircleCI, and OSS entities such as Apache Software Foundation (ASF). You can find more information about programs on our Pricing Page as well as links to contact us for information about programs for ISVs and companies with more than 500 users. Besides the Apache Software Foundation, we are working with many Open Source Software projects from Cloud Foundry and Jenkins to many other open source projects of all sizes, so they can freely use Docker in their project development and distribution. Updates and details on the program are available in this blog from Docker’s Marina Kvitnitsky. We have assembled a page of information updates, as well as relevant resources to understand and manage the transition, at https://www.docker.com/increase-rate-limits. We’ve had a big week delivering new features and integrations for developers this week. Along with the changes outlined above, we also announced new vulnerability scan results incorporated into Docker Desktop, a faster, integrated path into production from Desktop into Microsoft Azure, and improved support for Docker Pro and Team subscribers. We are singularly focused on creating a sustainable, innovative company focused on the success of developers and development teams, both today and tomorrow, and we welcome your feedback. The post Updates on Hub Rate Limits, Partners and Customer Exemptions appeared first on Docker Blog. View the full article

On August 13th, we announced the implementation of rate limiting for Docker container pulls for some users. Beginning November 2, Docker will begin phasing in limits of Docker container pull requests for anonymous and free authenticated users. The limits will be gradually reduced over a number of weeks until the final levels (where anonymous users are limited to 100 container pulls per six hours and free users limited to 200 container pulls per six hours) are reached. All paid Docker accounts (Pro, Team or Legacy subscribers) are exempt from rate limiting. The rationale behind the phased implementation periods is to allow our anonymous and free tier users and integrators to see the places where anonymous CI/CD processes are pulling container images. This will allow Docker users to address the limitations in one of two ways: upgrade to an unlimited Docker Pro or Docker Team subscription, or adjust application pipelines to accommodate the container image request limits. After a lot of thought and discussion, we’ve decided on this gradual, phased increase over the upcoming weeks instead of an abrupt implementation of the policy. An up-do-date status update on rate limitations is available at https://www.docker.com/increase-rate-limits. Docker users can get an up-to-date view of their usage limits and updated status messages in the CLI, in terms of querying for current pulls used as well as header messages returned from Docker Hub. This blog post walks developers through how they can access their current account usage as well as understanding the header messages. And finally, Docker users can avoid rate limits completely by upgrading to a Pro or Team subscription: subscription details and upgrade information is available at https://docker.com/pricing. And open source projects can apply for a sponsored no-cost Docker account by filling out this application. The post What you need to know about upcoming Docker Hub rate limiting appeared first on Docker Blog. View the full article

Continuing with our move towards consumption-based limits, customers will see the new rate limits for Docker pulls of container images at each tier of Docker subscriptions starting from November 2, 2020. Anonymous free users will be limited to 100 pulls per six hours, and authenticated free users will be limited to 200 pulls per six hours. Docker Pro and Team subscribers can pull container images from Docker Hub without restriction as long as the quantities are not excessive or abusive. In this article, we’ll take a look at determining where you currently fall within the rate limiting policy using some command line tools. Determining your current rate limit Requests to Docker Hub now include rate limit information in the response headers for requests that count towards the limit. These are named as follows: RateLimit-Limit RateLimit-Remaining The RateLimit-Limit header contains the total number of pulls that can be performed within a six hour window. The RateLimit-Remaining header contains the number of pulls remaining for the six hour rolling window. Let’s take a look at these headers using the terminal. But before we can make a request to Docker Hub, we need to obtain a bearer token. We will then use this bearer token when we make requests to a specific image using curl. Anonymous Requests Let’s first take a look at finding our limit for anonymous requests. The following command makes a request to auth.docker.io for an authentication token for the ratelimitpreview/test image and saves that token in an environment variable named TOKEN. You’ll notice that we do not pass a username and password as we will for authenticated requests. $ TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token) Now that we have a TOKEN, we can decode it and take a look at what’s inside. We’ll use the jwt tool to do this. You can also paste your TOKEN into the online tool located on jwt.io $ jwt decode $TOKEN Token header ------------ { "typ": "JWT", "alg": "RS256" } Token claims ------------ { "access": [ { "actions": [ "pull" ], "name": "ratelimitpreview/test", "parameters": { "pull_limit": "100", "pull_limit_interval": "21600" }, "type": "repository" } ], ... } Under the Token header section, you see a pull_limit and a pull_limit_interval. These values are relative to you as an anonymous user and the image being requested. In the above example, we can see that the pull_limit is set to 100 and the pull_limit_interval is set to 21600 which is the number of seconds for the limit. Now make a request for the test image, ratelimitpreview/test, passing the TOKEN from above. NOTE: The following curl command emulates a real pull and therefore will count as a request. Please run this command with caution. $ curl -v -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1 | grep RateLimit < RateLimit-Limit: 100;w=21600 < RateLimit-Remaining: 96;w=21600 The output shows that our RateLimit-Limit is set to 100 pulls every six hours – as we saw in the output of the JWT. We can also see that the RateLimit-Remaining value tells us that we now have 96 remaining pulls for the six hour rolling window. If you were to perform this same curl command multiple times, you would observe the RateLimit-Remaining value decrease. Authenticated requests For authenticated requests, we need to update our token to be one that is authenticated. Make sure you replace username:password with your Docker ID and password in the command below. $ TOKEN=$(curl --user 'username:password' "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token) Below is the decoded token we just retrieved. $ jwt decode $TOKEN Token header ------------ { "typ": "JWT", "alg": "RS256" } Token claims ------------ { "access": [ { "actions": [ "pull" ], "name": "ratelimitpreview/test", "parameters": { "pull_limit": "200", "pull_limit_interval": "21600" }, "type": "repository" } ], ... } The authenticated JWT contains the same fields as the anonymous JWT but now the pull_limit value is set to 200 which is the limit for authenticated free users. Let’s make a request for the ratelimitpreview/test image using our authenticated token. NOTE: The following curl command emulates a real pull and therefore will count as a request. Please run this command with caution. $ curl -v -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1 | grep RateLimit < RateLimit-Limit: 200;w=21600 < RateLimit-Remaining: 176;w=21600 You can see that our RateLimit-Limit value has risen to 200 per six hours and our remaining pulls are at 176 for the next six hours. Just like with an anonymous request, If you were to perform this same curl command multiple times, you would observe the RateLimit-Remaining value decrease. Error messages When you have reached your Docker pull rate limit, the resulting response will have a http status code of 429 and include the below message. HTTP/1.1 429 Too Many Requests Cache-Control: no-cache Connection: close Content-Type: application/json Retry-After: 21600 { "errors": [{ "code": "DENIED", "message": "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" }] } Conclusion In this article we took a look at determining the number of image pulls allowed based on whether we are an authenticated user or anonymous user. Anonymous free users will be limited to 100 pulls per six hours, and authenticated free users will be limited to 200 pulls per six hours. If you would like to avoid rate limits completely, you can purchase or upgrade to a Pro or Team subscription: subscription details and upgrade information is available at https://docker.com/pricing. For more information and common questions, please read our docs page and FAQ. And as always, please feel free to reach out to us on Twitter (@docker) or to me directly (@pmckee). To get started using Docker, sign up for a free Docker account and take a look at our getting started guide. The post Checking Your Current Docker Pull Rate Limits and Status appeared first on Docker Blog. View the full article