Domain Sharding revisited

September 5, 2013 10:41 am | 13 Comments

With the adoption of SPDY and progress on HTTP 2.0, I hear some people referring to domain sharding as a performance anti-pattern. I disagree. Sharding resources across multiple domains is a major performance win for many websites. However, there is room for debate. Domain sharding isn’t appropriate for everyone. It may hurt performance if done incorrectly. And it’s utility might be short-lived. Is it worth sharding domains? Let’s take a look.

Compelling Data

The HTTP Archive has a field call “maxDomainReqs”. The explanation requires a few sentences: Websites request resources from various domains. The average website today accesses 16 different domains as shown in the chart below. That number has risen from 12.5 a year ago. That’s not surprising given the rise in third party content (ads, widgets, analytics).

The HTTP Archive counts the number of requests made on each domain. The domain with the most requests is the “max domain” and the number of requests on that domain is the “maxDomainReqs”. The average maxDomainReqs value has risen from 47 to 50 over the past year. That’s not a huge increase, but the fact that the average number of requests on one domain is so high is startling.

50 is the average maxDomainReqs across the world’s top 300K URLs. But averages don’t tell the whole story. Using the HTTP Archive data in BigQuery and bigqueri.es, both created by Ilya Grigorik, it’s easy to find percentile values for maxDomainReqs: the 50th percentile is 39, the 90th percentile is 97, and the 95th percentile is 127 requests on a single domain.

This data shows that a majority of websites have 39 or more resources being downloaded from a single domain. Most browsers do six requests per hostname. If we evenly distribute these 39 requests across the connections, each connection must do 6+ sequential requests. Response times per request vary widely, but I use 500 ms as an optimistic estimate. If we use 500 ms as the typical responsive time, this introduces a 3000 ms long pole in the response time tent. In reality, requests are assigned to whatever connection is available, and 500 ms might not be the typical response time for your requests. But given the six-connections-per-hostname limit, 39 requests on one domain is a lot.

Wrong sharding

There are costs to domain sharding. You’ll have to modify your website to actually do the sharding. This is likely a one time cost; the infrastructure only has to be setup once. In terms of performance the biggest cost is the extra DNS lookup for each new domain. Another performance cost is the overhead of establishing each TCP connection and ramping up its congestion window size.

Despite these costs, domain sharding has great benefit for websites that need it and do it correctly. That first part is important – it doesn’t make sense to do domain sharding if your website has a low “maxDomainReqs” value. For example, if the maximum number of resources downloaded on a single domain is 6, then you shouldn’t deploy domain sharding. With only 6 requests on a single domain, most browsers are able to download all 6 in parallel. On the other hand, if you have 39 requests on a single domain, sharding is probably a good choice. So where’s the cutoff between 6 and 39? I don’t have data to answer this, but I would say 20 is a good cutoff. Other aspects of the page affect this decision. For example, if your page has a lot of other requests, then those 20 resources might not be the long pole in the tent.

The success of domain sharding can be mitigated if it’s done incorrectly. It’s important to keep these guidelines in mind.

  • It’s best to shard across only two domains. You can test larger values, but previous tests show two to be the optimal choice.
  • Make sure that the sharding logic is consistent for each resource. You don’t want a single resource, say main.js, to flip-flop between domain1 and domain2.
  • You don’t need to setup different servers for each domain – just create CNAMEs. The browser doesn’t care about the final IP address – it only cares that the hostnames are different.

These and other issues are explained in more detail in Chapter 11 of Even Faster Web Sites.

Short term hack?

Perhaps the strongest argument against domain sharding is that it’s unnecessary in the world of SPDY (as well as HTTP 2.0). In fact, domain sharding probably hurts performance under SPDY. SPDY supports concurrent requests (send all the request headers early) as well as request prioritization. Sharding across multiple domains diminishes these benefits. SPDY is supported by Chrome, Firefox, Opera, and IE 11. If your traffic is dominated by those browsers, you might want to skip domain sharding. On the other hand, IE 6&7 are still somewhat popular and only support 2 connections per hostname, so domain sharding is an even bigger win in those browsers.

A middle ground is to alter domain sharding depending on the client: 1 domain for browsers that support SPDY, 2 domains for non-SPDY modern browsers, 3-4 domains for IE 6-7. This makes domain sharding harder to deploy. It also lowers the cache hit rate on intermediate proxies.

There’s no need for domain sharding in the world of HTTP 2.0 across all popular browsers. Until then, there’s no silver bullet answer. But if you’re one of the websites with 39+ resources on a single hostname, domain sharding is worth exploring.

13 Responses to Domain Sharding revisited

  1. Will Chan talked about the network congestion produced by too many shards here – https://insouciant.org/tech/network-congestion-and-web-browsing/#Google_Kitten_Search

    The post has a really interesting line that Chrome is already limiting parallelism for some types of resources “Indeed, we’ve experimented with doing so and found compelling data to support limiting the number of concurrent image requests to 10, which should roll out to stable channel in the upcoming Chrome 27 release and dramatically mitigate congestion related issues.”

  2. Repost due to some bad edits:

    Hi Steve,

    Thanks for the interesting article!

    According to what I know, the HTTP standard recommended a limit of 2 connections per host – a limit no modern browser does comply to, anymore. browsers did select to use no more 6-8 parallel connections per host to balance between Parallelism and Congestion. As far as I read, this number is based on exhaustive testing.

    If browser makers would believe 10 or even 20 parallel connections per host is preferable, I believe they would have just done that.

    I understand the browser doesn’t know how many requests are expected per domain, while the site’s developer does. But… if the median site has 39 requests + I make a guess (from sites I know) almost all of the other domain require less than 6 request – it sound logical that increasing the browser amount of connections per domain would provide a benefit here. Yeah… this is just guessing.

    Is there any data available about the effect of “max domain sharding” on different web sites?

    Thanks in advance,
    Lior

  3. Andy: Thanks for the pointer to Will’s (awesome) article. Note that this only applies to images, which are given a lower download priority than scripts & stylesheets. Thus, a website might still benefit from domain sharding wrt non-image assets and to fill that queue of 10 image requests when it’s time.

    Lior: It is possible that browsers may revisit their max-connections-per-domain settings. Opera dropped there’s from 8 connections in Opera 8 down to 6 connections starting in Opera 11. It looks like IE 10 raised its value from 6 to 8 connections. Domain sharding is a way for the web developer to work with the browser to decide what’s best. Both are important. I’m not aware of any public data on this topic.

  4. Nice post Steve – a couple additional thoughts:

    1] the size of the resources being transferred should be a primary factor in deciding whether or not to shard aggressively. Small resources should be sharded more aggressively. Sharding large hi-res images is a great way to create self induced packet loss and retransmissions, but not sharding buttons and icons is a great way to leave unused bandwidth on the table.

    2] Both chrome and firefox will automatically unshard transparently for you when using spdy and both of the sharded hosts are at the same IP address and covered under one SSL cert (e.g. *.example.com). That’s pretty cool because you can automatically shard for HTTP/1 but not for HTTP/2 (spdy) without doing UA sniffing. Cause nothing ever goes wrong with UA sniffing :)

    3] At high levels of sharding-induced parallelism firefox will actually engage in pacing requests a few milliseconds apart.. they are staggered but by a factor much smaller than a RTT. This really helps with the packet loss problem.

  5. Patrick: Great to hear from you! I didn’t know about Chrome & Firefox UNsharding – that’s very cool. And Firefox’s staggered multiple requests is great. Were those changes motivated by sharding specifically, or do those issues arise on popular websites even without sharding?

  6. Thank you for the interesting article, Steve, but I disagree with your view of SPDY.

    SPDY requires a https connection which means that all those people using Internet Explorers will have see that the website is slower. So, using SPDY might bring a little improvement for one group (FF, Chrome), but it would bring make the website slower for IE users.

    According to this current report IE has a worldwide market share of 56 % (with IE8 as most important browser):
    http://thenextweb.com/insider/2013/08/01/ie9-falls-below-10-market-share-firefox-hits-50-month-low-and-chrome-again-gains-the-most/

    On our German-speaking financial news website we have these numbers:

    1. Internet Explorer 33,98 %
    2. Firefox 30,80 %
    3. Safari 15,36 %
    4. Chrome 10,42 %
    5. Android Browser 6,52 %
    6. Opera 1,28 %

  7. Markus:
    In what way it is slower? If it is badly configured server then, not surprising.
    (http://blogs.msdn.com/b/ieinternals/archive/2011/03/26/https-and-connection-close-is-your-apache-modssl-server-configuration-set-to-slow.aspx)

  8. Klimax, no, the problem would not be the server, but the browsers.
    Maybe half of the users have a browser which supports (already) SPDY. For them it could bring a little advantage.

    On the other handside the website would become slower for the others as users as building up a https connection takes extra time. When you look e.g. at tests with Chrome on WPT, you can see in the column ‘SSL Negotiation’ the additional time to build up the https connection:
    http://www.webpagetest.org/result/130912_CP_W1J/1/details/

    https://apis.google.com … 55 ms
    https://twitter.com … 171 ms
    https://ssl.gstatic.com … 65 ms
    https://googleads.g.doubleclick.net … 69 ms

    So, SPDY might bring an advantage for half of the users if you then download many files through this connection, but in case you load only 1 or 2 files there might be no advantage.
    And there should definitely be a disadvantage for all other people, e.g. those using browsers like the popular IE8 …

  9. Nice article Steve,

    Assume if we are having an index.html page, If that page requires like 10 external Js and Css files, Can we enable something in the server so that those files will be pushed for us from the server rather than we requesting the resources.

    This will reduce the number of HTTP requests that we create from client right. Is it possible Steve?

  10. Vignesh: You can use LINK REL PREFETCH if you’re navigating from another page you own to index.html. If index.html is the first page the user is navigating to, then you can use LINK REL SUBRESOURCE but it’s not supported in many browsers. If you’re mostly thinking about scripts & stylesheets then you can put those at the top of the HEAD (use async) and the browser will start fetching them immediately. Stay tuned for some future posts about prefetching.

  11. Vignesh: one other neat thing – I did some testing and found that simply using link rel=”prefetch subresource” not only gave the expected super-fast performance in Chrome and improved the situation in Firefox and IE10, so you can get a nice win simply by adding one tag per script or stylesheet:

    http://chris.improbable.org/experiments/browser/prefetch-timing.html

  12. Steve : Nicely done.. Thanks a lot for the answer. That made a tremendous improvement in performance.

    Chris : Cool thing Chris. That was amazing :)

    I am digging in to performance more.. these things are getting interesting a lot.

  13. @Markus | 12-Sep-13 at 1:17 pm:
    Not sure.
    Fresh set of results:
    Chrome:
    http://www.webpagetest.org/result/131008_2W_ZBB/1/details/
    IE9
    http://www.webpagetest.org/result/131008_X0_ZBR/1/details/
    IE10
    http://www.webpagetest.org/result/131008_PN_ZC2/1/details/
    IE 8:
    http://www.webpagetest.org/result/131008_0V_12FP/1/details/

    It seems Chrome is between IE9/10 and IE8 respectively Windows platforms. (IE uses a lot of existing components in Windows ,thus its performance is more dependant on them)

    It seems, that you need to distinguish between IE8 (possibly on older platform) and IE9+ on newer.

    BTW: I took exception mostly to your assertion:
    “SPDY requires a https connection which means that all those people using Internet Explorers will have see that the website is slower. So, using SPDY might bring a little improvement for one group (FF, Chrome), but it would bring make the website slower for IE users.” As mine links show, it is not clear cut at all.

    Note: IE8 doesn’t support SPDY at all, so no idea why that would matter or how would SPDY affect them. (Also for some reason SSL handshake with https://apis.google.com is much slower then other https cases)

    P.S.: Sorry for late reply, forgot I posted here…