Sharding Dominant Domains

May 12, 2009 7:05 pm | 13 Comments

This post is based on a chapter from Even Faster Web Sites, the follow-up to High Performance Web Sites. Posts in this series include: chapters and contributing authors, Splitting the Initial Payload, Loading Scripts Without Blocking, Coupling Asynchronous Scripts, Positioning Inline Scripts, Sharding Dominant Domains, Flushing the Document Early, Using Iframes Sparingly, and Simplifying CSS Selectors.

 

Rule 9 from High Performance Web Sites says reducing DNS lookups makes web pages faster. This is true, but in some situations, it’s worthwhile to take a bunch of resources that are being downloaded on a single domain and split them across multiple domains. I call this domain sharding. Doing this allows more resources to be downloaded in parallel, reducing the overall page load time.

To determine if domain sharding makes sense, you have to see if the page has a dominant domain being used for downloading resources. The following HTTP waterfall chart, for Yahoo.com, is indicative of a site being slowed down by a dominant domain. This waterfall chart shows the page being downloaded in Internet Explorer 7, which only allows two parallel downloads per hostname. The vertical lines show that typically there are only two simultaneous downloads at any given time. Looking at the resource URLs, we see that almost all of them are served from “l.yimg.com.” Sharding these resources across two domains, such as “l1.yimg.com” and “l2.yimg.com”, would cause the resources to download in about half the time.

Most of the U.S. top ten web sites do domain sharding. YouTube uses i1.ytimg.com, i2.ytimg.com, i3.ytimg.com, and i4.ytimg.com. Live Search uses ts1.images.live.com, ts2.images.live.com, ts3.images.live.com, and ts4.images.live.com. Both of these sites are sharding across four domains. What’s the optimal number? Yahoo! released a study that recommends sharding across at least two, but no more than four, domains. Above four, performance actually degrades.

Not all browsers are restricted to just two parallel downloads per hostname. Opera 9+ and Safari 3+ do four downloads per hostname. Internet Explorer 8, Firefox 3, and Chrome 1+ do six downloads per hostname. Sharding across two domains is a good compromise that improves performance in all browsers.

13 Responses to Sharding Dominant Domains

  1. Great article!

    I think that in some cases could be possible to check the browser type and define dynamically the number of hostnames. The script has a time cost, but it reduce the DNS lookup time for newest browsers.

  2. Good article – on thing to remember, however, is NOT to implement this technique for SSL (https) unless you’re able to secure ALL the ‘sharded’ subdomains. There are certificates that can handle multiple subdomains, but most don’t. And if you don’t have one, your user will end up with all sorts of warning messages about mixed resources and a broken ‘padlock’ symbol.

  3. Also, for SSL the cost of a new connection is higher, so having to much domain / to much connections could be negative far more quicker than with non-ssl domains.

    With new browers that have 6 threads per domain, it’s probably safe to stay with only one domain on SSL (and never more than two).

  4. I’ve been doing quite a bit of research regarding domain shredding. Is there a tool or method i could use ( short of proto-typing) to test using 2 vs. 4 host manes for static images across my website? My testing is limited to only http requests as i would prefer to leave all of my assets for ssl under a single hostname

  5. “Not all browsers are restricted to just two parallel downloads per hostname.”

    How sure are you about that? My impression was that installing IE8 raises the simultaneous connections limit by changing a system-wide setting.

    Have you tested Firefox or Chrome on a fresh install of Windows that’s never seen IE8?

  6. Will domain sharding work if I serve images from different TCP ports?

  7. @Guillermo: The best test is to do a test on your live traffic with JavaScript timing the page (see the Jiffy Web project).

    @Matt: You can look at the source code and see that these are the correct settings.

    @Michael: I have not tested ports. It would be great if you created a test page with a bunch of images from different ports, and then watched them in Firebug Net Panel to see if the number of parallel connections exceeds the max. Please comment back here if you do so.

  8. This article was written two years ago when IE7 may have been the dominant browser. Now that the proportions of browser usage has changed, do you think we need to revisit the optimal number of domains? And revise it upwards?

  9. @Brendan: Since newer browsers open more connections per domain, it’s probably better to revise the number downwards. I think 2 is a good compromize, but that’s just a hunch. It’d be great if some production property ran a test to determine the optimal number.

  10. Thanks Steve. I’ll see if we can do some of our own tests on this and feed the results back into the community.

  11. Steve, do you consider domain sharding still a good practice considering higher parallel download limits in modern browsers?

  12. @Daniel: Definitely! If you have 10 or more resources on a single domain I recommend sharding across 2 domains.

  13. Hi Steve,

    I really enjoyed and learned a lot from both of your books and first I would like to say thx! Now, I have a dillema which I’ve posted on Stack Overflow: http://stackoverflow.com/questions/14808953/resizing-and-optimizing-images-and-loading-them-into-cdn-vs-normaly-linking-them so if you would please be so kind to take a look at it?