Unexpected prerender in Chrome

April 30, 2014 2:55 pm | 12 Comments

Over the last week I’ve been investigating the cacheability of resources from Shopify. I would visit the page every few hours and track the number of 200 OK versus 304 Not Modified responses. To my surprise, Chrome’s Network tab indicated that almost all the responses were “from cache”.

This didn’t make sense. In many cases the resource URLs changed between test loads. How could a never-before-seen URL be “from cache”? In cases where the URL was the same, I noticed that the Date response header had changed from the previous test but Chrome still marked it “from cache”. How could the Date change without a 200 response status code?

I started thinking about my “Prebrowsing” work (blog post, slides, video). In my findings I talk about how browsers, especially Chrome, are doing more work in anticipation of what the user needs next. This proactive work includes doing DNS lookups, establishing TCP connections, downloading resources, and even prerendering entire pages.

Was it possible that Chrome was prerendering the entire page?

I started by looking at chrome://predictors. Given characters entered into Omnibox (the location field), this shows which URL you’re predicted to go to. In my tests, I had always typed the URL into the location bar, so the predictions for “shopify” could affect Chrome’s behavior in my tests. Here’s what I found in chrome://predictors:

chrome-predictors-shopify

Chrome predicted that if I entered “www.s” into the Omnibox I would end up going to “http://www.shopify.com/” with confidence 1.0 (as shown in the rightmost column). In fact, just typing “ww” had a 0.9 confidence of ending up on Shopify. In other words, Chrome had developed a deep history mapping my Omnibox keystrokes to the Shopify website, as indicated by rows colored green.

From my Prebrowsing research I knew that if the chrome://predictors confidence was high enough, Chrome would pre-resolve DNS and pre-connect TCP. Perhaps it was possible that Chrome was also proactively sending HTTP requests before they were actually needed. To answer this I opened Chrome’s Net panel and typed “www.s” in the Omnibox but never hit return. Instead, I just sat there and waited 10 seconds. But nothing showed up in Chrome’s Net panel:

shopify-omnibox

Suspecting that these background requests might not show up in Net panel, I fired up tcpdump and repeated the test – again only typing “www.s” and NOT hitting return. I uploaded the pcap file to CloudShark and saw 86 HTTP requests!

cloudshark-shopify

I looked at individual requests and saw that they were new URLs that had never been seen before but were in the HTML document. This confirmed that Chrome was prerendering the HTML document (as opposed to prefetching individual resources based on prior history). I was surprised that no one had discovered this before, so I went back to High Performance Networking in Google Chrome by Ilya Grigorik and scanned the Omnibox section:

the yellow and green colors for the likely candidates are also important signals for the ResourceDispatcher! If we have a likely candidate (yellow), Chrome may trigger a DNS pre-fetch for the target host. If we have a high confidence candidate (green), then Chrome may also trigger a TCP pre-connect once the hostname has been resolved. And finally, if both complete while the user is still deliberating, then Chrome may even pre-render the entire page in a hidden tab.

Takeaways

What started off as a quick performance analysis turned into a multi-day puzzler. The puzzle’s solution yields a few takeaways:

  • Remember that Chrome may do DNS prefetch, TCP pre-connect, and even prerender the entire page based on the confidences in chrome://predictors.
  • Not all HTTP requests related to a page or tab are shown in Chrome’s Net panel. I’d like to see this fixed, perhaps with an option to show these behind-the-scenes requests.
  • Ilya knows everything. Re-read his posts, articles, and book before running multi-day experiments.

 

12 Responses to Unexpected prerender in Chrome

  1. From a server perspective is it possible to detect prefetches from Firefox and block them. Is that possible with Chrome? Or is there nothing special about the headers?

  2. Firefox sends the X-moz: prefetch request header (see here).

    Chrome does not send a request header. They suggest detecting prerenders on the clientside using the Page Visibility API (see here).

  3. How does this affect caching? Presumably, Chrome is sufficiently clever to ignore any page it is in the process of grabbing when I’m trying to clear the cache.

    I do occasionally have problem clearing cache and was wondering if this is related.

  4. I haven’t tried it but I think this flag (or one of the similar ones) might disable that behaviour.

    http://peter.sh/experiments/chromium-command-line-switches/#prefetch-search-results

  5. chrome://net-internals/#prerender will show prerendering actions as well.

    #4: You can disable prerender (and prefetch) in Chrome by disabling “Predict network actions to improve page load performance”

  6. If you use boomerang to measure RUM data, then it will tell you if the page has been pre-rendered as well as the difference between pre-render time and actual “request intent” time to load time.

  7. I did not realize that Chrome did this. Very neat.

  8. chrome://net-internals in general can show you any request made by the browser. It is just a bit hard to figure out (if at all, I never tried) that a request comes from a certain tab (or a background service), but everything should be there, under Events.

  9. Wow, 86 pre browsing requests is crazy. I use my chromebook through my iPhone personal hotspot thingy quite often, I bet this is affecting my data charges.

    Wonder if there’s a way to turn that off?

  10. @Mika Indeed, it can waste your traffic. You can disable it in Settings > Privacy > Predict network actions to improve page load performance.

    Another way to speed up loading web pages is to use Data Compression Proxy, even on desktop. You can find it in Chrome Web Store.

  11. This is similar to the impact of robots on server stacks.

    As Orasi (my parent consulting company) typically represents the performance engineering of the server stack this is good information to consider.

    Have you investigated other browsers too?
    How does this show up in google analytics data?

    In other words did this change in Chrome occur recently? can we see if page visit count has increased as a result which might lead site owners to believe they have seen an uptick in site stickiness?

  12. 3. bob: I don’t think this prerender behavior is affecting your ability to clear the cache. It’s extremely unlikely that the cache clear and prerender would happen at the exact same time. Try clearing the cache and then looking in the cache. If it’s not clear then the issue lies elsewhere.

    9. Mika: I expect browsers would be less aggressive about prerender on mobile devices for the reason you mention: data plans. Would be a good test to run. Jerzy (comment #10) also mentions a way to disable prerender.

    11. dave: Please see my Prebrowsing work (cited in this post) for more info about prerender support in other browsers, etc. As mentioned in comment #2, Google recommends analytics companies use the Page Visibility API to avoid sending impression counters until a prerendered page is actually visible to the user. It appears that Google Analytics does that. (Note that there are NOT any requests to google-analytics.com in the list of 86 requests above. Also, see this line of code in ga.js: if("prerender"==J.visibilityState)