UA switching: be careful
At least once a day I’m in a conversation, email thread, or twitter exchange about monitoring websites. Lately this has focused on mobile. Tools like WebPagetest make it easier to monitor websites from the perspective of a desktop browser, but doing this from the perspective of a mobile device is still a significant challenge.
This issue is a current topic of discussion around HTTP Archive Mobile. Blaze.io supports the project through its Mobitest framework: every two weeks I submit 1000 URLs to their framework which downloads each URL 3 times on a real iPhone. I love that the resultant waterfall chart and screenshots are gathered from a real phone. But our next step is to scale this up to 100K and then 1M URLs. It’s going to be hard to scale up to this using real phones due to cost and maintenance issues.
Another alternative is to use an emulator. The timings won’t be identical to the actual mobile device, but HTTP Archive Mobile is more focused on HTTP headers, size and number of responses, etc. These structural aspects of the page should be identical between the actual phone and its emulator. We’ll soon do side-by-side tests to confirm this.
But this post is about a third alternative: UA switching. Changing the User-Agent string of a desktop browser to mimic a mobile browser and using that to gather the data could be accomplished this afternoon. The issue is that the results might differ from what is seen on an actual phone. Websites that don’t do anything special for mobile would probably be similar enough. And websites that look at the UA string on the serverside to determine how to alter the page would also be okay. But websites that adapt the page based on browser feature detection on the clientside, e.g. responsive web design, would differ dramatically.
When asked for an example of such a site I recalled seeing Scott Jehl at Breaking Development Conference talking about the recent launch of Boston Globe using responsive web design. It’s an amazing feat of engineering. Its adaptability from a single code base across so many devices is beautiful to watch in this promo video.
Because the Boston Globe uses responsive web design, the UI varies depending on the browser – not the UA string. Here’s a screenshot from my iPhone. The content has been modified to fit on the iPhone’s smaller screen.
Figure 1. iPhone browser
Figure 2 shows the Boston Globe as rendered inside Chrome on my laptop. Since the screen is much bigger the content is laid out quite differently compared to the iPhone. We see three columns instead of one, a wider horizontal navigation bar, etc.
Figure 2. Chrome
Figure 3 is also from Chrome, but in this case I changed the User-Agent string to mimic an iPhone:
Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_3_1 like Mac OS X; en-us) AppleWebKit/533.17.9 (KHTML, like Gecko) Version/5.0.2 Mobile/8J2 Safari/6533.18.5Even though the UA string says “iPhone”, the page is laid out exactly as it is for normal Chrome. (I confirmed the UA string by inspecting HTTP headers.)
Figure 3. Chrome with iPhone User-Agent string
Responsive web design is fairly new. There are still a number of websites that modify the HTML layout based on serverside UA detection. In fact, while generating the screenshot for Figure 3 I tried several other websites and most of them returned a layout customized for iPhone.
If you’re working on a framework to monitor mobile websites, be careful about taking the UA switching approach. If the websites you’re monitoring do serverside UA detection, you’ll probably be okay. But if the web app is based on clientside feature detection, the results you receive from UA switching won’t match what is seen on real mobile devices. As the adoption of responsive web design increases so will the number of websites that fall into this gap of mismeasurement. Real devices or emulators are a safer bet.
App cache & localStorage survey
In preparation for my talk at HTML5 Dev Conf I surveyed the Alexa US Top 10 websites to see which ones use app cache and localStorage. I mostly focus on mobile these days so it’s natural to think I ran these tests from a mobile browser, which I did. But I also tested with a desktop browser.
Some people might ask, Why a desktop browser?
To which I would reply, Why not a desktop browser?
I blogged previously about how Google and Bing use localStorage on mobile devices. It’s a powerful performance technique. They break up their JavaScript and CSS into smaller blocks and save them in localStorage. Simultaneously they set a cookie so that the server knows not to send that payload on subsequent searches, a savings of 150-170 kB before gzipping.
In the comments on that post Simon asked:
I’m curious why the techniques with LocalStorage are used for the mobile versions of the search sites but not for the standard desktop versions – I would think that this technique would work well [on] all clients, no?
I agree – this and other HTML5 web storage techniques make sense for the desktop, too. There are some reasons why we’re seeing these used first on mobile:
- Mobile latencies are higher and connection speeds are lower, so clientside caching is more important on mobile.
- Mobile disk cache sizes are smaller than desktop sizes, so a better alternative is needed for mobile.
- There are still desktop browsers with significant market share that are missing many HTML5 capabilities, whereas mobile browsers have more support for HTML5.
Even though the motivation for app cache and localStorage are stronger for mobile, they should also be used when the website is accessed from a desktop browser. I did a quick survey to see which of the top 10 websites were using app cache or localStorage on either mobile or desktop. Here are the results:
| Website | Mobile | Desktop | ||
|---|---|---|---|---|
| app cache | localStorage | app cache | localStorage | |
| Google Search | no | yes | no | yes [1] |
| Bing | no | yes | no | no |
| no | yes | no | no | |
| Yahoo! Front Page | no | yes [2] | no | no |
| YouTube | no | yes | no | no |
| Amazon | no | no | no | yes [3] |
| yes | yes | no | yes | |
| no | no | no | no | |
| eBay | no | no | no | no |
| MSN.com | no | no | no | no |
[2] Yahoo! Front Page only stores two numbers.
[3] Amazon on the desktop only stores a ~30 character string.
Ignoring the small uses of localStorage, 5 of these top 10 websites use localStorage on mobile, but only 2 out of 10 use localStorage (or sessionStorage) on desktop. None of them use app cache, either on mobile or desktop except for Twitter on mobile. I’m surprised no one is using app cache. It’s not appropriate for all applications, such as search, but I would enjoy catching up on Twitter, Facebook, and LinkedIn on the plane – potentially from my laptop in addition to my mobile device. App cache not only brings this offline capability, but provides better caching than the browser’s disk cache.
I’ll repeat this survey in a few months to track the progress. I expect we’ll see the use of localStorage and app cache increase, and for desktop to catch up to mobile.
Making a mobile connection
I just returned from Breaking Development Conference, an amazing gathering of many of the brightest minds in mobile web development. On the flight home I watched the video ($$) and slides from Rajiv Vijayakumar’s talk on Understanding Mobile Web Browser Performance at Velocity 2011. Rajiv works at Qualcomm where his team has done extensive performance analysis of the Android browser. Some of their findings include:
- Android 2.2 has a max of only 4 HTTP connections which limits parallel downloads. (This was increased to 8 in Android 2.3 and 35 in Android 3.1 according to Browserscope.)
- It supports pipelining for reduced HTTP overhead.
- Android’s cache eviction is based on expiration date. This is a motivation for setting expiration dates 10+ years in the future.
- Android closes TCP sockets after 6 seconds of inactivity.
This last bullet leads to an interesting discussion about the tradeoffs between power consumption and web performance.
Radio link power consumption
3G devices surfing the Web (do people still say “surfing”?) establish a radio link to the carrier’s cell tower. Establishing and maintaining the radio link consumes battery power. The following graph from Rajiv’s slides shows power consumption for an Android phone while loading a web page. It rises from a baseline of 200 mA to ~400 mA as the radio link is initialized. After the page is loaded the phone drops to 300 mA while the network is inactive. After 10 seconds of inactivity, the radio link reaches an idle state and power consumption returns to the 200 mA baseline level.
The takeaway from this graph is that closing the radio link sooner consumes less battery power. This graph shows that the radio link continues to consume battery power until 10 seconds of inactivity have passed. The 10 second radio link timer begins once the web page has loaded. But there’s also a 6 second countdown after which Android closes the TCP connection by sending a FIN packet. When Android sends the FIN packet the radio link timer resets and continues to consume battery power for another 10 seconds, resulting in a total of 16 seconds of higher battery consumption.
One of the optimizations Rajiv’s team made for the Android browser running on Qualcomm chipsets is to close the TCP connections after the page is done loading. By sending the FIN packet immediately, the radio link is closed after 10 seconds (instead of 16 seconds) resulting in longer battery life. Yay for battery life! But how does this affect the speed of web pages?
Radio link promotion & demotion
The problem with aggressively closing the phone’s radio link is that it takes 1-2 seconds to reconnect to the cell tower. The way the radio link ramps up and then drops back down is shown in the following figure from an AT&T Labs Research paper. When a web page is actively loading, the radio link is at max power consumption and bandwidth. After the radio link is idle for 5 seconds, it drops to a state of half power consumption and significantly lower bandwidth. After another 12 seconds of inactivity it drops to the idle state. From the idle state it takes ~2 seconds to reach full power and bandwidth.
These inactivity timer values (5 seconds & 12 seconds in this example) are sent to the device by the cell tower and thus vary from carrier to carrier. The “state machine” for promoting and demoting the radio link, however, is defined by the Radio Resource Control protocol with the timer values left to the carrier to determine. (The protocol dubs these timer values “T1″, “T2″, and “T3″. I just find that funny.) If the radio link is idle when you request a web page, you have to wait ~2 seconds before that HTTP request can be sent. Clearly, the inactivity timer values chosen by the carrier can have a dramatic impact on mobile web performance.
What’s your carrier’s state machine?
There’s an obvious balance, sort of a yin and yang, between power consumption and web performance for 3G mobile devices. If a carrier’s inactivity timer values are set too short, users have better battery life but are more likely to encounter a ~2 second delay when requesting a web page. If the carrier’s inactivity timer values are set too long, users might have a faster web experience but shorter battery life.
This made me wonder what inactivity timer values popular carriers used. To measure this I created the Mobile State Machine Test Page. It loads a 1 kB image repeatedly with increasing intervals between requests: 2, 4, 6, 11, 13, 16, and 20 seconds. The image’s onload event is used to measure the load time of the image. For each interval the image is requested three times, and the median load time is the one chosen. The flow is as follows:
- choose the next interval
i(e.g, “2″ seconds) - wait
iseconds - measure
t_start - request the image
- measure
t_endusing the image’s onload - record
t_end - t_startas the image load time - repeat steps 2-6 two more times and choose the median as the image load time for interval
i - goto step 1 until all intervals have been tested
The image should take about the same time to load on every request for a given phone and carrier. Increasing the interval between requests is intended to see if the inactivity timer changes the state of the radio link. By watching for a 1-2 second increase in image load time we can reverse engineer the inactivity timer values for a given carrier.
I tweeted the test URL about 10 days ago. Since then people have run the test 460+ times across 71 carriers. I wrote some code that maps IP addresses to known carrier hostnames so am confident about 26 of the carriers; the others are self-reported. (Max Firtman recommended werwar for better IP-to-carrier mapping.) I’d love to keep gathering data so:
I encourage you to run the test!
The tabular results show that there is a step in image load times as the interval increases. (The load time value shown in the table is the median collected across all tests for that carrier. The number of data points is shown in the rightmost column.) I generated the chart below from a snapshot of the data from Sept 12.
The arrows indicate a stepped increase in image load time that could be associated with the inactivity timer for that carrier. The most pronounced one is for AT&T (blue) and it occurs at the 5 second mark. T-Mobile (yellow) appears to have an inactivity timer around 3 seconds. Vodafone is much larger at 15 seconds. Sprint and AT&T Verizon have similar profiles but the step is less pronounced.
There are many caveats about this study:
- This is a small sample size.
- The inactivity timer could be affected by other apps on the phone doing network activity in the background. I asked people to close all apps, but there’s no way to verify they did that.
- A given carrier might have different kinds of networks (3G, 4G, etc.). Similarly, they might have different inactivity timer values in different regions. All of those different conditions would be lumped together under the single carrier name.
What’s the point?
Hats off to Rajiv’s team at Qualcomm for digging into Android browser performance. They don’t even own the browser but have invested heavily in improving the browser user experience. In addition to closing TCP connections once the page is loaded, they increased the maximum number of HTTP connections, improved browser caching, and more.
I want to encourage this holistic approach to mobile performance and will write about that in more depth soon. This post is pretty technical, but it’s important that mobile web developers have greater insight into the parts of the mobile experience that go beyond HTML and JavaScript – namely the device, carrier network, and mobile browser.
For example, in light of this information about inactivity timers, mobile web developers might choose to do a 1 pixel image request at a set interval that keeps the radio link at full bandwidth. This would shorten battery life, so an optimization would be to only do a few pings after which it’s assumed the user is no longer surfing. Another downside is that doing this would use more dedicated channels at the cell tower, worsening everyone’s experience.
The right answer is to determine what the tradeoffs are. What is the optimal value for these inactivity timers? Is there a sweet spot that improves web performance with little or no impact on battery life? How did the carriers determine the current inactivity timer values? Was it based on improving the user’s web experience? I would bet not, but am hopeful that a more holistic view to mobile performance is coming soon.