A few years ago, I set a goal to foster the creation of an Internet Performance Archive. The idea is similar to the Internet Archive. But whereas the Internet Archive’s Wayback Machine provides a history of the Web’s state of HTML, IPA would provide a history of the Web’s state of performance: total bytes downloaded, number of images|scripts|stylesheets, use of compression, etc. as well as results from performance analysis tools like YSlow and Page Speed. Early versions of this idea can be seen in Show Slow and Cesium.
I realized that a key ingredient to this idea was a way to capture the page load experience, basically, a way to save what you see in a packet sniffer. Wayback Machine archives HTML, but it doesn’t capture critical parts of the page load experience like the page’s resources (scripts, stylesheets, etc.) and their HTTP headers. It’s critical to capture this information so that the performance results can be viewed in the context of what was actually loaded and analyzed.
What’s needed is an industry standard for archiving HTTP information. The first step in establishing that industry standard took place today with the announcement of HttpWatch and Firebug joint support of the HTTP Archive format.
HttpWatch has long supported exporting HTTP information. That’s one of the many reasons why it’s the packet sniffer I use almost exclusively. Earlier this year, as part of the Firebug Working Group, I heard that Firebug wanted to add an export capability for Net Panel. I suggested that, rather than create yet another proprietary format, Firebug team up with HttpWatch to develop a common format, and drive that forward as a proposal for an industry standard. I introduced Simon Perkins (HttpWatch) and Jan “Honza” Odvarko (main Net Panel developer), then stepped back as they worked together to produce today’s announcement.
The HTTP Archive format (HAR for short – that was my contribution ;-) is in JSON format. You can see it in action in HttpWatch 6.2, released today. HAR has been available in Firebug for a month or so. You need Firebug 1.5 alpha v26 or later and Honza’s NetExport add-on (v0.7b5 or later).
Here’s what the end-to-end workflow looks like. After installing NetExport, the “Export” button is added to Firebug Net Panel. Selecting this, I can save the HTTP information for my Google flowers search to a file called “google-flowers.har”.
After saving the file, it’s automatically displayed in Honza’s HTTP Archive Viewer web page:
I can then open the file in HttpWatch:
I’m incredibly excited to see this milestone reached, thanks to the work of Honza and Simon. I encourage other vendors and projects to add support for HAR files. The benefits aren’t limited to performance analysis. Having a format for sharing HTTP data across tools and browsers is powerful for debugging and testing, as well. One more block added to the web performance foundation. Thank you HttpWatch and Firebug!
Rohit | 19-Oct-09 at 7:26 pm | Permalink
Hi Steve,
This is great news !
But whenever I click on the export tab it complains saying ‘nothing to export’. I am trying to figure it out but any clue from your side will be highly appreciated.
regards,
rohit
Rohit | 19-Oct-09 at 9:01 pm | Permalink
Hi Steve,
I could figure out the problem – all the firebug panels were not enabled !
Wim Leers | 20-Oct-09 at 2:23 am | Permalink
Very, very nice. Thanks for connecting those developers :)
It’s very unfortunate though, that it’s 1) an expensive app (and that it’s not open source, but I wouldn’t mind that all that much) and 2) Windows-only. Or so it seems.
Steve Souders | 20-Oct-09 at 6:49 am | Permalink
@Rohit: Yes, I should have mentioned Net Panel must be enabled.
@Wim: That’s the beautiful thing about HAR. I love HttpWatch and do so much performance analysis it’s worth the cost. But if you prefer another tool, we can still share our files (eventually). I review a bunch of packet sniffers in the appendix of Even Faster Web Sites: AOL Pagetest, WebPagetest, VRTA, Fiddler, Charles, Wireshark, and IBM Page Detailer. The key is – how many of these will adopt HAR? Honza pointed out that DebugBar has already added HAR support.
Greg | 20-Oct-09 at 8:12 am | Permalink
Never heard of HttpWatch until now. Thanks for the info and resources; as a web programmer and avid firebug user I will definitely have to give this a try.
badulla | 20-Oct-09 at 3:27 pm | Permalink
Not everybody wants to purchase an app just to read HTTP logs. There should be a way to load .har files into Firebug. Opening the file in a text editor and pasting its content into the web-based viewer is a bit convoluted, and it renders the data just like Firebug, so why not use the Firebug pane?
Edward Betts | 21-Oct-09 at 12:52 am | Permalink
The Internet Archive does capture HTTP headers. They use the WARC file format.
Tom | 22-Oct-09 at 8:45 am | Permalink
will httpwatch ever be released as a mac app?
Patrick Meenan | 23-Oct-09 at 6:55 pm | Permalink
I’ll commit that AOL Pagetest will support HAR files in both the desktop and online versions (online first) and for both export and import (and potentially remotely through JSON URLs). I have most of the work done for exporting the files (just have to add the headers) and it will work with any test, including those run over a year ago.
Steve Souders | 23-Oct-09 at 8:53 pm | Permalink
@Patrick: Great news! Thanks for your support.
Sergey Chernyshev | 27-Oct-09 at 7:19 pm | Permalink
Steve, thanks for mentioning Show Slow – I hope it is useful for the community.
As for HAR, it’d be great if this kind of data can be posted to beacons by different monitoring tools so servers can store and report on this data.
Yoni Mizrachi | 08-Nov-09 at 6:30 am | Permalink
This is great news, however I found a bug(?) that makes it impossible for me to open firebug exported har in HTTPWatch. Some of the cookies use the “version” attribute, in FireBug they are exported correctly but when trying to open it with HTTPWatch is says there is a problem. According to cookies spec this is a valid attribute… the har “spec” I found did not cover this attribute.
Is there a central group where this kind of mismatches can be reported?
Steve Souders | 08-Nov-09 at 1:18 pm | Permalink
@Yoni: Send a message to the Firebug Working Group or submit a bug for HAR issues.