Benchmarking Browsers with Real Websites: Chrome, Firefox, Opera, Safari, IE

When it comes to a browser benchmark people act like this:

The problem with browser benchmarks today is that they are extremely synthetic. A majority of them are Javascript only because its easier to test.

Why Do We Need a Better Benchmark?

  • Sunspider – Javascript only.
  • V8 - Javascript only.
  • Dromaeo – Mostly Javascript, some CSS.
  • Acid Test – Random web standards. No one really cares unless you get close to 100 or 0.
  • Peacekeeper – It runs Java and then allows itself to download and run an EXE. It spends most of the time doing a 3D benchmark on your browser as though you were running 3dMark.

As you can see, Javascript benchmarks dominate the landscape. There is a complete lack of benchmarks with anything remotely applying to real life situations.

Thus there are many who completely dismiss these benchmarks, stating that they don’t really measure the “true speed” of a browser. Maybe they are right.

How to Make a Better Benchmark?

According to Microsoft:

Knowing the top speed of a car doesn’t tell you how fast you can drive in rush hour. To actually see the difference in page loads between all three browsers, you need slow-motion video. This one’s also a tie.

Using slow-motion video? What a great idea. Maybe we can benchmark operating systems like that.

However there is a built-in timer which can let us know when the browser finishes loading a page. The hard part is getting it to load in a new page while using websites that people actually use in real life. Details explained later.

Setup

Core 2 Duo 1.8 GHZ 2GB Mem
Vista 32bit

Chrome 3.0.189.0
Firefox 3.5 RC 2
Internet Explorer 8.0.6001.18702
Opera 10.00 Beta Build 1551
Safari 4.0 (530.17)

0 Extensions.

Websites

Here are the websites used in this test.

  • Baidu.com – Chinese Google
  • Blogger.com – Popular blogsite
  • Facebook.com – Popular social site
  • Google.com – Popular search Engine
  • Havenworks.com – A poorly designed site weighing 820 kb with over 280 images
  • Live.com – Microsoft search engine
  • Myspace.com/tom – Typical Myspace profile
  • Reddit.com – Link aggregator
  • Wikipedia.org – Online encyclopedia

All websites were saved to disk and opened beforehand into the browser to minimize the impact of the few remaining external links and hard drive seek times.

Results: Average Time to Load 1 Page – 10 Repetitions (in ms)

Chrome

  • Baidu – 229
  • Blogger – 38.6
  • Facebook – 146.1
  • Google – 11
  • Havenworks – 482.2
  • Live – 42.3
  • Myspace – 536.5
  • Reddit – 141.4
  • Wikipedia – 222.8
  • Total Average – 205.5

Firefox

  • Baidu – 37.3
  • Blogger – 93.9
  • Facebook – 263.9
  • Google – 56.4
  • Havenworks – 1377.8
  • Live – 76.9
  • Myspace – 1386.4
  • Reddit – 337.3
  • Wikipedia – 114.1
  • Total Average – 416

Internet Explorer

  • Baidu – 32.6
  • Blogger – 92.2
  • Facebook – 238.7
  • Google – 34.4
  • Havenworks – 2722.2
  • Live – 107.7
  • Myspace – 922.6
  • Reddit – 451
  • Wikipedia – 410.2
  • Total Average – 556.8

Opera

  • Baidu – 24
  • Blogger – 77.5
  • Facebook – 214
  • Google – 27.5
  • Havenworks – 584
  • Live – 47
  • Myspace – 500
  • Reddit – 279
  • Wikipedia – 83
  • Total Average – 204

Safari

  • Baidu – 238.3
  • Blogger – 58.9
  • Facebook – 203.4
  • Google – 22.4
  • Havenworks – 438.3
  • Live – 47
  • Myspace – 442.8
  • Reddit – 161.8
  • Wikipedia – 233.6
  • Total Average – 205.2

loading

Conclusion

Average Loading Times (in ms)

Opera             204
Safari            205.2
Chrome            205.5
Firefox           416
Internet Explorer 556.8

Opera is the winner, but not by far. Interestingly, the browsers all perform differently on different websites.

Before You Complain

Download the benchmark here and use your own websites if you like. Follow the instructions in the source code.

Frequently Asked Questions

Q: According to some outdated website, webkit fires onload way too early. Therefore your benchmark is flawed.
A: It was fixed 2 years ago.

Q: Your statistics suck. You should have used a geometric mean with n degrees of freedom and 95th percentile measurements with a chi-squared test to calculate the deviation from a Weibull distribution … etc…
A: I have yet to hear a convincing reason why a simple average is not good enough. The only complaints I have so far is from people saying that the Havenworks bench is unfair. Why should I weigh it any differently than the other sites? Run your own tests.

Q: Can you include my favorite X browser and X platform so your benchmark doesn’t suck?
A: I don’t have infinite time, money, computers, and incentives. If you can offer any of those, maybe I will oblige.

Q: Why save websites to disk? Why not just load them online?
A: Because we’re testing loading speed, not your connection and how slow the website is.

Q: But what about all the super duper network code that my browser has?
A: The only feature that would make any appreciable difference would be concurrent connections (which I believe all modern browsers have). Other features like DNS look-ahead (likely cached) or different networking engines probably costs microseconds compared to the milliseconds it takes for a page to load.

Q: Why isn’t my favorite website there?
A: Because I couldn’t find a way to reliably download and adapt them. Wget refuses to work correctly, and some websites have some Javascript that messes with the script. So I just used Firefox to download. (Yahoo, Slashdot, and Digg don’t play nicely)

Q: Some of your test sites still have external links.
A: I tried HTTrack, wget, and the browsers for saving pages, and none of them work 100%. You can fix it by hand editing, but I don’t have the time, money, and incentive to do so completely. I have removed some by hand, but websites nowadays like to obfuscate their javascript making it a hassle.

Q: Why didn’t you use DOMContentLoaded etc..?
A: This was what I wanted to use. But it doesn’t work on IE8, and 90% of the time, the browser is stuck and useless in that stage (especially when loading from hard drive instead of online).

Q: You have to paste that tiny script into every webpage you want to benchmark. That’s lame.
A: My first version did not require that. Blame IE8 for being the only browser to break it.

Q: My browser is still the best.
A: Ok

Q: Your benchmark is worthless because it doesn’t show the benefit of extensions or how it matches the color of my eyes, I will never consider using another browser.
A: Ok

Resources: Codexon