I wanted to touch base on this article that's been getting quoted a lot lately: CNet's Performance showdown: Windows 7 vs. Snow Leopard. I did find one individual who actually called out the seriously biased nature of this article, but everyone else seems to be jumping on board. I find that it really gives Windows 7 a bad rap, one that it does not deserve at all.

First off, let me explain the hardware issue before I get into the much more serious issues with the software debate. This reviewer thought it was a good idea to use two hard drives for this test, which sounds like a good plan to the layman. Unfortunately, he used two different hard drives, only stating that they must perform similarly because they have similar specifications. As all of us geeks know unless you have two identical hard drives, this entire test is worthless. Just look at this graphic from the Tech Report's recent 2.5" hard drive roundup:

 Notice that the very similarly spec'd Scorpios and Momentus drives actually perform very differently in this media encoding test. Furthermore, let's look at power consumption for these very similar drives:

As you can see, under a given workload, each drive can have different power characteristics. Am I suggesting that .2 watts makes the difference between an hour and an hour and a half of battery life? Not by any means, it's just one of the statistical discrepancies we need to be aware of.

Another thing that we don't know is whether or not Apple uses the acoustic dampening features of their hard drives to make them quieter, but perform slower. It's quite possible that they do set this to make the machine as quiet as possible, much like Dell did with their terribly-loud Seagate drives, but without the SMART data, we can't know. If Windows 7 was installed on a drive that had dampening turned on, it's performance would be affected significantly.

But enough of the hardware issues. The fact is Windows 7 was installed on a Macbook Pro and compared to Snow Leopard on the same machine, just with slightly different hard drives. The next biggest issue is this one: 

"For each OS, during the tests, the computer had the following software installed: iTunes 9, QuickTime, Call of Duty 4 Modern Warfare, and Cinebench R10. All are 64-bit except Call of Duty and QuickTime. Windows 7 was tested with QuickTime 7 (version 7.6.4), which is 32-bit, and Snow Leopard was tested with QuickTime X, which comes with the OS. The reason is that QuickTime X is not currently available for Windows and you can't install QuickTime 7 on Snow Leopard."

Someone hasn't used Windows very much, or they'd know that iTunes 9 is not 64-bit at all on Windows platforms. Yes, Apple provides a 64-bit installer, but it only contains a little different driver setup that is more compatible on 64-bit Windows. The executable itunes.exe is still very 32-bit, as evidenced by the "*32" next to it in Task Manager.

The version of Quicktime being so far apart is another big issue for me, since Quicktime X was designed to work with Grand Central, which probably means some of it's encoding is being done by a GPU. We have software that does this in Windows too, but obviously not Quicktime. This is one place where comparing Quicktime to Quicktime is really an Apple to Orange situation.

So what can we actually take away from this review?

1. In Cinebench, an open source, very comparable benchmark, Windows 7 was around 7% faster.
2. In Call of Duty, a very comparable benchmark, Windows 7 was around 24% faster.
3. In Quicktime, not comparable at all, Windows 7 was much slower, 63%
4. Shutdown and Boot up times, while very different percentage wise, were within 6 seconds of each other. Try counting that out and see if you'd notice.
5. iTunes is around 8% faster on Mac OS, probably due to the 64-bit optimizations that aren't there for Windows, or possibly hard drive differences.

I wanted to talk about the battery life issue that's so important. This reviewer contradicts himself in the same paragraph, which is very frustrating:

Windows 7 lasted 78 minutes, while Snow Leopard managed to stay on for 111 minutes. These numbers are, of course, the worst case scenario--in real life, you'll get much longer battery life for each OS with regular usage. Personally, I could easily get about 3 hours with Windows 7 when running the MacBook Pro using the operating system's recommended "Balanced" power management scheme. Nonetheless, it's obvious that Windows 7's battery life is just about two-thirds of Snow Leopard's on the MacBook Pro.

Notice he says that Windows only lasts 78 minutes with the maximum performance mode selected and running benchmarks versus 111 minutes in Mac OS. Great, Microsoft tunes the performance of the CPU a lot higher in that mode, and it hits it's lower power states less frequently. This is something you can't control on Mac OS. He then says he gets 3 hours with Windows 7, but doesn't give us a comparable number in Mac OS. Why is that? Was it lower?

Anandtech's Windows 7 Performance Guide shows the OS in a very different light:

Down the page a bit, they say:

Starting with the NV52, our Athlon 64 X2 laptop, we see some immediate advantages for Windows 7. In terms of battery life it edges out Windows XP in all situations, and clearly surpasses Vista, particularly in DVD playback.

I'd love to see just how well Mac OS fits in that graph, using it's balanced settings, not Max Performance ones. Max Performance is a desktop thing, and it shouldn't even be available to laptops.

After tearing this article apart, there's one last thing I wanted to mention because I hear it so often. "Apples are better at graphics." This is simply not the case any more. I wish this review included Adobe CS4 applications, which that statement always refers to, because I'm pretty sure you'd see the truly 64-bit Photoshop CS4 wipe the floor with it's Mac OS counterpart. If anyone has those numbers, I'd love to see them. We did see that Windows is roughly 7% faster at rendering 3D scenes, and I'd imagine that is a pretty good idea of what you'd see across the board in CS4.

Just remember, even if a huge conglomerate like CNet puts their name on it, don't always believe what you read. Mac OS X has a lot of catching up to do before they catch this version of Windows in the performance world.