Busting AnandTech's Woodcrest vs Opteron benchmark
Anand reproduced MySQL bug #15815, and quickly termed it a feature called "negative scaling"
We have proven that Anand is a paid Intel pumper here. Now, AnandTech has done a series of Woodcrest vs Opteron benchmarks. Again, AnandTech showed Intel having substantial lead.
Really? Can a paid pumper be trusted ever again? Let's forget about the past, and look at AnandTech's newest presentations without prejudice.
One year ago, in June 2005, AandTech's Johan De Gelas did similar benchmarks on dual core Opteron. You can see from that page that Opteron MySQL performance was 17% lower under Gentoo Linux than under SuSe Linux. On the same page, AnandTech compared the MySQL (InnoDB) performance of two (2) dual core Opteron 875 processors against one (1) dual core Opteron processor under Gentoo Linux. The conclusion was that adding a processor led to about 10% total performance increase. This was extremely poor scaling, but there was at least some increase.
AnandTech's system configuration on Woodcrest and Opteron is here. Pay attention to the motherboard AnandTech chose for the Opteron. It was the K8N Master2-FAR. What's special about this board? It is probably the only 2P Opteron board with only one bank of memory. With this board, the second Opteron always has to get its memory from the first CPU, incurring high latency. This MSI board (pictured below) is definitely not designed for server use, it doesn't even have onboard video.
A typical Opteron 2P server board looks likt this Tyan S2892, with two banks of memory, one for each CPU. The only AMD recommened MSI Opteron board is the K8D Master3-FA4R.
For both Woodcrest and Opteron, Gentoo Linux 2.6.15-gentoo-r7 and Mysql5.0.21 were used. Gentoo has about 1% of the Linux market and is getting more popular. Now, look at the MySQL performance numbers AnandTech got here.
First, we look at the comparison between one(1) 3GHZ Woodcrest Xeon 5160 and one (1) 2.4GHZ Opteron 280. The Woodcrest did 996 queries/second, while the single Opteron did 805 queries/second. The Woodcrest had 25% clockspeed advantage and 23% lead over the Opteron 280 on MySQL query speed.
Now, look at the next row, which was TWO Woodcrest CPUs against TWO Opteron 280 CPUs. The result? The 2P Opteron 280 system showed a 23% performance DECREASE from 1P Opteron 280. It did 622 queries/second. Adding a CPU in a 2P system leads to 23% drop in total MySQL performance? Two processors slower than one?
Anyone with half a brain would immediately conclude that something was seriously wrong with this benchmark. The result was simply stupid. If some moron told his boss that adding a CPU actually causes performance drop, he should be fired immediately. Because he should have used his brain to fixed the sucker instead of making such stupid conclusions. You need to dig deeper, you need to debug the software and see where it's hanging, and then solve it. MySQL has a lot of parameters to tune, so does the Linux SMP kernel. You either try out all available combinations to find a rational outcome or you use your brain to identify and fix the problem. Of course, doing this requires deeper understanding of the operating system and MySQL. We don't expect AnandTech's people possess this level of knowledge, otherwise they'd be working as software or system engineers who make a lot more money and accumulate a lot of stock options. But, they should at least contact the OS and DB vendor and Google the WEB* to find a solution. If the problem was identified as a major bug in the OS or DB or was otherwise persistent, they should try some other OS such as RedHat Enterprise, SuSe Enterprise, or different versions of OS and DB. A nonsense software result simply can't be used to represent CPU performance. A nonsense result only makes the benchmarker look stupid.
Yet, AnandTech concluded that "[q]uad-core and Dual-core x86, you'll notice that the scaling is negative... It seems like an anomaly, but this is not the case. These benchmarks have been checked, verified and checked again. "
So, not only AnandTech is a paid pumper, they are also surprisingly incompetent*.
Stupidity lends no confidence.
Actually, if you look at the first graph on this page, you can see that the 2P Opteron 280 quickly reached a saturation point at concurrency level of 2. But the 1P Opteron 280 scaled up quite nicely up to concurrency level 8. Apparently, there was a resoucre contention problem in the 2P setting. You just need to dig deeper...
Performing a Google search with the words "Linux opteron dual core MySQL bug", I quickly found this MySQL.com's page for bug #15815. The #15815 bug is highly reproducible, it was a bug due to mutex locks and cache line "ping-pong". There was a fix for it. In the discussion of the "negative scaling" on AnandTech, "mutex ping-pong" was mentioned. This is indication that someone at AnandTech actually knew about this. At the bottom of the bug #15815 page, a user reported:
"I'm using MySQL 5.0.21 on a 2.6.16 Gentoo system, I've tried both Gentoo compiled and precompiled Mysql distribution, and the Xeon machine (dual xeon 3.8Ghz with 8GB RAM) acts too much better then the dual Opteron (Opteron 275 with 16GB RAM)...better as 640 tps (Xeon) versus 110 (!!!) tps for the Opteron."
Note this guy was using the Gentoo OS and the same MySQL 5.0.21* Anand was using. He was also running dual Opteron 275.
Maybe someone noticed this bug report on mysql.com and instructed Anand to use the same setup to reproduce it but present it as an Opteron benchmark?
* The most current version of MySQL is 5.0.22.