Benchmarks, Cheats, Optimizations…

So, let me tell you about the benchmarks used in the industry before we pass to some real tests.

Earlier this year both NVIDIA and ATI Technologies were caught on cheating in the 3DMark03 benchmark. The latter explained its position and eventually removed the cheats. The former went on saying that this industry benchmark had been developed with an evil intent – to show NVIDIA GPUs in bad light. This was a bit ridiculous and no one really believed that since 3DMark is co-developed by Microsoft, Dell Computer and a lot of other industry players. Unfortunately for NVIDIA, other cheats in its drivers were later disclosed by the community. Today no one believes the company is fully capable of developing a powerful graphics processor to run games fast enough without software “support”.

It is now perfectly clear that NVIDIA’s latest architecture is not as powerful as the company wants and claims it to be, but is still fast enough to play the vast majority of games available in the market, especially those that make little use of pixel and vertex shaders. The problem lies deeper, though. NVIDIA used to be an extremely successful company in the past and its solutions were tangibly faster and more feature-rich than products from 3dfx or ATI Technologies. Furthermore, this company had been adored by the industry and PC enthusiasts for so long that it simply couldn’t and wouldn’t be without being the number one. So, NVIDIA did not want to lose its King-of-the-Hill position and resorted to cheating. This dethroned the company faster than anything else possibly could, since a lot of enthusiasts feel now pretty suspicious about NVIDIA’s products. 

NVIDIA got trapped just like 3dfx did back in 2000. The community of PC enthusiasts relies on benchmarks. When users saw that NVIDIA’s GPUs were slow in a majority of benchmarks used by the press, they started to bash NVIDIA heavily, just like they had criticized 3dfx.

It was largely the tech community and the press who first saluted 3dfx, then pulled NVIDIA on the very top of the 3D Everest, and finally helped ATI Technologies to rise and shine. This same community of PC enthusiasts and this same press dragged 3dfx down and took the crown away from NVIDIA. No doubt, ATI Technologies will be done with the same way, if the company makes a mistake.

All problems obviously come from the fact that advanced end-users sometimes too blindly believe in the benchmarks and believe what the benchmarks show them. Less advanced end-users believe enthusiasts, who believe in benchmarks, and that’s how it is – even if a product is not a competitive one, but scores high in popular benchmarks, the industry eats it. If it is not, it will never be accepted by the industry. If the crowd feels deceived, it totally disregards the product and forgets about all its real advantages. However, there is no use at all trying to convince the public that benchmarks are not to be worshipped, but approached carefully. The benchmark is God!