Futuremark Caught NVIDIA and ATI Technologies On Cheating In 3DMark03

by Anton Shilov
05/23/2003 | 03:28 PM

Modern graphics cards and processors are very powerful and capable of performing numerous operations and features. The highest-end solutions like the GeForce FX 5900 Ultra from NVIDIA Corporation and the RADEON 9800 PRO from ATI Technologies are able to provide enough performance in almost all modern games, however, that is not enough and we are looking forward even more eye-candy 3D games that require even more powerful graphics cards. Since we do not have those powerful 3D games now, in order to determine the performance of expensive graphics cards we use synthetic benchmarks, such as Codecult Codecreatures and Futuremark 3DMark Although this is the only opportunity to measure the performance in upcoming applications, IHVs usually try to tweak their drivers to show better results in such benchmarks.

Today Futuremark released its patch and a document explaining how NVIDIA and ATI cheated the results of the 3DMark03 benchmark. Below you find some statements by the company.

Futuremark’s audit revealed cheats in NVIDIA Detonator FX 44.03 and 43.51 WHQL drivers. Earlier GeForce FX drivers include only some of the cheats listed below. Note that neither NVIDIA Corporation nor ATI Technologies commented on this.

• The loading screen of the 3DMark03 test is detected by the driver. This is used by the driver to disregard the back buffer clear command that 3DMark03 gives. This incorrectly reduces the workload. However, if the loading screen is rendered in a different manner, the driver seems to fail to detect 3DMark03, and performs the back buffer clear command as instructed.
• A vertex shader used in game test 2 (P_Pointsprite.vsh) is detected by the driver. In this case the driver uses instructions contained in the driver to determine when to obey the back bufferclear command and when not to. If the back buffer would not be cleared at all in game test 2, the stars in the view of outer space in some cameras would appear smeared as have been reported in the articles mentioned earlier. Back buffer clearing is turned off and on again so that the back buffer is cleared only when the default benchmark cameras show outer space. In free camera mode one can keep the camera outside the spaceship through the entire test, and see how the sky smearing is turned on and off.
• A vertex shader used in game test 4 (M_HDRsky.vsh) is detected. In this case the driver adds two static clipping planes to reduce the workload. The clipping planes are placed so that the sky is cut out just beyond what is visible in the default camera angles. Again, using the free camera one can look at the sky to see it abruptly cut off. Screenshot of this view was also reported in certain articles by ExtremeTech and Beyond3D. This cheat was introduced in the 43.51 drivers as far as Futuremark recognizes.
• In game test 4, the water pixel shader (M_Water.psh) is detected. The driver uses this detection to artificially achieve a large performance boost - more than doubling the early frame rate on some systems. In their inspection Futuremark noticed a difference in the rendering when compared either to the DirectX reference rasterizer or to those of other hardware. It appears the water shader is being totally discarded and replaced with an alternative more efficient shader implemented in the drivers themselves. The drivers produce a similar looking rendering, but not an identical one.
• In game test 4 there is detection of a pixel shader (m_HDRSky.psh). Again it appears the shader is being totally discarded and replaced with an alternative more efficient shader in a similar fashion to the water pixel shader above. The rendering looks similar, but it is not identical.
• A vertex shader (G_MetalCubeLit.vsh) is detected in game test 1. Preventing this detection proved to reduce the frame rate with these drivers, but Futuremark have not yet determined the cause.
• A vertex shader in game test 3 (G_PaintBaked.vsh) is detected, and preventing this detection drops the scores with these drivers. This cheat causes the back buffer clearing to be disregarded; Futuremark are not yet aware of any other cheats.
• The vertex and pixel shaders used in the 3DMark03 feature tests are also detected by the driver. When Futuremark prevented this detection, the performance dropped by more than a factor of two in the 2.0 pixel shader test. Futuremark boys have used various techniques to prevent NVIDIA drivers from performing the above detections. Futuremark have been extremely careful to ensure that none of the changes Futuremark have introduced causes differences in either rendering output or performance. In most case, simple alterations in the shader code – such as swapping two registers – has been sufficient to prevent the detection.

A test system with NVIDIA GeForce FX 5900 Ultra and the 44.03 drivers gets 5806 3DMarks with 3DMark03 build 320. The new build 330 of 3DMark03 in which Detonator FX 44.03 drivers cannot identify 3DMark03 or the tests in that build gets 4679 3DMarks – a 24.1% drop.

Futuremark's investigations reveal that some drivers from ATI also produce a slightly lower total score on this new build of 3DMark03. The drop in performance on the same test system with a RADEON 9800 PRO using the CATALYST 3.4 drivers is 1.9%. This performance drop is almost entirely due to 8.2% difference in the game test 4 result, which means that the test was also detected and somehow altered by the ATI drivers. Futuremark crew is currently investigating this further.

Are synthetic benchmarks good if they can be cheated? You will decide it next week.