by Anton Shilov
06/21/2011 | 10:59 PM
Advanced Micro Devices on Tuesday officially said that it would not endorse SYSmark 2012 benchmark and will also quit BAPCo (Business Applications Performance Corp.) that developers to PC performance measurement software. AMD and, according to some reports, Nvidia Corp. and Via Technologies have disagreements with BAPCo over the benchmark results.
SYSmark 2012 is an application-based benchmark that reflects usage patterns of business users in the areas of office productivity, data/financial analysis, system management, media creation, 3D modeling and web development. Applications used in SYSmark 2012 were selected based on market research and include Microsoft Office, Adobe Creative Suite, Adobe Acrobat, WinZip, Autodesk AutoCAD and 3ds Max, and others. The benchmark is used by many enterprises, PC makers, government organizations and other institutions to determine the performance in business applications and make the right choice of PCs or components.
The main concern of AMD, and presumably other semiconductor companies, is that BAPCo SYSmark 2012 does not utilize graphics processing units (GPUs) for general purpose computing tasks (GPGPU) and solely relies on performance of central processing units (CPUs). According to AMD, such approach is misleading as many applications nowadays take huge advantage of GPGPU technologies, including Adobe Flash 10.2 (SYSmark 2012 uses 10.1), Microsoft Office 11 (SM2012 uses Office 10), Microsoft Internet Explorer 9 (SM2012 uses IE8), Microsoft Movie Maker and many others. Besides, AMD accuses BAPCo of implementing unrepresentative workloads into the benchmark in order to favour AMD's competitor Intel.
"The SYSmark benchmark is not only comprised of unrepresentative workloads (workloads that ignore the importance of heterogeneous computing and, frankly, favor our competitor’s designs), but it actually generates misleading results that can lead to very poor purchasing decisions, causing governments worldwide to historically overspend somewhere in the area of approximately $8 billion," said Nigel Dessau, chief marketing officer of AMD.
BAPCo does not agree with AMD and claims that the sole purpose of the company's demarche is to devaluate SYSmark 2012 in order to prevent the customers from using it to evaluate performance of AMD-powered systems.
"BAPCo is disappointed that a former member of the consortium has chosen once more to violate the confidentiality agreement they signed, in an attempt to dissuade customers from using SYSmark to assess the performance of their systems. BAPCo believes the performance measured in each of the six scenarios in SYSmark 2012, which is based on the research of its membership, fairly reflects the performance that users will see when fully utilizing the included applications," a statement by BAPCo reads.
The company also states that AMD voted in support of over 80% of the SYSmark 2012 development milestones, and were supported by BAPCo in 100% of the SYSmark 2012 proposals they put forward to the consortium.
Scandals about alleged bias of benchmarks happen regularly and mostly due to the fact that certain systems or components cannot get competitive results in those measurement programs. This is by far not the first scandal between BAPCo and AMD over the last ten years. Previously AMD simply accused BAPCo of using Intel-optimized software within the test suite.
At present AMD's microprocessors cannot compete against Intel's chips head to head across a wide variety of applications. BAPCo's SYSmark 2007 shows results that are pretty similar with other benchmarks: Intel is well ahead of AMD. The problem is that SYSMark 2012 will be used for several years to come and software in 2013 will be different from programs 2010 as developers learn how to take advantage of GPU. Naturally, nowadays the majority of programs still rely on CPU, but several years down the road many more applications will use heterogeneous computing model involving both CPU and GPU (including GPUs integrated into Ivy Bridge and Haswell microprocessors).
What AMD and other apologists of heterogeneous model want is a benchmark that would show the glory of their graphics processing units here and now.