| Benchmark (computing) |
Article Index for Benchmark |
Information AboutBenchmark (computing) |
|
In Computing , a benchmark is the act of running a Computer Program , a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it. The term, benchmark, is also commonly used for specially-designed benchmarking programs themselves. Benchmarking is usually associated with assessing performance characteristics of computer hardware, for example, the floating point operation performance of a CPU , but there are circumstances when the technique is also applicable to software. Software benchmarks are, for example, run against Compiler s or Database Management System s. Benchmarks provide a method of comparing the performance of various subsystems across different chip/system architectures. Benchmarking is helpful in understanding how the database manager responds under varying conditions. You can create scenarios that test deadlock handling, utility performance, different methods of loading data, transaction rate characteristics as more users are added, and even the effect on the application of using a new release of the product. PURPOSE As Computer Architecture advanced, it became more and more difficult to compare the performance of various computer systems simply by looking at their specifications. Therefore, tests were developed that could be performed on different systems, allowing the results from these tests to be compared across different architectures. For example, while Intel Pentium 4 processors generally operate at a higher clock frequency than AMD Athlon XP processors, this does not necessarily translate to more computational power. In other words a 'slower' AMD processor, with regards to clock frequency, can perform as well on benchmark tests as an Intel processor operating at a higher frequency. Benchmarks are designed to mimic a particular type of workload on a component or system. "Synthetic" benchmarks do this by specially-created programs that impose the workload on the component. "Application" benchmarks, instead, run actual real-world programs on the system. Whilst application benchmarks usually give a much better measure of real-world performance on a given system, synthetic benchmarks still have their use for testing out individual components, like a Hard Disk or networking device. Benchmarks are particularly important in Semiconductor Microprocessor design, giving processor architects the ability to measure and make tradeoffs in Microarchitectural decisions. For example, if a benchmark extracts the key Algorithms of an application, it will contain the performance-sensitive aspects of that application. Running this much smaller "snippet" on a cycle-accurate simulator, can give clues on how to improve performance. Prior to 2000, computer and microprocessor architects used SPEC to do this, although SPEC's Unix-based benchmarks were quite lengthy and thus unwieldy to use intact. Computer manufacturers have a long history of trying to set up their systems to give unrealistically high performance on benchmark tests that is not replicated in real usage. For instance, during the 1980s some compilers could detect a specific mathematical operation used in a well-known floating-point benchmark and replace the operation with a mathematically-equivalent operation that was much faster. However, such a transformation was rarely useful outside the benchmark until the mid-1990s, when RISC and VLIW architectures emphasized the importance of Compiler technology as it related to performance. Benchmarks are now regularly used by Compiler companies to improve not only their own benchmark scores, but real application performance. Manufacturers commonly report only those benchmarks (or aspects of benchmarks) that show their products in the best light. They also have been known to mis-represent the significance of benchmarks, again to show their products in the best possible light. Taken together, these practices are called ''bench-marketing.'' Users are recommended to take benchmarks, particularly those provided by manufacturers themselves, with ample quantities of salt unless the benchmarks are certified and relate directly to a recognizable application workload. Ideally benchmarks should only substitute for real applications if the application is unavailable, or too difficult or costly to port, to a specific processor or computer system. If performance is really critical, the only benchmark that matters is the actual workload that the system is to be used for. If that is not possible, benchmarks that resemble real workloads as closely as possible should be used, and even then used with skepticism unless independently certified. It is quite possible for system A to outperform system B when running a certain program on the workload in the benchmark, and the order to be reversed with the same program on a real life workload. CHALLENGES Benchmarking is not easy and often involves several iterative rounds in order to arrive at predictable, useful conclusions. Interpretation of benchmarking data is also extraordinarily difficult. Here is a partial list of common challenges:
TYPES OF BENCHMARKS #Real program
#Kernel
#Toy Benchmark/ micro-benchmark
#Synthetic Benchmark
# I/O benchmarks # Parallel benchmarks: used on machines with multiple processors or systems consisting of multiple machines. COMMON BENCHMARKS Industry Standard (audited and verifiable)
Others
Open source benchmarks
Microsoft Windows benchmarks
Mac OS X benchmarks Java benchmarks BOOKS Jim Gray (Editor), The Benchmark Handbook for Database and Transaction Systems (2nd Edition), Morgan Kaufmann, 1993, ISBN 1-55860-292-5 Database Benchmarking Practical Methods for Oracle & SQL Server Dr. Bert Scalzo, Kevin Kline, Claudia Fernandez, Donald K. Burleson, Mike Ault, 2007, ISBN 0-9776715-3-4 SEE ALSO
EXTERNAL LINKS
|
|
|