Measuring Experimental Error in Microprocessor Simulation
Rajagopalan Desikan, Doug Burger, Stephen W. Keckler · Proceedings - International Symposium on Computer Architecture/Proceedings · 2001
Abstract: We measure the experimental error that arises from the use of non-validated simulators in computer architecture research, with the goal of increasing the rigor of simulation- based studies. We describe the methodology that we used to validate a microprocessor simulator against a Compaq DS-10L workstation, which contains an Alpha 21264 processor. Our evaluation suite consists of a set of 21 microbenchmarks that stress different aspects of the 21264 microarchitecture. Using the microbenchmark suite as the set of workloads, we describe how we reduced our simulator error to an arithmetic mean of 2%, and include details about the specific aspects of the pipeline that required extra care to reduce the error. We show how these low-level optimizations reduce average error from 40% to less than 20% on macrobenchmarks drawn from the SPEC2000 suite. Finally, we examine the degree to which performance optimizations are stable across different simulators, showing that researchers would draw different conclusions, in some cases, if using validated simulators.
Cited by 150 works. doi:10.1145/379240.565338
Authors
References
- The SimpleScalar tool set, version 2.0
- lmbench: Portable Tools for Performance Analysis
- SimICS/sun4m: a virtual workstation
- Complete computer system simulation: the SimOS approach
- Lockup-free instruction fetch/prefetch cache organization
- The Alpha 21264 microprocessor
- Instruction issue logic for high-performance, interruptible, multiple functional unit, pipelined computers
- Clock rate versus IPC
- Tradeoffs in Two-Level On-Chip Caching
- The Alpha 21264 microprocessor architecture