Yamaguchi-san asked for help to better reproduce a CPU performance measurement on Twitter. Here’s the long reply.
Performance measurement is a pick-your-poison kind of situation.
Host’s physical characteristics matters. High end Xeon may have different multi-minutes throttling behavior than low end Celeron. Low end mobile CPUs can only burst for time measurable in seconds. Active cooling versus passive cooling influences sustained peak performance a lot too.
What matters is simple: heat dissipation 🔥 and how the OS manages CPU cores frequency.
Back in the days, things were simple. A good old Intel 8088 would run at full speed, continuously, with no heatsink! Happily dissipating the same amount of power all the time. Running a performance test would lead consistent reproducible results, if you didn’t mind running at 4.77 MHz. 🐢
That hasn’t been true for a while.
As transistors became smaller, processors started to pack more and more, now with billions of them in a single die. We saw a big shift in the late 90s: a die can pack more transistors than what can be used simultaneously, so not all transistors can be active simultaneously!
With higher transistor density, thermal throttling becomes inevitable even on desktop, just like how it happens on mobile CPUs. Two techniques are used: dynamic frequency scaling coupled with dynamic voltage scaling. While initially it was meant as a way to save power for laptops, it eventually became inevitable even for desktop class CPUs.
On a recent high end desktop CPU, single threaded performance will likely give you the turbo frequency for a while, but only if you carefully reduce all other background threads. Also, there’s a delay before one CPU core scales all the way up to its maximum frequency. A delay which can be relatively high.
Consummers CPUs are now great at bursting, not much at broad sustained workload. Get a Xeon for that. Googlers are spoiled here, yet the original question was likely asked while trying to reproduce a benchmark on a high end CPU.
Windows 10 has a Game Mode and I’m pretty sure it’s designed explicitly for this but the documentation is sadly vague. See GetExpandedResourceExclusiveCpuCount() to get exclusive access to one or more cores. The OS decides which one you get, not the app. It’s the OS that has the power to tell the CPU to lock a specific core into turbo boost mode. I am not aware of such functionality on other OSes.
Linux has a different knob which is the CPU governor, which is configurable as root. Not all governors are available on all systems, the list of supported CPU governors is especially spotty on mobile CPUs.
But before diving more into performance, first, I lied.
CPUs are not throttled by heat dissipation limit. They are throttled by heat accumulation. Think of the CPU as a heat machine. The system’s input is heat production in Watts. The system’s output is cooling, also in Watts, negative. The capacity is amount of accumulated heat, in Joules (W·s). In this case, the energy accumulated is measurable as temperature, in Kelvin. The CPU can measure its own temperature, and uses it to control thermal throttling.
So in the end, you have a energy accumulation budget, where you can get the CPU die to ramp up and accumulate more energy than it dissipate, but that’s temporary by design. And changing each core frequency implies delays, so there’s a trade off the operating system must make.
The current state of CPU designs means that in practice performance measurement is affected by:
You need to address all three variables to improve performance measurements.
The first one can only be achieved by running less processes on your system. It’s not as much about background processes priority, it’s about not having them be scheduled at all. While running a single Go binary in user mode can be an extreme solution, typical workstations with many cores and no anti-virus installed will ‘generally’ be fine for CPU bound benchmarks.
The two later points can be worked around in two ways:
go test -bench does by default), then you may as well
to prewarm with a build, so that all CPU cores are warmed up.
For Go users, the most practical way to improve the reproducibility remains to tell the go test tool to use a single thread:
go test -bench=. -cpu 1
and run it a few times in a row. Yes this sounds silly, but this is the closest you’ll get without messing around with your CPU settings on a turbo boost enabled CPU.
Mobile is worth its own post because it is very touchy, here’s a short summary from what we learned on the Chrome infrastructure.
In practice mobile devices are limited by the heat dissipation, so having active cooling on the physical device, without a case, will help. That’s where thermodynamics University courses come handy.
The best way to increase the heat accumulation budget is to only start benchmarks when the CPU is under 35°C or so, so that the CPU can exceed the budget for a moment as the CPU temperature increases.
If the device is rooted, change its cpu governor to powersave to accelerate
cool down between benchmarks.
Then you should be able to reliably measure peak performance.
An alternative is heat the device first and only start your benchmark once it’s near breakdown so you measure the worst case performance. This is not something I personally tried.
