Benchmarking Results Can Be Very Different When Run in CI Tools

Most CI (continuous integration) tools run inside VMs in the cloud sometimes nested in other VMs. This causes pauses in execution as the VM may be applying different techniques to manage multi-tenency—throttling, pausing execution, and so on. Benchmarking code can have different results compared to running locally on your machine for this reason.

See also:

  • iai is a rust library with more accurate results for benchmarking by counting CPU instructions