criterion performance measurements
overview
want to understand this report?
Pure/base
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.691984511911869 | 4.709399274860819 | 4.726369925774634 |
| Standard deviation | 7.346869135896217e-3 | 2.0005422511818634e-2 | 2.6469903933151958e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Pure/strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.000243951100856 | 3.0194942499510944 | 3.0342213194817305 |
| Standard deviation | 1.0188148356974125e-2 | 1.8540272431805522e-2 | 2.224644983866258e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Pure/strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.8154793744906783 | 1.8229019242959719 | 1.8291732299452028 |
| Standard deviation | 2.6586844936229096e-3 | 7.733322935289231e-3 | 9.954828989513552e-3 |
Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.
Pure (unsafe idx)/base
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.6919822469353676 | 3.699673792036871 | 3.7109876861795783 |
| Standard deviation | 3.544460273786536e-3 | 1.0795128851947656e-2 | 1.4218402265562225e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
Pure (unsafe idx)/strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3800935242325068 | 2.391192656631271 | 2.3982876539230347 |
| Standard deviation | 6.172731518745422e-3 | 1.1630791872245637e-2 | 1.6380002360378763e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Pure (unsafe idx)/strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6110730757936835 | 1.621438062749803 | 1.6344517236575484 |
| Standard deviation | 5.157930976800758e-3 | 1.3082652943852578e-2 | 1.7996891516655823e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/base
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.637827694416046 | 5.674357505359997 | 5.690264259620259 |
| Standard deviation | 1.0225772101852139e-2 | 2.681279721150718e-2 | 3.6735075416381456e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.410884842276573 | 3.4603390203167996 | 3.4777270521347723 |
| Standard deviation | 1.8646822621424253e-3 | 3.3097093294198306e-2 | 4.015593940313717e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.0094971591606736 | 3.0466641428259513 | 3.0758797690893216 |
| Standard deviation | 1.597856678775121e-2 | 3.812793441501523e-2 | 5.149203196304982e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/tailrec/base
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.910259452648461 | 3.9171399011587105 | 3.922488048672676 |
| Standard deviation | 3.1229257583618164e-3 | 7.13441491772394e-3 | 9.74787962632509e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/tailrec/strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6882026884704828 | 1.6935315146110952 | 1.6975928354077041 |
| Standard deviation | 2.813728819146691e-3 | 5.130884606409714e-3 | 6.166932867714603e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
Arr/tailrec/strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.9437559749931097 | 0.9621164494504532 | 0.9819784052670002 |
| Standard deviation | 1.0363897308707237e-2 | 2.3941378589873195e-2 | 3.0434501679514385e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
Arr/tailrec/unsafe + strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.9174792841076851 | 0.944078465923667 | 0.9538939148187637 |
| Standard deviation | 8.163480414917247e-4 | 1.7944097092001683e-2 | 2.266780824720456e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/tailrec/unsafe + strict + LLVM + no read
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.7802201602607965 | 0.7885221159085631 | 0.8026247415691614 |
| Standard deviation | 2.337506040930748e-3 | 1.3611268861680114e-2 | 1.7633869184817623e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
C++ FFI
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.9111685231328011 | 0.9148959179098407 | 0.9201243193820119 |
| Standard deviation | 1.9671525806188583e-3 | 4.971105861116358e-3 | 6.3365773676847224e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.