criterion performance measurements
overview
want to understand this report?
Pure/base
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.293592917267233 | 5.309209522946428 | 5.331057018134743 |
| Standard deviation | 6.714443571005625e-3 | 2.1437793826786684e-2 | 2.8839878294405468e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
Pure/strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.4902448505163193 | 3.4951762249693274 | 3.5001075994223356 |
| Standard deviation | 3.223184918640454e-4 | 6.228133174410739e-3 | 7.632087213610237e-3 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
Pure/strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0960616419712705 | 2.107513846208652 | 2.1189660504460335 |
| Standard deviation | 6.95944684829887e-3 | 1.4473970621614152e-2 | 1.898484307604335e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
Pure (unsafe idx)/base
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.41637946665287 | 4.439133661178251 | 4.448025447626908 |
| Standard deviation | 2.632899209856987e-3 | 1.575453663484724e-2 | 2.0534701198836325e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
Pure (unsafe idx)/strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.6181550584733486 | 2.620990508235991 | 2.622171862050891 |
| Standard deviation | 4.234127700328827e-4 | 1.9759940079072586e-3 | 2.5584960511565343e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Pure (unsafe idx)/strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6844604685902596 | 1.6898449715226889 | 1.7004752960056067 |
| Standard deviation | 1.018717885017395e-4 | 1.0585231739810876e-2 | 1.2333657108085613e-2 |
Outlying measurements have moderate (0.18749999999999994%) effect on estimated standard deviation.
Arr/base
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 6.217116388492286 | 6.237041675640891 | 6.255234256697198 |
| Standard deviation | 1.7297518750031937e-2 | 2.1240773385775263e-2 | 2.397496752854617e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
Arr/strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.0836621932685375 | 4.1460321334501105 | 4.172066926956177 |
| Standard deviation | 3.5192687012806506e-3 | 4.4891691592980586e-2 | 5.660551145481312e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.7457999335601926 | 3.7840473630155125 | 3.807873321697116 |
| Standard deviation | 3.579307428041851e-2 | 4.33977923291775e-2 | 4.778411646715307e-2 |
Outlying measurements have moderate (0.18749999999999997%) effect on estimated standard deviation.
Arr/tailrec/base
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.683134809602052 | 4.6988912915500505 | 4.711313776206225 |
| Standard deviation | 6.996320835113156e-3 | 1.7284070233509096e-2 | 2.169217860587315e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/tailrec/strict
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.906794399023056 | 1.923723266304781 | 1.9300515990083416 |
| Standard deviation | 6.444677876693266e-4 | 1.1460253080089815e-2 | 1.4614658359688877e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/tailrec/strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.9859063783660531 | 0.994222316890955 | 1.0019737053662539 |
| Standard deviation | 7.081058779907661e-3 | 9.086871235574095e-3 | 1.0499437765178336e-2 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/tailrec/unsafe + strict + LLVM
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.9245669902302325 | 0.9283734906154374 | 0.9352149143815041 |
| Standard deviation | 9.790342301130295e-5 | 6.6031666171660714e-3 | 7.956320273587886e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
Arr/tailrec/unsafe + strict + LLVM + no read
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 0.8323013326153159 | 0.8334633220608035 | 0.8345797037084897 |
| Standard deviation | 6.421630581219806e-4 | 1.393630664769469e-3 | 1.7125028527659547e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
C++ FFI
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.3541298061609268 | 1.3556543577772875 | 1.3564664696653683 |
| Standard deviation | 2.0138381656192015e-4 | 1.4456310581448515e-3 | 1.8610924804796478e-3 |
Outlying measurements have moderate (0.1875%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.