What a benchmark leaves out

The same score can hide very different behaviour. A worked example on averages, variation and the limits of a result.

Example post · Not a Duskfields research result.

On this page

A score is a summary. It is not an explanation. Before using it to choose a system, we need to know what was measured, how the observations were combined, and what the summary leaves out.

This worked example uses invented numbers. It makes no claim about a Duskfields model or experiment; its purpose is to show how two very different profiles can disappear into the same average.

The information an average removes

Consider two systems evaluated in eight conditions. In this example, every condition has the same weight and a higher score is better.

Profile A: 48, 49, 49, 50, 50, 51, 51, 52.
Profile B: 15, 25, 35, 45, 55, 65, 75, 85.

Both sum to 400. Dividing by eight gives the same mean: 50. That equality is real, but it does not tell us that the systems are interchangeable.

Two illustrative sets of eight scores share a mean of 50. Profile A is grouped between 48 and 52; profile B spans 15 to 85.
Eight conditions, equally weighted. Invented scores, not measurements of a real system.
SummaryProfile AProfile B
Mean score5050
Lowest score4815
Highest score5285
Range470

Profile A stays close to its mean. Profile B reaches much higher in some conditions and much lower in others. A single ranking would hide that distinction.

The average answers one question. It cannot answer every question we might ask of the system.

The conditions are part of the result

Neither profile is automatically preferable. If poor outcomes are costly, the lower end may matter more than the peak. If a system will only be used in a narrow set of conditions, those particular observations may matter more than the overall mean.

There is another limit here: eight invented scores tell us nothing about repeatability. We have not specified repeated trials, uncertainty, or how the conditions relate to a real use case. The arithmetic is complete; the evidence is not.

Changing the weights would also change the comparison. Equal weighting is an assumption in this example, not a universal property of a benchmark.

Make the claim inspectable

A useful presentation keeps the result close to the conditions that produced it. The reader should be able to distinguish what was observed from what is being inferred.

For an actual experimental report, that means making a few things explicit:

  1. The question. Which decision or hypothesis is the experiment intended to inform?
  2. The setup. What was evaluated, against which comparison, under what conditions?
  3. The observations. What happened across conditions and repeated trials—not just in aggregate?
  4. The boundary. Which conclusions remain unsupported, and what would need to be tested next?

These details are not a substitute for a clear result. They are what make a clear result possible to interpret.

What this example establishes

Two sets of observations can have an identical mean and very different ranges. That is all the numerical example demonstrates. It does not establish which profile would be better in deployment, or whether either would be reliable outside these eight conditions.

Keeping that boundary visible is a small but important part of reporting a result honestly.

Keep reading