[StanFromIreland]

Sometimes, executables may be missing certain benchmarks:

This results in useless rows. However, adding an option to hide such incomplete comparisons, we get a more concise comparison:

[mattip]

What if only one executable has a result, will it show the single bar?

[StanFromIreland]

Sorry, I'm not sure I understand, can you please clarify, are there two or more executables or just one?

[mattip]

Your graph shows a comparison of two executables. What happens when one executable has a result and the other does not? Is the single result shown or not? I would think the logic should be:

• raw results: hide only benchmarks where there are no results from any executable, otherwise show whatever data exists for ones with incomplete data
• normalized results: hide benchmarks where the baseline has no results, otherwise show whatever normalized data exists

[StanFromIreland]

What happens when one executable has a result and the other does not? Is the single result shown or not?

With the option off (the default), the single result is shown exactly as before. However, if selected, any benchmark missing a result for one of the selected executables is hidden, and a note under the chart says how many were dropped.

hide only benchmarks where there are no results from any executable

I went stricter than "hide only when nobody has data" on purpose. A benchmark with a bar for one executable and nothing for the other reads as a huge regression or incredible win, when it is really just missing data, for example, the view that prompted this:

On first glance, it looks like k_core has severely regressed.

[mattip]

That is why I suggested different hiding when normalized (show any normalized results) and not normalized (show any results). k core is not normalized, so it should be hidden.

Sorry, I misunderstood the graph. Why does it look to you like k_core has regressed? The graph is not normalized.

[StanFromIreland]

Sorry, I misunderstood the graph. Why does it look to you like k_core has regressed? The graph is not normalized.

It is comparing raw data, and this is a slightly simplified example, but in my opinion on first glance it still looks like a significant difference (as one may mistake the orange bar as simply being tiny, like in neighbouring benchmarks).

[mattip]

I was thinking more of a scenario when there are three or four executables being compared, and one of them is flaky. Personally, I find the raw data confusing and always normalize.