R-words

[rougier]

The idea is to converge on definition of these words according to our respective scientific domains. At this point it is not even sure that all words are relevant to all domains and this may also depend on the kind of software we consider (see discussion in #4)

Rerunable:

Repeatable:

Replicable:

Reproducible:

Reusable:

Remixable:

Reimplementable:

[oliviaguest]

I will only attempt one because the rest seem too similar/unknown to me.

Reimplementable: Is there enough information in the specification (i.e., journal article, or referenced within the journal article) to recreate the model (within the theory or account) from scratch? If yes, then the model is reimplementable. If no, then even if the experiments can be carried out within the original (presumably opaque) codebase, then the model is not able to be reimplemented (given the current specification).

[rougier]

Rerunable: Is it possible to re-run the model (same computer, same system, same program) and get the exact same results ? It may seem obvious that the answer is yes but it is not that obvious actually. For example, if you're using a random generator and did not set or record the seed, then you cannot guarantee a re-run. Same if you fed manually some parameters when starting your model without a mechanism to save them or read them from a file that is changed after the run.

[oliviaguest]

What's the history/reason behind the capitalisation of rerun?

[rougier]

Replicable: Is it possible to re-run the model (same program) on a different computer using a different system or different version. Does your specification give enough information concerning required library and their respective version number ? Does your model relies on system specific libraries (use of system specific library) ? Does it correctly handle system-specific features (float precision, endianess, etc.)

[rougier]

@oliviaguest None, just correct it.

[rougier]

Reproducible = Reimplementable (for me)

[oliviaguest]

Ah, so for me it's more complex, like (although I might need to think about it more):

(Rerun + Reimplement) ~ Reproduce = Replicate

R-words on LHS are somehow weighted.

Edited. OK, not sure. But I think maybe best to nest them? I have seen other definitions out there.

[oliviaguest]

Something needs to be said about replicating the data also, in my opinion. If you are modelling data, then surely the experiment that produced the training data and testing data has to also be replicable.

[khinsen]

@oliviaguest Your definition of reimplementable looks fine to me, but it should be made clear that it applies to a human-readable document (such as a paper), not to software or computed results like the other R-words. I also think that this term matters to us, because it defines the ideal candidate paper for a replication to be published in ReScience.

[khinsen]

@rougier I am fine with your definition of "rerunable". The tricky part is the transition from there to "replicable".

The idea is that there are aspects of a computation that should be modifiable without affecting the results, according to expectations shared in the community. A computation is then called "replicable" if it satisfies those expectations. Typically the expectations include results independent of minor version changes in everything, and of the use of different compilers and operating systems.

The big problem is of course that these expectations are never written down explicitly, and it is unlikely that there is a complete consensus about them in any community. But without a clear list of criteria, it is impossible to verify if a computation is replicable. To make it worse, some people's expectations are about obtaining the exact same results at the bit level, whereas others consider it normal that "small" variations happen, though nobody ever seems to be able to define "small" in this context.

[khinsen]

Repeatable: = rerunable.

[khinsen]

Reproducible: the result of a computational study is reproducible if its human-readable description is reimplementable and if a reimplementation leads to results whose scientific interpretation is the same as for the original results.

Note that reproducibility can change over time, for two reasons:

1. A new reimplementation can lead to different results than any previous one.
2. The differences between the results of different implementations can become scientifically relevant if the state of the art of the field makes significant progress.

[khinsen]

Reusable: a piece of code or a dataset is reusable if its characteristics are sufficiently well described that it can safely be transferred to another context.

For a piece of code, there are two interesting relations to other R-words:

1. Reusable code is well-documented code, meaning that its documentation is reimplementable.
2. Reusable code is related to replicable computations through a clear statement of dependencies in the documentation.

[oliviaguest]

Where do changes that are not central to theory, so can be abstracted away in ideal circumstances, get relegated to? For example I have come across cases where a non-theoretically important implementation detail which should not affect the model (e.g., quicksort vs another sorting algo) ends up affecting the model because the authors were not careful. The type of sorting algo used is categorically not part of the theory and should not be, but was still integral to the replication of the results (because careful consistent modelling was not carried out). It should be part of the spec, ideally, but it was neither part of the spec nor abstracted away enough during investigations of the model so the results ended up depending on a theoretically irrelevant point.

And - very relatedly - where do details that are important to the theory but have not been discovered as such belong in this r-hierarchy of words? It is a similar but importantly different case in which an implementation detail needs to be promoted to the theory-level because it is actually theoretically important, e.g., it is important for quicksort to be mentioned as the theory depends on it and not just because of sloppy modelling.

[oliviaguest]

PS: I mentioned "r-word" as a slightly flippant comment. I am now a little sorry it has caught on as it makes me feel conflicted.