What is data provenance?

Data provenance is the record of where a piece of data came from, who or what produced it, and what has happened to it since. A complete record names the origin, the actor, the time of each version, and something that proves the contents have not changed, such as a content hash.

The four things a provenance record has to name

Origin: where the data came from, whether that is an instrument, a run, a dataset it was derived from, or a person typing it in. An origin is what separates a measurement from an assertion.

Actor: who or what produced it. Until recently this meant a person. It now routinely means a person, an instrument and an automated agent, and a record that cannot distinguish them is already behind how the work is done.

Time: when each version came into being, in an order that can be checked rather than recalled. Sequence turns out to be most of what anyone is really asking about when they question a result.

Integrity: whether the contents are still what they were. A content hash answers this in seconds for anyone holding a copy, without their having to trust the system that stored it.

Why it is usually missing

Provenance cannot be added afterwards. A record assembled at the end of the work is an account of what people remember, which is exactly the thing the record was supposed to replace. It has to be written by the system doing the writing, at the moment the file is written, or it is testimony rather than evidence.

Keep your research in a lab

Sign up with your email and create a lab in a few minutes.