Data provenance

Data provenance: what it is and what a record contains

Data provenance is the record of where a piece of data came from and what has happened to it since, including who or what produced each version.

One file, end to end

The full provenance record for a single file

The file's origin and every later version sit in one record, each with its author and date.
A file's record: its fingerprint, version 5 recorded today, and the earlier versions with who made each and when.

How to capture data provenance

Each practice below depends on recording at the moment the work happens, since a history added later can't be checked.

01

Record when the file is written

The system that writes the file should create the record at the same moment.

02

Give every actor its own identity

When several people or tools share one login, the record can only name the login. Each person needs a separate identity, and so does each instrument or agent, for the work to be attributed correctly.

03

Add versions instead of overwriting

Saving a new version keeps the earlier state available, so anyone checking later can see what the data looked like before it changed.

04

Keep the access history

Who could open each file, and when, is part of its provenance, so changes to access need to be recorded alongside the files.

What a data provenance record contains

A complete record covers origin, actor, time and integrity. Each one answers a different question a reviewer is likely to ask.

Origin

Where the data came from, such as the instrument run that produced it or the dataset it was derived from. If the origin is missing, nothing later in the record can be traced back to a source.

Actor

Who or what produced the data. In many labs that now includes AI agents and automated instruments as well as people, so the record needs to show which one did the work.

Time

When each version was created, recorded in an order that can be checked. Much of what a reviewer asks comes down to sequence, such as what existed before a result and what changed after it.

Integrity

Evidence that the file's contents haven't changed. A content hash lets anyone with a copy check this in seconds.

Reading the record

What a provenance record can and can't tell you

Provenance shows what happened to a file and who did it. Whether the science holds up is still a judgement for the people reviewing it.
A file's versions, each with who made it and when, with one version highlighted.

How provenance differs from data lineage

Data lineage traces how data moves and changes through a pipeline. Provenance starts earlier and goes further, recording where the data originated and who or what produced it, along with proof that it hasn't changed since, which makes it closer to a chain of custody.

Why provenance is harder in research

A warehouse pipeline runs the same steps every night, so its lineage can be read from the pipeline itself. A research result may draw on a one-off experiment and a collaborator's spreadsheet, and increasingly on work by an AI agent. The person who could explain it could move on when their contract ends, so the record has to hold that explanation instead.

Common questions about data provenance

Data provenance is the record of where a piece of data came from, who or what produced it, and what has happened to it since. A complete record names the origin, the actor, the time of each version, and something that proves the contents have not changed, such as a content hash.

See what this looks like in a real lab

For every file, a lab keeps a version, a date, a content hash and who made each change, by default, including when the work was done by an agent rather than a person.