Which output do I need?

A pipeline run writes a lot of files. Most of them are caches or review artifacts; only a few are the actual product. This page tells you which is which.

All of them are written to the data directory:

Set the CONSENSUS_FLOW_LIST_DATA_DIR environment variable to override this — useful for keeping a test run away from real artifacts.

Start here

harmonised-flows-simple.json.gz — the published list

This is what most consumers want. Gzip-compressed JSON, one entry per non-deprecated flow, with a consistent set of fields regardless of which source list the flow came from. Characterisation factors, internal identifiers, and processing bookkeeping are stripped out.

{
  "@context": { … },
  "schema_version": 5,
  "flows": [
    {
      "identifier": "0000b186-aea3-4c0a-b0c2-c284de7cdf92",
      "@id": "https://vocab.brightway.dev/elementary-flows/0000b186-…",
      "@type": ["http://www.w3.org/2004/02/skos/core#Concept",
                "https://w3id.org/chemrof/NeutralMolecule"],
      "source": "EF 3.1",
      "cas_numbers": ["64896-70-4"],
      "ec_numbers": ["807-840-4"],
      "context_iri": "https://vocab.brightway.one/flow-contexts/envi-air-indr-unkn",
      "unit": "kg",
      "unit_iri": "https://vocab.brightway.one/units/unit/KiloGM",
      "prefLabel": "…",
      "altLabel": ["…"],
      "properties": {
        "https://w3id.org/chemrof/molecular_formula": "C22H38O6",
        "https://w3id.org/chemrof/smiles_string": "CCCCCCCC(=O)OC1COC2C(OC(=O)CCCCCCC)COC12"
      },
      "references": ["https://…"],
      "definition": ["…"]
    }
  ],
  "redirects": [
    {
      "identifier": "0005ab9c-ad0b-4776-9ea5-5e6374140008",
      "replaced_by_identifier": "d626c3bb-c0b0-427d-a671-af3de83d1df4",
      "@id": "https://vocab.brightway.dev/elementary-flows/0005ab9c-…",
      "http://purl.org/dc/terms/isReplacedBy": {"@id": "https://…/d626c3bb-…"},
      "http://www.w3.org/2002/07/owl#deprecated": true,
      "https://vocab.brightway.one/terms/deprecationReason": {
        "@id": "https://vocab.brightway.one/deprecation-reasons/context-collapse"
      }
    }
  ],
  "concept_schemes": [ … ],
  "correspondences": [ … ]
}

Use it when you want a substance list, a name-to-identifier lookup, or a translation table.

flows holds only non-deprecated flows, and redirects says where every identifier that left it went — 130 of them on the 2026-08-12 build. A lookup that misses both is a flow this list has never carried; a lookup that hits redirects is one it merged into another.

lcia-factors.json.gz — the characterisation factors

Written by characterise, which runs after a build. Three implementations of EF 3.1 in one file: the European Commission's JRC, the ecoinvent Centre's, and this list's, which publishes the numbers the first two agree on and asks a curator about the rest.

{
  "schema_version": 1,
  "methods": [ { "id": "3c980711-…", "name": "EF", "meta": { … } } ],
  "impact_categories": [
    {
      "id": "3897dc04-68ec-5953-a76d-d40db54cc82a",
      "method": { "id": "3c980711-…", "name": "EF", "meta": { … } },
      "name": "Climate change",
      "version": "3.1",
      "implemented_by": "European Commission — JRC",
      "unit_iri": "https://vocab.brightway.one/units/unit/KiloGM-CO2eq",
      "indicator": "Radiative forcing as Global Warming Potential (GWP100)",
      "timeframe": "Long term impacts (no time horizon)",
      "meta": {"iri": "https://vocab.brightway.one/lcia/impact-category/ef/3.1/jrc/long/climate-change", "…": "…"}
    }
  ],
  "characterization_factors": [
    {
      "elementary_flow_uuid": "08a91e70-3ddc-11dd-923d-0050c2490048",
      "impact_category_id": "3897dc04-68ec-5953-a76d-d40db54cc82a",
      "amount": 1.0,
      "geography": null,
      "derivation": null,
      "source_flow_uuid": null
    }
  ],
  "stats": { … }
}

Four things to know before you read a number out of it:

• There is one method, not three. The method is EF; its version and its implementer are fields on the category, so three implementations of EF 3.1 are 75 categories under one method rather than three methods with one name.
• A factor has no identifier of its own. It is identified by the three things that make it one: impact_category_id, elementary_flow_uuid and geography. Those are the names the SQLite tables use too — an artifact and the table beside it describing one row two ways is a translation step nobody should have to make. There is exactly one factor per triple.
• implemented_by is the field that matters. There is no correct implementation of a method. EF 3.1 as the JRC published it and EF 3.1 as the ecoinvent Centre implemented it are two renderings of one method against two flow lists; both are published here, unchanged, beside this list's judgement.
• derivation is empty on a transcription. It says how this list arrived at the number — agreed where the implementations stated the same one, sole where only one spoke, ruled where a curator decided between them — and is null on the transcriptions, which arrive at nothing: they say what their publisher said.

geography is the publisher's own code, published verbatim: EF states one on 42,871 of its factors, mostly ISO 3166-1 alpha-2. Two places are two factors — Land use for one flow is -522.81 in ES-CA and -227.0 in YE.

lcia-differences.json — where the two implementations disagree

The comparison, as a deliverable rather than a page to browse. One row per (flow, category, place) triple both published implementations state, with each side's number, the ratio and a band:

{
  "schema_version": 1,
  "differences": [
    {
      "elementary_flow_uuid": "1c9d4f7b-803f-4b74-b5fa-8bf642c94b2c",
      "category_slug": "ecotoxicity-freshwater", "geography": "",
      "implemented_by": "European Commission — JRC", "amount": 6297400.0,
      "source_flow_uuid": null,
      "band": "over-100x", "ratio": 1441.48, "derivation": null
    },
    {
      "elementary_flow_uuid": "1c9d4f7b-803f-4b74-b5fa-8bf642c94b2c",
      "category_slug": "ecotoxicity-freshwater", "geography": "",
      "implemented_by": "ecoinvent Centre", "amount": 4368.7,
      "source_flow_uuid": "0ec92f76-932f-4407-bd4f-eb9aa426e099",
      "band": "over-100x", "ratio": 1441.48, "derivation": null
    }
  ],
  "coverage": [
    {"implemented_by": "European Commission — JRC", "dimension": "category",
     "value": "photochemical-ozone-formation-human-health", "flows": 881}
  ],
  "stats": { … }
}

One row per implementation, not per pair. Two implementations of one triple are two rows sharing a band and a ratio; three would be three. A column named after a source list would have to be added for every list this project ever compares, and would be missing from every file written before it existed — so the report is long form, which loads into a dataframe as it stands and groups in SQL without a pivot. 22,107 triples are 44,214 rows.

A difference is not an error. The headline of this file is the agreement: the two implementations agree exactly, same float at full precision, on 21,800 of the 22,107 triples they both state — 98.6%. Of the 307 that differ, most are modelling choices between two competent teams; the 20 over 100× are worth a conversation, not a correction.

Where only one implementation speaks there is no comparison to record, so those are counted in stats and not listed: 297,468 rows of "nobody disagreed, because nobody else spoke" is a fact about two differently sized flow lists.

coverage is the second report and a different question: where does one implementation characterise a substance in one context and skip the context beside it? It is a summary by compartment and by category and names no flow, deliberately — asked as a list it is thousands of rows, most of them a substance with no global-warming potential, where absence is the right answer.

consensus-flows.sqlite3 — everything else

The two layers, the change log, the PROV-O trail, the decision queues and the merge outcome, in one file. Use it when you need the identity/occurrence split explicitly — all the contexts one substance appears in, all the substances sharing a property — or when you want to run ad-hoc queries rather than parse JSON.

elementary_flows.lcia_factor_count is the JRC's non-zero factors and keeps that meaning now that there are three implementations, so a query written before characterise existed still answers what it always answered. The question it cannot answer — how many each implementation states — is lcia_flow_factor_counts, a view over the factors themselves, so the two cannot drift apart.

flow_objects.flow_object_json and elementary_flows.flow_json hold the whole record, so anything the old JSON layer files carried is a json_extract away. Recipes has worked queries.

Skip the underscore-prefixed keys in flow_json — _provided, _sources and _transformed are the transform stage's working state, not part of the flow. They are a bit over a quarter of the column, no reader downstream of the transform touches them, and they are absent from the published export; see the underscore keys.

Rebuilt from scratch on every transform run — do not edit it and expect the changes to survive.

Everything else

Intermediate artifacts

Two rows are gone from this table. elementary-flows-merged-ecoinvent-<version>.json was the merge's output before it wrote to the database, and additional-flow-input-*.json was a second way into the transform before a list became a --source and nothing else. Neither is written.

Review queues

These are the human-decision backlog. See The review application for the interfaces that present them.

These are tables in consensus-flows.sqlite3, not files. Each was a JSON side-car until the review web application was consolidated onto the database; three of them had stopped being written at all, and the pages reading them had shown an empty state ever since, because a missing file and an empty file look the same from a route.

ecoinvent-merge-report-<version>.json was replaced by the merge tables: it was one file per source version and could not describe a run merging several.

Logs

transform-log.json and provenance.json were these three tables, written only when a run was given --write-transform-log / --write-provenance. Both flags are gone: a review page that works only when someone remembered a flag is a review page that does not work.

Caches

chebi.json.gz, pubchem-data.json, commonchemistry-cache.json, compound-profile-cache.json, web-lookup-cache.json, wikidata-cache.json, chemlin-isotopes.json, pubchem-elements-isotopes.json, EF-v3.1.zip.

These hold downloaded reference data. Together they are tens of gigabytes and represent hours of rate-limited API traffic — do not delete them casually. Deleting one means re-downloading it on the next run.

The compound profile cache self-heals: entries recorded before the primary-CAS rule was introduced are detected and recomputed rather than trusted.

pubchem-elements-isotopes.json and chemlin-isotopes.json carry a cache_version, and both were bumped when the nuclide handling was corrected. An older file is discarded rather than read, so the first run after that change re-downloads the periodic table, 118 element pages, and one ChemLIN page per nuclide. That is deliberate and not optional: the decay modes in an older element cache are one row out of step on 18 elements, and there is no way to put them back without the blank cells that were discarded when it was written.

A caution about staleness

Files in the data directory are whatever the last run left there, and a bounded run (--max-flows 500) overwrites the layered artifacts and the SQLite database with a small subset. A flow-objects.json of a few megabytes is a test run, not a full list.

Two habits avoid the confusion:

• Point test runs at a separate directory with CONSENSUS_FLOW_LIST_DATA_DIR.
• Check the stats block and the file's modification time before trusting it.

Precise field definitions

File schemas documents every field of every output file, with types and what they mean.

If you want to validate rather than read, JSON Schemas for the four published artifacts are checked in under src/consensus_flow_list/data/schemas/. They are generated from the record classes and tested against real artifacts, so they are the authority when they and the prose disagree.