File schemas

Field-by-field reference for the record shapes the pipeline reads and writes. For advice on which output you want, see Which output do I need?.

Files live in the data directory — ~/Library/Application Support/consensus-flow-list/ on macOS, overridable with CONSENSUS_FLOW_LIST_DATA_DIR.

schema_version

Every wrapped file carries schema_version, currently 6 for the flow artifacts. It is the signal a reader uses to tell "this artifact does not have the field I want" from "this artifact is older than that field": domain.schema.check_schema_version raises on a version it cannot read, rather than letting a caller merge against fields that have moved.

The two LCIA artifacts are versioned separately, by LCIA_SCHEMA_VERSION, currently 1. lcia-factors.json.gz and lcia-differences.json describe characterisation factors rather than flows, and a flow-list bump has nothing to say about a factor; a reader of one passes expected=LCIA_SCHEMA_VERSION to check_schema_version. That one number covers both the files and the lcia_* tables, because both are renderings of the same records.

Bump it only when a change would break a reader. Adding an optional field to a record does not; adding a top-level key to a document does, because the documents are additionalProperties: False.

elementary-flows.json

Written by build.

{
  "schema_version": 1,
  "stats": { … },
  "elementary_flows": [ <ElementaryFlow>, … ]
}

ElementaryFlow

Required on every record: elementary_flow_id, flow_object_id, source, context, context_iri, unit, unit_iri, lcia_methods, general_comment, cas_match_labels. The remaining four are omitted when unset rather than serialised as null, so an active flow does not carry empty deprecation keys.

Source-specific keys not listed here are round-tripped verbatim at the top level, so an unrecognised input field is preserved rather than dropped.

cas_match_labels maps each CAS number to how confidently it matched — skos:exactMatch, skos:closeMatch, or skos:relatedMatch.

Invariant: (flow_object_id, context_iri) is unique across all non-deprecated elementary flows.

Every record in this file now has the same shape, whichever path produced it. Flows added by the merge used to be built as raw dicts and omitted fields that transform-created flows carried; they are now built through the record classes, so one schema covers both.

SourceRef

Not present in the published layered file — see the warning above. This is the shape it has in memory, in the elementary_flow_sources SQLite table, and in the merge report.

{
  "list_name": "EF",
  "list_version": "3.1",
  "source_flow_uuid": "0000b186-aea3-4c0a-b0c2-c284de7cdf92",
  "source_flow_name": "…",
  "source_metadata": {
    "input_file": "/…/ef-31-flows.json",
    "input_dataset": "EF 3.1",
    "original_context": ["Emissions", "Emissions to air", "Emissions to air, indoor"]
  }
}

source_flow_name and original_context are verbatim as the source list wrote them, not normalised.

ConceptAssociation

{
  "@type": "xkos:ConceptAssociation",
  "xkos:sourceConcept": {
    "@id": "https://vocab.brightway.one/ef/3.1/flow/<uuid>",
    "http://www.w3.org/2004/02/skos/core#prefLabel": "<source flow name>",
    "context": "Emissions/Emissions to air/Emissions to air, indoor",
    "qudt:hasUnit": {"@id": "https://vocab.brightway.one/units/unit/KiloGM"},
    "http://www.w3.org/2004/02/skos/core#exactMatch": {"@id": "<target IRI>"}
  },
  "xkos:targetConcept": {"@id": "<target IRI>"},
  "provenance": { … },
  "qudt:conversionMultiplier": 9.41
}

There is no xkos:mapType. XKOS defines no property for the type or strength of a mapping — the specification says so explicitly. The match kind is carried on the source concept as the SKOS mapping property itself. Readers should look for whichever of skos:exactMatch, skos:closeMatch, skos:broadMatch or skos:relatedMatch is present.

skos:broadMatch is used for manual ecoinvent additions, where one consensus flow groups several source flows. Everything else uses skos:exactMatch.

qudt:conversionMultiplier

Multiply an amount of the source flow by it to get the amount of the target flow.

It is published only where units.json could not already have supplied the factor. Litres to cubic metres, tonnes to kilograms, MJ to J: those are properties of the units themselves, carried on the unit via conversion_multiplier and reference_unit_iri, and restating them per mapping would be a second copy of a fact that has a home. Two kinds of conversion have no such home, and both are here.

Crossing quantity kinds. Mass to Energy, Volume to Mass. There is no unit-table factor to defer to, because how many megajoules a kilogram is depends on the substance. Seven ecoinvent flows, in every version from 3.8 to 3.12:

Changing what is measured. ecoinvent gives the ore; EF 3.1 gives the metal. Both sides are in kilograms, so the units raise no objection at all, and the unit table's answer for the pair — 1.0 — is the wrong one. 0.599 is titanium's share of titanium dioxide by mass (47.867 / 79.866):

ecoinvent replaced the ore flows with elemental Titanium at 3.10.1, so these three appear in 3.8 and 3.9.1 only. Barite, 15% in crude ore is deliberately not here: it maps onto EF 3.1's baryte, which is the same compound, so no conversion applies. It would need one only against elemental barium, which is not the target this project takes.

Crossing a time dimension, and only where a curator said so. Three obsolete ecoinvent land-occupation flows are published in m² against an m²·a flow:

A factor across a time dimension asserts a duration, which is a modelling choice rather than a fact about the land. Every ecoinvent correspondence table states 1.0 for these, commented "Assumed conversion based on land use through an entire year" — and the merge refuses a time-crossing factor that arrives from a table, so that assumption is never inherited silently. The same number is asserted in ecoinvent-match-overrides.json instead, where it carries its reasoning: the occupation lasted one year. An inventory using these flows for a shorter or longer occupation is mischaracterised by exactly the ratio, which is why ecoinvent made them obsolete.

One source list, one flow, two units. A list can ship the same flow twice in two spellings of one scale. BAFU does: Noise, Road, Lorry, Average in kilometres and again in metres, and the same for the passenger car. Both rows land on one consensus flow, because (flow_object_id, context_iri) is unique, and the flow has to declare one unit.

It declares the one units.json makes the reference unit for that quantity kind — the metre for a length — so the answer does not depend on which row the merge read first. It used to: the lorry came out in metres and the passenger car in kilometres, from one list, on one day.

Where none of the units offered is a reference unit, the table has no opinion and a curator writes one into created-flow-unit-decisions.json, naming the row whose unit wins. BAFU's Heat, waste is the case that exists: megajoules against kilowatt-hours, where the coherent unit for an energy is the joule and BAFU offers neither. Its entry chooses megajoules, because that is what BAFU itself writes in the other thirteen compartments it ships the flow in. Where neither a decision nor a reference unit settles it, the first row stands and the run says so — and the row is listed under Checks → Unit disagreements as needing a decision.

Either way, a row whose own unit is not the one the flow states is flagged has_unit_mismatch on its merge outcome, which means on a creation exactly what it means on a match: this source row is measured in something other than the flow it landed on. Which of the two situations produced it is the outcome's own outcome column.

Each row keeps its own unit where its own unit belongs: on its source ref and on its qudt:hasUnit. No multiplier is published for the difference, by the rule above — kilometres to metres is exactly what the unit table already says. A group whose units share no reference unit is left as it arrived and reported, because choosing there would be picking rather than deciding.

One source list, one resource, two quantity kinds. The harder version of the same case, where the two units are not one scale. BAFU ships standing wood by volume and again by mass, and no reference unit joins a cubic metre to a kilogram. There the list is rebased by hand, in that list's manual fixes, and the fix states the factor:

0.49 oven-dry tonnes per cubic metre of fresh volume, a 50:50 mix of Picea abies and Fagus sylvatica from Table 4.14 of the 2006 IPCC Guidelines. The mapping states kg, because that is the unit BAFU's amounts are in; the flow is in m³, because that is what every other standing-wood flow in every list uses. This is the only kind of multiplier that does not come from a correspondence table — BAFU publishes none — and it is why one can be authored on a fix at all. See Rebasing a row onto another unit.

Read the multiplier with the units on both ends: qudt:hasUnit on the source concept, and the target flow's own unit. The number is bare, so a consumer that assumes one pairing gets the others wrong — 36.0 is per standard cubic metre, not per kilogram, and 1025.0 is a density rather than a calorific value.

Absence does not mean 1.0. It means no factor was stated. A pair whose units disagree and which carries no multiplier is either a reviewed exception in unit-change-allowlist.json or a defect; treating it as parity is the fossil-depletion underestimate of #257. One such pair exists today, documented rather than converted: ecoinvent's Manganese-55 (kBq) onto elemental manganese in kg, where 55Mn is manganese's stable isotope and so has no activity to convert by. No ecoinvent 3.8 dataset uses the flow, so nothing an inventory carries is affected.

The values are the EF 3.1 LCIA method as implemented by the ecoinvent Centre, reaching this project through the randonneur_data correspondence tables. They are not in the JRC's own EF 3.1 distribution: every one of the target flow datasets in EF-v3.1.zip carries a single flow-property meanValue of 1.0, and no characterisation factor in its LCIA methods has these values. The ecoinvent implementation is where they exist.

What none of them states is a basis — a calorific value is net or gross, and 26 MJ/kg dry-ash-free is a different claim from 26 MJ/kg as-received. Uranium's is documented as a gross (HHV) fission energy content; the coal and oil values say nothing either way. Giving the field a unit, an explicit basis and provenance is tracked in #70.

It never appears on a flow or a flow object. A factor is a statement about a pair — 9.41 is not a fact about brown coal until it is 9.41 MJ per kg — so it belongs to the mapping and to nothing else.

Source IRI prefixes:

stats

Counts from the run that produced the file, present on both layered files:

flow_object_count, elementary_flow_count, override_hits, cas_conflict_guard_hits, nuclide_object_hits, nuclide_object_count, element_flow_object_count_added, element_flow_object_names_added, element_flow_object_count_enriched, element_flow_object_count_ignored_unlinked, isotope_flow_object_count_added, isotope_kbq_candidate_flow_object_count, isotope_kbq_matched_flow_object_count, isotope_kbq_unmatched_flow_object_count, isotope_kbq_unmatched_reasons, isotope_kbq_withheld_flow_object_count, isotope_kbq_withheld_reasons, isotope_rows_unparsed_count, isotope_unlinked_element_count, isotope_unlinked_elements, short_lived_isotopes_not_in_consensus_count, monoatomic_ion_flow_object_count_marked, monoatomic_ion_charge_detected_count, monoatomic_ion_cas_updates, monoatomic_ion_ec_updates, duplicate_elementary_group_count, deprecated_elementary_flow_count.

The isotope counts split three ways rather than into matched and unmatched. A label that is not a nuclide name is the expected outcome for an aggregate; a nuclide the tables have never heard of is a correction waiting to be written; and a record that failed its own consistency checks is neither. One number covered all three, which is how 37 wrong records and one unresolvable label sat behind the same figure. isotope_kbq_unmatched_reasons and isotope_kbq_withheld_reasons are objects keyed by reason, and isotope_unlinked_elements is keyed by element name. See What kind of thing is this flow?.

flow-objects.json

Written by build.

{
  "schema_version": 1,
  "stats": { … },
  "flow_objects": [ <FlowObject>, … ]
}

FlowObject

origin_qualifier is one of biogenic, fossil, land_use_change, biogenic_resource_correction, biogenic_100yr, green_water, blue_water, grey_water, alpha_emitters, delayed_emission_correction, biogenic_delayed_emission_correction, fossil_delayed_emission_correction. See Flow objects and elementary flows.

Property keys are ChemROF IRIs:

Property values are wrapped as {"@value": …} with provenance alongside. A value may be a list where several sources contributed — see Known limitations, which explains why that is a problem worth knowing about.

harmonised-flows-simple.json.gz

Written by build, updated by build. Gzip-compressed. The published export — this is what downstream consumers should read.

{
  "@context": { … },
  "schema_version": 5,
  "flows": [ <SimpleFlow>, … ],
  "redirects": [ <FlowRedirect>, … ],
  "concept_schemes": [ <ConceptScheme>, … ],
  "correspondences": [ <Correspondence>, … ]
}

Six top-level keys, all required. concept_schemes and correspondences hold the mappings to other flow lists, which moved out of the flows in schema version 4; see JSON-LD.

Deprecated flows are excluded from flows. Every flow has the same fields regardless of source.

FlowRedirect

One per deprecated flow: which surviving flow its identifier resolves to, and whether the two were ever the same flow. Added in schema version 5 (#265).

The key is always present, and empty rather than absent when nothing was deprecated: a missing redirects would leave a consumer unable to tell "nothing was deprecated" from "an export written before redirects existed", which is the ambiguity the key exists to remove.

SimpleFlow

No concept_associations: the links to source-list flows left the flow in schema version 4 and are the top-level concept_schemes and correspondences, one correspondence per source list.

Present in harmonised-flows.json but not here: lcia_methods, uuid, elementary_flow_id, name, context, general_comment, synonyms, cas_match_labels, flow_object_id, _sources, input_datasets.

Note the shape difference from the layered files: labels are plain strings rather than language-tagged objects, and property values are unwrapped from their {"@value": …} envelope, losing the provenance that goes with it. Property keys are not shortened — they are the same ChemROF IRIs the layered records use. This file is built for consumption, not for round-tripping.

harmonised-flows.json

No longer a file. Every flow after all processing, before layering. It was written as a flat list — several gigabytes, and a denormalised duplicate of elementary-flows.json joined to flow-objects.json — until #82. The record is still produced and still the one the published export is projected from: it is the flow_json column of elementary_flows in consensus-flows.sqlite3, read by pipeline.sqlite.read_published_flow_payloads.

That column no longer holds the whole record. The keys describing the substance rather than the occurrence — altLabel, properties, references, prefLabel, @type, the SKOS definition, and RO:0000087 roles — are stored once per flow object in flow_object_payloads, because 94,433 flows share 7,730 objects and a copy each came to 1.57 GiB. read_published_flow_payloads joins them back, and a key a flow still carries wins over the object's, so the record it returns is what the column used to hold. Reading flow_json directly, without that join, now gives a partial record.

The section keeps the old filename because that is what the schema file is called; see the warning at the top of this page.

{ … }   -- one object per row, in `elementary_flows.flow_json`

Selected fields:

Unrecognised source keys are preserved at the top level.

Required on every record: _provided, _sources, _transformed, altLabel, cas_numbers, concept_associations, context, context_iri, ec_numbers, flow_object_id, identifier, input_datasets, lcia_methods, properties, references, source, synonyms, unit_iri, uuid.

The underscore keys are transform state — readers should skip them

_provided, _sources and _transformed are the transform stage's working state, and the underscore is the signal: they are in flow_json because that column is where the transform keeps its records, not because anything reading a flow wants them. The published export drops all three — _strip_lcia_from_flows in pipeline/exporting.py projects an explicit field list, and none of the three is on it — and the review webapp, which hands flow_json to its templates wholesale, renders none of them either.

If you are reading flow_json for the flow record, ignore every key that starts with an underscore. What does read them is the pipeline itself — the transformer chain and the merge, which run over these records rather than consume them — and that is exactly what makes them working state. A reader that treats them as part of the record is reading the pipeline's notes-to-self.

They are not free. Measured on the 2026-08-07 build, across all 94,433 rows, after #248 hoists the substance body into flow_object_payloads:

So a bit over a quarter of the 350.7 MiB the column holds is state no reader wants. That is the trade this page is recording rather than fixing: the keys stay where the pipeline writes them, and readers skip them (#251). Whether they belong in a table of their own, or on disk at all, is still open there.

_transformed and _provided

The chain is ordered for a single pass over raw input and is not idempotent: normalize_name_case runs fifth and re-title-cases labels that consensus_match sets at fourteenth. _transformed records that the chain has finished with a flow, so a later stage — the merge, which shows the transformers the consensus flows alongside each new source list — can hand both to the same loop without either re-transforming the consensus or having to say which side is which.

_provided holds name, synonyms, context, cas_numbers, ec_numbers and unit as the source list gave them. The pipeline replaces as well as adds: bootstrap_labels moves name into prefLabel and then purges name and synonyms, default_context_mapping rewrites context into a consensus context dict, and the CAS transformers substitute a different registry number for the one the list gave. Without this there is no way back to the input, and the merge — which matches on the source list's own names and keys its context lookup on the source list's own strings — has nothing to read.

It is not a copy of the input row: it holds what would otherwise be destroyed.

_sources

_sources maps a field name to the transformer that last wrote it. Unlike the other two it has no reader: apply_transformers writes it (pipeline/engine.py), and nothing in src/ reads it back — not the merge, not the export, not the webapp. The only code that looks at it is the test asserting it gets written.

It is also already stored elsewhere, twice over. The same loop that sets flow.pipeline_sources[field] appends a ChangeEvent for that identical edit, which is what changelog and changelog_flows are written from: changelog holds the transformer and field of every edit, changelog_flows holds which flows it landed on. _sources is that join with everything but the last write per field thrown away — 34.4 MiB restating, less completely, what 273,384 changelog rows already say.

That makes it the weakest of the three: the argument for keeping _provided is that the input is otherwise unrecoverable, and for _transformed that the chain reads it within the run, but neither applies here. It stays because #251 chose to document this state rather than move it, and dropping a persisted field is a schema change; it is the part of that issue most likely to be revisited.

source_refs is stripped from this file on write, exactly as it is from elementary-flows.json. For source traceability use the elementary_flow_sources table in consensus-flows.sqlite3, or the merge report for merged flows.

As with the layered file, every record now has the same shape regardless of whether the transform or the merge produced it.

Source list flows — input format

The shape ef-31-flows.json and every SourceList.flows_path are read in. There is no separate "additional input" any more: a list is a --source, and the same normaliser reads both sides. See Choosing sources.

Either shape is accepted:

[ <InputFlow>, … ]

{"flows": [ <InputFlow>, … ]}

(flow_data is also accepted as the wrapper key.)

InputFlow

Non-object rows are ignored. A base file yielding no valid rows fails the run.

consensus-flows.sqlite3

A denormalised cache rebuilt on every transform, backing the review applications. Not a durable artifact — do not edit it.

Query examples are in Recipes.