Harmonisation steps

A transform runs twenty-four processing steps ("transformers") over the flows, in a fixed order, and then thirteen further passes over the two layers those steps produce. This page says what each one does and why it sits where it does.

For the reasoning behind the rules — why PubChem is distrusted, what the guards protect against — read How decisions are made first. This page is the ordered inventory.

How a step works

No step edits a flow. Each reads the flows it is shown and hands back a list of proposals — this flow, this field, this new value, and a sentence saying why. The run applies them and records every one it applied.

Three consequences matter when you are reading the output.

• The change log is complete. Nothing reaches a published flow without a row in the changelog table naming the step that wrote it and the reason. That is what /changes, and the history on a flow's own page, are showing you.
• A later step overrules an earlier one. Proposals are applied in step order, and for the same field on the same flow the last one wins. Carbon Dioxide is title-cased at step 5 and may be replaced outright at step 14; the published label is step 14's, and step 5's attempt is still in the log.
• Each flow goes through the chain once. The order assumes raw input — step 5 would re-title-case what step 14 wrote — so a flow is marked finished when the chain has been through it, and nothing may write to it again. This is what lets a later source list be enriched alongside the settled consensus without disturbing it.

What each step is shown

Nineteen of the twenty-four steps ask a question about one flow: what this registry number resolves to, what this structure implies, how this flow's own names should be spelled. Those are shown only the flows the run can still write to.

Five ask a question about the set, and are shown every flow, finished ones included:

The distinction is why bringing in a small list is now quick. Building the base list shows every step every flow, because nothing is finished yet. Bringing in a source list afterwards is different: the consensus is already settled, so a step that answers per flow sees only the arriving rows. Before that was true, adding ecoinvent 3.8's 4,892 flows cost as much as building the 94,000-flow base list, because every step re-derived an answer for the whole consensus and the run discarded it — about 210,000 discarded answers per merged list. Merging one list went from 189 seconds to 60 (#445).

A step that does not say which kind of question it asks is shown everything. That costs time and cannot cost correctness.

The order

A PubChemTransformer exists that proposes name updates from PubChem compound data. It is not in the default chain — PubChem names are not trusted as a primary naming source.

A PubChem readable name step used to run just before the dedupe, promoting a PubChem record title over the current label. It was removed in #207: it rewrote 17,942 published labels a run from a single source with no agreement requirement and no curator ruling, and being last in line its label was the one that shipped. Nivalenol was published as Epitope ID:2151205.

Phase by phase

Steps 1–5: make the data comparable

Nothing here makes a judgement. Labels move into the structured SKOS form, compartments resolve to consensus contexts, units resolve to the canonical vocabulary, whitespace and casing are normalised.

These run first because every later step compares flows to each other, and comparison is only meaningful once representation is uniform. Unit normalisation in particular has to happen before matching, since a unit that does not resolve fails the run at export.

Context resolution is where the context_iri is set. From this point on the IRI, not the display strings, is the authoritative form of the context.

Steps 6–9: fix the identifiers

Registry numbers are checked before anything is enriched from them.

• Common Chemistry CAS review reconciles CAS numbers against the highest-quality source available, and records disagreements for review rather than silently overwriting.
• Registry number from the name fills in a number where the row has none and its own name says which it is. BAFU ships four rows called Silver-110; two of them carry 14391-76-5 and two carry nothing at all, and the two without it ended up published as a second silver-110 with no half-life, no decay mode and no characterisation. A row takes the number the same list gives that name on another row, and only where the list is silent does it take the one ChEBI and PubChem publish — which is what keeps BAFU's Barite on the number BAFU gave it, 13462-86-7, rather than on barium sulfate. Nothing is written where a name reaches more than one number: Borax is the decahydrate and the anhydrous salt, and a name that means two substances has identified neither.
• Check digits catch transcription errors — CAS and EC numbers both carry a check digit — and propose corrections where the intended number is unambiguous. Failures go to the ec-malformed review queue.
• EC cross-check tests each CAS ↔ EC pairing against the official ECHA inventory. Contradictions go to the ec-cross-check review queue.

Order matters: enrichment steps look substances up by their identifiers, so a wrong identifier here becomes wrong chemistry later.

Steps 10–14: enrich, then clean up

Enrich references and ChEBI synonyms pull properties, cross-references and synonyms from ChEBI and PubChem. Both apply the CAS cross-check and formula guards described in How decisions are made — this is where a mis-linked ChEBI record would do the most damage.

Enrichment resolves a CAS to PubChem compounds in three narrowing passes, in this order:

1. The curated-CAS gate. A compound whose own curated CAS section does not list the number searched on is dropped, provided another candidate is attested. This runs first because it is the only one that consults PubChem's own judgement rather than inferring one, and the only one that works on a flow carrying neither a formula nor a ChEBI id — which is most of them at this point in the chain.
2. Formula similarity, where the flow already has a molecular formula.
3. ChEBI alignment, where the flow already has a ChEBI id. If the CAS was ambiguous and no candidate carries a matching ChEBI cross-reference, all of them are rejected rather than one being picked.

Passes 2 and 3 ask whether the CAS was ambiguous, not whether it still is — so the gate in pass 1 cannot make an ambiguous CAS look settled and let a candidate through that pass 3 would have rejected.

RDKit (pre-consensus) computes structural properties from whatever structures are now available, so consensus matching has them to weigh.

Then two cleanup passes remove synonyms that would mislead the matching step:

• synonyms identical to a different flow object's preferred name;
• element symbols (Rn, [Rn]) on anything that is not the pure element.

Both deliberately run before consensus matching, so the scoring step never sees a contaminated label set.

Step 15: consensus matching

The main naming and identity decision. It evaluates candidate preferred and alternative names across ChEBI, the EC inventory, PubChem and Common Chemistry; classifies how identifiers and names relate; and applies an update only when explicit confidence rules are met — typically several independent sources agreeing.

Where a CAS is unique within its context, name selection uses readability and complexity scoring with a margin requirement, so a marginally-better candidate does not displace an established name.

When the evidence does not meet the bar it does not guess. The case goes to the consensus-match review queue, at /queue/consensus-match. Changes it does apply are in the changelog table, under transformer = 'consensus_match'.

Steps 16–18: derive structure authoritatively

Once identity is settled, structural data can be recomputed rather than inherited.

OPSIN parses IUPAC names into structures. A flow qualifies when exactly one unique label across its preferred and alternative names parses — compared case-insensitively. For those flows the IUPAC name property is set to that label, and SMILES and InChI are replaced with values derived from the parse. All labels for the run are batched into a single OPSIN call, so the JVM starts once.

RDKit (authoritative) then replaces every structural property — SMILES, InChI, InChIKey, formula, masses — but only where identity is unambiguous:

1. exactly one IUPAC name value, set by OPSIN in this run or seeded from PubChem in an earlier one;
2. an InChI resolving to a standard-layer InChIKey that is globally unique across all flows.

OPSIN deliberately leaves InChIKey, formula and masses to this step, so they are all derived from one authoritative InChI rather than from a mixture of sources.

RDKit (post-consensus) fills in structural properties for the flows that did not qualify, without overriding what is already there.

Both RDKit stages ask whether the flow already carries the structure, not whether it carries the string. InChI, InChIKey, formula and masses are canonical across toolkits, so those match on the string as they always did; a canonical SMILES is canonical only per toolkit, so those are compared after canonicalisation. Where the structure is already there, RDKit is recorded as a further source on it rather than adding a second spelling — see one structure, one value.

Neither stage adds an InChIKey with no stereochemistry to a flow that already carries a stereochemistry-bearing key for the same skeleton, and a key that does carry it withdraws a flat one already there. The flat key is the same substance with a layer deleted, and it is another substance's real key — the two side by side in one slot are indistinguishable from two candidate identities. It is still computed and still published for a flow that has no more specific key for that skeleton. See Known limitations.

Steps 19–24: final polish

• Withhold contradicted composition withdraws a structure whose composition the registry's own formula contradicts. A registry writes a salt as its acid and a ratio of metal — nickel acetate is C2H4O2 · 1/2 Ni, two acetic acids to one nickel — and neither SMILES nor InChI can hold half an atom, so the structure derived from that formula loses the ratio and describes one acetic acid beside one nickel. Both end up published. This step multiplies the ratio out, ignores hydrogen (a salt is its acid with the acidic hydrogens gone) and compares what is left against each published formula.

  It withdraws only where the registry backs at least one of the substance's own formulas: where it agrees with none of them the disagreement is real and the row stays open for a curator, and a substance publishing a single formula is never touched. The same test cuts both ways — it keeps the name-derived answer for nickel acetate and withdraws it for 1,4-diazabicyclooctane, whose incomplete name parses to a different molecule. See Salts: the formula and the structure do not say the same thing. - Withhold ambiguous mass publishes molecular_mass only where one structure is settled — a mass is a property of a structure, so a flow carrying several candidates has none to report. - Strip qualifier synonyms removes synonyms from a qualified flow that are not specific to its qualifier — so the biogenic CO₂ flow does not accumulate names that describe fossil CO₂. - Strip catalogue synonyms removes the strings that are catalogue entries rather than names: supplier grades (S 100, A 1), trade names with a product number (Garlon 480, Prifrac 2981) and database accessions (NSC 147337, C.I. 77120). It runs here because both writers of an alternative label have finished by now — ChEBI at step 11 and Common Chemistry at step 14 — and before the dedupe, so the deduplicated list is the published one.

  It matches on the shape of the string, so it can be wrong; when it is, altlabel-keep-list.json overrules it for a named label without a code change, and the step's change-log comment names every value it removed. See A product code is not a name. - Strip product families removes what the shapes above cannot see, because the evidence is not in any single label. Silicon Dioxide published 4,003 synonyms, 173 of them beginning Snowtex and 126 Aerosil (#242); a brand token matches no anchored prefix list and there is an open-ended supply of them. Repetition within one object is the signal instead: several labels sharing a leading token and ending in a free-standing designation are a supplier catalogue, not a naming history.

  It runs only on objects carrying at least 100 alternative labels. That is a scope gate rather than a quality signal — the median object has 9 synonyms and the 96 above 100 hold 23.9% of every synonym in the list, so the rule is confined to where a repeated token has no innocent reading. It takes 8,643 labels from 69 objects, 6.6% of the published total.

  Names that are shaped like product codes are settled by a register rather than a pattern, because Ponceau 4R and Acetoquinone Blue R do not differ as strings: ChEBI is consulted at run time and the public part of the Colour Index ships in colour-index-names.json. Corroboration may only keep a label, never publish one. Methyl Violet 10B is in no open register and is the documented gap; the keep list covers that case. - Dedupe synonyms removes duplicates, keeping the first occurrence so provenance ordering survives.

After the steps: the layering, and the passes that follow it

When every step has run, the flows are split into the two published layers, and fourteen further passes run over the result. Much of what you will notice about the published list is decided here rather than in the chain above: what kind of thing each substance is, which occurrences are duplicates of one another, and which source-list flow each one answers to.

Each pass counts its own work into run_stats, under the stage name below, so a number that moves between builds can be read against the pass that moved it. Figures quoted in this section are from the 2026-08-14 build unless another is named.

A pass that writes to a flow writes through the same change log the steps do. So a change in a flow's history with no step behind it is one of these, and run_stats files them together under layer_writes, by pass. Before that was true, a published label being replaced outright — or a flow being deprecated — happened with nothing in the log to show for it.

1. Flow objects and elementary flows are separated. Substances are grouped by identity; occurrences are grouped by (flow_object_id, context_iri). Identity is the CAS number, falling back to the normalised name — except for a flow whose name is exactly a nuclide, which is grouped by that nuclide. Both source lists put the element's registry number on some nuclide rows, and grouping on it merged uranium ore with a single nuclide (#200).
2. Origin qualifiers split flow objects. Flows named as biogenic, fossil, land-use-change, or as green/blue/grey water get their own flow object linked to the base substance. Detection reads the flow's current display name first and falls back to the original names in source_refs — necessary because a naming step may already have normalised the qualifier out of the preferred name. Where the qualifier has to be recovered that way, the original qualified name is restored as the preferred label. Among names sharing a qualifier and CAS, the shorter wins.
3. A registry number that two flow names share is a question, not a merge. The layering groups on the CAS number, so two names in one source list carrying the same number become one substance — which is right for a synonym and wrong for two substances the list happened to register alike. The 2026-08-14 build weighed 794 such numbers. Eleven carry a curated ruling that they name two substances, which keeps 318 flows off the object their shared number would otherwise have put them on; 370 have no ruling, stay merged, and go to the contested-cas queue with the names that share them. Staying merged is the conservative answer: splitting a substance nobody asked to split would strand every characterisation factor pointing at it.

4. Duplicates are deprecated, not deleted. Elementary flows with the same substance-and-context signature keep one active row; the others are marked owl:deprecated with a dcterms:isReplacedBy pointer.

   A name is not part of that signature, so two flows of one substance in one context and unit differing only in their name are collapsed here — which is #255. What holds a pair apart is a field the signature reads, and it is almost always the free-text note: of the 116 pairs standing in the 2026-08-12 build, 114 differed only in general_comment and the other two only in which source named their CAS. On 90 of those the note was 01.00.000, a version string EF writes into a description field, which is not a description and is no longer read as one — collapsing 77 of the 116 (#374) and leaving 39, of which 37 are held apart by a note that really is prose. Those are the pairs the elementary-flow-collision queue asks about, and each item names the differing fields so a curator can see what is doing the holding.

   Where a curator has answered — the ruling is in elementary-flow-collision-decisions.json, keyed on the collision so it survives a rebuild — the collapse happens just before this step, on the curator's terms rather than the sort's: the ruling names which flow survives, the other flow's characterisation factors and source references are carried onto it, and a source list that mapped onto the collapsed identifier reaches the survivor through the replacement.

   The characterisation factors are not part of the signature either, so two rows being collapsed can publish the same factor and give it different numbers — usually because one of them is the other rounded. Whichever collapse it is, the number is now settled rather than inherited from whichever row the sort reached: where the two are one number written to different precision, the more precise one is published, and the number not published is recorded on the factor that superseded it, under superseded_values, naming the flow that published it. Where the gap is too wide to be rounding — 66 of the 68 doubly-published factors on the 2026-08-12 build, by up to 400 times — the number is still settled the same way, and the run logs merged_factor_values_conflict and counts duplicate_factor_values_conflict, because discarding a number that far from the one kept is a decision about which source to believe rather than a duplicate publishing itself twice.

   The queue is asked again by the merge, of the flows it is about to write, and its answer replaces this one — because a merge adds flows, and while only this step asked, the queue described 116 groups of a published 145 (#361). Each item says which stage produced it, and a group holding flows the merge minted is review rather than info. The two stages' counts are filed separately in run_stats, under elementary_flow_collisions and merge_elementary_flow_collisions, so a number that moves can be read against the stage that moved it. This step and the ruling above both run before the merge, so a group the merge produced is one neither has seen — and a ruling written against one is counted absent rather than applied. The factor settling above never reaches those flows either.

   The merge asks the other half of the same question as well, which nothing used to ask: whether one substance is published in two places (#442). Where a substance is released is a fact about the process, so releases are left out of it — a metal emitted to air and to a river is two ordinary flows. Where it is taken from is a fact about the substance, so two intake contexts that are two different places rather than one described at two levels of detail are two answers to a question that has one, and the flow the merge wrote into the second of them is what nothing watched. Those reach the substance-in-two-places queue, counted in run_stats under merge_substance_places. Unlike a collision, there is nothing here one field away from collapsing: the two halves are separate identifiers that can never meet, so an inventory using one and a method characterising the other do not connect. 5. Concept associations are attached, linking each occurrence back to the source-list flows it corresponds to. Which lists those are is a property of the run: the base list, plus every list named by --source whose manifest declares a concept_associations block. 6. Each of those links is told how strong a claim it may make. An exactMatch says that one source flow and one consensus flow mean the same thing. That is only true where the source list did not put several of its own flows onto the same consensus flow — where it did, the consensus flow is broader than any one of them, and each link is weakened to a broadMatch. The count is over the whole source list, so the answer is not knowable until every one of its flows is placed. On the 2026-08-14 build 93,799 links kept exactMatch and 50 were weakened, all of them EF 3.1's. Before this pass existed, 28 EF 3.1 flows each claimed two exact matches, one of them a retired duplicate's (#405). 7. Those links are copied onto the flow. The published export and the flow's own record are flow-shaped, so a link resolved on the occurrence layer has to travel or no consumer sees it. A link a merge wrote directly is kept rather than replaced; the two are told apart by the activity named in the link's own provenance. 8. Flows that are not substances are recognised, and grouped into families. A land-use class, a delayed-emission correction and an alpha-emitter aggregate are all things an inventory measures without their being a substance, and where several of them are the same kind of thing a parent object is minted for the family. The number to watch is non_material_unfamilied: it counts the candidates no family claimed, it is zero today, and a run where it is not is a run that met a kind of non-material flow nothing has been told about. 9. Each substance is typed — what kind of thing it is, derived from its chemistry rather than from its name. Nickel(2+) becomes a MonoatomicIon whose Atom is nickel; Alcohols, C12-15, ethoxylated becomes an ImpreciseChemicalMixture, which is what a UVCB is. Deriving it from the formula, charge, structure, registry numbers and emission contexts rather than from a pattern over the label is what catches Chloride and Sodium ion, which a pattern missed. The 2026-08-14 build typed 5,409 neutral molecules, 91 elements, 81 isotope records, 36 monoatomic ions, 24 polyatomic ions and 7 zwitterions, and left 32 objects untyped for having neither a structure nor a registry number. See Semantic types. 10. Property values are normalised. A semantic property leaves here typed the way the ontology types it, a whole-entity charge leaves on the predicate that means whole-entity charge, and one structure leaves as one value however its sources spelled it — 30,940 duplicate SMILES spellings and 226 duplicate isomeric ones were collapsed on that build. Four counters here have to read zero, and do: a flow or an object publishing a stereochemistry in the field defined as carrying none, and a flat InChIKey left beside a more specific one for the same skeleton. 11. The substance's chemistry travels to its flows. Formula, masses, structures, registry strings, charge and the RDF classes are held once, on the substance, and the flow shows what is in it. That is a derivation rather than a copy, so a later correction to the substance cannot reach one layer and not the other. It used to be a copy, and the two drifted: on the 2026-08-12 build 815 of 7,674 substances differed from their own flows, and 8,542 published flows stated a molecular shape in the field defined as carrying none (#345). 12. A flow that is not one substance may not state a chemistry. An alpha-emitter aggregate or a delayed-emission correction has no molecular formula to report, and 216 flows are in that position. properties_withdrawn reads zero, because step 12 above no longer gives them one — and it is still counted, because a non-zero reading means a flow the typing missed is publishing a molecular formula for a quantity measured in kg·a. 13. An element's name reaches its flows, titlecased — so a source list saying nitrogen publishes as Nitrogen. Only where that is a change of casing, or where a curator has approved the rename: a flow whose own name is more specific than its substance's — an allotrope, an isotope, a phase — keeps it, and the proposal goes to the undecided-label-replacement queue. That was 856 titlecasings and 2 undecided renames on the 2026-08-14 build. The rulings are in preferred-label-decisions.json, the same file that gates consensus matching's renames. 14. The substance's name reaches its flows — the general rule the element pass is a special case of, and the last writer of a published name. The name of a substance is written twice, once on the substance and once on each of its flows, and until this pass a correction to the first copy stopped there: the 2026-08-19 build published 886 flows under a name their own substance had superseded, and 149 substances appeared in the export under two different names at once (#128). Each flow now takes its substance's name, and the name it had stays findable as a synonym of the flow. The exception is deliberate: a flow whose own name answers for a different substance is an identification bug — either two substance records are one substance, or the flow sits on the wrong one (#588) — and renaming it would delete the only visible symptom. Such a flow keeps its name and the pair goes to the substance-label-conflict queue, unless a ruling in the same decisions file has answered it: approve renames and withholds the colliding synonym (a name that answers for two substances is published for neither, #571), reject lets the flow's name stand. The merge applies the same rule to the whole database after its own label writes, under the merge_substance_labels stage, because a merge moves substance names on flows it never otherwise rewrites. assess checks the result from the published payloads, as flows.label_disagrees_with_substance.

Element and isotope enrichment is seeded from the base list only. A substance that arrives solely through a merged list gets no element or isotope facts: the periodic table is matched by label against the one list whose labels are curated for it, and a merged list's row reaches the enrichment only once it has been matched onto a consensus flow the base list already named. This was an undeclared consequence of an if row.source == "EF 3.1" comparison until #173; it is now a stated policy, at the guard and in the module's docstring.

See Flow objects and elementary flows.

What a run writes

Every run writes:

• harmonised-flows-simple.json.gz — the published export, the one artifact with a downstream consumer
• consensus-flows.sqlite3 — both layers, the change log, the PROV-O activity trail, every review queue, what each stage counted, and the merge outcome

Nothing is optional any more. elementary-flows.json, flow-objects.json and harmonised-flows.json are no longer written; the records they held are JSON columns in the database. The review files that used to be written on request — transform-log.json, provenance.json, consensus-match-review.json — and the four written unconditionally are tables in it.

See Which output do I need?.