I built Praviar's vision pipeline to extract chemical structures from patent drawings and compare them with the compound being investigated. The work involved finding the drawings on each page, running chemical recognition models and deciding what to do when their outputs differed.
I integrated existing models and evaluated them on individual drawings and complete patent pages. Some disagreements came down to a bond or a stereochemical marker; some located drawings produced no accepted structure at all. The retained examples let us inspect those outcomes alongside the scores.
Praviar is now open source, and I am no longer developing it. The system design essay describes the wider research and review application; this companion focuses on the vision work.
1. Finding a drawing and reading it are separate tasks
A patent page is an awkward input for a structure recogniser. It can contain text, numbered compounds, reaction schemes, tables and several drawings. Before attempting to read a molecule, the system needs to identify the relevant region and preserve enough of it to make recognition possible.
The segmenter locates the drawing; an OCSR model, short for optical chemical structure recognition, attempts to interpret the atoms and bonds within it. Errors can occur at either stage. A detector may find the right region and the recogniser misread its contents, or the crop may cut off a substituent before recognition begins.
DECIMER-Segmentation extracts chemical depictions from documents. Recognition approaches such as MolScribe and MolGrapher recover molecular structure through graph-based representations. I used existing components for these tasks, then evaluated the assembled paths under the configurations described below.
The implementation acquires patent drawing pages, obtains regions from the configured segmenter and classifies crops before recognition. A crop may be a fixed molecule, a reaction, a Markush depiction or non-chemical content. That routing matters because those images do not all describe the same sort of object. A variable-group drawing cannot be made equivalent to a single fixed molecule just by choosing one plausible interpretation.
Figure 1. Page 08158684_19, reference region 3, from the run containing 128 page entries. The selected structure matches the reference, including its recorded stereochemistry. The highlighted box and crop come from the reference dataset; the exact crop passed to the recogniser was not retained in this result record.
The archived application runtime retains page and crop context, bounding boxes and image hashes. Preprocessing can change the image sent to a recogniser while the original crop hash identifies the acquired evidence. Those runtime records can distinguish the source material from the transformed input. The surviving experiment rows shown here contain less: they do not preserve the exact recognition crops.
A source lookup can succeed without returning a drawing, and segmentation can finish without finding a usable crop. Failure handling depends on whether drawing evidence may influence analysis. When it may, an acquisition failure that leaves no pages after fallback, or a segmentation failure, raises an error. In shadow mode, where drawings cannot influence analysis, those failures can be logged while the result still appears empty. That empty result alone cannot establish that there was nothing to read.
The recorded full-page evaluation used MolDet detection and a five-model recognition ensemble. The final checked-in configuration defaults to DECIMER segmentation and names MolScribe and MolSight in its ensemble roster. The historical scores therefore describe a different setup from the default installation.
2. What counts as the same recovered structure?
Recognition produces a candidate representation that the rest of the pipeline can examine. SMILES encodes a molecular structure as text, but the same molecular graph can have different textual encodings. Comparing the raw strings would count some equivalent representations as errors. The evaluation therefore parses and canonicalises structures before comparing them.
RDKit provides the chemistry operations used in this work. A candidate must first be interpretable as a molecular structure. The system can then normalise it, derive identifiers and apply further checks. Parseability is a necessary property for downstream processing, but it says little by itself about whether the drawing was read correctly.
In the 447-crop experiment, all 447 fused outputs were valid structures, while 410 matched the reference under the evaluation's policy. The remaining 37 were valid outputs that did not match. A misread bond can still produce a chemically permissible structure, so parsing alone would not reject those errors.
The page-to-structure evaluator canonicalises candidates and labels, retaining the largest fragment for comparison. Minor fragments can therefore be removed before scoring. An “exact” match under this policy does not establish that every component of the drawing was recovered. Salt forms and other representation choices may still matter to the research question even when they make no difference to that score.
The implementation also checks confidence, plausibility and available text signals, including chemical names or formula information. A molecular formula can help reject an implausible candidate, but it does not specify how the atoms connect. These checks can inform selection without supplying an independent reference answer.
Scoring a stored answer and choosing that answer are separate operations. The paired comparison below scores retained outputs; the runtime code shows how candidate selection works in the inspected implementation. It does not reconstruct every historical run's settings.
I compared candidates using confidence-based selection and voting. The inspected runtime groups candidates by their largest fragment with stereochemical distinctions removed. Structures with the same connectivity but different spatial arrangements can therefore share a vote. A separate rule chooses the stereochemical representation to return from the winning group. The page evaluator retains those stereo distinctions when it scores the result, so agreement during voting is weaker than an exact match under evaluation.
Within the winning group, that rule prefers candidates specifying double-bond geometry, known as E/Z, then candidates with atom-centred stereo annotations, then unmarked candidates. If several E/Z candidates remain, it canonicalises them with stereochemistry retained and chooses the most common representation, using summed model weights to break ties. The highest-weighted voter within the chosen subgroup supplies the returned string. This preserves stereo information that grouping removed, but a preference for an annotated candidate is not evidence that its annotations are correct.
Text-derived structures use that flattened comparison key too. A matching name-derived candidate may confirm connectivity without confirming exact stereochemistry; formula information can adjust the voting weights. The returned candidate still passes the resolution checks.
Shared training data, similar conventions or difficulty with the same image could lead models to make correlated errors. Independence cannot be assumed from the fact that they are separate implementations. The retained predictions for inputs shared by all the voters allow a comparison with the combined result, as described in the next section.
I kept unresolved cases as a possible outcome. A candidate that fails the relevant checks can leave an abstention with diagnostic context for inspection. The evaluation needs to count those targets as well as scoring the structures that were returned.
On page 08158684_19, the selected structure preserves the stereochemistry in the reference. MolScribe and MolSight supply the matching representation; MolGrapher's candidate omits the stereo markers. Solid wedges and hashed bonds indicate bonds directed towards or away from the viewer. They carry information that an ordinary line does not. Figure 2 places the reference and selected structure beside the source drawing.
On page US20100151379A1_19, the selected candidate contains a chain of oxygen atoms where the reference has two sulfur–oxygen double bonds. MolGrapher supplies the reference-matching candidate, but the recorded selection follows a three-of-four majority. In this case, agreement favoured a valid structure with the wrong bonding.
On page 07169802_58, a fixed-molecule reference is routed through the experimental Markush worker. That worker records an invalid output and no structure is accepted. This is a located but unresolved target; the record does not establish a deliberate confidence-based abstention or diagnose why that route was chosen.
Figure 2. Selected cases from the same recorded page run. Molecular graphs are rendered from the retained strings; the unresolved case has no predicted graph. These cases explain outcomes, not their frequency. Reference crops are shown because the exact historical model crops are not retained in the result rows.
3. What the ensemble added
The March crop-level experiment ran five recognisers across 447 chemical-structure crops from 50 USPTO patents. It recorded 410 exact canonical matches, or 91.7%, and valid fused outputs for all 447 crops. The run made live inference calls rather than replaying an inference cache. Its recorded elapsed time was 20,034.5 seconds, approximately five hours and 34 minutes.
The elapsed time includes this run's environment and execution choices. It gives no service latency guarantee, and dividing it by the crop count would not isolate each model's latency. Additional recognisers consume resources even when their result agrees with a candidate already available.
The most direct comparison comes from a separate experiment containing 128 patent-page entries and 86 labelled target occurrences. Of those targets, 76 have retained entries for all five voters. This subset lets us compare model outputs; the full 86-target denominator measures the path from page detection to structure recovery, examined in the next section.
On the 76 comparison rows, the recorded ensemble selection recovered 62 exact structures; the best individual voter, MolScribe, recovered 59. The individual scores compare raw valid predictions with fusion gates bypassed, rather than separately deployed single-model systems.
The ensemble achieved a net gain of three exact recoveries, about 3.95 percentage points on this subset. Comparing the stored outputs row by row shows where that gain came from:
Here, fusion preserved all 59 raw MolScribe successes and added three. That is a useful result for these rows; it does not establish that every added model was necessary or that fusion will preserve a voter's successes on other material.
All five model entries are present, but entries can contain invalid outputs. Only 74 MolScribe entries are marked valid, and the ensemble returned structures on 70 rows. Invalid and unresolved outputs remain in the 76-row denominator. Routes without the relevant voter entries are outside this comparison.
One complete comparison record appears twice, leaving 75 distinct crop identifiers in the retained records. The repeated row belongs to “neither exact”. Removing that identical record gives 59 shared successes, three ensemble-only successes and 13 neither: 62/75 versus 59/75. Different record identifiers do not prove different image content. This sensitivity check concerns retained records, not a new experiment or independent inputs.
The ensemble's 91.7% on 447 crops and MolScribe's 77.6% on these 76 crops come from different samples and surrounding runs. They cannot isolate the contribution of combining models. The 62-versus-59 comparison uses shared inputs, but compares the complete recorded fusion policy with raw voter outputs. It does not isolate the effect of voting from candidate selection and gating.
The paired table comes from rescoring the stored ensemble selection and raw MolScribe output under the same largest-fragment, stereochemistry-retaining policy. Every recomputed ensemble outcome agreed with its recorded exact-match flag. This calculation ran neither recognition nor fusion again; it compares what the run actually retained.
The archive also contains a replay that removes one voter at a time from those stored outputs. Exact results fall from 62 to 57 without MolSight and to 58 without MolScribe. Removing MolGrapher leaves the aggregate at 62. That unchanged total does not establish that its predictions were redundant: the sulfur–oxygen example shows a correct minority candidate, and an aggregate can conceal gains and losses on different cases.
Figure 3. Removing one voter from the retained predictions and replaying fusion. Every bar uses the same 76 rows, including the repeated record. No fresh inference or compute saving was measured.
The replay measures changes in fusion using existing predictions. Its artifact does not bind a complete historical settings and code snapshot, and it contains no measurement of the cost or accuracy of running a smaller roster from scratch.
4. What changed when the input was a whole page
The full-page artifact contains 128 US patent-page entries with 117 distinct page identifiers. Some repeated identifiers have different results, so simply removing repeats would change the experiment. The historical counts include all occurrences: 1,069 annotated drawing regions, of which the detector matched 1,035, giving 96.8% recall. Recognition was scored on 86 target occurrences with a single reference structure available for comparison. Those are two different denominators within one experiment.
The assembled path recovered 62 exact structures from those 86 labelled target occurrences. Its end-to-end exact rate was therefore 72.1%. It resolved 70 of the 86, and 62 of the 70 resolved targets were exact, giving the conditional 88.6% figure.
Within the 86 labelled targets, 83 were matched by the detector. Thirteen matched targets produced no accepted structure, and three were unmatched. Of the 70 returned structures, 62 were exact and eight were not. The conditional score excludes all 16 unresolved targets, including both detection misses and failures after detection. The 72.1% overall rate includes them.
Figure 4. Different tasks retain their denominators. The stage counts concern 86 labelled target occurrences, not all 1,069 drawing-region occurrences. The conditional 62/70 result excludes unresolved targets; the ensemble comparison uses 76 rows with retained voter entries. Repeated inputs remain in these historical counts.
The 96.8% detection recall cannot be multiplied by the crop benchmark's 91.7% to produce a reliable end-to-end estimate. The crop benchmark used another sample and setup. The relationship between detection and recognition errors also depends on which regions are found and how they are cropped. The full-page run provides a direct measurement for its labelled targets instead.
There is a separate detector-only evaluation on 50 US patent pages with 393 annotated regions. At an intersection-over-union threshold of 0.5, it recorded 380 true positives, 20 false positives and 13 false negatives. That gives 95.0% precision and 96.69% recall. Intersection over union measures how well the predicted and reference boxes overlap; it does not score the chemical structure within either box.
The 50-page detector trial and the run with 128 page entries used different inputs and purposes, so their counts cannot be pooled into one trial. Only the latter includes the labelled progression from locating a region to returning an exact structure.
The case records locate some of those losses without explaining every cause. Among the 13 matched but unresolved targets are molecule, Markush and no_detections routing outputs. That last label belongs to an already matched region; it is distinct from the three unmatched targets. Without the exact historical recognition crops, the available records cannot fully separate crop defects from recognition or routing errors.
5. Different material, different results
The separate multi-jurisdiction OCSR evaluation scored 358 rows representing 355 unique images, with three images appearing in two dataset partitions. It recorded 254 exact matches, or 70.95% of the scored rows.
These groups differ in their drawings and sample sizes as well as their jurisdiction. The variation describes the tested material; it cannot be attributed to jurisdiction alone or used to predict performance on every patent from that source.
A later analysis summarised 12,560 cached evaluation rows across USPTO, CLEF, UOB and ACS material. It counted 11,416 stored exact-match flags, or 90.9%. The summary generator counted existing scores; it did not compare molecular structures again or run new inference.
These benchmarks contain different kinds of material. USPTO and CLEF contain molecular images from US patents. UOB comes from scanned Maybridge compound-catalogue pages, while ACS contains journal images labelled by the MolScribe authors. The counts below describe Praviar's cached evaluation rows, which do not cover every row of every source benchmark. The archive does not fully establish the historical exclusions or explain all differences between cached reference strings and their present-day normalisation.
USPTO contributed 5,360 exact matches out of 5,633 images, or 95.2%. CLEF contributed 800/905, or 88.4%; UOB 5,040/5,740, or 87.8%; and ACS 216/282, or 76.6%. The roughly 18.6-point difference between USPTO and ACS would disappear from view if only the aggregate were reported.
This larger summary reports the recognition results retained in the caches. The page trial includes the earlier work of locating and preparing regions from document images. The larger sample does not replace that measurement of the assembled path.
Markush drawings need a representation that can preserve variable groups and their alternatives. Praviar includes an experimental specialist path for that work, with separate validation and influence conditions. The fixed-structure evaluations do not measure how well it interprets the members of a generic chemical claim.
The ordinary similarity calculation also has a narrower meaning. A fingerprint or substructure comparison can indicate a chemical relationship between recovered structures. It does not establish the legal scope of a claim. The specialist Markush path does not simply become one more fixed-structure similarity score, and the recorded fixed-structure results should not be reused as its evaluation.
6. From an experimental result to usable evidence
Conditions for using a result
The archived runtime includes an optional confidence cascade to control which recognisers run. With that cascade enabled and MolScribe available, it tries MolScribe first. A valid result with sufficiently high confidence and chemical plausibility can return early, after the shared resolution gates. A medium-confidence result can bring in available MolSight or DECIMER runners; otherwise, the path requests the remaining configured voters. These are available workers, not a promise that all five historical models are installed.
A high-confidence candidate that fails plausibility can still lead to more recognition. If it passes those initial checks but fails the shortcut's final resolution gate, however, the path returns unresolved without automatically requesting another round of voters. The point at which the candidate fails determines whether more computation is requested. Selected results still require available confidence, a minimum confidence level, parseability and an atom-count bound. The historical evaluations do not establish the compute saving or accuracy of each cascade branch.
The final checked-in system deliberately separates obtaining a visual result from allowing it to influence analysis. Its default rollout is shadow. Experimental evidence may be retained for inspection while the decision-facing path receives no drawing input. Influence-capable use requires the relevant configuration, tool and calibration bindings to pass.
Where permitted, drawing evidence can affect routing and appear in report fields. The normal drawing-triggered route did not forward the drawing payload to the claim agent, and drawings did not directly populate the later matter evidence index and graph. The system design essay examines these unfinished connections. Correct recognition alone would not repair them.
What can be reproduced
Reproducing a historical experiment would require its worker setup, model files, preprocessing and source path. Reusing those components also requires checking their licences and provenance. The archive identifies MolDet's non-commercial licence and unresolved MolNexTR provenance, so the historical roster cannot be treated as a ready deployment choice on the strength of its score.
The evaluations do not exclude training-data overlap or establish patent-family-independent sampling. Nor does every artifact cryptographically identify every historical model checkpoint. Source hashes identify the result files used here, but cannot recover that missing metadata or establish independent generalisation. The paired rescore is repeatable on the stored predictions; it does not recreate the conditions under which those predictions were produced.
The sulfur–oxygen case remains a useful test of any proposed change to selection: a recogniser already supplied the reference-matching answer, yet fusion chose another structure. The paired comparison establishes three gains over raw MolScribe on the retained rows, with no losses against that voter. It does not tell us how to select the correct minority answer in this case. A changed policy would have to recover it and then be checked against the other drawings, including those the current selection got right.
The historical results and their recorded configurations are in the project's evaluation summary. The paired comparison and repeated-record audit were calculated for this essay from the retained outputs. The analysis files and reproduction instructions accompany this article. Upstream methods are described by Qian et al. for MolScribe, Morin et al. for MolGrapher and Rajan et al. for DECIMER-Segmentation. Those papers describe the component methods; the Praviar results come from its own recorded evaluations.
For the surrounding application, see Praviar: from patent drawings to reviewable evidence.