Thermocatalysis

End-to-end workflow for ammonia-decomposition catalysts: extract synthesis procedures and conversion-vs-temperature curves from a folder of PDFs, then score several vision models against human-annotated ground truth.

This is the most elaborate of the three case studies, because it doubles as the VLM benchmark harness used in the paper. If you only want catalysis data and not a model comparison, lemat-synth batch papers/ domain=catalysis with_performance=true does the extraction half in one command.

Source — examples/scripts/case_study_thermocatalysis/

An interactive single-paper version of the same pipeline lives in examples/notebooks/dev/catalysis_synthesis_with_performance.ipynb.

Prerequisites#

ANTHROPIC_API_KEY=...              # plot reading
GEMINI_API_KEY=...                 # materials, synthesis, linking
MISTRAL_API_KEY=...                # OCR
OPENROUTER_QWEN_API_KEY=...        # only if benchmarking Qwen VLMs
OPENROUTER_DEEPSEEK_API_KEY=...    # only if benchmarking DeepSeek VLMs

Quickstart — everything in one script#

bash examples/scripts/case_study_thermocatalysis/run_case_study.sh

Edit the VLMS=() array at the top to choose which models to benchmark. The script runs all three phases in order and can be launched from any directory.

The two-phase workflow#

Synthesis extraction (OCR → materials → synthesis → figure detection) is slow (~30 min/paper) and identical for every VLM. Plot reading (~5 min/paper) is the only VLM-dependent part. Splitting them means you pay the expensive half once:

Phase 1 — run once.

uv run examples/scripts/case_study_thermocatalysis/run.py \
    --pdf-dir data/papers_catalysis \
    --output  data/results_cache \
    --phase   synthesis \
    --match-gt-only \
    --no-eval \
    --skip-existing

Phase 2 — run once per VLM. Reads the cache, so no re-extraction:

uv run examples/scripts/case_study_thermocatalysis/run.py \
    --output data/results_catalysis/claude-sonnet-4.6 \
    --phase  vlm \
    --cache  data/results_cache \
    --vlms   claude-sonnet-4.6 \
    --single-dir

Repeat with --vlms gemini-3-flash, --vlms gpt-4o, and so on, changing --output each time.

data/results_cache/_cache/
    Teng_2024_Ru/
        synthesis.json   ← materials + synthesis + paper text
        figures.json     ← detected figures with base64 image data
    Zhou_2021_.../
        synthesis.json
        figures.json

Evaluation#

Compare every VLM you ran against the ground truth:

uv run examples/scripts/case_study_thermocatalysis/run.py \
    --output data/results_catalysis \
    --gt     data/results_catalysis_human \
    --vlms   claude-sonnet-4.6 gemini-3-flash gpt-4o \
    --eval-only \
    --metric rmse \
    --csv    data/results_catalysis/ranking.csv

Prints a ranked table and writes vlm_ranking_rmse.json plus the CSV.

Figures#

uv run examples/scripts/case_study_thermocatalysis/catalysis_map.py \
    data/results_catalysis/claude-sonnet-4.6 \
    --out-dir data/results_catalysis/claude-sonnet-4.6/figures

Writes PNG + PDF for seven figures — conversion landscape, metal/support heatmap, synthesis network, radar charts, promoter analysis, conversion by synthesis method, and a 3D waterfall — plus landscape_data.csv.

Optional: --use-llm (LLM-assisted parsing of material names), --ref-temp 500 (reference temperature for cross-material comparison), --debug (print a data inventory instead of plotting).

Output layout#

data/results_catalysis/
    <vlm_name>/
        <paper_id>/
            <material>.json            ← synthesis procedure + plot_data coordinates
            performance_mappings.json  ← which plot series → which material
            linking_summary_llm.json   ← linking stats + quality scores
            batch_summary.json         ← run timing + material counts
        figures/                       ← catalysis_map.py output
    manifest.json                      ← which PDFs ran + ground-truth mapping
    vlm_ranking_rmse.json              ← VLM ranking by mean RMSE
    ranking.csv                        ← per-material scores for all VLMs

Each <material>.json carries the standard synthesis object plus the digitised curve:

{
  "material": "Ru/MgO(110)",
  "synthesis": { "…synthesis procedure…" },
  "performance": {
    "material_name": "Ru/MgO(110)",
    "plot_data": [{
      "series_name": "Ru/MgO(110)",
      "coordinates": [[350, 12.4], [400, 41.9], [450, 78.2]],
      "x_axis_label": "Temperature", "x_axis_unit": "°C",
      "y_axis_label": "NH3 conversion", "y_axis_unit": "%"
    }]
  }
}

Every field is explained in Output Format.

All flags#

Available VLMs#

Any key from LLM_REGISTRY in src/llm_synthesis/utils/llms.py works with --vlms. Commonly benchmarked:

The full table with cost guidance is in Configuration & Models.