Transformers

All extractors inherit from ExtractorInterface[T, R]. Each implements a forward(input) method (synchronous) and gets an aforward(input) method (async) for free.

Base interface#

ExtractorInterface#

class ExtractorInterface(Module, Generic[T, R])

Bases: Module, Generic[T, R]

Generic interface for an extractor that takes an input of type T and returns an output of type R.

Material extraction#

DspyTextExtractor#

class DspyTextExtractor(signature, lm, retry_temperatures=None)

Bases: MaterialExtractorInterface

A text extractor that uses dspy to extract any arbitrary text from the publication text.

Implements temperature escalation retry on failures to improve robustness against transient validation errors.

Initialize the extractor with a dspy signature and language model.

Parameters:

Methods#

forward(input)#

def forward(input)

Extract text from the given str using the language model and signature.

Retries at escalating temperatures on failure.

Parameters:

Returns:

make_dspy_text_extractor_signature#

def make_dspy_text_extractor_signature(signature_name='DspyTextExtractorSignature', instructions='Extract the synthesis paragraph from the publication text.', input_description='The publication text to extract the synthesis paragraph from.', output_name='synthesis_paragraph', output_description='The extracted synthesis paragraph.')

Create a dspy signature for extracting text from publication text.

Parameters:

Returns:

Synthesis extraction#

DspySynthesisExtractor#

class DspySynthesisExtractor(signature, lm, retry_temperatures=None)

Bases: SynthesisExtractorInterface

Extractor that uses dspy to extract a structured synthesis ontology for a specific material from the entire paper text.

Implements temperature escalation retry and bare-JSON recovery on failures to improve robustness without changing the happy-path behavior.

Initialize the extractor with a dspy signature and language model.

Parameters:

Methods#

forward(input)#

def forward(input)

Extract a structured synthesis ontology for a specific material from the given paper text.

Retries at escalating temperatures on failure. Attempts bare-JSON recovery before escalating. Falls back to a minimal ontology if all attempts are exhausted.

Parameters:

Returns:

make_dspy_synthesis_extractor_signature#

def make_dspy_synthesis_extractor_signature(signature_name='DspySynthesisExtractorSignature', instructions='Extract structured synthesis for a specific material from the paper. Output only a valid JSON with the structured_synthesis field.', paper_text_description='Complete paper text to search for the material synthesis procedure.', material_name_description='The name of the specific material to extract synthesis for.', output_name='structured_synthesis', output_description='The extracted structured synthesis for specific material as a JSON.')

Create signature for extracting a materials-specific synthesis ontology.

Parameters:

Returns:

PDF extraction#

DoclingPDFExtractor#

class DoclingPDFExtractor(pipeline='standard', table_mode='accurate', add_page_images=False, use_gpu=True, scale=2.0, format='markdown')

Bases: PdfExtractorInterface

An extractor for extracting content from PDF files using the Docling library.

This class provides functionality to convert PDF files into various formats such as Markdown, doctags, JSON, or tokens. It supports different modes for handling images within the PDF, including embedding, referencing, or using placeholders. The extractor can be configured with various options such as pipeline type, table extraction mode, GPU usage, and scaling.

Attributes:

Methods:

extract_to_markdown(pdf_data: bytes) -> str:
    Converts a PDF file to Markdown format. Supports different image
    modes and raises a ValueError if an invalid image mode is provided.

Methods#

forward(input)#

def forward(input)

Extracts text and figures from a PDF and returns them as markdown with embedded figures.

Parameters:

Returns:

MistralPDFExtractor#

class MistralPDFExtractor(structured=False, mistral_api_key=None, max_retries=3, retry_base_delay=2.0)

Bases: PdfExtractorInterface

A PDF extractor that uses the Mistral OCR API to extract content from PDF files and optionally convert it to Markdown format. This extractor supports embedding images as data URIs and can return structured JSON output if required.

Attributes:

Methods:

Methods#

forward(input)#

def forward(input)

Extracts text and figures from a PDF and returns them as markdown with embedded figures.

Parameters:

Returns:

Plot data extraction#

ClaudeLinePlotDataExtractor#

class ClaudeLinePlotDataExtractor(model_name, prompt=resources.LINE_CHART_PROMPT_WITH_CONTEXT, max_tokens=1024, temperature=0.0, use_figure_context=True)

Bases: LinePlotDataExtractorInterface

Methods#

get_cost()#

def get_cost()

Get cumulative cost from Claude client.

reset_cost()#

def reset_cost()

Reset costs in Claude client.

LiteLLMPlotDataExtractor#

class LiteLLMPlotDataExtractor(model, prompt=resources.LINE_CHART_PROMPT_WITH_CONTEXT, max_tokens=8192, temperature=0.0, api_key=None, api_base=None, extra_kwargs=None, retry_temperatures=None)

Bases: LinePlotDataExtractorInterface

Plot data extractor using litellm — works with any vision model.

Uses the same prompt and parsing logic as ClaudeLinePlotDataExtractor, but routes API calls through litellm for multi-provider support.

Performance linking#

SeriesMaterialLinker#

class SeriesMaterialLinker(lm, prompt_template=DEFAULT_MATCHING_PROMPT)

Bases: PerformanceLinkingInterface

LLM-based transformer for matching plot series names to material names.

This transformer uses an LLM to semantically match series names from plots (e.g., “575”, “Ni/Al2O3”, “Sample A”) to the actual material names extracted from the paper (e.g., “Mo2(C,N)Tx-575”, “10%Ni/Al2O3”).

Attributes:

Initialize the linker.

Parameters:

Methods#

forward(input)#

def forward(input)

Match plot series names to material names using LLM.

Parameters:

Returns: