Metrics & Judges

Synthesis judge#

DspyGeneralSynthesisJudge#

class DspyGeneralSynthesisJudge(lm, enable_reasoning_traces=False, confidence_threshold=0.7, signature=None, retry_temperatures=None)

Bases: SynthesisJudgeInterface

Enhanced DSPy module for evaluating GeneralSynthesisOntology extraction quality against source synthesis text.

Implements a two-level fallback chain for robust structured output.

Within each strategy, temperature is escalated on validation failures. API-level format errors (400/unsupported) skip immediately to the next strategy without wasting temperature retries.

Initialize the unified synthesis judge.

Parameters:

Methods#

forward(input)#

def forward(input)

Evaluate extracted GeneralSynthesisOntology against source text.

Tries each format strategy in order. Within a strategy, retries at escalating temperatures on content-validation failures. API-level format errors skip immediately to the next strategy.

Parameters:

Returns:

GeneralSynthesisEvaluation#

class GeneralSynthesisEvaluation(BaseModel)

Bases: BaseModel

Complete evaluation of GeneralSynthesisOntology extraction quality.

GeneralSynthesisEvaluationScore#

class GeneralSynthesisEvaluationScore(BaseModel)

Bases: BaseModel

Evaluation scores for GeneralSynthesisOntology extraction quality. Scores are on a scale of 1.0 (poor) to 5.0 (excellent) with 0.5 increments.

make_general_synthesis_judge_signature#

def make_general_synthesis_judge_signature(signature_name='GeneralSynthesisJudgeSignature', instructions=None, source_text_description='Original synthesis text for ontology extraction evaluation.', extracted_ontology_description='JSON representation of extracted GeneralSynthesisOntology.', target_material_description='Target material for synthesis context.', evaluation_description='Comprehensive evaluation of ontology extraction quality. CRITICAL: populate ALL fields — reasoning, confidence_level, all seven *_score and *_reasoning pairs inside scores, and scores.overall_reasoning. Omitting any field is invalid.')

Create a DSPy signature for GeneralSynthesisOntology evaluation.

Parameters:

Returns:

Linking judge#

DspyLinkingJudge#

class DspyLinkingJudge(lm, enable_reasoning_traces=False, confidence_threshold=0.7, signature=None)

Bases: LinkingJudgeInterface

DSPy module for evaluating synthesis-to-performance linking quality.

The judge receives:

1. The full paper text (source of truth).
2. The extracted synthesis ontologies (JSON list).
3. The extracted plot data (JSON list).
4. The linking output mapping syntheses to plot series (JSON).

It produces a LinkingEvaluation with four criterion scores (1-5 in 0.5 increments), nine failure-mode flags, and supporting reasoning.

Methods#

forward(input)#

def forward(input)

Evaluate linking output against the paper and extracted data.

Parameters:

Returns:

LinkingEvaluation#

class LinkingEvaluation(BaseModel)

Bases: BaseModel

Complete evaluation of synthesis-to-performance linking quality.

make_linking_judge_signature#

def make_linking_judge_signature(signature_name='LinkingJudgeSignature', instructions=None, source_text_description='Full paper text for linking evaluation.', synthesis_json_description='JSON list of extracted synthesis ontologies.', plot_data_json_description='JSON list of extracted plot data with series and coordinates.', linking_output_json_description='JSON linking output mapping syntheses to plot series.', evaluation_description='Comprehensive evaluation of linking quality.')

Factory for creating a customised LinkingJudge DSPy signature.

Follows the same pattern as make_general_synthesis_judge_signature.

Figure extraction metric#

FigureExtractionMetric#

class FigureExtractionMetric(LinePlotExtractionMetric)

Bases: LinePlotExtractionMetric

Methods#

__call__(preds, refs, error_metric='rmse')#

def __call__(preds, refs, error_metric='rmse')

Compute average RMSE or MAE across all matching series. For each series, it uses normalized-to-axis-sclae nearest-neighbor matching to find the closest points in the ground truth data to the extracted points from the LLM output. And then computes the error metric (RMSE or MAE) based on these matches.

compute_scale(ground_truth)#

def compute_scale(ground_truth)

Compute normalization scales for x and y.

pointwise_rmse(extracted_coords, gt_coords, x_scale, y_scale)#

def pointwise_rmse(extracted_coords, gt_coords, x_scale, y_scale)

Compute RMSE using nearest-neighbor matching for one series.

pointwise_mae(extracted_coords, gt_coords, x_scale, y_scale)#

def pointwise_mae(extracted_coords, gt_coords, x_scale, y_scale)

Compute MAE using nearest-neighbor matching for one series.