How TrialDiff builds amendment Evidence Records

Severity Calibration Status

TrialDiff v0.2.1 failed the blinded review-priority calibration gate. The critical tier needed at least 24 of 30 fresh TrialDiff-critical records to be independently confirmed as critical; fresh rubric applications confirmed 4/30, 5/30, 12/30, and 17/30.

The finding is that the critical review-priority boundary was not stable enough to validate severity as an external priority standard in this design. Severity is therefore retained only as deterministic uncalibrated triage metadata. The underlying Evidence Records, changed paths, source hashes, canonical JSON, and claim boundaries remain citable and reproducible. The calibration record is published as SEVERITY_CALIBRATION_v0.2.1.md , with the decoupling decision recorded in SEVERITY_DECOUPLING_v0.2.1.md .

The operational primitive is specified in EVIDENCE_RECORD_PRIMITIVE.md .

Corpus Scope

The regenerated 100-study breast-cancer corpus contains 100 trials, 4,485 adjacent patches, and 868 materiality events under the v0.2.1 triage-rule generation. Evidence Record counts are generation-specific: the intermediate materiality-filter population contained 483 records, the published v0.1.1 event-class generation contains 100 records and carries erratum E1, and the published v0.1.2 generation contains 97 records and now carries erratum E4 for three missed secondary-outcome memberships. The corrected v0.1.3 generation also contains 97 records, has been active in production since 2026-08-04, and was published on 2026-08-05 as Zenodo DOI 10.5281/zenodo.21811845.

The corpus was selected for amendment density and demonstration value. Dashboard counts are evidence-engine outputs for this bounded corpus, not population estimates for ClinicalTrials.gov, oncology trials, or breast-cancer trials generally.

The frozen v0.1-alpha package remains a historical 25-study slice (25 trials, 280 adjacent patches, 122 materiality events, 86 Evidence Records, 40 exported) preserved byte-identically in the repository.

External Context

TrialDiff does not claim to have discovered the problem of trial amendment opacity. Independent literature documents it. Florez MA, Jaoude JA, Patel RR, et al. reported that among 755 active cancer phase 3 randomized clinical trials, 145 trials (19.2%) had primary end point changes, and 102 of those 145 changes (70.3%) were not disclosed in the manuscript.

TrialDiff's narrower contribution is inspectability: making individual registry changes citable, replayable, and bounded by explicit claims-supported and claims-not-supported language.

Citation: Florez MA et al. JAMA Netw Open. 2023;6(5):e2313819. doi:10.1001/jamanetworkopen.2023.13819.

Evidence Records, Not A Closed Feed

Closed commercial monitoring products can answer whether a trial record changed. TrialDiff is intentionally a different surface: it publishes the evidence object itself. Each exported Evidence Record exposes the exact changed paths, patch payload, rules or value signals, source links, hashes, timing context, and claim boundaries.

The methodology is part of the artifact. A reader should be able to challenge the deterministic triage label, inspect the source, and cite the record without trusting a hidden scoring system.

The active v0.1.3 records are byte-verifiable but not fully source-closed: 80 of 109 class memberships are decidable from the embedded adjacent-version patches, while 29 require registry-sourced slices that the package does not include. Independent clean-room reconstruction of every membership from packaged bytes alone remains unfinished and is not claimed.

Critical-Triage Density Denominator

Dashboard critical-triage density is computed as critical-triage events per amendment patch, not validated critical review-priority events. The denominator is total adjacent patches for the trial.

This means trials with high administrative-only amendment volume show lower density than trials where most amendments yield material changes. The dashboard also applies a patch_count >= 10 floor to reduce small-denominator distortion.

Evidence Record Filter

New Evidence Record packages are generated as first-class citable artifacts for patches satisfying deterministic event-class predicates. A record can carry several event classes, and the triage severity remains uncalibrated metadata rather than the inclusion rule.

The frozen v0.1-alpha package remains a historical high/critical triage slice and is not rewritten. For current package generation, event-class membership is the cleaner citation criterion: reconciliation co-occurrence is recorded as a tag, not used to erase the patch or to claim the change was benign.

Rule Taxonomy

The classifier starts with deterministic JSON Pointer rules over the ClinicalTrials.gov schema. Outcome, design, intervention, eligibility, status, timeline, enrollment, and adverse-event paths are mapped to review categories and base triage severities. Each event stores the rules that fired, the rule set hash, and source provenance.

Large language models do not set severity. They are not part of the deterministic triage decision in v0.1-alpha.

Timing Modifier

Timing context (pre-recruitment, early recruitment, late recruitment, post-recruitment) is recorded on every event. Automatic timing escalation was disabled in the v0.2.1 rule tightening (see V0.2.1_RULE_TIGHTENING_DIAGNOSTIC.md in the repository), so severity currently always equals severity_pre_timing for the displayed generation.

Both fields are stored so a future calibrated rule set can re-enable escalation per category.

Source Provenance Caveat

The demo uses public ClinicalTrials.gov study pages and API payloads. Adjacent history patches come from ClinicalTrials.gov history endpoints that expose JSON Patch-like changes between versions. TrialDiff stores the returned patch payload and hashes it so an exported Evidence Record remains inspectable even if a source endpoint later changes.

The current alpha is therefore evidence-preserving, not endpoint-independent. Long-term durability would require a broader source archiving strategy.

Status And Stopped-Trial Rules

Status and termination rules are treated as triage signals because stopped-trial explanations are structured public-record fields. 42 CFR 11.64 references the ClinicalTrials.gov "Why Study Stopped" data element when applicable. TrialDiff can surface whether this field changed, was empty, or matched a low-information value.

TrialDiff does not determine whether an explanation satisfies the regulation, whether a sponsor complied with law, or whether a stopped trial was scientifically justified.

Dual Distribution Finding

Amendment volume and critical-triage density answer different questions. Patch volume measures operational activity. Critical-triage density measures how often the deterministic rules assign the critical triage label. Heavy-amendment platform protocols can have low critical-triage density, while smaller trials can have a high share of critical-triage changes.