Additional held-out IDs
Give search a second check
A small model checks a shortlist; compare quality and processing on fresh queries.
A small model can read a search query beside a likely match and give it a relevance score. Here it checks only the first 20 keyword matches.
200 additional test IDs
These are the remaining official SciFact test IDs, excluded from our earlier 100-query experiments. One claim duplicates training text under another ID; removing it leaves 199 queries and the same correct-first totals below. Each search starts with the same collection of real scientific abstracts.
| Method | Relevant first | Mean time (50-query subset) |
|---|---|---|
| Keywords | 105 / 200 | 6.0 ms |
| Meaning | 105 / 200 | 5.8 ms |
| Learned combination | 119 / 200 | 12.4 ms |
| Keywords + small second check | 111 / 200 | 99.6 ms |
| Second check only when uncertain | 113 / 200 | 23.4 ms |
Quality totals cover all 200 test IDs. Times cover only the first 50 queries, measured separately after other model experiments finished: warm CPU means, three repeats per query. Input processing and ranking are included; model loading and building the document index are excluded. Background operating-system load is uncontrolled.
The frozen learned combination performed best here: 119/200. Compared with keywords, it corrected 15 first results and lost one. It still missed 81 first results.
The optional check ran for 42 of 200 queries. Its rule was selected on 209 separate development claims before these results were measured.
In the separate timing subset, 9/50 queries used the check. Its mean includes those slower queries; the 95th-percentile time was 109.2 ms. Timing results.
Checking every shortlist put a relevant paper first on 6 more queries overall than keywords alone. It also adds processing; it is not a free speed improvement.
Only 171/200 shortlists contained a judged relevant paper. A second check cannot recover papers that keyword search left out.
What model runs, and what gets skipped?
The second check uses pretrained MS MARCO TinyBERT: two transformer layers, 128 hidden dimensions and about 4.39 million parameters. Both layers run for each of the 20 query-document pairs. When the keyword winner is clearly ahead, the optional route skips the entire second model. This is conditional reranking, not stopping inside the model.
The learned combination reuses our frozen five-weight model. Meaning search uses six-layer MiniLM and stored document vectors. Neither was retrained on these 200 claims. The live search demo supports keywords, meaning and their combinations; this new second check is an offline experiment.
Relevant means human-judged evidence about the claim, which may support or refute it. This is not a claim-truth classifier, production search traffic, or proof of a general improvement. Model pretraining overlap is unknown.
Method, all metrics and limitations · Every query and result · Frozen protocol · Experiment code