SciFactScientific claims
Small scientific claim retrieval set for checking whether research-like evidence can be surfaced in the top results.
Full-corpus measured results across three BEIR datasets
Every score below comes from a complete public BEIR corpus and every official test-qrels query, with zero query failures. Each card shows Full Stack first and Embedding + Reranking second; no score is projected or normalized. The macro card is an unweighted three-dataset average. Per-dataset scores are the primary evidence.
FiQAFinance QA
Finance-language retrieval where short, practical questions need the right supporting material rather than loose semantic matches.
NFCorpusBiomedical retrieval
Small corpus with dense relevance labels, useful for seeing whether ranking quality holds when many documents may be relevant.
Unweighted across complete corpora and official test queries.
Recall@100.5207Full Stack0.4304 Embedding + Reranking
NDCG@100.5366Full Stack0.3839 Embedding + Reranking
Full Stack
Embedding + Reranking
Quality came with higher measured query time in every paired run. Full Stack versus Embedding + Reranking p50 latency was 34,232 versus 7,311 ms on SciFact, 106,172 versus 26,521 ms on FiQA, and 29,504 versus 5,905 ms on NFCorpus. Compare only within each dataset and hardware pair.
Why these three BEIR datasets?
SciFact, FiQA, and NFCorpus cover scientific claims, finance questions, and biomedical retrieval. They show three different public BEIR shapes under one paired full-corpus protocol, not every domain and not a global ranking of dataset difficulty.
Selection logic
Three domains, one paired full-corpus protocol.
The public BEIR inputs vary from a 3,633-document biomedical corpus to a 57,638-document finance corpus. Each dataset uses the same Cortrix source lock within its Full Stack and Embedding + Reranking pair, and every official test-qrels query is evaluated.
All official test queries completed with zero query failures. The NFCorpus and SciFact Full Stack runs had partial coverage in LLM-dependent feature verification. The retained FiQA cells predate the strict feature-completeness contract, so their run-specific feature completeness was not formally evaluated. These statuses limit feature attribution; the scores shown are the complete measured retrieval results.
| Dataset | Corpus docs | Official test queries | Scientific cells |
|---|---|---|---|
| SciFact | 5,183 | 300 | 2 of 2 VALID |
| FiQA | 57,638 | 648 | 2 of 2 VALID |
| NFCorpus | 3,633 | 323 | 2 of 2 VALID |
A retrieval benchmark, not an end-to-end agent demo
BEIR-style evaluation asks a narrow question: when a query is issued against a known corpus, does the system retrieve the relevant documents near the top? Each run ingests the complete corpus and evaluates every official test-qrels query. That is why Recall@10 and NDCG@10 are the main page metrics.
Recall finds relevant material. NDCG rewards ranking it higher.
Recall@10 checks whether relevant material appears in the top ten results. NDCG@10 also rewards placing stronger relevant results higher in that top-ten list. The runner records Recall and NDCG at 10 and 50.
Retrieval quality only
This benchmark does not claim third-party product superiority, business outcome gains, memory quality, audit coverage, or end-to-end RAG answer quality.
Source-backed runner
- Pin Cortrix commit
4a6299c, cortrix-benchmarks commit6defa0a, both profile configurations, and each BEIR archive checksum. - Use the BEIR runner README and local reproduction guide to prepare the runtime; download the public archive, select the official test split, and verify corpus, query, and qrels counts.
- Create paired Embedding + Reranking and Full Stack cells with isolated namespaces and identical corpus, qrels, seed, retrieval depth, and top-k settings.
- Import the complete corpus and wait for required tasks to drain. Keep BGE-M3 and bge-reranker-v2-m3 active in both cells; for Full Stack, verify DeepSeek-V4-Flash enrichment, contextual retrieval, HyPE, document summaries, RAG Fusion, and listwise reranking.
- Freeze database state and send one cold end-to-end request for every official test-qrels query through the benchmark runner; require zero query failures.
- Map returned source-document IDs to qrels, calculate Recall@10/50 and NDCG@10/50, and record within-pair latency p50/p95.
- Judge the scientific scorecard separately from strict persistent-feature completeness using the published methodology.
- Save run, sample, and profile manifests; per-cell results and scorecards; query and latency records; resource snapshots; paired deltas; and the source lock so another evaluator can audit the comparison.
Start with quick validation
Want to check your setup before a full run? Use the deterministic fiqa-mini-120 or fiqa-mini-600 fixtures, or cap a sampled run with --max-queries and --max-corpus-docs. These paths validate installation, pipeline behavior, and configuration. Their scores are not comparable to the full-corpus measurements shown here.
Reproduction details
Each result carries its profile, source lock, hardware, dataset boundary, and reproduction method.
| Evidence field | Recorded value or source |
|---|---|
| Run date | 2026-07-14 to 2026-07-16. |
| Hardware profile | NFCorpus and SciFact: AWS g5.2xlarge; FiQA: AWS c7i.8xlarge. Compare latency only within each paired dataset. |
| Source lock | Cortrix 4a6299c; cortrix-benchmarks 6defa0a. Dataset-pair manifests also lock the orchestration source. |
| Dataset version | Full public BEIR SciFact, FiQA, and NFCorpus corpora with all official test-qrels queries: 300, 648, and 323 respectively. |
| Methodology | README, runner, methodology, and local reproduction. |
| Result bundle | The machine-readable bundle contains a manifest, six full-corpus scorecards, profile records, summaries, checksums, and exact reproduction steps. Cite the bundle from an immutable release tag or commit. |
| Scope | Retrieval quality only: Recall@10/50, NDCG@10/50, paired-run latency, and artifact reproducibility. |