Model Benchmarks & Performance
Transparent validation metrics comparing NLPinSEO's proprietary architectures with state-of-the-art information retrieval systems.
Overview
Our benchmarking process evaluates how effectively models capture search intent, semantic richness, and text density compared to traditional term-matching approaches (BM25) and dense embeddings. The goal is to provide measurable, reproducible indicators of how algorithms rank content based on semantic quality rather than simple keyword stuffing.
Performance Tables
Below is a zero-shot retrieval performance comparison evaluated on the SEO-QA-2026 dataset (higher scores are better).
| Model / Architecture | NDCG@10 | MRR@10 | Latency (ms) | Footprint |
|---|---|---|---|---|
| BM25 (Baseline) | 0.412 | 0.428 | 1.4 ms | Minimal (CPU) |
| MiniLM-L6-v2 (Bi-Encoder) | 0.518 | 0.534 | 12.8 ms | 90M params (GPU) |
| TDFM (Proprietary) | 0.598 | 0.612 | 4.2 ms | Hybrid CPU/GPU |
| SGF (Proprietary) | 0.614 | 0.630 | 8.6 ms | Graph Sparse |
| TDFM-SGF (Hybrid) | 0.672 (est.) | 0.688 (est.) | TBD | In Development |
Validation Datasets
SEO-QA-2026
A custom validation corpus of 50,000 highly structured commercial search queries paired with expert-annotated content relevance labels.
Target: Semantic Similarity MatchMTEB (Selected Subsets)
Evaluation on standard Massive Text Embedding Benchmark tasks, focusing on web search retrieval and semantic textual similarity.
Target: NDCG@10 & MRR@10MS MARCO (Passage Retrieval)
Re-ranking evaluation against standard passage retrieval queries to benchmark zero-shot out-of-domain transfer capability.
Target: MRR@10Evaluation Metrics
- NDCG@10: Normalized Discounted Cumulative Gain at rank 10. Focuses on the top-ranking documents to ensure the absolute best match is surfaced first.
- MRR@10: Mean Reciprocal Rank. Measures the query position of the first highly relevant passage, validating navigation query accuracy.
- Semantic Density Coefficient (SDC): An internal density measure assessing how much unique factual semantic potential resides within text fragments.
Validation Methodology
To prevent overfitting, all models are evaluated in a zero-shot environment without local finetuning on the test splits. Performance runs are cross-validated across multiple web scraping runs to minimize variability introduced by server latency, rendering formats, and random seeding.
Future Benchmark Results
We plan to release evaluation results for the **TDFM-SGF Hybrid** model on standard MTEB task suites by Q4 2026. Code and reproducibility scripts will be open-sourced on our Open Source platform.