Research map / Learning theory and algorithms / Prediction and decision losses
Results of the Big ANN: NeurIPS’23 competition
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Summarizes the 2023 Big ANN NeurIPS competition with four ANN track variants (filtered, out-of-distribution, sparse, streaming), their datasets, evaluation protocol, baselines, and top-performing submissions.
Problem
Classical ANN benchmarks emphasize scaling dense search; practical variants such as filtered, OOD, sparse and streaming ANN search lacked standard tasks, datasets and evaluation protocols.
Approach
Defines four industry-motivated ANN tracks on modest-size (10M-scale) datasets runnable on accessible hardware, with a standardized open-source benchmarking framework, and reports the competition results and winning techniques.
Key results
- Top filtered-track entry ParlayANN reached about 37,902 QPS at recall@10 >= 0.9, more than 11x the Faiss-based baseline.
- In the OOD track, MysteryANN (RoarANN) and PyANNS were joint winners, reaching about 22,500 and 22,300 QPS at recall 0.9.
- In the sparse track, PyANNS achieved 6,500 QPS (private) at recall@10 >= 0.9, versus 95 QPS for the Linscan baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use tag-keyed indexing with tag-specific search strategies and candidate intersections | ParlayANN · Filtered YFCC | QPS at recall@10 ≥ 0.9 | Helped |
| Use a query-distribution-aware graph construction | MysteryANN (RoarANN) · OOD Yandex T2I | QPS at recall@10 ≥ 0.9 | Helped |
| Use 8-bit scalar quantization, VNNI, and adaptive prefetching | PyANNS · OOD Yandex T2I | QPS at recall@10 ≥ 0.9 | Helped |
| Quantize sparse vectors and prune smaller query values, followed by refinement with higher-precision vectors | PyANNS · Sparse MSMARCO/SPLADE | QPS at recall@10 ≥ 0.9 | Helped |
| Co-locate sparse-vector coordinates and values and use value bounds to terminate dot products early | GrassRMA · Sparse MSMARCO/SPLADE | QPS at recall@10 ≥ 0.9 | Helped |
| Use the public rather than private query workload | PyANNS, GrassRMA, and SUSTech-WHU · Sparse MSMARCO/SPLADE | QPS at recall@10 ≥ 0.9: around 25% slower on the harder private dataset | Mixed |
| Use the public rather than private query workload | NLE · Sparse MSMARCO/SPLADE | QPS at recall@10 ≥ 0.9: around 45% slower | Hurt |
| Apply the corrected whole-runbook recall computation instead of the first-snapshot computation | Puck · Streaming MS Turing | recall: 0.985 -> 0.0921 | Hurt |
| Add consolidation to free deleted vectors and reorganize the graph around deleted nodes | DiskANN streaming index · Streaming MS Turing | recall | Helped |
Surprising findings
- The streaming track winner under the initial scoring (Puck, recall 0.985) fell to recall 0.0921 after a caching error in recall computation was corrected, and PyANNS became the winner.
- Several sparse-track entries were about 25% slower on the private query set than the public one, and NLE about 45% slower, suggesting overfitting to public queries.
Limitations
- Filter predicates restricted to one or two tags, which encouraged specialized data structures less general than real-world filters.
- Some datasets are frozen, so private query sets could not be produced for all tracks; a near-duplicate removal oversight introduced minor measurement jitter.
- Streaming track recall computation error was discovered six months after the competition and required rerunning entries.
Experimental setup
Datasets: YFCC 100M Yandex Text-to-Image 10M MSMARCO passage (SPLADE sparse vectors) MS Turing 10M
Benchmarks: Big ANN NeurIPS'23 filtered track Big ANN NeurIPS'23 OOD track Big ANN NeurIPS'23 sparse track Big ANN NeurIPS'23 streaming track
Metrics: recall@10 (10-recall@10) queries per second (QPS) at 90% recall average recall over runbook
Baselines: Faiss IVF DiskANN in-memory index Linscan FreshDiskANN streaming index
Builds on: DiskANN HNSW FreshDiskANN Faiss SPLADE
Closest papers by meaning
- LotusFilter: Fast Diverse Nearest Neighbor Search via a Learned Cutoff Table (CVPR 2025 · similarity 0.84)
- CSPG: Crossing Sparse Proximity Graphs for Approximate Nearest Neighbor Search (NeurIPS 2024 · similarity 0.84)
- Probabilistic Routing for Graph-Based Approximate Nearest Neighbor Search (ICML 2024 · similarity 0.83)
- Discovering Data Structures: Nearest Neighbor Search and Beyond (NeurIPS 2025 · similarity 0.83)
- Adaptive Retrieval and Scalable Indexing for k-NN Search with Cross-Encoders (ICLR 2024 · similarity 0.83)
- Optimistic Query Routing in Clustering-based Approximate Maximum Inner Product Search (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.