Research map / Architectures and efficiency / Pruning and efficient tuning
Some Like It Small: Czech Semantic Embedding Models for Industry Applications
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops small (14M-parameter, 12-layer, 256-hidden) Czech BERT-based sentence embedding models via RetroMAE pre-training, multilingual distillation from all-mpnet-base-v2, and unsupervised contrastive fine-tuning, and evaluates them intrinsically and extrinsically against Base-sized Czech models and OpenAI Ada.
Problem
Sentence embedding models for Czech are scarce and typically large; real-time industry applications in resource-constrained settings need small, efficient Czech embeddings without abundant labeled Czech data.
Approach
Combine RetroMAE auto-encoder pre-training and multilingual distillation from an English teacher (all-mpnet-base-v2) with a merged Czech-English vocabulary to train Small-size bi-encoders, then apply unsupervised SimCSE fine-tuning on Czech Wikipedia; the resulting models match Base-size Czech models at roughly 8x smaller size and 5x faster speed.
Key results
- Dist-MPNet-CzEng (Small) reaches 87.60 STS-average versus 87.60 for SimCSE-Dist-MPNet-CzEng at 87.83, both above OpenAI Ada (82.59), and surpasses Base-size Czech models on STS-average (LaBSE 82.91).
- In fine-tuning on CTDC, Dist-MPNet-CzEng reaches 85.85 F1, close to Base-sized Czert (88.69) and FERNET-C5 (89.56) but at roughly 8x smaller size.
- Replacing Small-E-Czech with RetroMAE-Small in featured snippets reduced error rate by 20% and gave a 3.5% average recall boost at 95% precision in production.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multilingual distillation from all-mpnet-base-v2 with czeng20-csmono rather than the Small-E-Czech baseline | Czech sentence embeddings · STS-Average | Spearman's correlation: 48.30 -> 87.60 | Helped |
| Add SimCSE fine-tuning (ablation) | Dist-MPNet-CzEng · STS-Average | Spearman's correlation: 87.60 -> 87.83 | No clear effect |
| Add SimCSE fine-tuning (ablation) | RetroMAE-Small · STS-Average | Spearman's correlation: 76.30 -> 78.66 | No clear effect |
| Use SimCSE instead of the original Small-E-Czech embeddings (ablation) | Small-E-Czech · CFD linear probing | F1 score: 32.38 ± 2.35 -> 54.78 ± 2.59 | Helped |
| Use SimCSE instead of the original Small-E-Czech embeddings (ablation) | Small-E-Czech · CTDC linear probing | F1 score: 25.92 ± 0.56 -> 50.06 ± 0.45 | Helped |
| Add TSDAE fine-tuning (ablation) | Small-E-Czech · CFD linear probing | F1 score: 32.38 ± 2.35 -> 66.99 ± 1.24 | Helped |
| Add contrastive fine-tuning (ablation) | pre-trained models other than Small-E-Czech · Linear probing on CFD and CTDC | F1 score | No clear effect |
| Add SimCSE fine-tuning (ablation) | Dist-MPNet-ParaCrawl · CTDC fine-tuning | F1 score: 86.02 ± 0.12 -> 86.10 ± 0.32 | No clear effect |
| Add SimCSE fine-tuning (ablation) | Dist-MPNet-CzEng · CFD fine-tuning | F1 score: 78.73 ± 1.39 -> 78.73 ± 1.43 | No clear effect |
| Add SimCSE fine-tuning (ablation) | Small-E-Czech · CTDC fine-tuning | F1 score: 58.12 ± 1.52 -> 68.33 ± 1.34 | Helped |
| Use RankCSE or InfoCSE instead of SimCSE (ablation) | RetroMAE-Small · STS-Average | Spearman's correlation: 78.66 -> 79.01; 78.66 -> 76.40 | No clear effect |
| Train SimCSE with the larger Czech-plus-English Wikipedia dataset (ablation) | sentence embedding models · STS | score: +1 percentage point for small models; +2 percentage points for base models | Helped |
Surprising findings
- Contrastive fine-tuning (SimCSE) gave little or no gain for RetroMAE and distilled models on STS, suggesting autoencoder pre-training and contrastive learning are partly interchangeable.
- Newly initialized bilingual student with a merged Czech-English vocabulary outperformed pre-trained multilingual models as distillation initialization.
Limitations
- Pre-training corpus is non-public, limiting reproducibility of RetroMAE pre-training.
- Many contrastive variants (InfoCSE, RankCSE) gave no improvement over SimCSE, and the paper did not exhaustively tune hyperparameters.
- Costra evaluation modified by omitting basic and modality categories because they fell below random baseline.
Experimental setup
Datasets: Czech corpus (Seznam.cz, non-public) czeng20-csmono Paracrawler v9 Czech Wikipedia Costra CNA SVOB-IMG SVOB-HL CTDC CFD DaReCzech
Benchmarks: Costra STS (CNA, SVOB-IMG, SVOB-HL) DaReCzech CTDC CFD
Metrics: Spearman correlation accuracy P@10 micro-F1 macro-F1 NDCG
Baselines: Small-E-Czech Avg. fastText OpenAI Ada Embedding Czert-b-base-cased FERNET-C5 RobeCzech LaBSE SimCSE-Small-E-Czech
Builds on: RetroMAE SimCSE TSDAE RankCSE InfoCSE
Closest papers by meaning
- Repurposing Language Models into Embedding Models: Finding the Compute-Optimal Recipe (NeurIPS 2024 · similarity 0.87)
- Narrowing the Gap between Supervised and Unsupervised Sentence Representation Learning with Large Language Model (AAAI 2024 · similarity 0.86)
- The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding (NeurIPS 2024 · similarity 0.86)
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models (ICLR 2025 · similarity 0.85)
- MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025 · similarity 0.85)
- Test-Time Training on Nearest Neighbors for Large Language Models (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.