Research map / Large language models / Language, safety and interpretability
Towards Making Learnware Specification and Market Evolvable
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ELSI, an evolvable learnware specification with indexes (RKME indexed tree and LSH-based specification hash table) that lets a learnware market estimate model loss on unseen user tasks without raw data and without scanning the whole market.
Problem
Identifying the most helpful learnware(s) for a new user task in a continuously growing learnware market without accessing raw user data, while characterizing model abilities beyond their original training tasks and avoiding examining every learnware.
Approach
Use RKME specifications of all learnwares as a bridge: each model gets a loss vector over the other learnwares' RKME anchors, the user's task is approximated as a weighted combination of RKMEs via interactive anchor queries, and the model loss on the user task is estimated by a weighted inner product. Learnwares are organized into an RKME indexed tree and sign-random-projection LSH hash table so t
Key results
- On Postures at 100% of learnwares, ELSI-traverse reduces error rate to 11.27 vs 51.59 for Random, 23.43 for RKME-task and A-distance.
- On Mushroom at 100%, ELSI-hash achieves error 6.23 vs 16.90 for the best baseline (A-distance) and 43.66 for Random.
- ELSI-hash has the lowest execution time in deploying stage, significantly faster than RKME-task and A-distance, with time that stays nearly constant as the market ratio grows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ELSI-hash instead of RKME-task | Learnware identification · Postures | Error Rate: 23.43 -> 11.08 | Helped |
| Use ELSI-hash instead of RKME-task | Learnware identification · Bank | Error Rate: 14.13 -> 12.11 | Helped |
| Use ELSI-hash instead of RKME-task | Learnware identification · Mushroom | Error Rate: 16.29 -> 6.23 | Helped |
| Use ELSI-traverse instead of RKME-task | Learnware identification · PPG-DaLiA | RMSE: 20.62 -> 11.11 | Helped |
| Use ELSI-traverse instead of RKME-task | Learnware identification · PFS | RMSE: 2.21 -> 1.97 | Helped |
| Use ELSI-hash instead of RKME-task | Learnware identification · M5 | RMSE: 2.19 -> 2.14 | No clear effect |
| Use ELSI-hash rather than ELSI-traverse | ELSI · Postures | Error Rate: 11.27 -> 11.08 | No clear effect |
| Use ELSI-traverse rather than ELSI-hash | ELSI · PPG-DaLiA | RMSE: 13.88 -> 11.11 | Helped |
Surprising findings
- On the M5 scenario, A-distance performs worse than Random at some ratios (e.g., 5.25 vs 3.36 RMSE at 100%).
Limitations
- Only the single-model case (Cu=1) is handled; multi-model extension is deferred to future work.
- Market prototype is built from 3090 models on six tabular scenarios, which may not reflect large heterogeneous real markets.
- ELSI-hash trades some accuracy for speed compared with ELSI-traverse.
Experimental setup
Datasets: Postures Bank Mushroom PPG-DaLiA PFS M5
Benchmarks: Postures Bank Mushroom PPG-DaLiA PFS M5
Metrics: error rate RMSE execution time
Baselines: Random RKME-task A-distance
Builds on: Reduced Kernel Mean Embedding (RKME) RKME-task Locality Sensitive Hashing multi-index hashing k-medoids clustering
Closest papers by meaning
- Handling Learnwares from Heterogeneous Feature Spaces with Explicit Label Exploitation (NeurIPS 2024 · similarity 0.86)
- Learning Classifiers That Induce Markets (ICML 2025 · similarity 0.84)
- Pricing and Competition for Generative AI (NeurIPS 2024 · similarity 0.84)
- A Market for Accuracy: Classification Under Competition (ICML 2025 · similarity 0.84)
- Learnware Specification via Dual Alignment (ICML 2025 · similarity 0.83)
- Fairshare Data Pricing via Data Valuation for Large Language Models (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.