Research map / Large language models / AI and society

Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges

AAAI 2025 · Large language models › AI and society · approach: Critiques of LLM claims and judging · 8 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Position/overview paper that delineates the challenges of Language-Models-as-a-Service (accessibility, replicability, reliability, trustworthiness), compares commercial LMaaS licences and interfaces, and proposes a tentative agenda for mitigation.

Problem

Proprietary LMs accessed only via web or API restrict evaluation, benchmarking, replication, and trust; the paper systematically examines how the lack of model information impedes accessibility, replicability, reliability, and trustworthiness.

Approach

Frames LMaaS challenges along four axes (accessibility, replicability, reliability, trustworthiness) and provides a synthesized overview of licences, opt-out/training-use policies and API availability of major commercial LMaaS, with recommendations such as hashed model commits, deterministic modes, a contamination registry, and interpretable benchmarking.

Key results

• Summarises terms of service and opt-out/training policies for major LMaaS (AI21, Anthropic, Cohere, Google, Inflection, Microsoft, OpenAI), showing most offer opt-out or API access but not all.
• Notes that OpenAI's API opt-out became the default from March 1, 2023, while the web interface is opt-in by default.
• Reports that GPT-3.5 and GPT-4 have been observed to be non-deterministic even at temperature zero, so temperature alone does not guarantee replicability.

Surprising findings

• Reports that GPT-3.5 and GPT-4 are non-deterministic even at temperature zero.
• Argues that a 'hash of outputs' test for detecting model replacement is unsound because different models can output identical answers or differ due to non-determinism.

Limitations

• Table of LMaaS licences is illustrative and not exhaustive; only a short list of services is covered.
• Many recommendations (registry, deterministic modes, latent-factor benchmarking) are proposals without empirical validation.

Experimental setup

Datasets: Hugging Face model license statistics (accessed 01/08/2023)

Builds on: HELM Chain-of-Thought prompting model cards

Closest papers by meaning

• SWE-bench: Can Language Models Resolve Real-world Github Issues? (ICLR 2024 · similarity 0.87)
• Gorilla: Large Language Model Connected with Massive APIs (NeurIPS 2024 · similarity 0.86)
• BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks (NeurIPS 2025 · similarity 0.86)
• LooGLE v2: Are LLMs Ready for Real World Long Dependency Challenges? (NeurIPS 2025 · similarity 0.86)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets (ICLR 2024 · similarity 0.86)
• RedPajama: an Open Dataset for Training Large Language Models (NeurIPS 2024 · similarity 0.86)