Research map / Architectures and efficiency / RNNs and normalization
CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CATransformers, a multi-objective Bayesian co-optimization framework that jointly searches Transformer pruned architectures and edge accelerator designs to minimize total (operational + embodied) carbon, producing CarbonCLIP models.
Problem
Existing hardware-aware NAS and co-design methods optimize latency or energy and ignore embodied carbon, or fix either the model or hardware, so they cannot minimize total lifecycle carbon of Transformer inference on edge devices.
Approach
Treat carbon (embodied from chip area plus operational from energy and latency over a deployment lifetime) as a first-class objective in joint model-pruning and accelerator-parameter search, using multi-objective Bayesian optimization (qNEHVI) with a fine-tuning-based accuracy proxy.
Key results
- Carbon-optimized co-design reduces total carbon by an average of about 30% versus latency-optimized designs (with 7.7x higher latency), and energy-optimized designs reduce carbon by 24%.
- CarbonCLIP-S achieves a 17% carbon reduction versus TinyCLIP-39M/16 with no accuracy regression; CarbonCLIP-XS gives an 8% accuracy gain at 3% lower carbon versus TinyCLIP-8M/16.
- Fine-tuning proxy accuracy shows Spearman correlation of 0.98 with final post-pruning accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Optimize total carbon in the joint model–hardware search rather than latency | CATransformers · 20 TOPS, language/vision/CLIP models | total carbon: 30\% | Helped |
| Optimize total carbon rather than latency | CATransformers · 20 TOPS, language/vision/CLIP models | latency: 7.7$\times$ | Hurt |
| Optimize energy rather than latency | CATransformers · 20 TOPS, language/vision/CLIP models | total carbon: 24\% | Helped |
| Optimize energy rather than latency | CATransformers · 20 TOPS, language/vision/CLIP models | latency: 4$\times$ | Hurt |
| Jointly optimize carbon and latency instead of carbon alone | CATransformers · 20 TOPS, language/vision/CLIP models | total carbon reduction: 18\% | Mixed |
| Use CarbonCLIP-XS instead of TinyCLIP-8M/16 | CLIP-ViT-B/16 · CLIPBenchmark, 41 datasets | average accuracy: 30.7 -> 38.7 | Helped |
| Use CarbonCLIP-XS instead of TinyCLIP-8M/16 | CLIP-ViT-B/16 · CLIP-ViT-B/16 edge inference | total carbon: 0.34 -> 0.32 | Helped |
| Use CarbonCLIP-S instead of TinyCLIP-39M/16 | CLIP-ViT-B/16 · CLIPBenchmark, 41 datasets | average accuracy: 45.0 -> 45.0 | No clear effect |
| Use CarbonCLIP-S instead of TinyCLIP-39M/16 | CLIP-ViT-B/16 · CLIP-ViT-B/16 edge inference | total carbon: 0.44 -> 0.36 | Helped |
| Remove 50\% of any model dimension through pruning (ablation) | CLIP models · MSCOCO retrieval | accuracy | Hurt |
| Prune the embedding dimension instead of other dimensions (ablation) | CLIP models · MSCOCO retrieval | accuracy | Hurt |
| Prune the vision encoder rather than the text encoder (ablation) | CLIP models · MSCOCO retrieval | accuracy | Hurt |
Surprising findings
- Carbon-optimal designs incur substantially higher latency (about 7.7x) than latency-optimal ones, so carbon and latency objectives trade off strongly.
- Under tight compute constraints (1 TOPS), energy optimization becomes nearly equivalent to total carbon optimization because operational carbon dominates.
Limitations
- Targets Transformer models and domain-specific edge accelerators with limited GPU support; no evaluation on CNNs.
- Proxy-based accuracy estimation may deviate from real outcomes due to overfitting or underestimating pruning effects.
- Does not account for broader environmental impacts such as e-waste, water use, and rare minerals.
Experimental setup
Datasets: MS COCO GLUE MRPC CIFAR-10 MetaCLIP-2.5B DataComp-1B DataComp-Medium CLIPBenchmark (41 tasks)
Benchmarks: CLIPBenchmark (41 zero-shot tasks) MS COCO retrieval (proxy) GLUE MRPC CIFAR-10
Metrics: carbon (kgCO2e) latency (ms) energy average zero-shot accuracy hypervolume
Baselines: CLIP ViT-B/16 (DataComp) CLIP ViT-B/32 (DataComp) TinyCLIP-8M/16 TinyCLIP-39M/16 TinyCLIP-40M/32 TinyCLIP-61M/32
Builds on: Ax BoTorch qNEHVI ACT Accelergy SCALE-Sim
Closest papers by meaning
- CO2-Meter: A Comprehensive Carbon Footprint Estimator for LLMs on Edge Devices (AAAI 2026 · similarity 0.84)
- Carbon Footprint Reduction for Sustainable Data Centers in Real-Time (AAAI 2024 · similarity 0.83)
- LLMCarbon: Modeling the End-to-End Carbon Footprint of Large Language Models (ICLR 2024 · similarity 0.83)
- CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework (NeurIPS 2024 · similarity 0.83)
- Holistically Evaluating the Environmental Impact of Creating Language Models (ICLR 2025 · similarity 0.82)
- Learning Rate Transfer for Hybrid Transformer-SSM Architectures (NeurIPS 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.