Research map / Vision-language and multimodal / Human and embodied modeling
Tabular Insights, Visual Impacts: Transferring Expertise from Tables to Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CHARMS, which uses optimal transport to align image channels with tabular attributes and maximizes mutual information so image classifiers gain tabular expert knowledge without tabular inputs at test time.
Problem
Transferring expensive expert-derived tabular knowledge into image models for prediction when tabular data is unavailable at inference, given heterogeneity between modalities and the need to handle numerical and categorical attributes differently.
Approach
Align image channels with tabular attributes via sample-wise similarity distributions and optimal transport, then train image channels to predict the aligned tabular attributes (MSE for numerical, cross-entropy for categorical) so only relevant tabular knowledge is transferred.
Key results
- CHARMS reaches 0.9175 accuracy on DVM versus 0.8743 for ResNet image-only and 0.9748 for LightGBM tabular-only.
- CHARMS achieves 0.8661 on SUNAttribute, surpassing the tabular-only LightGBM (0.8501) and RTDL (0.8563) and the image-only ResNet (0.8361).
- On PetFinder-pawpularity, CHARMS reduces RMSE to 18.4314 from 18.6150 for ResNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer tabular knowledge to image channels with CHARMS instead of image-only Resnet | image classifier · DVM | accuracy: 0.8743 -> 0.9175 | Helped |
| transfer tabular knowledge to image channels with CHARMS instead of image-only Resnet | image classifier · SUN | accuracy: 0.8361 -> 0.8661 | Helped |
| transfer tabular knowledge to image channels with CHARMS instead of image-only Resnet | image classifier · CelebA | accuracy: 0.8146 -> 0.8220 | Helped |
| transfer tabular knowledge to image channels with CHARMS instead of image-only Resnet | image classifier · Adoption | accuracy: 0.3477 -> 0.3603 | Helped |
| transfer tabular knowledge to image channels with CHARMS instead of image-only Resnet | image regressor · Pawpularity | root mean square error: 18.6150 -> 18.4314 | Helped |
| transfer tabular knowledge to image channels with CHARMS instead of image-only Resnet | image regressor · Avito | root mean square error: 0.2512 -> 0.2495 | Helped |
| use 20 clusters instead of 40 (ablation) | CHARMS · SUNAttribute | accuracy: 0.8661 -> 0.8494 | Hurt |
| use 60 clusters instead of 40 (ablation) | CHARMS · SUNAttribute | accuracy: 0.8661 -> 0.8494 | Hurt |
| use 80 clusters instead of 40 (ablation) | CHARMS · SUNAttribute | accuracy: 0.8661 -> 0.8556 | Hurt |
| use 100 clusters instead of 40 (ablation) | CHARMS · SUNAttribute | accuracy: 0.8661 -> 0.8522 | Hurt |
Surprising findings
- CHARMS outperforms the tabular-only model on SUNAttribute despite tabular data being the stronger modality on some datasets.
- CLIP-based text-serialized tabular transfer performs poorly, with CLIP-LP reaching only 0.7619 on DVM.
Limitations
- On DVM, CHARMS (0.9175) still trails the tabular-only LightGBM (0.9748), so transfer does not fully close the gap to the tabular modality.
- Results on the regression tasks are mixed; the MMCL baseline cannot be applied to regression, and the method's gains on Avito are limited relative to LightGBM.
Experimental setup
Datasets: DVM SUNAttribute CelebA PetFinder-adoption PetFinder-pawpularity Avito MFEAT
Benchmarks: DVM SUNAttribute CelebA PetFinder-adoption PetFinder-pawpularity Avito
Metrics: accuracy root mean square error mutual information
Baselines: LightGBM FT-Transformer (RTDL) ResNet KD MFH FMR MMCL CLIP-LP CLIP-FT
Builds on: Fixed Model Reuse Multimodal Contrastive Learning Modality Focus Hypothesis MINE GradCAM
Closest papers by meaning
- Large Scale Transfer Learning for Tabular Data via Language Modeling (NeurIPS 2024 · similarity 0.88)
- Multimodal Tabular Reasoning with Privileged Structured Information (NeurIPS 2025 · similarity 0.87)
- TabGLM: Tabular Graph Language Model for Learning Transferable Representations Through Multi-Modal Consistency Minimization (AAAI 2025 · similarity 0.86)
- MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Image (NeurIPS 2026 · similarity 0.86)
- TRL-Bench: Standardizing Cross-Paradigm Representation-Level Evaluation of Tabular Encoders (NeurIPS 2026 · similarity 0.86)
- TabPrep: Closing the Feature Engineering Gap in Tabular Benchmarks (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.