AI Paper Atlas

Research map / Probabilistic and scientific modeling / Molecules and proteins

Great Models Think Alike: Improving Model Reliability via Inter-Model Latent Agreement

Ailin Deng, Miao Xiong, Bryan Hooi

ICML 2023 · Probabilistic and scientific modeling › Molecules and proteins · approach: Neuro-symbolic probabilistic layers · 0 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add agreement score (CLIP ViT/L-14 + ViT/L-16 multiple foundation models) to temperature scalingMSP/temperature scaling failure detection · ID datasets (CIFAR10, CIFAR100, STL, BIRDS, FOOD, ImageNet), CNN and ViTAUROC (%): e.g. CIFAR10 CNN 93.76 -> 98.07Improved
add agreement score with a single foundation model (CLIP ViT/L-14) to temperature scalingtemperature scaling · ID datasets, CNN and ViTAUROC (%): e.g. CIFAR100 CNN 87.21 -> 89.63Improved
add agreement score with multiple foundation models vs single (ablation)temperature scaling with agreement score · ID datasets, CNN and ViTAUROC (%): multiple better on CIFAR10/CIFAR100/BIRDS/ImageNet CNN; single better on FOOD (92.33 vs 92.17 CNN, 93.81 vs 93.64 ViT)Mixed
add agreement score to temperature scalingtemperature scaling under distribution shift · CIFAR10.1, CIFAR10-C, CIFAR100-C, ImageNetV2, ImageNet-SketchAUROC (%): e.g. CIFAR10-C CNN 78.14 -> 92.76; ImageNetV2 ViT 83.68 -> 84.77Improved
replace NDCG agreement with Spearman rank correlation (ablation)agreement measure · ID average over datasetsAvg AUROC (%): 92.67 -> 90.87Hurt
replace NDCG agreement with CKA (ablation)agreement measure · ID average over datasetsAvg AUROC (%): 92.67 -> 90.99Hurt
replace NDCG agreement with Jaccard similarity of k-hop neighborhoods (ablation)agreement measure · ID average over datasetsAvg AUROC (%): 92.67 -> 92.59Hurt
increase neighbor candidate pool size n from 2000 to 50000 (ablation)agreement score with CIFAR10/CIFAR100 neighbors · CIFAR10 and CIFAR100AUROC (%): CIFAR10 97.26 -> 97.50; CIFAR100 89.08 -> 90.38Improved
use a small neighbor pool (n=2000) instead of larger pools (ablation)agreement score · CIFAR10 and CIFAR100AUROC (%): still above MSP (94.15 / 88.42) and T.S. baselinesNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy