ROBIN: A Benchmark for Robustness to Individual Nuisances
in Real-World Out-of-Distribution Shifts
Bingchen Zhao
Shaozuo Yu
Wufei Ma
Mingxin Yu
Shenxiao Mei
Angtian Wang
Ju He
Alan Yuille
Adam Kortylewski
[Paper]
[GitHub]

Abstract

Enhancing the robustness in real-world scenarios has been proven very challenging. One reason is that existing robustness benchmarks are limited, as they either rely on synthetic data or they simply measure robustness as generalization between datasets and hence ignore the effects of individual nuisance factors. In this work, we introduce ROBIN, a benchmark dataset for diagnosing the robustness of vision algorithms to individual nuisances in real-world images. ROBIN builds on 10 rigid categories from the PASCAL VOC 2012 and ImageNet datasets, and includes out-of-distribution examples of the object's 3D pose, shape, texture, context and the weather conditions. ROBIN is richly annotated to enable benchmark models for image classification, object detection, and 3D pose estimation. We provide results for a number of popular baselines and make several interesting observations: 1) Some nuisance factors have a much stronger negative effect on the performance compared to others. Moreover, the negative effect of an OOD nuisance depends on the downstream vision task. 2) Current approaches to enhance OOD robustness using strong data augmentation have only marginal effects in real-world OOD scenarios, and sometimes even reduce the OOD performance. 3) We do not observe any significant differences between convolutional and transformer architectures in terms of OOD robustness. We believe our dataset provides a rich testbed to study OOD robustness of vision algorithms and will help to significantly push forward research in this area.

Paper and Supplementary Material

B. Zhao, S. Yu, W. Ma, M. Yu, S. Mei, A. Wang, J. He, A. Yuille, A. Kortylewski.
ROBIN: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts.
(hosted on ArXiv)


BibTex

			
    @article{zhao21robin,
	author  = {Bingchen Zhao and Shaozuo Yu and Wufei Ma and Mingxin Yu and Shenxiao Mei and Angtian Wang and Ju He 
and Alan Yuille and Adam Kortylewski}, title = {ROBIN: A Benchmark for Robustness to Individual Nuisances in Real-World Out-of-Distribution Shifts}, journal = {arXiv preprint arXiv:2111.14341}, year = {2021} }


This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.