|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Enhancing the robustness of vision algorithms in real-world scenarios is challenging. One reason is that existing robustness benchmarks are limited, as they either rely on synthetic data or ignore the effects of individual nuisance factors. We introduce ROBIN, a benchmark dataset that includes out-of-distribution examples of 10 object categories in terms of pose, shape, texture, context and the weather conditions, and enables benchmarking models for image classification, object detection, and 3D pose estimation. Our experiments using popular baseline methods reveal that: 1) Some nuisance factors have a much stronger negative effect on the performance compared to others, also depending on the vision task. 2) Current approaches to enhance robustness have only marginal effects, and can even reduce robustness. 3) We do not observe significant differences between convolutional and transformer architectures. We believe our dataset provides a rich testbed to study robustness and will help push forward research in this area. Our dataset can be accessed from \href{here}{http://www.ood-cv.org/challenge.html}. |
To access the full dataset for research, access [Classification] [Detection] [Pose Estimation]. The data in Classification and Detection contains three folders, phase-1 phase-2 train, the train folder contains the training set, and the phase-1 and phase-2 folders contain the test sets for the two phases of the challenge. One can use phase-1 data as a validation set and then test on phase-2 data, or the two dataset can be combined as one validation set.
Additionally, we provide a data processing tool and baseline for the 3D pose estimation task, which can be accessed from [GitHub]. |
| To access the dataset we use for the challenge at ECCV 2022 and reproduce the performance of the challenge participants, access [Phase-1 Data] and [Phase-2 Data] |
 |
B. Zhao, S. Yu, W. Ma, M. Yu, S. Mei, A. Wang, J. He, A. Yuille, A. Kortylewski. OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images. In ECCV, 2022. (hosted on ArXiv) |
@inproceedings{zhao22oodcv,
author = {Bingchen Zhao and Shaozuo Yu and Wufei Ma and Mingxin Yu and Shenxiao Mei and Angtian Wang and Ju He
and Alan Yuille and Adam Kortylewski},
title = {OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images},
booktitle = {ECCV},
year = {2022}
}
|
|