Abstract
We present a simple self-training method that achieves 87.4% top-1 accuracy
on ImageNet, which is 1.0% better than the state-of-the-art model that requires
3.5B weakly labeled Instagram images. On robustness test sets, it improves
ImageNet-A top-1 accuracy from 16.6% to 74.2%, reduces ImageNet-C mean
corruption error from 45.7 to 31.2, and reduces ImageNet-P mean flip rate from
27.8 to 16.1.
To achieve this result, we first train an EfficientNet model on labeled
ImageNet images and use it as a teacher to generate pseudo labels on 300M
unlabeled images. We then train a larger EfficientNet as a student model on the
combination of labeled and pseudo labeled images. We iterate this process by
putting back the student as the teacher. During the generation of the pseudo
labels, the teacher is not noised so that the pseudo labels are as good as
possible. But during the learning of the student, we inject noise such as data
augmentation, dropout, stochastic depth to the student so that the noised
student is forced to learn harder from the pseudo labels.
Users
Please
log in to take part in the discussion (add own reviews or comments).