To Annotate or Not? Predicting Performance Drop under Domain Shift

Partner responsible/main author: Naver Authors: Hady Elsahar, Matthias Gallé Published in: EMNLP, Association for Computational Linguistics Performance drop due to domain-shift is an endemic problem for NLP models in production. This problem creates an urge to continuously annotate evaluation datasets to measure the expected drop in the model performance which can be prohibitively expensive and…