Active Learning in Black-Box Settings
Austrian Journal of Statistics, 40 (1&2):
125-135 (2011)Abstract
Active learning refers to the settings in which a machine learning
algorithm (learner) is able to select data from which it learns (selecting
points and then obtaining their labels), and by doing so aims to
achieve better accuracy (e.g., by avoiding obtaining training data
that is redundant or unimportant). Active learning is particularly
useful in cases where the labeling cost is high. A common assumption
is that an active learning algorithm is aware of the details of the
underlying learning algorithm for which it obtains the data. However,
in many practical settings, obtaining precise details of the learning
algorithm may not be feasible, making the underlying algorithm in
essence a black box � no knowledge of the internal workings of the
algorithm is available, and only the inputs and corresponding output
estimates are accessible. This makes many of the traditional approaches
not applicable, or at the least not effective. Hence our motivation
is to use the only data that is accessible in black box settings
� output estimates. We note that accuracy will improve only if the
learner�s output estimates change. Therefore we propose active learning
criterion that utilizes the information contained within the changes
of output estimates.