Mixup is the latest data augmentation technique that linearly interpolates
input examples and the corresponding labels. It has shown strong effectiveness
in image classification by interpolating images at the pixel level. Inspired by
this line of research, in this paper, we explore i) how to apply mixup to
natural language processing tasks since text data can hardly be mixed in the
raw format; ii) if mixup is still effective in transformer-based learning
models, e.g., BERT. To achieve the goal, we incorporate mixup to
transformer-based pre-trained architecture, named "mixup-transformer", for a
wide range of NLP tasks while keeping the whole end-to-end training system. We
evaluate the proposed framework by running extensive experiments on the GLUE
benchmark. Furthermore, we also examine the performance of mixup-transformer in
low-resource scenarios by reducing the training data with a certain ratio. Our
studies show that mixup is a domain-independent data augmentation technique to
pre-trained language models, resulting in significant performance improvement
for transformer-based models.
Description
Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks
%0 Generic
%1 sun2020mixuptransformer
%A Sun, Lichao
%A Xia, Congying
%A Yin, Wenpeng
%A Liang, Tingting
%A Yu, Philip S.
%A He, Lifang
%D 2020
%K DA level token
%T Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks
%U http://arxiv.org/abs/2010.02394
%X Mixup is the latest data augmentation technique that linearly interpolates
input examples and the corresponding labels. It has shown strong effectiveness
in image classification by interpolating images at the pixel level. Inspired by
this line of research, in this paper, we explore i) how to apply mixup to
natural language processing tasks since text data can hardly be mixed in the
raw format; ii) if mixup is still effective in transformer-based learning
models, e.g., BERT. To achieve the goal, we incorporate mixup to
transformer-based pre-trained architecture, named "mixup-transformer", for a
wide range of NLP tasks while keeping the whole end-to-end training system. We
evaluate the proposed framework by running extensive experiments on the GLUE
benchmark. Furthermore, we also examine the performance of mixup-transformer in
low-resource scenarios by reducing the training data with a certain ratio. Our
studies show that mixup is a domain-independent data augmentation technique to
pre-trained language models, resulting in significant performance improvement
for transformer-based models.
@misc{sun2020mixuptransformer,
abstract = {Mixup is the latest data augmentation technique that linearly interpolates
input examples and the corresponding labels. It has shown strong effectiveness
in image classification by interpolating images at the pixel level. Inspired by
this line of research, in this paper, we explore i) how to apply mixup to
natural language processing tasks since text data can hardly be mixed in the
raw format; ii) if mixup is still effective in transformer-based learning
models, e.g., BERT. To achieve the goal, we incorporate mixup to
transformer-based pre-trained architecture, named "mixup-transformer", for a
wide range of NLP tasks while keeping the whole end-to-end training system. We
evaluate the proposed framework by running extensive experiments on the GLUE
benchmark. Furthermore, we also examine the performance of mixup-transformer in
low-resource scenarios by reducing the training data with a certain ratio. Our
studies show that mixup is a domain-independent data augmentation technique to
pre-trained language models, resulting in significant performance improvement
for transformer-based models.},
added-at = {2023-05-12T17:51:29.000+0200},
author = {Sun, Lichao and Xia, Congying and Yin, Wenpeng and Liang, Tingting and Yu, Philip S. and He, Lifang},
biburl = {https://www.bibsonomy.org/bibtex/2065db84eada33c24444ddfa2c1092915/hassanpour71},
description = {Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks},
interhash = {a49db1c8b9582017c267aa7f3d8d11f4},
intrahash = {065db84eada33c24444ddfa2c1092915},
keywords = {DA level token},
note = {cite arxiv:2010.02394Comment: Accepted by COLING 2020},
timestamp = {2023-05-12T17:51:29.000+0200},
title = {Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks},
url = {http://arxiv.org/abs/2010.02394},
year = 2020
}