An Experimental Evaluation of Transformer-based LanguageModels in the Biomedical Domain

Paul Grouchi (Untether AI); Shobhit Jain (Manulife); Michael Liu (Tealbook); Kuhan Wang (CIBC); Max Tian (Adeptmind); Nidhi Arora (Intact); Hillary Ngai (University of Toronto); Faiza Khan Khattak (Manulife); Elham Dolatabadi and Sedef Akinli Kocak (Vector Institute)

Abstract: With the growing amount of text in health data, there have beenrapid advances in large pre-trained models that can be applied to awide variety of biomedical tasks with minimal task-specific mod-ifications. Emphasizing the cost of these models, which renderstechnical replication challenging, this paper summarizes experi-ments conducted in replicating BioBERT and further pre-trainingand careful fine-tuning in the biomedical domain. We also inves-tigate the effectiveness of domain-specific and domain-agnosticpre-trained models across downstream biomedical NLP tasks. Ourfinding confirms that pre-trained models can be impactful in somedownstream NLP tasks (QA and NER) in the biomedical domain;however, this improvement may not justify the high cost of domain-specific pre-training.