Hands-on Tutorials
Transformers, can you rate the complexity of reading passages?
Fine-tuning RoBERTa with PyTorch to predict reading ease of text excerpts
Transformers, what are they actually? They are not the devices used in power transmission of electric energy, nor the fictional living autonomous robots Optimus Prime or Bumblebee who can transform into other objects like vehicles. Transformers in our context here is referring to BERT, ALBERT, RoBERTa, and the like, where they are used in the world of data science to solve all kinds of Natural Language Processing tasks, such as machine translation, text summarization, speech recognition, sentiment analysis and many more. They are state-of-the-art language models for Natural Language Processing, and they have gained tremendous popularity over the past few years.
This post will demonstrate the fine-tuning of Transformer model, specifically RoBERTa, on our dataset of interest. Fine-tuning is done for a downstream task to predict the reading ease of excerpts from literature for grades 3–12 classroom use.
This piece of work is motivated by an initiative from CommonLit, a nonprofit education technology organization. It sponsored a competition hosted on Kaggle (you can read more about it here), aiming to use machine…