Member-only story
Hands-on Tutorials
Transformers, can you rate the complexity of reading passages?
Fine-tuning RoBERTa with PyTorch to predict reading ease of text excerpts
Transformers, what are they actually? They are not the devices used in power transmission of electric energy, nor the fictional living autonomous robots Optimus Prime or Bumblebee who can transform into other objects like vehicles. Transformers in our context here is referring to BERT, ALBERT, RoBERTa, and the like, where they are used in the world of data science to solve all kinds of Natural Language Processing tasks, such as machine translation, text summarization, speech recognition, sentiment analysis and many more. They are state-of-the-art language models for Natural Language Processing, and they have gained tremendous popularity over the past few years.
This post will demonstrate the fine-tuning of Transformer model, specifically RoBERTa, on our dataset of interest. Fine-tuning is done for a downstream task to predict the reading ease of excerpts from literature for grades 3–12 classroom use.
This piece of work is motivated by an initiative from CommonLit, a nonprofit education technology organization. It sponsored a competition hosted on Kaggle (you can read more about it here), aiming to use machine learning to seek improvement over existing readability rating methods. This will aid literacy curriculum developers as well as educators in choosing appropriate reading passages for students. Presenting engaging passages with the right level of complexity and reading challenge will greatly benefit students in developing essential reading skills.
Table of contents
1. About the Dataset
2. Splitting the Data
3. Creating the Dataset Class
4. roberta-base As Our Model
5. What are the typical raw outputs from Transformers?
6. Defining the Model Class
∘ Using pooler_output
∘ Building your own Transformer custom head
∘ (A) Attention head
∘ (B) Concatenate hidden layers
7. Model Training
∘ Evaluation Metric and Loss Function
∘ Training Function
∘ Validation Function
∘ Run Training
Summary
References
