Hands-on Tutorials

Transformers, can you rate the complexity of reading passages?

Fine-tuning RoBERTa with PyTorch to predict reading ease of text excerpts

Peggy Chang
Towards Data Science
14 min readAug 18, 2021

--

Article cover for “Transformers, can you rate the complexity of reading passages?” Fine-tuning RoBERTa with PyTorch to predict reading ease of text excerpts. Author: Peggy Chang
Photo by Jeffery Ho on Unsplash

Transformers, what are they actually? They are not the devices used in power transmission of electric energy, nor the fictional living autonomous robots Optimus Prime or Bumblebee who can transform into other objects like vehicles. Transformers in our context here is referring to BERT, ALBERT, RoBERTa, and the like, where they are used in the world of data science to solve all kinds of Natural Language Processing tasks, such as machine translation, text summarization, speech recognition, sentiment analysis and many more. They are state-of-the-art language models for Natural Language Processing, and they have gained tremendous popularity over the past few years.

This post will demonstrate the fine-tuning of Transformer model, specifically RoBERTa, on our dataset of interest. Fine-tuning is done for a downstream task to predict the reading ease of excerpts from literature for grades 3–12 classroom use.

This piece of work is motivated by an initiative from CommonLit, a nonprofit education technology organization. It sponsored a competition hosted on Kaggle (you can read more about it here), aiming to use machine…

--

--

Published in Towards Data Science

Your home for data science and AI. The world’s leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.

Written by Peggy Chang

Machine Learning | Data Science | “Getting 1% better every day” | https://www.linkedin.com/in/peggy1502/

Responses (6)