ProjectBlog: FastAI Sentiment Analysis

Blog: FastAI Sentiment Analysis

Analyze the sentiment of Tweets using the FastAI deep learning library

Sentiment analysis refers to the use of natural language processing, text analysis, computational linguistics, and other techniques to identify and quantify the sentiment (i.e. positive, neutral, or negative) of text or audio data.

Because it’s really hard for a model to learn language when only provided with a single value — the sentiment — FastAI lets you first train a language model — a model that predicts the next word — and then use that encoder for the model that actually classifies the sentiment.

This allows the classification model to focus on learning the sentiment instead of having to learn the whole language from scratch. This approach offers astonishingly good results.

Loading data

For our data, we’ll use the Twitter US Airline Sentiment dataset from Kaggle. The dataset contains tweets about US Airlines, annotated with their respective sentiments.

After downloading the dataset, we need to import the text module from the FastAI library, specify the path to our dataset, and load in our csv using pandas.

Figure 2: Dataframe head

The two columns that are important for our problem are the airline_sentiment column, which contains a sentiment value —if you’ll recall, either negative, neutral, or positive — and the text column.

As mentioned above, we’ll create two models. A language model and a text classification model; therefore we’ll create two different data bunches using the TextLMDataBunch and TextClasDataBunch classes:

We can get a few examples of each data bunch using the show_batch method:

Figure 3: Language model data

Figure 4: Classification data

You may notice that there are quite a few strange tokens starting with xx. These are special FastAI tokens that have the following meanings:

  • xxunk: Token used instead of unknown words (words not found in the vocabulary).
  • xxbos: Beginning of a text.
  • xxfld: Represents separate parts of your document (several columns in a dataframe) like headline, body, summary, etc.
  • xxmaj: Indicates that the next word starts with a capital, e.g. “House” will be tokenized as “xxmaj house”.
  • xxup: Indicates that next word is written in all caps, e.g. “WHY” will be tokenized as “xxup why ”.
  • xxrep: Token indicates that a character is repeated n times, e.g. if you have 10 $ in a row it will be tokenized as “xxrep 10 $” (in general “xxrep n {char}”)
  • xxwrep: Indicates that a word is repeated n times.
  • xxpad : Token used as padding (so every text has the same length)

Language model

In FastAI, a language model can be created using the language_model_learner method. We’ll pass this method three arguments. Our data, a pre-trained model (trained on Wikipedia text), and a dropout percentage.

After creating the model, we’ll follow the standard FastAI training pipeline, which includes finding the best learning rate, training the top layers, unfreezing all layers, and repeating the above process.

If you aren’t familiar with this process yet, I’d highly recommend checking out the Practical Deep Learning for Coders course, which is excellent for anyone who wants to learn about cutting edge deep learning approaches. Or you can check out my first FastAI tutorial, which goes into this process in detail.

Figure 5: Learning rate helps us find the optimal learning rate as described here.

Figure 6: Training results (shows loss and accuracy metrics after the end of the epoch)

Figure 7: Learning rate plot

Figure 8: Training results

Now that we have our trained language model, we can make predictions and save the encoder to use for our classification model:

Figure 9: Text generation result

Classification model

With our language model ready and the encoder saved, we can now create a text classification model, load in the encoder, and train the network.

Figure 10: Learning rate plot

For training the classification model, we’ll start off by training the decoder, and then we’ll unfreeze one layer after another—this approach was found to give better results for text classification models.

With our model trained, we can now make predictions using the predict method:

This outputs a tuple containing the class as a string and integer as well as the probability values for each class.

Out: (Category positive, tensor(2), tensor([0.1067, 0.0125, 0.8808]))


The FastAI library offers us a high-level API capable of creating deep learning models for a lot of different applications, including text generation, text analysis, image classification, and image segmentation.

For more information on the FastAI library check out the Practical Deep Learning for Coders course, which goes through a lot of cutting edge deep learning techniques.

If you liked this article consider subscribing to my YouTube Channel and following me on social media.

The code covered in this article is available as a GitHub Repository.

If you have any questions, recommendations or critiques, I can be reached via Twitter or the comment section.

Editor’s Note: Ready to dive into some code? Check out Fritz on GitHub. You’ll find open source, mobile-friendly implementations of the popular machine and deep learning models along with training scripts, project templates, and tools for building your own ML-powered iOS and Android apps.

Join us on Slack for help with technical problems, to share what you’re working on, or just chat with us about mobile development and machine learning. And follow us on Twitter and LinkedIn for all the latest content, news, and more from the mobile machine learning world.

Source: Artificial Intelligence on Medium

Leave a Reply

Your email address will not be published. Required fields are marked *

Back To Top

Display your work in a bold & confident manner. Sometimes it’s easy for your creativity to stand out from the crowd.