Month: November 2022

This tutorial explains how to perform multiple-label text classification using the Hugging Face transformers library. Hugging Face library implements advanced transformer architectures, proven to be state-of-the-art for various natural language processing tasks, including text classification.

Hugging Face library provides trainable transformer models in three flavors:

1. Via the Trainer Class API
2. Via PyTorch Models
3. Via TensorFlow Models

The HuggingFace documentation for Trainer Class API is very clear and easy to use. However, I wanted to train my text classification model in TensorFlow. After some research, I found that the Hugginface API lacks documentation on fine-tuning transformers models for multilabel text classification in TensorFlow.

In this tutorial, I will explain how I fine-tuned a Hugging Face transformers model for multilabel text classification in TensorFlow.

I will use the Toxic Comment Dataset From Kaggle to fine-tune my transformer model. Download the dataset's CSV file and import it into your Python script using the Pandas dataframe, as shown in the following script:

import pandas as pd

dataset = pd.read_csv('/content/fake-and-real-news-dataset/train.csv')
print(dataset.shape)
dataset.head()

Output:

The above output shows that the dataset contains more than 159k records. The dataset consists of 8 columns. The text comment_text column contains user comments. A comment can be categorized into one or more categories: toxic, severe toxic, obscene, threat, insult, or identity hate. A one is added in a column if a comment belongs to the column category, else a zero is added.

Several comments in the dataset do not fall into any of the comment categories. The following script returns these records:

no_toxic_comments_df = dataset[(dataset[['toxic', 
                                         'severe_toxic',
                                         'obscene', 
                                         'threat', 
                                         'insult',
                                         'identity_hate']] == 0).all(axis=1)]

print(no_toxic_comments_df.shape)

Output:

(143346, 8)

The above output shows that more than 143k records do not fall into any comment category. I will remove these records since I am only interested in comments assigned to at least one category.

toxic_comments_df = dataset[(dataset[['toxic', 
                                         'severe_toxic',
                                         'obscene', 
                                         'threat', 
                                         'insult',
                                         'identity_hate']] != 0).any(axis=1)]

print(toxic_comments_df.shape)

Output:

(16225, 8)

Like every machine learning problem, we need to divide our dataset into features and labels set before model training. Subsequently, we need to divide our dataset into training and test sets. The following script does that.

X = list(toxic_comments_df['comment_text'])
y = toxic_comments_df.drop(['id', 'comment_text'], axis = 1).values.tolist()


from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)

Hugging Face transformers model accept data input data to be in a particular format. You can use tokenizers to convert raw text into the Hugging Face complaint format.

The script below installs the Hugging Face library.

! pip install datasets transformers[sentencepiece]

The following script defines the transformer model (English Distil Bert), and the tokenizer for the model.

model_name = "distilbert-base-uncased-finetuned-sst-2-english"

from transformers import AutoTokenizer, TFAutoModel
tokenizer = AutoTokenizer.from_pretrained(model_name, do_lowercase = True)
bert = TFAutoModel.from_pretrained(model_name, from_pt = True)

The Tokenizer class object converts the input training and test sets into Distil Bert complaint input format in the following script.

train_encodings = tokenizer(X_train, truncation=True, padding="max_length", max_length=512)
test_encodings = tokenizer(X_test, truncation=True, padding="max_length", max_length=512)

And the script below creates TensorFlow datasets (including output labels) for TensorFlow model training and testing.

train_dataset = tf.data.Dataset.from_tensor_slices((
    dict(train_encodings),
    y_train
))
test_dataset = tf.data.Dataset.from_tensor_slices((
    dict(test_encodings),
    y_test
))

The Hugging Face model can be added as an encoder layer to the TensorFlow model. The input is passed to the encoder layer.

Following are the steps to incorporate a Hugging Face transformer model for fine-tuning as a TensorFlow model:

1. Create a Python Class that inherits from the Keras.Model class.
2. Pass the Hugging Face transformer model to the constructor of the Python class you created in the first step. The encoder variable stores the Huggin Face model in the following script.
3. In the Call() method of the class, pass the input (TF datasets) to the encoder layer.
4. Subsequently, add the standard TensorFlow layer to define the overall model architecture.

The following script defines the TensorFlow model that fine-tunes the Hugging Face transformer. I added three dense layers after the encoder layer. The final dense layer contains six nodes since we have six comment categories in the output.

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

import numpy as np

class TextClassificationModel(keras.Model):
  def __init__(self, encoder):
    super(TextClassificationModel, self).__init__()
    self.encoder = encoder
    self.encoder.trainable = True
    self.dropout1 = layers.Dropout(0.1)
    self.dropout2 = layers.Dropout(0.1)
    self.dropout3 = layers.Dropout(0.1)
    self.dense1 = layers.Dense(100, activation="relu")
    self.dense2 = layers.Dense(50, activation="relu")
    self.dense3 = layers.Dense(6, activation="sigmoid")

  def call(self, input):
    x = self.encoder(input)
    x = x['last_hidden_state'][:, 0, :]
    x = self.dropout1(x)
    x = self.dense1(x)
    #x = self.dropout2(x)
    x = self.dense2(x)
    #x = self.dropout3(x)
    x = self.dense3(x)
    return x

The script below initializes and compiles our TensorFlow model.

text_classification_model = TextClassificationModel(bert)

metric = "binary_crossentropy"

text_classification_model.compile(
    optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5), 
    loss = tf.keras.losses.BinaryCrossentropy(from_logits=True), 
    metrics=tf.keras.metrics.BinaryCrossentropy(
    name="binary_crossentropy", dtype=None, from_logits=False, label_smoothing=0))

Finally, you can train the model using the TensorFlow model class's fit() method.

history = text_classification_model.fit(
    train_dataset.shuffle(1000).batch(16), 
    epochs=3, 
    validation_data=test_dataset.batch(16)
)

Let's plot the model loss against the number of epochs.

import seaborn as sns
import matplotlib.pyplot as plt

sns.set(rc={'figure.figsize':(10,8)})


sns.set_context('poster', font_scale = 1)

plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss - ' + metric)
plt.ylabel('loss - ' + metric)
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()

Output:

The above plot shows that we achieved the lowest loss after the first epoch, and the model started to overfit after that.

The following script makes predictions on the test set.

y_pred = text_classification_model.predict(test_dataset.batch(16))
y_pred[0]

Output:

array([9.9412137e-01, 2.0126806e-04, 1.7747579e-03, 7.2536163e-04,
       3.4322389e-03, 9.6116390e-04], dtype=float32)

The dataset input labels are binary, while the output predictions are continuous numeric values. We will convert continuous output values to binary to compare the test labels with output predictions. All the output values greater than 0.5 are converted to 1, while the values less than or equal to 0.5 are converted to 0.

The following script evaluates the model performance.

from sklearn.metrics import roc_auc_score, classification_report

y_pred = (y_pred >0.50)


print(classification_report(y_test, y_pred))

roc_auc = roc_auc_score(y_test, y_pred, average = 'macro')
print(roc_auc)

Output:

In this tutorial, you will see how to convert the text in CSV file columns to other languages using the DeepL API in the Python programing language.

DeepL is one of the most popular and accurate text translation platforms. DeepL, as the name suggests, incorporates advanced deep learning algorithms for training text translation models.

In addition to raw text strings, DeepL supports translating documents in PDF, MS Word, and PowerPoint formats. However, I wanted to translate text in CSV file columns, which DeepL does not support.

In this tutorial, I will explain how I achieved translating text in CSV columns using the DeepL API. The resultant CSV will have new columns containing translated text.

I chose Python language for translating text in CSV files since DeepL has an official Python client that you can exploit for text translation in your code.

The official GitHub repository explains installing the DeepL API along with sample scripts.

Here I will provide a simple example for your reference. The following are the steps:

Create an Object of the Translator class and pass it your DeepL authorization key.
Pass the text you want to translate to the translate_text() method of the Translator class. You must also pass the ISO 639-1 standard language code to the target_lang attribute of the translate_text() method.
To get the translated text, access the text attribute of the object returned by the translate_text() function.

Here is an example:

import os
from deepl import Translator

auth_key = os.environ['deepL-key']
dl_translator = Translator(auth_key)

result = dl_translator.translate_text("Python is awesome", target_lang="FR")
print(result.text) 

Output:

Python est gnial

Lets now see how to translate a CSV file. For example, I will translate a CSV file containing Yelp Reviews. The file looks like the one in the following screenshot. The text column contains reviews in the English language.

A Pandas Dataframe is ideal for reading, manipulating, and saving CSV files in Python. The following script imports the yelp.csv file into a Pandas dataframe.

import pandas as pd

dataset = pd.read_csv("D:\Datasets\yelp.csv")

dataset.head()

Output:

I will remove all the columns except the text column. You can only translate 500,000 characters with the free DeepL API. Therefore, for the sake of demonstration, I will keep the first 20 rows of the dataset.

import pandas as pd

dataset = pd.read_csv("D:\Datasets\yelp.csv")

dataset = dataset.filter(["text"], axis = 1).head(20)
dataset.head()

Output:

Lets now translate reviews in the text column. I want to add a new column that contains a translation of the reviews in the corresponding row in the text column.

To do so, I will define a translate_df() function that accepts a Pandas dataframe. The function will call the translate_text() method to translate text in the individual rows of the text column.

Next, I can use the apply() method of Pandas dataframe and pass the function variable translate_df to this method. The apply() method applies the translate_df() function to each row of the input Pandas dataframe. I will add a new column named text_fr, which contains the translated text.

The following script translates text into the French language.

import os
import pandas as pd
from deepl import Translator

auth_key = os.environ['deepL-key']
dl_translator = Translator(auth_key)

def translate_df(df):
     return dl_translator.translate_text(df["text"], target_lang="FR").text


dataset["text_fr"] = dataset.apply(translate_df, axis = 1)
dataset.head()

Output:

Instead of hardcoding the target language in the translate_df() function, you can pass the target language as a parameter value to the translate_df() function. Here is an example that translates the input text into the Spanish language.

import os
import pandas as pd
from deepl import Translator

auth_key = os.environ['deepL-key']
dl_translator = Translator(auth_key)

def translate_df(df, lang):
     return dl_translator.translate_text(df["text"], target_lang= lang).text

language = "es"
dataset["text_es"] = dataset.apply(translate_df, 
                                   lang = language, 
                                   axis = 1)
dataset.head()

Output:

You might want to convert text into multiple languages. To do so, you can modify the translate_df() and pass it a list of target languages. You can then iterate through the target languages list to translate the input text.

The translated text for each language is appended to a list that you can convert to a Pandas series. Finally, you can return the Pandas series to the calling function. The following script modifies the translate_text() function.

import os
import pandas as pd
from deepl import Translator

auth_key = os.environ['deepL-key']
dl_translator = Translator(auth_key)

def translate_df(df, lang):

    translate_list = []
    for i in lang:
        translate_list.append(dl_translator.translate_text(df["text"], target_lang= i).text)

    return pd.Series(translate_list)

The script below converts the text in the text column of the input Pandas dataframe to Spanish and French.

import numpy as np
languages = ["es", "fr"]
col_names = ["text_es", "text_fr"]

for newcol in col_names:
    dataset[newcol]=np.nan

dataset[col_names] = dataset.apply(translate_df, 
                                   lang = languages, 
                                   axis = 1)

dataset.head()

In the output below, you can see that the input text is converted to Spanish and French.

Output:

Once you translate text in a Pandas dataframe, you can easily store the Pandas dataframe as a CSV file using the to_csv() function. Here is an example:

dataset.to_csv("D:\Datasets\yelp_translated.csv", index = False)

The output below shows the CSV file containing the input text and corresponding translations in Spanish and French.

Output:

DeepL Library is a convenient library for text translation. It provides raw text and document translation services for PDF, MS Word, and PowerPoint documents. However, you cannot translate text in CSV file columns by default. In this tutorial, I explained how you could achieve that using DeepL API Python client and Pandas dataframes. I hope you will find it helpful. If you have any suggestions or improvements, feel free to comment.

I was working on a problem where I had to scrape tweets related to the T20 Cricket World Cup 2022, which is currently taking place in Australia.

I wanted tweets containing location names (cities) and the keyword T20. In the response, I want the user names of tweet authors, tweet texts, creation time, and the location keyword used to search the tweet. Finally, I wanted to create a Python Pandas Dataframe that contains these values in columns.

In this article, I will explain how you can return scrape tweets containing location information and how to store these tweets in a Pandas Dataframe.

Developers can scrape tweets from Twitter using the Twitter REST API. In most cases, the Twitter API returns tweet-type objects that contain various attributes for extracting tweet information. However, by default, the Twitter API doesn't return a Pandas Dataframe.

You must sign up with Twitter Developer Account and create your API Key and Token to access the Twitter REST API. The official documentation explains signing up for the Twitter Developer Account.

I will use the Python Tweepy library for accessing the Twitter API. Tweepy is an unofficial Python client for accessing the Twitter API.

The following script demonstrates a basic example of Twitter scraping with the Python Tweepy library.

I use the search_all_tweets() function to search 100 English language tweets containing keywords Sydney and T20. I set a filter for removing retweets.

import pandas as pd
import os
import tweepy

bt = os.environ['twitter-bt']

client = tweepy.Client(bearer_token = bt,
                      wait_on_rate_limit= False)

location = "Sydney T20"
language = "lang:en "
no_tweet = "-is:retweet "


query = '"'+location+'" ' + language + no_tweet

tweets = client.search_all_tweets(query = query, 
                                  max_results=100)

tweets.data

You can use the data attribute of the response object to print the returned tweets. You can access tweet text using the text attribute as shown below:

Output:

Now you know how to scrape tweets with Tweepy, lets scrape tweets with the following information and store them in a Pandas dataframe:

1. Username of the tweet author.
2. Tweet text.
3. Creation time of the tweet.

I will scrape tweets with the following filters:

1. Location keywords, e.g., Sydney T20
2. English language
3. No Retweets
4. Between October 01, 2022, to October 30, 2022.

In this case, I passed values for the tweet_fields, user_fields, and expansions attributes of the search_all_tweets() function. These attributes extract tweet creation time and user information, e.g., username.

location = "Sydney T20"
language = "lang:en "
no_tweet = "-is:retweet "

start_time = '2022-10-01T00:00:00Z'
end_time = '2022-10-30T00:00:00Z'


query = '"'+location+'" ' + language + no_tweet

tweets = client.search_all_tweets(query = query, 
                                  tweet_fields=['created_at'],
                                  user_fields=['username'],
                                  expansions=['author_id'],
                                  start_time=start_time,
                                  end_time=end_time, 
                                  max_results=100)

tweets.includes['users']

In addition to the data attribute, which contains tweet information, you can retrieve user information using the tweets.includes[users] list.

Output:

You can iterate through lists of tweet data and user data to extract user name, tweet, text, and tweet creation time and append these values to Python lists.

Finally, you can create a Pandas dataframe using these lists. Here is an example script:

user_name = []
text = []
date_time = []


for tweet, user  in zip(tweets.data, tweets.includes['users']):
    text.append(tweet.text)
    date_time.append(tweet.created_at)
    if user is not None:
        user_name.append(user.name)


df = pd.DataFrame(list(zip(user_name, text, date_time)),
               columns =['User', 'Text', 'Date Time'])

df.head(20)

Output:

Since I wanted to extract tweets by multiple location keywords, I defined a Python function that accepts a list of locations, along with the start and end date of tweets. The function iterates through all the locations and appends the extracted tweet information in Python lists.

The get_location_tweets() function in the following script returns a Pandas dataframe containing the tweet text, the tweet author's username, the tweet's creation time, and the location keyword that parameterizes the tweet search.

import time
def get_location_tweets(loc_list, st, et):


    user_name = []
    text = []
    date_time = []
    loc_keyword = []

    i = 0

    for loc in loc_list:

        i = i + 1

        location = loc + " T20"
        language = "lang:en "
        no_tweet = "-is:retweet "

        start_time = st
        end_time = et


        query = '"'+location+'" ' + language + no_tweet

        tweets = client.search_all_tweets(query = query, 
                                          tweet_fields=['created_at'],
                                          user_fields=['username'],
                                          expansions=['author_id'],
                                          start_time=start_time,
                                          end_time=end_time, 
                                          max_results=100)


        user_name = []
        text = []
        date_time = []


        for tweet, user  in zip(tweets.data, tweets.includes['users']):
            text.append(tweet.text)
            date_time.append(tweet.created_at)
            if user is not None:
                user_name.append(user.name)

            loc_keyword.append(loc)

        time.sleep(3)


    df = pd.DataFrame(list(zip(user_name, text, date_time, loc_keyword)),
                   columns =['User', 'Text', 'Date Time', 'Loc Keyword'])


    return df

The following script calls the get_location_tweets() function to search for tweets containing either Sydney, Perth, Melbourne and the keyword T20.

loc_list = ["Sydney", "Perth", "Melbourne"]

start_time = '2022-10-01T00:00:00Z'
end_time = '2022-10-30T00:00:00Z'

locations_df = get_location_tweets(loc_list, start_time, end_time)

print(locations_df.shape)
locations_df.sample(frac=1).head(10)

Output:

Extracting tweets using Twitter API is straightforward. However, storing desired information from tweets in a data structure, e.g., a Pandas dataframe, is tricky. In this tutorial, I explained how you could scrape tweets using Twitter API and store them in a Pandas Dataframe in your desired format.