Text Classification

Graduate Quantitative Economics and Datascience

Jesse Perla
jesse.perla@ubc.ca

University of British Columbia

Overview

This Lecture and the Next

  • Here: lexicons, word counts, TF-IDF, and supervised logistic regression. No LLM calls or pretrained embeddings
  • Goal: turn text into economic variables, check held-out errors, and use the predictions in regressions
  • Next: the NLP and LLM lecture revisits the same Newswire task with embeddings, zero-shot prompts, and one-shot prompts
  • Optional here: measurement-error corrections and downloading datasets from the Hugging Face Hub

The Pipeline

  • The classification lecture coded job titles; this one builds variables from text: the sentiment of a sentence, the topic of a news article. The top row is what ML tutorials cover; the dashed row is the classification lecture’s measurement error applied to an outcome, run on Newswire in the optional section after the coverage regressions
  • Running example: Newswire, \(2.7\) million wire-service articles from \(1878\) to \(1977\) with topic labels (Silcock, Arora, D’Amico-Wong, and Dell 2024)

Extra Materials

Packages

import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split, cross_val_predict
from sklearn.metrics import confusion_matrix, classification_report
import statsmodels.formula.api as smf
  • The optional Hugging Face section also uses huggingface_hub to download datasets

Text as Features

From Documents to Vectors

  • Two ways to turn a document into \(x_n \in {\mathbb{R}}^M\)
    1. Bag of words: count each vocabulary word. Sparse, interpretable, ignores order
    2. TF-IDF (term frequency, inverse document frequency): the same counts, reweighted so that words common to every document count less
  • With term frequency \(\text{tf}_{n,j}\) and document frequency \(\text{df}_j\) over \(N\) documents,

\[ x_{n,j} = \text{tf}_{n,j} \cdot \text{idf}_j, \qquad \text{idf}_j = \log \frac{1 + N}{1 + \text{df}_j} + 1 \]

  • TfidfVectorizer normalizes rows to unit length; sublinear_tf=True, stop_words="english", and min_df=5 tame counts and vocabulary

Where Do Labels Come From?

  • Somebody reads a random sample of documents and labels each one. That is the expensive step; two coders on a subset measure how vague the definition is
  • Split the labelled sample into training and held-out parts. A representative held-out sample lets us estimate error rates for the corpus
  • Keep the held-out labels out of training and model selection; compare predictions with those labels only for the final evaluation

A Simple Sentiment Measure

Vibe-coding Warning!

This material has not yet been fully vetted and checked.

Financial PhraseBank

  • \(2{,}264\) sentences from financial news, labelled positive, negative, or neutral by annotators who all agreed (Malo et al. 2014)
def data_path(name):
    url = "https://jlperla.github.io/grad_econ_datascience/slides/data/"
    return f"data/{name}" if os.path.exists(f"data/{name}") else url + name

phrases = pd.read_csv(data_path("financial_phrasebank.csv.gz"))
print(phrases["label"].value_counts())
for _, row in phrases.sample(3, random_state=3).iterrows():
    print(f"[{row['label']}] {row['sentence'][:90]}")
label
neutral     1391
positive     570
negative     303
Name: count, dtype: int64
[neutral] LCS 's services cover the whole life cycle of software and information systems , from proc
[negative] The company 's profit before taxes fell to EUR 21.1 mn in the third quarter of 2008 , comp
[positive] EPS from continuing operations came in at 0.30 eur , up from 0.17 .

A Lexicon Score

  • The oldest approach: positive count minus negative count from hand-written word lists, no training data
positive = ["profit", "increase", "increased", "rose", "growth", "higher",
            "improved", "gain", "record", "strong", "up"]
negative = ["loss", "losses", "decrease", "decreased", "fell", "lower",
            "decline", "cut", "weak", "drop", "down"]
n_pos = CountVectorizer(vocabulary=positive).transform(phrases["sentence"]).sum(axis=1)
n_neg = CountVectorizer(vocabulary=negative).transform(phrases["sentence"]).sum(axis=1)
score = np.asarray(n_pos - n_neg).ravel()
phrases["lexicon"] = np.select([score > 0, score < 0], ["positive", "negative"],
                               default="neutral")
print(pd.crosstab(phrases["label"], phrases["lexicon"]))
lexicon   negative  neutral  positive
label                                
negative       118      163        22
neutral          5     1326        60
positive        29      225       316
  • Rows are the human label. The lexicon finds many positives but calls most negative sentences neutral: financial news says “decreased” in many ways

A Trained Sentiment Classifier

  • Same sentences, TF-IDF features, multinomial logistic regression, a held-out quarter
train, test = train_test_split(phrases, test_size=0.25, random_state=526)
vec_fpb = TfidfVectorizer(min_df=2, sublinear_tf=True)
sentiment = LogisticRegression(max_iter=1000).fit(
    vec_fpb.fit_transform(train["sentence"]), train["label"])
pred = sentiment.predict(vec_fpb.transform(test["sentence"]))
print(classification_report(test["label"], pred, digits=2))
              precision    recall  f1-score   support

    negative       0.85      0.52      0.65        79
     neutral       0.83      0.99      0.90       341
    positive       0.86      0.66      0.74       146

    accuracy                           0.84       566
   macro avg       0.85      0.72      0.76       566
weighted avg       0.84      0.84      0.83       566
  • \(1{,}700\) labelled sentences beat the hand-written lexicon on every class

The Learned Lexicon

  • The largest coefficients per class are the words that push hardest toward that class
coef = pd.DataFrame(sentiment.coef_.T, index=vec_fpb.get_feature_names_out(),
                    columns=sentiment.classes_)
print({k: list(coef[k].nlargest(6).index) for k in coef.columns})
{'negative': ['decreased', 'down', 'fell', 'loss', 'dropped', 'by'], 'neutral': ['is', 'be', 'and', 'will', 'not', 'the'], 'positive': ['rose', 'increased', 'up', 'increase', 'grew', 'from']}
  • The negative and positive lists are what a person would write, plus forms the hand list missed. Neutral is function words: what remains when nothing directional is said

Newswire: A Century of Wire Stories

Vibe-coding Warning!

This material has not yet been fully vetted and checked.

The Dataset

  • Local papers ran wire-service stories (AP, UP, INS). Silcock, Arora, D’Amico-Wong, and Dell (2024) found them on scanned pages, deduplicated the copies, and released \(2.7\) million articles, \(1878\) to \(1977\), with the newspapers that ran each one and their states
  • Seven topic flags (politics, civil rights, labor movement, antitrust, government regulation, protests, crime) from RoBERTa classifiers trained on hand-labelled articles: about \(2{,}400\) for politics, \(940\) for civil rights, \(250\) for labor movement, F1 between \(85\) and \(94\)
  • We treat the flags as labels. They are classifier output distilled from human labels, so our classifier learns to imitate a model. Hub page, DOI 10.57967/hf/2423, CC BY 4.0

Our Sample

  • The yearly files total \(35\) GB, so the course repository holds a stratified sample, \(400\) articles per year for \(1947\) to \(1977\) and \(40\) per year for \(1900\) to \(1946\), drawn by reading random byte windows of each file (the Hub section shows how)
  • Four caveats
    1. Records straddling a window edge are dropped, so very long articles are slightly under-sampled
    2. paper_states lists which newspapers carried a wire story: regional differences are differential carrying, not local production
    3. The topic flags are model output, not human coding
    4. Years have fixed counts, so pooled means are not corpus-representative. Report by year

Base Rates by Year

news = pd.read_csv(data_path("newswire_sample.csv.gz"))
topics = ["politics", "civil_rights", "labor_movement", "crime"]
by_year = news.groupby("year")[topics].mean()
print(by_year.loc[[1920, 1940, 1950, 1955, 1960, 1965, 1970, 1977]].round(3))
      politics  civil_rights  labor_movement  crime
year                                               
1920     0.350         0.000           0.350  0.125
1940     0.450         0.000           0.050  0.075
1950     0.318         0.002           0.098  0.108
1955     0.335         0.008           0.085  0.132
1960     0.378         0.055           0.058  0.122
1965     0.288         0.058           0.070  0.095
1970     0.335         0.042           0.052  0.120
1977     0.435         0.015           0.075  0.125
  • Politics is a third or more of the wire in every year. Civil rights is under one percent in the early 1950s and about six percent by the mid 1960s
  • Labor movement sits between five and ten percent after 1947 with no trend, the cleaner target to learn. The 1920 figure is \(40\) articles in a strike wave

Coverage over the Century

smooth = by_year[["politics", "labor_movement", "civil_rights"]].rolling(
    5, center=True).mean()
smooth.plot(subplots=True, layout=(1, 3), figsize=(12, 3.0), legend=False,
            title=["politics", "labor movement", "civil rights"])
plt.tight_layout()
plt.show()

  • Five-year centered moving averages. The pre-1947 years have \(40\) articles each, so the early series are noisy

Training a Topic Classifier

Reminder: What Is TF-IDF?

TF-IDF = term frequency × inverse document frequency

  • TF: how often word \(j\) appears in article \(n\)
  • DF: how many of the \(N\) training articles contain word \(j\), counted once per article
  • IDF: give less weight to words that appear across many articles

\[ w_{n,j} = \text{tf}_{n,j} \cdot \text{idf}_j, \qquad \text{idf}_j = 1 + \log \frac{1 + N}{1 + \text{df}_j} \]

  • At equal TF, strike gets more weight than news if it occurs in fewer articles
  • Our sublinear_tf=True uses \(1 + \log(\text{count})\) for present words and zero for absent words. The vectorizer then normalizes each article’s vector to unit length

Features and Split

  • TF-IDF on the article text, a quarter of the articles held out. The vocabulary and the document frequencies come from the training part only, so the held-out quarter never touches the fit
train_idx, test_idx = train_test_split(np.arange(len(news)), test_size=0.25,
                                       random_state=526)
vec = TfidfVectorizer(min_df=5, stop_words="english", sublinear_tf=True)
vec.fit(news["text"].iloc[train_idx])
X = vec.transform(news["text"])
X_tr, X_te = X[train_idx], X[test_idx]
print(f"{X.shape[0]} articles, {X.shape[1]} words, "
      f"{len(train_idx)} train, {len(test_idx)} test")
14280 articles, 15340 words, 10710 train, 3570 test

Labor Movement: the Default Fit

y_tr = news["labor_movement"].iloc[train_idx]
y_te = news["labor_movement"].iloc[test_idx]
labor = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print(confusion_matrix(y_te, labor.predict(X_te)))
print(classification_report(y_te, labor.predict(X_te), digits=2))
[[3296    0]
 [ 157  117]]
              precision    recall  f1-score   support

           0       0.95      1.00      0.98      3296
           1       1.00      0.43      0.60       274

    accuracy                           0.96      3570
   macro avg       0.98      0.71      0.79      3570
weighted avg       0.96      0.96      0.95      3570
  • Perfect precision, poor recall: with \(8\%\) positives the penalized fit is conservative and misses most labor stories

Rebalancing the Classes

  • With \(8\%\) positives the ordinary likelihood is dominated by the negatives, so the fit only calls a story “labor” when it is very sure: high precision, but most labor stories missed (a high false negative rate)
  • class_weight="balanced" multiplies each observation’s term in the log-likelihood by the inverse of its class share, so the rare class carries as much total weight as the common one. The fit then accepts more false alarms in order to miss fewer positives
labor = LogisticRegression(max_iter=1000, class_weight="balanced").fit(X_tr, y_tr)
print(confusion_matrix(y_te, labor.predict(X_te)))
print(classification_report(y_te, labor.predict(X_te), digits=2))
[[3226   70]
 [  64  210]]
              precision    recall  f1-score   support

           0       0.98      0.98      0.98      3296
           1       0.75      0.77      0.76       274

    accuracy                           0.96      3570
   macro avg       0.87      0.87      0.87      3570
weighted avg       0.96      0.96      0.96      3570
  • Precision falls, recall nearly doubles: the cost argument from the classification lecture

What the Classifier Learned

weights = pd.Series(labor.coef_[0], index=vec.get_feature_names_out())
print(list(weights.nlargest(15).index))
print(list(weights.nsmallest(8).index))
['strike', 'union', 'labor', 'workers', 'association', 'cio', 'afl', 'contract', 'negotiations', 'wage', 'unions', 'agreement', 'coal', 'protest', 'premier']
['ap', 'eisenhower', 'soviet', 'military', 'nixon', 'states', 'american', 'old']
  • The largest positive weights are the vocabulary of a strike story; the most negative are the rest of the wire: presidents, the Cold War, and the AP byline

Civil Rights

  • Rarer (\(2.5\%\) overall, \(3\) to \(6\%\) in the years that matter) and the outcome of the first regression, so balanced weights from the start
y_tr = news["civil_rights"].iloc[train_idx]
y_te = news["civil_rights"].iloc[test_idx]
civil = LogisticRegression(max_iter=1000, class_weight="balanced").fit(X_tr, y_tr)
print(confusion_matrix(y_te, civil.predict(X_te)))
print(classification_report(y_te, civil.predict(X_te), digits=2))
[[3459   26]
 [   7   78]]
              precision    recall  f1-score   support

           0       1.00      0.99      1.00      3485
           1       0.75      0.92      0.83        85

    accuracy                           0.99      3570
   macro avg       0.87      0.96      0.91      3570
weighted avg       0.99      0.99      0.99      3570

Why Cross-Fit?

  • We need a prediction for every article for the coverage regression. Predicting the training articles with their own fitted model can make its errors look too small

Five-fold cross-fitting: each of five models trains on four article groups and predicts only the remaining group. The held-out group rotates across the five fits, giving one held-out prediction per article.

  • A fold is one group of articles. With \(K=5\), each fit uses \(4/5\) to train and \(1/5\) to predict; every article is held out exactly once
  • Cross-validation often compares models by their scores. Cross-fitting keeps the predictions for later analysis; it does not eliminate classification error

Cross-Fitting in Python

  • A pipeline refits TF-IDF and logistic regression inside each training fold. Pass the raw text, so held-out articles influence neither the vocabulary nor the weights
from sklearn.model_selection import StratifiedKFold
from sklearn.pipeline import make_pipeline

folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=526)
model = make_pipeline(
    TfidfVectorizer(min_df=5, stop_words="english", sublinear_tf=True),
    LogisticRegression(max_iter=1000, class_weight="balanced"),
)
news["civil_rights_hat"] = cross_val_predict(
    model, news["text"], news["civil_rights"], cv=folds, n_jobs=-1,
)
print(confusion_matrix(news["civil_rights"], news["civil_rights_hat"]))
[[13849    81]
 [   32   318]]
  • StratifiedKFold keeps similar class shares in each fold. cross_val_predict returns the held-out predictions in the original article order
  • The confusion matrix pools those predictions. The next regression uses civil_rights_hat as its outcome; no article’s own label trained its prediction

Coverage Regressions

Civil Rights Coverage in the South

  • Did papers in the former Confederacy carry more or less civil rights news than the rest of the country between \(1954\) (Brown) and \(1968\)? The unit is an article \(\times\) newspaper pair, one explode of paper_states
south = ["alabama", "arkansas", "florida", "georgia", "louisiana",
         "mississippi", "north carolina", "south carolina", "tennessee",
         "texas", "virginia"]
era = news[news["year"].between(1954, 1968)]
panel = era.assign(state=era["paper_states"].str.split(";")).explode("state")
panel["south"] = panel["state"].isin(south).astype(int)
print(f"{len(era)} articles, {len(panel)} article-newspaper pairs, "
      f"{panel['state'].nunique()} states, south share {panel['south'].mean():.2f}")
6000 articles, 85448 article-newspaper pairs, 47 states, south share 0.20

The Regression, Twice

  • Outcome: the civil rights flag, then our cross-fitted prediction of it. Regressors: the South indicator and year effects, standard errors clustered by state
def south_effect(outcome):
    fit = smf.ols(f"{outcome} ~ south + C(year)", data=panel).fit(
        cov_type="cluster", cov_kwds={"groups": panel["state"]})
    return fit.params["south"], fit.bse["south"]

results = pd.DataFrame([south_effect("civil_rights"),
                        south_effect("civil_rights_hat")],
                       index=["Newswire label", "our classifier"],
                       columns=["south", "se"])
print(results.round(4))
                 south      se
Newswire label  0.0175  0.0038
our classifier  0.0184  0.0043
  • Southern papers carried civil rights stories about \(1.8\) percentage points more often, on a base of about \(4\%\). The generated outcome gives nearly the same answer, closer than the attenuation of the classification lecture predicts; the optional section that follows says why

Labor Coverage and Strikes

strikes = pd.read_csv(data_path("work_stoppages.csv"))
yearly = news[news["year"] >= 1947].groupby("year")[
    "labor_movement"].mean().to_frame()
yearly = yearly.join(strikes.set_index("year"))
fit = smf.ols("labor_movement ~ workers_thousands",
              data=yearly).fit()
ax = yearly.plot.scatter(x="workers_thousands",
                         y="labor_movement",
                         figsize=(5, 4))
ax.set_title(f"t = {fit.tvalues.iloc[1]:.1f}, "
             f"R2 = {fit.rsquared:.2f}")
plt.show()

  • Yearly share of labor stories against workers in major BLS work stoppages, \(1947\) to \(1977\). Weak: wire coverage follows particular disputes (steel in 1952 and 1959, coal in 1949), not the volume of strikes
  • Each share comes from \(400\) articles, so it carries about a point of sampling noise, which lowers \(R^2\) without biasing the slope

Correcting the Coverage RegressionOptional

Applying the Correction

  • With error rates that do not depend on \(x_n\), every coefficient on a generated outcome shrinks by exactly the factor \(1 - p_0 - p_1\) and the intercept also picks up \(p_0\) (classification lecture), so divide by the factor with the rates from the cross-fitted confusion matrix
cm = confusion_matrix(era["civil_rights"], era["civil_rights_hat"])
p0, p1 = cm[0, 1] / cm[0].sum(), cm[1, 0] / cm[1].sum()
factor = 1 - p0 - p1
raw, label = results.loc["our classifier", "south"], results.loc["Newswire label", "south"]
print(f"p0 = {p0:.4f}, p1 = {p1:.3f}, 1 - p0 - p1 = {factor:.3f}")
print(f"label {label:.4f}, expected if attenuated {factor * label:.4f}, "
      f"raw {raw:.4f}, corrected {raw / factor:.4f}")
p0 = 0.0069, p1 = 0.066, 1 - p0 - p1 = 0.927
label 0.0175, expected if attenuated 0.0163, raw 0.0184, corrected 0.0199
  • The prediction-based coefficient should have come out attenuated and the division should have recovered the label’s coefficient. Instead the raw coefficient was already there and the correction overshoots

Diagnosing Differential Error

  • Check the assumption directly: error rates by the regressor
def rates(group):
    cm = confusion_matrix(group["civil_rights"], group["civil_rights_hat"])
    return pd.Series({"p0": cm[0, 1] / cm[0].sum(), "p1": cm[1, 0] / cm[1].sum(),
                      "base rate": group["civil_rights"].mean()})

print(panel.groupby("south").apply(rates, include_groups=False).round(4))
           p0      p1  base rate
south                           
0      0.0094  0.0828     0.0392
1      0.0111  0.0654     0.0568
  • Slightly more false positives and fewer false negatives for stories carried in the South, where the base rate is higher: words like “southern” and “school” are features
  • Small differences, but so is the coefficient, and they offset the whole attenuation. When the outcome is rare and the effect is small, check the confusion matrix by subgroup before trusting a correction

The Hugging Face HubOptional

What the Hub Holds

  • huggingface.co hosts datasets, models, and spaces (demo apps) as git repositories with large-file storage, each with a card describing contents, licence, and citation
  • The datasets library loads many of them with one call; load_dataset(name, streaming=True) iterates parquet or JSON-lines data without downloading it. Newswire is one pretty-printed JSON array per year, which streaming cannot handle
  • Cite the card’s citation and DOI and read the licence: Newswire is CC BY 4.0, Financial PhraseBank CC BY-NC-SA 3.0, which forbids commercial use

Downloading a File

import zipfile
from huggingface_hub import hf_hub_download
archive = hf_hub_download("takala/financial_phrasebank",
                          "data/FinancialPhraseBank-v1.0.zip", repo_type="dataset")
with zipfile.ZipFile(archive) as z:
    fpb = pd.read_csv(z.open("FinancialPhraseBank-v1.0/Sentences_AllAgree.txt"),
                      sep="@", header=None, names=["sentence", "label"],
                      encoding="latin-1")
print(fpb.shape, fpb["label"].value_counts().to_dict())
(2264, 2) {'neutral': 1391, 'positive': 570, 'negative': 303}
  • hf_hub_download caches the file locally and returns its path. This is how the course’s financial_phrasebank.csv.gz was built

Sampling a 35 GB Dataset Without Downloading It

  • The yearly files are JSON arrays with one record per block. The CDN serves byte ranges, and the separator between records cannot occur inside a JSON string, so a random window resynchronizes on it
import json
import requests
BASE = ("https://huggingface.co/datasets/dell-research-harvard/newswire"
        "/resolve/main/{y}_data_clean.json")
SEP = "\n    },\n    {\n"

def window_records(y, offset, size=1_000_000):
    r = requests.get(BASE.format(y=y), timeout=180,
                     headers={"Range": f"bytes={offset}-{offset + size - 1}"})
    r.raise_for_status()
    fragments = r.text.split(SEP)[1:-1]
    return [json.loads("{\n" + frag + "\n    }") for frag in fragments]
  • The first and last fragments are partial records and are dropped. utilities/build_newswire_sample.py in the course repository loops over this function

One Window of 1960

records = window_records(1960, 300_000_000)
print(f"{len(records)} complete records in one window")
for a in records[:3]:
    flags = [a[k] for k in ["politics", "civil_rights", "labor_movement"]]
    print(a["dates"][0], flags, a["cleaned_article"][:60].replace("\n", " "))
85 complete records in one window
Dec-19-1960 [0, 0, 0] LONDON (UPD) — Emperor Haile Selassie, once again firmly ent
Oct-08-1960 [0, 0, 0] LULING, Tex. (AP)—A housewife who gained nationwide attentio
Oct-07-1960 [0, 0, 0] MUSKEGON, Mich. (AP)—Dale Young and Tom Skeels, both 15, wal
  • Under a second and no token. The files are not sorted by date, so a window at a random offset is close to a random draw from the year; the course sample took windows at random offsets until each year had its quota, about \(250\) requests in a few minutes, drawn once and committed
  • The same trick works for any large file on a server that honours Range requests, including the parquet shards on the Hub, where pyarrow can read a single row group from a byte range

References