Natural Language Processing and Large Language Models

Graduate Quantitative Economics and Datascience

Jesse Perla

jesse.perla@ubc.ca

University of British Columbia

Overview

From Word Counts to Language Models

Previously: text classification with word counts, TF-IDF, and supervised classifiers
Today: revisit the same news articles with pretrained embeddings and LLM prompts. Start with the application, then explain tokens, embeddings, and next-token prediction
Here: hosted OpenAI models through a Python API
Next: open-weight LLMs on your own machine, reusing the chat and embedding workflow. No local model setup is needed for this lecture

References

These notes are a bare-bones introduction. See references for more
See Melissa Dell’s Survey Deep Learning for Economists and Course
- Topic and Sentiment Classification
- Semantic and Syntactic Similarity
The interactive Transformer Explainer walks through how GPT-2 turns tokens into the next-token distribution

API Options for LLMs

There are many ways to access these tools, including vendor-specific APIs and those which wrap things in as much of a cross-vendor way as possible
A big variation is between LLMs that are hosted and ones that run locally.
Local use of LLM/NLP is often facilitated by the package Hugging Face Transformers
- We will not use this approach here because it requires far more background, but if you are serious this is useful to know about
Remote use here goes through OpenAI’s API, whose request shape Ollama, Google, and most hosts also serve (see the local LLM lecture)

Packages

The openai package calls OpenAI’s hosted models

import os
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.manifold import TSNE
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import numpy as np
import tiktoken
from openai import OpenAI

OpenAI API Keys

Sign up for the OpenAI Platform
- Sometimes it is free for a limited number of tokens. See pricing for more
Go to the API keys tab and create a key
In your terminal, set OPENAI_API_KEY to this value (see here)

Creating the Client

One client, created once; it reads OPENAI_API_KEY from the environment
The model= is chosen per request for speed, cost, etc.
temperature= sets how random the output is (0 is nearly deterministic, default 1). Will discuss more later.

client = OpenAI(max_retries=10)
chat_model = "gpt-4o-mini"
embedding_model = "text-embedding-3-large"

Messages, Roles, and No State

A request is a list of messages, each with a role: system sets the behaviour, user is the prompt, assistant is what the model said
The reply is choices[0].message, itself an assistant message
The model keeps no state or “memory” between requests: every request carries the whole history, and the answer conditions on all of it

Does an LLM Do as Well?

Vibe-coding Warning!

This material has not yet been fully vetted and checked.

Same Articles, Same Economic Question

The text classification lecture measured civil rights coverage across regions. We now compare ways to assign the article-level flag
Use the same \(14{,}280\)-article extract of Newswire. Its topic flags come from RoBERTa classifiers trained on human labels; our scores measure agreement with those flags

def data_path(name):
    url = "https://jlperla.github.io/grad_econ_datascience/slides/data/"
    return f"data/{name}" if os.path.exists(f"data/{name}") else url + name

news = pd.read_csv(data_path("newswire_sample.csv.gz"))

Four Ways to Classify an Article

Method	What we supply	Task-specific labels
TF-IDF + logistic	Word counts, reweighted by document frequency	\(10{,}710\)
Embeddings + logistic	A pretrained model’s vector for each article	\(400\)
Zero-shot prompt	A question and the article	\(0\)
One-shot prompt	The question, one labelled example, and the article	\(1\) in the prompt

Zero-shot / one-shot counts examples in the prompt. The language model is already pretrained; we do not update its weights
Each method predicts the same held-out articles. The one-shot exemplar is selected using the trained TF-IDF model, so its selection also uses those training labels

Rebuild the TF-IDF Baseline

Reproduce the previous lecture’s \(75\%\) training / \(25\%\) test split. Learn both the vocabulary and the classifier from training articles only

train_idx, test_idx = train_test_split(np.arange(len(news)), test_size=0.25,
                                     random_state=526)
pool = news.iloc[train_idx]
vec = TfidfVectorizer(min_df=5, stop_words="english", sublinear_tf=True)
X_train = vec.fit_transform(pool["text"])
civil = LogisticRegression(max_iter=1000, class_weight="balanced").fit(
    X_train, pool["civil_rights"])
print(f"{len(pool)} training articles, {len(test_idx)} held out")

10710 training articles, 3570 held out

The \(400\) embedding-training articles and the prompt exemplar also come from this training pool. Test labels are reserved for scoring

Setting Up the Comparison

Keep every held-out civil rights article from \(1954\) to \(1968\) and an equal number of other articles: \(108\) articles, \(54\) per class

held = news.iloc[test_idx]
held = held[held["year"].between(1954, 1968)]
held_pos = held[held["civil_rights"] == 1]
compare = pd.concat([held_pos, held[held["civil_rights"] == 0].sample(
    len(held_pos), random_state=526)])
print(f"{len(compare)} articles, {len(held_pos)} about civil rights")

108 articles, 54 about civil rights

This balanced comparison makes misses and false alarms visible. Its accuracy and precision describe this \(50/50\) sample, not the corpus’s much lower civil rights share
TF-IDF reads the full text; the API examples use the first \(2{,}000\) characters to limit cost. This compares practical workflows, not equal inputs or label budgets

Embeddings as Features

An embedding is a learned vector representing the article. We use those vectors as \(x_n\) in the same logistic regression; the embedding model stays fixed

subset = pd.concat([pool[pool["civil_rights"] == 1].sample(100, random_state=526),
                    pool[pool["civil_rights"] == 0].sample(300, random_state=526)])

def embed(texts):
    resp = client.embeddings.create(model=embedding_model, input=texts)
    return np.array([d.embedding for d in resp.data])

E_train = embed(subset["text"].str[:2000].tolist())
E_compare = embed(compare["text"].str[:2000].tolist())
embed_clf = LogisticRegression(max_iter=1000, class_weight="balanced").fit(
    E_train, subset["civil_rights"])
compare["embedding"] = embed_clf.predict(E_compare)
compare["tfidf"] = civil.predict(vec.transform(compare["text"]))

\(508\) embeddings in two calls: \(400\) training articles and \(108\) held-out articles

Zero-Shot: Asking the Model Directly

A question, the article, and a yes or no; one request per article. seed makes replies reproducible on a best-effort basis

topic_question = (
    "Is this newspaper article about the civil rights movement: racial "
    "segregation, desegregation, voting rights, or racial discrimination? "
    "Answer yes or no.\n\n")

def classify(texts, prefix=""):
    labels = []
    for t in texts:
        prompt = prefix + topic_question + t[:2000]
        resp = client.chat.completions.create(
            model=chat_model, temperature=0, seed=526,
            messages=[{"role": "user", "content": prompt}])
        answer = resp.choices[0].message.content
        labels.append(int(answer.strip().lower().startswith("yes")))
    return labels

compare["zero_shot"] = classify(compare["text"])
print(confusion_matrix(compare["civil_rights"], compare["zero_shot"]))

[[53  1]
 [ 5 49]]

Most errors are misses: articles Newswire’s labels count as civil rights that the model, reading the question cold, does not

One Labelled Article as an Exemplar

Pick one short civil rights article from the training pool, the one our classifier is surest about among those of a few hundred characters

positives = pool[(pool["civil_rights"] == 1)
                 & pool["text"].str.len().between(400, 900)]
scores = civil.predict_proba(vec.transform(positives["text"]))[:, 1]
exemplar = positives.iloc[scores.argmax()]["text"]
print(exemplar)

- WASHINGTON UP—The National Association for the Advancement of Colored People will call on both parties to adopt platforms pledging the full power of the government to end racial segregation in the South, it was learned today.

The Negro organization will ask both the Democratic and Republican national conventions to endorse strongly the Supreme Court's decisions outlawing segregation in the public schools.

The demand for an all-out integration plank, which will be supported by some northern “liberals”

(Continued on page 2, column 2)

One-Shot

one_shot = ("Example of an article about the civil rights movement:\n"
            f"{exemplar}\n\nNow the article to classify.\n")
compare["one_shot"] = classify(compare["text"], one_shot)
print(confusion_matrix(compare["civil_rights"], compare["one_shot"]))

[[53  1]
 [ 4 50]]

The exemplar shows what this dataset’s label covers: a story about the movement even when the words “civil rights” never appear
This is in-context learning: change the prompt, keep the model’s weights fixed. Fine-tuning would update weights using training examples

The Comparison

from sklearn.metrics import accuracy_score, precision_score, recall_score
rows = {}
for method in ["tfidf", "embedding", "zero_shot", "one_shot"]:
    rows[method] = [f(compare["civil_rights"], compare[method])
                    for f in (accuracy_score, precision_score, recall_score)]
print(pd.DataFrame(rows, index=["accuracy", "precision", "recall"]).T.round(2))

           accuracy  precision  recall
tfidf          0.96       1.00    0.93
embedding      0.94       0.93    0.96
zero_shot      0.94       0.98    0.91
one_shot       0.95       0.98    0.93

“Correct” means “agrees with Newswire’s RoBERTa model”. Differences of one or two articles are noise at this sample size, and the LLM rows move by that much between runs even at temperature zero

What the Example Teaches

A pretrained model brings language knowledge. Task-specific labels help align its answers with the labeller’s definition of civil rights coverage
One example in a prompt is useful supervision, but our exemplar was selected by a classifier trained on \(10{,}710\) labels. This is not a controlled comparison of annotation budgets
Better agreement is still not ground truth. Before using predictions in a regression, audit errors on human-labelled data and by relevant subgroups
Next we unpack the tokens and representations behind these API calls. The local LLM lecture then shows how to run chat and embedding models on your own computer

Tokens, Vocabulary, and Corpus

Tokens and Vocabulary

We can define a vocabulary of \(K\) possible “tokens” enumerated \(k = 1, \ldots, K\)
- For NLP, this might be common words, parts of words, and characters.
- The vocabulary is model and sometimes application specific
A splitter will take an input and break it into pieces
- e.g., take a sentence and split on the whitespace
A tokenizer will take an input and map it into a sequence of token indices
- e.g., take a sentence, split on the spaces, and then map to tokens in the vocabulary
The corpus is the set of text/etc. you use
- In specialized cases you might want to build a custom vocabulary based on how common certain words are in your corpus

Example with OpenAI Tokens

When using LLMs this sometimes happens in the background, but other times you may need more manual control
We can use a package to reverse-engineer what OpenAI does under the hood.
- Splits on whitespace, tokens may be whole words, subwords, or characters

enc = tiktoken.encoding_for_model(embedding_model)
print(f"Tokens in vocabulary for {embedding_model}: {enc.n_vocab}")
print(f"'Hello world' -> {enc.encode('Hello world')}")
print(f"'hello world' -> {enc.encode('hello world')}")
print(f"'hello world 67' -> {enc.encode('hello world 67')}")

Tokens in vocabulary for text-embedding-3-large: 100277
'Hello world' -> [9906, 1917]
'hello world' -> [15339, 1917]
'hello world 67' -> [15339, 1917, 220, 3080]

Tasks with Tokens

Most tasks in NLP involve working with these tokenized inputs. For example,
- Simple counts of tokens (e.g., bag of words) for comparison
- Using a sequence of tokens as input to a model to predict the next token (or missing tokens)
- Mapping a sequence of tokens into an embedding of the entire sequence for similarity/classification/etc.
Not specific to text though. Can use with any sequence of discrete or discretized outcomes

Embedding a Token

The ordering of the tokens from \(k = 1, \ldots, K\) is arbitrary
However, we can assign a real-valued vector to each token
- This is called an embedding of the token and could capture things like conceptual similarity between them, etc.
- The real-valued vectors can then be clustered, classified, etc.

Embeddings

What is an Embedding?

messages = [
    {"role": "system", "content": "Two short, technical bullet points."},
    {"role": "user", "content": "What is an embedding?"},
]
resp = client.chat.completions.create(model=chat_model, messages=messages)
answer = resp.choices[0].message.content
print(answer)

- An embedding is a mathematical representation of objects, such as words or sentences, in a continuous vector space, allowing for semantic relationships to be captured based on the proximity of their vectors.
- Commonly used in natural language processing, embeddings facilitate efficient computation and enhanced performance in tasks like classification and clustering by transforming discrete items into dense, lower-dimensional numerical formats.

Followup: What does it have to do with Latent Spaces?

messages += [
    {"role": "assistant", "content": answer},
    {"role": "user", "content": "What is the relationship to latent spaces?"},
]
resp = client.chat.completions.create(model=chat_model, messages=messages)
print(resp.choices[0].message.content)

- Embeddings are often a way to project data into a latent space, which is an abstract, lower-dimensional space that captures the underlying structures and relationships in the data, facilitating pattern recognition and generalization.
- The latent space representation allows for the preservation of similarities and semantic meanings, enabling models to learn about complex datasets by visualizing and manipulating high-dimensional data as points in a continuous space.

Embeddings Overview

Used loosely and inconsistently, but the spirit is the same
A mapping of some \(x \in {\mathcal{X}}\) to a latent space \(\phi(x) \in {\mathcal{Z}}\)
Typically we want that \({\mathcal{Z}}\) is a continuous vector space of finite dimension
- Crucially, we can think of distances in that space (e.g., \(||\phi(x) - \phi(y)||\))
The dimension of \({\mathcal{Z}}\) may be smaller or larger than \({\mathcal{X}}\)
- If larger, typically in only small regions of the higher-dimensional space.
Often want to preserve norms in cases where the original space itself had a norm
- e.g., if \(||x - y||\) is small, then \(||\phi(x) - \phi(y)||\) is small

Cosine Similarity

Cosine similarity is the “angle” between two vectors in the embedding space \(z = \phi(x)\) and \(z' = \phi(x')\)

\[ \mathrm{sim}(z, z') = \frac{z \cdot z'}{||z|| ||z'||} \]

Interpretation of Cosine Similarity

Interpretation: If \(z\) and \(z'\) are
1. close in the embedding space, then the cosine similarity is close to 1
2. orthogonal, then the cosine similarity is 0
3. opposite, then the cosine similarity is -1
Norm comparison \(||z - z'||\) instead? Works but not invariant to scaling

Implementation of Cosine Similarity

def cos_sim(a, b):
  denom = np.linalg.norm(a) * np.linalg.norm(b)
  return np.dot(a, b) / denom
print(f"cos_sim([1, 0], [0, 1]) = {cos_sim(np.array([1, 0]), np.array([0, 1]))}")
print(f"cos_sim([1, 0], [0, 2]) = {cos_sim(np.array([1, 0]), np.array([0, 2]))}")
print(f"cos_sim([1, 0], [1, 0]) = {cos_sim(np.array([1, 0]), np.array([1, 0]))}")
print(f"cos_sim([1, 0], [2, 0]) = {cos_sim(np.array([1, 0]), np.array([2, 0]))}")
print(f"cos_sim([1, 0], [-2, 0]) = {cos_sim(np.array([1, 0]), np.array([-2, 0]))}")

cos_sim([1, 0], [0, 1]) = 0.0
cos_sim([1, 0], [0, 2]) = 0.0
cos_sim([1, 0], [1, 0]) = 1.0
cos_sim([1, 0], [2, 0]) = 1.0
cos_sim([1, 0], [-2, 0]) = -1.0

OpenAI Embeddings

The same embed function used for Newswire also works on single words: one vector per input, in order

words = ["bank", "banks", "river", "money"]
E = embed(words)
print(E.shape)
for i, j in [(0, 1), (0, 2), (0, 3), (2, 3)]:
    print(f"sim({words[i]}, {words[j]}) = {cos_sim(E[i], E[j]):.4f}")

(4, 3072)
sim(bank, banks) = 0.7883
sim(bank, river) = 0.4143
sim(bank, money) = 0.4347
sim(river, money) = 0.3746

Where Did the Embedding Come From?

OpenAI’s text embeddings (e.g., text-embedding-3-large) are trained with self-supervised objectives to produce general-purpose semantic vectors.
Conceptually, a representation map \(\phi(x)\) is learned so that cosine similarity reflects semantic relatedness.
Always check provider docs for the training objective and intended use; not all embeddings are suitable for every task.

Bigger Embeddings

Bag of Words

Words usually occur in sentences, paragraphs, etc.
Instead of embedding a single word, we can embed a block of text.
Simple starting point: compare the relative frequency of words
1. Turn the text into blocks of tokens with unique identifiers
2. Filter out tokens that are not useful (e.g., “the”, “a”, etc.)
3. Count the frequency of each token
4. The embedding is the vector of the frequency of each token
Many issues with this, but crucially it does not capture any sense of context dependent meaning of words, and is invariant to word order

Sentence Embeddings with LLMs

Encode an entire sentence into a single embedding and compare for similarity

e_1, e_2, e_3 = embed(["The man bites the dog",
                       "The dog chased the man",
                       "The man was chased by the dog"])
print(f"sim(e_1, e_2) = {cos_sim(e_1, e_2):.4f}")
print(f"sim(e_1, e_3) = {cos_sim(e_1, e_3):.4f}")
print(f"sim(e_2, e_3) = {cos_sim(e_2, e_3):.4f}")

sim(e_1, e_2) = 0.5415
sim(e_1, e_3) = 0.5226
sim(e_2, e_3) = 0.7913

Note that the 2nd and 3rd are the most similar, and the 1st and 3rd the least.

Be Cautious, Interpretation is Tricky

e_4 = embed(["The man chased the dog"])[0]
print(f"sim(e_2, e_3) = {cos_sim(e_2, e_3):.4f}")
print(f"sim(e_3, e_4) = {cos_sim(e_3, e_4):.4f}")

sim(e_2, e_3) = 0.7913
sim(e_3, e_4) = 0.8598

Note that sim(e_3, e_4) > sim(e_3, e_2) even though they seem to have the exact opposite meaning!
Why? There are many embeddings (e.g. bag of words) which make these close, and others where they are far apart. Hard to know without a sense of how it was trained.

Clustering/Visualization

Since embeddings provide a measure of similarity/distance, we can cluster it into groups or use other tools to find interpretations
There are various algorithms to cluster based on representations, and find the closest elements within the set of data for a new element
One approach for visualization: come up with a lower-dimensional embedding which approximately preserves local neighborhoods in 2D/3D as in t-SNE (t-Distributed Stochastic Neighbor Embedding)

Get Some Random Embeddings

sample_words = [
  "bank", "river", "money", "finance", "water", "dog", "cat", "animal", "pet", "tree",
  "forest", "city", "village", "road", "car", "bus", "train", "doctor", "nurse", "hospital",
  "school", "student", "teacher", "book", "library", "music", "guitar", "piano", "art", "painting",
  "computer", "algorithm", "data", "model", "economics", "market", "price", "inflation", "policy",
  "riverbank", "beach", "mountain", "valley", "ocean", "lake", "software", "hardware", "network", "cloud"
]
embeddings = embed(sample_words)
print(embeddings.shape)

(49, 3072)

Visualize with t-SNE

tsne = TSNE(n_components=2, random_state=0)
embeddings_2d = tsne.fit_transform(embeddings)
plt.figure()
for i, token in enumerate(sample_words):
    x, y = embeddings_2d[i]
    plt.scatter(x, y)
    plt.text(x+0.1, y+0.1, token, fontsize=9)
plt.title('t-SNE visualization of OpenAI text embeddings')
plt.xlabel('Dimension 1')
plt.ylabel('Dimension 2')
plt.show()

Visualize with t-SNE

Sequential Data and Token Prediction

Distribution of Tokens

In many cases, data has an inherent sequential ordering. e.g. time series, language, etc.
Consider conditional probabilities over \(x_t \in \{1, \ldots K\}\). Generically call these “tokens”
Objects of interest: for \(x_1, x_2, \ldots, x_T\) for a sequence of length \(T\) sampled from some population distribution
We may want to condition on the past to predict the next token, condition on the entire sequence to predict missing ones, etc.

Multinomial Logit/Softmax

Often want to map a latent space to a probability distribution over \(K\) outcomes, as the multinomial logit of the classification lecture does
Given a set of \(K\) outcomes, define the logit/softmax mapping a vector \(y \in {\mathbb{R}}^K\) to a probability distribution over \(K\) outcomes as

\[ \mathrm{softmax}(y)_k = \frac{e^{y_k}}{\sum_{j=1}^K e^{y_j}}, \quad k = 1, \ldots, K \]

Typically this is combined with a mapping of a latent space, \(z \in {\mathbb{R}}^L\), with some matrix \(W \in {\mathbb{R}}^{K \times L}\) so that \(\mathrm{softmax}(W z)\) is a probability distribution over \(K\) outcomes

\[ \mathbb{P}\left(x = k \,|\, z\right) = \mathrm{softmax}(W z)_k \]

Token Prediction from Conditional Probabilities

Given a sequence \(x_1, x_2, \ldots, x_{t-1}\), model the conditional distribution:

\[ \mathbb{P}(x_t | x_{t-1}, x_{t-2}, \ldots, x_1) \]
- Where in the background these are done with marginal and conditional probabilities sampling the population distribution
Basic strategy for sequence models:
1. Do an embedding for each token \(\phi_1(x_t)\)
2. Map those embeddings to some combined representation, \(\phi_2(\phi_1(x_1), \ldots, \phi_1(x_{t-1}))\)
3. Map that with \(\mathrm{softmax}\) to get probabilities over \(k = 1, \ldots, K\) for \(x_t\)
4. Fit with methods such as maximum likelihood given the multinomial logit structure

From Tokens to the Next-Token Distribution

Embed each token, combine the embeddings into one latent \(z\), and map \(z\) through the softmax to a distribution over the \(K\) possible next tokens

Conditioning on Prior Tokens

The prompt is the conditioning set: the same question after a different history is a draw from a different \(\mathbb{P}(x_t \,|\, x_{t-1}, \ldots, x_1)\)
See Melissa Dell’s Prompting lecture and Prompts for Economists

question = {"role": "user",
            "content": "Which meaning of the word bank fits our conversation?"}
river = [{"role": "system", "content": "Answer in one sentence."},
         {"role": "user", "content": "We talked about fishing on the river."},
         {"role": "assistant", "content": "Rivers are great places to fish."},
         question]
money = [{"role": "system", "content": "Answer in one sentence."},
         {"role": "user", "content": "We talked about interest rates."},
         {"role": "assistant", "content": "Interest rates matter for savings."},
         question]
for label, history in [("river", river), ("money", money)]:
    resp = client.chat.completions.create(model=chat_model, messages=history)
    print(f"after {label} talk: {resp.choices[0].message.content}")

Conditioning on Prior Tokens

after river talk: In the context of fishing on the river, "bank" refers to the land along the edge of the river.
after money talk: In our conversation, "bank" likely refers to a financial institution that handles deposits, loans, and interest rates.

Mental Model of Sequential Generation

Start with a set of tokens, \(x_1, x_2, \ldots, x_{t-1}\) (include EVERYTHING)
Condition on those tokens to find the \(\mathbb{P}(x_t | x_{t-1}, x_{t-2}, \ldots, x_1)\) distribution
Sample from that to get the \(x_t\) token
- “Temperature” setting mechanically increases/decreases the randomness for the logit (e.g., rescales \(W z\))
Add its own generated token to the set of tokens, \(x_1, x_2, \ldots, x_{t-1}, x_t\)
Condition on those tokens to find the \(\mathbb{P}(x_{t+1} | x_t, x_{t-1}, \ldots, x_1)\) distribution
Repeat until it hits a special “done” token (itself predicted) or reaches a maximum length

Cardinality of the Estimated Conditional Distribution

ChatGPT and others generate text token-by-token auto-regressively
Not directly Markov. \(K^T\) possible sequences of \(K\) tokens of length \(T\)
Modest sized LLMs in 2025 might support \(T=4,000\) with \(K=50,000\) tokens
- Estimating a \(\mathbb{P}: 50,000 \times 50,000^{4,000}\to [0,1]\) conditional PMF
- And growing. Many recent models support \(128k\) tokens for context (i.e., conditioning) and \(16k\) for “output”

How Can This Possibly Work?

It sometimes feels like it has compressed and memorized the entire internet
We don’t always know the parameterization or number of tokens in the training data (remember, this largely uses MLE to train) but OpenAI’s GPT-3 was disclosed
- The \(\mathbb{P}(\cdot | \cdot)\) is approximated by approximating 175 billion parameters
- Trained on 300 billion tokens from close to a terabyte of cleaned text
Which is an absurdly small amount of data given the cardinality of \(\mathbb{P}\)
- Paraphrasing Mikhail Belkin: this is like estimating the Library of Congress with a molecule of ink.