Introduction

Political debates are a big part of the presidential races in the United States. They allow candidates to talk about important issues together as well as share important ideas; this allows for some neat and relevant textual analysis.

While I was looking at data sets for this final project I stumbled upon an interesting data set on Kaggle that were the 2020 Democratic Debate transcripts. After recently learning about topic modeling I decided to choose that data set and apply my new topic modeling skills to answer some questions I had:

Some important things to note:

Getting the Data

Full link: https://www.kaggle.com/brandenciranni/democratic-debate-transcripts-2020

Importing the Data

# Packages used to clean
library(tidyverse)
## -- Attaching packages ------------------------------------------------------------------------------------- tidyverse 1.3.0 --
## v ggplot2 3.3.0     v purrr   0.3.3
## v tibble  3.0.1     v dplyr   0.8.4
## v tidyr   1.0.2     v stringr 1.4.0
## v readr   1.3.1     v forcats 0.5.0
## -- Conflicts ---------------------------------------------------------------------------------------- tidyverse_conflicts() --
## x dplyr::filter() masks stats::filter()
## x dplyr::lag()    masks stats::lag()
library(tidytext)
library(lubridate)
## 
## Attaching package: 'lubridate'
## The following objects are masked from 'package:dplyr':
## 
##     intersect, setdiff, union
## The following objects are masked from 'package:base':
## 
##     date, intersect, setdiff, union
library(stringr)
library(ggplot2)
library(tidyr)
library(citr)

Read the data into R.

# Reads the data in from csv format
debate_data <-
  read.csv(
    here::here("data", "raw", "debate_transcripts_v3_2020-02-26.csv"),
    stringsAsFactors = FALSE,
    encoding = "UTF-8"
  )

# I perfer to use tibbles :)
debate_data <- tibble(debate_data)

I specify stringAsFactors = FALSE and encoding = "UTF-8" due to conflict between the characters encoding from the original data set.

A peek into the data:

# Structure of the data
str(debate_data)
## tibble [5,911 x 6] (S3: tbl_df/tbl/data.frame)
##  $ date                 : chr [1:5911] "02-25-2020" "02-25-2020" "02-25-2020" "02-25-2020" ...
##  $ debate_name          : chr [1:5911] "South Carolina Democratic Debate Transcript: February 25 Democratic Debate" "South Carolina Democratic Debate Transcript: February 25 Democratic Debate" "South Carolina Democratic Debate Transcript: February 25 Democratic Debate" "South Carolina Democratic Debate Transcript: February 25 Democratic Debate" ...
##  $ debate_section       : chr [1:5911] "Part 1: South Carolina Democratic Debate Transcript" "Part 1: South Carolina Democratic Debate Transcript" "Part 1: South Carolina Democratic Debate Transcript" "Part 1: South Carolina Democratic Debate Transcript" ...
##  $ speaker              : chr [1:5911] "Norah O<U+0092>Donnell" "Gayle King" "Norah O<U+0092>Donnell" "Gayle King" ...
##  $ speech               : chr [1:5911] "Good evening and welcome, the Democratic presidential primary here in South Carolina. The first primary in the "| __truncated__ "And Super Tuesday is just a week away and this is the biggest primary day of the year as voters in 14 states ca"| __truncated__ "And CBS News is proud to bring you this debate along with our co-sponsors. They are the Democratic National Com"| __truncated__ "And we are partnering tonight also with Twitter. So you at home can participate in this debate. How do you do t"| __truncated__ ...
##  $ speaking_time_seconds: num [1:5911] 8 22 14 10 31 59 5 5 19 27 ...
# Names of the columns
colnames(debate_data)
## [1] "date"                  "debate_name"           "debate_section"       
## [4] "speaker"               "speech"                "speaking_time_seconds"
# This takes a look at the dimensions of our data in a clear format
paste("Our data has", nrow(debate_data), "rows and", ncol(debate_data), "columns")
## [1] "Our data has 5911 rows and 6 columns"

Speaking Time

One of the questions asked was: Who talked the most?

Since the analyses are of candidates that continued through-out the debates (last one being the South Carolina Debate on Feb. 25) a list of them must be provided.

# This will filter out only the last debate items
list_of_speakers_last <-
  debate_data %>%
  filter(debate_name == "South Carolina Democratic Debate Transcript: February 25 Democratic Debate")

# This gives the names of the speakers in the last debate
unique(list_of_speakers_last$speaker)
##  [1] "Norah O<U+0092>Donnell" "Gayle King"           "Bernie Sanders"      
##  [4] "Michael Bloomberg"    "Pete Buttigieg"       "Elizabeth Warren"    
##  [7] "Tom Steyer"           "Joe Biden"            "Amy Klobuchar"       
## [10] "Bill Whitaker"        "Major Garrett"        "Speaker 1"           
## [13] "Margaret Brennan"

The speakers can be seen above.

Creating a list of candidates.

# This is a list of the candidates of the last debate
list_of_candidates_last <- c("Bernie Sanders", "Joe Biden", "Elizabeth Warren", "Michael Bloomberg", "Pete Buttigieg", "Tom Steyer", "Amy Klobuchar")

I saw some of the speaking times were 0, I don’t think that made sense with the words found in those occurrences so I changed them to 1 second.

# There were some speaking times of 0 (which doesn't seem to make sense) so I replaced them with 1 second.
debate_data <-
  debate_data %>%
  mutate(speaking_time_seconds = ifelse(speaking_time_seconds == 0, 1, speaking_time_seconds))

Now we can start making a graph of the data to visualize it.

# Here I am going to graph the speaking time in minutes for our candidates
speaking_time_graph <-
  debate_data %>%
  tidyr::drop_na() %>% # Drops NA
  group_by(speaker) %>% # I have to tell it to group by speaker because there are multiple rows of own speaker
  filter(speaker %in% list_of_candidates_last) %>%
  summarise(total_speaking_time_minutes = sum(speaking_time_seconds) / 60) %>% # Creates our minutes column
  mutate(speaker = fct_reorder(speaker, total_speaking_time_minutes)) %>% # reoders for clarity
  ggplot(aes(speaker, total_speaking_time_minutes)) +
  geom_col()

# Cleans it up a little bit and adds labels
speaking_time_graph +
  labs(
    title = "Total Speaking Time in Minutes",
    subtitle = "2020 Democratic Debates",
    x = "Speaker",
    y = "Speaking Time in Minutes"
  ) +
  geom_text(aes(label = round(total_speaking_time_minutes)), position = position_dodge(width = 0.9), vjust = -0.25)

We can see here that Sanders had the most speaking time (157 minutes) and Bloomberg had the least (27 minutes) and that makes sense because Bloomberg didn’t have start his campaign until later.

Tokenization

Tokenization of the text (split into individual words) must be done in order to perform textual analysis. This can be done using the tidytext package.

library(tidytext)

From here on only one candidate will be shown; however, changing the name will result in different analyses.

# I'm choosing Bernie Sanders because he has the most data.
candidate_name <- "Bernie Sanders"

Filter the transcripts for candidate_name.

candidate_transcripts <-
  debate_data %>%
  filter(speaker == candidate_name) %>%
  mutate(document = (1:nrow(.))) # Add's a document column

Splitting the text into documents is an important step in textual analysis. Each time the candidates speak are chosen as the document type.

Stop Words

Stop words must be removed from the text (words that carry little information like: a, we, I, me). (University of Liechtenstein et al., 2016, p. 10)

The tidy text packages has commonly used stop words; however, ;I’ve added my own.

custom_stop_words <- tibble::tribble(
  ~word,        ~lexicon,
  "america",    "custom",
  "american",   "custom",
  "americans",  "custom",
  "people",     "custom",
  "country",    "custom",
  "bring",      "custom",
  "'",          "custom", # The reason I added this one is because of the UTF-8 coding
  "don",        "custom",
  "ve",         "custom",
  "crosstalk",  "custom",
  "ain",        "custom",
  "ll",         "custom",
  "didn",       "custom",
  "president",  "custom",
  "donald",     "custom",
  "time",       "custom",
  "tonight",    "custom",
)

# Adds my custom stopwords with the already made stopwords
stop_words2 <- stop_words %>% # stop_words comes from the tidytext package
  bind_rows(custom_stop_words)

I’ve added words like “America”, “Tonight”,“President” and others since we are using debate data about the president/presidential race in America it doesn’t really convey information. I’ve added “Donald” because I believe they’re talking about Donald Trump but I have no way of knowing so I assume that the candidates will use “Trump” as well.

Now we tokenize the data.

# Unnested tokens data
candidate_token <-
  candidate_transcripts %>%
  unnest_tokens(word, speech, token = "words") %>%
  anti_join(stop_words2) %>% # Stop words
  arrange(word)
## Joining, by = "word"

Remove any digits (dollar amounts, statistics, etc).

candidate_token <-
  candidate_token %>%
  filter(!grepl("[[:digit:]]", word)) # removes any digits

Word Cloud

One nice visualization is a word cloud when it comes to textual data. They provide a nice intuitive way of looking at words that show up a lot.

Imposing thresholds allow for focus in word clouds. As an arbitrary measure, words that show up more than 7 times are shown on the word cloud.I chose 7 because I thought it brought out the key features of the data.

# Creating a word list with words being mentioned > 7 times
candidate_word <- candidate_token %>%
  count(word) %>%
  filter(n > 7)

The wordcloud package will be used for producing word clouds.

library(wordcloud)
## Loading required package: RColorBrewer
# Color-blind friendly palette
cbPalette <- c("#999999", "#D55E00", "#56B4E9")

candidate_cloud <- wordcloud(
  words = candidate_word$word,
  freq = candidate_word$n,
  color = cbPalette,
  random.order = FALSE,
  rot.per = 0,
  scale = c(2.5, 1),
  max.words = 70
) # Only shows 70 words

Descriptive Statistics

Before conducting topic modeling, its important to look at descriptive statistics of the data beforehand. Two questions that can lead to fruitful topic modeling are:

# gets the count of words per document
words_per_document <-
  candidate_token %>%
  count(document, speaking_time_seconds)

# Creates a dot plot of words by speaking time
words_per_time_plot <-
  ggplot(words_per_document, aes(x = speaking_time_seconds, y = n)) +
  geom_point() +
  geom_smooth() +
  labs(
    x = "Speaking Time in Seconds",
    y = "Words",
    title = "Words per Seconds of Speaking Time"
  )

# Creates a distribution plot of the number of words per document
distribution_of_words <- ggplot(words_per_document, aes(x = n)) +
  geom_histogram(aes(y = ..density..), # Histogram with density instead of count on y-axis
    binwidth = 1
  ) +
  geom_density(alpha = .2, fill = "#FF6666") +
  labs(
    x = "Number of Words",
    y = "Density",
    title = "Number of words per Document"
  )

# Creates a boxplot with the documents and words
boxplot_amt_words <-
  ggplot(words_per_document, aes(x = document, y = n)) +
  geom_boxplot() +
  geom_jitter(alpha = .5) +
  labs(
    x = "Documents",
    y = "# of Words",
    title = "Amount of words per document"
  )

# Loads patchwork
library(patchwork)

# Creates a plot with all three graphs
tri_plot <-
  boxplot_amt_words + distribution_of_words + words_per_time_plot +
  plot_annotation(
    title = "Descriptive Statistics of Documents",
    subtitle = paste("Candidate:", candidate_name)
  )
tri_plot
## Warning: Continuous x aesthetic -- did you forget aes(group=...)?
## `geom_smooth()` using method = 'loess' and formula 'y ~ x'

Looking at the first plot, a large amount of document have a higher density of smaller words. This can be problematic because of the data-hungry nature of topic modeling. It disrupts not only the creation of the model but also the diagnostics of determining the correct number of topics. This is why imposing a threshold of a speaking_time_seconds of more than 2 can be beneficial. I chose more than 2 seconds because after looking at the actual words it wasn’t until the 3 seconds mark I think we started seeing meaningful text/speech.

cut_off_time <- 2

candidate_token <-
  candidate_token %>%
  filter(speaking_time_seconds > cut_off_time)
## Warning: Continuous x aesthetic -- did you forget aes(group=...)?
## `geom_smooth()` using method = 'loess' and formula 'y ~ x'

Displayed above is the change in distribution when enacting the cut off time.

Topic Modeling

Topic modeling is a unsupervised machine learning technique. An initial number of topics must be given to the algorithm. The fun part is trying to figure out what number of topics are in the data. Showcased are four diagnostics used to determine the optimal number of topics: “held-out likelihood”,“semantic coherence”,“residuals”, and “lower-bound” metrics.(Training, Evaluating, and Interpreting Topic Models | Julia Silge, n.d.; Wallach et al., 2009)

The stm package is a great package for doing topic modeling.(Roberts et al., 2014; Training, Evaluating, and Interpreting Topic Models | Julia Silge, n.d.)

library(stm)
## stm v1.3.5 successfully loaded. See ?stm for help. 
##  Papers, resources, and other materials at structuraltopicmodel.com

One way the optimal number of topics can be found is by making a model for each number of topics (2-20) and graphing it.

First it is necessary to create something called a “Document-Frequency Matrix.”(Training, Evaluating, and Interpreting Topic Models | Julia Silge, n.d.) Basically this is a way of gathering our data in a format that shows the frequency of each word in a collection of documents. It looks like this:

Document Word1 Word2 Word3 Word 4
1 1 0 0 1
2 0 1 1 1
3 1 0 1 1
# Creating dfm
candidate_dfm <- candidate_token %>%
  count(document, word, sort = TRUE) %>%
  tidytext::cast_dfm(document, word, n)

A sparse matrix is also necessary (Training, Evaluating, and Interpreting Topic Models | Julia Silge, n.d.)

# Creates a cast_spares
candidate_sparse <- candidate_token %>%
  count(document, word) %>%
  tidytext::cast_sparse(document, word, n)

This way of evaluating an optimal amount of topics is computationally heavy. That’s why the furrr package for parallel computing will come in handy.

library(furrr)
## Loading required package: future
library(ggthemes) # For some themes for graphs

This tells the computer to plan for a multiprocess form of the package

future::plan(multiprocess)

This makes a data frame that will have all of our models (2-20). I chose the init.type = "Spectral" because that is what is suggested from the stm authors. (Roberts et al., 2014, p. 10)

set.seed(278)

many_models <-
  tibble(K = c(2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20)) %>%
  mutate(topic_model = future_map(K, ~ stm(
    candidate_sparse,
    K = .,
    verbose = FALSE,
    init.type = "Spectral"
  )))
## Warning in stm(candidate_sparse, K = ., verbose = FALSE, init.type =
## "Spectral"): K=2 is equivalent to a unidimensional scaling model which you may
## prefer.

Here, a selection is made of what documents to hold out from the model when evaluating it (Training, Evaluating, and Interpreting Topic Models | Julia Silge, n.d.). This is a similar process used for other numerical machine learning. We take a portion of the documents out and compare the model’s output to the held out and see how likely it fits.

# Makes a heldout
heldout <- make.heldout(candidate_sparse)
# heldout <- make.heldout(candidate_sparse, N = (.05*nrow(candidate_transcripts)), proportion = .25)

Now create the data frame in which will output the metrics.

# Creates a df of our metrics *from Julia Silge's Blog*
k_result <- many_models %>%
  mutate(
    exclusivity = map(topic_model, exclusivity),
    semantic_coherence = map(topic_model, semanticCoherence, candidate_sparse),
    eval_heldout = map(topic_model, eval.heldout, heldout$missing),
    residual = map(topic_model, checkResiduals, candidate_sparse),
    bound = map_dbl(topic_model, function(x) {
      max(x$convergence$bound)
    }),
    lfact = map_dbl(topic_model, function(x) {
      lfactorial(x$settings$dim$K)
    }),
    lbound = bound + lfact,
    iterations = map_dbl(topic_model, function(x) {
      length(x$convergence$bound)
    })
  )

Now visualize the output.

# Cleans the data for the graph
clustergraph_data <- k_result %>%
  transmute(
    K,
    `Lower bound` = lbound,
    Residuals = map_dbl(residual, "dispersion"),
    `Semantic coherence` = map_dbl(semantic_coherence, mean),
    `Held-out likelihood` = map_dbl(eval_heldout, "expected.heldout")
  ) %>%
  gather(Metric, Value, -K)

# Graphs the metrics.
clustergraph <-
  clustergraph_data %>%
  ggplot(aes(K, Value, color = Metric)) +
  geom_line(
    size = 1.5,
    alpha = 0.7,
    show.legend = FALSE
  ) +
  geom_point(alpha = 1, color = "Black") +
  facet_wrap(~Metric, scales = "free_y") +
  labs(
    x = "K (number of topics)",
    y = NULL,
    title = paste("Model diagnostics by number of topics for", candidate_name),
    subtitle = paste("Cut off time:", cut_off_time)
  )
clustergraph

# Optimal number of models
number_of_clusters <- 15

As seen above, the held-out-likelihood and semantic coherence are optimal at 15 topics. (Mimno et al., 2011)

Extract the topic with 15 topics.

topic_model <- k_result %>%
  filter(K == number_of_clusters) %>%
  pull(topic_model) %>%
  .[[1]]

The beta refers to topic-word density and will answer the question: What topics were most common?. (LDA Alpha and Beta Parameters - the Intuition - the Thought Vector Blog - Blog Vector, n.d.)

# tidy format for the beta measurment
td_beta <- tidy(topic_model)

# visualize the topics
topic_graph <-
  td_beta %>%
  group_by(topic) %>%
  top_n(10) %>%
  ungroup() %>%
  mutate(term = reorder(term, beta)) %>% # ordering it by their beta rank
  ggplot(aes(term, beta, fill = topic)) + # plotting it
  geom_col(show.legend = FALSE) +
  facet_wrap(~topic, scales = "free") + # by topic
  coord_flip() +
  labs(
    title = "Spectral Topic Model",
    subtitle = paste("Candidate:", candidate_name, "-", "Number of topics:", number_of_clusters),
    x = "",
    y = ""
  )
## Selecting by beta
topic_graph

Analysis

Looking at the different topics per candidate its interesting to see that one common topic was summed up as “Defeating Trump” across candidates. It also was cool to see how the topics found for each candidate were comprehensive and show words associated around political issues (environment, gun laws, etc). With this type of analysis it can be seen that debates are filled with key words associated with political issues;also showing a glimpse of how each candidate expresses each issue.

Other Candidates

Elizabeth Warren:

knitr::include_graphics(here::here("finalproject", "images", "10_cluster_elizabethwarren_topics.png"))

Joe Biden:

knitr::include_graphics(here::here("finalproject", "images", "10_cluster_joebiden_topics.png"))

Amy Klobuchar:

knitr::include_graphics(here::here("finalproject", "images", "9_cluster_amyklobuchar_topics.png"))

Tom Steyer:

knitr::include_graphics(here::here("finalproject", "images", "15_cluster_tomsteyer_topics.png"))

Notes

There was trouble with Michael Bloomberg’s and Pete Buttigeig’s topic models. Unfortunately, after looking for solutions on various help sites and GitHub, there was not a solution found. Topic modeling works best when theres a lot of data. Some candidates had more data than others making topic models easy in some cases and hard in others. For Bloomberg, it may have to do to the shortage of data; however, that doesn’t explain Buttigeig’s error. Below is the error that was received.

knitr::include_graphics(here::here("finalproject", "images", "error_console_image.png"))

References

LDA Alpha and Beta Parameters - The Intuition - the Thought Vector blog - Blog Vector. (n.d.). https://www.thoughtvector.io/blog/lda-alpha-and-beta-parameters-the-intuition/.

Mimno, D., Wallach, H. M., Talley, E., Leenders, M., & McCallum, A. (2011). Optimizing semantic coherence in topic models. Proceedings of the Conference on Empirical Methods in Natural Language Processing, 262–272.

Roberts, M. E., Stewart, B. M., & Tingley, D. (2014). Stm: R Package for Structural Topic Models. Journal of Statistical Software, 10(2), 1–40.

Training, evaluating, and interpreting topic models | Julia Silge. (n.d.). https://juliasilge.com/blog/evaluating-stm/.

University of Liechtenstein, Debortoli, S., Müller, O., IT University of Copenhagen, Junglas, I., Florida State University, vom Brocke, J., & University of Liechtenstein. (2016). Text Mining for Information Systems Researchers: An Annotated Topic Modeling Tutorial. Communications of the Association for Information Systems, 39, 110–135. https://doi.org/10.17705/1CAIS.03907

Wallach, H. M., Murray, I., Salakhutdinov, R., & Mimno, D. (2009). Evaluation methods for topic models. Proceedings of the 26th Annual International Conference on Machine Learning - ICML ’09, 1–8. https://doi.org/10.1145/1553374.1553515