Skip to content
Nagendra Inuguri
Go back

Summarising thousands of support chats before LLMs: what extractive NLP got right

Support teams sit on one of the richest data sets a business has: thousands of conversations a month in which customers explain, in their own words, what’s broken. Most of it is never read twice. Each month someone skims a sample, pulls out a few themes for a business review, and the rest disappears.

For a large online advertising platform, we automated that. The goal was to turn every support chat into a short, readable summary, and to roll those up into themes for each part of the product, so the monthly business reviews could be built from all the conversations rather than a sample. The work ran through 2022, before LLMs were an option for this, and the automation saved about $300K in manual analysis effort.

Why chat data is harder than it looks

Support chats are messy in very specific ways:

  • Most of the text is boilerplate. Greetings, sign-offs, “please hold while I check”, satisfaction-survey prompts. In our data, roughly 40% of chats were filtered out by removing the most frequently repeated sentences alone.
  • The useful line is buried. In a 40-message chat, one or two sentences actually describe the problem.
  • It isn’t clean text. Links, HTML fragments, masked personal data, typos and the occasional non-English conversation.
  • Domain words dominate. Product and feature names appear in almost every chat, so they say nothing about any one issue.

Feeding that straight into any model gives you summaries full of “thank you for contacting support”.

The pipeline

We built it as a set of Databricks notebooks, scheduled as a pipeline, iterating every couple of weeks with the business.

The pipeline: split and strip, filter noise, normalise and score sentences, producing chat-level summaries and category-level insight for monthly business reviews

1. Split and strip. Extract links first (they’re useful later), remove HTML, and split each message into sentences before any other processing.

2. Filter noise. Drop sentences of three words or fewer, unless they contain a word from a curated “important words” list. Remove the most frequent sentences across all chats, after first checking which of them were actually informative. Remove greetings by cosine similarity to a set of known greetings, mask personal data, and drop non-English chats.

3. Normalise. Replace common phrases with short tokens, words first, then bigrams, then trigrams, so “campaign budget limited” counts as one idea. Remove stop words from three lists: standard ones, common words we identified, and support-specific ones. We also split NLTK’s stop words into negative and neutral sets, because dropping “not” from “ads not showing” changes the meaning. spaCy’s lemmatisation worked better than NLTK’s for this text.

4. Score sentences. Score every remaining sentence with TF-IDF (and plain term frequency as a comparison), and keep the highest-scoring one.

Two kinds of output

A chat-level summary. For every conversation: the most important sentence, plus the sentence before and after it for context, shown in its original wording rather than the cleaned version. Each sentence is tagged with who said it (customer or agent), and any help links the agent shared are listed with their titles, which we also automated.

Category-level insight. Every chat is filed under a five-level issue category tree. For each node in that tree, we pulled the most representative sentences and the top words, using the same TF-IDF and term-frequency scores across all the chats in that category. That’s what made the monthly business reviews possible: for any product area, you could read the dominant problems in the customers’ own words.

Topic modelling: lots of attempts, one winner

We also wanted themes that cut across the category tree, so we tried most of the standard toolbox:

What worked: TF-IDF sentence scoring, domain stop-word lists, phrase replacement and NMF topic models. What didn't: affinity propagation, K-means and DBSCAN, LDA on nouns only, and spaCy language detection

  • Clustering with K-means and DBSCAN produced blurry, overlapping clusters. Affinity propagation didn’t converge on around 6,000 sentences, and on smaller samples it produced far too many clusters.
  • LDA, including a version that used only nouns identified by spaCy, produced topics that were hard to name.
  • Non-negative matrix factorisation (NMF), using Gensim, gave the clearest topics. We picked the number of topics by coherence score rather than by eye, suppressed generic words and category names so topics weren’t just restating the tree, and checked whether the data had real cluster structure at all with Hopkins statistics.

The output was a topic for every chat, and named topics for every category, built from the top words.

Lessons learned

  • Cleaning is the model. Most of the quality came from removing boilerplate and normalising language, not from the scoring method. The stop-word and phrase lists were the most valuable artefacts of the project.
  • Domain stop words aren’t optional. Product names and support phrases dominate frequency counts. Without suppressing them, every summary and topic says the same thing.
  • Be careful with negation. Generic stop-word lists quietly remove “not” and “no”, which reverses the meaning of a problem description.
  • Show the original sentence. Score on cleaned text, but show people the original wording, with context either side. Readers trust what they recognise.
  • Measure topics, don’t eyeball them. Coherence scores for choosing the number of topics, and Hopkins statistics for checking cluster structure, saved weeks of debate.

Would I do it with an LLM today?

Partly. An LLM would write far more fluent summaries and name topics better. But the hard-won parts of this pipeline still matter: removing boilerplate before you pay for tokens, masking personal data, keeping the original sentence as evidence, and rolling summaries up a category tree. Extractive scoring is also cheap, deterministic and easy to explain, which counts for a lot when the output feeds a business review.

The takeaway

Before LLMs, we got useful, trusted summaries from thousands of support chats a month with careful cleaning, simple scoring and a well-tuned topic model, and replaced a large manual effort in the process. The models were never the hard part. Understanding what makes the text noisy, and removing it carefully, was.

If you’re working on support or conversation analytics, I’d love to compare notes. Find me on LinkedIn.



Previous Post
Forecasting support demand for markets with almost no history
Next Post
Classifying user feedback when 97% of it is one class