01
An N-gram Is a Sliding Token Window
After text has been tokenized, an n-gram analyzer moves a window of length n across the tokens one position at a time. Every window becomes a phrase candidate. Identical sequences are grouped and counted.
In a word-level analyzer, tokens are usually normalized words. Character n-grams are also common in machine learning, but they answer a different question. The Text Analysis Tools calculator uses word-level n-grams from one to ten tokens.
For tokens w₁, w₂, …, wₘ, each n-gram is a consecutive sequence (wᵢ, …, wᵢ₊ₙ₋₁).
02
Unigrams, Bigrams, Trigrams, and Longer Sequences
Unigram · n=1
Each individual word becomes a feature. This is equivalent to a basic word-frequency table.
Bigram · n=2
Each adjacent word pair becomes a feature, preserving short expressions such as “text analysis.”
Trigram · n=3
Three-word windows capture more specific fragments such as “free text analysis.”
Longer n-grams
Four-to-ten-token windows can expose repeated clauses, templates, labels, and copied phrase fragments.
Larger n creates more specific features, but repetitions become rarer. The vocabulary can also grow quickly because a small change in one token creates a different sequence.
03
A Worked Bigram Example
Take the six-token sentence:
text analysis makes text analysis clearIts five overlapping bigram windows are:
“Text analysis” appears twice. Every other bigram appears once. Because there are five bigram windows, the phrase has a window percentage of 2 ÷ 5 × 100 = 40% and a normalized rate of 400 per 1,000 windows.
04
The Denominator Is the Number of Available Windows
For a document containing m tokens, the number of length-n windows is:
window count = max(0, m − n + 1)Each row uses that window count as its denominator:
percentage = phrase count / window count × 100per 1,000 = phrase count / window count × 1,000This differs from keyword-density calculations that divide phrase occurrences by the total word count. Both conventions can be useful, but the denominator must be stated before values are compared.
05
Stop Words and Minimum Counts Change What You See
The analyzer builds n-gram windows from the original normalized token sequence. When default stop words are excluded, it hides a sequence only when every token in that sequence is a stop word. It does not remove words first and then join previously non-adjacent tokens. This preserves the phrase order that actually appeared in the source.
A minimum-count filter can hide one-off sequences from the visible table. It does not alter the underlying token count or denominator. Exports contain the rows that match the selected filter and result limit.
Compare n-gram tables only when n, language, stop-word behavior, and input-cleanup rules are the same.
06
What N-gram Analysis Is Good For
Pattern inspection
- Finding repeated multi-word phrases
- Auditing templates and boilerplate
- Building features for classifiers
- Comparing phrase usage between revisions
- Detecting recurring labels or clauses
Interpretation
- A repeated phrase is not automatically a problem
- A rare phrase is not automatically important
- Different tokenizers create different windows
- Longer sequences need more source text
- Counts do not establish meaning
Use the comparison workflow when the main question is how phrases changed between two versions rather than which sequences dominate one document.
07
N-grams Preserve Local Order, Not Meaning
An n-gram model knows that tokens appeared next to one another. It does not understand why they appeared, whether they are factual, or whether two different phrases express the same concept. Sparse longer sequences also make cross-document matching difficult.
For transparent lexical diagnostics, that simplicity is an advantage. For semantic retrieval or paraphrase detection, use n-grams as one signal rather than the complete decision.
Technical references
1–10 TOKENS · FULL EXPORT
Count phrase windows in text or a webpage
Choose n, inspect the exact denominator, filter recurring sequences, and export the table.
Open the N-gram analyzer →