01
What Does Word Frequency Mean?
Word frequency is the number of times a word appears in a text. If “analysis” occurs five times, its raw frequency is 5. A complete frequency table repeats this calculation for every unique token and usually sorts the result from the most common word to the least common.
Raw counts are easy to understand, but they are tied to document length. Five mentions in a 100-word product description are much more concentrated than five mentions in a 3,000-word guide. That is why frequency reports often add a percentage or a normalized rate.
Use count for the exact number of occurrences, percentage for the share of analyzed words, and per 1,000 for an intuitive normalized comparison.
02
Word Frequency Formulas
The raw formula is simply the number of exact matches after the text has been normalized. Percentage and per-1,000 rates divide that count by the same total number of analyzed words.
count(word)(count ÷ total words) × 100(count ÷ total words) × 1,000count(unique words)The denominator must be defined. If stop words are excluded, the total should be the number of words remaining after that filter. If every word is kept, the denominator should include every token. Mixing a filtered numerator with an unfiltered denominator makes the output harder to interpret.
03
A Worked Word-Frequency Example
Consider the sentence: “Text tools analyze text and compare text.” After lowercasing and removing punctuation, the tokens are:
[text, tools, analyze, text, and, compare, text]There are 7 total words and 5 unique words. “Text” occurs 3 times, so its percentage is 3 ÷ 7 × 100 = 42.86%. Its normalized rate is 3 ÷ 7 × 1,000 = 428.6 per 1,000 words.
3
42.86%
1
14.29%
1
14.29%
1
14.29%
1
14.29%
If “and” is treated as a stop word and removed, the filtered total becomes 6. “Text” still occurs 3 times, but its percentage becomes 50%. Neither answer is universally correct; the preprocessing rule determines which question the table answers.
04
Tokenization Choices Change the Count
A calculator must decide how to split and normalize text before counting. Lowercasing normally combines “Text” and “text.” Punctuation is usually removed, while apostrophes may remain inside a token. Numeric-only tokens may be excluded. HTML analysis should remove markup, scripts, and styles before counting visible language.
Case normalization
Combines capitalized and lowercase forms unless case itself matters to the task.
Stop-word filtering
Can make topic vocabulary easier to inspect, but changes both the table and its denominator.
Stemming or lemmatization
May combine related forms such as “analyze” and “analyzing,” but a plain counter normally keeps them separate.
N-grams
Count consecutive phrases such as “word frequency” instead of treating each word independently.
Production NLP libraries expose similar choices. For example, scikit-learn documents tokenization and occurrence counting with CountVectorizer, including configurable word and n-gram features.
05
How to Compare Texts of Different Lengths
Use percentages or per-1,000 rates when documents have different word counts. Suppose “conversion” appears 6 times in a 600-word draft and 8 times in a 1,200-word page. The second page has the higher raw count, but the first has the higher normalized rate: 10 versus 6.7 occurrences per 1,000 words.
Normalization does not make the texts equivalent. Templates, navigation, quoted material, and different purposes can all change vocabulary. Treat the number as a comparable measurement, then inspect the language in context.
Keep tokenization and stop-word settings identical across A and B. Otherwise the difference may come from preprocessing rather than the documents.
06
Practical Uses for a Frequency Table
Editors can find accidental repetition, researchers can inspect vocabulary patterns, and developers can build transparent Bag of Words features. SEO teams can compare a draft with a published page, but frequency is not a ranking score and should not replace a review of intent, usefulness, facts, and readability.
- Which terms dominate the text?
- Does a template distort the vocabulary?
- Which words appear only once?
- How did a revision change emphasis?
- Are expected terms completely absent?
- Which phrases deserve n-gram analysis?
For a deeper explanation of how counts become document vectors, continue to the Bag of Words model guide. To calculate the table immediately, use the free word frequency counter.
FREE · NO SIGN-UP
Calculate Word Frequency Now
Paste text or enter a public URL, search and sort the vocabulary, edit stop words, and export the complete result as CSV or JSON.
Open the word frequency counter →