TF-IDF Formula: How the Weight Is Calculated

01

What Does TF-IDF Measure?

Term frequency–inverse document frequency is a weighting method for document-term matrices. It starts with a term's relative frequency inside a document, then increases or reduces that value according to how many documents in the corpus contain the term.

A raw word count cannot distinguish a term that appears everywhere from one that helps identify a particular document. TF-IDF adds that corpus-level context. A term found in every document keeps the minimum inverse-document weight in this implementation. A term found in only one document receives a higher weight.

02

The Exact TF-IDF Formula Used by the Calculator

The calculator uses normalized term frequency. For term t in document d, divide its count by the number of analyzed tokens in that document:

Inverse document frequency uses natural logarithm and add-one smoothing. Let N be the number of documents and df(t) the number of documents containing the term:

The final weight is the product:

Add-one smoothing keeps the calculation defined and gives a term present in every document an IDF of exactly 1. Other libraries can use raw counts, logarithmic TF, a different logarithm base, or vector normalization. Those results can all be valid, but they should not be compared as though they came from one formula.

03

A Worked TF-IDF Example with Three Documents

Consider this small corpus after tokenization:

The corpus contains three documents. “apple” appears in one document, “carrot” in two, and “banana” in all three. Their smoothed inverse-document weights are:

“Apple” receives the strongest weight in Document A because it combines high within-document frequency with low document frequency. “Banana” occurs in every document, so corpus rarity does not increase its weight. “Carrot” has a positive IDF, but its weight in A is zero because it does not occur there.

04

Why the Corpus Changes Every IDF Score

IDF is not an intrinsic property of a word. It belongs to a word inside a particular corpus. If you add ten documents containing “apple,” its document frequency rises and its IDF falls. If you compare only two drafts about the same subject, many topical words may appear in both and receive the minimum weight.

This makes corpus design part of the analysis. A two-document comparison answers “what distinguishes these two inputs?” A ten-document corpus can answer “what distinguishes each document within this selected set?” Neither result automatically represents rarity across the web, a language, or an industry.

05

Tokenization and Stop Words Change the Matrix

Before TF-IDF can be calculated, every document is converted into a Bag of Words vector. The calculator lowercases text, removes markup and punctuation, excludes numeric-only tokens, and applies the same language and stop-word configuration to every source.

06

When TF-IDF Is Useful

TF-IDF is useful when transparent lexical features matter: document classification, clustering baselines, search indexing experiments, corpus exploration, revision analysis, and pre-filtering before manual review. It also supplies weighted vectors for cosine similarity.

The method is especially helpful when raw frequency is dominated by terms that appear throughout the corpus. Because every value can be traced to a count, a document frequency, and a formula, it is easier to audit than an opaque similarity score.

07

What TF-IDF Cannot Tell You

TF-IDF does not understand synonyms, word order, negation, factual accuracy, or intent. Two documents can discuss the same idea with different vocabulary and receive weak overlap. Two documents can repeat the same terms while making opposite claims and receive strong overlap.

It is also not an SEO score. A high weight means that a term is concentrated in a document relative to the selected corpus. It does not mean the term should be added more often or that the document will rank.