Text Mining Word Analyzer

This simulation shows whether a selected word acts as a useful signal or unhelpful noise in a collection of documents.

The TF-IDF Formula

$$w_{t,d} = \text{TF}(t,d) \times \log\left(\frac{N}{n_t}\right)$$
  • \(w_{t,d}\): TF-IDF weight of term \(t\) in document \(d\)
  • \(\text{TF}(t,d)\): occurrences of term \(t\) in one document
  • \(N\): total number of documents
  • \(n_t\): number of documents containing term \(t\)

How the Analyzer Removes Noise

  • General stop words: common words such as the, a, and is.
  • Corpus stop words: words that occur in many documents in this particular collection and do not distinguish documents.
  • Threshold (max_df): words occurring above the selected percentage are classified as corpus noise.

Cosine Similarity

$$\text{sim}(d_1,d_2)=\frac{\vec{d_1}\cdot\vec{d_2}}{\lVert\vec{d_1}\rVert\,\lVert\vec{d_2}\rVert}$$
  • \(\cos(\theta) \approx 1\): documents are highly similar.
  • \(\cos(\theta) \approx 0\): documents have few shared important words.

Analyze Your Documents

Each line is treated as one document. Try contract for corpus noise, or customer and battery for more specific signals.
Corpus-noise threshold: 0.80 means words found in more than 80% of documents are treated as corpus noise.