How to use remove duplicate words
- Paste your text into Your text.
- Keep Ignore case ticked so “The” and “the” match.
- Keep Only remove immediate repeats (“the the”) ticked to fix typing slips in normal writing. Untick it to keep only the first use of every word.
- Check the Removed count, then copy the result.
What it does and when to use it
Doubled words are among the most common typing mistakes, and they are hard to see. The eye skips over “the the”, especially across a line break. Spell checkers do not always catch it. This tool finds and deletes them in one pass.
The second mode does a different job. It keeps only the first use of every word, which is useful for:
- cleaning a keyword or tag list where the same term was added twice;
- making a list of the distinct words in a short text while keeping their order of first use.
If the items are on separate lines, remove duplicates works on whole lines instead. To list each distinct word in alphabetical order, use alphabetize words with its own duplicate option.
How it works
The text is processed line by line, and each line is split into words and the spaces between them. For each word the tool builds a comparison key: punctuation at both ends is removed and, with Ignore case, the key is made lowercase.
- In immediate repeats mode, a word is dropped when its key equals the key of the word just before it. This check carries over line breaks, so a word at the end of one line and the same word at the start of the next also counts.
- In all repeats mode, a word is dropped when its key has been seen anywhere earlier in the text.
A dropped word is removed together with the space before it, so no gaps are left. The count of dropped words is shown as Removed. The text stays in your browser.
Worked examples
These outputs come from the tool’s own function.
Typing slips. “This is is a test test of the the duplicate word remover. This test is short.” gives This is a test of the duplicate word remover. This test is short. with Removed: 3. The second “This test is” was kept, because those words were not doubled.
A tag list. “red blue red green blue” with immediate repeats unticked gives red blue green.
Case matters when you ask it to. “The the end. End” with Ignore case unticked gives Removed: 0.
Limits and tips
- Closing punctuation on a removed copy is kept: “It is the end end.” becomes “It is the end.” A removed copy in brackets or quotes, such as “(the)”, goes with its brackets.
- Some doubled words are correct, as in “I had had enough” or “that that”. Read the result before you replace your original.
Frequently asked questions
What is the difference between the two modes?
Does punctuation stop a match?
Is "The" the same word as "the"?
Which copy is kept?
How do I just find repeated words without deleting them?
How do we know nothing is uploaded? Test it yourself on the privacy proof page.