Remove Duplicate Lines — Keep the First of Each, in Order

How to use Remove Duplicate Lines

1. Paste your list into One item per line.
2. Tick Ignore case if “Gmail.com” and “gmail.com” should count as the same line.
3. Leave Trim spaces before comparing ticked to ignore spaces at the start and end of each line, or untick it for an exact match.
4. Read the Kept and Removed counts, then copy or download the unique lines.

What it does and when to use it

Lists collect repeats. Mailing lists are merged from two exports, keyword lists are built over several sessions, and log files repeat the same message. This tool leaves exactly one copy of each line and tells you how many it took out.

The order of your list stays the same. That is the main difference from the classic command-line method of sort | uniq, which has to sort the list first and loses the order you had. Here, a list in priority order, date order or the order a database returned it keeps that order.

When to use each option:

• Ignore case off suits identifiers, file paths, API keys and passwords, where “ABC” and “abc” really are different.
• Ignore case on suits email addresses, domain names, tags and most lists typed by people, where keeping both spellings is the mistake.
• Trim on suits data pasted from a spreadsheet or PDF, which often carries invisible trailing spaces.

How it works

The tool splits your text into lines. For each line it builds a comparison key: the line itself, trimmed if Trim is on and lowercased if Ignore case is on. It remembers every key it has seen. A line whose key is new is kept, exactly as you typed it. A line whose key has been seen before is removed. This all runs in your browser, so customer lists and exported data never leave your device.

Worked examples

Case-sensitive (the default). The list apple, banana, apple, cherry, Banana gives Kept: 4, Removed: 1. Only the second “apple” goes. “Banana” with a capital B stays, because case matters.

Ignore case on. The same list gives Kept: 3, Removed: 2. Both the second “apple” and “Banana” go, leaving apple, banana, cherry.

Exact match. With both options off, the lines “anna@example.com”, “ anna@example.com” (with a leading space) and “Anna@Example.com” are all different, so the result is Removed: 0. Turning on Trim and Ignore case would reduce them to one line.

Limits and tips

• The tool works on whole lines. To remove repeated words inside a sentence, use Remove Duplicate Words. To list repeated words without removing them, use the Duplicate Word Finder.
• Blank lines count as lines. Clear them out with Remove Empty Lines.
• Messy spacing inside lines (two spaces between words, for example) still makes lines different. Run Remove Extra Spaces first if that is a problem.
• Ignore case uses simple lowercasing, so the kept line keeps its original capitals.