Remove Duplicate Lines
Deduplicate a list without losing its order.
- Files stay on your device
- No upload, no waiting
- No sign-up, no watermark
Related tools
All text toolsQuestions
Does removing duplicates change the order of my list?
No. The first occurrence of each line stays exactly where it was and later repeats are dropped. That is deliberately different from the usual sort | uniq approach on the command line, which has to sort the list first and therefore destroys the original order. If your list is a ranking, a chronology or a sequence of steps, order-preserving deduplication is the only correct behaviour.
Are 'Apple' and 'apple' treated as duplicates?
By default yes — matching is case-insensitive, and the first spelling encountered is the one kept. Switch on 'Case sensitive' when case actually carries meaning, which it does for passwords, API keys, code identifiers, Linux filenames and anything destined for a case-sensitive database.
Why are lines that look identical not being removed?
Almost always invisible whitespace. Lists copied from spreadsheets, PDFs and web pages routinely carry trailing spaces, tabs or non-breaking spaces that make two visually identical lines different strings. Leave 'Trim whitespace' switched on — it is on by default for exactly this reason — and they will collapse correctly.
Can I see which lines were duplicated instead of removing them?
Yes. Switch on 'Show only duplicates' and the tool inverts: it lists every line that appeared more than once, one entry each. That is the quick way to audit a mailing list for repeated addresses, spot SKUs entered twice in an inventory export, or check whether a merged CSV has double-counted rows.
How large a list can it handle?
Comfortably into the hundreds of thousands of lines. Deduplication uses a hash set, so it runs in roughly linear time rather than comparing every line against every other. Everything happens in your browser, so there is no upload, no row cap, and nothing for us to store.
About remove duplicates
What counts as a duplicate
Two lines are duplicates when they match after the options you have chosen are applied. That sounds obvious, but the options change the answer a great deal, and picking the wrong ones is the usual reason a deduplicated list still looks wrong.
- Trim whitespace (on by default) —
appleandapplebecome the same line. Copy-pasted lists from spreadsheets and PDFs are full of invisible trailing spaces, so leaving this on is almost always right. - Case sensitive (off by default) — with it off,
Appleandappleare one entry, and the first spelling encountered is the one kept. Turn it on for anything case-significant: passwords, API keys, code identifiers, Linux filenames. - Remove blank lines — blank lines otherwise deduplicate down to a single blank, which is rarely what anyone wants.
Order is preserved, and that matters
The first occurrence of each line is kept in its original position; later repeats are dropped. That is different from the usual command-line approach, where sort | uniq requires sorting first and therefore destroys the original order. If your list is a ranking, a chronology or a set of steps, order-preserving deduplication is the only correct answer.
If you do want it sorted, the Sort A–Z option applies after deduplication, using locale-aware comparison so accented characters land where a reader would expect rather than after Z.
Finding duplicates instead of removing them
Switch on “Show only duplicates” and the tool inverts: you get the lines that appeared more than once, one entry each. This is the fast way to audit a mailing list for repeated addresses, find SKUs entered twice in an inventory export, or check whether a merged CSV has double-counted anything — without opening a spreadsheet.
Related
Browse the rest of the text tools, or count what you are left with using the word counter.