Duplicate Line Remover
Removes duplicate lines, keeps only duplicates, or counts how many times each line appeared.
Removes repeated lines from a list, or shows you only the ones that repeat — with the comparison rules made explicit rather than guessed at. Trailing spaces and Unicode normalisation are the two reasons two identical-looking lines both survive an ordinary deduplication.
Lines
Rules
Keep the first copy of each line. The list, deduplicated.
Korean text is compared after Unicode normalisation, so a name copied from a Mac matches the same name typed on Windows — they are different byte sequences that look identical.
Result
Lines in
6
Distinct
3
Repeated
2
Lines out
3
How to use it
Paste the list
One value per line — email addresses, order numbers, names out of a spreadsheet column.
Choose what to keep
The deduplicated list, only the values that appeared exactly once, or only the ones that repeated. The last is what you want when duplicates are the problem rather than the noise.
Set the comparison rules
Case sensitivity and surrounding spaces are the two that change the answer most. Turning on counts shows how many times each value appeared, which is a different question and often the real one.
Common questions
Two identical names were both kept
Check the trailing spaces first — turn on "ignore surrounding spaces". If that is not it, the two are different byte sequences that render the same, which is what Unicode normalisation is for; this tool already compares after normalising, so a Mac-typed name matches a Windows-typed one.
Does it keep the original order?
Yes. The first copy of each value stays where it was; the later ones go. Nothing is sorted unless you take the output to the sorter.
Can I deduplicate by one column of a CSV?
Not here — this compares whole lines. The CSV deduplicator does it by key column and keeps the rest of the row intact, which is what you want for tabular data.
Feed it from
Line Sorter & Shuffler
Sorts lines alphabetically, numerically, by length or randomly, with reverse and case options.
Whitespace Cleaner
Trims trailing spaces, collapses runs of blanks, converts tabs and removes empty lines.
List Formatter
Turns lines into a comma list, quoted SQL IN clause, JSON array or bullet list, and back again.
Pattern Extractor
Pulls every email, URL, number, date or custom regex match out of a blob of text into a list.
Next
These accept what this tool produces, so the result can go straight in.
CSV Deduplicator
Removes duplicate rows on all or selected key columns, keeping the first, last or most complete row.
List Formatter
Turns lines into a comma list, quoted SQL IN clause, JSON array or bullet list, and back again.
Line Sorter & Shuffler
Sorts lines alphabetically, numerically, by length or randomly, with reverse and case options.
Word Frequency
Counts how often each word or n-gram appears, with Korean-aware tokenising and a stopword list.
Find & Replace
Replaces text by literal, whole word or regular expression, with capture groups and a preview of every hit.
Markdown Editor
Edits Markdown with a live preview and exports the rendered HTML, with GFM tables and task lists.