Order-preserving deduplication
Unlike a set-based approach that shuffles results, this keeps your original ordering — essential when the list has meaning, like a ranked keyword set.
TextRuns in your browser
Remove Duplicate Lines
Remove duplicate lines from a list in one paste, and keep the order you want.
- Input
- No upload needed — instant
- Privacy
- Nothing is uploaded
- Cost
- Free · no sign-up · no watermark
Loading tool…
The tool is loading its code on your device. This happens once and is cached for later visits.
Processed entirely on your device
Everything you type or paste is handled by JavaScript running in this tab. No request is sent, nothing is logged and nothing is stored. Close the page and it is gone.
Overview
About the Remove Duplicate Lines
Remove duplicate lines from text online. Keep the first or last occurrence, ignore case and blank lines, trim whitespace, and see exactly how many lines were removed.
Duplicate lines creep in from everywhere: copy-pasting from several sources into one list, exporting a database view with joined rows, scraping results that overlap between pages, or a keyword export where the same term appears under two different match types. This tool removes duplicate lines from text while preserving the original order — exactly what you need when the list has meaning and a naive dedupe would scramble it.
Why order matters when you remove duplicate lines
The obvious way to deduplicate is to push lines into a Set and read them back. That works, but a Set in JavaScript preserves insertion order only for the first occurrence of each value, and many quick implementations sort as a side effect. When your list is ranked — top keywords by volume, a priority queue, a chronological log — an unexpected reorder silently destroys the information you actually cared about.
This duplicate line remover walks the list once, records what it has already seen, and emits lines in their original positions.
The normalisation toggles for removing duplicate lines
Exact matching is rarely what you want on real data. Three problems account for almost all "missed" duplicates:
Trailing whitespace. Copying from a PDF or a spreadsheet often appends a space or a non-breaking space. Trimming before comparison catches these. Note that trimming also cleans the output, not just the comparison.
Case variation. SEO, seo and Seo are the same term for almost every purpose. Case-insensitive matching compares lowercased copies while preserving the original casing of whichever line survives.
Blank lines. Multiple consecutive empty lines are visually noisy and break some import formats. Removing them is a separate toggle so you can keep intentional paragraph breaks if you need them.
Choosing first or last occurrence when removing duplicates
Consider a list of key=value settings where a key appears twice. Keeping the first gives you the original default; keeping the last gives you the effective override. Neither is universally correct — it depends on whether later entries are meant to supersede earlier ones. The toggle exists because the answer genuinely varies.
Working with other tools
A common pipeline: paste raw data → remove duplicates → sort lines → copy into a spreadsheet. Another: remove duplicates → keyword density checker, to get accurate frequency counts that are not inflated by repeated export rows.
If the duplicates you are chasing are subtler — near-matches, or duplicates only within a field of a delimited row — the regex tester with a global match is the more precise instrument.
Common use cases for duplicate line removal
A few scenarios where removing duplicate lines saves real time: deduplicating a keyword list exported from two different SEO tools (the overlap is often 40% or more), cleaning a mailing list assembled from several sign-up forms, or deduplicating a log file where the same event was captured by two monitoring agents.
The tool runs entirely in the browser, which means confidential data — customer lists, internal logs, unpublished research — is never uploaded. Deduplication uses a hash set and runs in linear time, so a hundred thousand lines process in well under a second. The limit is your browser's memory, not the algorithm.
Step by step
How to use the Remove Duplicate Lines
Paste the list or text block containing duplicates.
Choose whether to keep the first or the last occurrence of each repeated line.
Toggle case-insensitive matching, blank-line removal and whitespace trimming as needed.
Press Remove duplicates and read the summary of what changed.
Copy the cleaned list or download it as a text file.
Why use it
Benefits and common use cases
What this tool is good for, and what it deliberately does not try to do.
First or last occurrence
Keeping the first preserves the original appearance; keeping the last is what you want when later entries are updates that supersede earlier ones.
Normalisation options
Trim trailing spaces, ignore case and drop blank lines — the three sources of duplicates that a naive exact-match dedupe misses.
Reports the delta
You see input lines, output lines and how many were removed, so you can sanity-check before trusting the result.
Questions
Frequently asked questions
Short, honest answers about quality, limits and privacy.
Does removing duplicates change the line order?
No. Lines are processed top to bottom and kept in place, so the output preserves your original sequence with repetitions removed. Sorting is a separate operation — use the Sort Lines tool if you want alphabetical order.
What counts as a duplicate?
By default, an exact character-for-character match of a whole line. With case-insensitive matching on, 'Apple' and 'apple' are duplicates. With trimming on, 'Apple' and 'Apple ' are duplicates. The first version encountered is the one that survives.
Should I keep the first or the last occurrence?
Keep the first when order of appearance matters — a reading list, a ranked set of keywords, a log you want to preserve chronologically. Keep the last when later lines override earlier ones, such as a configuration list or a changelog where the newest entry wins.
Can it handle very large lists?
Yes. Deduplication uses a hash set, so it runs in linear time — a hundred thousand lines process in well under a second. The limit is your browser's memory for holding the text, not the algorithm.
Does it remove duplicate words within a line?
No — this tool operates on whole lines. For repeated words inside a sentence you want a text cleaner or a manual edit; removing intra-line duplicates automatically would damage ordinary prose.