Duplicate Content Checker: How to Find and Fix Duplicate Pages
Most duplicate content is not copied on purpose. It comes from the way a site builds URLs: the same page with and without www, with a tracking parameter, sorted a different way, or listed under two categories. Search engines then have to guess which version you mean. A duplicate content checker finds every copy, groups them, and shows which one should stay.
What a Duplicate Content Checker Does
A duplicate content checker compares the pages of a site with each other and reports the ones that are the same or nearly the same. A good one looks at four things:
- URLs that differ only in protocol, host, letter case, trailing slash or parameters but return the same page.
- Main text that is identical on two or more pages.
- Main text that is very similar, with a few words, a product name or a city swapped.
- Titles, meta descriptions and headings repeated across pages, which often point to the same problem from a different angle.
It should also tell you which pages are already handled. A page that names another URL as its canonical, or that redirects, is a duplicate you have dealt with, and should not be counted with the ones you have not.
Does Duplicate Content Hurt SEO?
Not in the way people often fear. Google's canonicalization documentation explains that when it finds several pages with the same content, it groups them, picks one as the canonical, and crawls the others less often. There is no penalty for having a www and a non-www version of a page. Copying other sites' content to manipulate rankings is a different matter, and Google's spam policies cover it as scraped content.
The real costs are quieter:
- Google may pick the wrong version. The URL in search results can be the one with a parameter, the HTTP one or the print view.
- Signals are split. Links and internal links spread across several URLs instead of adding up on one.
- Crawling is wasted. On a large site, thousands of parameter variants can use up time a search engine would rather spend on new pages. The crawl budget guide covers that side.
- Reports get noisy. Clicks and impressions are spread across URLs, so it is harder to see how a page really performs.
SEO ProCheck separates the myths from the real fixes in duplicate content: causes, myths, and fixes.
The Kinds of Duplicate Content
| Kind | Example | Usual fix |
|---|---|---|
| URL variants | http, www, case, slash | Redirect to one form |
| Parameter variants | ?utm_source=, ?sort=price | Canonical to clean URL |
| Exact duplicates | Two categories, print view | Redirect or canonical |
| Near duplicates | Only the city swapped | Merge, rewrite or canonical |
| Duplicate metadata | One title, many pages | One title per page |
| Copies on other sites | Syndication, scraping | Their noindex or canonical |
The first two kinds are technical and usually fixed once, in the server or the template. The middle two are content decisions. The last one depends on someone else's site.
Where Duplicate Pages Come From
When a checker finds hundreds of duplicates, they rarely come from hundreds of decisions. Look for the pattern:
- Host and protocol. The site answers on both http and https, or both with and without www, and nothing redirects.
- Trailing slashes and case. The server returns 200 for /page and /page/, or for /Page and /page.
- Tracking parameters in internal links. A utm_ tag added to a banner link creates a new URL for the same page.
- Sorting and filtering. Faceted navigation can produce more URLs than there are products. Google's faceted navigation guide asks sites to keep filter parameters in the same order in every URL.
- One item, several paths. A product reachable under /sale/ and /shoes/, or a post under each of its tags.
- Templates with little text. Tag archives, empty category pages and location pages where the template outweighs the content.
- Leftovers. A staging host that is open to crawlers, an old domain that still answers, a print or AMP version.
We found two of these on our own sites. On neova.app, November 20, 2026, http:// answered 200 instead of a 301 to https, so every page had a twin until we fixed it. And Neova's full crawl of seoprocheck.com on October 17, 2026 found:
| seoprocheck.com, Oct 17, 2026 | Count |
|---|---|
| HTML pages answering 200 | 4,561 |
| Duplicate titles | 56 |
| Titles over 60 characters | 557 |
| Meta descriptions over 155 characters | 582 |
| Canonicals to another internal URL | 5 |
Neova groups pages by template, so when duplicates cluster in one page group, you know the template is the place to fix them.
How Exact and Near-Duplicate Detection Works
Exact duplicates are the easy part. The checker extracts the main text of each page, leaving out navigation, header and footer, normalizes it, and compares. Two pages with the same main text are duplicates even if their menus differ.
Near duplicates need a measure of similarity. A common method breaks the text into overlapping runs of words, called shingles, and counts how many runs two pages share:
page A: "red running shoes for trail and road"
5-word runs: "red running shoes for trail", "running shoes for trail and", ...
similarity = runs both pages share / runs on the page
| Setting in Neova | Value |
|---|---|
| Compared text | Main text only |
| Run length (shingle) | 5 words |
| Shared runs to flag | 90% or more |
| Minimum main text | 50 words |
| Pages canonicalized elsewhere | Left out of exact check |
Neova flags two pages when one shares 90% or more of its 5-word sequences with the other, on pages with 50 words or more of main text. No search engine publishes a threshold, so treat a near-duplicate flag as a prompt to look at the two pages side by side, not as a verdict.
Two details make a big difference. Compare main text, not the whole HTML, or every page on a site with a big menu will look alike. And if the content is built by JavaScript, compare the rendered page, or every page will look empty and identical.
How to Check a Site for Duplicate Content, Step by Step
- Crawl the whole site, and turn on JavaScript rendering if the content is built in the browser.
- Start with URL variants. They are the cheapest to fix and often explain most of the rest.
- Review exact duplicates group by group. For each group, choose the URL that should stay.
- Review near duplicates next to each other. Decide whether they serve different needs. A size variant may not; two cities with different services may.
- Scan repeated titles and descriptions. Where the content is different, write new ones. Where it is the same, they confirm a duplicate group.
- Check the canonicals on every duplicate you keep, and make sure they point at the URL you chose. The canonical tag checker guide covers what can go wrong there.
- Crawl again after the fix and compare the two crawls to see which flags were resolved.
How to Fix Duplicate Content
Google's page on consolidating duplicate URLs lists the main ways to tell it which URL you prefer. Redirects and rel=canonical are strong signals; listing a URL in a sitemap is a weaker one. Match the fix to the case:
| Situation | Fix | Signal strength |
|---|---|---|
| Duplicate URL has no purpose | 301 to the preferred URL | Strong |
| Variant must stay reachable | rel=canonical to preferred URL | Strong |
| Two pages, same topic | Merge, redirect the other | Strong |
| Similar pages, different readers | Rewrite each one | n/a |
| One title on many pages | A title per page | n/a |
| Listed in the sitemap only | Add a redirect or canonical | Weak |
| Blocked in robots.txt | Unblock, then canonical | None, page unread |
Whatever you choose, send the same message everywhere. Internal links, the XML Sitemap, hreflang and the canonical should all name the same URL. Google advises against using robots.txt for canonicalization, since a blocked page cannot be read at all. After a redirect, update the links that pointed at the old URL so they do not go through a hop; the redirect chains guide covers that.
For canonicals in depth, SEO ProCheck keeps a complete reference on canonical tags.
Check Duplicate Content With Neova
Neova checks for duplicates in the Content and Indexability categories of every crawl. Pages that already name another URL as canonical are left out of the exact duplicate check, so that list shows what is still open. The checks behind it:
- URL Variants Return the Same HTML, for protocol, www, case, trailing slash, parameter order and tracking parameters
- Trailing Slash Variant Redirects or Duplicates the URL and Letter Case Variant Redirects or Duplicates the URL
- Same Query Parameters in a Different Order
- Duplicate Content Across HTML Pages, after JavaScript rendering when the crawl rendered the pages
- Near-Duplicate Content Across HTML Pages
- Duplicate Title Across HTML Pages, Duplicate Meta Description Across HTML Pages and Duplicate H1 Across HTML Pages
- Syndicated Copy Has No Noindex or Canonical to the Original
- Indexable URL Uses Search or Filter Parameters and Indexable URL Uses Sort Parameters

Page Groups show which template the duplicates come from, and Compare Crawls shows which flags were resolved after a fix. Every table exports to CSV. See Features or download Neova; the Free plan runs every check.
Questions
What is a duplicate content checker?
A tool that compares the pages of a site and reports the ones that return the same or very similar content, including URL variants of one page and titles or descriptions repeated across pages.
Does Google penalize duplicate content?
Not duplicates within your own site. Google groups them, picks one URL as the canonical and shows that one. The risk is that it picks a version you did not want and that signals are split. Content copied from other sites to manipulate rankings falls under Google's spam policies.
How similar do pages have to be to count as duplicates?
No search engine publishes a percentage. Neova flags near duplicates when one page shares 90% or more of its 5-word sequences with another, as a prompt to compare them, not a ruling.
Should I use a canonical or a redirect for duplicate pages?
Redirect when the duplicate URL does not need to exist. Use a canonical when it must stay reachable for visitors, such as a sorted list or a URL with a tracking parameter.
Can I block duplicate pages in robots.txt?
Google advises against it. A blocked URL cannot be read, so Google cannot see its canonical or that it is a duplicate, and it may still index the URL from links.