NeovaAdvanced SEO Platform

Duplicate Content Checker: How to Find and Fix Duplicate Pages

Most duplicate content is not copied on purpose. It comes from the way a site builds URLs: the same page with and without www, with a tracking parameter, sorted a different way, or listed under two categories. Search engines then have to guess which version you mean. A duplicate content checker finds every copy, groups them, and shows which one should stay.

On-Page Contentduplicate content checkerUpdated November 21, 2026By the Neova team

What a Duplicate Content Checker Does

A duplicate content checker compares the pages of a site with each other and reports the ones that are the same or nearly the same. A good one looks at four things:

  • URLs that differ only in protocol, host, letter case, trailing slash or parameters but return the same page.
  • Main text that is identical on two or more pages.
  • Main text that is very similar, with a few words, a product name or a city swapped.
  • Titles, meta descriptions and headings repeated across pages, which often point to the same problem from a different angle.

It should also tell you which pages are already handled. A page that names another URL as its canonical, or that redirects, is a duplicate you have dealt with, and should not be counted with the ones you have not.

Does Duplicate Content Hurt SEO?

Not in the way people often fear. Google's canonicalization documentation explains that when it finds several pages with the same content, it groups them, picks one as the canonical, and crawls the others less often. There is no penalty for having a www and a non-www version of a page. Copying other sites' content to manipulate rankings is a different matter, and Google's spam policies cover it as scraped content.

The real costs are quieter:

  • Google may pick the wrong version. The URL in search results can be the one with a parameter, the HTTP one or the print view.
  • Signals are split. Links and internal links spread across several URLs instead of adding up on one.
  • Crawling is wasted. On a large site, thousands of parameter variants can use up time a search engine would rather spend on new pages. The crawl budget guide covers that side.
  • Reports get noisy. Clicks and impressions are spread across URLs, so it is harder to see how a page really performs.

SEO ProCheck separates the myths from the real fixes in duplicate content: causes, myths, and fixes.

The Kinds of Duplicate Content

KindExampleUsual fix
URL variantshttp, www, case, slashRedirect to one form
Parameter variants?utm_source=, ?sort=priceCanonical to clean URL
Exact duplicatesTwo categories, print viewRedirect or canonical
Near duplicatesOnly the city swappedMerge, rewrite or canonical
Duplicate metadataOne title, many pagesOne title per page
Copies on other sitesSyndication, scrapingTheir noindex or canonical

The first two kinds are technical and usually fixed once, in the server or the template. The middle two are content decisions. The last one depends on someone else's site.

Where Duplicate Pages Come From

When a checker finds hundreds of duplicates, they rarely come from hundreds of decisions. Look for the pattern:

  • Host and protocol. The site answers on both http and https, or both with and without www, and nothing redirects.
  • Trailing slashes and case. The server returns 200 for /page and /page/, or for /Page and /page.
  • Tracking parameters in internal links. A utm_ tag added to a banner link creates a new URL for the same page.
  • Sorting and filtering. Faceted navigation can produce more URLs than there are products. Google's faceted navigation guide asks sites to keep filter parameters in the same order in every URL.
  • One item, several paths. A product reachable under /sale/ and /shoes/, or a post under each of its tags.
  • Templates with little text. Tag archives, empty category pages and location pages where the template outweighs the content.
  • Leftovers. A staging host that is open to crawlers, an old domain that still answers, a print or AMP version.

We found two of these on our own sites. On neova.app, November 20, 2026, http:// answered 200 instead of a 301 to https, so every page had a twin until we fixed it. And Neova's full crawl of seoprocheck.com on October 17, 2026 found:

seoprocheck.com, Oct 17, 2026Count
HTML pages answering 2004,561
Duplicate titles56
Titles over 60 characters557
Meta descriptions over 155 characters582
Canonicals to another internal URL5

Neova groups pages by template, so when duplicates cluster in one page group, you know the template is the place to fix them.

How Exact and Near-Duplicate Detection Works

Exact duplicates are the easy part. The checker extracts the main text of each page, leaving out navigation, header and footer, normalizes it, and compares. Two pages with the same main text are duplicates even if their menus differ.

Near duplicates need a measure of similarity. A common method breaks the text into overlapping runs of words, called shingles, and counts how many runs two pages share:

page A: "red running shoes for trail and road"
5-word runs: "red running shoes for trail", "running shoes for trail and", ...
similarity = runs both pages share / runs on the page
Setting in NeovaValue
Compared textMain text only
Run length (shingle)5 words
Shared runs to flag90% or more
Minimum main text50 words
Pages canonicalized elsewhereLeft out of exact check

Neova flags two pages when one shares 90% or more of its 5-word sequences with the other, on pages with 50 words or more of main text. No search engine publishes a threshold, so treat a near-duplicate flag as a prompt to look at the two pages side by side, not as a verdict.

Two details make a big difference. Compare main text, not the whole HTML, or every page on a site with a big menu will look alike. And if the content is built by JavaScript, compare the rendered page, or every page will look empty and identical.

How to Check a Site for Duplicate Content, Step by Step

  1. Crawl the whole site, and turn on JavaScript rendering if the content is built in the browser.
  2. Start with URL variants. They are the cheapest to fix and often explain most of the rest.
  3. Review exact duplicates group by group. For each group, choose the URL that should stay.
  4. Review near duplicates next to each other. Decide whether they serve different needs. A size variant may not; two cities with different services may.
  5. Scan repeated titles and descriptions. Where the content is different, write new ones. Where it is the same, they confirm a duplicate group.
  6. Check the canonicals on every duplicate you keep, and make sure they point at the URL you chose. The canonical tag checker guide covers what can go wrong there.
  7. Crawl again after the fix and compare the two crawls to see which flags were resolved.

How to Fix Duplicate Content

Google's page on consolidating duplicate URLs lists the main ways to tell it which URL you prefer. Redirects and rel=canonical are strong signals; listing a URL in a sitemap is a weaker one. Match the fix to the case:

SituationFixSignal strength
Duplicate URL has no purpose301 to the preferred URLStrong
Variant must stay reachablerel=canonical to preferred URLStrong
Two pages, same topicMerge, redirect the otherStrong
Similar pages, different readersRewrite each onen/a
One title on many pagesA title per pagen/a
Listed in the sitemap onlyAdd a redirect or canonicalWeak
Blocked in robots.txtUnblock, then canonicalNone, page unread

Whatever you choose, send the same message everywhere. Internal links, the XML Sitemap, hreflang and the canonical should all name the same URL. Google advises against using robots.txt for canonicalization, since a blocked page cannot be read at all. After a redirect, update the links that pointed at the old URL so they do not go through a hop; the redirect chains guide covers that.

For canonicals in depth, SEO ProCheck keeps a complete reference on canonical tags.

Check Duplicate Content With Neova

Neova checks for duplicates in the Content and Indexability categories of every crawl. Pages that already name another URL as canonical are left out of the exact duplicate check, so that list shows what is still open. The checks behind it:

All Flags: each check with the URLs it flagged out of those checked, here with Duplicate Content Across HTML Pages and Duplicate Title Across HTML Pages among the Must Fix checks.
All Flags: each check with the URLs it flagged out of those checked, here with Duplicate Content Across HTML Pages and Duplicate Title Across HTML Pages among the Must Fix checks.

Page Groups show which template the duplicates come from, and Compare Crawls shows which flags were resolved after a fix. Every table exports to CSV. See Features or download Neova; the Free plan runs every check.

Questions

What is a duplicate content checker?

A tool that compares the pages of a site and reports the ones that return the same or very similar content, including URL variants of one page and titles or descriptions repeated across pages.

Does Google penalize duplicate content?

Not duplicates within your own site. Google groups them, picks one URL as the canonical and shows that one. The risk is that it picks a version you did not want and that signals are split. Content copied from other sites to manipulate rankings falls under Google's spam policies.

How similar do pages have to be to count as duplicates?

No search engine publishes a percentage. Neova flags near duplicates when one page shares 90% or more of its 5-word sequences with another, as a prompt to compare them, not a ruling.

Should I use a canonical or a redirect for duplicate pages?

Redirect when the duplicate URL does not need to exist. Use a canonical when it must stay reachable for visitors, such as a sorted list or a URL with a tracking parameter.

Can I block duplicate pages in robots.txt?

Google advises against it. A blocked URL cannot be read, so Google cannot see its canonical or that it is a duplicate, and it may still index the URL from links.

Take this with you

The assistant links open with a question about this page already written.

Crawl your site with NeovaThe Free plan runs every check on Mac, Windows and Linux. No credit card required.

Keep Reading