Skip to content
SBMM Cloud
Sign inGet started free

Free Content Duplicate Checker

Find near-duplicate pages on your site that may be triggering Helpful Content penalties.

What a run costs
One free run a day, no account needed
How it runs
In the background
Who it is for
Free: 2 a day. Silver: 5 a day. Gold: 10 a day. Ultimate: unlimited.

A free account raises the daily allowance and keeps your results. Create one — no card needed.

Loading…

Overview

Internal content duplication is a known trigger for the Google Helpful Content Update. When the same paragraph or page appears at multiple URLs, Google has to pick a canonical, and the choice is often wrong. This free duplicate content checker crawls up to 30 pages of your site and surfaces every near duplicate pair.

How It Detects Duplicates

The crawler shingles every page body into rolling five word phrases and compares the shingle sets between every URL pair. Pages that share more than the threshold percentage of shingles are flagged as near duplicates with a similarity score. Boilerplate content like footer, sidebar, and navigation is detected and separated so it does not skew the report.

How To Fix What It Finds

For exact duplicates, pick one canonical URL and 301 redirect the others. For near duplicates that are intentional (product variants, archive pages), add a rel canonical tag pointing to the primary version. For thin or low value duplicates, noindex or merge them into a richer page. Re run the audit after each fix to confirm the count drops.

About the tool

What Is The SBMM Content Duplicate Checker?

The SBMM Content Duplicate Checker is a free internal duplicate content scanner that crawls up to 20 pages of your site, shingles each page body into rolling five-word phrases, compares every URL pair, and surfaces every near-duplicate match with a similarity score. It is the fastest way to spot the kind of content overlap Google's Helpful Content Update penalises.

Internal content duplication is one of the clearest signals Google uses to demote a site under the Helpful Content Update. When the same paragraph or page appears at multiple URLs, Google has to pick a canonical, and the choice is often wrong. The pages compete with each other for ranking, split the link equity, and confuse the rater grading the helpfulness of your content set.

This checker finds the duplicates before Google does. It detects exact duplicates, near duplicates (a product variant page that copies most of its content from the master), boilerplate bleed (footer, sidebar, or template text dominating the body), and thin-content overlap. Every flagged pair shows you exactly which shingles match so you can verify the report and pick the right canonical or merge target.

Step by step

How To Use This Tool In 3 Steps

  1. Enter Your Website URL

    Drop the homepage URL of any public website you own or have permission to crawl. The scanner respects robots.txt and limits itself to 20 pages without an account, so it never overloads your origin.

  2. Shingle + Compare Every Pair

    The crawler fetches each page, strips boilerplate (header, footer, sidebar, navigation), shingles the body into rolling five-word phrases, and compares the shingle sets between every URL pair using a Jaccard similarity index.

  3. Fix What Is Flagged

    See every duplicate pair sorted by similarity score, with a list of matched shingles per pair. Choose to 301 redirect, canonicalise, noindex, or merge the duplicate. Re-run the audit after fixing to confirm the count drops.

Why this tool

Why Use This Tool

  • 5-Word Shingle Fingerprinting

    Each page body is shingled into rolling five-word phrases that capture both exact and near-duplicate overlap. Five-word shingles are the standard duplicate-content detection unit used by Google's own internal deduper.

  • Boilerplate Detection

    The scanner identifies and excludes repeating template content (header, footer, sidebar, nav, related-posts widgets) so duplicate detection focuses on real body content, not the chrome that legitimately repeats.

  • Similarity Score Per Pair

    Every flagged pair gets a 0 to 100 similarity score. Pairs above 70 are exact-duplicate candidates; 40 to 70 are near-duplicate candidates worth canonicalising; below 40 is usually safe overlap.

  • Crawls Up To 20 Pages Free

    Twenty pages without an account, which catches duplicate patterns on small to mid-sized sites and gives a clean directional signal on larger ones. Signed in, the crawl goes to 50 pages.

  • HCU-Aligned Reporting

    The findings are organised the way Google's Helpful Content rater handbook reads: thin content overlap, near-duplicate product variants, paginated archives, and tag-archive bloat all surface as distinct categories.

  • Free, One Run A Day

    One free run of this tool a day without an account. A free account gives you two runs a day of this tool, keeps your results and lets you download them. Paid plans raise the allowance further.

FAQ

Frequently Asked Questions

What is duplicate content in SEO?

Duplicate content is text that appears at more than one URL, either within a single domain (internal duplication) or across multiple domains (external duplication). Internal duplication forces Google to pick a canonical, splits link equity between competing URLs, and is a known Helpful Content Update demotion trigger.

Will Google penalise me for duplicate content?

Google does not issue a formal penalty for duplicate content, but it actively demotes pages and sites with significant internal duplication under the Helpful Content Update. The practical effect is identical to a penalty: the duplicate pages stop ranking, and in severe cases the whole subfolder or domain loses visibility.

What is the difference vs Copyscape or Siteliner?

Copyscape is built for external plagiarism detection (someone else copying your content). Siteliner is the closest equivalent to this tool but is paid for serious crawls. The SBMM Duplicate Checker focuses on internal duplication, runs free for small sites, and integrates with the rest of the SBMM SEO toolkit.

How does boilerplate detection work?

The scanner identifies blocks of HTML that repeat verbatim across multiple pages (header, footer, sidebar, navigation, related-posts widgets) and excludes them from the shingle comparison. This prevents the legitimate template from triggering false duplicate alerts in the body content audit.

How do I fix near-duplicate pages?

For exact duplicates, pick one canonical URL and 301 redirect the others. For intentional near-duplicates (product variants, archive pages), add a rel canonical tag pointing to the primary version. For thin or low-value duplicates, noindex them or merge their content into a richer page. Re-run the audit after each fix to confirm the count drops. Run our Site Audit Pro afterward to confirm the canonical chain is clean and the redirects do not loop.

When should I use canonical instead of 301?

Use rel canonical when both URLs need to stay reachable for users (product variants by colour, paginated archives, tracking-parameter URLs) but only one should rank. Use a 301 redirect when the duplicate URL has no reason to exist (an old slug, a content merge, a category rename). 301 passes equity; canonical concentrates it.

Does pagination count as duplicate content?

Paginated archives are not duplicate content in the strict sense (each page has a different URL list) but they often share boilerplate intro text that can trigger soft-duplicate flags. The scanner detects pagination patterns and separates them from real duplicate cases in the report.

Can I scan a site I do not own?

Technically yes (the scanner only fetches public URLs the same way Google does) but practically you should only audit sites you own or have written permission to audit. Respect the target site's robots.txt, crawl-rate signals, and bandwidth.