NeovaAdvanced SEO Platform

How to Crawl a Website for SEO, Step by Step

Crawling a website means fetching its pages the way a search engine does, following links from page to page. Here is how to crawl a website properly: politely, completely and without getting blocked.

Crawling and Indexinghow to crawl a websiteUpdated November 21, 2026By the Neova team

What Crawling a Website Means

A website crawler for SEO starts at one URL, downloads it, reads the links in the HTML and adds them to a queue. Then it takes the next URL from the queue and does the same, until nothing new turns up or it hits a limit. Search engines do this at web scale. An SEO crawler does it for one site, and records what each URL answered so you can audit it.

For every URL, a crawl typically records the status code, the response headers, the title and meta description, the canonical, robots directives, the links in and out, and how many clicks it sits from the start page. Google describes how its own crawlers and fetchers work in its overview of Google crawlers.

Before You Crawl: Permission, robots.txt and Scope

Three questions to answer before you press start:

  • Is it your site, or do you have permission? Crawling your own site or a client's is routine. For anyone else's site, stay polite: respect robots.txt and keep the speed low.
  • What does robots.txt say? Open /robots.txt and read it. It tells crawlers what they may not fetch, and usually lists the XML Sitemaps. The rules follow RFC 9309, and Google explains how it reads them in its robots.txt introduction. The complete robots.txt reference on SEO ProCheck goes line by line.
  • How much do you need? The whole site, one section, or a sample of each template. A sample answers most template questions in minutes; a full crawl is what you need for broken links and orphan pages.

How to Crawl a Website Step by Step

  1. Find the right start URL. Request http and https, with and without www. One should answer 200 and the others should redirect to it. Start there.
  2. Choose a mode. Spider mode follows links from the start URL. List mode crawls only the URLs you give it, which is useful for checking a set of old URLs after a migration.
  3. Add the XML Sitemaps. Crawling the sitemap URLs after the linked pages shows which pages the site lists but never links to.
  4. Set limits. A URL limit, a depth limit and rules for query parameters keep filters from running away with the crawl. The faceted navigation SEO guide on SEO ProCheck shows which filter URLs deserve crawling.
  5. Exclude what you do not need. Admin areas, cart and checkout URLs, internal search results. Use include and exclude patterns, and test a URL against them before you start.
  6. Pick a user agent. Your crawler's own name is the honest default. Crawling as Googlebot shows what Google is served, but some servers treat a fake Googlebot as an attack.
  7. Start, then watch the first few minutes. If the count of new URLs climbs much faster than the site's size, look at where they come from. Calendars and endless filters are the usual cause.
  8. Let the checks run. When the crawl ends, start with the issues that touch the most indexable pages.
New Crawl, Speed and Rendering: the speed in URLs a second, reduced when the server slows down, the user agent sent with every request, and JavaScript rendering
New Crawl, Speed and Rendering: the speed in URLs a second, reduced when the server slows down, the user agent sent with every request, and JavaScript rendering

How to Crawl a Website for All URLs

Following links finds only the pages that links reach. To crawl a website for all URLs, combine several sources:

SourceLinked pagesUnlinked pagesComes from
Link crawl✓–The start URL
XML Sitemaps◐✓robots.txt or /sitemap.xml
Seed URLs◐✓Your own list
Server logs◐✓Server or CDN
Search Console◐✓Your Google property

We found how much this matters on our own site. In the full Neova crawl of seoprocheck.com on October 17, 2026:

Measureseoprocheck.com
URLs fetched10,222
Time61 minutes
HTML pages answering 2004,561
URLs in the XML Sitemaps1,835
Reached by links, in no sitemap2,727
In a sitemap, reached by no link1
Completeness99.98%
Page bodies1.66 GB raw, 340 MB stored

A sitemap-only audit would have missed 2,727 URLs, and a link-only crawl would have missed the one orphan. Pages that come only from sitemaps, logs or Search Console are orphans. Indexable orphans that get search traffic usually need internal links. Checks worth knowing here: Orphan URL Returns 200 and Is Indexable, Indexable URL Is Not in Any XML Sitemap and XML Sitemap Cannot Be Fetched or Parsed.

How to Crawl a Website Without Getting Blocked

Most blocks come from crawling too fast or looking like a bot the server does not trust. Crawl the way the site owner would want:

  • Keep the speed low. 2 or 3 URLs a second is enough for most audits. Faster only on your own site.
  • Say who you are. A clear user agent with a link to a page about the crawler is easier to allow than a fake browser string.
  • Respect robots.txt. It is the site's stated wish, and ignoring it on someone else's site is the fastest way to get blocked.
  • Ask to be allowed in. On a client site behind a firewall or bot protection, ask the IT team to allow your crawler, for example by a secret header only your crawler sends.

Then read the answers. Each status code says something different:

AnswerMeansDo
429Too many requestsHalve speed, honor Retry-After
503Server busy or downSlow down, retry later
403RefusedStop and ask
401Login requiredAsk for access
5xx runServer strugglingPause the crawl

Google's page on HTTP status codes and network errors explains how Googlebot itself treats 429 and 5xx answers.

Crawling JavaScript Websites

If the site is built with React, Vue, Angular or a similar framework, the HTML the server sends may be almost empty, with the content added by scripts. A plain crawl then finds few links and little text. Turn on JavaScript rendering, which loads each page in a headless browser, and compare the rendered page with the raw HTML. What matters is what is missing before rendering: main text, links, titles and canonicals that only appear after scripts run. Rendering is slower, so render a sample of each template first.

Crawl My Website With Neova

To crawl your own website with Neova, download the app, then choose Crawl, New Crawl, type the address and click Start Crawl. The Your First Crawl page shows each screen.

SettingDefault
Runs on
User agentNeovaBot
Speed3 URLs a second
On 429 or 503Halves speed, honors Retry-After
robots.txtRespected
CAPTCHANever solved
Free plan1,000 URLs a crawl
  • Start URL Check picks the version of the site (http or https, with or without www) that answers 200 and says what the others do.
  • Modes and scope: Spider or List, Full Site, Specific Folders or Sample per Page Group, plus XML Sitemaps, seed URLs and include and exclude rules.
  • Your own site: once you confirm it is yours, Neova can go faster and ignore robots.txt. The Allowlist Header makes a secret header sent only to that site, with the rule to hand the IT team.
  • Blocked Crawl Explained: when a site refuses the crawl, Neova says why (rate limited, blocked by a firewall, forbidden) and what to do next.
  • Discovery and URL Breakdown: while it runs, every URL found lands in one bucket and crawl trap shapes are flagged, with Skip or Exclude on the spot.
  • Pause, Stop and Resume: stop when you like and run the checks on what was crawled, or Resume later from where it stopped.

Site owners can read how the crawler identifies itself, and how to block it, on the NeovaBot page.

What to Do With the Crawl

A crawl is a snapshot. Its value is in what you fix. Sort the findings by how many indexable pages they touch, fix template issues once rather than page by page, and crawl again to confirm. A short first pass that usually pays off:

The technical SEO audit checklist puts the whole review in order.

Questions

How do I crawl a website for all URLs?

Combine a link crawl with the XML Sitemaps, then add URLs from Google Search Console and the server logs. Any page that only those sources find, and no internal link reaches, is an orphan worth a look.

How do I crawl a website without getting blocked?

Crawl slowly, use an honest user agent, respect robots.txt and slow down when the server answers 429 or 503. On a client site behind a firewall, ask the IT team to allow your crawler instead of trying to get around the block.

Is it legal to crawl a website?

Crawling public pages is common practice, but rules vary by country and by the site's terms. Crawl your own and your clients' sites freely, respect robots.txt everywhere, and read the site's terms when in doubt. This is not legal advice.

How long does it take to crawl a website?

Roughly the number of URLs divided by the speed. Our full crawl of seoprocheck.com on October 17, 2026 fetched 10,222 URLs in 61 minutes, about 2.8 URLs a second. At 3 URLs a second, 1,000 URLs take about 6 minutes and 100,000 about 9 hours, before rendering. Sampling each template is much faster.

Can I crawl my website for free?

Yes. Neova's Free plan crawls up to 1,000 URLs per crawl with every check, on Mac, Windows and Linux. See pricing for the larger plans.

Take this with you

The assistant links open with a question about this page already written.

Crawl your site with NeovaThe Free plan runs every check on Mac, Windows and Linux. No credit card required.

Keep Reading