Technical SEO Field Reference, Question 02 of 66. The interview answer, checked against what Google currently documents.
What is Google crawling? Google crawling is the first stage of Search: Googlebot downloads pages it has found so they can be indexed and served. Whether and how often a URL gets crawled is influenced by three groups of factors: how Google discovers the URL (links from known pages, sitemaps, previous visits), how Google schedules crawling (an algorithmic process weighing demand against your server’s capacity), and whether anything blocks access (robots.txt, login walls, server errors, rendering failures) (Google Search Central, last updated 18 December 2025).
Most candidates answer this question with one word: links.
Links are a factor. But an interviewer asking this wants to know whether you can see the whole pipeline: how a URL becomes known to Google at all, what decides its place in the crawl queue, and what silently takes it out of the running before crawling even starts.
The strongest answers organise the factors instead of listing them.
What Google actually says about crawling
Google describes crawling in its in-depth guide to how Search works, and the factors split into two stages.
Stage 1: URL discovery. There is no central registry of web pages, so Google constantly looks for new and updated pages. A URL becomes known because Google has visited it before, or because Google extracts a link to it from a known page. Sitemaps are the third documented path: you hand Google the list directly.
Stage 2: crawl scheduling and fetching. Once a URL is known, Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site. Two documented modifiers:
- Googlebot deliberately avoids overloading your server, slowing down based on the site’s responses (HTTP 500-class errors mean “slow down”) and Search Console settings. This is the crawl capacity side.
- Demand: how much Google wants to crawl your URLs, weighing perceived inventory, popularity, and staleness. Capacity plus demand is crawl budget, covered in the full breakdown.
The access gates. The documentation is explicit that Googlebot does not crawl every page it discovers. Documented reasons: the owner disallows crawling (robots.txt), the page requires login, and pages that are unreachable. Rendering belongs here too: Google renders pages and runs JavaScript during crawling, and without successful rendering Google might not see the content.
| Factor group | What it controls | Documented inputs | Where you check it |
|---|---|---|---|
| Discovery | Whether Google knows the URL exists | Internal links, external links, sitemaps, previous visits | URL Inspection, sitemap reports, server logs |
| Scheduling | When and how often Google crawls it | Crawl demand (popularity, staleness, inventory) + crawl capacity (server health) | Crawl Stats report, server logs |
| Access | Whether anything blocks the crawl | robots.txt, login walls, server errors (5xx), rendering failures | robots.txt tester, URL Inspection, server logs |
The interview answer
Crawling is influenced by three groups of factors. Discovery: Google has to find the URL first, through links from pages it already knows, sitemaps, or previous visits. Scheduling: an algorithmic process decides which sites to crawl, how often, and how deeply, balancing Google’s interest in the URLs against what the server can handle. Access: robots.txt rules, login walls, server errors, and rendering failures can take a discovered URL out of the running entirely. Crawl budget is the scheduling piece, not the whole answer.
How often does Google crawl a website?
There is no fixed interval. Crawl frequency is per-URL, driven by the demand factors: URLs Google perceives as popular or frequently changing get recrawled more; stale or low-interest URLs can go weeks or months. The Crawl Stats report in Google Search Console shows your site’s actual request volume per day, and that number is the answer for your site.
Real-world example
A publisher launches a new content section: 400 articles, live, indexable, in the XML sitemap. Three weeks later, almost none have been crawled.
The instinct is “crawl budget problem.” The logs say otherwise: Googlebot is crawling the rest of the site at its normal rate. The new section’s articles are linked only from each other. The section hub is reachable solely from a JavaScript-rendered carousel on the homepage that outputs no crawlable links in the served HTML.
Diagnosis path:
- Server logs: is Googlebot requesting the new URLs at all? (No, meaning a discovery problem, not scheduling.)
- Rendered vs served HTML on the homepage: does the carousel produce crawlable links?
- GSC URL Inspection on a sample article: “Discovered, currently not indexed” or entirely unknown?
- Sitemap check: submitted, fetched, no errors, which proves sitemaps alone were not enough to drive crawling here.
Fix: one crawlable HTML link block from the homepage and category pages to the section hub. Measure: Googlebot requests to the new URL pattern in the logs over the following two weeks.
The technical detail: how Google crawling works in practice
The practical hierarchy of discovery, as observed in logs: internal links first, sitemaps second. In practice sitemaps reliably make URLs known but are weak at making them crawled. A sitemap-only URL with no inbound links is exactly the pattern behind the example above.
Depth matters observationally too: URLs many clicks from strong pages tend to be crawled less. No documented click-depth threshold exists, so do not state “three clicks” as a rule.
Where you measure all of this: GSC Crawl Stats for the aggregate, server access logs for per-URL-pattern truth, URL Inspection for individual state.
The interview trap: what is Google crawling really about?
The one-word answer: “links.” Links are one discovery input, and discovery is one of three factor groups. An answer that stops at links misses scheduling entirely and misses every gate that removes a URL before crawling starts.
The subtler trap: “submit a sitemap and Google will crawl it.” Documented behaviour is that sitemaps aid discovery; nothing in the documentation promises crawling of discovered URLs, and the documentation states outright that Googlebot does not crawl all the pages it discovers. Sitemaps make URLs known. They do not make URLs wanted.
Takeaway
Wrong question: “How do I get Google to crawl more?” Better question: “Can Google find it, is anything blocking it, and has Google been given a reason to want it?”
Have questions about this topic? Contact The GEO Lab · Return to homepage
Have questions about this topic? Contact The GEO Lab · Return to homepage