Skip to content
← All posts
By the Numbers

What Does Google See When It Crawls a Contractor’s Site?

What Does Google See When It Crawls a Contractor's Site?

Mentioned in this postGoogle

Google does not see your design. It reads HTML, twice: the raw file your server sends and the page after JavaScript runs. Every figure below covers the whole measured web, about 12.9 million sites in HTTP Archive’s July 2025 crawl, mobile pages; nobody publishes them for contractors alone.

What does Google actually see when it crawls your site?

Code, not pixels. Google crawls, renders the page in a headless browser, then indexes the rendered HTML (Google Search Central, 2026).

94.1%
of sites serve a robots.txt file
Web Almanac, 2025
16%
of home page words appear after rendering
Web Almanac, 2025
2%
add structured data by JavaScript
Web Almanac, 2025
One crawl, in order

Google Search Central, 2026.

View data
StepValue
Request01 (Googlebot asks for the URL)
robots.txt02 (may it crawl this path?)
Raw HTML03 (what your server sends)
Render04 (a headless browser runs the JavaScript)
Index05 (the rendered page is read and stored)

What happens before Google reads a single word?

It asks for robots.txt. Almost every site answers; the risk is a file whose rules nobody has read, because a 200 only proves the file exists.

What robots.txt requests return, mobile sites

HTTP Archive, Web Almanac 2025, SEO chapter · July 2025 crawl, mobile

View data
HTTP Archive, Web Almanac 2025, SEO chapter · July 2025 crawl, mobile
ItemValue
200 success84.9%
404 not found13.2%
Timeout1.1%
403 forbidden0.5%
500 server error0.1%
94.1%
serve robots.txt with a directive
Web Almanac, 2025
77.1%
of files address every bot at once
Web Almanac, 2025
1.7%
serve a completely empty file
Web Almanac, 2025
A file with rules

Fine, if someone has read the rules. A 200 response confirms the file exists, not that its directives are right for the site.

A clean 404

Also fine. It tells crawlers there are no restrictions, and the chapter calls it the safer choice when none are intended.

An empty file

The risky middle. Major crawlers read it as permissive, but the standard (RFC 9309) never defined it, so lesser-known bots may not.

Which signals does Google look for in the head?

The title is nearly universal. The meta description and the canonical are missing on about a third of mobile pages.

Share of mobile pages with each tag

HTTP Archive, Web Almanac 2025, SEO chapter · July 2025 crawl, mobile

View data
HTTP Archive, Web Almanac 2025, SEO chapter · July 2025 crawl, mobile
ItemValue
Title98.5%
Viewport95.2%
Meta description67.2%
Canonical67%
Title

Google’s main source for the title link, which it can rewrite when the tag does not match the page (Google Search Central, 2026). Present on 98.5% of mobile pages.

Meta description

The snippet you propose for the result. Present on 67.2% of mobile pages, so missing on about a third.

Canonical

A strong hint, not a directive, about which URL is the main one. Present on 67% of mobile pages.

A valid head

10.3% of mobile pages carry an invalid element in the head, most often an image (23.0% of mobile sites). A title or canonical below it can stop counting.

What is missing most often on a typical page?

Headings and alt text. The h1 is absent on 3 in 10 mobile pages, and the median page describes only 60% of its images.

Share of pages using each heading level

HTTP Archive, Web Almanac 2025, SEO chapter · July 2025 crawl, mobile

View data
HTTP Archive, Web Almanac 2025, SEO chapter · July 2025 crawl, mobile
ItemValue
h170%
h271%
h360%
h436%

Structured data helps Google understand a page, though the chapter itself says it is not a direct ranking factor.

60%
of images carry alt, median page
Web Almanac, 2025
15%
of images carry a blank alt
Web Almanac, 2025
50%
of home pages have structured data
Web Almanac, 2025

Does Google see the content your JavaScript loads?

It renders it, but the raw HTML is the safe place. On the median mobile home page, 16% of the visible words appear only after rendering.

Median visible words per page, mobile

HTTP Archive, Web Almanac 2025, SEO chapter · July 2025 crawl

View data
HTTP Archive, Web Almanac 2025, SEO chapter · July 2025 crawl
ItemValue
Home page, raw HTML316
Home page, rendered378
Inner page, raw HTML271
Inner page, rendered323

Part of that gap is content hidden in mobile accordions, present in the page but not shown.

16%
of home page words appear after rendering
Web Almanac, 2025
2%
add structured data by JavaScript
Web Almanac, 2025
3.02%
change the canonical when rendered
Web Almanac, 2025
Words that matter

Put the service, the city and the phone number in the HTML the server sends, not only in what a script loads later.

Structured data

Only about 2% of crawls add it by JavaScript. Google can process injected schema, but the initial HTML avoids potential delays.

Canonical

3.02% of mobile sites change it during rendering, which sends Google two different answers about the main URL.

What should a contractor fix first?

Anything that blocks crawling. Google’s own SEO checker weights that one check about four times any other, so failing it alone fails the category.

  1. 01Do not block it
  2. 02Name it: title and description
  3. 03Point it: canonical and h1
  4. 04Describe it: alt text and structured data

Page isn’t blocked from indexing

blockedcrawlable

Weight 93/23, about 4.04 against 1 for each of the other nine (Lighthouse, 2026).

Document has a meta description

missing on about a thirdpresent

67.2% of mobile pages have one.

Document has a valid rel=canonical

missing on about a thirdpresent

67% of mobile pages have one.

Images have alt text

60% on the median pageevery image

15% of images carry a blank alt.

View data
MetricDay zero → target
Page isn’t blocked from indexingblocked → crawlable
Document has a meta descriptionmissing on about a third → present
Document has a valid rel=canonicalmissing on about a third → present
Images have alt text60% on the median page → every image

That weight works out to about 31% of the category score, our arithmetic on the published weights. For the rest of the checklist, see what a website audit actually checks; for speed, how fast a service business website needs to load.

Questions owners ask

What does Googlebot actually read on my site?

The HTML, not the design. It asks for robots.txt first, then reads the raw HTML your server sends, then the page after JavaScript runs. On the median mobile home page those two versions differ by 16% of the visible words (Web Almanac, 2025).

Why isn’t my new site showing up in Google?

Start with whether anything blocks it. Google’s own SEO checker weights the is-crawlable check about four times every other check, by design, so one blocked page fails the whole category. Then check the title, the meta description and the canonical.

Does my site need a robots.txt file?

94.1% of crawled sites serve one with at least one directive. A valid file or a clean 404 is fine. An empty file is the risky middle: the standard never defined how bots should read it, and 1.7% of mobile sites serve one anyway.

Is alt text an SEO thing or an accessibility thing?

Both, and it is one of the thinnest signals on a typical page: on the median mobile page only 60% of images carry an alt attribute, and 15% carry a blank one (Web Almanac, 2025).

Sources

  1. HTTP Archive, Web Almanac 2025, SEO chapter (July 2025 crawl; mobile pages; read from the chapter's source text, commit 46992bf, October 2026) (2025)
  2. HTTP Archive, Web Almanac 2025, Generative AI chapter (~12.9 million sites analyzed for robots.txt) (2025)
  3. Google Lighthouse, SEO category audit weights (open-source code, core/config/default-config.js, commit 1c45a6f, October 2026) (2026)
  4. Google Search Central: Understand the JavaScript SEO basics (2026)
  5. Google Search Central: Influencing your title links in search results (2026)

The first meeting starts with findings, not promises.The free Business Health Report reads your business and shows you what it found, in days.

Get Your Free Business Health Report
← All posts