Craig Campbell SEO Training & Consultancy

Tutorials

Log file analysis for SEO: see what crawlers actually requested

By Craig Campbell · 15 Nov 2026

4 min read

SEO log file analysis reviews recorded requests to understand which URLs crawlers visited and what responses they received. It answers a different question from a crawl tool: what happened, not just what could be discovered now.

Logs can help investigate persistent errors or large groups of unwanted URLs. For a small healthy site, simpler Search Console checks may be enough.

This guide uses Northgate Property Listings, a fictional Edinburgh website. All counts, URLs and findings are illustrative only, not Craig’s client work or a real log analysis. Sources checked 15 November 2026.

1. Check what the logs actually cover

Ask the hosting owner for access logs covering the relevant hostnames and a useful period. Establish whether they come from the CDN, web server or another layer.

  • Record the time zone and date range.
  • Check whether requests are sampled or missing.
  • Confirm whether cached requests appear.
  • Keep URL, time, response status and crawler-identification evidence where available.

Origin logs alone may miss requests answered by the CDN. Do not call an export complete until its coverage is understood.

2. Protect the data before sharing it

Access logs can contain IP addresses, query strings and other sensitive information. Work with authorised people and keep raw files private.

  • Remove unnecessary personal data and token-bearing query strings before sharing.
  • Restrict access and agree a retention period.
  • Check a tool’s data handling before uploading logs.
  • Use aggregated, redacted examples in reports.

Keep the verification data needed for the analysis under controlled access. Do not publish raw logs to make an article look more convincing.

3. Verify the crawler before counting it

A user-agent string saying “Googlebot” is not proof. Google documents verification through published IP ranges or reverse DNS followed by a confirming forward lookup.

  • Distinguish common search crawlers from other Google fetchers.
  • Use the correct published range for the crawler being checked.
  • Have a technical owner verify uncertain requests.
  • Keep unverified bot traffic separate from verified crawler counts.

Only then group requests by page type, response status and date. Avoid mixing image requests with HTML page requests when comparing page crawling.

4. Turn patterns into investigations

Northgate Property Listings — fictional UK example; illustrative only

Assume a complete seven-day export containing 1,000 verified Googlebot HTML requests.

  • 650: current listing and area pages. Check whether commercially useful pages are included.
  • 250: filter combinations. Review whether those URLs serve distinct searches.
  • 80: redirects. Inspect old internal links and unnecessary chains.
  • 20: server errors. Check timing, affected URLs and whether errors recur.

The counts total 1,000 and are invented. Filter requests are not automatically wasted; redirects and errors need context before changing anything.

A query such as “flats to rent in Edinburgh” is an illustrative business target, not an observed ranking.

5. Compare logs with indexing and site structure

A successful crawl does not prove indexing. Combine the logs with Search Console, a current crawl and a list of important pages.

  • Investigate important URLs absent from a sufficiently complete export.
  • Check internal links and discovery routes before assuming a crawl-budget problem.
  • Review repeated errors with the hosting owner.
  • Use the redirect guide for unnecessary hops.
  • Use the shop-filter guide when analysing faceted URL patterns.

The crawl-budget guide explains when scale and update frequency make this a meaningful concern.

Limitations and next step

Logs show recorded requests and responses, not Google’s ranking decisions, indexing outcome or the rendered page. Missing data can make an apparent problem disappear or exaggerate it.

Choose one question, such as repeated server errors on important URLs. Agree the coverage, verify the crawler, investigate and repeat the same comparison after the fix. Use Search Console to connect crawl evidence with indexing and visibility, without claiming more crawling guarantees more traffic.

Sources

Sources checked 15 November 2026. The workflow and examples are editorial recommendations, not firsthand testing or client results.

Let's work together.

Whether you need an SEO partner, a consultant, a speaker for your event or training for your team — get in touch and let's chat.

Get in touch
Get in touch