Craig Campbell SEO Training & Consultancy

Tutorials

XML Sitemaps: Build, Submit and Troubleshoot

By Craig Campbell · 18 Nov 2026

11.477272727272727 min read

XML Sitemaps: Build, Submit and Troubleshoot

An XML sitemap is a straightforward way to tell search engines about the URLs you want them to discover. Unfortunately, it is also easy to turn one into a dumping ground for redirects, filtered pages and content you never intended to index. Submitting that file does not repair the rest of your website.

I use sitemaps as part of a broader discovery and maintenance process. They should reflect your preferred indexable URLs, update reliably and give you useful groups to investigate when indexing goes wrong. This tutorial covers building, submitting and troubleshooting them without confusing discovery with a promise of rankings. All example websites and addresses are illustrative.

Quick answer: what should an XML sitemap contain?

A standard XML sitemap lists preferred, indexable URLs you want search engines to discover. Include fully qualified addresses and keep them consistent with your canonical tags, redirects and internal links. Normally, those URLs should return a successful response and should not be blocked from crawling or marked noindex.

A single sitemap can contain up to 50,000 URLs and must not exceed 50 MB uncompressed. If you exceed either limit, split the URLs into multiple files and reference them through a sitemap index. Compression can reduce transfer size, but it does not increase the uncompressed size limit.

Submit the sitemap or sitemap index in Google Search Console and consider declaring it in robots.txt. Submission helps discovery; it does not guarantee crawling, indexing or ranking. A sitemap should support a navigable website, not become the only route search engines have to important pages.

Plan the URL inventory before generating XML

I start with the website’s content model rather than a list scraped from whatever a crawler happens to find. Identify the templates you actually want indexed: products, useful categories, articles, service pages and other genuine landing pages. Then define the eligibility rules for each type.

For a product, that might mean published, accessible, canonical to itself and not excluded by a deliberate indexing policy. An article might need a published status and a public URL. Write these rules down so your sitemap generator and your website templates use the same logic rather than slowly drifting apart.

Do not automatically include every URL in your database. Drafts, internal search results, account areas and tracking variants are not suddenly valuable search landing pages because they have an address. Equally, do not exclude an entire template just because a few examples are poor. Make the decision at the right level.

Compare three sources where available: the content database, a crawl of the site and the existing sitemap. URLs found only in the sitemap may be orphaned or obsolete. URLs found only in a crawl may expose missing generation rules. URLs published in the database but absent from both deserve investigation before you assume discovery is working.

This comparison is especially useful after platform changes. A new content type might be visible to users but absent from the sitemap generator. An old plugin may continue publishing archived URLs. The inventory should describe today’s website, not a historical list nobody owns.

Step by step: build a valid XML sitemap

Step 1: choose a generation method. A reliable CMS feature or maintained plugin is usually more practical than editing XML by hand. For a custom platform, generate the file from the publishing system or a controlled URL source. The right method is the one your team can keep accurate.

Step 2: create valid XML. A basic file uses a urlset root with the sitemap protocol namespace. Each URL entry contains a loc element. For example: <url><loc>https://example.com/services/</loc></url>. That entry belongs inside the correctly declared root element; it is not a complete sitemap by itself.

Step 3: encode and escape correctly. Use UTF-8 and ensure XML special characters are escaped. An ampersand inside a URL must appear as &amp; in the XML source. URL encoding and XML escaping solve different problems, so check both rather than replacing characters indiscriminately.

Step 4: use absolute preferred URLs. Include the protocol and hostname. Check for production domains, consistent HTTPS usage and your chosen path format. A sitemap should not list one version while the canonical tag nominates another. Avoid fragment identifiers as entries for ordinary indexable pages.

Step 5: add accurate modification dates if available. The optional lastmod value should describe a significant update to the page, not the time the sitemap was regenerated. Use a valid date or date-time format. If your system cannot provide trustworthy values, leaving the field out is better than manufacturing freshness.

Step 6: split files when necessary. Respect both the URL count and uncompressed size limits. A sitemap index can reference your child sitemap files. You can also split earlier for operational clarity, provided the arrangement remains manageable and the index points to files that actually exist.

Step 7: publish and validate. Check the public endpoint, response status, XML structure and actual listed URLs. Do this without relying on a logged-in browser session. A file visible to an administrator may still be inaccessible to a crawler because of authentication, firewall or environment rules.

Organise sitemap files for useful diagnosis

For a small website, one sitemap may be perfectly adequate. For a larger website, I prefer useful groupings over one enormous undifferentiated list. Splitting by content type, language section or stable business area can make it easier to see where problems are concentrated.

GroupingWhy it helpsWhat to avoid
ProductsSeparates catalogue issues from editorial contentIncluding unavailable URLs without an indexing decision
CategoriesHighlights valuable listing pagesAdding every filter combination
ArticlesSupports publishing checksResetting dates for unchanged articles
Language sectionsMakes regional diagnosis easierAssuming file grouping replaces hreflang
Stable date batchesKeeps large archives manageableMoving unchanged URLs between files unnecessarily

There is no ranking bonus for giving a sitemap an elaborate filename. The benefit is operational. Your team should be able to identify which generator owns a file and what belongs in it. Avoid creating hundreds of tiny files without a reason, because complexity brings its own monitoring burden.

Usually, I place a sitemap index at the root of the host to keep scope straightforward. Be careful with files hosted under subdirectories, alternate hosts or cross-site arrangements. Those setups have scope and verification requirements; do not assume a sitemap anywhere on the internet can freely submit any URL.

Step by step: submit and verify discovery

Step 1: open the correct Search Console property. Confirm that it covers the host and protocol you are submitting. Domain properties can cover multiple protocol and subdomain variants, while URL-prefix properties have a narrower scope. Confusing the two can make a working sitemap look absent from the place you are checking.

Step 2: submit the sitemap address. Use the Sitemaps report and submit the index if you have one. You do not normally need to submit every child file individually. Keep the address stable so automated generation can update the contents without requiring repeated manual submissions.

Step 3: add a robots.txt reference. A line such as Sitemap: https://example.com/sitemap_index.xml advertises the file location. This is a discovery reference, not an instruction to index everything inside it. It also does not override rules elsewhere in robots.txt.

Step 4: check the processing result. Confirm that Google can fetch and read the file. Review reported errors and whether child sitemaps are being discovered. A successful fetch means the file was processed; it does not mean every listed page has been accepted into the index.

Step 5: inspect real pages. Choose examples from each group, including newly published content and pages you know have changed. Verify that their sitemap inclusion, canonical tags and indexing directives agree. This catches logical problems that a syntactically valid XML document cannot reveal.

Other search engines have their own webmaster submission routes. Use the appropriate official tools where relevant to your audience, but keep one accurate source of sitemap data. Maintaining contradictory files for different engines is usually an avoidable source of mistakes.

Troubleshoot fetch and parsing errors

If a sitemap cannot be fetched, start with the response. Does the requested address return XML successfully, redirect somewhere unexpected or show an HTML error page? Check without cookies and from outside your own network where possible. A content delivery network may treat verified crawlers differently from your browser.

Inspect firewall rules, rate limits and server logs before assuming the XML is wrong. Intermittent server errors can make a valid file unreliable. If your generator performs an expensive database operation on every request, consider caching or pre-generating it with safe refresh behaviour rather than allowing bursts of requests to exhaust resources.

For parsing errors, examine the raw response. Typical causes include unescaped characters, broken nesting, incorrect namespace declarations and extra output from application warnings. A visually attractive browser stylesheet is not a validator. The underlying XML must remain valid even when the presentation looks tidy.

Watch for systems that return a successful status with an error message inside the body. That can mislead basic uptime checks. I want monitoring that confirms both a healthy response and recognisable sitemap content, then tests a sample of the URLs listed inside it.

If files exceed protocol limits, split them and update the index. If the index references missing child files, fix the publishing sequence so new files become available before the index advertises them. Reliable deployment matters just as much as correct generation code.

Troubleshoot submitted URLs that are not indexed

Separate file-level problems from page-level decisions. A sitemap can be perfectly readable while its URLs are duplicates, blocked, thin, redirected or simply not selected for indexing. Resubmitting the same file repeatedly will not change the quality or technical state of those pages.

Sample affected URLs and group them by reason. Check response codes, robots access, noindex directives, canonical destinations and rendered content. Compare an affected page with an indexed page from the same template. Differences often reveal a publishing or linking issue more quickly than staring at an aggregate total.

For pages reported as crawled but not indexed, use my crawled currently not indexed guide. The useful next step is understanding why those pages are not being selected, not adding them to yet another sitemap.

If discovery is happening but crawling is slow on a large website, examine server reliability and the URLs the site exposes. My crawl budget guide covers that distinction. Sitemaps do not prevent crawlers spending resources on endless filters, broken links or duplicate paths elsewhere.

Server evidence can help establish whether requested pages were actually fetched. With log file analysis, you can compare sitemap membership with verified crawler activity. Logs still do not prove indexing, but they help separate a discovery assumption from an observed request.

Common XML sitemap mistakes

Including redirected or non-canonical URLs. These entries nominate addresses you do not actually want indexed. Update the source rules rather than accepting permanent noise. Occasional change is normal; a sitemap that routinely publishes obsolete versions needs fixing.

Publishing artificial freshness. Setting every lastmod value to the current date on each build makes the field misleading. A navigation update or a routine rendering event is not necessarily a significant change to the page’s main content. Tie the value to meaningful page changes.

Relying on priority and changefreq. Google ignores these sitemap fields. Do not spend hours tuning them while your important pages are buried or your server returns errors. Clear internal links, sound content and trustworthy URL data are more useful work.

Forgetting unpublished content. A deleted page should not remain indefinitely in the sitemap because the generator runs from a stale export. Equally, a newly published landing page should not wait for someone to remember a manual spreadsheet. Connect sitemap updates to the publishing workflow.

Confusing removal with deindexing. Taking a URL out of the sitemap does not remove it from search. Use the appropriate page status or indexing directive for the actual situation. Search engines can still discover that URL through links, historical knowledge or other sources.

Test the update process as well as the finished file. Publish an eligible test page in a controlled environment, confirm its inclusion, then change its status and check that the generator responds correctly. Repeat for a canonical change. These simple lifecycle checks reveal whether the sitemap follows the publishing rules or merely produces valid XML from outdated data.

Keep a record of generation failures and the last successful publication. If a scheduled job breaks, serving the previous valid file temporarily may be preferable to replacing it with an empty document, but the failure still needs an alert and a prompt fix.

XML sitemap maintenance checklist

  1. Define which published templates and URLs qualify for inclusion.
  2. List absolute preferred URLs that match canonical and internal-link choices.
  3. Exclude redirects, errors and deliberately non-indexable pages.
  4. Validate XML structure, escaping and protocol limits.
  5. Use truthful lastmod values or omit them.
  6. Publish stable child files and a working index where needed.
  7. Submit the correct address in the relevant Search Console property.
  8. Add a clear sitemap reference in robots.txt.
  9. Monitor fetching, parsing and representative page eligibility.
  10. Recheck the generator after migrations and publishing-system changes.

Give the process an owner. Someone should know whether a missing product is a publishing decision, a generation bug or a reporting delay. Without ownership, sitemap errors tend to become background noise until a larger release exposes how far the data has drifted.

Frequently asked questions

Does every website need an XML sitemap?

Not necessarily. Search engines can discover a small, well-linked website without one. I still usually provide a clean sitemap because it makes the intended URL set explicit and supports monitoring. Larger websites, new sections and less well-connected content can particularly benefit from that additional discovery route.

How often should I update my sitemap?

Update it when eligible URLs are published, removed or meaningfully changed. The implementation depends on your platform: event-driven updates, scheduled generation or sensible caching can all work. There is no benefit in rewriting an unchanged file constantly simply to make the timestamp look active.

Should out-of-stock products stay in the sitemap?

That depends on whether the product page remains an indexable, useful destination. A temporarily unavailable product may still deserve its page and sitemap entry. A permanently removed product needs a separate decision about replacement, redirection or removal. Stock status alone is not a complete indexing strategy.

Can I have more than one sitemap?

Yes. Multiple sitemap files are normal, especially for larger sites. Use a sitemap index to organise them and respect the limits for individual files and the index itself. Choose meaningful groups where possible so troubleshooting does not become an exercise in opening randomly numbered files.

Why does successful submission not mean every URL is indexed?

Submission confirms a discovery mechanism, not acceptance of every page. Search engines still evaluate accessibility, duplication, canonicalisation and other indexing considerations. Review representative excluded URLs and fix the actual reason. A clean sitemap can make that investigation easier, but it cannot replace it.

Your sitemap should be a dependable summary of the pages you stand behind, not a wish list disconnected from the live site. If discovery and indexing still do not line up, an SEO audit can help identify the underlying blockers.

Let's work together.

Whether you need an SEO partner, a consultant, a speaker for your event or training for your team — get in touch and let's chat.

Get in touch
Get in touch