Technical SEO for developers: what to check in your code

SEO is often associated with keywords and content, but a significant part of it depends on technical choices that are entirely in the hands of the developer: how the server responds, how URLs are built, what the head of each page contains. A mistake at this level can make even the best content invisible. Here are the checks worth building into every project.

Crawling and indexing are two different things

A search engine first crawls a page (downloads it and reads its content), then decides whether to index it (include it in the results). The tools for controlling the two phases are different, and confusing them is a classic mistake.

• robots.txt controls crawling: it tells crawlers which paths not to visit.
• Meta robots noindex (or the X-Robots-Tag header) controls indexing: it asks for the page not to be shown in results.

# robots.txt
User-agent: *
Disallow: /private-area/
Sitemap: https://www.example.com/sitemap.xml

<meta name="robots" content="noindex">

The critical point: if you block a page with robots.txt, the crawler does not download it and therefore cannot see your noindex. A blocked page can still appear in results, perhaps without a description, if other sites link to it. To truly exclude a page from results, leave it crawlable and use noindex.

Remember too that robots.txt is public and is not a security measure: private areas must be protected with authentication.

Correct status codes

The server must respond with the code that describes the real situation:

A frequent mistake in sites built as JavaScript applications is the soft 404: the "not found" page is served with a 200 code. The search engine treats it as a valid page and may index hundreds of empty pages. Make sure non-existent URLs really return 404.

For redirects, avoid chains (A goes to B which goes to C): point every old URL directly at the final destination.

One version of each page

The same content is often reachable from several addresses: with and without www, with and without a trailing slash, with tracking parameters, in upper and lower case. To a search engine they are different pages with duplicate content.

The solutions, to be used together:

1. 301 redirects to a single form for host and protocol (for example always https://www.).
2. Internal consistency: the site's links always point to the canonical form.
3. A canonical tag on every page, indicating the preferred URL:

<link rel="canonical" href="https://www.example.com/guides/technical-seo/">

The canonical is a strong hint, not an order; it works well when it agrees with redirects, internal links and the sitemap. Always use absolute URLs.

XML sitemap

The sitemap lists the URLs you want indexed. It does not guarantee indexing, but it helps engines discover pages, especially on new or large sites.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/guides/technical-seo/</loc>
    <lastmod>2025-02-01</lastmod>
  </url>
</urlset>

Include only canonical, indexable URLs returning 200. A sitemap full of redirects or noindex pages sends contradictory signals. If you use a static generator, have it created automatically at build time.

Title and description

Every page should have a unique, descriptive title and a relevant meta description. The title is one of the most important signals and appears in results; the description does not directly affect ranking, but it is often the text shown under the title and influences the user's choice. Generate them systematically from the page data rather than leaving them identical across the site.

Multilingual sites: hreflang

If you have versions of the same content in different languages, hreflang tells engines which one to show to which audience:

<link rel="alternate" hreflang="it" href="https://www.example.com/it/guida/">
<link rel="alternate" hreflang="en" href="https://www.example.com/en/guide/">
<link rel="alternate" hreflang="x-default" href="https://www.example.com/en/guide/">

The annotations must be reciprocal: the Italian page points to the English one and vice versa, and each includes itself. Without reciprocity, engines may ignore them.

JavaScript and rendering

The main search engines can execute JavaScript, but rendering may happen later and not every crawler does it. If the main content and links appear only after scripts run, you expose yourself to delays and risks. Some rules:

• serve the main content in the initial HTML, with server-side rendering or static generation;
• use real <a href="..."> links for navigation, not clickable elements with JavaScript handlers;
• do not leave the title, canonical and meta robots to later script changes alone.

URL structure

Short, readable and stable URLs help users and engines alike. Use words separated by hyphens, avoid session identifiers and unnecessary parameters and, above all, do not change URLs without a strong reason. If you must, set up 301 redirects from the old address to the new one.

In short

• robots.txt controls crawling, noindex controls indexing: do not block what you want deindexed.
• Return real status codes and avoid soft 404s and redirect chains.
• Unify the versions of each page with 301 redirects, consistent internal links and canonicals.
• Keep a sitemap with canonical URLs only, and reciprocal hreflang annotations.
• Serve content and links in the initial HTML and keep URLs stable.