Sitemap & robots.txt

Two small files tell search engines how to crawl your site: sitemap.xml lists the URLs worth indexing, and robots.txt tells crawlers which parts of the site they may (or may not) visit. Next.js lets you generate both dynamically with plain TypeScript files instead of hand-maintaining static XML and text files.

Why both matter for SEO

File	What it does	Why it matters
`sitemap.xml`	Lists every URL you want indexed, often with last-modified dates and priority hints	Helps search engines discover pages faster, especially ones with few internal links pointing at them
`robots.txt`	Allows or disallows crawler access to specific paths, and can point to the sitemap	Keeps crawlers away from admin routes, search-result pages, or duplicate content, and saves crawl budget for pages that matter

robots.ts

Add a app/robots.ts file that exports a default function returning a MetadataRoute.Robots object. Next.js serves it at /robots.txt automatically.

app/robots.ts

TSX

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/admin', '/api'],
      },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  }
}

sitemap.ts

A static site can hardcode an array of URLs. A content-driven site should generate the sitemap dynamically from whatever database or CMS holds the actual pages, so new content is picked up automatically without a manual sitemap update.

app/sitemap.ts — dynamic, database-driven

TSX

import type { MetadataRoute } from 'next'
import { getAllPosts } from '@/libs/posts'

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getAllPosts()

  const postEntries = posts.map((post) => ({
    url: `https://example.com/blog/${post.slug}`,
    lastModified: post.updatedAt,
    changeFrequency: 'weekly' as const,
    priority: 0.7,
  }))

  return [
    {
      url: 'https://example.com',
      lastModified: new Date(),
      changeFrequency: 'monthly',
      priority: 1,
    },
    {
      url: 'https://example.com/blog',
      lastModified: new Date(),
      changeFrequency: 'daily',
      priority: 0.9,
    },
    ...postEntries,
  ]
}

Next.js serves this at /sitemap.xml, converting the returned array into valid sitemap XML for you — you never write XML by hand.

Note

For very large sites (tens of thousands of URLs), a single sitemap file can hit size limits search engines enforce. Next.js supports generating multiple sitemap files by exporting a generateSitemaps function alongside sitemap.ts, splitting the URL set across several numbered sitemap files that get linked from a sitemap index.

app/robots.ts exports a default function returning crawl rules, served automatically at /robots.txt.
app/sitemap.ts exports a default (async) function returning an array of URL entries, served at /sitemap.xml.
A dynamic sitemap that pulls from your database/CMS keeps itself up to date as content changes, unlike a hand-maintained static file.
Point robots.ts's sitemap field at your sitemap URL so crawlers discover it directly.
Very large sites can split into multiple sitemap files via generateSitemaps.