robots.txt

Definition

A text file (`robots.txt`) at the root of a website that explicitly tells search engine crawlers which parts of the site they are allowed to scan, and which they must ignore.

Explain Like I'm New

You don't want Google showing your private `/admin` dashboard or your `/checkout` page in public search results. `robots.txt` is a sign on the door that says 'Google Bot: Do Not Enter this specific room'.

Real World Example

Preventing AI web scrapers from stealing the data on your proprietary `/api` routes by blocking them in `robots.txt`.

Common Use Cases

•SEO management
•Crawler control

Interactive Example

// app/robots.ts
import { MetadataRoute } from 'next';

// Generates /robots.txt automatically
export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*', // Applies to ALL bots (Google, Bing, etc)
      allow: '/',     // They can crawl the main site
      disallow: ['/admin/', '/private/', '/api/'], // DO NOT crawl these!
    },
    // Crucial: Tell bots where to find your map!
    sitemap: 'https://mywebsite.com/sitemap.xml',
  };
}

Interview Questions

basic

Does `robots.txt` physically stop a malicious hacker from accessing a URL?

intermediate

How do you programmatically generate a `robots.txt` file in Next.js?

Flash Cards

Question

Stop hackers?

Click to reveal answer

Answer

Absolutely not. It relies entirely on the 'honor system'. Good bots (like Google) obey it. Malicious bots completely ignore it. Never use it for security.

Question

How to generate?

Click to reveal answer

Answer

By creating a `robots.ts` file in the `app` directory that exports a function returning the rules.