HTML & CSS
/Intermediate
robots.txt
Definition
A simple text file placed in the root directory of a website that tells search engine crawlers which pages or files they can or cannot request from your site.
Explain Like I'm New
It's the bouncer at the door of your website. When Googlebot arrives, the very first thing it does is ask for `robots.txt`. If the file says 'Do not enter the /admin/ folder', Googlebot respectfully turns around and skips that folder so your private admin pages don't show up on Google.
Real World Example
Blocking Google from indexing your `cart` or `checkout` pages. There is no reason a user's shopping cart URL should ever appear in a Google search result.
Common Use Cases
- •Preventing sensitive pages from being indexed
- •Saving crawl budget by blocking low-value pages
Interactive Example
Interview Questions
basic
- Does `robots.txt` physically prevent hackers from accessing a URL?
intermediate
- What does the asterisk (`*`) mean in a User-agent line?