Skip to main content
Settings
Color Mode
Theme Skin
Background

Appearance preferences are saved in this browser only.

Environment
Current Environment Production

Built with JEKYLL_ENV=production. Changes require deployment.

Quick Links
Theme & Build
Theme
Jekyll v3.10.0
Last Build Jul 27, 03:09
Repository
bamr87/bamr87.github.io
Changelog Docs Jekyll
Admin
Configuration
Page Location
Page Info
Layout article
Collection posts
Path _posts/2023-12-04-robots-dot-txt.md
URL /posts/2023/12/04/robots-dot-txt/
Date 2023-12-04
Source Code
View on GitHub Edit in GitHub.dev
Configuration
Site Config _config.yml

Table of Contents

robots dot txt

• 1 min read • Updated: May 16, 2024

A robots.txt file is used to instruct web robots (typically search engine robots) which pages on your website to crawl and which not to. Here’s an example of a robots.txt file that you might use for a Jekyll site:

User-agent: *
Disallow: /secret/
Disallow: /private/
Disallow: /tmp/
Sitemap: https://bamr87.github.io/sitemap.xml

In this example:

  • User-agent: * means that the instructions apply to all web robots.
  • Disallow: /secret/ tells robots not to crawl pages under the /secret/ directory.
  • Disallow: /private/ and Disallow: /tmp/ do the same for these directories.
  • Sitemap: https://www.yoursite.com/sitemap.xml provides the location of your site’s sitemap, which is helpful for search engines to find and index your content.

Remember to replace https://www.yoursite.com/sitemap.xml with the actual URL of your sitemap. Also, the Disallow entries should be adjusted based on the specific directories or pages you want to keep private or don’t want to be indexed by search engines.