Simple robots.txt Package

This is a simple package for adding a robots.txt file to your site.

Summary and Content fields are not used. The first usabled field is the free-form Robots Txt textarea. The default value in this field (below) will allow all robots to all pages.

User-agent: *
Disallow:

You can add your own exclusions here based on actual file names or folder paths.

User-agent: * 
Disallow: /somepage
Disallow: /landing-pages/

The Exclude All of Types checkbox list allows you to disallow every page of a given entity type. If you select Tags then all tags will be added to the robots.txt file.

Additionally, any entities with a URL that have NOINDEX in the Robots field will also be listed in this automatically.