Syntax: one or more user-agent strings, one per line
This is a list of user agents to respect when checking
robots.txt on a site. The
robots.txt group with the
User-agent string that is a case-insensitive substring of the
earliest agent listed in Robots.txt Agents will be used; i.e.
the Robots.txt Agents should be listed highest-priority first.
robots.txt groups match the same agent, the group
with the longest substring-matching
User-agent is used. If no
agents match, and a group for agent "
*" is present, it is
used. The default value for this setting is
For example, changing this setting to
and given this
then the Search Appliance will not walk
/some/other/dir, but will
/some/google/dir: while both agents
is listed first in Robots.txt Agents and is thus higher
robots.txt with the same setting:
then the Search Appliance would not walk
/some/bot/dir, because while
both agents substring-match
Googlebot is the