/src/robots.txt

http://github.com/perlorg/cpanorg · Plain Text · 8 lines · 6 code · 2 blank · 0 comment · 0 complexity · 3146ee3515b68345e93078ad07230b6b MD5 · raw file 

    

  1. # Hello Robots!
    2. 

  2. # Welcome to CPAN.
    3. 

  3. 

  4. User-agent: *
    5. 

  5. Disallow: /badrobot/
    6. 

  6. 

  7. User-Agent: The Knowledge AI
    8. 

  8. Disallow: /
    9.