/public/robots.txt

https://bitbucket.org/opus85/rottentomatoes · Plain Text · 5 lines · 5 code · 0 blank · 0 comment · 0 complexity · be860f6d2760c884906a9180451d799f MD5 · raw file 

    

  1. # See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
    2. 

  2. #
    3. 

  3. # To ban all spiders from the entire site uncomment the next two lines:
    4. 

  4. # User-Agent: *
    5. 

  5. # Disallow: /
    6.