PageRenderTime 60ms CodeModel.GetById 23ms RepoModel.GetById 0ms app.codeStats 0ms

/alert/assets/templates/robots/robots.txt

https://bitbucket.org/jbward/search-and-awareness-platform-courtlistener
Plain Text | 62 lines | 54 code | 8 blank | 0 comment | 0 complexity | 0f7d9b6c703ca745cb095056f088efd2 MD5 | raw file
Possible License(s): LGPL-2.0, AGPL-3.0, BSD-3-Clause, MPL-2.0-no-copyleft-exception 
    

  1. # Google, AOL, Bing, Yahoo!
    2. 

  2. # These support meta robots and x-robots-tag.
    3. 

  3. # Crawl everything except a few explicit blocks
    4. 

  4. User-agent: Googlebot
    5. 

  5. Disallow: /robots.txt
    6. 

  6. Disallow: /dumps/
    7. 

  7. Disallow: /assets/
    8. 

  8. Disallow: /api/rest/v1/
    9. 

  9. 

  10. User-agent: bingbot
    11. 

  11. Disallow: /robots.txt
    12. 

  12. Disallow: /dumps/
    13. 

  13. Disallow: /assets/
    14. 

  14. Disallow: /api/rest/v1/
    15. 

  15. 

  16. User-agent: Slurp
    17. 

  17. Disallow: /robots.txt
    18. 

  18. Disallow: /dumps/
    19. 

  19. Disallow: /assets/
    20. 

  20. Disallow: /api/rest/v1/
    21. 

  21. 

  22. # Yandex, Ask
    23. 

  23. # These support meta robots, but not x-robots-tag
    24. 

  24. # Crawl everything except real files
    25. 

  25. User-agent: YandexBot
    26. 

  26. Disallow: /robots.txt
    27. 

  27. Disallow: /dumps/
    28. 

  28. Disallow: /assets/
    29. 

  29. Disallow: /api/rest/v1/
    30. 

  30. Disallow: /pdf/
    31. 

  31. Disallow: /wpd/
    32. 

  32. Disallow: /txt/
    33. 

  33. Disallow: /doc/
    34. 

  34. 

  35. User-agent: teoma
    36. 

  36. Disallow: /robots.txt
    37. 

  37. Disallow: /dumps/
    38. 

  38. Disallow: /assets/
    39. 

  39. Disallow: /api/rest/v1/
    40. 

  40. Disallow: /pdf/
    41. 

  41. Disallow: /wpd/
    42. 

  42. Disallow: /txt/
    43. 

  43. Disallow: /doc/
    44. 

  44. 

  45. User-agent: ia_archiver
    46. 

  46. Disallow: /robots.txt
    47. 

  47. Disallow: /dumps/
    48. 

  48. Disallow: /assets/
    49. 

  49. Disallow: /api/rest/v1/
    50. 

  50. Disallow: /pdf/
    51. 

  51. Disallow: /wpd/
    52. 

  52. Disallow: /txt/
    53. 

  53. Disallow: /doc/
    54. 

  54. 

  55. 

  56. # Duckduckgo, Baidu, Blekko, Others
    57. 

  57. # No support for robots meta tag nor x-robots-tag.
    58. 

  58. # Be conservative; Block everything.
    59. 

  59. User-agent: *
    60. 

  60. Disallow: /
    61. 

  61. 

  62. Sitemap: https://www.courtlistener.com/sitemap.xml
    63. 

  63.