PageRenderTime 45ms CodeModel.GetById 20ms RepoModel.GetById 1ms app.codeStats 0ms

/alert/assets/templates/simple_pages/robots.txt

https://bitbucket.org/mlissner/search-and-awareness-platform-courtlistener
Plain Text | 69 lines | 59 code | 10 blank | 0 comment | 0 complexity | 1c63eaf458ce45f438142babe9d90cd5 MD5 | raw file
Possible License(s): MPL-2.0-no-copyleft-exception 
    

  1. # Google, AOL, Bing, Yahoo!, DuckDuckGo
    2. 

  2. # These support meta robots and x-robots-tag or are otherwise harmless (DDG)
    3. 

  3. # Crawl everything except a few explicit blocks
    4. 

  4. User-agent: Googlebot
    5. 

  5. Disallow: /robots.txt
    6. 

  6. Disallow: /dumps/
    7. 

  7. Disallow: /assets/
    8. 

  8. Disallow: /api/rest/v1/
    9. 

  9. 

  10. User-agent: bingbot
    11. 

  11. Disallow: /robots.txt
    12. 

  12. Disallow: /dumps/
    13. 

  13. Disallow: /assets/
    14. 

  14. Disallow: /api/rest/v1/
    15. 

  15. 

  16. User-agent: Slurp
    17. 

  17. Disallow: /robots.txt
    18. 

  18. Disallow: /dumps/
    19. 

  19. Disallow: /assets/
    20. 

  20. Disallow: /api/rest/v1/
    21. 

  21. 

  22. User-agent: DuckDuckBot
    23. 

  23. Disallow: /robots.txt
    24. 

  24. Disallow: /dumps/
    25. 

  25. Disallow: /assets/
    26. 

  26. Disallow: /api/rest/v1/
    27. 

  27. 

  28. 

  29. # Yandex, Ask
    30. 

  30. # These support meta robots, but not x-robots-tag
    31. 

  31. # Crawl everything except real files
    32. 

  32. User-agent: YandexBot
    33. 

  33. Disallow: /robots.txt
    34. 

  34. Disallow: /dumps/
    35. 

  35. Disallow: /assets/
    36. 

  36. Disallow: /api/rest/v1/
    37. 

  37. Disallow: /pdf/
    38. 

  38. Disallow: /wpd/
    39. 

  39. Disallow: /txt/
    40. 

  40. Disallow: /doc/
    41. 

  41. 

  42. User-agent: teoma
    43. 

  43. Disallow: /robots.txt
    44. 

  44. Disallow: /dumps/
    45. 

  45. Disallow: /assets/
    46. 

  46. Disallow: /api/rest/v1/
    47. 

  47. Disallow: /pdf/
    48. 

  48. Disallow: /wpd/
    49. 

  49. Disallow: /txt/
    50. 

  50. Disallow: /doc/
    51. 

  51. 

  52. User-agent: ia_archiver
    53. 

  53. Disallow: /robots.txt
    54. 

  54. Disallow: /dumps/
    55. 

  55. Disallow: /assets/
    56. 

  56. Disallow: /api/rest/v1/
    57. 

  57. Disallow: /pdf/
    58. 

  58. Disallow: /wpd/
    59. 

  59. Disallow: /txt/
    60. 

  60. Disallow: /doc/
    61. 

  61. 

  62. 

  63. # Baidu, Blekko, Others
    64. 

  64. # No support for robots meta tag nor x-robots-tag.
    65. 

  65. # Be conservative; Block everything.
    66. 

  66. User-agent: *
    67. 

  67. Disallow: /
    68. 

  68. 

  69. Sitemap: https://www.courtlistener.com/sitemap.xml
    70. 

  70.