PageRenderTime 61ms CodeModel.GetById 23ms RepoModel.GetById 1ms app.codeStats 0ms

/Pycon2009/scrape/scraping-the-web.rst

https://github.com/toastdriven/pydanny-event-notes
ReStructuredText | 109 lines | 80 code | 29 blank | 0 comment | 0 complexity | 0ff3b2a93b73c091b5a5650d713c9705 MD5 | raw file
    

  1. ===============
    2. 

  2. Scrap the Web
    3. 

  3. ===============
    4. 

  4. 

  5.     * Asheesh Laroia
    6. 

  6.     * http://pycon09.asheesh.org/
    7. 

  7.  
    8. 

  8. Things to remember
    9. 

  9. ===================
    10. 

  10.  
    11. 

  11.  * Scrapists are people who scrape web pages. 
    12. 

  12.  * Ignore terms of service at your on peril.
    13. 

  13.  * DO NOT BECOME AN EVIL COMMENT SPAMMER
    14. 

  14.  * Hidden form fields
    15. 

  15.  * make traps for automated web scrapers look for emails
    16. 

  16.  * If its on the web, you can scrape it
    17. 

  17.     - Now you have an API for everything 
    18. 

  18.  
    19. 

  19. Why scrape the web?
    20. 

  20. --------------------
    21. 

  21.  
    22. 

  22.  * You can maintain interoperation with unmaintained systems
    23. 

  23.  * "Rogue interoperability"
    24. 

  24.  
    25. 

  25. Three perspectives
    26. 

  26. ---------------------
    27. 

  27. 

  28.     * Be a maverick
    29. 

  29.     * Use the old standby XML tools
    30. 

  30.     * "Use the web tools against the web"
    31. 

  31.     
    32. 

  32. 

  33. Choosing a parser
    34. 

  34. ------------------
    35. 

  35. 

  36.     * Performance
    37. 

  37.     * Ease-of-use
    38. 

  38.     
    39. 

  39.  
    40. 

  40. ----
    41. 

  41. 

  42. Some tools
    43. 

  43. ===========
    44. 

  44. 

  45.  * BeautifulSoup is old and not maintained anymore
    46. 

  46.  * html5lib 
    47. 

  47.     - builds BeautifulSoup objects
    48. 

  48.     - builds elementTrees
    49. 

  49.  * lxml provides XPath
    50. 

  50.  * mechanize (so much better than urllib2!)
    51. 

  51.  * pyquery
    52. 

  52.  * DOMForm is a python module for web scraping but it is not maintained
    53. 

  53.  * python-spidermonkey is a bridge between python and spidermonkey
    54. 

  54.     - lets you build a python engine that lets you run Javascript inside of Python!
    55. 

  55.  * Firefox via SeleniumRC. Really powerful. Downside is that instances of Selenium is not cheap.
    56. 

  56. 

  57. Mechanize
    58. 

  58. -----------
    59. 

  59.     
    60. 

  60.     * You may need to turn off robots control 
    61. 

  61.         - mechanize.Browser().set_handle_robots(False)
    62. 

  62.     * See some other tricks
    63. 

  63.     * Handles hidden fields and cookies
    64. 

  64.     
    65. 

  65. 

  66. HTTP Headers for urllib
    67. 

  67. ------------------------
    68. 

  68. 

  69.     * Need to support headers in stuff via urllib2
    70. 

  70.     * 2xx: Success
    71. 

  71.     * 3xx: Redirection
    72. 

  72.     * 4xx: Error
    73. 

  73.     * JavaScript behavior does not mirror Firefox
    74. 

  74.     * Image download behavior
    75. 

  75.     * Cookie behavior
    76. 

  76.     * Invalid HTML handling behavior
    77. 

  77.     * Not emulating web browsers can cause all kinds of fun and random things to happen
    78. 

  78. 

  79. ----
    80. 

  80. 

  81. HTTP Methods
    82. 

  82. =============
    83. 

  83. 

  84.     * GET is for requesting a page
    85. 

  85.     * POST is for submitting a form to a server
    86. 

  86.     * PUT is for uploading files in WebDav
    87. 

  87.     * BREW is because POST is trademarked by Post.
    88. 

  88.     
    89. 

  89. robots.txt
    90. 

  90. -----------
    91. 

  91. 

  92.     * Check out http://www.robotstxt.org
    93. 

  93. 

  94. Getting around IP address limits
    95. 

  95. ================================
    96. 

  96. 

  97.     * ssh -D to borrow the IP the machine you can log in to
    98. 

  98.     * ssh -D 1080 asheesh.org
    99. 

  99.         - Binds the 1080 port number to your local
    100. 

  100.         - Sets up a soft proxy redirect to fake out people wondering who you are
    101. 

  101. 

  102. ----
    103. 

  103.     
    104. 

  104. Handling CAPTCHA
    105. 

  105. =================
    106. 

  106. 

  107.      * Sometimes people only have a limited set of images
    108. 

  108.      * Audio analysis
    109. 

  109.      * Use SeleniumRC to let you do the CAPTCHA for things.
    110.