/extras/perl/site_perl/lwpcook.pod

http://github.com/perigrin/android-scripting-environment-perl · Unknown · 309 lines · 212 code · 97 blank · 0 comment · 0 complexity · bef2208a32baf1f10b0614dea7437094 MD5 · raw file

  1. =head1 NAME
  2. lwpcook - The libwww-perl cookbook
  3. =head1 DESCRIPTION
  4. This document contain some examples that show typical usage of the
  5. libwww-perl library. You should consult the documentation for the
  6. individual modules for more detail.
  7. All examples should be runnable programs. You can, in most cases, test
  8. the code sections by piping the program text directly to perl.
  9. =head1 GET
  10. It is very easy to use this library to just fetch documents from the
  11. net. The LWP::Simple module provides the get() function that return
  12. the document specified by its URL argument:
  13. use LWP::Simple;
  14. $doc = get 'http://www.linpro.no/lwp/';
  15. or, as a perl one-liner using the getprint() function:
  16. perl -MLWP::Simple -e 'getprint "http://www.linpro.no/lwp/"'
  17. or, how about fetching the latest perl by running this command:
  18. perl -MLWP::Simple -e '
  19. getstore "ftp://ftp.sunet.se/pub/lang/perl/CPAN/src/latest.tar.gz",
  20. "perl.tar.gz"'
  21. You will probably first want to find a CPAN site closer to you by
  22. running something like the following command:
  23. perl -MLWP::Simple -e 'getprint "http://www.perl.com/perl/CPAN/CPAN.html"'
  24. Enough of this simple stuff! The LWP object oriented interface gives
  25. you more control over the request sent to the server. Using this
  26. interface you have full control over headers sent and how you want to
  27. handle the response returned.
  28. use LWP::UserAgent;
  29. $ua = LWP::UserAgent->new;
  30. $ua->agent("$0/0.1 " . $ua->agent);
  31. # $ua->agent("Mozilla/8.0") # pretend we are very capable browser
  32. $req = HTTP::Request->new(GET => 'http://www.linpro.no/lwp');
  33. $req->header('Accept' => 'text/html');
  34. # send request
  35. $res = $ua->request($req);
  36. # check the outcome
  37. if ($res->is_success) {
  38. print $res->decoded_content;
  39. }
  40. else {
  41. print "Error: " . $res->status_line . "\n";
  42. }
  43. The lwp-request program (alias GET) that is distributed with the
  44. library can also be used to fetch documents from WWW servers.
  45. =head1 HEAD
  46. If you just want to check if a document is present (i.e. the URL is
  47. valid) try to run code that looks like this:
  48. use LWP::Simple;
  49. if (head($url)) {
  50. # ok document exists
  51. }
  52. The head() function really returns a list of meta-information about
  53. the document. The first three values of the list returned are the
  54. document type, the size of the document, and the age of the document.
  55. More control over the request or access to all header values returned
  56. require that you use the object oriented interface described for GET
  57. above. Just s/GET/HEAD/g.
  58. =head1 POST
  59. There is no simple procedural interface for posting data to a WWW server. You
  60. must use the object oriented interface for this. The most common POST
  61. operation is to access a WWW form application:
  62. use LWP::UserAgent;
  63. $ua = LWP::UserAgent->new;
  64. my $req = HTTP::Request->new(POST => 'http://www.perl.com/cgi-bin/BugGlimpse');
  65. $req->content_type('application/x-www-form-urlencoded');
  66. $req->content('match=www&errors=0');
  67. my $res = $ua->request($req);
  68. print $res->as_string;
  69. Lazy people use the HTTP::Request::Common module to set up a suitable
  70. POST request message (it handles all the escaping issues) and has a
  71. suitable default for the content_type:
  72. use HTTP::Request::Common qw(POST);
  73. use LWP::UserAgent;
  74. $ua = LWP::UserAgent->new;
  75. my $req = POST 'http://www.perl.com/cgi-bin/BugGlimpse',
  76. [ search => 'www', errors => 0 ];
  77. print $ua->request($req)->as_string;
  78. The lwp-request program (alias POST) that is distributed with the
  79. library can also be used for posting data.
  80. =head1 PROXIES
  81. Some sites use proxies to go through fire wall machines, or just as
  82. cache in order to improve performance. Proxies can also be used for
  83. accessing resources through protocols not supported directly (or
  84. supported badly :-) by the libwww-perl library.
  85. You should initialize your proxy setting before you start sending
  86. requests:
  87. use LWP::UserAgent;
  88. $ua = LWP::UserAgent->new;
  89. $ua->env_proxy; # initialize from environment variables
  90. # or
  91. $ua->proxy(ftp => 'http://proxy.myorg.com');
  92. $ua->proxy(wais => 'http://proxy.myorg.com');
  93. $ua->no_proxy(qw(no se fi));
  94. my $req = HTTP::Request->new(GET => 'wais://xxx.com/');
  95. print $ua->request($req)->as_string;
  96. The LWP::Simple interface will call env_proxy() for you automatically.
  97. Applications that use the $ua->env_proxy() method will normally not
  98. use the $ua->proxy() and $ua->no_proxy() methods.
  99. Some proxies also require that you send it a username/password in
  100. order to let requests through. You should be able to add the
  101. required header, with something like this:
  102. use LWP::UserAgent;
  103. $ua = LWP::UserAgent->new;
  104. $ua->proxy(['http', 'ftp'] => 'http://username:password@proxy.myorg.com');
  105. $req = HTTP::Request->new('GET',"http://www.perl.com");
  106. $res = $ua->request($req);
  107. print $res->decoded_content if $res->is_success;
  108. Replace C<proxy.myorg.com>, C<username> and
  109. C<password> with something suitable for your site.
  110. =head1 ACCESS TO PROTECTED DOCUMENTS
  111. Documents protected by basic authorization can easily be accessed
  112. like this:
  113. use LWP::UserAgent;
  114. $ua = LWP::UserAgent->new;
  115. $req = HTTP::Request->new(GET => 'http://www.linpro.no/secret/');
  116. $req->authorization_basic('aas', 'mypassword');
  117. print $ua->request($req)->as_string;
  118. The other alternative is to provide a subclass of I<LWP::UserAgent> that
  119. overrides the get_basic_credentials() method. Study the I<lwp-request>
  120. program for an example of this.
  121. =head1 COOKIES
  122. Some sites like to play games with cookies. By default LWP ignores
  123. cookies provided by the servers it visits. LWP will collect cookies
  124. and respond to cookie requests if you set up a cookie jar.
  125. use LWP::UserAgent;
  126. use HTTP::Cookies;
  127. $ua = LWP::UserAgent->new;
  128. $ua->cookie_jar(HTTP::Cookies->new(file => "lwpcookies.txt",
  129. autosave => 1));
  130. # and then send requests just as you used to do
  131. $res = $ua->request(HTTP::Request->new(GET => "http://www.yahoo.no"));
  132. print $res->status_line, "\n";
  133. As you visit sites that send you cookies to keep, then the file
  134. F<lwpcookies.txt"> will grow.
  135. =head1 HTTPS
  136. URLs with https scheme are accessed in exactly the same way as with
  137. http scheme, provided that an SSL interface module for LWP has been
  138. properly installed (see the F<README.SSL> file found in the
  139. libwww-perl distribution for more details). If no SSL interface is
  140. installed for LWP to use, then you will get "501 Protocol scheme
  141. 'https' is not supported" errors when accessing such URLs.
  142. Here's an example of fetching and printing a WWW page using SSL:
  143. use LWP::UserAgent;
  144. my $ua = LWP::UserAgent->new;
  145. my $req = HTTP::Request->new(GET => 'https://www.helsinki.fi/');
  146. my $res = $ua->request($req);
  147. if ($res->is_success) {
  148. print $res->as_string;
  149. }
  150. else {
  151. print "Failed: ", $res->status_line, "\n";
  152. }
  153. =head1 MIRRORING
  154. If you want to mirror documents from a WWW server, then try to run
  155. code similar to this at regular intervals:
  156. use LWP::Simple;
  157. %mirrors = (
  158. 'http://www.sn.no/' => 'sn.html',
  159. 'http://www.perl.com/' => 'perl.html',
  160. 'http://www.sn.no/libwww-perl/' => 'lwp.html',
  161. 'gopher://gopher.sn.no/' => 'gopher.html',
  162. );
  163. while (($url, $localfile) = each(%mirrors)) {
  164. mirror($url, $localfile);
  165. }
  166. Or, as a perl one-liner:
  167. perl -MLWP::Simple -e 'mirror("http://www.perl.com/", "perl.html")';
  168. The document will not be transferred unless it has been updated.
  169. =head1 LARGE DOCUMENTS
  170. If the document you want to fetch is too large to be kept in memory,
  171. then you have two alternatives. You can instruct the library to write
  172. the document content to a file (second $ua->request() argument is a file
  173. name):
  174. use LWP::UserAgent;
  175. $ua = LWP::UserAgent->new;
  176. my $req = HTTP::Request->new(GET =>
  177. 'http://www.linpro.no/lwp/libwww-perl-5.46.tar.gz');
  178. $res = $ua->request($req, "libwww-perl.tar.gz");
  179. if ($res->is_success) {
  180. print "ok\n";
  181. }
  182. else {
  183. print $res->status_line, "\n";
  184. }
  185. Or you can process the document as it arrives (second $ua->request()
  186. argument is a code reference):
  187. use LWP::UserAgent;
  188. $ua = LWP::UserAgent->new;
  189. $URL = 'ftp://ftp.unit.no/pub/rfc/rfc-index.txt';
  190. my $expected_length;
  191. my $bytes_received = 0;
  192. my $res =
  193. $ua->request(HTTP::Request->new(GET => $URL),
  194. sub {
  195. my($chunk, $res) = @_;
  196. $bytes_received += length($chunk);
  197. unless (defined $expected_length) {
  198. $expected_length = $res->content_length || 0;
  199. }
  200. if ($expected_length) {
  201. printf STDERR "%d%% - ",
  202. 100 * $bytes_received / $expected_length;
  203. }
  204. print STDERR "$bytes_received bytes received\n";
  205. # XXX Should really do something with the chunk itself
  206. # print $chunk;
  207. });
  208. print $res->status_line, "\n";
  209. =head1 COPYRIGHT
  210. Copyright 1996-2001, Gisle Aas
  211. This library is free software; you can redistribute it and/or
  212. modify it under the same terms as Perl itself.