{*_* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *



Author:       François PIETTE

Creation:     November 23, 1997

Version:      7.08

Description:  THttpCli is an implementation for the HTTP protocol

              RFC 1945 (V1.0), and some of RFC 2068 (V1.1)

Credit:       This component was based on a freeware from by Andreas

              Hoerstemeier and used with his permission.

              andy@hoerstemeier.de http://www.hoerstemeier.com/index.htm

EMail:        francois.piette@overbyte.be         http://www.overbyte.be

Support:      Use the mailing list twsocket@elists.org

              Follow "support" link at http://www.overbyte.be for subscription.

Legal issues: Copyright (C) 1997-2010 by François PIETTE

              Rue de Grady 24, 4053 Embourg, Belgium. Fax: +32-4-365.74.56

              <francois.piette@overbyte.be>

              SSL implementation includes code written by Arno Garrels,

              Berlin, Germany, contact: <arno.garrels@gmx.de>



              This software is provided 'as-is', without any express or

              implied warranty.  In no event will the author be held liable

              for any  damages arising from the use of this software.



              Permission is granted to anyone to use this software for any

              purpose, including commercial applications, and to alter it

              and redistribute it freely, subject to the following

              restrictions:



              1. The origin of this software must not be misrepresented,

                 you must not claim that you wrote the original software.

                 If you use this software in a product, an acknowledgment

                 in the product documentation would be appreciated but is

                 not required.



              2. Altered source versions must be plainly marked as such, and

                 must not be misrepresented as being the original software.



              3. This notice may not be removed or altered from any source

                 distribution.



              4. You must register this software by sending a picture postcard

                 to the author. Use a nice stamp and mention your name, street

                 address, EMail address and any comment you like to say.



Quick Reference:

HTTP component can retrieve documents or files using HTTP protocol; that is

connect to a HTTP server also known as a webserver. It can also trigger a

CGI/ISAPI/NSAPI script and post data using either GET or POST method.

Syntax of an URL: protocol://[user[:password]@]server[:port]/path

Path can include data: question mark followed by URL encoded data.

HTTP component is either asynchonous (non-blocking) or synchonous (blocking).

Highest performance is when using asynchonous operation. This is the

recommended way to use HTTP component.

To request several URL simultaneously, use asynchronous operation and as much

HTTP components as you wants to request URLs. All requests will be executed

simultaneously without using multi-threading and without blocking your app.

Methods:

    GetASync    Asynchronous, non-blocking Get

                Retrieve document or file specified by URL, without blocking.

                OnRequestDone event trigered when finished. Use HTTP GET

                method (data contained in URL)

    PostASync   Asynchronous, non-blocking Post

                Retrieve document or file specified by URL, without blocking.

                OnRequestDone event trigered when finished. Use HTTP POST

                method (data contained in request stream)

    HeadASync   Asynchronous, non-blocking Head

                Retrieve document or file header specified by URL, without

                blocking. OnRequestDone event trigered when finished. Use HTTP

                HEAD method.

    Get         Synchronous, blocking Get. Same as GetAsync, but blocks until

                finished.

    Post        Synchronous, blocking Post. Same as PostAsync, but blocks until

                finished.

    Head        Synchronous, blocking Head. Same as HeadAsync, but blocks until

                finished.

    Abort       Immediately close communication.



Updates:

11/29/97 RcvdStream and SendStream properties moved to public section

11/30/97 Document name bug corrected

12/02/97 Removed bug occuring with terminating slash in docname

12/03/97 Added properties RcvdCount and SentCount to easily add a progress

         bar feature (On receive, the ContentLength is initialized with the

         value from the header. Update the progress bar in the OnDocData event,

         or the OnSendData event).

         Added the OnSendBegin, OnSendData and OnSendEnd events.

12/07/97 Corrected Head function to work as expected. Thanks to

         R. Barry Jones <rbjones@therightside.demon.co.uk

29/12/97 V0.96 Added ModifiedSince property as following proposition made by

         Aw Kong Koy" <infomap@tm.net.my>.

30/12/97 V0.97 Added a Cookie property to send cookies

11/01/98 V0.98 Added WSocket read-only property which enable to access the

         socket component used internally. For example to close it to abort

         a connection.

13/01/98 V0.99 Added MultiThreaaded property to tell the component that it is

         working in a thread and should take care of it.

15/01/98 V1.00 Completely revised internal working to make it work properly

         with winsock 2. The TimeOut property is gone.

         Changed OnAnswerLine event to OnHeaderData to be more consistent.

         Replaced AnswserLine property by readonly LastResponse property.

         Added OnRequestDone event. Added GetAsync, PostAsync, HeadAsync

         asynchronous, non-blocking methods. Added Abort procedure.

16/01/98 V1.01 Corrected a bug which let some data be lost when receiving

         (thanks to  Fulvio J. Castelli <fulvio@rocketship.com>)

         Added test for HTTP/1.1 response in header.

31/01/98 V1.02 Added an intermediate message posting for the OnRequestDone

         event. Thanks to Ed Hochman <ed@mbhsys.com> for his help.

         Added an intermediate PostMessage to set the component to ready state.

04/02/98 V1.03 Added some code to better handle DocName (truncating at the

         first question mark).

05/02/98 V1.04 Deferred login after a relocation, using WM_HTTP_LOGIN message.

         Added workarounf to support faulty webservers which sent only a single

         LF in header lines. Submitted by Alwin Hoogerdijk <alwin@lostboys.nl>

15/03/98 V1.05 Enlarge buffers from 2048 to 8192 bytes (not for D1)

01/04/98 V1.06 Adapted for BCB V3

13/04/98 V1.07 Made RcvdHeader property readonly and cleared the content at the

         start of a request.

         Protected Abort method from calling when component is ready.

         Ignore any exception triggered by CancelDnsLookup in Abort method.

14/04/98 V1.08 Corrected a relocation bug occuring with relative path

26/04/98 V1.09 Added OnLocationChange event

30/04/98 V1.10 Added ProxyUsername and ProxyPassword. Suggested by

         Myers, Mike <MikeMy@crt.com>.

26/05/98 V1.11 Corrected relocation problem when used with ASP webpages

09/07/98 V1.12 Adapted for Delphi 4

         Checked argument length in SendCommand

19/09/98 V1.13 Added support for HTML document without header

         Added OnSessionConnected event, httpConnected state and

         httpDnsLookupDone state.

         Corrected a problem with automatic relocation. The relocation

         message was included in data, resulting in wrong document data.

         Added two new events: OnRequestHeaderBegin and OnRequestHeaderEnd.

         They replace the OnHeaderBegin and OnHeaderEnd events that where

         called for both request header (to web server) and response

         header (from web server)

22/11/98 V1.14 Added a Location property than gives the new location in

         case of page relocation. Suggested by Jon Robertson <touri@pobox.com>

21/12/98 V1.15 Set ContentLength equal to -1 at start of command.

31/01/99 V1.16 Added HostName property

01/02/99 V1.17 Port was lost in DoRequestAsync when using a proxy.

         Thanks to David Wright <wrightd@gamespy.com> for his help.

         Report Dns lookup error and session connect error in OnrequestDOne

         event handler as suggested by Jack Olivera <jack@token.nl>.

14/03/99 V1.18 Added OnCookie event.

16/03/99 V1.19 Added Accept property.

               Added a default value to Agent property.

               Changed OnCookie event signature (not fully implemented yet !).

07/05/99 V1.20 Added code to support Content Ranges by Jon Robertson

               <touri@pobox.com>.

24/07/99 V1.21 Yet another change in relocation code.

Aug 20, 1999  V1.22 Changed conditional compilation so that default is same

              as latest compiler (currently Delphi 4, Bcb 4). Should be ok for

              Delphi 5. Added Sleep(0) in sync wait loop to reduce CPU usage.

              Added DnsResult property as suggested by Heedong Lim

              <hdlim@dcenlp.chungbuk.ac.kr>. This property is accessible from

              OnStateChange when state is httpDnsLookupDone.

              Triggered OnDocData after writing to the stream.

Sep 25, 1999  V1.23 Yet another change in relocation code when using proxy

              Francois Demers <fdemers@videotron.ca> found that some webserver

              do not insert a space after colon in header line. Corrected

              code to handle it correctly.

              Cleared ContentType before issuing request.

Oct 02, 1999  V1.24 added AcceptRanges property. Thanks to Werner Lehmann

              <wl@bwl.uni-kiel.de>

Oct 30, 1999  V1.25 change parameter in OnCommand event from const to var to

              allow changing header line, including deleting or adding before

              or after a given line sent by the component.

Nov 26, 1999  V1.26 Yet another relocation fix !

Jun 23, 2000  V1.27 Fixed a bug in ParseURL where hostname is followed by a '?'

              (that is no path but a query).

Jul 22, 2000  V1.28 Handle exception during DnsLookup from the login procedure.

              Suggested by Robert Penz <robert.penz@outertech.com>

Sep 17, 2000  V1.29 Eugene Mayevski <Mayevski@eldos.org> added support for

              NOFORMS.

Jun 18, 2001  V1.30 Use AllocateHWnd and DeallocateHWnd from wsocket.

              Renamed property WSocket to CtrlSocket (this require code change

              in user application too).

Jul 25, 2001  V1.31 Danny Heijl <Danny.Heijl@cevi.be> found that ISA proxy adds

              an extra space to the Content-length header so we need a trim

              to extract numeric value.

              Ran Margalit <ran@margalit.com> found some server sending

              empty document (Content-Length = 0) which crashed the component.

              Added a check for that case when header is finished.

              Andrew N.Silich" <silich@rambler.ru> found we need to handle

              handle relative path using "../" and "./" when relocating. Thanks

              for his code which was a good starting point.

Jul 28, 2001  V1.32 Sahat Bun <sahat@operamail.com> suggested to change POST to

              GET when a relocation occurs.

              Created InternalClear procedure as suggested by Frank Plagge

              <frank@plagge.net>.

              When relocation, clear FRcvdHeader. If port not specified, then

              use port 80. By Alexander O.Kazachkin <kao@inreco.ru>

Jul 30, 2001 V1.33 Corected a few glitches with Delphi 1

Aug 18, 2001 V1.34 Corrected a bug in relocation logic: when server send only a

             header, with no document at all, relocation was not occuring and

             OnHeaderEnd event was not triggered.

             Corrected a bug in document name when a CGI was invoked (a '?'

             found in the URL). Now, ignore everything after '?' which is CGI

             parameter.

Sep 09, 2001 V1.35 Beat Boegli <leeloo999@bluewin.ch> added LocalAddr property

             for multihomed hosts.

Sep 29, 2001 V1.36 Alexander Alexishin <sancho@han.kherson.ua> corrected

             ParseUrl to handle the case where http:// is not at start of url:

             'first.domain.com/cgi-bin/serv?url=http://second.domain.com'

             Yet another relocation code change.

Oct 28, 2001 V1.37 Corrected SocketSessionClosed which called

             LocationSessionClosed when it was not needed.

Nov 10, 2001 V1.38 Fixed a bug where component was trying to connect to proxy

             using default port instead on specified port after a relocation.

             Corrected a bug when relocating to a relative path. Current path

             was not taken into account !

Mar 06, 2002 V1.39 Fixed a bug in relocation when content-length was 0: no

             relocation occured ! (Just check for relocation before checking

             content length empty).

Mar 12, 2002 V1.40 Added UrlEncode and UrlDecode utility functions.

Mar 30, 2002 V1.41 Export a few utility functions: IsDigit, IsXDigit, XDigit,

             htoin and htoi2.

Apr 14, 2002 V1.42 Paolo S. Asioli <paolo.asioli@libero.it> found a bug in

             relocation code where new user/pass are specified.

             On relocation, change DocName according to the relocation.

             When DocName has no extension and ContentType is text/html the

             add extension .htm (could be expanded do other content type)

Apr 20, 2002 V1.43 Added Socks code from Eugene Mayevski <mayevski@eldos.org>

Apr 21, 2002 V1.44 In LocationSessionClosed, clear status variables from

             previous operation.

Sep 06, 2002 V1.45 Made a few more methods virtual.

Sep 10, 2002 V1.46 Added AcceptLanguage property.

Sep 11, 2002 V1.47 Wilfried Mestdagh <wilfried@mestdagh.biz> added

             OnBeforeHeaderSend event to help add/remove/change header lines.

             He also corrected SocketSessionClosed to report error code.

Feb 08, 2003 V1.48 Implemented more HTTP/1.1 features notably TCP session

             persistance.

Feb 22, 2003 V1.49 Corrected a bug related to document length computation.

             Thanks to Dav999 <transmaster@ifrance.com> who found a

             reproductible case with perso.wanadoo.fr webserver.

Apr 27, 2003 V1.50 OnLocationChange was not called when a relocation occured

             and server handle HTTP/1.1 and new location on same server.

May 01, 2003 V1.51 Location and URL properties where incorrect after relocation

             to same HTTP/1.1 server.

             Change POST to GET after relocation to same HTTP/1.1 server.

May 09, 2003 V1.52 Implemented PUT method

May 31, 2003 V1.53 Corrected a problem with relocation when a proxy was used

             and a relative path was given.

Aug 21, 2003 V1.54 Removed HTTPCliDeallocateHWnd virtual attribute for BCB

             because of a bug in BCB preventing linking any derived

             component with a HWND argument in a virtual method.

             With help from Steven S. Showers and Stanislav Korotky.

Nov 26, 2003 V1.55 Implemented OnDataPush event for server push handling.

             Added OnSessionClosed event.

             Corrected ParseUrl to correctly handle protocol specified in

             uppercase. Thanks to "Nu Conteaza" <osfp@personal.ro>.

             Implemented OnDataPush2 event. Not the same as OnDataPush:

               OnDataPush: Need to call Receive to get data

               OnDataPush2: Data already received and delivered in LastResponse

Dec 28, 2003 V1.56 Implemented TransferEncoding = chunked

             Moved code for relocation after document receive so that even in

             case of a relocation, any available document is preserved.

             Thanks to Steve Endicott <Endi@pacbell.net> for his help.

Jan 09, 2004 V1.57 Fixed a relocation not done when end of document is in the

             same data packet as the last header line. With help of S. Endicott.

             Steve Endicott <Endi@pacbell.net> implemented "Connection: close"

             header line.

Jan 12, 2004 V1.58 "Ted T?raasen" <Ted@extreme.no> added proprety

             FollowRelocation (default to TRUE) to have the component follow

             relocation or just ignore them.

Jan 15, 2004 V1.59 Set FRcvdCount to zero in StartRelocation (S. Endicott).

             Started to implement NTLM authentication. Doesn't work yet !

Jan 26, 2004 V1.60 Reordered uses clause for FPC compatibility.

Feb 16, 2004 V1.61 Fixed GetHeaderLineNext to start relocation at the right

             moment. See annotation "16/02/2004".

Mar 12, 2004 Fixed GetHeaderLineNext to check for StatusCode < 200, 204 and

             304 in order to not wait for body.

             Thanks to Csonka Tibor <bee@rawbite.ro> for finding a test case.

Jul 12, 2004 Just this warning: The component now doesn't consider 401 status

             as a fatal error (no exception is triggered). This required a

             change in the application code if it was using the exception that

             is no more triggered for status 401.

Jul 18, 2004 V1.63 Use CompareText to check for http string is relocation

             header. Thanks to Roger Tinembart <tinembart@brain.ch>

Jul 23, 2004 V1.64 Fixed a line too long exception when requesting HEAD or URL

             http://de.news.yahoo.com:80/. The server was sending a document

             even after we requested just the header. The fix make the

             component ignore data and abort the connection. This is really an

             error at server side !

Aug 08, 2004 V1.65 Moved utility function related to URL handling into IcsUrl

             unit for easy reuse outside of the component.

Aug 20, 2004 V1.66 Use MsgWaitForMultipleObjects in DoRequestSync to avoid

             consumming 100% CPU while waiting.

Sep 04, 2004 V1.67 Csonka Tibor <bee@rawbite.ro> worked a lot on my NTLM code,

             fixing it and making it work properly.

             I removed NTLM specific usercode and password properties to use

             FUsername and FPassword which are extracted from the URL.

             Define symbol UseNTLMAuthentication for Delphi 5 and up.

Sep 13, 2004 V1.68 Added option httpoNoNTLMAuth by Csonka Tibor

             Fixed TriggerRequestDone for NTLM authentication

             Moved NTLM code out of DoBeforeConnect which was intended for

             socket setup and not for protocol handling.

Oct 02, 2004 V1.69 Removed second copy of IntToStrDef.

Oct 06, 2004 V1.70 Miha Remec fixed THttpCli.GetHeaderLineNext to add

             status check for 301 and 302 values.

Oct 15, 2004 V1.71 Lotauro.Maurizio@dnet.it enhanced basic and NTLM

             authentifications methods. Event OnNTLMAuthStep has been

             removed. Now basic authentication is not automatically sent with

             a request. It is only sent when the server request it by replying

             with a 401 or 407 response. Sending basic authentication in the

             first request was a kind of security threat for NTLM:

             usercode/password is sent unencrypted while NTLM is made to send

             it encrypted (DES). This has the side effect of requiring two

             request where one was needed. This could be a problem when posting

             data: data has to be posted twice ! This is transparent to the user

             except for performance :-( A future enhancement could be a new

             option to always send basic authentication.

Oct 30, 2004 V1.72 Made SendRequest virtual.

Nov 07, 2004 V1.73 Added CleanupRcvdStream. Lotauro.Maurizio@dnet.it found that

             document must be cleaned if received in intermediate authentication

             steps.

Nov 09, 2004 V1.74 Cleared FDoAuthor from InternalClear. Thanks Maurizio.

Nov 11, 2004 V1.75 Added CleanupRcvdStream when starting relocation.

             Thanks Maurizio.

             Removed second TriggerHeaderEnd in GetHeaderLineNext.

             Thanks Ronny Karl for finding this one.

Nov 20, 2004 V1.76 Angus Robertson found a problem with SendStream because of

             authentication (post restarted because authentication need to be

             done).

             Maurizio fixed the issue above an a fix others:

             - added a CleanupSendStream procedure, and added a call to it in

               every place where the CleanupRcvdStream is called.

             - changed the Content-Length calculation: if the position of the

               stream is not 0 then the length was wrong

             - changed the the test in DoRequestAsync: if the position of the

               stream is at the end then it will send nothing

Nov 22, 2004 V1.77 Only a single error code for httperrInvalidAuthState.

Dec 14, 2004 V1.78 Excluded code 407 and added code 400 in DoRequestSync

Dec 19, 2004 V1.79 Revised CleanupRcvdStream to make it compatible with D1.

Dec 22, 2004 V1.80 Changed SocketDataAvailable so that header lines that are

             too long to fit into the receive buffer (8K) are simply truncated

             instead of triggering an exception.

Jan 05, 2005 V1.81 Maurizio Lotauro <Lotauro.Maurizio@dnet.it> optimized NTLM

             authentication by not sending content in the first step.

Jan 29, 2005 V1.82 Fixed socks properties propagation to control socket.

Feb 05, 2005 V1.83 Fixed GetHeaderLineNext in the case Abort is called from

             OnHeaderEnd event (Bug reported by Csonka Tibor).

             Fixed relocation to https destination when USE_SSL is not

             defined. It is handled as a non implemented protocol.

Mar 19, 2005 V1.84 Changed CleanupRcvdStream to check for COMPILER3_UP instead

             of checking DELPHI3_UP (BCB compatibility issue).

             Changed StrToIntDef to check for COMPILER5_UP instead of

             DELPHI5_UP (BCB compatibility issue). Thanks to Albert Wiersch.

Mar 21, 2005 V1.85 Added port in "host:" header line as suggested by Sulimov

             Valery <99valera99@rambler.ru>.

             In DoRequestAsync, allow to continue even with no data to be sent.

Apr 14, 2005 V1.86 Fixed PrepareBasicAuth to ignore charcase when checking for

             'basic' keyword in header line. Thanks to abloms@yahoo.com for

             finding this bug which affected use thru iPlanet Web Proxy Server.

Apr 16, 2005 V1.87 Applyed the fix above to PrepareNTLMAuth (two places),

             StartAuthNTLM and StartProxyAuthNTLM.

May 26, 2005 V1.88 Fixed DoRequestAsync to set http as default protocol.

Aug 15, 2005 V1.89 Implemented bandwidth control for download (Get).

Oct 31, 2005 V1.89a rework of authentication handling to fix a problem that could

               happen when both proxy and remote server needed an authentication

               causing an infinite loop, by Maurizio Lotauro

             - removed the TXXXAuthType type and the variables of that types

             - properties Username and Password will no more replaced by the

               credential contained in the URL. Two new variables are introduced

               FCurrUsername and FCurrPassword. These will contain the credential

               in the URL (if specified), otherwise the content of the property.

            - same logic for Connection and ProxyConnection (introduced

              FCurrConnection and FCurrProxyConnection)

Nov 19, 2005 V1.89b supports SSL v5 by Arno Garrels 

Nov 27, 2005 V1.90 implemented redirection limiting to avoid continuous

             looping to same URL, by Angus Robertson, Magenta Systems

             test example is http://www.callserve.com/ that keeps looping

             Implemented ContentEncoding (can be disabled with define

             UseContentCoding), mostly done in new HttpContCod.pas,

             add HttpCCodzlib to your project for GZIP compression support,

             and set httpoEnableContentCoding in Options to enable it

             by Maurizio Lotauro <Lotauro.Maurizio@dnet.it>

Dec 20, 2005 V1.91 new configurable DebugOptions to replace IFDEF DEBUG_OUTPUT,

             see wsocket for more information 

Apr 10, 2006 V6.00.1 Added LowerCase for FTransferEncoding? Thanks to Fastream.

Dec 10, 2006 V6.00.2 Jack <jlist9@gmail.com> fixed BandwidthTimerTimer to clear

             the count.

Mar 17, 2007 V6.00.3 Introduced THttpBigInt to support documents longer

             than 2GB.

Mar 19, 2007 V6.00.3 A.Garrels fixed a memory leak of FSendBuffer and

             FReceiveBuffer. Check for negative value of FReceiveLen in

             SocketDataAvailable.

May 27, 2008 V6.00.4 A.Garrels Workaround in GetHeaderLineNext. Ignore body data

             sent in the HEAD response by buggy servers.

Jun 25, 2008 V6.00.5 A. Garrels SSL code merged.

Jul 17, 2008 V6.00.6 A. Garrels made a few changes to prepare code for Unicode,

             added OverbyteIcsUtils to the uses clause, optimized

             MoveTBytesToString a bit, removed some stuff for older compilers,

             removed typedef of TBytes, now uses TBytes of OverbyteIcsTypes.pas.

Sep 28, 2008 V6.00.7 Maurizio Lotauro fixed a bug with premature received

             401/407 responses while data was still being sent with POST and PUT

             requests. A. Garrels small fix in SendCommand().

Sep 29, 2008 V6.00.8 A. Garrels added OverbyteIcsUtils to the uses clause

             for all compilers.

Dec 06, 2008 V7.00.9 A. Garrels fixed function EncodeStr and EncodeLine.

Jan 11, 2009 V7.01 A. Garrels fixed a bug with proxies which do not

             send a Content-Length header in non-persistent connections by

             simply always setting FReceiveLen to zero in SocketSessionClosed.

Jan 11, 2009 V7.02 A. Garrels - Added Digest Access Authentication.

             In order to disable this feature entirely undefine directive

             UseDigestAuthentication below.

             Added new event OnBeforeAuth that can be used to skip internal

             authorization by setting argument Allow to FALSE.

             ** Also cleaned up the source code a bit, thus a comparison

             with previous version, unfortunately won't be fun. **

Jan 12, 2009 V7.02a Arno added two missing lines for digest auth to the SSL code.

Jan 22, 2009 V7.02b Sorry guys! Re-added property OnBeforeHeaderSend again

             which I (Arno) removed in V7.02 accidently :(

Jan 22, 2009 V7.02c Arno - Conditional define UseDigestAuthentication was not

             set properly in THttpCli.StateChange.

Apr 25, 2009 V7.03 Steve Endicott fixed a relocation bug with HTTPS.

Sep 17, 2009 V7.04 Arno added property Timeout, works only with synchronous

             methods!

Dec 02, 2009 V7.05 Bjornar found a HTTPS POST bug with proxy basic

             authentication that added two Content-Length header lines.

Feb 15, 2010 V7.06 Yuri Semenov fixed a bug with content coding and chunked

             transfer encoding.

Feb 25, 2010 V7.07 Fix by Bj?rnar Nielsen: TSslHttpCli didn't work when used

             against Websense-Content_Gateway (http://www.websense.com) and 

             some others. The problem was that this (and some other proxies too)

             answer 200 OK to notify client that connection to remote server 

             is established. Usually proxies use 200 OK and an error text when

             something is wrong. In that case Content-Length is not 0.

May 24, 2010 V7.08 Angus ensure Ready when relocations exceed maximum to avoid timeout





 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}

unit OverbyteIcsHttpProt;



interface



{$I OverbyteIcsDefs.inc}

{$IFDEF COMPILER14_UP}

  {$IFDEF NO_EXTENDED_RTTI}

    {$RTTI EXPLICIT METHODS([]) FIELDS([]) PROPERTIES([])}

  {$ENDIF}

{$ENDIF}

{$IFNDEF COMPILER7_UP}

    'Sorry, we do not want to support ancient compilers any longer'

{$ENDIF}

{$IFDEF DELPHI6_UP}

    {$WARN SYMBOL_PLATFORM   OFF}

    {$WARN SYMBOL_LIBRARY    OFF}

    {$WARN SYMBOL_DEPRECATED OFF}

{$ENDIF}

{$B-}             { Enable partial boolean evaluation   }

{$T-}             { Untyped pointers                    }

{$X+}             { Enable extended syntax              }

{$H+}             { Use long strings                    }

{$J+}             { Allow typed constant to be modified }

{$IFDEF BCB3_UP}

    {$ObjExportAll On}

{$ENDIF}

{$IFNDEF NO_ADVANCED_HTTP_CLIENT_FEATURES}

    {$DEFINE UseNTLMAuthentication}

    {$DEFINE UseDigestAuthentication}

    {$DEFINE UseBandwidthControl}

    {$DEFINE UseContentCoding}

{$ENDIF}

{$IFDEF CLR}

    {$UNDEF UseNTLMAuthentication}

    {$UNDEF UseDigestAuthentication}

    {$UNDEF UseContentCoding}

{$ENDIF}



uses

    Messages,

{$IFDEF USEWINDOWS}

    Windows,

{$ELSE}

    WinTypes, WinProcs,

{$ENDIF}

{$IFDEF CLR}

    System.Text, System.IO, System.Threading,

{$ENDIF}

    SysUtils, Classes,

{$IFNDEF NOFORMS}

    Forms, Controls,

{$ENDIF}

{ You must define USE_SSL so that SSL code is included in the component.   }

{ Either in OverbyteIcsDefs.inc or in the project/package options.         }

{$IFDEF USE_SSL}

    OverbyteIcsSSLEAY, OverbyteIcsLIBEAY,

{$ENDIF}

{$IFDEF UseNTLMAuthentication}

    OverbyteIcsNtlmMsgs,

{$ENDIF}

{$IFDEF UseContentCoding}

    OverbyteIcsHttpContCod,

{$ENDIF}

{$IFNDEF NO_DEBUG_LOG}

    OverbyteIcsLogger,

{$ENDIF}

    OverbyteIcsUrl, OverbyteIcsTypes,

    OverbyteIcsUtils,

{$IFDEF UseDigestAuthentication}

    OverbyteIcsDigestAuth,

{$ENDIF}

    OverbyteIcsWinSock, OverbyteIcsWndControl, OverbyteIcsWSocket;



const

    HttpCliVersion       = 708;

    CopyRight : String   = ' THttpCli (c) 1997-2010 F. Piette V7.08 ';

    DefaultProxyPort     = '80';

    HTTP_RCV_BUF_SIZE    = 8193;

    HTTP_SND_BUF_SIZE    = 8193;

    { EHttpException error code }

    httperrNoError                  = 0;

    httperrBusy                     = 1;

    httperrNoData                   = 2;

    httperrAborted                  = 3;

    httperrOverflow                 = 4;

    httperrVersion                  = 5;

    httperrInvalidAuthState         = 6;

    httperrSslHandShake             = 7;

    httperrCustomTimeOut            = 8;  { V7.04 }



type

    THttpBigInt = Int64;



    EHttpException = class(Exception)

        ErrorCode : Word;

        constructor Create(const Msg : String; ErrCode : Word);

    end;



    // TBytes = array of Byte;



    THttpEncoding    = (encUUEncode, encBase64, encMime);

    THttpRequest     = (httpABORT, httpGET, httpPOST, httpPUT,

                        httpHEAD,  httpCLOSE);

    THttpState       = (httpReady,         httpNotConnected, httpConnected,

                        httpDnsLookup,     httpDnsLookupDone,

                        httpWaitingHeader, httpWaitingBody,  httpBodyReceived,

                        httpWaitingProxyConnect,

                        httpClosing,       httpAborting);

    THttpChunkState  = (httpChunkGetSize, httpChunkGetExt, httpChunkGetData,

                        httpChunkSkipDataEnd, httpChunkDone);

{$IFDEF UseNTLMAuthentication}

    THttpNTLMState   = (ntlmNone, ntlmMsg1, ntlmMsg2, ntlmMsg3, ntlmDone);

{$ENDIF}

{$IFDEF UseDigestAuthentication}

    TAuthDigestInfo  = TAuthDigestResponseInfo;

    THttpDigestState = (digestNone, digestMsg1, digestDone);

{$ENDIF}

    THttpBasicState  = (basicNone, basicMsg1, basicDone);

    THttpAuthType    = (httpAuthNone, httpAuthBasic, httpAuthNtlm, httpAuthDigest);

    THttpBeforeAuthEvent   = procedure(Sender  : TObject;

                                 AuthType      : THttpAuthType;

                                 ProxyAuth     : Boolean;

                                 const AuthHdr : String;

                                 var Allow     : Boolean) of object;

    TOnCommand       = procedure (Sender : TObject;

                                  var S: String) of object;

    TDocDataEvent    = procedure (Sender : TObject;

                                  {$IFDEF CLR}

                                  var Buffer : TBytes;

                                  Offset     : Integer;

                                  {$ELSE}

                                  Buffer : Pointer;

                                  {$ENDIF}

                                  Len    : Integer) of object;

    TCookieRcvdEvent = procedure (Sender       : TObject;

                                  const Data   : String;

                                  var   Accept : Boolean) of object;

    THttpRequestDone = procedure (Sender  : TObject;

                                  RqType  : THttpRequest;

                                  ErrCode : Word) of object;

    TBeforeHeaderSendEvent = procedure (Sender       : TObject;

                                        const Method : String;

                                        Headers      : TStrings) of object;

    THttpCliOption = (httpoNoBasicAuth, httpoNoNTLMAuth, httpoBandwidthControl,

                  {$IFDEF UseContentCoding}

                      httpoEnableContentCoding, httpoUseQuality,

                  {$ENDIF}

                      httpoNoDigestAuth);

    THttpCliOptions = set of THttpCliOption;

    TLocationChangeExceeded = procedure (Sender              : TObject;

                                  const RelocationCount      : Integer;

                                  var   AllowMoreRelocations : Boolean) of object;  {  V1.90 }





    THttpCli = class(TIcsWndControl)

    protected

        FMsg_WM_HTTP_REQUEST_DONE : UINT;

        FMsg_WM_HTTP_SET_READY    : UINT;

        FMsg_WM_HTTP_LOGIN        : UINT;

        FCtrlSocket           : TWSocket;

        //FWindowHandle         : HWND;

        FMultiThreaded        : Boolean;

        FState                : THttpState;

        FLocalAddr            : String;

        FHostName             : String;

        FTargetHost           : String;

        FTargetPort           : String;

        FPort                 : String;

        FProtocol             : String;

        FProxy                : String;

        FProxyPort            : String;

        FUsername             : String;

        FPassword             : String;

        FCurrUsername         : String;

        FCurrPassword         : String;

        FProxyUsername        : String;

        FProxyPassword        : String;

        FProxyConnected       : Boolean;

        FLocation             : String;

        FCurrentHost          : String;

        FCurrentPort          : String;

        FCurrentProtocol      : String;

        FConnected            : Boolean;

        FDnsResult            : String;

//      FSendBuffer           : array [0..HTTP_SND_BUF_SIZE - 1] of char;

        FSendBuffer           : TBytes;  // FP 09/09/06

        FRequestType          : THttpRequest;

//      FReceiveBuffer        : array [0..HTTP_RCV_BUF_SIZE - 1] of char;

        FReceiveBuffer        : TBytes;  // FP 09/09/06

        FReceiveLen           : Integer;

        FLastResponse         : String;

        FHeaderLineCount      : Integer;

        FBodyLineCount        : Integer;

        FAllowedToSend        : Boolean;

        FDelaySetReady        : Boolean;        { 09/26/08 ML }

        FURL                  : String;

        FPath                 : String;

        FDocName              : String;

        FSender               : String;

        FReference            : String;

        FConnection           : String;         { for Keep-alive }

        FProxyConnection      : String;         { for proxy keep-alive }

        FCurrConnection       : String;

        FCurrProxyConnection  : String;

        FAgent                : String;

        FAccept               : String;

        FAcceptLanguage       : String;

        FModifiedSince        : TDateTime;      { Warning ! Use GMT date/Time }

        FNoCache              : Boolean;

        FStatusCode           : Integer;

        FReasonPhrase         : String;

        FResponseVer          : String;

        FRequestVer           : String;

        FContentLength        : THttpBigInt;

        FContentType          : String;

        FTransferEncoding     : String;

{$IFDEF UseContentCoding}

        FContentEncoding      : String;

        FContentCodingHnd     : THttpContCodHandler;

        FRcvdStreamStartSize  : Integer;

{$ENDIF}

        FChunkLength          : Integer;

        FChunkRcvd            : Integer;

        FChunkState           : THttpChunkState;

        FDoAuthor             : TStringList;

        FContentPost          : String;         { Also used for PUT }

        FContentRangeBegin    : String;

        FContentRangeEnd      : String;

        FAcceptRanges         : String;

        FCookie               : String;

        FLocationFlag         : Boolean;

        FFollowRelocation     : Boolean;    {TED}

        FHeaderEndFlag        : Boolean;

        FRcvdHeader           : TStrings;

        FRcvdStream           : TStream;     { If assigned, will recv the answer }

        FRcvdCount            : THttpBigInt; { Number of rcvd bytes for the body }

        FSentCount            : THttpBigInt;

        FSendStream           : TStream; { Contains the data to send         }

        FReqStream            : TMemoryStream;

        FRequestDoneError     : Integer;

        FNext                 : procedure of object;

//      FBodyData             : PChar;

        FBodyData             : Integer;  // Offset in FReceiveBuffer (FP 09/09/06)

        FBodyDataLen          : THttpBigInt;

        FOptions              : THttpCliOptions;

        FSocksServer          : String;

        FSocksLevel           : String;

        FSocksPort            : String;

        FSocksUsercode        : String;

        FSocksPassword        : String;

        FSocksAuthentication  : TSocksAuthentication;

{$IFDEF UseNTLMAuthentication}

        FNTLMMsg2Info         : TNTLM_Msg2_Info;

        FProxyNTLMMsg2Info    : TNTLM_Msg2_Info;

        FAuthNTLMState        : THttpNTLMState;

        FProxyAuthNTLMState   : THttpNTLMState;

{$ENDIF}

{$IFDEF UseDigestAuthentication}

        FAuthDigestState      : THttpDigestState;

        FProxyAuthDigestState : THttpDigestState;

        FAuthDigestInfo       : TAuthDigestInfo;

        FAuthDigestProxyInfo  : TAuthDigestInfo;

        { As specified in RFC 2617, section 3.2.2.4, used only with auth-int }

        FAuthDigestEntityHash : THashHex;

{$ENDIF}

        FOnBeforeAuth         : THttpBeforeAuthEvent;

        FAuthBasicState       : THttpBasicState;

        FProxyAuthBasicState  : THttpBasicState;

        //FServerAuth           : String;

        //FProxyAuth            : String;

        FServerAuth           : THttpAuthType;

        FProxyAuth            : THttpAuthType;

{$IFDEF UseBandwidthControl}

        FBandwidthLimit       : Integer;  // Bytes per second

        FBandwidthSampling    : Integer;  // mS sampling interval

        FBandwidthCount       : Int64;    // Byte counter

        FBandwidthMaxCount    : Int64;    // Bytes during sampling period

        FBandwidthTimer       : TIcsTimer;

        FBandwidthPaused      : Boolean;

        FTimerOldEnabled      : Boolean;

{$ENDIF}

        FOnStateChange        : TNotifyEvent;

        FOnSessionConnected   : TNotifyEvent;

        FOnSessionClosed      : TNotifyEvent;

        FOnRequestHeaderBegin : TNotifyEvent;

        FOnRequestHeaderEnd   : TNotifyEvent;

        FOnHeaderBegin        : TNotifyEvent;

        FOnHeaderEnd          : TNotifyEvent;

        FOnHeaderData         : TNotifyEvent;

        FOnDocBegin           : TNotifyEvent;

        FOnDocEnd             : TNotifyEvent;

        FOnDocData            : TDocDataEvent;

        FOnSendBegin          : TNotifyEvent;

        FOnSendEnd            : TNotifyEvent;

        FOnSendData           : TDocDataEvent;

        FOnTrace              : TNotifyEvent;

        FOnCommand            : TOnCommand;

        FOnCookie             : TCookieRcvdEvent;

        FOnDataPush           : TDataAvailable;

        FOnDataPush2          : TNotifyEvent;

        FOnRequestDone        : THttpRequestDone;

        FOnLocationChange     : TNotifyEvent;

        FLocationChangeMaxCount   : Integer;  {  V1.90 }

        FLocationChangeCurCount   : Integer;  {  V1.90 }

        FOnLocationChangeExceeded : TLocationChangeExceeded;  {  V1.90 }

        { Added by Eugene Mayevski }

        FOnSocksConnected     : TSessionConnected;

        FOnSocksAuthState     : TSocksAuthStateEvent;

        FOnSocksError         : TSocksErrorEvent;

        FOnSocketError        : TNotifyEvent;

        FOnBeforeHeaderSend   : TBeforeHeaderSendEvent;     { Wilfried 9 sep 02}

        FCloseReq             : Boolean;                    { SAE 01/06/04 }

        FTimeout              : UINT;  { V7.04 }            { Sync Timeout Seconds }

        procedure AllocateMsgHandlers; override;

        procedure FreeMsgHandlers; override;

        function  MsgHandlersCount: Integer; override;

        procedure CheckDelaySetReady;                       { 09/26/08 ML }

{$IFNDEF NO_DEBUG_LOG}

        function  GetIcsLogger: TIcsLogger;                 { V1.91 }

        procedure SetIcsLogger(const Value: TIcsLogger);    { V1.91 }

        procedure DebugLog(LogOption: TLogOption; const Msg : string); virtual;   { V1.91 }

        function  CheckLogOptions(const LogOption: TLogOption): Boolean; virtual; { V1.91 }

{$ENDIF}

{$IFDEF UseBandwidthControl}

        procedure BandwidthTimerTimer(Sender : TObject);

{$ENDIF}

        procedure CreateSocket; virtual;

        procedure DoBeforeConnect; virtual;

        procedure DoSocksConnected(Sender: TObject; ErrCode: Word);

        procedure DoSocksAuthState(Sender : TObject; AuthState : TSocksAuthState);

        procedure DoSocksError(Sender : TObject; ErrCode : Integer; Msg : String);

        procedure SocketErrorTransfer(Sender : TObject);

        procedure SendRequest(const method, Version: String); virtual;

        procedure GetHeaderLineNext; virtual;

        procedure GetBodyLineNext; virtual;

        procedure SendCommand(const Cmd : String); virtual;

        procedure Login; virtual;

        procedure Logout; virtual;

        procedure InternalClear; virtual;

        procedure StartRelocation; virtual;

{$IFDEF UseNTLMAuthentication}

        procedure StartAuthNTLM; virtual;

        procedure StartProxyAuthNTLM; virtual;  {BLD proxy NTLM support }

        function  GetNTLMMessage1(const ForProxy: Boolean) : String;

        function  GetNTLMMessage3(const HttpMethod: String;

            const ForProxy: Boolean): String;

        procedure ElaborateNTLMAuth;

        function  PrepareNTLMAuth(var FlgClean : Boolean) : Boolean;

{$ENDIF}

{$IFDEF UseDigestAuthentication}

        procedure ElaborateDigestAuth;

        function  GetDigestAuthorizationHeader(const HttpMethod: String;

           ProxyAuth : Boolean): String;

        function  PrepareDigestAuth(var FlgClean : Boolean) : Boolean;

        procedure StartAuthDigest; virtual;

        procedure StartProxyAuthDigest; virtual;

{$ENDIF}

        function GetBasicAuthorizationHeader(

            const HttpMethod: String; ProxyAuth: Boolean): String;

        procedure CleanupRcvdStream;

        procedure CleanupSendStream;

        procedure StartAuthBasic; virtual;

        procedure StartProxyAuthBasic; virtual;

        procedure ElaborateBasicAuth;

        function  PrepareBasicAuth(var FlgClean : Boolean) : Boolean;

        procedure SocketDNSLookupDone(Sender: TObject; ErrCode: Word); virtual;

        procedure SocketSessionClosed(Sender: TObject; ErrCode: Word); virtual;

        procedure SocketSessionConnected(Sender : TObject; ErrCode : Word); virtual;

        procedure SocketDataSent(Sender : TObject; ErrCode : Word); virtual;

        procedure SocketDataAvailable(Sender: TObject; ErrCode: Word); virtual;

        function  StartsWithText(Source : TBytes; Find : PAnsiChar) : Boolean; {Bjornar}

        function  ContainsText(Source : TBytes; Find : PAnsiChar) : Boolean; {Bjornar}

        procedure LocationSessionClosed(Sender: TObject; ErrCode: Word); virtual;

        procedure DoRequestAsync(Rq : THttpRequest); virtual;

        procedure DoRequestSync(Rq : THttpRequest); virtual;

        procedure SetMultiThreaded(newValue : Boolean); virtual;

        procedure StateChange(NewState : THttpState); virtual;

        procedure TriggerStateChange; virtual;

        procedure TriggerCookie(const Data : String;

                                var   bAccept : Boolean); virtual;

        procedure TriggerSessionConnected; virtual;

        procedure TriggerSessionClosed; virtual;

        procedure TriggerBeforeHeaderSend(const Method : String;

                                          Headers : TStrings); virtual;

        procedure TriggerRequestHeaderBegin; virtual;

        procedure TriggerRequestHeaderEnd; virtual;

        procedure TriggerHeaderBegin; virtual;

        procedure TriggerHeaderEnd; virtual;

        procedure TriggerDocBegin; virtual;

        procedure TriggerDocData(

{$IFDEF CLR}

            var Data : TBytes;

            Offset   : Integer;

{$ELSE}

            Data : Pointer;

{$ENDIF}

            Len : Integer); virtual;

        procedure TriggerDocEnd; virtual;

        procedure TriggerSendBegin; virtual;

        procedure TriggerSendData(

{$IFDEF CLR}

            var Data : TBytes;

            Offset   : Integer;

{$ELSE}

            Data     : Pointer;

{$ENDIF}

            Len      : Integer); virtual;

        procedure TriggerSendEnd; virtual;

        procedure TriggerRequestDone; virtual;

        procedure WndProc(var MsgRec: OverbyteIcsTypes.TMessage); override;

        procedure SetReady; virtual;

        procedure AdjustDocName; virtual;

        procedure SetRequestVer(const Ver : String);

        procedure WMHttpRequestDone(var msg: TMessage);

        procedure WMHttpSetReady(var msg: TMessage);

        procedure WMHttpLogin(var msg: TMessage);

{$IFDEF UseContentCoding}

        function  GetOptions: THttpCliOptions;

        procedure SetOptions(const Value : THttpCliOptions);

{$ENDIF}

{$IFDEF USE_SSL}

        procedure SslHandshakeDone(Sender         : TObject;

                                   ErrCode        : Word;

                                   PeerCert       : TX509Base;

                                   var Disconnect : Boolean);

{$ENDIF}

    public

        constructor Create(Aowner:TComponent); override;

        destructor  Destroy; override;

        procedure   Get;        { Synchronous blocking Get         }

        procedure   Post;       { Synchronous blocking Post        }

        procedure   Put;        { Synchronous blocking Put         }

        procedure   Head;       { Synchronous blocking Head        }

        procedure   Close;      { Synchronous blocking Close       }

        procedure   Abort;      { Synchrounous blocking Abort      }

        procedure   GetASync;   { Asynchronous, non-blocking Get   }

        procedure   PostASync;  { Asynchronous, non-blocking Post  }

        procedure   PutASync;   { Asynchronous, non-blocking Put   }

        procedure   HeadASync;  { Asynchronous, non-blocking Head  }

        procedure   CloseAsync; { Asynchronous, non-blocking Close }

        procedure   ThreadAttach; override;

        procedure   ThreadDetach; override;

        property CtrlSocket           : TWSocket     read  FCtrlSocket;

        //property Handle               : HWND         read  FWindowHandle;

        property State                : THttpState   read  FState;

        property LastResponse         : String       read  FLastResponse;

        property ContentLength        : THttpBigInt  read  FContentLength;

        property ContentType          : String       read  FContentType;

        property TransferEncoding     : String       read  FTransferEncoding;

{$IFDEF UseContentCoding}

        property ContentEncoding      : String       read FContentEncoding;

        property ContentCodingHnd     : THttpContCodHandler read FContentCodingHnd;

{$ENDIF}

        property RcvdCount            : THttpBigInt  read  FRcvdCount;

        property SentCount            : THttpBigInt  read  FSentCount;

        property StatusCode           : Integer      read  FStatusCode;

        property ReasonPhrase         : String       read  FReasonPhrase;

        property DnsResult            : String       read  FDnsResult;

        property AuthorizationRequest : TStringList  read  FDoAuthor;

        property DocName              : String       read  FDocName;

        property Location             : String       read  FLocation

                                                     write FLocation;

        property RcvdStream           : TStream      read  FRcvdStream

                                                     write FRcvdStream;

        property SendStream           : TStream      read  FSendStream

                                                     write FSendStream;

        property RcvdHeader           : TStrings     read  FRcvdHeader;

        property Hostname             : String       read  FHostname;

        property Protocol             : String       read  FProtocol;

{$IFDEF UseDigestAuthentication}

        property AuthDigestInfo       : TAuthDigestInfo

                                                     read  FAuthDigestInfo

                                                     write FAuthDigestInfo;

        property AuthDigestProxyInfo  : TAuthDige…