Showing posts with label InternetExplorer. Show all posts
Showing posts with label InternetExplorer. Show all posts

June 9, 2013

Dynamic web page using Delphi, ICS and DWScript


ICS has a web application server component which allows you to build dynamic web page very easily. Delphi code for each web page is encapsulated in a TUrlHandler class and compiled into your application, making a standalone webserver application.

In the article, we will create a TUrlHandler class which will read a DWScript script from disc and execute it. The script is responsible for build a valid answer which ICS will send back to the client.

A simple “Hello World” script is made of a single line like this:

   Response.Write('Server time is ' + DateTimeToStr(Now) + ' ');

To invoke it, assuming the script is located in “Hello.pas”, the user must enter this URL into his browser:
http://localhost:20105/DWScripts/Hello.pas

Remember it is a script. You can change the file on disc and the changes will be immediately reflected for the next request, without recompiling your application. This looks much like PHP but use Delphi syntax instead of PHP. To be honest, PHP may use embedded HTML code within the script which is not supported here.

A more complex script may make use of request parameters. For example, the script “Add.pas” could looks like:

var Value1 : String;
var Value2 : String;
Response.ContentType := 'text/html';
Response.Status := '200 OK';
Response.Write('');
Response.Write('ICS and DWScript demo
');
Response.Write('Server time is ' + DateTimeToStr(Now) + '
');
if not Request.CheckParamByName('Value1', Value1) then
    Response.Write('Missing Value1 parameter
')
else if not Request.CheckParamByName('Value2', Value2) then
    Response.Write('Missing Value2 parameter
')
else Response.Write(Value1 + ' + ' + Value2 + ' = ' + 
    IntToStr(StrToIntDef(Value1, 0) + StrToIntDef(Value2, 0)));
Response.Write('');

The URL to use in the browser looks like:
http://localhost:20105/DWScripts/Add.pas?Value1=123&Value2=456

Getting ICS and DWScript

ICS:    http://wiki.overbyte.be/wiki/index.php/ICS_Download
DWScript: http://code.google.com/p/dwscript/

Implementation


Less than 200 Delphi code is required to implement this behavior in an ICS based web application server. The code I will show you below can be plugged in OverbyteIcsWebAppServer demo application you can find in ICS V8 distribution. Two lines must be added to the main file in order to add the feature: add the unit in the uses clause and add a line to map “/DWScript/*” to the TUrlHandler taking care of the script.

Using the above wild card mapping, the TUrlHandler will be invoked for any URL beginning with “/DWScript/”. It will then access the full path to get the script filename. In the above examples, I used a “.pas” file extension for ease, but this is not mandatory at all. You can use any extensions and even no extension if you don’t like to have the user know you are using a DWScript.

In the implementation, I managed to have the DWScript not directly accessible thru an URL. This protects your source code. The user can’t access it. He can just execute it (Note that ICS THttpServer component has an option which allows access to any file, so what I just said maybe wrong).

The script has two objects instances readily available: “Request” and “Response”. They maps to the corresponding Delphi object instances and classes:

    THttpResponse = class(TPersistent)
    private
        FStatus      : String;
        FContentType : String;
    public
        DocStream    : TStream;
    published
        property Status      : String read FStatus      write FStatus;
        property ContentType : String read FContentType write FContentType;
        procedure Write(const S : String);
    end;

    THttpRequest = class(TPersistent)
    public
        Params : String;
        class function ReadTextFile(const FileName : String) : String;
    published
        function  GetParamByName(const ParamName: String): String;
        function  CheckParamByName(const ParamName  : String;
                                   var   ParamValue : String): Boolean;
    end;

When using DWScript ExposeRTTI function, you are exposing a Delphi class. With the options I selected, only the published methods and properties will be exposed to the script.

THttpResponse exposes Status and ContentType properties as well as Write method. The two properties allows the script to select the HTTP status return and the HTTP content type. They default to “200 OK” and “text/html” which are the most common values. You can use anything you need for your application.

The Write method will be used by the script to produce the document part of the HTTP response sent back to the client. Obviously, the document format must match the content type.

THttpRequest exposes the incoming request. Using it you may access the parameters passed thru the URL. One method gets a parameter value when you provides his name while the other return the value as well as a Boolean value telling if the parameter exists or not.

The most important part of the code is a TUrlHandler derived class. I named it TUrlHandlerDWScript. His declaration looks like this:

    TUrlHandlerDWScript = class(TUrlHandler)
    protected
        FScript                  : IdwsProgram;
        FCompileMsgs             : String;
        FDelphiWebScript         : TDelphiWebScript;
        FUnit                    : TdwsUnit;
        FExec                    : IdwsProgramExecution;
        FHttpRequest             : THttpRequest;
        FHttpResponse            : THttpResponse;
        procedure ExposeInstancesAfterInitTable(Sender: TObject);
    public
        procedure Execute; override;
    end;

The five first member variables are required for DWScript operation. You should look at DWScript documentation for help about their use.

The last two member variables are the object instances we talk above. Their published parts are exposed to the script.

There is also an event handler ExposeInstancesAfterInitTable which is required by DWScript engine to expose the Delphi object instances to the script.

Finally, there is a single method which is called by ICS web application server framework to handle the mapped URL. This is where almost everything happens.

procedure TUrlHandlerDWScript.Execute;
var
    SrcFileName : String;
    Source      : String;
begin
    FDelphiWebScript  := TDelphiWebScript.Create(nil);
    FUnit             := TdwsUnit.Create(nil);
    FHttpResponse     := THttpResponse.Create;
    FHttpRequest      := THttpRequest.Create;
    try
        DocStream.Free;
        DocStream := TMemoryStream.Create;

        FHttpResponse.DocStream    := DocStream;
        FHttpResponse.Status       := '200 OK';
        FHttpResponse.ContentType  := 'text/html';
        FHttpRequest.Params        := Params;

        FUnit.OnAfterInitUnitTable := ExposeInstancesAfterInitTable;
        FUnit.UnitName             := 'WebPage';
        FUnit.Script               := FDelphiWebScript;
        FUnit.ExposeRTTI(TypeInfo(THttpResponse), [eoNoFreeOnCleanup]);
        FUnit.ExposeRTTI(TypeInfo(THttpRequest),  [eoNoFreeOnCleanup]);
        // In this application, we have placed DWScripts source code in
        // a directory at the same level as the "Template" folder.
        SrcFileName := ExcludeTrailingPathDelimiter(
                           ExtractFilePath(Client.TemplateDir)) +
                       StringReplace(Client.Path, '/', '\', [rfReplaceAll]);
        if not FileExists(SrcFileName) then
            FHttpResponse.Write('Script not found')
        else begin
            Source       := FHttpRequest.ReadTextFile(SrcFileName);
            FScript      := FDelphiWebScript.Compile(Source);
            FCompileMsgs := FScript.Msgs.AsInfo;
            if FScript.Msgs.HasErrors then begin
                FHttpResponse.Write('' + FCompileMsgs + '');
            end
            else begin
                FExec    := FScript.Execute;
            end;
        end;
        AnswerStream(FHttpResponse.Status, FHttpResponse.ContentType, NO_CACHE);
    finally
        FreeAndNil(FUnit);
        FreeAndNil(FDelphiWebScript);
        FreeAndNil(FHttpResponse);
        FreeAndNil(FHttpRequest);
    end;
    Finish;
end;


Basically the code creates all the required object instances, initializes default values, expose Delphi classes, read the script source code from file, compile the script and either create an error message should any compilation fails, or execute the script so that it can produce the document. Finally, the document is sent back and all object instances are destroyed.

Full source code:


The code is available from my website, see
   http://www.overbyte.be/frame_index.html?redirTo=/blog_source_code.html


unit OverbyteIcsWebAppServerDWScriptUrlHandler;

interface

uses
    Classes, SysUtils, OverbyteIcsHttpAppServer, OverbyteIcsHttpSrv,
    dwsVCLGUIFunctions,
    dwsMagicExprs,
    dwsRTTIExposer,
    dwsFunctions,
    dwsSymbols,
    dwsExprs,
    dwsComp;


type
    THttpResponse = class(TPersistent)
    private
        FStatus      : String;
        FContentType : String;
    public
        DocStream    : TStream;
    published
        property Status      : String read FStatus      write FStatus;
        property ContentType : String read FContentType write FContentType;
        procedure Write(const S : String);
    end;

    THttpRequest = class(TPersistent)
    public
        Params : String;
        class function ReadTextFile(const FileName : String) : String;
    published
        function  GetParamByName(const ParamName: String): String;
        function  CheckParamByName(const ParamName  : String;
                                   var   ParamValue : String): Boolean;
    end;

    TUrlHandlerDWScript = class(TUrlHandler)
    protected
        FScript                  : IdwsProgram;
        FCompileMsgs             : String;
        FDelphiWebScript         : TDelphiWebScript;
        FUnit                    : TdwsUnit;
        FExec                    : IdwsProgramExecution;
        FHttpRequest             : THttpRequest;
        FHttpResponse            : THttpResponse;
        procedure ExposeInstancesAfterInitTable(Sender: TObject);
    public
        procedure Execute; override;
    end;

implementation


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}

{ TUrlHandlerDWScript }

{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
procedure TUrlHandlerDWScript.Execute;
var
    SrcFileName : String;
    Source      : String;
begin
    FDelphiWebScript  := TDelphiWebScript.Create(nil);
    FUnit             := TdwsUnit.Create(nil);
    FHttpResponse     := THttpResponse.Create;
    FHttpRequest      := THttpRequest.Create;
    try
        DocStream.Free;
        DocStream := TMemoryStream.Create;

        FHttpResponse.DocStream    := DocStream;
        FHttpResponse.Status       := '200 OK';
        FHttpResponse.ContentType  := 'text/html';
        FHttpRequest.Params        := Params;

        FUnit.OnAfterInitUnitTable := ExposeInstancesAfterInitTable;
        FUnit.UnitName             := 'WebPage';
        FUnit.Script               := FDelphiWebScript;
        FUnit.ExposeRTTI(TypeInfo(THttpResponse), [eoNoFreeOnCleanup]);
        FUnit.ExposeRTTI(TypeInfo(THttpRequest),  [eoNoFreeOnCleanup]);
        // In this application, we have placed DWScripts source code in
        // a directory at the same level as the "Template" folder.
        SrcFileName := ExcludeTrailingPathDelimiter(
                           ExtractFilePath(Client.TemplateDir)) +
                       StringReplace(Client.Path, '/', '\', [rfReplaceAll]);
        if not FileExists(SrcFileName) then
            FHttpResponse.Write('Script not found')
        else begin
            Source       := FHttpRequest.ReadTextFile(SrcFileName);
            FScript      := FDelphiWebScript.Compile(Source);
            FCompileMsgs := FScript.Msgs.AsInfo;
            if FScript.Msgs.HasErrors then begin
                FHttpResponse.Write('' + FCompileMsgs + '');
            end
            else begin
                FExec    := FScript.Execute;
            end;
        end;
        AnswerStream(FHttpResponse.Status, FHttpResponse.ContentType, NO_CACHE);
    finally
        FreeAndNil(FUnit);
        FreeAndNil(FDelphiWebScript);
        FreeAndNil(FHttpResponse);
        FreeAndNil(FHttpRequest);
    end;
    Finish;
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
procedure TUrlHandlerDWScript.ExposeInstancesAfterInitTable(Sender: TObject);
begin
    FUnit.ExposeInstanceToUnit('Response', 'THttpResponse', FHttpResponse);
    FUnit.ExposeInstanceToUnit('Request',  'THttpRequest',  FHttpRequest);
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}

{ THttpResponse }

{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
procedure THttpResponse.Write(const S: String);
var
   Ch : Char;
   B  : Byte;
begin
    for Ch in S do begin
        // We should convert the Unicode string to whatever the document
        // is supposed to be. Here we just convert it, brute force, to ASCII.
        // This won't work eastern character sets.
        B := Ord(AnsiChar(Ch));
        DocStream.Write(B, 1);
    end;
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}

{ THttpRequest }

{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
function THttpRequest.CheckParamByName(
    const ParamName  : String;
    var   ParamValue : String): Boolean;
begin
    Result := ExtractURLEncodedValue(Params, ParamName, ParamValue);
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
function THttpRequest.GetParamByName(const ParamName: String): String;
begin
    ExtractURLEncodedValue(Params, ParamName, Result);
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
class function THttpRequest.ReadTextFile(const FileName : String) : String;
var
    Stream : TFileStream;
    AnsiBuf : AnsiString;
begin
    Stream := TFileStream.Create(FileName, fmOpenRead);
    try
        SetLength(AnsiBuf, Stream.Size);
        Stream.Read(AnsiBuf[1], Stream.Size);
        Result := String(AnsiBuf);
    finally
        FreeAndNil(Stream);
    end;
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}

end.


Download source code from: http://www.overbyte.be/frame_index.html?redirTo=/blog_source_code.html
Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be

May 21, 2013

Internet Explorer Automation Part 4


I this article, I will explain how to extract statistics from Blogger stats page. This follows the previous article in which you learned how to automate the login process and get the stats page.

The stats page is organized in a number of HTML elements. The one which is interesting for us is a table. Since there are many tables in the page, I had to find out a way to detect the correct one, even if the page layout changes.

The idea is to enumerate all the HTML elements in the page, check for the table tag and check the labels against string constants. This is easy since the table is organized in two columns, one with the label and one with the number.

To iterate all the HTML elements is easy. As we saw in previous article, there is a property of the HTML document which is a collection (a kind of array) name “all”. It is enough to enumerate it and for each item in the collection query the interface IID_IHTMLElement to get hand on the HTML element.

Having the HTML element, we can check the tagName property which is actually the tag type. We are looking for ‘Table’. If it is a table, we get the innertext property which as its name implies is the raw text inside the tag. Raw text means it is the text without any embedded tags. In the case of an HTML table, we get the content of all table cells. We will search that text for the labels such as “Pageviews today” and then extract the number just after. For that purpose I wrote a little utility function I will show you in a moment.

Here is the code to do what I’ve just described:
    Coll := FHtmlDoc.all;
    for I := 0 to Coll.Length - 1 do begin
        pDisp := Coll.item(I, var2);
        if pDisp.QueryInterface(IID_IHTMLElement, HtmlElem) = S_OK then begin
            if SameText(HtmlElem.tagName, 'TABLE') then begin
                Txt := String(HtmlElem.innertext);
                if not ExtractNumberAfterText(Txt, TxtToday,
                                              CountToday) then
                    continue;
                if not ExtractNumberAfterText(Txt, TxtYesterday,
                                              CountYesterday) then
                    continue;
                if not ExtractNumberAfterText(Txt, TxtLastMonth,
                                              CountLastMonth) then
                    continue;
                if not ExtractNumberAfterText(Txt, TxtAllTime,
                                              CountAllTime) then
                    continue;

                Buf := AnsiString(
                          FormatDateTime('YYYY/MM/DD;HH:NN:SS;', Now) +
                          '"_' + FBlogId + '";' +
                          IntToStr(CountToday) + ';' +
                          IntToStr(CountYesterday) + ';' +
                          IntToStr(CountLastMonth) + ';' +
                          IntToStr(CountAllTime));
                Result := TRUE;
                break;
            end;
        end;
    end;

The utility function ExtractNumberAfterText is rather simple. It is just simple Delphi code to parse the string. We just have to pay attention to skip all spaces and line breaks because they are not significant in HTML.

function ExtractNumberAfterText(
    const Source : String;
    const Text   : String;
    out   Number : Integer) : Boolean;
var
    J : Integer;
begin
    Result := FALSE;
    Number := 0;
    J := Pos(Text, Source);
    if J <= 0 then
        Exit;
    // Search for first digit right after searched text,
    // ignore anything not a digit
    J := J + Length(Text);
    while (J <= Length(Source)) and
          (not CharInSet(Source[J], ['0'..'9'])) do
        Inc(J);
    // After first digit, scan all digit and ',' or '.' (which
    // are used as thousand separator (Depends on language, any will do)
    repeat
        // If we have a digit, use it to build the final number
        if CharInSet(Source[J], ['0'..'9']) then
            Number := Number * 10 + Ord(Source[J]) - Ord('0');
        Inc(J);
    until (J > Length(Source)) or
          (not CharInSet(Source[J], ['0'..'9', ',', '.']));
    Result := TRUE;
end;

About the design of the application


I explained how to automate Internet Explorer. I showed the actual code used. But I didn’t gave any explanation about how I have designed the whole application.

I always like to separate the user interface from data processing. For that purpose, I created two source files: one with the user interface and one with a class having the automation code.

My user interface is very basic: a simple form with a memo showing messages about what is going on. I could as well write a console mode application or a service application. This doesn’t really matters.

My data processing code is encapsulated in a class I named TQueryBloggerStatistics. It explains what it does. The class is a kind of container. It exposes a few methods and properties to permit what has to be done with that kind of automation.

The class declaration is as follow:

    TQueryBloggerStatistics = class
    private
        FWebBrowser   : IWebBrowser2;
        FBlogID       : String;
        FUserEMail    : String;
        FUserPassword : String;
        FLogFileName  : String;
        FVisible      : Boolean;
        FOnDisplay    : TDisplayEvent;
        function WaitComplete(const URL : String = ''): IHTMLDocument2;
        function FindTag(const Coll    : IHTMLElementCollection;
                         const TagName, TagID: String): IHTMLElement;
        procedure Display(const Msg : String);
    public
        constructor Create;
        function  Execute : Boolean;
        procedure Quit;
        procedure LoadConfig(const IniFileName : String); overload;
        procedure LoadConfig; overload;
        function  SaveConfig(const IniFileName: String) : Boolean; overload;
        function  SaveConfig : Boolean; overload;
        property  BlogID       : String        read  FBlogID
                                               write FBlogID;
        property  UserEMail    : String        read  FUserEMail
                                               write FUserEMail;
        property  UserPassword : String        read  FUserPassword
                                               write FUserPassword;
        property  LogFileName  : String        read  FLogFileName
                                               write FLogFileName;
        property  Visible      : Boolean       read  FVisible
                                               write FVisible;
        property  OnDisplay    : TDisplayEvent read  FOnDisplay
                                               write FOnDisplay;
    end;
I won’t reproduce the implementation here because I already showed most interesting part. You can download the full source code for the class and the complete demo application from my website at:
http://www.overbyte.be/frame_index.html?redirTo=/blog_source_code.html

Previous article: http://francois-piette.blogspot.be/2013/05/internet-explorer-automation-part-3.html

Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be

May 12, 2013

Internet Explorer Automation Part 3


Today I will present an Internet Explorer automation which will query Blogger stats page automatically. IE automation is required because Blogger website makes heavy use of JavaScript to dynamically construct the stats page. Downloading the webpage with a HTTP component won’t work because the numbers we are looking for are not in clear! JavaScript must be executed to get hand of it.

The code I will show you will also take care of authentication. Asking for the stats page without being first authenticated and you get the authentication page instead. The code I’ll present will detect the login page, fill the form automatically, submit it and then request the stats page again and finally extract the data.

For those not accustomed with Blogger author interface and his stats page, the screen dump shows an actual view of the page. It shows the stats for this week (At the time of writing this article). What we are interested in is to get the column on the right showing “Pageviews today 149”, “Pageviews yesterday 434” and the two other lines. This is an HTML table that we have to extract from the document.



As I said above before getting this stats page, you must be authenticated. This means that if you are not authenticated, Blogger will show you the login page whatever you asked in the first place. For your reference, here is a screen dump of the authentication page:



On that page, we see a form with two fields for Email and Password and a button “Sign in” to click. The program will locate those fields, assign a value and then click on the button.

Document Object Model (DOM)


The World Wide Web Consortium (W3C) Document Object Model (DOM) is a platform- and language-neutral interface that permits programs or scripts to access and update the content, structure, and style of a document. The W3C DOM includes a model for how a standard set of objects representing HTML and XML documents are combined, and an interface for accessing and manipulating them.

Internet Explorer exposes DOM thru a set of COM interfaces available to external programs such as our Delphi application. This is documented on MSDN website at:
      http://msdn.microsoft.com/en-us/library/ie/hh772384(v=vs.85).aspx

I will only scratch the surface of DOM. Just enough to get you started and to accomplish the task for the sample application.

We saw in previous article that we can connect to IE by calling this line:
    FWebBrowser := CreateComObject(CLASS_InternetExplorer) as IWebBrowser2;

And that we can navigate to an URL with this line of code:
        FWebBrowser.Navigate(Url, EmptyParam, EmptyParam, EmptyParam, EmptyParam);

To get hand on the interface which is the entry point for the DOM, we must get the document (whatever it is) and the get the interface to the HTML document (if it exists):
      Doc := FWebBrowser.Document;
      Doc.QueryInterface(IID_IHTMLDocument2, HtmlDoc);

Those code lines are easy but wait! There can be some glitches. Internet Explorer takes some time to fetch URL and build document. A document can be quite complex and could requires a lot of downloads for HTML, images, CSS, scripts and more. And once everything is downloaded, scripts have to be executed. There are various status available to be sure everything is OK. The method WaitComplete here after takes an URL, navigate to it and wait until the HTML document interface is available and the document is ready:

function TQueryBloggerStatistics.WaitComplete(
    const URL : String = ''): IHTMLDocument2;
var
    Doc : IDispatch;
begin
    Result := nil;
    if URL <> '' then
        FWebBrowser.Navigate(Url, EmptyParam, EmptyParam, EmptyParam, EmptyParam);
    while FWebBrowser.Busy do
        Sleep(250);
    while FWebBrowser.Document = nil do
        Sleep(250);
    Doc := FWebBrowser.Document;
    if Doc.QueryInterface(IID_IHTMLDocument2, Result) <> S_OK then
        Exit;
    while not SameText(Result.readyState, 'complete') do
        Sleep(250);
end;

WaitComplete takes and optional URL and returns the IHTMLDocument2 interface required for handling the document. Tests are made to be sure everything is ready or complete. The code is quite straightforward but this must be done like that.

Once we’ve got an IHTMLDocument2 interface, we can use it to traverse the document object model (DOM) to find the HTML elements we need and to get or set their properties.

The HTML document has a number of collections like images, links, scripts and the likes. And there is a special collection returning absolutely everything. It is named “all”. We will use it to find what we need. For example, in the login form, we need to get hand on the HTML INPUT tag for each field and submit buttons. Each HTML tag has a TagName such as “input” and a tagID. TagName is an HTML standard while TagID is chosen by the web developer, in this case by Blogger. Fortunately at Blogger, they used very clear and meaningful TagId sucha as “Email” (for the Email input field), “Passwd” (for the password input field) and “Signin” for the submit button.

Since we have to get hand on several HTML elements, I wrote a little function FindTag:

function TQueryBloggerStatistics.FindTag(
    const Coll    : IHTMLElementCollection;
    const TagName : String;
    const TagID   : String) : IHTMLElement;
var
    PDisp : IDispatch;
    Var2  : OleVariant;
    I     : Integer;
begin
    for I := 0 to Coll.Length - 1 do begin
        pDisp := Coll.item(I, var2);
        if pDisp.QueryInterface(IID_IHTMLElement, Result) = S_OK then begin
            if SameText(Result.tagName, TagName) and
               SameText(Result.Id, TagID) then
                Exit;
        end;
    end;
    Result := nil;
end;
FindTag has to be called like this:

    HtmlElem := FindTag(FHtmlDoc.All, 'INPUT', 'EMail');
    if Assigned(HtmlElem) then
        HtmlElem.setAttribute('Value', FUserEMail, 0);

This excerpt find tag name “input” tag having an ID “Email”. The result, if found, is the interface to handle that HTML element. Here I use the interface to set the attribute “value” to the user email (variable FUserEMail hold the Email address).

FindTag code is relatively simple although accessing the collection items is a little bit tricky and must pass thru the use of another interface. Sorry but this is how Microsoft designed IE to handle the DOM.

Detecting and handling the login page

The code I’ll show you below will query a webpage by his URL. Nere this URL is supposed to be the stats page of a given Blogger’s blog. We’ll come back to that URL later. It makes use of WaitComplete to fetch the URL, wait until it is ready and complete and then use FindTag to see it the page conatins an “input” tag with and ID “Email”. If this is the case, then it is assumed we have received the login page. The conde then fetch in cascade all other required tags in that page, fill it with user data and then claa the “Click” method of the HTML element which is the submit button. And guess what… IE will send the form to Blogger and authentication take place.

    FHtmlDoc := WaitComplete(URL);
    if not Assigned(FHtmlDoc) then
        Exit;

    // Check for login page
    // If found, fill in the form and subit it before continuing
    HtmlElem := FindTag(FHtmlDoc.All, 'INPUT', 'EMail');
    if Assigned(HtmlElem) then begin
        HtmlElem.setAttribute('Value', FUserEMail, 0);
        HtmlElem := FindTag(FHtmlDoc.All, 'INPUT', 'Passwd');
        if Assigned(HtmlElem) then begin
            HtmlElem.setAttribute('Value', FUserPassword, 0);
            HtmlElem := FindTag(FHtmlDoc.All, 'INPUT', 'PersistentCookie');
            if Assigned(HtmlElem) then
                HtmlElem.setAttribute('Checked', '', 0);
            HtmlElem := FindTag(FHtmlDoc.All, 'INPUT', 'Signin');
            if Assigned(HtmlElem) then begin
                HtmlElem.click;
                Display('Login...');
                // We have found login form and must wait for login to occur
                FHtmlDoc := WaitComplete;
                if not Assigned(FHtmlDoc) then
                    Exit;
                // Login is finished, we must navigate again to the target URL
                FHtmlDoc := WaitComplete(URL);
                if not Assigned(FHtmlDoc) then
                    Exit;
                HtmlElem := FindTag(FHtmlDoc.All, 'INPUT', 'EMail');
                if Assigned(HtmlElem) then begin
                    Display('Login failed');
                    Exit;
                end;
            end;
        end;
    end;


The next step is to extract the statistics from the stat page.
We will do that in the next article. Stay tuned!

Read also part 1 and part 2.

Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be

January 31, 2013

Internet Explorer Automation Part 2


Internet Explorer is a very nice program to automate. There are a large number of actions you can do programmatically from your own application. But when IE is already opened with a bunch of tabs, it is not a trivial task to programmatically select and activate the tab you want.

Here after, I will present all the code required to do that. It has been developed using Delphi XE3 but of course as automating IE is independent of the language, you should be able to translate my code to C#, C++ or any language supporting COM programming.

The code I present is basically in a single function with a number of small supporting functions. The main function is:

function WebBrowserSelectTabByUrl(
    const Wb           : IWebBrowser2;
    const Url          : String;
    out   HwndTopLevel : HWND) : Boolean;


You pass an existing IWebBrowser interface (see for example my previous article at http://francois-piette.blogspot.com/2013/01/internet-explorer-automation-part-1.html) and an URL. The function will select the tab having the given URL loaded, if any. It will also return a window handle that can be used to bring the actual window in the foreground or to restore it if it was minimized.

To achieve his goal, WebBrowserSelectTabByUrl is using a seldom know interface. I mean IAccessible (http://msdn.microsoft.com/en-us/library/windows/desktop/dd318466(v=vs.85).aspx). This interface is normally used by software written for the visual impaired person. This kind of software is able to discover almost every interface gadget on screen, return a description and perform a default action such as clicking on it if it is a button.

Internet Explorer is exposing a complete IAccessible interface for its entire user interface. And this is what I use to search for the tab rows displaying IE tabs, and get the URL assigned to each of the tab.

IAccessible interface and related definitions is defined in OleAcc unit which is an import from OLEACC.DLL type library. This unit also contains a lot of constants that were not included in the type library.

Beside the interface, there are a few API functions which give an IAccessible interface from a window handle or the reverse. We need two functions which are not defined in OleAcc and you’ll find the required import in the code at the end of this article. It is WindowFromAccessibleObject and AccessibleChildren.

IAccessible is just the programmatic way to interact with the underlying user interface gadgets. It is organized in an hierarchical tree. One you get an IAccessible interface for something, you can “travel” thru the tree to find what you need. Each gadget has a name. We are looking for “Tab Row” item. In Internet Explorer user interface, this represents the row usually below the address bar, where IE shows all tabs for all opened URL.

Once we get hand on the “Tab Row” gadget, we can iterate all of its descendants to find the one with the URL we are looking for. The URL is associated with each tab as a description. Actually the tab description is composed of the text IE show on the tab and the associated URL that IE shows in the address bar when the tab is selected.

Finally, when we have the IAccessible for the exact tab were looking for, we can invoke his default action programmatically. The net effect is the same as the effect a user produce when clicking on the tab.

There is still an issue: As IAccessible is made to help visually impaired users, the name of each gadget is localized. So “Tab Row” in English becomes “Onglet Ligne” in French! I have not found any way to discover the translation so I have to code a small routine querying the language from Windows configuration and use it to select the correct translation. If you use my code, you must add the language you need because I only programmed the English and French translation. See WebBrowserGetLocalizedTabRowName function at the end of this article. [The translation is probably somewhere in one resource in IE executable or DLL. Let me know if you know where it is]

The fastest way to find the first IAccessible interface we need is to travel Internet Explorer window tree. I used Microsoft Spy++ tool to see how those windows are organized. The outermost window handle is given my IWebBrowser interface in his HWND property. Then the hierarchy of window classes is “WorkerW” (or “CommandBarClass” depending on IE version), “ReBarWindow32”, “TabBandClass” and finally “DirectUIHWND”. In used the API function FindWindowEx to navigate thru the hierarchy. Yhe result is the functions WebBrowserGetDirectUIHWND.

From the DirectUIHwnd, we can get the IAccessible interface calling AccessibleObjectFromWindow. Let’s name it AccDirectUI.

The, as I said above, we have to traverse the IAccessible tree to find one with name “Tab Row” (Or the translated is you don’t use an English IE). This is FindAccessibleDescendantByName function. This is a classical tree traversal algorithm. The only complex thing is that there is a variant in the process. A cast and a call to QueryInterface are required to get hand on the IAccessible interface of the child.

Almost the same tree traversal is used from the “Tab Row” to find the right tab. Instead of checking the name, I check the description which contain the URL.

Enough story, here is the code:

function WebBrowserSelectTabByUrl(
  const Wb           : IWebBrowser2;
  const Url          : String;
  out   HwndTopLevel : HWND) : Boolean;
var
  HwndDirectUI     : HWND;
  AccDirectUI      : IAccessible;
  TabRow           : IAccessible;
  CandidateTab     : IAccessible;
  I                : Integer;
  LocalUrl         : String;
  HwndCandidateTab : HWND;
  ChildArray       : array of OleVariant;
  ChildDispatch    : IDispatch;
  ChildCount       : Integer;
  CountObtained    : Integer;
begin
  Result       := FALSE;
  HwndDirectUI := WebBrowserGetDirectUIHWND(Wb);
  AccessibleObjectFromWindow(HwndDirectUI, OBJID_WINDOW,
                             IID_IAccessible, AccDirectUI);
  if not Assigned(AccDirectUI) then
    Exit;

  TabRow := FindAccessibleDescendantByName(AccDirectUI, 

                           WebBrowserGetLocalizedTabRowName);
  TabRow.Get_accChildCount(ChildCount);
  if ChildCount <= 0 then
    Exit;
  SetLength(ChildArray, ChildCount);
  if AccessibleChildren(Pointer(TabRow), 0, ChildCount,

                        ChildArray[0], CountObtained) <> S_OK then
    Exit;
  for I := 0 to CountObtained - 1 do begin
    if VarType(ChildArray[i]) = varDispatch then begin
      ChildDispatch := TVarData(ChildArray[i]).VDispatch;
      if (ChildDispatch <> nil) and
         (ChildDispatch.QueryInterface(Ole2.TGUID(IID_IAccessible),

                   CandidateTab) = S_OK) then begin
        if not Assigned(CandidateTab) then
          continue;
        LocalUrl := WebBrowserUrlForTab(CandidateTab);
        if SameText(LocalUrl, Url) then begin
          CandidateTab.accDoDefaultAction(0);
          WindowFromAccessibleObject(CandidateTab, HwndCandidateTab);
          HwndTopLevel := FindIEFrameWnd(HwndCandidateTab);
          Result := TRUE;
          Exit;
        end;
      end;
    end;
  end;
end;



function WebBrowserUrlForTab(AccTab : IAccessible) : String;
var
  Desc : WideString;
  I    : Integer;
begin
  try
    SetLength(Desc, 1024);
    AccTab.Get_accDescription(CHILDID_SELF , Desc);
    if Desc <> '' then begin
      I := Pos(String(#13#10), String(Desc));
      if I > 1 then
        Result := Copy(Desc, I + 2, MAXINT)
      else
        Result := Desc;
      Exit;
    end;
  except

    Result := '??';
  end;
end;


// The IAccessible name for the tab row in Internet explorer is localized
// This function fetch the language code and return the appropriate value
// according to the current system default language
function WebBrowserGetLocalizedTabRowName : String;
var
  Lang : String;
begin
  Lang := GetLocaleStr(LOCALE_SYSTEM_DEFAULT, LOCALE_SISO639LANGNAME, '');
  if Lang = 'fr' then
    Result := 'Onglet Ligne'
    // YOU MUST ADD a "else if" clause for each language you want to support
  else
    Result := 'Tab Row'; // English
end;

function WebBrowserGetDirectUIHWND(Wb : IWebBrowser2): HWND;
begin
  // try IE 9 first:
  Result := FindWindowEx(Wb.HWND, 0, 'WorkerW', nil);
  if Result = 0 then begin
    // IE8 and IE7
    Result := FindWindowEx(Wb.HWND, 0, 'CommandBarClass', nil);
  end;
  Result := FindWindowEx(Result, 0, 'ReBarWindow32', nil);
  Result := FindWindowEx(Result, 0, 'TabBandClass', nil);
  Result := FindWindowEx(Result, 0, 'DirectUIHWND', nil);
end;


// Recursively trave the tree of descendant IAccessible interface object
// to search for the one having a given name.
function FindAccessibleDescendantByName(
  const AParent : IAccessible;
  const AName   : String) : IAccessible;
var
  ChildArray    : array of OleVariant;
  Child         : IAccessible;
  ChildName     : WideString;
  ChildDispatch : IDispatch;
  ChildCount    : Integer;
  CountObtained : Integer;
  I             : Integer;
begin
  Result := nil;
  Aparent.Get_accChildCount(ChildCount);
  if ChildCount <= 0 then
    Exit;
  SetLength(ChildArray, ChildCount);
  if AccessibleChildren(Pointer(AParent), 0, ChildCount,

                        ChildArray[0], CountObtained) <> S_OK then
    Exit;
  for I := 0 to CountObtained - 1 do begin
    if VarType(ChildArray[i]) = varDispatch then begin
      ChildDispatch := TVarData(ChildArray[i]).VDispatch;
      if (ChildDispatch <> nil) and
         (ChildDispatch.QueryInterface(Ole2.TGUID(IID_IAccessible),

                                       Child) = S_OK) then begin
        if not Assigned(Child) then
          continue;
        Child.Get_accName(0, ChildName);
        if SameText(AName , ChildName) then begin
          Result := Child;
          Exit;
        end;
        Result := FindAccessibleDescendantByName(Child, AName);
        if Assigned(Result) then
          Exit;
      end;
    end;
  end;
end;


// Given a HWND for a window deep in the hierarchy of windows, go back to
// the top level window which has the class name 'IEFrame'.
function FindIEFrameWnd(Hndl : HWND) : HWND;
var
    H     : HWND;
begin
    H := Hndl;
    while TRUE do begin
        if SameText(GetClassName(H), 'IEFrame') then begin
            Result := H;
            Exit;
        end;
        H := GetParent(H);
    end;
end;


function WindowFromAccessibleObject(

             pAcc      : IACCESSIBLE;
             var phwnd : HWND) : HRESULT; stdcall;
             external 'oleacc.dll';

function AccessibleChildren(

             paccContainer     : Pointer;
             iChildStart       : LongInt;
             cChildren         : LongInt;
             out rgvarChildren : OleVariant;
             out pcObtained    : LongInt) : HRESULT; stdcall;
             external 'oleacc.dll';


The first part of this article is at:
   http://francois-piette.blogspot.be/2013/01/internet-explorer-automation-part-1.html

This article is at:
   http://francois-piette.blogspot.be/2013/01/internet-explorer-automation-part-2.html

Follow me on Twitter

January 28, 2013

Internet Explorer Automation Part 1


Internet Explorer can be automated just like Word or Excel. Most automation is done using IWebBrowser2 interface. Getting hand on a IWebBrowser interface is easy. It is enough to call CreateComObject, passing the Internet Explorer ID. This will create a new instance of Internet Explorer:

FWebBrowser := CreateComObject(CLASS_InternetExplorer) as IWebBrowser2;

Once the instance is created (A new IE window will open), we can call for example the Navigate method to load a page:


FWebBrowser.Navigate('http://www.overbyte.be', EmptyParam,
                     EmptyParam, EmptyParam, EmptyParam);

Sometimes, we do not need a new Internet Explorer Window but access an existing window to automate some processing on that window.

There exists several ways of finding an existing Internet Explorer window. One of the easiest is to use the Windows Explorer API. There is a bunch of interfaces to work with Windows Explorer. IShellWindows handle a collection of Explorer windows and this is exactly what we need. We will iterate thru all the windows and locate the Internet Explorer. Since there can be several IE opened windows, we will use the URL to find the one we are looking for.

Here is the code:

function GetIERunningInstanceByUrl(const Url : String): IWebBrowser2;
var
    ShWindows : IShellWindows;
    I         : Integer;
begin
    ShWindows := CoShellWindows.Create;
    for I := 0 to ShWindows.Count - 1 do begin
        Result := ShWindows.Item(I) as IWebBrowser2;
        if Assigned(Result) then begin
            if SameText(GetClassName(Result.HWND), 'IEFrame') then begin
                if SameText(Url, Result.LocationURL) then
                    Exit;
            end;
        end;
    end;
    // Not found
    Result := nil;
end;


GetClassName is a simple wrapper around Windows API to make it easier to use with Delphi:

function GetClassName(Hndl : HWND) : String;
var
    L : Integer;
begin
    SetLength(Result, MAX_PATH * SizeOf(Char));
    L := WinApi.Windows.GetClassName(Hndl, PChar(Result), Length(Result));
    SetLength(Result, L);
end;


Share this article if you like it!


http://francois-piette.blogspot.com/2013/01/internet-explorer-automation-part-1.html

See aldo the second part:
    http://francois-piette.blogspot.be/2013/01/internet-explorer-automation-part-2.html