DirWatch: Delphi 10 in action

In this article, I will show you how I used Delphi 10 Seattle to create a nice application aimed at watching a directory tree. Source code provided, see at the end.

Background:

I frequently use shared directories as repository for files. Some user on the network just drops the file in the directory where I can grab it. The user is supposed to notify me when he has dropped a file.  Unfortunately some user forgets to notify e and the file stay there for a long time before I notice it myself. In other situation, I like to know which files are affected by a given application or by a setup.

DirWatch will be of great help in those situation and many others. DirWatch will monitor a whole directory tree, even a full disk, for changes and send me an email when a change is detected. A change means a file or subdirectory is added / renamed / deleted / modified.

How it works:

Part 1: Getting notifications

Using Windows, it is actually very easy to be notified of changes in the file system. There is a specific and simple API for the purpose. The main API functions are FindFirstChangeNotification, FindNextChangeNotification and ReadDirectoryChanges. The actual work is done by Windows!

This notification API works much like listing a directory: you do a FindFirst, then if successful, you read the changes that occurred and then call FindNext to be notified for the next change. As simple as that.

OK, things can be a little bit more complex if you want your application to stay responsive you should resort to asynchronous Input/Output and multithreading. That code already exists on the web in numerous instances. I selected to use DWScript implementation because it is simple and efficient and also because I use DWScript for other project and find this open source product excellent. DWScript is available under Mozilla Public License Version 1.1.

dwsDirectoryNotifier is available from DWScript project (https://bitbucket.org/egrange/dwscript). Direct link to the source file at the time of writing is

https://bitbucket.org/egrange/dwscript/raw/5514c091096d0ca0217a2154a7ec8f5d95de443e/Libraries/SimpleServer/dwsDirectoryNotifier.pas

Part 2: Sending an email

To programmatically send an email, you have many solutions available. In my opinion, the best solution is to directly address the email server, either company email server or Internet Service Provider email server. This may sound difficult but it is not much. An email server makes use of a protocol named SMTP, just like your browser make use of a protocol name HTTP. SMTP stands for Simple Mail Transfer Protocol and as its name implies, it is simple! SMTP is a text based protocol and involves the exchange of a few messages to send an email.

SMTP becomes a little bit more complex when you start talking authentication, compression, and security but it basically remains simple.

I have implemented the SMTP protocol in an open source freeware I published nearly 20 year ago. I mean the Internet Component Suite (ICS for short). See http://wiki.overbyte.be. ICS is a huge component library. We will only use the SMTP component.

ICS (Internet Component Suite) is available from http://wiki.overbyte.be/wiki/index.php/ICS_Download and also can be installed directly in Delphi 10 Seattle using Embarcadero GetIT (Delphi menu Tools / GetIi Package Manager, type ICS into the search box and select "ICS for VCL").

Part 3: be discreet

When I use a utility which I don't have to interact with, I like to have it very discreet. I don't even want to see it on my screen. There are two good solutions for this: write a service application or write a "tray-icon" application. I selected the second which is slightly easier to write and debug.

A tray-icon application is a normal Windows application which once minimized is only noticeable by its icon displayed in the systray (You know, this little area usually on the bottom right of the screen where the time is displayed).

Using Delphi 10, you create an application as a "VCL form application" and drop a TTrayIcon component on it. You have just two event handler to write: FormResize and TrayIconDoubleClick. FormResize is used to hide the form once it is minimized and TrayIconDoubleClick is used to make the form visible again when the user double clicks on the small icon.

There is a side problem when building a tray-icon application: you need a small 16x16 icon. If you don't create your own 16x16 icon, Windows will shrink the large icon to create the small one. The result is frequently horrible.

To create the multiple resolution icon file, I used IconMagick (http://www.imagemagick.org) command line utility "convert" to combine icons created using my favorite paint application.

convert DirWatchIcon32x32.ico DirWatchIcon16x16.ico DirWatchIcon.ico

Once DirWatchIcon.ico was created, I associated it with the application using Delphi project options dialog.

Please note that TrayIcon component dropped on the form must be loaded with the DirWatchIcon16x16.ico file otherwise window will use the 32x32 icon file resized to 16x16 which gives an horrible result.

Part 4: Work offline

I wanted to be notified even for changes occurring while DirWatch was not running. This is especially useful if the watched directory is not local but a shared resource on a file server.

When DirWatch starts, I create notifications for the changes occurred since last run by comparing the list of files actually on disk with the list of files that where on disk when DirWatch was stopped.

This means that at program startup, DirWatch enumerates all the files and directories in the watched directory. This could take some time if the directory contains millions of files!  On my hard drive with about one million files, it takes approximately 15 seconds to enumerate the files, compare that list with the previous list and send the email for the changes.

The implementation I made makes use of Delphi 10 Seattle generics. Internally the list of files is not a list but a dictionary. This gives fast access to each file data. Each file (Or directory entry) is stored with his name, attributes, size and timestamp. This is used to find out what has changed with a very low chance to miss something. Of course it could be possible to write a program which saves the timestamp of a file, update some bytes in the file without changing the size and then restore the timestamp. Actually this never occurs in real life!

It is interesting to look at the code I wrote for the dictionary. There are a few involved classes: TDirDictionary, TDirEntry, TDirDictionaryPair, TDirDictionaryObject and TDirDictionaryEnumerator.

I implemented TDirDictionary by delegation instead of inheritance. I frequently do that to have complete freedom about the interface I expose. In most cases, delegation is a better encapsulation that inheritance. Look at the source code for details and you'll quickly see the code is really very simple.

Part 5: Data persistence

DirWatch must be configured with some parameters. Of course, the directory which must be watched but also the SMTP server information (Those information are the same that Outlook uses for a classical POP3/SMTP account).

I also like that my applications remember where their window was and which size they had when I run the application again.

For all those purposes, I like to use the good old INI file. It is easy to use within the program and easy you me to edit with my favorite text editor. I store the INI file in the "Loacl AppData" folder in the user profile.  The Windows API function SHGetFolderPath is used to query the exact location of the "Local AppData" folder.

Part 6: Hide sensitive data

Sending an email may sometimes involve a usercode and password that the DirWatch must know. As explained above, those values are stored in the INI file and this is a problem if they are saved in clear text.

I wrote two functions Decrypt and EncryptRestore for the purpose. One trick is used to allow encryption without using a special program. When the data is enclosed in angle bracket in reverse order, then it's assumed not encrypted but needing to be encrypted. Then once encrypted the resulting value is saved between angle brackets.  So "]MyName[" will result in "MyName" and a flag is set so that EncryptRestore will produce "[Qyaze2==]" which is an encrypted value. If no angle bracket is used at all, then the value is assumed not encrypted. Here in this implementation I used very weak encryption: base64 encoding. In a fully secured application you should use a real encryption mechanism providing strong security.

Part 7: Command line options

Some information cannot be stored in the INI file and yet be changed at run time. For example the name and location of the INI file itself can be changed.

-i IniFileName        Select INI file

-h            Show this help

-m            Start minimized

-a AppName        Select application name

-c CompanyName    Select company name

To parse command line argument, I used code largely inspired by by code from Stehan Huberdoc  (http://stefan.huberdoc.at/comp/software/delphi/sandkasten.html) which is an implementation of the well-known GNU GetOpt command line parsing function (https://en.wikipedia.org/wiki/Getopt).

Part 8: Source code

Source code is split into two units: DirWatchMain.pas which is the main and only form of the application and DirWatchProcess.pas which contains the code which does the actual work without any user interface. Of course other units are used as explained above: some from ICS, some from DWScript and GetOpt.

Download links:

DirWatch: http://www.overbyte.be/eng/blog_source_code.html

ICS: http://wiki.overbyte.be/wiki/index.php/ICS_Download

DWScript: https://bitbucket.org/egrange/dwscript

Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be

Dynamic web page using Delphi, ICS and DWScript

ICS has a web application server component which allows you to build dynamic web page very easily. Delphi code for each web page is encapsulated in a TUrlHandler class and compiled into your application, making a standalone webserver application.

In the article, we will create a TUrlHandler class which will read a DWScript script from disc and execute it. The script is responsible for build a valid answer which ICS will send back to the client.

A simple “Hello World” script is made of a single line like this:

   Response.Write('Server time is ' + DateTimeToStr(Now) + ' ');

To invoke it, assuming the script is located in “Hello.pas”, the user must enter this URL into his browser:
http://localhost:20105/DWScripts/Hello.pas

Remember it is a script. You can change the file on disc and the changes will be immediately reflected for the next request, without recompiling your application. This looks much like PHP but use Delphi syntax instead of PHP. To be honest, PHP may use embedded HTML code within the script which is not supported here.

A more complex script may make use of request parameters. For example, the script “Add.pas” could looks like:

var Value1 : String;
var Value2 : String;
Response.ContentType := 'text/html';
Response.Status := '200 OK';
Response.Write('');
Response.Write('ICS and DWScript demo
');
Response.Write('Server time is ' + DateTimeToStr(Now) + '
');
if not Request.CheckParamByName('Value1', Value1) then
    Response.Write('Missing Value1 parameter
')
else if not Request.CheckParamByName('Value2', Value2) then
    Response.Write('Missing Value2 parameter
')
else Response.Write(Value1 + ' + ' + Value2 + ' = ' + 
    IntToStr(StrToIntDef(Value1, 0) + StrToIntDef(Value2, 0)));
Response.Write('');

The URL to use in the browser looks like:
http://localhost:20105/DWScripts/Add.pas?Value1=123&Value2=456

Getting ICS and DWScript

ICS:    http://wiki.overbyte.be/wiki/index.php/ICS_Download
DWScript: http://code.google.com/p/dwscript/

Implementation

Less than 200 Delphi code is required to implement this behavior in an ICS based web application server. The code I will show you below can be plugged in OverbyteIcsWebAppServer demo application you can find in ICS V8 distribution. Two lines must be added to the main file in order to add the feature: add the unit in the uses clause and add a line to map “/DWScript/*” to the TUrlHandler taking care of the script.

Using the above wild card mapping, the TUrlHandler will be invoked for any URL beginning with “/DWScript/”. It will then access the full path to get the script filename. In the above examples, I used a “.pas” file extension for ease, but this is not mandatory at all. You can use any extensions and even no extension if you don’t like to have the user know you are using a DWScript.

In the implementation, I managed to have the DWScript not directly accessible thru an URL. This protects your source code. The user can’t access it. He can just execute it (Note that ICS THttpServer component has an option which allows access to any file, so what I just said maybe wrong).

The script has two objects instances readily available: “Request” and “Response”. They maps to the corresponding Delphi object instances and classes:

    THttpResponse = class(TPersistent)
    private
        FStatus      : String;
        FContentType : String;
    public
        DocStream    : TStream;
    published
        property Status      : String read FStatus      write FStatus;
        property ContentType : String read FContentType write FContentType;
        procedure Write(const S : String);
    end;

    THttpRequest = class(TPersistent)
    public
        Params : String;
        class function ReadTextFile(const FileName : String) : String;
    published
        function  GetParamByName(const ParamName: String): String;
        function  CheckParamByName(const ParamName  : String;
                                   var   ParamValue : String): Boolean;
    end;

When using DWScript ExposeRTTI function, you are exposing a Delphi class. With the options I selected, only the published methods and properties will be exposed to the script.

THttpResponse exposes Status and ContentType properties as well as Write method. The two properties allows the script to select the HTTP status return and the HTTP content type. They default to “200 OK” and “text/html” which are the most common values. You can use anything you need for your application.

The Write method will be used by the script to produce the document part of the HTTP response sent back to the client. Obviously, the document format must match the content type.

THttpRequest exposes the incoming request. Using it you may access the parameters passed thru the URL. One method gets a parameter value when you provides his name while the other return the value as well as a Boolean value telling if the parameter exists or not.

The most important part of the code is a TUrlHandler derived class. I named it TUrlHandlerDWScript. His declaration looks like this:

    TUrlHandlerDWScript = class(TUrlHandler)
    protected
        FScript                  : IdwsProgram;
        FCompileMsgs             : String;
        FDelphiWebScript         : TDelphiWebScript;
        FUnit                    : TdwsUnit;
        FExec                    : IdwsProgramExecution;
        FHttpRequest             : THttpRequest;
        FHttpResponse            : THttpResponse;
        procedure ExposeInstancesAfterInitTable(Sender: TObject);
    public
        procedure Execute; override;
    end;

The five first member variables are required for DWScript operation. You should look at DWScript documentation for help about their use.

The last two member variables are the object instances we talk above. Their published parts are exposed to the script.

There is also an event handler ExposeInstancesAfterInitTable which is required by DWScript engine to expose the Delphi object instances to the script.

Finally, there is a single method which is called by ICS web application server framework to handle the mapped URL. This is where almost everything happens.

procedure TUrlHandlerDWScript.Execute;
var
    SrcFileName : String;
    Source      : String;
begin
    FDelphiWebScript  := TDelphiWebScript.Create(nil);
    FUnit             := TdwsUnit.Create(nil);
    FHttpResponse     := THttpResponse.Create;
    FHttpRequest      := THttpRequest.Create;
    try
        DocStream.Free;
        DocStream := TMemoryStream.Create;

        FHttpResponse.DocStream    := DocStream;
        FHttpResponse.Status       := '200 OK';
        FHttpResponse.ContentType  := 'text/html';
        FHttpRequest.Params        := Params;

        FUnit.OnAfterInitUnitTable := ExposeInstancesAfterInitTable;
        FUnit.UnitName             := 'WebPage';
        FUnit.Script               := FDelphiWebScript;
        FUnit.ExposeRTTI(TypeInfo(THttpResponse), [eoNoFreeOnCleanup]);
        FUnit.ExposeRTTI(TypeInfo(THttpRequest),  [eoNoFreeOnCleanup]);
        // In this application, we have placed DWScripts source code in
        // a directory at the same level as the "Template" folder.
        SrcFileName := ExcludeTrailingPathDelimiter(
                           ExtractFilePath(Client.TemplateDir)) +
                       StringReplace(Client.Path, '/', '\', [rfReplaceAll]);
        if not FileExists(SrcFileName) then
            FHttpResponse.Write('Script not found')
        else begin
            Source       := FHttpRequest.ReadTextFile(SrcFileName);
            FScript      := FDelphiWebScript.Compile(Source);
            FCompileMsgs := FScript.Msgs.AsInfo;
            if FScript.Msgs.HasErrors then begin
                FHttpResponse.Write('' + FCompileMsgs + '');
            end
            else begin
                FExec    := FScript.Execute;
            end;
        end;
        AnswerStream(FHttpResponse.Status, FHttpResponse.ContentType, NO_CACHE);
    finally
        FreeAndNil(FUnit);
        FreeAndNil(FDelphiWebScript);
        FreeAndNil(FHttpResponse);
        FreeAndNil(FHttpRequest);
    end;
    Finish;
end;

Basically the code creates all the required object instances, initializes default values, expose Delphi classes, read the script source code from file, compile the script and either create an error message should any compilation fails, or execute the script so that it can produce the document. Finally, the document is sent back and all object instances are destroyed.

Full source code:

The code is available from my website, see
   http://www.overbyte.be/frame_index.html?redirTo=/blog_source_code.html

unit OverbyteIcsWebAppServerDWScriptUrlHandler;

interface

uses
    Classes, SysUtils, OverbyteIcsHttpAppServer, OverbyteIcsHttpSrv,
    dwsVCLGUIFunctions,
    dwsMagicExprs,
    dwsRTTIExposer,
    dwsFunctions,
    dwsSymbols,
    dwsExprs,
    dwsComp;


type
    THttpResponse = class(TPersistent)
    private
        FStatus      : String;
        FContentType : String;
    public
        DocStream    : TStream;
    published
        property Status      : String read FStatus      write FStatus;
        property ContentType : String read FContentType write FContentType;
        procedure Write(const S : String);
    end;

    THttpRequest = class(TPersistent)
    public
        Params : String;
        class function ReadTextFile(const FileName : String) : String;
    published
        function  GetParamByName(const ParamName: String): String;
        function  CheckParamByName(const ParamName  : String;
                                   var   ParamValue : String): Boolean;
    end;

    TUrlHandlerDWScript = class(TUrlHandler)
    protected
        FScript                  : IdwsProgram;
        FCompileMsgs             : String;
        FDelphiWebScript         : TDelphiWebScript;
        FUnit                    : TdwsUnit;
        FExec                    : IdwsProgramExecution;
        FHttpRequest             : THttpRequest;
        FHttpResponse            : THttpResponse;
        procedure ExposeInstancesAfterInitTable(Sender: TObject);
    public
        procedure Execute; override;
    end;

implementation


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}

{ TUrlHandlerDWScript }

{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
procedure TUrlHandlerDWScript.Execute;
var
    SrcFileName : String;
    Source      : String;
begin
    FDelphiWebScript  := TDelphiWebScript.Create(nil);
    FUnit             := TdwsUnit.Create(nil);
    FHttpResponse     := THttpResponse.Create;
    FHttpRequest      := THttpRequest.Create;
    try
        DocStream.Free;
        DocStream := TMemoryStream.Create;

        FHttpResponse.DocStream    := DocStream;
        FHttpResponse.Status       := '200 OK';
        FHttpResponse.ContentType  := 'text/html';
        FHttpRequest.Params        := Params;

        FUnit.OnAfterInitUnitTable := ExposeInstancesAfterInitTable;
        FUnit.UnitName             := 'WebPage';
        FUnit.Script               := FDelphiWebScript;
        FUnit.ExposeRTTI(TypeInfo(THttpResponse), [eoNoFreeOnCleanup]);
        FUnit.ExposeRTTI(TypeInfo(THttpRequest),  [eoNoFreeOnCleanup]);
        // In this application, we have placed DWScripts source code in
        // a directory at the same level as the "Template" folder.
        SrcFileName := ExcludeTrailingPathDelimiter(
                           ExtractFilePath(Client.TemplateDir)) +
                       StringReplace(Client.Path, '/', '\', [rfReplaceAll]);
        if not FileExists(SrcFileName) then
            FHttpResponse.Write('Script not found')
        else begin
            Source       := FHttpRequest.ReadTextFile(SrcFileName);
            FScript      := FDelphiWebScript.Compile(Source);
            FCompileMsgs := FScript.Msgs.AsInfo;
            if FScript.Msgs.HasErrors then begin
                FHttpResponse.Write('' + FCompileMsgs + '');
            end
            else begin
                FExec    := FScript.Execute;
            end;
        end;
        AnswerStream(FHttpResponse.Status, FHttpResponse.ContentType, NO_CACHE);
    finally
        FreeAndNil(FUnit);
        FreeAndNil(FDelphiWebScript);
        FreeAndNil(FHttpResponse);
        FreeAndNil(FHttpRequest);
    end;
    Finish;
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
procedure TUrlHandlerDWScript.ExposeInstancesAfterInitTable(Sender: TObject);
begin
    FUnit.ExposeInstanceToUnit('Response', 'THttpResponse', FHttpResponse);
    FUnit.ExposeInstanceToUnit('Request',  'THttpRequest',  FHttpRequest);
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}

{ THttpResponse }

{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
procedure THttpResponse.Write(const S: String);
var
   Ch : Char;
   B  : Byte;
begin
    for Ch in S do begin
        // We should convert the Unicode string to whatever the document
        // is supposed to be. Here we just convert it, brute force, to ASCII.
        // This won't work eastern character sets.
        B := Ord(AnsiChar(Ch));
        DocStream.Write(B, 1);
    end;
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}

{ THttpRequest }

{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
function THttpRequest.CheckParamByName(
    const ParamName  : String;
    var   ParamValue : String): Boolean;
begin
    Result := ExtractURLEncodedValue(Params, ParamName, ParamValue);
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
function THttpRequest.GetParamByName(const ParamName: String): String;
begin
    ExtractURLEncodedValue(Params, ParamName, Result);
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}
class function THttpRequest.ReadTextFile(const FileName : String) : String;
var
    Stream : TFileStream;
    AnsiBuf : AnsiString;
begin
    Stream := TFileStream.Create(FileName, fmOpenRead);
    try
        SetLength(AnsiBuf, Stream.Size);
        Stream.Read(AnsiBuf[1], Stream.Size);
        Result := String(AnsiBuf);
    finally
        FreeAndNil(Stream);
    end;
end;


{* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *}

end.

Download source code from: http://www.overbyte.be/frame_index.html?redirTo=/blog_source_code.html
Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be