April 3, 2013

Inter Process Communication Using Pipes

A pipe is a communication channel between two ends. It is mostly used to communicate between processes running within a computer. As such it is an Inter Process Communication (IPC) mechanism.

The concept of pipe is well known in the Linux (Unix) world. It is used on the command line to direct the output of a command as input of the next command. The so called “pipe character” is used as a syntax gadget for that purpose. Windows is using the same syntax.

More generally, a pipe is an operating system object which is close to a file. An application opens a pipe, then read and writes data and finally closes it. The difference between a pipe and a file is that data is kept in memory and the pipe has two ends. The same or different processes may open the same pipe. What is written at one end becomes readable at the other end. This is why it is named a “pipe”: it fact as a pipe or tunnel between the two ends, data flows from one end and the other. Unlike pipes in the real world, computer pipes are bi-directional. One computer pipe is actually made of two distinct sub-pipes: one for each direction.

We can think of pipes as client/server architecture. The server side opens a pipe and waits for data to be available. The client side opens the pipe and then data may flow into the pipe in both directions independently. If several clients open the same pipe, the server sees it as several independent pipes. At server side, data may be written to individual client’s pipes or broadcasted to all clients.

As seen from Windows API, a pipe can be created with or without a name. Giving a name to a pipe allows processes to share the same pipe easily. When using anonymous pipes, a process knows about it because it has inherited his handle.

The process that creates a pipe is the pipe server. A process that connects to a pipe is a pipe client. One process writes data to the pipe, and then the other process reads the data from the pipe. This is bidirectional: each process can write data that will be read by the other. Read and writes are asynchronous. It means a lot of data may be written before it is read at the other end. Data is stored by the operating system into shared memory. The size of the shared memory area is managed by the operating system using advisory values passed to the pipe’s creation function.

Windows pipe API is relatively complex. It has been encapsulated into Delphi classes by Russell Libby in 2003. Although Russell website is no more available, his code is still around in the internet and has been enhanced by several peoples. I have updated his code for Delphi XE3 and made it available from my website at http://www.overbyte.be/eng/blog_source_code.html

Russell wrote a unit named “pipes.pas” containing 3 components:
- TPipeClient: client component
- TPipeServer: Server component
- TPipConsole: console pipe redirection component

For your convenience, I created two packages: one runtime package and one design time package. The components are installed in the component palette under “Pipes” tab.
Using the components is fairly simple. I made two simple demo programs: PipeClient and PipeServer. You run PipeServer first and then one or more PipeClient instances. PipeClient has an edit box and a “Send” button. When you click on “Send”, the edit box content is sent to the server which in turn displays it in a memo. At server side, the “send” button is replaced by “broadcast” to send the same message to all connected client, if any. Of course you may also send to a specific client.

To send the messages, I actually send strings, taking care of Unicode. When sending (Write or broadcast method), you have to pass a pointer to the first byte to be sent and a count of bytes. At receiver side, data is read from a stream which is also made of bytes.

Warning: a pipe is a byte oriented communication channel. There is no guarantee that each single write at one end will correspond to a single read at the other end. Sending strings has I’ve done in my demo is good for interactive application where a user clicks on a button. In a real world application, you have to design a protocol, just like you do when using socket, so that the reader knows exactly what the writer sent. For example, you send a message made of a byte count and the then actual bytes. Or you design a “line oriented” protocol much like most TCP/IP protocol are. Your messages are made you line of text terminated by a CRLF pair. The reader knows he received a complete message because he received the CRLF pair.

To say it in other words, a pipe is like a TFileStream. The read doesn’t know how the stream has been written. He knows how to read it, for example he knows the stream is made of text lines.

UPDATE 2013-10-04: arno.garrels@gmx.de added 64-bit support and fixed code to compile with Delphi 7 to XE5 (earlier versions may compile however untested).

Follow me on Twitter
Follow me on LinkedIn
Follow me on Google+
Visit my website: http://www.overbyte.be
This article is available from http://francois-piette.blogspot.be
Download source code from my website at
      http://www.overbyte.be/eng/blog_source_code.html

9 comments:

runner said...

I have Cromis.IPC available for quite some time on http://www.cromis.net/blog/downloads/cromis-ipc/

It is a little bit different as it is very high level data oriented. You work with data packets without even knowing that you are using pipes at low level. You don't need to know how pipes work. It eliminates a lot of code you need to write to simply communicate. Naturally in certain situations you need the fine grained control that Russels implemation offers.

Panagiotis Drivilas said...

How can I have a pipe client in a secondary thread like ICS'
.ThreadAttach;
.MessageLoop;
.ThreadDetach;
so you can receive data inside that thread instead of main?

François Piette said...

Not sure I understand your question. Anyway, if it is related to ICS, please use ICS support mailing list. See http://wiki.overbyte.be/wiki/index.php/Main_Page

Chris Hearn said...

Hi,
I can't seem to compile using Delphi 2010, as it does not like type SIZE_T
Is there an equiv. I can substitute so the design and runtime .bpl will compile in D2010 please?
Thanks

François Piette said...

Try to replace SIZE_T by Cardinal Or better, look at the declaration of the function using a variable of type size_t sucha as WriteProcessMemory. You just ctrl-click-left on the function name so that the IDE open if for you, or you let the cursor fly over the function name and have the editor show the syntax and data types (Not sure this feature already existed in D2010).

Chris Hearn said...

Thanks Francois, got it going fine now using Cardinal (or DWord would also be fine looking at the code)

François Piette said...

UPDATE 2013-10-04: arno.garrels@gmx.de added 64-bit support and fixed code to compile with Delphi 7 to XE5 (earlier versions may compile however untested).

Mikhail Mayurov said...

May be you (or someone who feels that he has rights to) can place code to GitHub (or something like this) where people could make co-ordinated enhancements in this project?

Anonymous said...

Yes, please publish code on bitbucket or github, code in zip on website is a very uncomfortable and outdated solution