Karma Home Page
Next: Security Up: Karma Programming Manual Previous: Channels: the basic Input/Output

# Interprocess Communications

# Purpose

Communications are useful when writing applications which are comprised of many modules (processes). Using communications, one can even write applications to perform parallel processing on a heterogeneous network.

# Architecture

The communications support in Karma is built on a layered approach. In fact, one of the original motivations for the layered approach to the Karma library was the need to provide communications at a high level of abstraction for most needs, but not to preclude access to more basic communications when required. Also, in order to leave the future of the communications open-ended, a layered approach, with increasing levels of abstraction, was also seen to be useful. The various packages in the Karma library which provide communications support are listed below.

## ``r'' package

The lowest level (most primitive) communications routines are in the r package in the Karma library. These routines provide a very low level interface to Unix sockets (on Unix systems). Routines exist for allocating ports and making connections to ports. These routines are not recommended for use, as they are inherently non-portable (outside of Unix, say) in their interactions, and too primitive.

The r package also provides other miscellaneous functions to manipulate the process environment, determining the hostname of a machine, computing port numbers, etc. These routines provide some OS independent functions, and hence are preferred over the vendor supplied routines. The r package is a key package in providing OS abstraction.

## ``ch'' package

The basic Input/Output class in Karma is the ``channel''. This is provided by the ch package. A channel is a (possibly) full duplex buffered stream of data. These are similar to the standard C library FILE * streams, but fix a number of deficiencies in FILE streams. See the chapter on for more details.

## ``dm'' package

When a process uses communication channels, it must have some way of responding to incoming data and events. Because the underlying I/O support is provided by ``file descriptors'' (in POSIX environments), a mechanism is required to respond to events on file descriptors. This is called ``descriptor management''. In Unix, there are two ways of doing this. One way is to install signal handlers. Unfortunately, becauses most code is not re-entrant, there are inherent dangers in using signals. The other alternative is to using a polling loop to watch for communication events. The dm package provides the means (when running under a plain Unix environment) to register event callback functions to process communications events. The dm package also provides the dm_native_poll routine which does the actual polling for events. On Unix systems, this can effectively place a process in hibernation until an event occurs.

When running under a different environment, say using the Xt toolkit, the dmx package provides the callback registration functions. This package does not provide a polling routine, as the toolkit choosen supplies this function.

## ``chm'' package

This is simply a layer built on top of the dm package so that channel objects may be managed. This is generally a more useful interface than that provided by the dm package.

## ``pio'' package

The pio package provides a simple means to transfer binary data between a unified ``network'' format and local (host) format. These routines write to/ read from channels. These routines are handy for simple communications.

## ``conn'' package

The real power in the communications support in Karma comes from the conn package. This provides sequenced, reliable full-duplex data streams.

This is the recommended interface for all user-level communications (and most library communications too). The conn package is initialised with the conn_initialise routine. This initialises the package. Note that the dm package should be setup prior to this. The conn package provides support for modules (processes) to connect to each other, using agreed upon "protocols". The conn package does not enforce the definition of a "protocol", it just provides a means for modules to differentiate between the various connections they may have. It is the responsibility of the code implementing the protocol to determine the format of data which may pass down a connection of a particular protocol.

Each client module client that wishes to make connections with a particular protocol to another server module must register the callback routines which will process events on connections of this protocol. If the client does not register support for this protocol, it cannot make connections with that protocol.

Each server module which wishes to receive connections of a particular protocol from other client modules must register the callback routines which will process events on connections of this protocol. If the server does not register support for this protocol, it cannot accept connections with that protocol.

Each module may have both client and server support for zero, one or more protocols. Also, each module may be operating both as a client and a server for different connections. The difference between a client and server (as far as the conn package is concerned) is that a client initiates a connection while the server accepts (or rejects) the connection. See the section on security below which details how access may be restricted.

In order to transfer data over a Connection, one must call the conn_get_channel routine to obtain the underlying Channel, since all I/O is based on the Channel.

The conn package, in conjunction with the ch package provides complete abstraction of communications. Higher level library code and user code has no knowledge of the underlying transport mechanisms used. This allows the library to choose the most efficient transport available. It is worth pointing out that the communications abstraction does not come at the cost of speed or efficiency, both in terms of raw throughput and turnaround times.

## ``arln'' package

The arln package allows the reading (with blocking: waiting for input) from the standard input of lines, while still polling for other events. This package uses the chm package for channel management, hence is not applicable to Xt or XView toolkit based modules.

## ``dsxfr'' package

Another fundamental part of the Karma library is the Karma Data Structure. This flexible, heirarchical data structure can be simply transferred between modules by using the dsxfr package. This package defines the ``multi_array'' protocol, and provides simple routines to read or write an entire data structure from/ to another modules (or disc file). Part of the definition of the ``multi_array'' protocol is that data always flows from the client to the server, never the other way around.

## ``kcmap'' package

Another package in the Karma library is the kcmap package. This package defines an opaque dynamic pseudocolourmap class. This provides a graphics system independent means to modify colourmaps. In addition, it provides for dynamic changes to colourmaps to be shared between modules by using the conn package to define the ``colourmap_indices'' and ``full_colourmap'' protocols. This allows processes attached to the same graphics display and separate graphics displays (i.e. separate X displays) to dynamically share colourmaps.

## ``iedit'' package

The iedit package defines an opaque image edit instruction class. This is designed for simple image painting operations. The iedit package also uses the conn package to define the ``2D_edit'' protocol which allows changes to an edit list to be propagated to co-operating modules. Using this package, it is trivial to implement a "shared paintboard" application for several users, where each user sees what every other user is painting, in real time.

## ``iarray'' package

The Karma library provides a simplified interface to the general Karma Data Structure in the iarray package. This package defines the ``Intelligent Array'' class. As with the dsxfr package, the iarray package provides similar routines to transfer data between modules and to/from disc files.

# Addressing and Network Layers

To make a connection from a Karma client to a Karma server, the
conn_attempt_connection function is given a hostname parameter which specifies which computer the Karma server is running on. This may be a standard Internet address. The special names `unix` and `localhost` refer to the same computer as where the client is running. The special name `local-slow` is the same, except that a slower transport mechanism is used instead.

By default the best reliable network protocol is used, which usually means that TCP/IP or Unix domain sockets are used. You can specify alternative network protocols by appending a protocol name after the hostname, with a colon character in between. Currently supported protocol names are:

- `stream` use the best reliable protocol available (the default)
- `udp` use UDP (User Datagram Protocol) to send datagrams
- `atmsvc` use a native ATM (Asynchronous Transfer Mechanism) SVC (Switched Virtual Circuit) connection, with AAL5, to send datagrams.

Note that when you ask for a datagram protocol, you may end up with an unreliable connection. This means that datagrams (packets) may be corrupted or lost, which means you may need to send them again. The normal, reliable communication channel is still available as the default channel, but you get an extra communication channel for the datagram protocol. You can use the conn_get_datagram_channel function to yield the datagram channel. The conn_get_channel function will return the normal, reliable channel. This channel allows you to send data reliably, and use the datagram channel for less critical, high-bandwidth data.

If you specify a hostname of `my.host:stream` this is the same as `my.host` which will yield a reliable connection. Specifying `my.host:atmsvc` will make a connection with an ATM SVC AAL5 datagram channel.

## Quality of Service Parameters

Some network protocols (such as ATM) allow you to specify quality of service parameters. This means you can negotiate with the network the sustained, minimum and maximum speed and other quality of service parameters such as delay variation, and your connection will not degrade in performance for its duration. Obviously, if the network does not have sufficient resources to meet your demands, the connection attempt will fail, in which case you may elect to try again later or try for a connection with lesser quality. In the future, networks will allow you to trade off service quality versus cost. During peak usage periods, the cost of a connection of a particular quality would be higher than during periods of low network activity (such as late at night).

Karma provides the mechanism to specify desired quality of service parameters. You specify these in the hostname string, by including `name=value` pairs after the connection protocol, using a comma character as a separator. The currently supported QOS parameters are:

- `class` the traffic class, either `ubr` (unassigned bit rate: i.e. ``best-effort'', which is the default) or `cbr` (constant bit rate)
- `txmin` minimum peak transmit rate (bandwidth). This only applies to `class=cbr`
- `txtyp` the sustained peak transmit rate (bandwidth). This only applies to `class=cbr`
- `txmax` maximum peak transmit rate (bandwidth). This only applies to `class=cbr`
- `txsdu` the maximum transmit service data unit (packet size) in bytes
- `txdvmax` the maximum transmit per byte delay variation, in nanoseconds. This only applies to `class=cbr`
- `rxmin` minimum peak receive rate (bandwidth). This only applies to `class=cbr`
- `rxtyp` the sustained peak receive rate (bandwidth). This only applies to `class=cbr`
- `rxmax` maximum peak receive rate (bandwidth). This only applies to `class=cbr`
- `rxsdu` the maximum receive service data unit (packet size) in bytes
- `rxdvmax` the maximum receive per byte delay variation, in nanoseconds. This only applies to `class=cbr`.

Note that bandwidth parameters are given in bytes per second. The following units are recognised:

- `b` bits per second
- `B` bytes per second
- `kb` kilo (1000) bits per second
- `kB` kilo (1000) bytes per second
- `Mb` mega (1000000) bits per second
- `MB` mega (1000000) bytes per second

and may be used as a suffix for the numerical value.

Thus, if you wanted an ATM SVC AAL5 CBR connection with a sustained transmit rate of 1000000 bits per second to host `my.host`, you would specify
`my.host:atmsvc,class=cbr,txtyp=1Mb` for the hostname.

Note that the syntax for QOS parameters is experimental and may change in future releases.

# Tutorials

In a simple module, the following sequence should be followed if communications are required:

- Register the channel management package applicable
- Register any client or server protocols
- Optionally allocate a server port
- Enter the polling loop

In the case where a module only wishes to transmit data, and is not interested in input (or closure) events on connections, it can omit the stage where the polling loop is entered.

Some examples follow.

## Example 1

This simple example is a module which generates some data (a 2 dimensional Intelligent Array) and then wishes to transmit this to a module (name ``receiver'') on another machine (with name ``localhost'').

    /*----------------------------------------------------------*/
    /*  Communications sample program: multi_array transmitter  */
    /*----------------------------------------------------------*/

    #include <stdio.h>
    #include <karma_iarray.h>
    #include <karma_dsxfr.h>
    #include <karma_conn.h>
    #include <karma_dm.h>
    #include <karma_r.h>

    main ()
    {
        /*  Declare variables  */
        int i, j;
        iarray a;

        /*  Initialise communications package  */
        dm_native_setup ();
        conn_initialise ( ( void (*) () ) NULL );
        /*  Register "multi_array" client protocol support  */
        dsxfr_register_connection_limits (-1, 1);
        /*  Attempt connection to module  */
        if ( !conn_attempt_connection ("localhost",
                                       r_get_def_port ("receiver", NULL),
                                       "multi_array") )
        {
            fprintf (stderr, "Error connecting\n");
            exit (1);
        }
        /*  Create a 10x10 integer 2D iarray  */
        a = iarray_create_2D (10, 10, K_INT);

        /*  Fill  a  with data  */
        for (i = 0; i < 10; i++)
        {
            for (j = 0; j < 10; j++)
            {
                I2 (a, i, j) = i * j;
            }
        }

        /*  Send  a  to module  */
        iarray_write (a, "connections");

        /*  Deallocate array  */
        iarray_dealloc (a);
    }

## Example 2

This simple example is a module which receives some data (an N-dimensional Intelligent Array) and writes it to a disc file ``out.kf''.

    /*----------------------------------------------------------*/
    /*  Communications sample program: multi_array receiver     */
    /*----------------------------------------------------------*/

    #include <stdio.h>
    #include <karma_iarray.h>
    #include <karma_dsxfr.h>
    #include <karma_conn.h>
    #include <karma_dm.h>
    #include <karma_r.h>

    void read_func (/* first_time_data */);

    main ()
    {
        unsigned int port_number;

        /*  Initialise communications package  */
        dm_native_setup ();
        conn_initialise ( ( void (*) () ) NULL );
        /*  Register "multi_array" server protocol support  */
        dsxfr_register_connection_limits (1, -1);
        /*  Register callback for new data  */
        dsxfr_register_read_func (read_func);
        /*  Allocate a port  */
        port_number = r_get_def_port ("receiver", NULL);
        if ( !conn_become_server (&port_number, 0) )
        {
            fprintf (stderr, "Could not become a server\n");
            exit (1);
        }

        /*  Enter polling loop (forever)  */
        while (TRUE) dm_native_poll (-1);
    }

    void read_func (first_time_data)
    /*  This routine will be called whenever new data arrives on a "multi_array"
        connection.
        Note that by the time this routine is called, the data structure has
        already been read and cached in the library.
        If data appears on a connection for the first time, the value of
        first_time_data  will be TRUE. Any subsqeuent data that appears on a
        connection will not set this flag.
        The routine returns nothing.
    */
    flag first_time_data;
    {
        iarray a;

        /*  Read (non-blocking) from connection  */
        if ( ( a = iarray_read_nD ("connection", TRUE, NULL, 0, (char **) NULL,
                                   NULL, K_CH_MMAP_NEVER) )
            == NULL )
        {
            fprintf (stderr, "Error getting Intelligent Array\n");
            exit (1);
        }
        /*  Write "out.kf"  */
        iarray_write (a, "out");
        /*  Exit module now that data has been read  */
        exit (0);
    }

## Example 3

This simple example is a module which reads lines from the standard input and transmits the lines to all modules connected with the ``experimental'' protocol. The module will attempt a connection to a module (name ``receiver'') on another machine (with name ``localhost'').

    /*-----------------------------------------------------------*/
    /*  Communications sample program: experimental transmitter  */
    /*-----------------------------------------------------------*/

    #include <stdio.h>
    #include <karma_arln.h>
    #include <karma_conn.h>
    #include <karma_dm.h>
    #include <karma_r.h>

    main ()
    {
        /*  Declare variables  */
        Connection conn;
        Channel channel;
        unsigned int num_conn;
        unsigned int count;
        char buffer[256];

        /*  Initialise communications package  */
        dm_native_setup ();
        conn_initialise ( ( void (*) () ) NULL );
        /*  Register "experimental" client protocol support  */
        /*  No callbacks registered because client will never receive data  */
        conn_register_client_protocol ("experimental", 0, 1, ( flag (*) () ) NULL,
                                       ( flag (*) () ) NULL, ( flag (*) () ) NULL,
                                       ( void (*) () ) NULL);
        /*  Attempt connection to module  */
        if ( !conn_attempt_connection ("localhost",
                                       r_get_def_port ("receiver", NULL),
                                       "experimental") )
        {
            fprintf (stderr, "Error connecting\n");
            exit (1);
        }

        /*  Loop waiting for input and transmit each line  */
        while ( arln_read_line (buffer, 256, "Hurry up> ") )
        {
            /*  Have line of data: transmit to all "experimental" servers  */
            /*  Get number of connections made (should be 1 in this example)  */
            num_conn = conn_get_num_client_connections ("experimental");
            for (count = 0; count < num_conn; ++count)
            {
                /*  Get a particular connection  */
                if ( ( conn = conn_get_client_connection ("experimental", count) )
                    == NULL )
                {
                    fprintf (stderr,
                                "Error getting \"experimental\" connection: %u\n",
                                    count);
                    exit (1);
                }
                /*  Get the channel  */
                channel = conn_get_channel (conn);
                /*  Write and flush  */
                ch_puts (channel, buffer, TRUE);
                ch_flush (channel);
            }
        }
    }

## Example 4

This simple example is a module which reads lines from modules connected with the ``experimental'' protocol and displays the lines.

    /*--------------------------------------------------------*/
    /*  Communications sample program: experimental receiver  */
    /*--------------------------------------------------------*/

    #include <stdio.h>
    #include <karma_arln.h>
    #include <karma_conn.h>
    #include <karma_dm.h>
    #include <karma_r.h>

    flag read_func (/* connection, info */);

    main ()
    {
       unsigned int port_number;

        /*  Initialise communications package  */
        dm_native_setup ();
        conn_initialise ( ( void (*) () ) NULL );
        /*  Register "experimental" server protocol support  */
        conn_register_server_protocol ("experimental", 0, 1, ( flag (*) () ) NULL,
                                       read_func, ( void (*) () ) NULL);
        /*  Allocate a port  */
        port_number = r_get_def_port ("receiver", NULL);
        if ( !conn_become_server (&port_number, 0) )
        {
            fprintf (stderr, "Could not become a server\n");
            exit (1);
        }

        /*  Wait forever for events  */
        while (TRUE) dm_native_poll (-1);
    }

    flag read_func (connection, info)
    /*  This routine will read in data from the connection given by  connection
        and will write any appropriate information to the pointer pointed to by
        info  .
        The routine returns TRUE on successful reading,
        else it returns FALSE (indicating the connection should be closed).
        Note that the  close_func  will be called if this routine returns FALSE
    */
    Connection connection;
    void **info;
    {
        Channel channel;
        char buffer[256];

        /*  Get channel  */
        channel = conn_get_channel (connection);
        if ( !ch_gets (channel, buffer, 256) )
        {
            fprintf (stderr, "Error reading line\n");
            return (FALSE);
        }
        fprintf (stderr, "Incoming line: \"%s\"\n", buffer);
        return (TRUE);
    }

# Connection Management

connection management

When developing a complex application comprised of many communicating modules, the establishment of connections (and even the starting of modules) can become rather difficult to manage, with connection attempts (calls to conn_attempt_connection) in many different modules. To streamline this, there exists the Connection Management shell, an interpreter which reads a script file. The script file specifies which hosts modules will run on, which modules should be run and the connections between all the modules. This script allows the application designer to centralise the startup of modules and connections: making the application somewhat self-documenting.

Modules started locally have the same working directory as the script, whereas modules started on other hosts have an undefined working directory.

Modules started with the Connection Management Shell have their output redirected to logfiles in /tmp

## Script Syntax

The syntax for Connection Management Shell scripts is processed in order, and consists of 5 sections. Each section is prefixed with a keyword and postfixed with ``END''. In between the prefix and postfix is the body of the section, which contains zero or more configuration line. The section keywords and their contents are:

- HOSTLIST each line contains a hostname where modules will be run on, and zero or more optional arguments. The format is as follows:
  `hostname [KARMABASE] [-manual] [- args...]`
  where `hostname` is the remote host where you may want to run modules. You can supply an optional `KARMABASE` which specifies the value of the `KARMABASE` environment variable for modules run on the remote host. The `-manual` switch is required if the host where the script runs cannot execute remote shell commands on the remote host. Any arguments following the `-` are passed to the slave programme run on the remote host
- MODULES each line contains a module to be run. The format is as follows:
  `hostname module_name [x y] [- args...]`
  where `hostname` is the host on which to run the module, `module_name` is the name of the module to run. You can optionally specify the position of the module icon in the Connection Management Tool with `x y`. Further module-specific arguments may be specified after the `-`. The first module started is given the index number zero, each subsequent module is given an index number one higher than the previous one
- CONNECTIONS each line specifies a connection to make between two modules. The format is:
  `client_module_num server_module_num protocol_name [transport]`
  where `client_module` is the index number of the module which is making the connection (the client) and `server_module_num` is the index number of the module to which the connection is made (the server). You specify the protocol name for the connection with `protocol_name` and an optional transport layer specification with `transport` (this may be used to specify UDP or ATM SVC connections)
- COMMANDS this section is optional. Each line contains a string command to send to the command line parser of a module. The format is:
  `module_num delay command`
  where `module_num` is the index number of the module to which to send the command to, `delay` is the number of milliseconds to wait after the command is sent and `command` is the command string to send
- SHELL_CLEANUP this section is optional. Each line contains a Bourne Shell command to execute. This section requires no ``END'' postfix.

    #! /usr/local/karma/bin/cmshell

    # This file is a sample script (executable) which runs the Connection
    # Management tool in batch mode.
    # Karma modules may be started on many hosts, connected together and then made
    # to perform some function.
    # Blank lines and comments ('#' is the comment character) are ignored.

    # First the list of hosts to run on. Shell escapes allowed here
    HOSTLIST
    # hostname [KARMABASE] [-manual] [-- args...]
    # hostname "local[host]" is always defined
    lynx
    !echo "Executing a Bourne shell command"  # Note the '!' shell escape character
    asterix /usr/local/karma
    phoenix:vx -- -stereo1152x900
    obelix -manual # Wait for the user to manually make a CM slave connection
    END

    # Now the list of modules to run
    MODULES
    # Format of each module line:
    # hostname  module_name  [x y] [-- args...]
    #   x  and  y  are optional icons positions used when the Connection Management
    #   tool is operating in Interactive mode.
    #   optional arguments to be passed to the module must be separated by "--"
    local kvis -- -bg black
    asterix tx 15 15
      DEFAULTS	# Optional
        animate off
        num_iterations 5
      END	# Defaults
    END	# Modules

    # Next the list of connections to make
    CONNECTIONS
    # client_module_num  server_module_num  protocol_name [transport]
    1 0 multi_array
    END

    # And the list of commands to send to modules (via their  stdin  )
    # Shell escapes allowed here
    COMMANDS	# Optional
    # module_num  delay (milliseconds)  command
    1 5000 girl
    1 1000 animate on
    !echo "Executing a Bourne shell command"  # Note the '!' shell escape character
    1 10 script
    1 1 abort
    END

    # After processing all commands, the Connection Management tool will wait
    # forever in an event loop. When any module dies or connection closes,
    # all slave processes and controlled modules should finish. A number of shell
    # commands may then be processed prior to the Connection Management tool
    # terminating.

    SHELL_CLEANUP	# Optional Bourne shell commands
    echo "Application finished"

    # No "END" directive is required: EndOfFile is sufficient

## Security (authentication and encryption)

### Authentication

In a networked environment, any client module is able to make a connection to a server module. On the Internet, this means every machine with Internet access can potentially connect to your server module. In many (most) cases, this is not a concern since your server module will only be running a short time. However, there may be circumstances which require a more prudent approach. The Karma communications infrastructure provides multiple levels of authentication verification to support your needs.

### Encryption

Distinct from authentication is the issue of privacy. Since the Karma communications infrastructure utilises the network transport facilities provided by the operating system, the level of privacy is the same as that provided by the network layer. On a local area network (LAN), you can expect that every machine on that network is capable of spying on your data as it passes over the network. For most people, this is probably not a serious concern since their data files are also transferred over this network. Of greater concern is the possibility of information to be captured as it passes between two machines, each on a different LAN. If you are transferring data between two machines across the Internet, you really have no idea of who might be spying on your data. Karma provides strong encryption to assure your privacy.

### Purpose

The Karma communcations security support is aimed at:

- Individual users who want to secure their communications
- Application developers who want to write secure, network-aware applications without worrying about the implemtation details
- Commercial service providers on the Internet who wish to ensure their own security and that of their customers.

### Details

All connections maintained by the conn package may be both authenticated and encrypted. The configuration of this is centralised in your /.KARMAauthority file (this file should bar all access except for the owner). You may set a uniform authentication/encryption requirement for all Karma protocols, or you may selectively protect some protocols. If a protocol is not specified in your authority file, the default authentication/encryption is used. The format of this file is a follows (one line per protocol):

`protocol_name security_type [extra information]`

The <protocol_name> may be any protocol name (a specific module need not support a protocol). The name ``RAW_PROTOCOL'' designates the raw Karma protocol used by the conn package prior to negotiating a specific application protocol.

<security_type> may be any of the following:

- `key-only` IDEA used to verify authentication. Data unencrypted
- `IDEA` IDEA used to verify authentication and data encrypted
- `PGPuserID-IDEA` PGP used for session key transfer, then use IDEA mode
- `drop-encryption` drop data encryption for a specific protocol

For both the `key-only` and `IDEA` modes, the security type field must be followed by a 24 byte IDEA session key and CFB initialisation vector. This data must be in ASCII hexadecimal format. Both these modes require the secure transfer of the authority file.

The `PGPuserID-IDEA` mode is the most secure: the server module uses PGP (Philip R. Zimmermann's Pretty Good Privacy) to encrypt a random IDEA session key which is sent the the client module. The client module uses PGPdaemon (part of the PGPsendmail/Unix Suite available from:
ftp://ftp.atnf.csiro.au/pub/people/rgooch/) to decode this session key. The security type field must be followed by the authorised PGP userID. You must have PGP and PGPsendmail/Unix 1.4.1 or later installed.

The `drop-encryption` mode allows you to drop encryption for a specified protocol. This is useful when you have specified an encryption mode for all protocols (using `RAW_PROTOCOL`) and you are concerned about the performance degradation of encrypting data.

Note that in all cases, encryption is automatically dropped after protocol authentication if the connection is local (i.e. a connection to the same machine).

### Restrictions

See the Karma security guide on information regarding restrictions.

Karma Home Page
Next: Security Up: Karma Programming Manual Previous: Channels: the basic Input/Output