E-MailRelay Internals
=====================

Module structure
----------------
The main C++ libraries in the E-MailRelay code are: "glib", providing low-level
classes for file-system abstraction, date and time representation, string
utility functions, logging, command line parsing etc.; "gssl", which is a thin
abstraction over OpenSSL; "gnet", which provides network classes using the
Berkley socket and Winsock APIs; "gauth", which implements various
authentication mechanisms; "gsmtp", containing SMTP and message-store classes;
and "gpop", which contains POP3 classes. All of these libraries are portable
between POSIX-like systems (eg. Linux) and Windows.

Under Windows there is an additional library for event handling. Windows has
historically built network event processing on top of the GUI event system which
means that the "gnet" library has to be able to create GUI windows in order to
process network events. The extra GUI and event classes are put into a separate
library in the "src/win32" directory, using the namespace "GGui".

There is also a separate configuration GUI program which uses the "glib" library
together with TrollTech's Qt.

Event model
-----------
The E-MailRelay server uses non-blocking socket i/o, with a select() event loop.
This event model means that the server can handle multiple clients
simultaneously from a single thread and the only significant blocking occurs
when external programs are executed (see "--filter" and "--verifier").

In some ways this makes the implementation more complicated than the equivalent
multi-threaded approach since (for example) it is not possible to wait for a
complete line of input to be received from a remote SMTP client because there
might be other connection that need servicing.

See *C10K Problem* [http://www.kegel.com/c10k.html] for a discussion of
different network event models.

At higher levels the C++ slot/signal design pattern is used to propagate events
between objects (not to be confused with operating system signals). The
slot/signal implementation has been simplified compared to Qt or boost by not
supporting signal multicasting, so each signal connects to no more than one
slot. For historical reasons the slot/signal pattern is not used in the lowest
layers of the network library.

Event handling and exceptions
-----------------------------
The use of non-blocking i/o in the network library means that most processing
operates within the context of an i/o event or timeout callback so the top level
of the call stack is nearly always the event loop code. This can make using C++
exceptions a bit awkward compared to a multi-threaded approach because it is not
possible to put a single catch block around a particular high-level feature.

The event loop delivers all asynchronous events to the abstract "EventHandler"
and "AbstractTimer" interfaces. If these callbacks throw exceptions then the
event loop will catch them and deliver them back to the same interface through
the virtual functions onException() and onTimerException() respectively. If
exceptions are thrown out of _these_ callbacks then the event loop code lets
them propagate back to main(), typically terminating the program.

The two callback interfaces are brought together by having a concrete "Timer"
class that requires an "EventHandler" object to be associated with each timer.
The "Timer" class routes any exceptions thrown out of the timeout callback to
the designated "EventHandler" interface so that both i/o and timeout exceptions
are delivered to the same place.

In common with other event-driven frameworks this leads to a programming
model where objects are instantiated on the heap and the objects delete
themselves when they receive certain events from the framework. In the
"gnet" library the "ServerPeer" and "HeapClient" classes do this lifetime
management; instances of these classes delete themselves when the associated
network connection goes away or when an exception is thrown out their
event-handling code.

Core class structure
--------------------
The message-store functionality uses three abstract interfaces: "MessageStore",
"NewMessage" and "StoredMessage". The "NewMessage" interface is used to create
messages within the store, and the "StoredMessage" interface is used for
reading and extracting messages from the store. The concrete implementation
classes based on these interfaces are respectively "FileStore", "NewFile" and
"StoredFile".

Protocol classes such as "GSmtp::ServerProtocol" receive network and timer
events from their container and use an abstract "Sender" interface to send
network data. This means that the protocols can be largely independent of the
network and event loop framework.

The interaction between the SMTP server protocol class and the message store is
mediated by the "ProtocolMessage" interface. Two main implementations of this
interface are available: one for normal spooling ("ProtocolMessageStore"), and
another for immediate forwarding ("ProtocolMessageForward"). The "Decorator"
pattern is used whereby the forwarding class uses an instance of the storage
class to do the message storing and pre-processing, while adding in an instance
of the "GSmtp::Client" class to do the forwarding.

Message pre-processing (see "--filter") is implemented via an abstract
"Processor" interface. Concrete implementations are provided for doing nothing,
running an external executable program and talking to an external network server.

The protocol, processor and message-store interfaces are brought together by the
high-level "GSmtp::Server" and "GSmtp::Client" classes. Dependency injection is
used to create the concrete instances of the "ProtocolMessage" and "Processor"
interfaces.

Simplified class diagrams for the *GNet* [gnet-classes.png] and
*GSmtp* [gsmtp-classes.png] namespaces are available.

Windows service
---------------
To get E-MailRelay to run as a Windows service there is a service wrapper
program called "emailrelay-service.exe". This program registers itself as
a service when run with the "--install" commandline option. When the service
runs the wrapper starts the actual E-MailRelay server by looking for a batch
file called "emailrelay-start.bat" in the same directory as service wrapper
executable. It reads the contents of this batch file in order to construct the
E-MailRelay command-line, adding "--no-daemon" and "--hidden" options if they
are not there already.

The service name and display name can be added to the wrapper's "--install"
command-line, and it is the service name that is used to derive the name of the
"start" batch file. This allows more than one server to be run as services,
using different server command-line options on each one.

Diagrams
--------
Class diagrams:
* *GNet namespace* [gnet-classes.png]
* *GSmtp namespace* [gsmtp-classes.png]

State transition diagrams:
* *GNet::Client* [gnet-client.png]
* *GSmtp::ServerProtocol* [gsmtp-serverprotocol.png]

Sequence diagrams:
* *Proxy mode forwarding* [sequence-3.png]

E-MailRelay GUI
---------------
The optional GUI program "emailrelay-gui" uses TrollTech Qt v4 for its user
interface components. The GUI can run as a stand-alone configuration helper
("--as-configure") or as part of a self-extracting installation
("--as-install"). When it runs it checks whether it has a payload of packed
files. If it has then it runs as an installer; if it does not then it runs as a
configuration helper. Refer to the comments in "src/gui/guimain.cpp" for more
details.

The user interface is structured as a "wizard" having a dialog box with the
forward and back buttons at the bottom and a single Qt layout object for the
main area. A stack of Qt widgets representing the various pages of the wizard
are installed into the main layout object in turn as the user navigates from
one page to the next.

Once the wizard is completed it asks each page to dump its state as a set of
key-value pairs into a stringstream (see "src/gui/pages.cpp"). These key-value
pairs are processed by an installer class into a list of action objects (in the
"Command" design pattern) and then the action objects are run in turn. In order
to display the progress of the installation each action object is run within a
timer callback so that the Qt framework gets a chance to update the GUI between
each one.

During development the user interface pages and the installer can be tested
separately since the interface between them is a simple text stream containing
key-value pairs.

When run in "--as-configure" mode the GUI normally ends up simply editing
the "emailrelay.conf" file and/or the "emailrelay.auth" secrets file.

When run in "--as-install" mode the GUI expects to unpack all the E-MailRelay
files from a "payload" archive into target directories.

The packing scheme used for a payload archive is a simple concatenation of the
"stub" executable followed by a table of contents for the payload files,
followed by the payload files themselves (possibly compressed by "zlib"), and
ending with an twelve-byte ascii representation of the offset of the table of
contents.

The payload may be appended to the GUI executable to make a self-extracting
installer, usually then called "emailrelay-setup".

However, on Windows two levels of packing are required: the "emailrelay-setup"
program has a stub executable written in "C" that prints an "extracting..."
message to the standard output, with a payload comprising another packed
executable and a small number of "C++" runtime library files. The inner packed
executable has the emailrelay GUI program as its stub and all the other
installable files, including the main emailrelay executable, as its payload.

On unix-like operating systems it is more natural to use some sort of package
derived from the "make install" process rather than an installer program, so
"emailrelay-setup" ia typically never built.

On Mac OS X the payload is better stored in the installer's application
bundle rather than simply appended to the binary. This is desribed in the next
section.

Windows build
-------------
On Windows E-MailRelay can be built using MinGW or Visual Studio. Makefiles and
Visual Studio 2012 project files are provided in the "src" directory.

For a MinGW build of the E-MailRelay server without TLS/SSL support it should
be sufficient to run make from the "src" directory:

	C:\emailrelay\src> PATH=c:\mingw\bin;%PATH%
	C:\emailrelay\src> mingw32-make -f mingw.mak

To add TLS/SSL support first install ActiveState perl and make sure that your
MinGW installation contains the MSYS subsystem. Unpack the OpenSSL tarball and
build it as follows:

	C:\openssl> PATH=c:\perl\bin;c:\mingw\msys\1.0\bin;c:\mingw\bin;%PATH%
	C:\openssl> bash -l
	$ ./config
	$ mingw32-make

Then edit the E-MailRelay "src/mingw-common.mak" file to enable openssl and run
"mingw32-make" as above.

If building the E-MailRelay GUI then it is best to use MinGW with Qt 5 static
libraries. Start by installing zlib source (eg. to c:/zlib) and then build Qt
using the following "configure" options:

	-prefix /c/qt/qt5-static
	-I c:/zlib
	-L c:/zlib
	-static
	-release
	-force-debug-info
	-separate-debug-info
	-opensource
	-confirm-license
	-no-c++11
	-fully-process
	-no-largefile
	-accessibility
	-no-sql-sqlite
	-no-sql-sqlite2
	-no-javascript-jit
	-no-qml-debug
	-platform win32-g++
	-no-sse2
	-no-sse3
	-no-ssse3
	-no-sse4.1
	-no-sse4.2
	-no-avx
	-no-avx2
	-no-neon
	-no-mips_dsp
	-no-mips_dspr2
	-no-pkg-config
	-system-zlib
	-no-gif
	-qt-libpng
	-no-libjpeg
	-no-openssl
	-qt-pcre
	-qt-xcb
	-qt-xkbcommon
	-gui
	-widgets
	-no-rpath
	-no-optimized-qmake
	-no-nis
	-no-cups
	-no-iconv
	-no-icu
	-strip
	-no-pch
	-no-dbus
	-no-xcb
	-no-eglfs
	-no-directfb
	-no-linuxfb
	-no-kms
	-no-opengl
	-no-system-proxies
	-no-glib

Start the Qt build by running "mingw32-make" from the "qtbase" directory and
finish off with "mingw32-make install".

Edit the E-MailRelay GUI makefile "src/gui/mingw.mak" so that the E-MailRelay
build uses similar compiler options to the Qt examples and then then build with:

	$ mingw32-make -f mingw.mak

Mac OS X packaging
------------------
On Mac OS X the standard "configure ; make ; make install" procedure works best
if the "configure" script is given Mac-like directories on its command-line. The
script "bin/configure-mac.sh" can be used to do this.

The "make" step in "src/main" on Mac OS X additionally builds a simple wrapper
program "emailrelay-start" from "start.cpp" that runs the E-MailRelay server
using a command-line assembled from the "emailrelay.conf" file. This is then
used from the "E-MailRelay-Start" application bundle to provide a Mac-friendly
way of running the server.

Similary the "make" in "src/gui" builds a wrapper program "emailrelay-start-gui"
from "guistart.cpp" that runs "emailrelay-gui.real" and this is used from the
"E-MailRelay-Configure" application bundle.

The self-extracting installer scheme (described above) does work on Mac OS X,
but it is more sensible to use a Mac application bundle (E-MailRelay-Install) to
hold the payload and then wrap the whold thing up in a ".dmg" disk image. The
GUI code supports this by looking for a separate file called "payload" and using
that in preference to any payload archive it finds appended to its own executable.

The format of the "payload" file in the application bundle is the same as is used
in a self-extracting installer.

The makefile in the "src/gui" directory provides the "image" target to create
the E-MailRelay-Install application bundle and the disk image.

Source control
--------------
The source code is stored in the SourceForge "svn" repository. A working
copy can be checked out as follows:

	$ svn co https://svn.code.sf.net/p/emailrelay/code/trunk

Directory structure
-------------------

# src

  Parent directory for source code.

# src/glib

  A low-level class library, including classes for file-system abstraction,
  date and time, string utility functions, logging, command line parsing etc.

# src/gnet

  A network library using Berkley sockets or Winsock.

# src/gssl

  A library for using OpenSSL.

# src/gauth

  A library for SASL and PAM authentication.

# src/gsmtp

  An SMTP library.

# src/gpop

  A POP3 library.

# src/win32

  Additional classes for windows event processing.

# src/main

  Application-level classes for E-MailRelay.

# src/gui

  Installation and configuration GUI program using Qt v4.

# lib

  Parent directory for ISO C++ fixups for various older compilers.

# test

  Test scripts and utilities.

Source file names
-----------------
Generally the source file names are follow the name of the principal class,
(often including the namespace) but all in lowercase. Any underscores in the
name indicate a choice of implementation, so class "G::Foo" might have two
implementations in the files "gfoo_main.cpp" and "gfoo_alternate.cpp".
The choice is normally made by the makefile.

Portability
-----------
The E-MailRelay code is written in ISO C++, although avoiding less-widely
supported language features such as "mutable", templated methods and "export".

The header files "gdef.h" in "src/glib", and "gnet.h" in "src/gnet" are intended
to be used to fix up compiler portability issues such as missing standard types,
non-standard system headers etc. Conditional compilation directives ("#if"
etc.) are confined to these headers as far as possible in order to improve
readability.

Deficiencies in the standard headers files provided by older compilers are fixed
up by files in the "lib" directory tree. For example, the msvc6.0 compiler
sometimes does not put its names into the "std" namespace, even though the
std-namespace headers are used. This can be worked round by additional "using"
declarations in the "lib/msvc6.0" headers.  These work-rounds are kept out of
the "src" tree because they are not necessary for more modern compilers.

Windows/unix portability is generally addressed by providing a common class
declaration with two implementations. Where necessary a "pimple" (or "Bridge")
pattern is used to hide the system-specific parts of the declaration.

A good example is the "G::Directory" class used for iterating through files in
a directory. The header file "src/glib/gdirectory.h" is common to both systems,
but two implementations are provided in "gdirectory_unix.cpp" and
"gdirectory_win32.cpp". The unix implementation uses opendir() and glob(),
while the windows implementation uses FindFirstFile().

Sometimes only small parts of the implementation are system-specific. In
these cases there are three source files per header. For example, "gsocket.cpp",
"gsocket_win32.cpp" and "gsocket_unix.cpp" in the "src/gnet" directory.

Compile-time features
---------------------
Compile-time features can be selected with options passed to the "configure"
script. These include the following:

* Debug-level logging ("--enable-debug")
* IPv6 (Linux only) ("--enable-ipv6")
* Configuration GUI ("--enable-gui")
* PAM support ("--with-pam")

Some functionality can be disabled at compile-time in order to reduce the size
of the executable, typically when building for embedded systems:

* Disable POP3 protocol, "--disable-pop"
* Disable authentication, "--disable-auth" (requires "--disable-pop")
* Disable administration interface, "--disable-admin"
* Disable execution of external programs, "--disable-exec"

The "--enable-small-config" option can be used to change the command-line
parsing code to use a configuration file instead, resulting in a smaller
executable. This also removes a lot of the configuration checking code, so it is
not recommended unless size is critical. (The format of the configuration file
is similar to the command-line using the long-form options without the
double-dash and using '=' to separate the option from the option value.)

Use "./configure --help" to see a complete list of options and refer to
"acinclude.m4" for more detailed comments.

Idioms
------
The "<<=" operator defined in "src/glib/gmemory.h" is used idiomatically
to reassign a std::auto_ptr<> since reset() is not always available.



Copyright (C) 2001-2013 Graeme Walker <graeme_walker@users.sourceforge.net>. All rights reserved.
