ZMailer Overview

[ This is mostly from 1988-1990 by Rayan Zachariassen,
  some additional notes by  Matti Aarnio, 1994-1997 ]


- What is this?

  This is a package that implements an internet message transfer agent
  called ZMailer.  It is intended for gateways or mail servers or other
  large site environments that have extreme demands on the abilities of
  the mailer.  It was motivated by the problems of the Sendmail design
  in such situations.

  ZMailer is intended and designed as a multi-protocol mailer.  The only
  protocol supported in this distribution is RFC822 (and variations).


- What good will it do me?

  Different people and sites differ in priorities.  Keep that in mind
  when reading this ``features'' list:

  Code and Design features:

  + Strong limits on host impact, system loading is deterministic.
  + Secure design (and hopefully implementation).
  + Natural fit for client/server environments.
  + Extremely customizable configuration mechanism.
  + Flexible database interface with support for: sorted files, unsorted
    files, dbm, ndbm, gdbm, nis (yellow pages), dns (BIND resolver),
    /etc/hosts file, and in-core data.
  + Efficient message queue management.
  + Low-technology implementation (with high-technology used,
    when available.)
  + Fast binary-transparent SMTP server and client.
    [ It is still binary transparent, but MIME support has slowed
      it slightly.  Most propably it will still be able to flood
      your Ethernet with a 10 MB SMTP transfer, so beware ;-) ]


  Default configuration file features:

  + Default configuration will work for most sites.
  + Network protocol support for: smtp, uucp, bitnet, mail to news.
  + An easy way of overriding any external routing information.
  + Automatic handling of mailing lists.

- What experience shows that it can do?

  Benchmarking mailers is difficult -- on the other hand, we can
  try to see how many single recipient messages are processable
  per day, or we can see how many multi-hundred recipients are
  processable per day..  The Zmailer is used in all kinds of
  environments with their extremely varying load behaviours.

  Original developement systems had loads of 1000-2000 messages
  a day (somewhat more route decissions, like 2-3 times that),
  which by todays (-96) standard is nothing to speak about, but
  multi-router mode has enabled serious processing to happen
  which sends out 20-50 THOUSAND messages a day per process,
  and total number becoming in range of millions..

  On one burst-load test the system did show up to handle about
  12500 messages a day per router process (three route lookups
  on each message from DNS across the Ethernet).
  If system can take more route processes, it is most likely
  possible to increase system performance to hundreds of thousands
  messages per day..   There is no conclusive evidence that next
  possible bottle-neck, scheduler, won't clog, but there appears
  to be ample power (if your machine has it..) -- 50 000 messages
  a day is no problem.  ( Sun SS-10/41 ) [ Apr-1994 ]

  Another burst-load test routed 1000 messages to "nobody" (via
  local alias db to "/dev/null") on an Sun SS-10/50 MHz/Solaris 2.4,
  gave speed of  120 000 messages per day per router process (the
  test was run with 4 parallel routers)

  On same machine with scheduler from zmailer-2.99.15, scheduling
  the messages took a bit under two minutes (1:55), which indicates
  speed of roughly 750 000 messages per day.  Very likely it can
  exceed million messages per day (not million recipients, like
  when expanding lists, but million individual messages!) [ Aug-1995 ]

  With rewritten scheduler (zmailer-2.99.19: scheduler-new) the speed
  improved still:
	- 1000 messages to same /dev/null:
		84 seconds -> 1.0 Million messages per day
	- 1000 messages to 10 different /dev/nulls:
		84 seconds ...
  The speed is actually dominated by the speed the scheduler can
  assimilate information from new entries.

  Newer versions tested for burst performance on my workstation
  with  Linux-2.1.7 kernel, AHA2940 controller, PPro-200, 64 M RAM,
  and multiple partitions on single Seagate ST15230N disk (suboptimal
  for performance) gives following numbers:
	- Submitting  1000  messages into router without anything
	  else running in the system: 70 seconds (SH script forking
	  off for each submission)
	- Routeing those messages from cold-started router: 42 secs
	  (with ONE router)
	- Getting them scanned into the scheduler: 8 seconds
	- Running deliveries with fsync() at every possible occasion
	  into my own mailbox: 30 seconds ( ==> 2-3 M deliveries
	  per 24h )
  Running it with all subsystems active goes in about 90 seconds
  thru submitting, routeing and delivering with system load-average
  raising only to 1.1 ...   The speed was (acoustically) dominated
  by disk-IO along with IPC thru a socketpair().

  A colleque of mine ran a recent ZMailer (2.99.44 I presume) between
  two PPro-200 Linux machines over a dedicated 10base-2 ethernet.
  He submitted messages at host-A into router as fast as he could,
  and routed them all to the other machine over SMTP to /dev/null
  there.  The transfer-rate at the Host-A was about 500 000 per hour;
  or 10-12 million per day...  The test was run for two hours to
  measure true throughput figures, not only bursts!
  (Heard about this on 3-Feb-97.)


  The ZMailer is used on various multi-domain machines, including
  ones running Internet, ( and BITNET, and/or UUCP ).


- How does it work?

  ZMailer is a multi-process mailer, using two daemon processes to
  manipulate messages.  One of these  processes is a router (which
  may consist of N parallel processes), and makes all decisions
  about what should happen to a message.  The other daemon is a
  message queue manager, used to schedule delivery of messages.

  The Router uses a configuration file that closely follows Bourne-
  shell script syntax and semantics with minimal magic.  Message files
  are moved around in a series of directories, and the Scheduler and
  its Transport Agents run off of control files created by the Router.

  The Router will process messages one at a time, as it finds them in a
  directory where User Agents submit their outgoing messages.  Envelope
  and Message Header information is all kept in the same message file
  along with the message body, and this file is never modified by any
  ZMailer program.  After parsing the envelope and RFC822 header
  information, the Router validates the information extracted, and
  calls functions defined in the configuration file to decide exactly
  how to deliver the message and how to transform the embedded addresses.
  The algorithms that do this are easily reconfigurable, since the control
  flow and address manipulation is specified by familiar shell script
  statements.  When the Router is finished, it will produce a message
  control file for use by the delivery processing stage of ZMailer, and
  move the original message file to another location.

  Once the Router has decided what to do with each of the addresses in
  a message, the Scheduler daemon builds an internal summary of this
  information by reading the control file created by the Router.
  This knowledge is merged with a data structure it maintains that stores
  which messages are supposed to be sent where, and how.  According to
  a pre-arranged agenda, the Scheduler will execute delivery programs to
  properly move the message envelope, header, and body, to the immediate
  destination.  These delivery programs are called Transport Agents, and
  communicate with the Scheduler using a simple protocol that tells them
  which messages to process and returns status reports to the Scheduler.
  The Scheduler also manages status reports, taking appropriate action on
  delivery errors, and when all delivery instructions for a message have
  been processed, deletes remnants of the message from the spool.

  There are several standard Transport Agents included with the ZMailer
  distribution.  The collection currently includes a local delivery program,
  an SMTP client implementation, and a Transport Agent that can run
  Sendmail-compatible delivery programs.

  A separate utility allows querying the Scheduler for the state of its
  mail queues.  For existing Sendmail installations, a Sendmail replacement
  program is included that simulates most of the Sendmail functionality
  in the ZMailer environment.  This allows ZMailer to replace a working
  Sendmail installation without requiring changes in standard User Agents.

  If you have a client/server host environment, mail clients need not run
  any daemons.  They only need two programs (the Sendmail replacement, and
  a mail queue querying program) and a view of the mail server's spool area.


- How will it affect my current system?

  Two programs will be replaced: /usr/lib/sendmail (with a functionality
  emulator), and /bin/rmail (for completeness).  Nothing else will be
  changed, so it is very easy to back out if you want to.

  Per default this replacement happens with moving the original binaries
  to alternate name, and then placeing appropriate symlinks to MAILBIN
  for those program.


- What problems can I expect?

  This software is in use on several major mail gateways.  The development
  machine is my workstation, and my main testbeds are very popular high-
  volume mail-servers.  This mailer has been running in them in some form
  for 5 years, and before that a lot in Toronto since 1980es..
  This means that you are unlikely to see crippling basic problems in
  the software, though occasionally my "unerring coding" appears to be
  without the "un"-prefix..

  You may well encounter problems due to your local Operating System,
  libraries, compiler (this software makes a good compiler test suite),
  or other things about your environment that I have not considered.

  The ZMailer Router RFC822 implementation is rather picky about adherence
  to the protocol standards.  You may run into cases where an error that
  has been ignorable or unknown so far, will generate automatic complaints
  from the mailer to users, and from users to you.  Much of this pickyness
  is required.  Some is not, and is deselectable.

  Due to the security mechanism, in particular the propagation and use of
  privilege levels associated with addresses, the default setup requires that
  the .forward and mailing list files be publically readable.  Otherwise,
  unprivileged addresses (e.g., those specified from outside the host), will
  not be able to use the contents of such files.  Similarly, if the source
  of an address is not ``secure'', the address will have no privileges.
  In particular, for an address to take on the privileges of whomever
  specified it (as in a .forward file), its source file must be owner-only
  writable, and in a directory which has the same ownership or is owned by
  root and is also owner-only writable.  These checks may be disabled.


- What level of support can I expect?

  I will promptly fix problems that I consider serious if they are
  reproducible on a system I have easy access to.  Any comments or
  reports you send in will be saved and considered.  Because of my
  other activities, I cannot commit to answering non-critical items
  in a timely manner, or at all.
  [940829: 'I' has latter turned out to be <mea@nic.funet.fi> ]

  ... and if you want to pay for support, ask  <mea@nic.funet.fi>

  There is a mailing list you can join if you would like the ear of
  a community.  Send subscription requests to:
			<zmailer-request@nic.funet.fi>
  or preferrably use <mailserver@nic.funet.fi> and send there a request:
		subscribe zmailer Your Name


- How do I install this?

  If you are an SMTP (and possibly UUCP) site running a supported OS,
  you can use a supplied host environment description and the default
  Router configuration files.

  Read the file "INSTALL" for detailed installation information.

  Read the file doc/guides/aliases for information on local address
  interpretation, and doc/guides/lists for a quick introduction to
  setting up mailing lists.
