
# $Id: README,v 1.4 2001/10/11 16:52:22 beck Exp $

     LZS (aka Stac aka HiFn) compression support for I4L SyncPPP

Sometime in 1998 I picked up a question in the de.alt.comm.isdn4linux
newsgroup asking "Is there Stac compression support for I4L ?". Due to a
strange coincidence, I had found out before on this same day that RFC1974
is a somewhat complete documentation of the LZS CCP stuff and contains enough
information to code a decompressor and compressor. Thus I answered the
question like "a decompressor requires only some effort, a compressor
is somewhat more complicated but clearly possible". At this time the real
problem was that the isdn_ppp stuff in I4L was not ready for CCP modules at
all, most hooks were still missing. But then Michael Hipp implemented
the hooks in the newer CVS source of I4L and got BSD compression basically
working, so the ball was at my foot again. Thus I decided to code that
decompressor and eventually code a compressor, too.


                            LICENSE

The code is Copyleft 1998-2001 by Andre Beck under terms of the
GNU GPL. While the code is provided free and under GPL, it may infringe
patents Stac has applied for in the US and maybe other countries. Verify
that this is not the case in your country before using it. This code is a
private development related in no way whatsoever to my employer, written in
Germany where software patents are still considered bogus. This
situation might change any day due to EU law. Linux distributors who
supply this stuff shrink wrapped should reconsider doing this (not only
for isdn_lzscomp, but as well for isdn_bsdcomp, which is likely to
infringe on the infamous Unisys LZW patent) with their law departement.
See also the PATENT ISSUES section.


                            STATUS

After three years without a lot of changes, the code should still be
considered beta. There are two different things here, the actual LZS
implementation (isdn_lzscomp.c) and the CCP framework (to be found in
isdn_ppp.c) in conjunction with the necessary changes to ipppd.

a) The compression module
   The module appears to be working well. It could still get some reworks
   (last but not least some comments are outdated and contrary to the
   actual code), though. One of the things to be reworked is the does-or-
   does-not-compress heuristics. Some code is already there, but not
   enabled yet asl long as there is no clear API for the module to know
   the MTU of the frames it can generate. Another potential improvement
   would be to change the default for tweaks to no tweaks at all - but
   then again, it's a lot of Ascends out there. If there would ever be
   more peers supporting multihistory LZS, trying a basic multihistory
   extension to the compressor would be quite interesting, but at the
   moment it appears mostly useless.

b) ipppd
   The changes to ipppd have settled since the first implementation,
   the version of ipppd in current distributions already contains them.
   I had supplied patches later that disabled CCP per default, but I'm
   not sure whether they made it everywhere. The code generally works,
   but it is known not to be flawless. Primarily, it doesn't gracefully
   terminate when the peer is rejecting our CCP ConfReqs, nor does it
   try other parameters that may be supported by the peer. It could be
   improved, ideally by querying for multihistory SEQ mode first, then
   falling back to single history SEQ and then trying EXT, with a proper
   bailout of CCP if all of them fail.

c) The CCP framework
   That's the most unstable part of the whole thing. We had to expand and
   change some portions of isdn_ppp, mainly those dealing with Reset
   Requests and -Acks. This code is raw and not really finished, primarily
   because I didn't want to hack around in isdn_ppp but just wanted to write
   the LZS module. But LZS CCP is more complicated than most other CCP
   engines and required those changes. I hope someone with more
   insight into isdn_ppp will adopt and complete them.
   - The state engine that generates and deals with ResetReqs and -Acks
     is quite a hack. I am not sure about it's fitting into the kernel
     parallelism aspects.
   - The code is trying to deal with both, the normal (bundle) and the
     link compression variants of CCP. The latter is normally not used,
     and still completely untested (and expected to fail).

As a conclusion, the more betaish aspect of the whole thing is the CCP
framework with which you have to deal if you try any CCP module. For LZS,
the module is somewhat well tested and seems to not have obvious bugs,
though some deficiencies. Interaction with MP in newer kernels seems to
be rather flakey at least in EXT mode. If you don't need MP, disable it
in ipppd removing "+mp" from the config options. We're working on it.


                           BUILDING

The easiest way to build the module is to copy isdn_lzscomp.c to
linux/drivers/isdn/ and to edit the Makefile in this directory, adding
a line for LZS. The structure of the kernel Makefiles has changed over
the years, so this is best done using the patch provided by Frank Elsner
for most kernel versions. The patch even includes changes to the kernel
configuration help and config file, so building the code can be triggered
with whatever config frontend you like.

The module will be built on the next "make modules" and installed with
"make modules_install", or the distribution dependent mechanisms that
replace that (like Debian make-kpkg and dpkg). But if you want, you still
can build the module standalone and copy it wherever you see fit, but deal
with the oddities (aka MODVERSIONS) yourself.


                            USAGE

If you want to use any CCP method with I4L, you must first load the
relevant module(s) explicitely into the kernel. This is best accomplished
either when loading the basic isdn modules, or in the script that sets up
the interfaces and ipppd for the connections. After loading the module(s)
you start the connection. You may need to give options to ipppd in order
to make sure negotiations succeed, see below.


                       Supported Modes

This set of sources supports LZS check mode 3 (the default of RFC1974)
and LZS check mode 4 (the derivative used by Micro$oft in Win0.95). If your
peer is an Ascend Max, the techs of the ISP might use another nomenclature:

What it is		Ascend NVRAM Menu	Ascend RADIUS

RFC1974 LZS SEQ Mode	Stac-9 (named after	Link-Comp-Stac-Draft-9
(check mode 3)          draft 9 of the RFC)

RFC1974 LZS EXT Mode	MS-Stac			Link-Comp-MS-Stac
(check mode 4)


Ask your ISP for what their NAS speaks. Either mode is Ok, but if you
insist on non-M$, I'm virtually yours. The compression deployed by german
mass dialin provider T-Online in mid-2001 (beeing the reason for much
increased usage of this code) is the EXT mode variant.

Note: If your peer sends you CCP with the LZS identity (0x11) but a
      length of 6 bytes, it is not compatible with RFC1974 but uses
      a proprietary predecessor of it. Typical case is an Ascend Max
      that has never been set to something else but the default "Stac"
      in "Answer" and doesn't override this per-profile, either. Ask
      your ISP about changing this, nobody (but other Ascend boxes)
      will ever be able to use this compression. Especially not Win95,
      if the ISP wants interoperability with Win95 he needs to set
      MS-Stac, and then this module works as well. In a later patch
      ipppd was modified to detect that variant of pre-RFC LZS and
      name it correctly in the logs.

If the NAS you are talking to will not negotiate gracefully you may want
to wire the exact mode ipppd is requesting. For instance, an Ascend Max
with a profile set to Stac-9, when asked for any number of histories but
one or asked for check mode 4 will not NAK these options with the ones
it likes better, but will reject LZS altogether. Your only chance to get
it to negotiate LZS is to use the exact options it wants in the first
place. To achieve that, ipppd got a new parameter:

	lzs [recv-hists[:recv-mode[,xmit-hists[:xmit-mode]]]]

		recv-hists	Downstream # of Histories	[8]
		recv-mode	Mode used downstream		[3]
		xmit-hists	Upstream # of Histories		[recv-hists]
		xmit-mode	Upstream mode			[recv-mode]

This option is, as always, accompanied by

	-lzs
	nolzs

to prevent any LZS negotiations from happening. Some examples:

	lzs 1			Mode 3 with 1 hist in both directions. This
				will negotiate against a Max at Stac-9

	lzs 1:4			Mode 4 with 1 hist in both directions. This
				will negotiate against a Max at MS-Stac

Multihistory is reported to work as expected by some users that found the
occasional NAS supporting it at their ISP. There is only downstream support
for multiple histories (improving significantly the overall compression
ratio as soon as more than one flow is active on the same PPP link), to
support it upstream would require nontrivial code extensions.


                        Module options

When loading the isdn_lzscomp module, some options are available to tweak
it for ones personal desires:

	comp=n			n=0..9 [0]
				Specifies the level of compression to be used.
				By default, compression is switched off, so
				only decompression (downstream) is available.
				Increasing numbers increase the ratio that
				can be reached as well as the CPU ressources
				eaten up for that.

				0:	No compression at all.
				1:	Absolutely minimal compression.
				...
				7:	Good compression.
				8:	Heavy compression.
				9:	Ultimate compression, but eats more
					cycles than most CPUs can dream of.

				If you want to try upstream compression,
				start with 7. Try 9 only for fun, it is
				rather useless (compresses only 1 or 2%
				better than 8, but a P100 facing that
				amount of brute force becomes sluggish,
				jumping mouse etc) - just not worth the
				burned CPU (even if you have a GHz oven).

	debug=n			n=0..3
				Increasing levels of debug messages from
				the LZS code. 3 includes full packet dumps
				before and after compression and decompression.
				More than 1 is overkill if you don't hunt
				bugs. Even 1 has some rather informational
				output. BTW, that doesn't control the
				messages from the CCP framework.

	tweak=n			n=tweak-flag-mask
				Tweaks some aspects of the module to behave
				strange or even violate RFCs in order to talk
				to RFC-impaired peers. For Ascend, use 0x7.
				If you want to know why, UTSL.
				For historic reasons, the tweak parameter, if
				left unspecified, defaults to 7 (the Ascend
				tweaks). If your peer is something else, you
				should therefore try tweak=0 to not upset the
				peer with erratic behavior that is not en-
				tirely wrong, but then again, is only right
				if you are an Ascend box.


                        PATENT ISSUES (argh)

Please check whether using this code constitutes a crime in your country.
While I just implemented a module that can source and sink the data format
described by Stac/HiFn in RFC1974, they claim to have US Patents on
practically every aspect of this format. Thus I assume that selling this
code violates these US Patents. As this code is

a) Freeware under GPL and
b) Developed in Germany

I hope I wont get any terror from the lawyers of the patent holders. I'm
sure that the availability of this code will actually help HiFn sales
because more users (the Linux community) will ask their ISPs for LZS
support on their NASes and these NASes will contain HiFn chips for that
purpose. Then again, after they claimed even deflate would be covered by
their patents and tried to sue Netscape for using deflate in the PNG code
of Mozilla, they are probably not on the planet for good will games.
We'll see.

FMI, surf http://www.delphion.com/ and search for "stac".


                               BUGS

Over the years, there have been some bugs tracked down and eliminated that
tended to crash the whole thing (the kernel, that is, so beware). Another
problem was introduced to the CCP framework in 2.4.0 and removed in 2.4.1.
There are problem reports for 2.2.19 currently under investigation. I'm still
running an older 2.2 version and don't have much reason to upgrade.


                               FMI

The source code isdn_lzscomp.c is heavily commented (so I can still read it
in a month). It contains a thourough description of the compression algo-
rithm used and how I came to it. This in turn involves Credits to the
net.persons who helped me with getting that compressor faster. When we
are at it:


                             Credits

Phil Katz and L. Peter Deutsch
	for the zlib and deflate format and specifications

Jean-loup Gailly and Mark Adler
	for deflate, gzip and zlib as well as the comp.compression FAQ

David Carr
	for a lot of general tips, hints and tricks speeding up a
	compressor, pointers to freeze and the sigma buffer and especially
	explaining to me the concept of the barrier hash until I finally
	got it

Leonid Broukis
	for the freeze implementation with the sigma buffer which
	inspired my 8-buffer and for pointing out how memcmp(3) can
	be used in match searching

Ojala Pasi 'Albert'
	for general tips and his website which was the first reading
	I had about speeding up a compressor beyond brute force

Frank Elsner
	for creating a full-blown kernel patch out of the module source
	and adapting it to all the changes that I had no time to follow,
	as well as supplying it on his website and giving support on this
	topic in Usenet and mailing lists.
	http://www.tu-berlin.de/~elsner/ISDN/


                             Contact

You can reach me by EMail: beck@ibh.de
URL: http://www.ibh.de/~beck/stuff/lzs4i4l/

Discussion will take place at de.comp.os.unix.linux.isdn, which recently
took over the remains of de.alt.comm.isdn4linux, the former Usenet mirror
of the I4L mailing list.


