# Copyright: 2001-2004 The Perl Foundation.  All Rights Reserved.
# $Id: rx.dev 7859 2005-04-18 03:49:02Z chromatic $

=head1 NAME

docs/dev/rx.dev - rx.c and rx.h

=head1 SUMMARY

rx.c and rx.h set up functions to be used by the regular expression engine.  
They also define internal helper functions that add a layer of abstraction to
the rx_is_X family of functions.  Please also see C<rx.ops>, C<rxstacks.c>, 
and C<rxstacks.h>.

=head2 rx.c

=over 4

=item B<rx_allocate_info>

Initializes a regular expression object and allocates the memory.

=item B<rx_is_word_character>

=item B<rx_is_number_character>

=item B<rx_is_whitespace_character>

=item B<rx_is_newline>

These functions check if the character passed as an argument is a 
word_character, number_character, whitespace_character, or a newline, 
respectively.  They each use bitmaps to add a layer of abstraction.  All a 
bitmap is in this case is a collection of characters.  Instead of manually 
looking at a string these functions create a bitmap of allowable characters 
(using predefined constants, like RX_WORDCHARS), and call the function 
C<bitmap_match>, which checks if the supplied character is in the 
bitmap.  Not only do bitmaps add abstraction, but they provide a significant 
increase in speed over a linear search.


NOTE: The C<rx_is_number_character> function breaks the abstraction and 
uses the following expression to test the argument:

	if (ch >= '0' && ch <= '9')

It explains that it is "faster to do less-than/greater-than"
Basically, this is just a speed hack for now, it will change when it needs to 
be changed (to add different encoding/language support).

=item B<bitmap_make>

This function makes a bitmap from its argument (of type STRING*).  Let us 
examine two cases, one is a character is one byte, the other is it is more.

=over 1

=item B<One byte>

First of all, (255 >> 3) = 31.  The code uses this for a little efficiency in 
storage/speed.  An internal array is created with 32 elements (each byte-sized).
If you take the input character and right shift it by 3, you will get a number
between 0 and 31, it just so happens that exactly 8 numbers between 0 and 
255 map to the same number between 0 and 31.  Then, each element in this array
is a bitfield, with a 1 or 0 in each bit to indicate if a particular character
is in the bitmap or not.  So, (ch >> 3) takes us to the right element in the
array for ch, but how do we get to the right element in the bitfield?  The 
code is 1 << (ch & 7).  This will give us a unique power of two for each
character that maps to that particular bitfield in the array.

=item B<More than one byte>

Here each character is appended to the internal string bigchars (of type
STRING*).

=back

=item B<bitmap_make_cstr>

This is the same thing at bitmap_make, except it is called with a const char*
argument.  Because of this, it knows there will be no bigchars, so it is only
concerned with byte-sized characters.

=item B<bitmap_add>

This function takes a bitmap and a single character, and adds that character
to the bitmap.  The code for adding the character is the same as in bitmap_make.

=item B<bitmap_match>

This functions takes a bitmap and a single character, and checks to see if that
character is in the bitmap.  If the character is more than one byte, then the
function searches the bigchars string linearly (one by one).  If it is a
byte-sized character than it checks the appropriate bitfield, as specified in
bitmap_make.

=item B<bitmap_destroy>

This deallocates the memory for the bitmap.

=back

=head2 rx.h

Here is the definition for rxinfo (all comments are mine)

	typedef struct rxinfo {
	    STRING *string;	//This is the string the regex tests to see if it matches or not
	    INTVAL index;	//This is the current spot in string we are checking
	    INTVAL startindex;	//This is where the regex started checking
	    INTVAL success;	//This is just a flag to see if the regex matched or not

	    rxflags flags;	//This is a set of flags to see what modifiers were used in the regex
	    UINTVAL minlength;  //The minumum length string can be and still be able to match
	    rxdirection whichway; //Is the regex going forwards or backwards?

	    PMC *groupstart;	//Indexes for where each group starts
	    PMC *groupend;	//Indexes for where each gruop ends
				//Groups here are capturing groups, ie. $1,$2, etc.

	    opcode_t *substfunc; //This is unused.  Originally regexes were going to
			 	 //handle their own substitutions (s///).  Now this
				 //is not the case.  This can probably be removed.

	    IntStack stack;	//Sets up an intstack for internal use (backtracking purposes)
	} rxinfo;


rx.h also sets up a series of macros for setting/unsetting flags in each regex,
advancing the regex one char (or a given number of chars), and finding the
current index.  Here is the list of the macros, check the rx.h file for their
definitions.

=over 4

=item B<RX_dUNPACK(pmc)>

=item B<RxCurChar(rx)>

=item B<RxAdvance(rx)>

=item B<RxAdvanceX(rx, x)>

=item B<RxCaseInsensitive_on(rx)>

=item B<RxCaseInsensitive_off(rx)>

=item B<RxCaseInsensitive_test(rx)>

=item B<RxSingleLine_on(rx)>

=item B<RxSingleLine_off(rx)>

=item B<RxSingleLine_test(rx)>

=item B<RxMultiline_on(rx)>

=item B<RxMultiline_off(rx)>

=item B<RxMultiline_test(rx)>

=item B<RxReverse_on(rx)>

=item B<RxReverse_off(rx)>

=item B<RxReverse_test(rx)>

=item B<RxFlagOn(rx, flag)>

=item B<RxFlagOff(rx, flag)>

=item B<RxFlagTest(rx, flag)>

=item B<RxFlagsOff(rx)>

=back

=head1 BITMAPS

This section describes the use of bitmaps in rx.[ch] and briefly 
describes the other possible choices and why bitmaps are used over them.  
The advantage to using bitmaps is both in speed and storage.  After 
starting to use bitmaps, there was a noticeable speed increase.

=over 4

=item B<Array>

Ideally it would be nice to have an array of bits.  If you wanted to 
see if the character represented by code 130, say, was in the bitmap, all 
you would do is check the 130th element ( C<array[129]> ) of the array, 
and see if it's a one or a zero.  Of course, we don't have access to an 
array of bits.  The closest we could come is an array of chars, which 
would waste 7*255=1785 bits of data.  It would still be the fastest 
method (constant time, only one operation required), but it would waste 
a considerable amount of space.  More on using arrays when we get to 
bitmaps.

=item B<Linear Search>

This method uses a string, and appends all characters to the end of it 
as they are added to the bitmap.  To check if a character were in the 
bitmap, you would need to scan each entry in the string one by one.  This 
is slow (linear time), and it still wastes space.  It turns out that this 
is actually how characters > 255 are handled.  This is a good method for 
that, but for characters < 256 we can make use of some optimization.  

=item B<Array/Bitfields>

In the linear search, we are storing a character (1 or more bytes)--it would
be nice if we could store a I<flag> (1 bit) instead, and use the character as
a key to get to the flag.  The array method did that, but it wasted space 
as well.  Since each element of the array in 8 bits, then it is possible 
to have an array of bitfields.  There will be an equation that maps the 
key to the right array element, and then another that maps it to the 
right bit in the bitfield.  So then, let's examine how we can do this:

First of all, to see how many array elements we need we can divide 256 by 
8 (since characters above 256 are handled differently, we are only 
concerned with byte-sized characters now).  The answer is 32, so we need 
some way to map a number in the range 0-255 to a number in the range 
0-31.  Moreover, there must be exactly 8 numbers from the range 0-255 
that map to a number in the range 0-31.  We note that 2^3=8, and thus if 
we ignore the 3 least significant bits of each number in the range 0-255, 
than for every number, there will be 7 others that are the same as it 
(since for each permutation of the 5 most significant bits, there are 8 
possibilities for the 3 least significant bits).  Now all we need to do is 
ignore the 3 least significant digits, and at the same time produce a 
number in the range 0-31.  The quickest way to do this is just to right 
shift a number 3 places, making the 3 least significant digits "fall 
off," and also creating the right sized number. Let's recap just a bit.  
We have an array (let's call it C<bmp>) of 32 elements.  Given a character 
(say, C<ch>), to get to the right bitfield in C<bmp> we do this: bmp[ch >> 3].
Now we have to come up with an equation to get to the right bit in the 
bitfield for ch.  Let's examine just one bitfield for simplicity, all the 
rest will naturally be the same.  For this bitfield, there are 8 numbers 
we need to consider.  Due to our above equation, they each have the same 
5 most significant bits, but their 3 least significant bits are all 
uniqe.  So, all we do is extract the 3 least significant bits and create 
a power of 2 using them.  Thus, the code for the right bit given C<ch> is 
1 << (ch & 7). (Because 7 is 111 in binary.)  Now, to sum it all 
together, we are given a character (C<ch>) and an array (C<bmp>), how do we
"store" that this character is in the bitmap?  Here's the code:

	bmp[ch >> 3] = 1 << (ch & 7)

=back
