# Tables

## Hello, tables!

The tables are the fundamental elements of state-keeping in Triceps. Let's start with a basic example:

    my $hwunit = Triceps::Unit->new("hwunit") or confess "$!";
    my $rtCount = Triceps::RowType->new(
    	address => "string",
    	count => "int32",
    ) or confess "$!";

    my $ttCount = Triceps::TableType->new($rtCount)
    	->addSubIndex("byAddress",
    		Triceps::IndexType->newHashed(key => [ "address" ])
    	)
    or confess "$!";
    $ttCount->initialize() or confess "$!";

    my $tCount = $hwunit->makeTable($ttCount, &Triceps::EM_CALL, "tCount")
    	or confess "$!";

    while(&readLine) {
    	chomp;
    	my @data = split(/\W+/);

    	# the common part: find if there already is a count for this address
    	my $rhFound = $tCount->findBy(
    		address => $data[1]
    	);
    	my $cnt = 0;
    	if (!$rhFound->isNull()) {
    		$cnt = $rhFound->getRow()->get("count");
    	}

    	if ($data[0] =~ /^hello$/i) {
    		my $new = $rtCount->makeRowHash(
    			address => $data[1],
    			count => $cnt+1,
    		) or confess "$!";
    		$tCount->insert($new);
    	} elsif ($data[0] =~ /^count$/i) {
    		&send("Received '", $data[1], "' ", $cnt + 0, " times\n");
    	} else {
    		&send("Unknown command '$data[0]'\n");
    	}
    }

What happens here? The main loop reads the lines from standard input, splits into words and uses the first word as a command and the second word as a key. Note that it's not CSV format, it's words with the non-alphanumeric characters separating the words. Hello, table!, hello world, count world are examples of the valid inputs. For someting different, the commands are compared with their case ignored (but the case matters for the key).

The example counts, how many times each key has been

    hello

-ed, and prints this count back on the command

    count

. Here is a sample, with the input lines printed in bold: > Hello, table! > Hello, world! > Hello, table! > count world Received 'world' 1 times > Count table Received 'table' 2 times

In this example the table is read and modified using the direct procedural calls. As you can see, there isn't even any need for unit scheduling and such. There is a scheduler-based interface to the tables too, it will be shown soon. But in many cases the direct access is easier. Indeed, this particular example could have been implemented with the plain Perl hashes. Nothing wrong with that either. Well, the Perl tables provide many more intersting ways of indexing the data. But if you don't need them, they don't matter. And at some future point the tables will be supporting the on-disk persistence, but no reason to bother much about that now: things are likely to change a dozen times yet before that happens. Feel free to just use the Perl data structures if they make the code easier.

A table is created through a table type. This allows to stamp out duplicate tables of the same type, which can get handy when the multithreading will be added. A table is local to a thread. A table type can be shared between threads. To look up something in another thread's table, you'd either have to ask it through a request-reply protocol or to keep a local copy of the table. Such a copy can be easily done by creating a copy table from the same type.

In reality, right now all the business with table types separated from the tables is more pain than gain. It not only adds extra steps but also makes difficult to define a template that acts on a table by defining extra features on it. Something will be done about it, I have a few ideas.

The table type gets first created and configured, then initialized. After a table type is initialized, it can not be changed any more. That's the point of the initialization call: tell the type that all the configuration has been done, and it can go immutable now. Fundamentally, configuting a table type just makes it collect bits and pieces. Nothing but the most gross errors can be detected at that point. At initialization time everything comes together and everything gets checked for consistency. A table type must be fully initialized in one thread before it can be shared with other threads. The historic reason for this API is that it mirrors the &Cpp; API, which has turned out not to look that good in Perl. It's another candidate for a change.

A table type gets the row type and at least one index. Here it's a hashed index by the key field

    address

. "Hashed" means that you can look up the rows by the key value but there are no promises about any specific row order. And the hashing is used to make the key comparisons more efficient. The key of a hashed index may consist of multiple fields.

The table is then created from the table type, enqueueing mode (another hold-over, just always use

    EM_CALL

, this argument will be removed in the future), and given a name.

The rows can then be inserted into the table (and removed too, not shown in this example yet). The default behavior of the hashed index is to replace the old row if a new row with the same key is inserted.

The search in the table is done by the method

    findBy()

with the key fields of the index. Which returns a RowHandle object. A RowHandle is essentially an iterator in the table. Even if the row is not found, a RowHandle will be still returned but it will be NULL, which is checked for by

    $rh->isNull()

.

No matter which command will be used, it's always useful to look up the previous row for the key: its contents would be either printed or provide the previous value for the increase. So the model does it first and gets the count from it. If it's not found, then the count is set to 0.

Then it looks at the command and does what it's been told. Updating the count amounts to creating a new row with the new values and inserting it into the table. It replaces the previous one.

This is just the tip of the iceberg. The tables in Triceps have a lot more features.

table
table insert
table procedural API
index hashed
table replacement
table find
RowHandle

## Tables and labels

A table does not have to be operated in a procedural way. It can be plugged into the the scheduler machinery. Whenever a table is created, three labels are created with it.

- The input label is for sending the modification rowops to the table. The table provides the handler for it that applies the incoming rowops to the table.
- The output label propagates the modifications done to the table. It is a dummy label, and does nothing by itself. It's there for chaining the other labels to it. The output rowop comes quite handy to propagate the table's modifications to the rest of the state.
- The pre-modification label is also a dummy label, for chaining other labels to it. It sends the rowops right before they are applied to the table. This comes very handy for the elements that need to act depending on the previous state of the table, such as joins. The pre-modification label doesn't simply mirror the input label. The rows received on the input label may trigger the automatic changes to the table, such as an old row being deleted when a new row with the same key is inserted. All these modifications, be they automatic or explicit, will be reported to the pre-modification label. Since the pre-modification label is used relatively rarely, it contains a special optimization: if there is no label chained to it, no rowop will be sent to it in the first place. Don't be surprised if you enable the tracing and don't see it in the trace.

tracing

Again, the rowops coming through these labels aren't necessarily the same. If a DELETE rowop comes to the input label, referring to a row that is not in the table, it will not propagate anywhere. If an INSERT rowop comes in and causes another row to be replaced, the replaced row will be sent to the pre-modification and output labels as a DELETE rowop first.

Anf of course the table may be modified through the procedural interface. These modifications also produce rowops on the pre-modification and output labels.

The labels of the table have names. They are produced by adding suffixes to the table name. They are "tablename.in", "tablename.pre" and "tablename.out".

In the no bundling spirit, a rowop is sent to the pre-modification label right before it's applied to the table, and to the output label right after it's applied. If the labels executed from there need to read the table, they can, and will find the table in the exact state with no intervening modifications. However, they can't modify the table neither directly nor by calling its input label. When these labels are called, the table is in the middle of a modification and it can't accept another one. Such attempts are treated as recursive modifications, forbidden, and the program will die on them. If you need to modify the table, use

    schedule()

or

    loopAt()

to have the next modification done later. However there are no guarantees about other modifications getting done in between. When the looped rowop executes, it might need to check the state of the table again and decide if its operation still makes sense.

So, let's make a version of Hello, table example that passes the modification requests as rowops through the labels. It will print the information about the updates to the table as they happen, so there is no more use having a separate command for that. But for another demonstration let's add a command that would clear the counter of hellos. Here is its code:

    my $hwunit = Triceps::Unit->new("hwunit") or confess "$!";
    my $rtCount = Triceps::RowType->new(
    	address => "string",
    	count => "int32",
    ) or confess "$!";

    my $ttCount = Triceps::TableType->new($rtCount)
    	->addSubIndex("byAddress",
    		Triceps::IndexType->newHashed(key => [ "address" ])
    	)
    or confess "$!";
    $ttCount->initialize() or confess "$!";

    my $tCount = $hwunit->makeTable($ttCount, "EM_CALL", "tCount") or confess "$!";

    my $lbPrintCount = $hwunit->makeLabel($tCount->getRowType(),
    	"lbPrintCount", undef, sub { # (label, rowop)
    		my ($label, $rowop) = @_;
    		my $row = $rowop->getRow();
    		&send(&Triceps::opcodeString($rowop->getOpcode), " '",
    			$row->get("address"), "', count ", $row->get("count"), "\n");
    	} ) or confess "$!";
    $tCount->getOutputLabel()->chain($lbPrintCount) or confess "$!";

    # the updates will be sent here, for the tables to process
    my $lbTableInput = $tCount->getInputLabel();

    while(&readLine) {
    	chomp;
    	my @data = split(/\W+/);

    	# the common part: find if there already is a count for this address
    	my $rhFound = $tCount->findBy(
    		address => $data[1]
    	);
    	my $cnt = 0;
    	if (!$rhFound->isNull()) {
    		$cnt = $rhFound->getRow()->get("count");
    	}

    	if ($data[0] =~ /^hello$/i) {
    		$hwunit->makeHashSchedule($lbTableInput, "OP_INSERT",
    			address => $data[1],
    			count => $cnt+1,
    		);
    	} elsif ($data[0] =~ /^clear$/i) {
    		$hwunit->makeHashSchedule($lbTableInput, "OP_DELETE",
    			address => $data[1]
    		);
    	} else {
    		&send("Unknown command '$data[0]'\n");
    	}
    	$hwunit->drainFrame();
    }

The table creation is the same as last time. The only difference is that it uses

    "EM_CALL"

instead of

    &Triceps::EM_CALL

, both being equivalent. The row finding in the table is also the same.

The printing of the modifications to the table is done with

    $lbPrintCount

, which is connected to the table's output label. It prints the opcode, the address of the greeting, and the count of greetings. It will show us what is happening to the table as soon as it happens. An unit trace could be used instead but a custom printout contains less noise. The pre-modification label is of no interest here, so it's not used.

The references to the labels of a table are gotten with:

    $label = $table->getInputLabel();
    $label = $table->getPreLabel();
    $label = $table->getOutputLabel();

The deletion does not require an exact row to be sent in. All it needs is a row with the keys for deletion, the rest of the fields in it are ignored. So the clear command puts only the key field in it.

Here is an example of input (in bold) and output:

> Hello, table! OP_INSERT 'table', count 1 > Hello, world! OP_INSERT 'world', count 1 > Hello, table! OP_DELETE 'table', count 1 OP_INSERT 'table', count 2 > clear, table OP_DELETE 'table', count 2 > Hello, table! OP_INSERT 'table', count 1

An interesting thing happens after the second Hello, table!: the code send only an

    OP_INSERT

but the output shows an

    OP_DELETE

and

    OP_INSERT

. The

    OP_DELETE

for the old row gets automatically generated when a row with repeated key is inserted. Now, depending on what you want, just sending in the first place the consequent inserts of rows with the same keys, and relying on the table's internal consistency to turn them into updates, might be a good thing or not. Overall it's a dirty way to write but sometimes it comes convenient. The clean way is to send the explicit deletes first. When the data goes through the table, it gets automatically cleaned. The subscribers to the table's output and pre-modification labels get the clean and consistent picture: a row never gets simply replaced, they always see an

    OP_DELETE

first and only then an

    OP_INSERT

.
label
label table
table label API
table replacement
scheduling recursion
recursion
schedule loop
bundling

## Basic iteration through the table

Let's add a dump of the table contents to the "Hello, table" example, either version of it. For that, the code needs to go through every record in the table:

    	elsif ($data[0] =~ /^dump$/i) {
    		for (my $rhi = $tCount->begin(); !$rhi->isNull(); $rhi = $rhi->next()) {
    			&send($rhi->getRow->printP(), "\n");
    		}
    	}

As you can see, the row handle works kind of like an STL iterator. Only the end of iteration is detected by receiving a NULL row handle. Calling

    next()

on a NULL row handle is OK but it would just return another NULL handle. And there is no decrementing the iterator, you can only go forward with

    next()

. The backwards iteration is in the plans but not implemented yet.

An example of this fragment's output would be:

> Hello, table! > Hello, world! > Hello, table! > count world Received 'world' 1 times > Count table Received 'table' 2 times > dump address="world" count="1" address="table" count="2"

The order of the rows in the printout is the same as the order of rows in the table's index. Which is no particular order, since it's a hashed index. As long as you stay with the same 64-bit AMD64 architecture (with LSB-first byte order), it will stay the same on consecutive runs. But switching to a 32-bit machine or to an MSB-first byte order (such as a SPARC, if you can still find one) will change the hash calculation, and with it the resulting row order. There are the ordered indexes as well, they will be described later.

table iteration

## Deleting a row

Deleting a row from a table through the input label is simple: send a rowop with

    OP_DELETE

, it will find the row with the matching key and delete it, as was shown above. In the procedural way the same can be done with the method

    deleteRow()

. The added row deletion code for the main loop of Hello, table (either version, but particularly relevant for the one from &xrsp;) is:

    	elsif ($data[0] =~ /^delete$/i) {
    		my $res = $tCount->deleteRow($rtCount->makeRowHash(
    			address => $data[1],
    		));
    		&send("Address '", $data[1], "' is not found\n") unless $res;
    	}

The result allows to differentiate between the situations when the row was found and deleted and the row was not found. On any error the call confesses. The

    insert()

method also follows this new convention and confesses on errors.

However we already find the row handle in advance in

    $rhFound

. For this case a more efficient form is available, and it can be added to the example as:

    		elsif ($data[0] =~ /^remove$/i) {
    			if (!$rhFound->isNull()) {
    				$tCount->remove($rhFound);
    			} else {
    				&send("Address '", $data[1], "' is not found\n");
    			}
    		}

It removes a specific row handle from the table. In whichever way you find it, you can remove it. An attempt to remove a NULL handle would be an error (and this method also confesses on errors).

The reason why

    remove()

is more efficient than

    deleteRow()

is that

    deleteRow()

amounts to finding the row handle by key and then removing it. And the

    OP_DELETE

rowop sent to the input label calls

    deleteRow()

.

    deleteRow()

never deletes more than one row, even if multiple rows match (yes, the indexes don't have to be unique). There isn't any method to delete multiple rows at once. Every row has to be deleted by itself. As an example, here is the implementation of the command clear for Hello, table that clears all the table contents by iterating through it:

    	elsif ($data[0] =~ /^clear$/i) {
    		my $rhi = $tCount->begin();
    		while (!$rhi->isNull()) {
    			my $rhnext = $rhi->next();
    			$tCount->remove($rhi);
    			$rhi = $rhnext;
    		}
    	}

After a handle is removed from the table, it continues to exist, as long as there are references to it. It could even be inserted back into the table. However until (and unless) it's inserted back, it can not be used for iteration any more. Calling

    next()

on a handle that is not in the table would just return a NULL handle. So the next row has to be found before removing the current one.
error handling
table procedural API
table remove row
table iteration

## A closer look at the RowHandles

A few uses of the RowHandles have been shown by now. So, what is a RowHandle? As Captain Obvious would say, RowHandle is a class (or package, in Perl terms) implementing a row handle.

A row handle keeps a table's service information (including the index data) for a single data row, including of course a reference to the row itself. Each row is stored in the table through its handle. The row handle is also an iterator in the table, and a special one: it's an iterator for all the table's indexes at once. For you SQLy people, an iterator is essentially a cursor on an index. For you Java people, an iterator can be used to do more than step sequentially through rows. So far only the table types with one index have been shown, but in reality multiple indexes are supported, potentially with quite complicated arrangements. More on the indexes later, for now just keep it in mind. A row handle can be found through one index and then used to iterate through another one. Or you can iterate through one index, find a certain row handle and continue iterating through another index starting from that handle. If you remember a reference on a particular row handle, you can always continue iteration from that point later. (unless the row handle gets removed from the table).

A RowHandle always belongs to a particular table, the RowHandles can not be shared nor moved between two tables, even if the tables are of the same type. Since the tables are single-threaded, obviously the RowHandles may not be shared between the threads either.

However a RowHandle may exist without being inserted into a table. In this case it still has a spiritual connection to that table but is not included in the index (the iteration attempts with it would just return end of the index), and will be destroyed as soon as all the references to it disappear.

The insertion of a row into a table actually happens in two steps:

1. A RowHandle is created for a row.
2. This new handle is inserted into the table.

This is done with the following code:

    $rh = $table->makeRowHandle($row) or confess "$!";
    $table->insert($rh);

Only it just so happens that to make life easier, the method

    insert()

has been made to accept either a row handle or directly a row. If it finds a row, it makes a handle for it behind the curtains and then proceeds with the insertion of that handle. Passing a row directly is also more efficient (if you don't have a handle already created for it for some other reason) because the row handle creation then happens entirely in the &Cpp; code, without surfacing into Perl.

A handle can be created for any row of a type matching the table's row type. For a while it was accepting only equal types but that was not consistent with what the labels are doing, so I've changed it.

The method

    insert()

has a return value. It's often ignored but occasionally comes handy. 1 means that the row has been inserted successfully, and 0 means that the row has been rejected. On errors it confesses. An attempt to insert a NULL handle or a handle that is already in the table will cause a rejection, not an error. Also the table's index may reject a row with duplicate key (though right now this option is not implemented, and the hash index silently replaces the old row with the new one).

There is a method to find out if a row handle is in the table or not:

    $result = $rh->isInTable();

Though it's used mostly for debugging, when some strange things start going on.

The searching for rows in the table by key has been previously shown with the method

    findBy()

. Which happens to be a wrapper over a more general method

    find()

: it constructs a row from its argument fields and then calls

    find()

with that row as a sample of data to find. The method

    find()

is similar to

    insert()

in the handling of its arguments: the proper way is to give it a row handle argument, but the more efficient way is to give it a row argument, and it will create the handle for it as needed before performing a search.

Now you might wonder: huh,

    find()

takes a row handle and returns a row handle? What's the point? Why not just use the first row handle? Well, those are different handles:

- The argument handle is normally not in the table. It's created brand new from a row that contains the keys that you want to find, just for the purpose of searching.
- The returned handle is always in the table (of course, unless it's NULL). It can be further used to extract back the row data, and/or for iteration.

Though nothing really prevents you from searching for a handle that is already in the table. You'll just get back the same handle, after gratuitously spending some CPU time. (There are exceptions to this, with the more complex indexes that will be described later).

Why do you need to create new a row handle just for the search? Due to the internal mechanics of the implementation. A handle stores the helper information for the index. For example, the hash index calculates the hash value of all the row's key fields once and stores it in the row handle. Despite it being called a hash index, it really stores the data in a tree, with the hash value used to speed up the comparisons for the tree order. It's much easier to make both the

    insert()

and

    find()

work with the hash value and row reference stored in the same way in a handle than to implement them differently. Because of this,

    find()

uses the exactly same row handle argument format as

    insert()

.

Can you create multiple row handles referring to the same row? Sure, knock yourself out. From the table's perspective it's the same thing as multiple row handles for multiple copies of the row with the same values in them, only using less memory.

There is more to the row handles than has been touched upon yet. It will all be revealed when more of the table features are described. The internal structure of the row handles will be described in the &xrsp;.

RowHandle
table insert
error handling
table replacement
table find

## A window is a FIFO

A fairly typical situation in the CEP world is when a model needs to keep a limited history of events. For a simple example, let's discuss, how to remember the last two trades per stock symbol. The size of two has been chosen to keep the sample input and outputs small.

This is normally called a window logic, with a sliding window. You can think of it in a mechanical analogy: as the trades become available, they get printed on a long tape. However the tape is covered with a masking plate. The plate has a window cut in it that lets you see only the last two trades.

Some CEP systems have the special data structures that implement this logic, that are called windows. Triceps has a feature on a table instead that makes a table work as a window. It's not unique in this department: for example Coral8 does the opposite, calls everything a window, even if some windows are really tables in every regard but name.

Here is a Triceps example of keeping the window for the last two trades and iteration over it:

    our $uTrades = Triceps::Unit->new("uTrades") or confess "$!";
    our $rtTrade = Triceps::RowType->new(
    	id => "int32", # trade unique id
    	symbol => "string", # symbol traded
    	price => "float64",
    	size => "float64", # number of shares traded
    ) or confess "$!";

    our $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    			->addSubIndex("last2",
    				Triceps::IndexType->newFifo(limit => 2)
    			)
    	)
    	or confess "$!";
    $ttWindow->initialize() or confess "$!";
    our $tWindow = $uTrades->makeTable($ttWindow,
    	&Triceps::EM_CALL, "tWindow") or confess "$!";

    # remember the index type by symbol, for searching on it
    our $itSymbol = $ttWindow->findSubIndex("bySymbol") or confess "$!";
    # remember the FIFO index, for finding the start of the group
    our $itLast2 = $itSymbol->findSubIndex("last2") or confess "$!";

    # print out the changes to the table as they happen
    our $lbWindowPrint = $uTrades->makeLabel($rtTrade, "lbWindowPrint",
    	undef, sub { # (label, rowop)
    		&send($_[1]->printP(), "\n"); # print the change
    	}) or confess "$!";
    $tWindow->getOutputLabel()->chain($lbWindowPrint) or confess "$!";

    while(&readLine) {
    	chomp;
    	my $rTrade = $rtTrade->makeRowArray(split(/,/)) or confess "$!";
    	my $rhTrade = $tWindow->makeRowHandle($rTrade) or confess "$!";
    	$tWindow->insert($rhTrade);
    	# There are two ways to find the first record for this
    	# symbol. Use one way for the symbol AAA and the other for the rest.
    	my $rhFirst;
    	if ($rTrade->get("symbol") eq "AAA") {
    		$rhFirst = $tWindow->findIdx($itSymbol, $rTrade);
    	} else  {
    		# $rhTrade is now in the table but it's the last record
    		$rhFirst = $rhTrade->firstOfGroupIdx($itLast2) or confess "$!";
    	}
    	my $rhEnd = $rhFirst->nextGroupIdx($itLast2) or confess "$!";
    	&send("New contents:\n");
    	for (my $rhi = $rhFirst;
    			!$rhi->same($rhEnd); $rhi = $rhi->nextIdx($itLast2)) {
    		&send("  ", $rhi->getRow()->printP(), "\n");
    	}
    }

This example reads the trade records in CSV format, inserts them into the table, and then prints the actual modifications reported by the table and the new state of the window for this symbol. And here is a sample log, with the input lines in bold:

> 1,AAA,10,10 tWindow.out OP_INSERT id="1" symbol="AAA" price="10" size="10" New contents: id="1" symbol="AAA" price="10" size="10" > 2,BBB,100,100 tWindow.out OP_INSERT id="2" symbol="BBB" price="100" size="100" New contents: id="2" symbol="BBB" price="100" size="100" > 3,AAA,20,20 tWindow.out OP_INSERT id="3" symbol="AAA" price="20" size="20" New contents: id="1" symbol="AAA" price="10" size="10" id="3" symbol="AAA" price="20" size="20" > 4,BBB,200,200 tWindow.out OP_INSERT id="4" symbol="BBB" price="200" size="200" New contents: id="2" symbol="BBB" price="100" size="100" id="4" symbol="BBB" price="200" size="200" > 5,AAA,30,30 tWindow.out OP_DELETE id="1" symbol="AAA" price="10" size="10" tWindow.out OP_INSERT id="5" symbol="AAA" price="30" size="30" New contents: id="3" symbol="AAA" price="20" size="20" id="5" symbol="AAA" price="30" size="30" > 6,BBB,300,300 tWindow.out OP_DELETE id="2" symbol="BBB" price="100" size="100" tWindow.out OP_INSERT id="6" symbol="BBB" price="300" size="300" New contents: id="4" symbol="BBB" price="200" size="200" id="6" symbol="BBB" price="300" size="300"

You can see that the window logic works: at no time is there more than two rows in each group. As more rows are inserted, the oldest rows get deleted.

Now let's dig into the code. The first thing to notice is that the table type has two indexes (strictly speaking, index types, but most of the time they can be called indexes without creating a confusion) in it. Unlike your typical database, the indexes in this example are nested.

    TableType
    +-IndexType Hash "bySymbol"
      +-IndexType Fifo "last2"

If you follow the nesting, you can see, that the first call

    addSubIndex()

adds an index type to the table type, while the textually second

    addSubIndex()

adds an index to the previous index.

The same can also be written out in multiple separate calls, with the intermediate results stored in the variables:

    $itLast2 = Triceps::IndexType->newFifo(limit => 2);
    $itSymbol = Triceps::IndexType->newHashed(key => [ "symbol" ]);
    $itSymbol->addSubIndex("last2", $itLast2);
    $ttWindow = Triceps::TableType->new($rtTrade);
    $ttWindow->addSubIndex("bySymbol", $itSymbol);

I'm not perfectly happy with the way the table types are constructed with the index types right now, since the parenthesis levels have turned out a bit hard to track. This is another example of following the &Cpp; API in Perl that didn't work out too well, and it will change in the future. But for now please bear with it.

The index nesting is kind of intuitively clear, but the details may take some time to get your head wrapped around them. You can think of it as the inner index type creating the miniature tables that hold the rows, and then the outer index holding not individual rows but those miniature tables. So, to find the rows in the table you go through two levels of indexes: first through the outer index, and then through the inner one. The table takes care of these details and makes them transparent, unless you want to stop your search at an intermediate level: such as, to find all the transactions with a given symbol, you need to do a search in the outer index, but then from that point iterate through all rows in the found inner index. For this you obviously have to tell the table, where do you want to stop in the search.

The outer index is the hash index that we've seen before, the inner index is a FIFO index. A FIFO index doesn't have any key, it just keeps the rows in the order they were inserted. You can search in a FIFO index but most of the time it's not the best idea: since it has no keys, it searches linearly through all its rows until it finds an exact match (or runs out of rows). It's a reasonable last-resort way but it's not fast and in many cases not what you want. This also sends a few ripples through the row deletion. Remember that the method

    deleteRow()

and sending the

    OP_DELETE

to the table's input label invoke

    find()

, which would cause the linear search on the FIFO indexes. So when you use a FIFO index, it's usually better to find the row handle you want to delete in some other way and then call

    remove()

on it, or use another approach that will be shown later. Or just keep inserting the rows and never delete them, like this example does.

A FIFO index may contain multiple copies of an exact same row. It doesn't care, it just keeps whatever rows were given to it in whatever order they were given.

By default a FIFO index just keeps whatever rows come to it. However it may have a few options. Setting the option

    limit

limits the number of rows stored in the index (not per the whole table but per one of those miniature tables). When you try to insert one row too many, the oldest row gets thrown out, and the limit stays unbroken. That's what creates the window behavior: keep the most recent N rows.

If you look at the sample output, you can see that inserting the rows with ids 1-4 generates only the insert events on the table. But the rows 5 and 6 start overflowing their FIFO indexes, and cause the oldest row to be automatically deleted before completing the insert of the new one.

A FIFO index doesn't have to be nested inside a hash index. If you put a FIFO index at the top level, it will control the whole table. So it would be not two last record per key but two last records inserted in the whole table.

Continuing with the example, the table gets created, and then the index types get extracted back from the table type. Now, why not just write out the table type creation with intermediate variables as shown above and remember the index references? At some point in the past this actually would have worked but not any more. It has to do with the way the table type and its index types are connected. It's occasionally convenient to create one index type and then reuse it in multiple table types. However for the whole thing to work, the index type must be tied to its particular table type. This tying together happens when the table type is initialized. If you put the same index type into two table types, then when the first table type is initialized, the index type will get tied to it. The second table type would then fail to initialize because an index in it is already tied elsewhere. To get around this dilemma, now when you call

    addSubIndex()

, it doesn't connect the original index type, instead it makes a copy of it. That copy then gets tied with the table type and later gets returned back with

    findSubIndex()

.

The table methods that take an index type argument absolutely require that the index type must be tied to that table's type. If you try to pass a seemingly the same index type that has not been tied, or has been tied to a different table type, that is an error.

One last note on this subject: there is no interdependency between the methods

    makeTable()

and

    findSubIndex()

, they can be done in either order.

The example output comes from two sources. The running updates on the table's modifications (the lines with

    OP_INSERT

and

    OP_DELETE

) are printed from the label

    $lbWindowPrint

. The new window contents is printed from the main loop.

The main loop reads the trade records in the simple CSV format without the opcode, and for simplicity inserts directly into the table with the procedural API, bypassing the scheduler. After the row is inserted, the contents of its index group (that miniature table) gets printed. The insertion could as well have been done with passing directly the row reference, without explicitly creating a handle. But that handle will be used to demonstrate an interesting point.

To print the contents of an index group, we need to find its boundaries. In Triceps these boundaries are expressed as the first row handle of the group, and as the row handle right after the group. There is an internal logic to that, and it will be explained later, but for now just take it on faith.

With the information we have, there are two ways to find the first row of the group:

- With the table's method

      findIdx()

  . It's very much like

      find()

  , only it has an extra argument of a specific index type. If the index type given has no further nesting in it,

      findIdx()

  works exactly like

      find()

  . In fact,

      find()

  is exactly such a special case of

      findIdx()

  with an automatically chosen index type. If you use an index type with further nesting under it,

      findIdx()

  will return the handle of the first row in the group under it (or the usual NULL row handle if not found).
- If we create the row handle explicitly before inserting it into the table, as was done in the example, that will be the exact row handle inserted into the table. Not a copy or anything but this particular row handle. After a row handle gets inserted into the table, it knows its position in the indexes. It knows, in which group it is. And we still have a reference to it. So then we can use this knowledge to navigate within the group, jump to the first row handle in the group with

      firstOfGroupIdx()

  . It also takes an index type but in this case it's the type that controls the group, the FIFO index in out case.

RowHandle

The example shows both ways. As a demonstration, it uses the first way if the symbol is AAA and the second way for all the other symbols.

The end boundary is found by calling

    nextGroupIdx()

on the first row's handle. The handle of the newly inserted row could have also been used for

    nextGroupIdx()

, or any other handle in the group. For any handle belonging to the same group, the result is exactly the same.

And finally, after the iteration boundaries have been found, the iteration on the group can run. The end condition comparison is done with

    same()

, to compare the row handle references and not just their Perl-level wrappers. The stepping is done with

    nextIdx()

, with is exactly like

    next()

but according to a particular index, the FIFO one. This has actually been done purely to show off this method. In this particular case the result produced by

    next()

,

    nextIdx()

on the FIFO index type and

    nextIdx()

on the outer hash index type is exactly the same. We'll come to the reasons of that yet.

Checking the results of find and iteration methods by

    or confess

would be quite inconvenient, so these methods have been already made to confess on errors.

Looking forward, as you iterate through the group, you could do some manual aggregation along the way. For example, find the average price of the last two trades, and then do something useful with it.

There is also a piece of information that you can find without iteration: the size of the group.

    $size = $table->groupSizeIdx($idxType, $row_or_rh);

This information is important for the joins, and iterating every time through the group is inefficient if all you want to get is the group size. Since when you need this data you usually have the row and not the row handle, this operation accepts either and implicitly performs a

    findIdx()

on the row to find the row handle. Moreover, even if it receives the argument of a row handle that is not in the table, it will also automatically perform a

    findIdx()

on it (though calling it for a row handle in the table is more efficient because the group would not need to be looked up first).

If there is no such group in the table, the result will be 0.

The

    $idxType

argument is the non-leaf parent index of the group. (Using a leaf index type is not an error but it always returns 0, because there are no groups under it). It's basically the same index type as you would use in

    findIdx()

to find the first row of the group or in

    firstOfGroupIdx()

or

    nextGroupIdx()

to find the boundaries of thr group. Remember, a non-leaf index type defines the groups, and the nested index types under it define the order in those groups (and possibly further break them down into sub-groups).

It's a bit confusing, so let's recap with another example. If you have a table type defined as:

    our $ttPosition = Triceps::TableType->new($rtPosition)
    	->addSubIndex("primary",
    		Triceps::IndexType->newHashed(key => [ "date", "customer", "symbol" ])
    	)
    	->addSubIndex("currencyLookup", # for joining with currency conversion
    		Triceps::IndexType->newHashed(key => [ "date", "currency" ])
    		->addSubIndex("grouping", Triceps::IndexType->newFifo())
    	)
    	->addSubIndex("byDate", # for cleaning by date
    		Triceps::SimpleOrderedIndex->new(date => "ASC")
    		->addSubIndex("grouping", Triceps::IndexType->newFifo())
    	)
    or confess "$!";

then it would make sense to call

    groupSizeIdx()

,

    firstOfGroupIdx()

and

    nextGroupIdx()

with the indexes currencyLookup or byDate but not with primary, currencyLookup/grouping nor byDate/grouping. You can call

    findIdx()

with any index, but for currencyLookup or byDate it would return the first row of the group while for primary, currencyLookup/grouping or byDate/grouping it would return the only matching row. On the other hand, for iteration in a group, it makes sense to call

    nextIdx()

only on primary, currencyLookup/grouping or byDate/grouping. Calling

    nextIdx()

on the non-leaf index types is not an error but it would in effect resolve to the same thing as using their first leaf sub-indexes.
window
FIFO
Coral8
index group
table replacement
type table
type index
index FIFO
index initialization
type index
table iteration
index group
table find
index find
table iteration
error handling
aggregation
index group size

## Secondary indexes

The last example dealt only with the row inserts, because it could not handle the deletions that well. What if the trades may get cancelled and have to be removed from the table? There is a solution to this problem: add one more index. Only this time not nested but in parallel. The indexes in the table type become tree-formed:

    TableType
    +-IndexType Hash "byId" (id)
    +-IndexType Hash "bySymbol" (symbol)
      +-IndexType Fifo "last2"

It's very much like the common relational databases where you can define multiple indexes on the same table. Both indexes

    byId

and

    bySymbol

(together with its nested sub-index) refer to the same set of rows stored in the table. Only

    byId

allows to easily find the records by the unique id, while

    bySymbol

is responsible for keeping then grouped by the symbol, in FIFO order. It could be said that

    byId

is the primary index (since it has a unique key) and

    bySymbol

is a secondary one (since it does the grouping) but from the Triceps'es standpoint they are pretty much equal and parallel to each other.

To illustrate the point, here is a modified version of the previous example. Not only does it manage the deletes but also computes the average price of the collected transactions as it iterates through the group, thus performing a manual aggregation.

    our $uTrades = Triceps::Unit->new("uTrades") or confess "$!";
    our $rtTrade = Triceps::RowType->new(
    	id => "int32", # trade unique id
    	symbol => "string", # symbol traded
    	price => "float64",
    	size => "float64", # number of shares traded
    ) or confess "$!";

    our $ttWindow = Triceps::TableType->new($rtTrade)
    	->addSubIndex("byId",
    		Triceps::IndexType->newHashed(key => [ "id" ])
    	)
    	->addSubIndex("bySymbol",
    		Triceps::IndexType->newHashed(key => [ "symbol" ])
    			->addSubIndex("last2",
    				Triceps::IndexType->newFifo(limit => 2)
    			)
    	)
    	or confess "$!";
    $ttWindow->initialize() or confess "$!";
    our $tWindow = $uTrades->makeTable($ttWindow,
    	&Triceps::EM_CALL, "tWindow") or confess "$!";

    # remember the index type by symbol, for searching on it
    our $itSymbol = $ttWindow->findSubIndex("bySymbol") or confess "$!";
    # remember the FIFO index, for finding the start of the group
    our $itLast2 = $itSymbol->findSubIndex("last2") or confess "$!";

    # remember, which was the last row modified
    our $rLastMod;
    our $lbRememberLastMod = $uTrades->makeLabel($rtTrade, "lbRememberLastMod",
    	undef, sub { # (label, rowop)
    		$rLastMod = $_[1]->getRow();
    	}) or confess "$!";
    $tWindow->getOutputLabel()->chain($lbRememberLastMod) or confess "$!";

    # Print the average price of the symbol in the last modified row
    sub printAverage # (row)
    {
    	return unless defined $rLastMod;
    	my $rhFirst = $tWindow->findIdx($itSymbol, $rLastMod);
    	my $rhEnd = $rhFirst->nextGroupIdx($itLast2) or confess "$!";
    	&send("Contents:\n");
    	my $avg;
    	my ($sum, $count);
    	for (my $rhi = $rhFirst;
    			!$rhi->same($rhEnd); $rhi = $rhi->nextIdx($itLast2)) {
    		&send("  ", $rhi->getRow()->printP(), "\n");
    		$count++;
    		$sum += $rhi->getRow()->get("price");
    	}
    	if ($count) {
    		$avg = $sum/$count;
    	}
    	&send("Average price: ", (defined $avg? $avg: "Undefined"), "\n");
    }

    while(&readLine) {
    	chomp;
    	my @data = split(/,/);
    	my $op = shift @data; # string opcode, if incorrect then will die later
    	my $rTrade = $rtTrade->makeRowArray(@data) or confess "$!";
    	my $rowop = $tWindow->getInputLabel()->makeRowop($op, $rTrade)
    		or confess "$!";
    	$uTrades->call($rowop) or confess "$!";
    	&printAverage();
    	undef $rLastMod; # clear for the next iteration
    	$uTrades->drainFrame(); # just in case, for completeness
    }

And an example of its work, with the input lines shown in bold:

> OP_INSERT,1,AAA,10,10 Contents: id="1" symbol="AAA" price="10" size="10" Average price: 10 > OP_INSERT,2,BBB,100,100 Contents: id="2" symbol="BBB" price="100" size="100" Average price: 100 > OP_INSERT,3,AAA,20,20 Contents: id="1" symbol="AAA" price="10" size="10" id="3" symbol="AAA" price="20" size="20" Average price: 15 > OP_INSERT,4,BBB,200,200 Contents: id="2" symbol="BBB" price="100" size="100" id="4" symbol="BBB" price="200" size="200" Average price: 150 > OP_INSERT,5,AAA,30,30 Contents: id="3" symbol="AAA" price="20" size="20" id="5" symbol="AAA" price="30" size="30" Average price: 25 > OP_INSERT,6,BBB,300,300 Contents: id="4" symbol="BBB" price="200" size="200" id="6" symbol="BBB" price="300" size="300" Average price: 250 > OP_DELETE,3 Contents: id="5" symbol="AAA" price="30" size="30" Average price: 30 > OP_DELETE,5 Contents: Average price: Undefined

The input has changed: now an extra column is prepended to it, containing the opcode for the row. The updates to the table are not printed any more, but the calculated average price is printed after the new contents of the group.

In the code, the first obvious addition is the extra index in the table type. The label that used to print the updates is gone, and replaced with another one, that remembers the last modified row in a global variable.

That last modified row is then used in the function

    printAverage()

to find the group for iteration. Why? Could not we just remember the symbol from the input data? Not always. As you can see from the last two input rows with

    OP_DELETE

, the trade id is the only field required to find and delete a row using the index

    byId

. So these trade cancellation rows take a shortcut and only provide the trade id, not the rest of the fields. If we try to remember the symbol fields from them, we'd remember an

    undef

. Can we just look up the row by id after the incoming rowop has been processed? Not after the deletion. If we try to find the symbol by looking up the row after the deletion, we will find nothing, because the row will already be deleted. We could look up the row in the table before the deletion, and remember it, and afterwards do the look-up of the group by it. But since on deletion the row with will come to the table's output label anyway, we can just ride the wave and remember it instead of doing the manual look-up. And this also spares the need of creating a row with the last symbol for searching: we get a ready pre-made row with the right symbol in it.

Note that in this example, unlike the previous one, there are no two ways of finding the group any more: after deletion the row handle will not be in the table any more, and could not be used to jump directly to the beginning of its group.

    findIdx()

has to be used to find the group.

By the time

    printAverage()

executes, it could happen that all the rows with that symbol will be gone, and the group will disappear. This situation is handled nicely in an automatic way:

    findIdx()

will return a NULL row handle, for which then

    nextGroupIdx()

will also return a NULL row handle. The for-loop will immediately satisfy the condition of

    $rhi->same($rhEnd)

, it will make no iterations, the

    $count

and

    $avg

will be left undefined. In result no rows will be printed and the average value will be printed as Undefined, as you can see in the reaction to the last input row in the sample output.

The main loop becomes reduced to reading the input, splitting the line, separating the opcode, calling the table's input label, and printing the average. The auto-conversion from the opcode name is used when constructing the rowop. Normally it's not a good practice, since the program will die if it finds a bad rowop in the input, but good enough for a small example. The direct use of

    $uTrades->call()

guarantees that by the time it returns, the last modified row will be remembered in

    $rLastMod

, available for

    printAverage()

to use.

After the average is calculated,

    $rLastMod

is reset to prevent it from accidentally affecting the next row. If the next row is an attempt to delete a trade id that is not in the table any more, the DELTE operation will have no effect on the table, and nothing will be sent from the table's output label.

    $rLastMod

will stay undefined, and

    printAverage()

will check it and immediately return. An attempt to pass an

    undef

argument to

    findIdx()

would be an error.

The final

    $uTrades->drainFrame()

is there purely for completeness. In this case we know that nothing will be scheduled by the labels downstream from the table, and there will be nothing to drain.

Now, an interesting question is: how does the table know, that to delete a row, it has to find it using the field

    id

? Or, since the deletion internally uses

    find()

, the more precise question is: how does

    find()

know that it has to use the index

    byId

? It doesn't use any magic. It simply goes by the first index defined in the table. That's why the index

    byId

has been very carefully placed before

    bySymbol

. The same principle applies to all the other functions like

    next()

, that use an index but don't receive one as an argument: the first index is always the default index. There is a bit more detail to it, but that's the rough principle.
index hashed
index secondary
index tree
aggregation manual
table find
index default
index order
index primary

## Sorted index

The hashed index provides a way to store rows indexed by a key. It is fast but it has a price to pay for that speed: when iterating through it, the records come in an unpredictable (though repeatable, within a particular machine architecture) order determined by the hash function. If the order doesn't matter, that's fine. But often the order does matter, and is desirable even at the tradeoff of the reduced performance.

The sorted index provides a solution for this problem. It is created with:

    $it = Triceps::IndexType->newPerlSorted($sortName,
      \&initFunc, \&compareFunc, @args);

The Perl in newPerlSorted refers to the fact that the sorting order is specified as a Perl comparison function.

    $sortName

is just a symbolic name for printouts. It's used when you call

    $it->print()

(directly or as a recursive call from the table type print) to let you know what kind of index type it is, since it can't print the compiled comparison function. It is also used in the error messages if something dies inside the comparison function: the comparison is executed from deep inside the &Cpp; code, and by that time the

    $sortName

is the only way to identify the source of the problems. It's not the same name as used to connect the index type into the table type hierarchy with

    addSubIndex()

. As usual, an index type may be reused in multiple hierarchies, with different names, but in all cases it will also keep the same

    $sortName

. This may be easier to show with an example:

    $rt1 = Triceps::RowType->new(
    	a => "int32",
    	b => "string",
    ) or confess "$!";

    $it1 = Triceps::IndexType->newPerlSorted("basic", undef, \&compBasic)
    	or confess "$!";

    $tt1 = Triceps::TableType->new($rt1)
    	->addSubIndex("primary", $it1)
    	or confess "$!";

    $tt2 = Triceps::TableType->new($rt1)
    	->addSubIndex("first", $it1)
    	or confess "$!";

    print $tt1->print(), "\n";
    print $tt2->print(), "\n";

The print calls in it will produce:

    table (
      row {
        int32 a,
        string b,
      }
    ) {
      index PerlSortedIndex(basic) primary,
    }
    table (
      row {
        int32 a,
        string b,
      }
    ) {
      index PerlSortedIndex(basic) first,
    }

Both the name of the index type in the table type and the name of the sorted index type are printed, but in different spots.

The

    initFunc

and/or

    compareFunc

references specify the sorting order. One of them may be left undefined but not both.

    @args

are the optional arguments that will be passed to both functions.

The easiest but least flexible way is to just use the

    compareFunc

. It gets two Rows (not RowHandles!) as arguments, plus whatever is specified in

    @args

. It returns the usual Perl-style <=> result. For example:

    sub compBasic # ($row1, $row2)
    {
    	return $_[0]->get("a") <=> $_[1]->get("a");
    }

Don't forget to use <=> for the numbers and cmp for the strings. The typical Perl idiom for sorting by more than one field is to connect them by ||.

Or, if we want to specify the field names as arguments, we could define a sort function that sorts first by a numeric field in ascending order, then by a string field in descending order:

    sub compAscDesc # ($row1, $row2, $numFldAsc, $strFldDesc)
    {
    	my ($row1, $row2, $numf, $strf) = @_;
    	return $row1->get($numf) <=> $row2->get($numf)
    		|| $row2->get($strf) cmp $row1->get($strf); # backwards for descending
    }

    my $sit = Triceps::IndexType->newPerlSorted("by_a_b", undef,
    	\&compAscDesc, "a", "b") or confess "$!";

This assumes that the row type will have a numeric field a and a string field b. The problem is that if it doesn't then this will not be discovered until you create a table and try to insert some rows into it, which will finally call the comparison function. Even then it won't be exactly obvious because this comparison function never checks

    "$!"

after

    get()

, and you'll see no failures but all the rows will be considered equal and will replace each other.

You could check that the arguments match the row type (

    $row1->getType()

) in the comparison function but that would add extra overhead, and the Perl comparisons are slow enough as they are.

The

    initFunc

provides a way to do that check and more. It is called at the table type initialization time. By this time all this extra information is known, and it gets the references to the table type, index type (itself, but with the class stripped back to Triceps::IndexType), row type, and whatever extra arguments that were passed. It can do all the checks once.

The init function's return value is kind of backwards to everything else: on success it returns

    undef

, on error it returns the error message. It could die too, but simply returning an error message is somewhat nicer. The returned error messages may contain multiple lines separated by \n, so it should try to collect all the error information it can.

The init function that would check the arguments for the last example can be defined as:

    sub initNumStr # ($tabt, $idxt, $rowt, @args)
    {
    	my ($tabt, $idxt, $rowt, @args) = @_;
    	my %def = $rowt->getdef(); # the field definition
    	my $errors; # collect as many errors as possible
    	my $t;

    	if ($#args != 1) {
    		$errors .= "Received " . ($#args + 1) . " arguments, must be 2.\n"
    	} else {
    		$t = $def{$args[0]};
    		if ($t !~ /int32$|int64$|float64$/) {
    			$errors .= "Field '" . $args[0] . "' is not of numeric type.\n"
    		}
    		$t = $def{$args[1]};
    		if ($t !~ /string$|uint8/) {
    			$errors .= "Field '" . $args[1] . "' is not of string type.\n"
    		}
    	}

    	if (defined $errors) {
    		# help with diagnostics, append the row type to the error listing
    		$errors .= "the row type is:\n";
    		$errors .= $rowt->print();
    	}
    	return $errors;
    }

    my $sit = Triceps::IndexType->newPerlSorted("by_a_b", \&initNumStr,
    	\&compAscDesc, "a", "b") or confess "$!";

The init function can do even better: it can create and set the comparison function. It's done with:

    $idxt->setComparator(\&compareFunc)
    	or return "Failed to set comparator: $!";

Instead of the usual

    or confess

, this snippet shows

    or return

because this is the error indication convention of the init function. But

    or confess

woudl work too.

When the init function sets the comparator, the compare function argument in

    newPerlSorted()

can be left undefined, because

    setComparator()

would override it anyway. But one way or the other, the compare function must be set, or the index type initialization and with it the table type initialization will fail.

By the way, the sorted index type init function is not of the same kind as the aggregator type init function. The aggregator type could use an init function of this kind too, but at the time it looked like too much extra complexity. It probably will be added in the future. But more about aggregators later.

A fancier example of the init function will be shown in the next section.

Internally the implementation of the sorted index shares much with the hashed index. They both are implemented as trees but they compare the rows in different ways. The hashed index is aimed for speed, the sorted index for flexibility. The common implementation means that they share certain traits. Both kinds have the unique keys, there can not be two rows with the same key in an index of either kind. Both kinds allow to nest other indexes in them.

index sorted
aggregation
index hashed

## Ordered index

To specify the sorting order in a more SQL-like fashion, Triceps has the class SimpleOrderedIndex. It's implemented entirely in Perl, on top of the sorted index. Besides being useful by itself, it shows off two concepts: the initialization function of the sorted index, and the template with code generation on the fly.

First, how to create an ordered index:

    $it = Triceps::SimpleOrderedIndex->new($fieldName => $order, ...)
    	or confess "$!";

The arguments are the key fields.

    $order

is one of

    "ASC"

for ascending and

    "DESC"

for descending. Here is an example of a table with this index:

    my $tabType = Triceps::TableType->new($rowType)
    	->addSubIndex("sorted",
    		Triceps::SimpleOrderedIndex->new(
    			a => "ASC",
    			b => "DESC",
    		)
    	) or confess "$!";

When it gets translated into a sorted index, the comparison function gets generated automatically. It's smart enough to generate the string comparisons for the

    string

and

    uint8

fields, and the numeric comparisons for the numeric fields. It's not smart enough to do the locale-specific comparisons for the strings and locale-agnostic for the

    uint8

, it just uses whatever you have set up in

    cmp

for both. It treats the NULL field values as numeric 0 or empty strings. It doesn't handle the array fields at all but can at least detect such attempts and flag them as errors.

A weird artifact of the boundary between &Cpp; and Perl is that when you get the index type back from the table type like

    $sortIdx = $tabType->findSubIndex("sorted") or confess "$!";

the reference stored in

    $sortIdx

will be of the base type Triceps::IndexType. That's because the &Cpp; internals of the TableType object know nothing about any derived Perl types. But it's no big deal, since there are no other useful methods for SimpleOrderedIndex anyway. For the future, I have an idea of a workaround, but it has to wait for the future.

If you call

    $sortIdx->print()

, it will give you an idea of how it was constructed:

    PerlSortedIndex(SimpleOrder a ASC, b DESC, )

The contents of the parenthesis is a sort name from the sorted index'es standpoint. It's an arbitrary string. But when the ordered index prepares this string to pass to the sorted index, it puts its arguments into it.

Now the interesting part, I want to show the implementation of the ordered index. It's not too big and it shows the flexibility and the extensibility of Triceps:

    package Triceps::SimpleOrderedIndex;
    use Carp;

    our @ISA = qw(Triceps::IndexType);

    # Create a new ordered index. The order is specified
    # as pairs of (fieldName, direction) where direction is a string
    # "ASC" or "DESC".
    sub new # ($class, $fieldName => $direction...)
    {
    	my $class = shift;
    	my @args = @_; # save a copy

    	# build a descriptive sortName
    	my $sortName = 'SimpleOrder ';
    	while ($#_ >= 0) {
    		my $fld = shift;
    		my $dir = shift;
    		$sortName .= quotemeta($fld) . ' ' . quotemeta($dir) . ', ';
    	}

    	$self = Triceps::IndexType->newPerlSorted(
    		$sortName, \&init, undef, @args
    	) or confess "$!";
    	bless $self, $class;
    	return $self;
    }

    # The initialization function that actually parses the args.
    sub init # ($tabt, $idxt, $rowt, @args)
    {
    	my ($tabt, $idxt, $rowt, @args) = @_;
    	my %def = $rowt->getdef(); # the field definition
    	my $errors; # collect as many errors as possible
    	my $compare = "sub {\n"; # the generated comparison function
    	my $connector = "return"; # what goes between the comparison operators

    	while ($#args >= 0) {
    		my $f = shift @args;
    		my $dir = uc(shift @args);

    		my ($left, $right); # order the operands depending on sorting direction
    		if ($dir eq "ASC") {
    			$left = 0; $right = 1;
    		} elsif ($dir eq "DESC") {
    			$left = 1; $right = 0;
    		} else {
    			$errors .= "unknown direction '$dir' for field '$f', use 'ASC' or 'DESC'\n";
    			# keep going, may find more errors
    		}

    		my $type = $def{$f};
    		if (!defined $type) {
    			$errors .= "no field '$f' in the row type\n";
    			next;
    		}

    		my $cmp = "<=>"; # the comparison operator
    		if ($type eq "string"
    		|| $type =~ /^uint8.*/) {
    			$cmp = "cmp"; # string version
    		} elsif($type =~ /\]$/) {
    			$errors .= "can not order by the field '$f', it has an array type '$type', not supported yet\n";
    			next;
    		}

    		my $getter = "->get(\"" . quotemeta($f) . "\")";

    		$compare .= "  $connector \$_[$left]$getter $cmp \$_[$right]$getter\n";

    		$connector = "||";
    	}

    	$compare .= "  ;\n";
    	$compare .= "}";

    	if (defined $errors) {
    		# help with diagnostics, append the row type to the error listing
    		$errors .= "the row type is:\n";
    		$errors .= $rowt->print();
    	} else {
    		# compile the comparison
    		#print STDERR "DEBUG Triceps::SimpleOrderedIndex::init: comparison function:\n$compare\n";
    		my $cmpfunc = eval $compare
    			or return "Triceps::SimpleOrderedIndex::init: internal error when compiling the compare function:\n"
    				. "$@\n"
    				. "The generated comparator was:\n"
    				. $compare;
    		$idxt->setComparator($cmpfunc)
    			or return "Triceps::SimpleOrderedIndex::init: internal error: can not set the compare function:\n"
    			. "$!\n";
    	}
    	return $errors;
    }

The class constructor simply builds the sort name from the arguments and offloads the rest of logic to the init function. It can't really do much more: when the index type object is constructed, it doesn't know yet, where it will be used and what row type it will get. It tries to enquote nicely the weird characters in the arguments when they go into the sort name. Not that much use is coming from it at the moment: the &Cpp; code that prints the table type information doesn't do the same, so there still is a chance of misbalanced quotes in the result. But perhaps the &Cpp; code will be fixed at some point too.

The init function is called at the table type initialization time with all the needed information. It goes through all the arguments, looks up the fields in the row type, and checks them for correctness. It tries to collect as much of the error information as possible. The returned error messages may contain multiple lines separated by \n, and the ordered index makes use of it. The error messages get propagated back to the table type level, nicely indented and returned from the table initialization. If the init function finds any errors, it appends the printout of the row type too, to make finding what went wrong easier. A result of a particularly bad call to a table type initialization may look like this:

    index error:
      nested index 1 'sorted':
        unknown direction 'XASC' for field 'z', use 'ASC' or 'DESC'
        no field 'z' in the row type
        can not order by the field 'd', it has an array type 'float64[]', not supported yet
        the row type is:
        row {
          uint8 a,
          uint8[] b,
          int64 c,
          float64[] d,
          string e,
        }

Also as the init goes through the arguments, it constructs the text of the compare function in the variable

    $compare

. Here the use of

    quotemeta()

for the user-supplied strings is important to avoid the syntax errors in the generated code. If no errors are found in the arguments, the compare function gets compiled with

    eval

. There should not be any errors, but it's always better to check. Finally the compiled compare function is set in the sorted index with

    $idxt->setComparator($cmpfunc)

If you uncomment the debugging printout line (and run

    make

, and maybe

    make install

afterwards), you can see the auto-generated code printed on stderr when you use the simple ordered index. It will look somewhat like this:

    sub {
    	return $_[0]->get("a") cmp $_[1]->get("a")
    	|| $_[1]->get("c") <=> $_[0]->get("c")
    	|| $_[0]->get("b") cmp $_[1]->get("b")
    	;
    }

That's it! An entirely new piece of functionality added in a smallish Perl snippet. This is your typical Triceps template: collect the arguments, use them to build Perl code, and compile it. Of course, if you don't want to deal with the code generation and compilation, you can just call your class methods and whatnot to interpret the arguments. But if the code will be reused, the compilation is more efficient.

index ordered
template
code generation

## The index tree

The index types in a table type can form a pretty much arbitrary tree. Following the common tree terminology, the index types that have no other index types nested in them, are called the leaf index types. Since there seems to be no good one-word naming for the index types that have more index types nested in them ("inner"? "nested" is too confusing), I simply call them non-leaf.

At the moment the Hashed, Sorted and Ordered index types can be used only in both leaf and non-leaf positions. The FIFO index types must always be in the leaf position, they don't allow the further nesting.

Now is the time to look deeper into what is going on inside a table. Note that I've been very carefully talking about index types and not indexes. In this section the difference matters. The index types are in the table type, the indexes are in the table. One index type may generate multiple indexes.

This will become clearer after you see the illustrations. First, the legend in the &xrsp;.

Drawings legend.

The nodes belonging to the table type are shown in red, the nodes belonging to the table are shown in blue, and the contents of the RowHandle is shown separately in yellow. The lines on the drawings represent not exactly pointers as such but more of the logical connections that may be more complicated than the simple pointers.

The lines in the RowHandle don't mean anything at all, they just show that the parts go together. In reality a RowHandle is a chunk of memory, with various elements placed in that memory. As far as indexes are concerned, the RowHandle contains an iterator for every index where it belongs. This lets it know its position in the table, to iterate along every index, and, most importantly, to be removed quickly from every index. A RowHandle belongs to one index of each index type, and contains the matching number of iterators in it.

The table type is shown as a normal flat tree. But the table itself is more complex and becomes 3-dimensional. Its view from above matches the table type's tree but the data grows up in the third dimension.

Let's start with the simplest case: a table type with only one index type. Whether the index type is hash or FIFO, doesn't matter here.

    TableType
    +-IndexType "A"

&xrsp; shows the table structure.

One index type.

The table here always contains exactly one index, matching the one defined index type, and the root index. The root index is very dumb, its only purpose is to tie together the multiple top-level indexes into a tree.

The only index of type A provides an ordering of the records, and this ordering is used for the iteration on the table.

For the next example let's look at the straight nesting in &xrsp;.

    TableType
    +-IndexType "A"
      +-IndexType "B"

Straight nesting.

The stack of row references is shown visually divided to match the indexing, but in reality there is no special division. This was done purely to make the picture easier to read.

There is still only one index of type A. And this is always the case with the top-level indexes, there is only one of them. This index divides the rows into 3 groups. Just like the rows in a leaf index, the groups in a non-leaf index are ordered in some index-specific way.

Each group then has its own second-level index of type B. Which then defines an order for the rows in it. To reiterate: the index of type A splits the rows by groups, then the group's index of type B defines the order of the rows in the group.

So what happens when we iterate through the table and ask for the next row handle? The current row handle contains the iterators in the indexes of types A and B. The easy thing is to advance the iterator of type B. Yeah, but in which index? The &xrsp; shows three indexes of type B, let's call them B1, B2 and B3. The iterator of type B in the row handle tells the relative position in the index, but it doesn't tell, which index it is. We need to step back and look at the index type A. It's the top-level index type, so there is always only one index for it. Then we take the iterator of type A and find this row's group in the index A. The group contains the index of type B, say B1. We can then take this index B1, take the iterator of type B from the row handle, and advance this iterator in this index. If the advance succeeded, then great, we've got the next row handle. But if the current row was the last row in B1, we need to step back to the index A again, advance the current row handle's iterator of type A there, find its index B2, and pick the first row handle of B2.

This process is what happens when we use

    $rh->nextIdx($itB)

. The iteration goes by the leaf index type B, however it relies on all the index types in the path from the table type to B. If we do

    $rh->next()

, the result is the same because the first leaf index type is used as the default index type for the iteration.

If we do

    $rh->next($itA)

, the semantics is still the same: return the next row handle (not the next group). There is no way to get to the row handle without going all the way through a leaf index. So when a non-leaf index type is used for the iteration, it gets implicitly extended to its first nested leaf index type.

What would happen if a new row gets inserted, and the index type A determines that it does not belong to any of the existing groups? A new group will be created and inserted in the appropriate position in A's order. This group will have a new index of type B created, and the new row inserted in that index.

What would happen if both rows in B1 are removed? B1 will become empty and will be collapsed. The index A will delete the B1's group and B1 itself, and will remain with only two groups. The effect propagates upwards: if all the rows are removed, the last index of type B will collapse, then the index A will become empty and also collapse and be deleted. The only thing left will be the root index that stays in the existence no matter what.

When a table is first created, it has only the root index. The rest of the indexes pop into the existence as the rows get inserted. If you wonder, yes, this does apply to a table type with only one index type as well. Just this point has not been brought up until now.

Among all this froth of creation and collapse the iterators stay stable. Once a row is inserted, the indexes leading to it are not going anywhere (at least until that row gets removed). But since other rows and groups may be inserted around it, the notion of what row is next, will change over time.

Let's go through how the other index-related operations work.

The iteration through the whole table starts with

    begin()

or

    beginIdx()

, the first being a form of the second that always uses the first leaf index type.

    beginIdx()

is fairly straightforward: it just follows the path from the root to the leaf, picking the first position in each index along the way, until it hits the RowHandle, as is shown in &xrsp;. That found RowHandle becomes its result. If the table is empty, it returns the NULL row handle.

    begin()

,

    beginIdx($itA)

and

    beginIdx($itB)

work the same for this table.

The next pair is

    find()

and

    findIdx()

(and

    findBy()

and

    findIdxBy()

are wrappers around those). As usual,

    find()

is the same thing as

    findIdx()

on the table's first leaf index type. It also follows the path from the root to the target index type. On each step it tries to find a matching position in the current index. If the position could not be found, the search fails and a NULL row handle is returned. If found, it is used to progress to the next index.

As has been mentioned in &xrsp;, the search always works internally on a RowHandle argument. If a plain Row is used as an argument, a new temporary RowHandle will be created for it, searched, and then freed after the search. This works well for two reasons. First, the indexes already have the functions for comparing two row handles to build their ordering. The same functions are reused for the search. Second, the row handles contain not only the index iterators but also the cached information from the rows, to make the comparisons faster. The exact kind of cached information varies by the index type. The FIFO, Sorted and Ordered indexes use none. The Hashed indexes calculate a hash of the key field values, that will be used as a quick differentiator for the search. This information gets created when the row handle gets created. Whether the row handle is then used to insert into the table or to search in it, the hash is then used in the same way, to speed up the comparisons.

In

    findIdx()

, the non-leaf index type arguments behave differently than the leaf ones: up to and including the index of the target type, the search works as usual. But then at the next level the logic switches to the same as in

    beginIdx()

, going for the first row handle of the first leaf sub-index. This lets you find the first row handle of the matching group under the target index type.

If you use

    $table->findIdx($itA, $rh)

, on &xrsp; it will go through the root index to the index A. There it will try to find the matching position. If none is found, the search ends and returns a NULL row handle. If the position is found, the search progresses towards the first leaf sub-index type. Which is the index type B, and which conveniently sits in this case right under A. The position in the index A determines, which index of type B will be used for the next step. Suppose it's the second position, so the second index of type B is used. Since we're now past the target index A, the logic used is the same as for

    beginIdx()

, and the first position in B2 is picked. Which then leads to the first row handle of the second sub-stack of handles.

    findIdx($itA, $rh)

goes through A and then switches to the

    beginIdx()

logic.

The method

    firstOfGroupIdx()

allows to navigate within a group, to jump from some row somewhere in the group to the first one, and then from there iterate through the group. The example in &xrsp; made use of it.

The &xrsp; shows an example of

    $table->firstOfGroupIdx($itB, $rh)

, where

    $rh

is pointing to the third record in B2. What it needs to do is go back to B2, and then execute the

    begin()

logic from there on. However, remember, the row handle does not have a pointer to the indexes in the path, it only has the iterators. So, to find B2, the method does not really back up from the original row. It has to start all the way back from the root and follow the path to B2 using the iterators in

    $rh

. Since it uses the ready iterators, this works fast and requires no row comparisons. But logically it's equivalent to backing up by one level, and I'll continue calling it that for simpicity. Once B2 (an index of type B) is reached, the

    begin()

logic goes for the first row in there.

    firstOfGroupIdx()

works on both leaf and non-leaf index type arguments in the same way: it backs up from the reference row to the index of that type and executes the

    begin()

logic from there. Obviously, if you use it on a non-leaf index type, the

    begin()

-like part will follow its first leaf index type.

    firstOfGroupIdx($itB, $rh)

.

The method

    nextGroupIdx()

jumps to the first row of the next group, according to the argument index type. To do that, it has to retrace one level higher than

    firstOfGroupIdx()

. &xrsp; shows that

    $table->nextGroupIdx($itB, $rh)

that starts from the same row handle as in &xrsp;, has to logically back up to the index A, go to the next iterator there, and then follow to the first row of B3.

    nextGroupIdx($itB, $rh)

.

As before, in reality there is no backing up, just the path is retraced from the root using the iterators in the row handle. Once the parent of index type B is reached (which is the index of type A), the path follows not the iterator from the row handle but the next one (yes, copied from the row handle, increased, followed). This gives the index of type B that contains the next group. And from there the same

    begin()

-like logic finds its first row.

Same as

    firstOfGroupIdx()

,

    nextGroupIdx()

may be used on both the leaf and non-leaf indexes, with the same logic.

It's kind of annoying that

    firstOfGroupIdx()

and

    nextGroupIdx()

take the index type inside the group while

    findIdx()

uses takes the parent index type to act on the same group. But as you can see, each of them follows its own internal logic, and I'm not sure if they can be reconciled to be more consistent.

At the moment the only navigation is forward. There is no matching

    last()

,

    prev()

or

    lastGroupIdx()

or

    prevGroupIdx()

. They are in the plan, but so far they are the victims of corner-cutting. Though there is a version of

    last()

in the AggregatorContext, since it happens to be particularly important for the aggregation.

Continuing our excursion into the index nesting topologies, the next example is of two parallel leaf index types:

    TableType
    +-IndexType A
    +-IndexType B

The resulting internal arrangement is shown in &xrsp;.

Two top-level index types.

Each index type produces exactly one index under the root (since the top-level index types always produce one index). Both indexes contain the same number of rows, and exactly the same rows. When a row is added to the table, it's added to all the leaf index types (one actual index of each type). When a row is deleted from the table, it's deleted from all the leaf index types. So the total is always the same. However the order of rows in the indexes may differ. The drawing shows the row references stacked in the same order as the index A because the index A is of the first leaf index type, and as such is the default one for the iteration.

The row handle contains the iterators for both paths, A and B. It's pretty normal to find a row through one index type and then iterate from there using the other index type.

The next example in &xrsp; has a primary index with a unique key and a secondary index that groups the records:

    TableType
    +-IndexType A
    +-IndexType B
      +-IndexType C

A primary and secondary index type.

The index type A still produces one index and references all the rows directly. The index of type B produces the groups, with each group getting an index of type C. The total set of rows referrable through A and through B is still the same but through B they are split into multiple groups.

And &xrsp; shows two leaf index types nested under one non-leaf.

    TableType
    +-IndexType A
      +-IndexType B
      +-IndexType C

Two index types nested under one.

As usual, there is only one index of type A, and it splits the rows into groups. The new item in this picture is that each group has two indexes in it: one of type B and one of type C. Both indexes in the group contain the same rows. They don't decide, which rows they get. The index A decides, which rows go into which group. Then if the group 1 contains two rows, indexes B1 and C1, would both contain two rows each, the exact same set. The stack of row references has been visually split by groups to make this point more clear.

This happens to be a pretty useful arrangement: for example, B might be a hash index type, or a sorted index type, allowing to find the records by the key (and for the sorted index, to iterate in the order of keys), while C might be a FIFO index, keeping the insertion order, and maybe keeping the window size limited.

That's pretty much it for the basic index topologies. Some much more complex index trees can be created, but they would be the combinations of the examples shown. Also, don't forget that every extra index type adds overhead in both memory and CPU time, so avoid adding indexes that are not needed.

One more fine point has to do with the replacement policies. Consider that we have a table that contains the rows with a single field:

    id int32

And the table type has two indexes:

    TableType
    +-IndexType "A" HashIndex key=(id)
    +-IndexType "B" FifoIndex limit=3

And we send there the rowops:

    INSERT id=1
    INSERT id=2
    INSERT id=3
    INSERT id=2

The last rowop that inserts the row with id=2 for the second time triggers the replacement policy in both index types. In the index A it is a duplicate key and will cause the removal of the previous row with id=2. In the index B it overflows the limit and pushes out the oldest row, the one with id=1. If both records get deleted, the resulting table contents will be 2 rows (shown in FIFO order):

    id=3
    id=2

Which is probably not the best outcome. It might be tolerable with a FIFO index and a hashed index but gets even more annoying if there are two FIFO index types in the table: one top-level limiting the total number of rows, another one nested under a hashed index, limiting the number of rows per group, and they start conflicting this way with each other.

The Triceps FIFO index is actually smart enough to avoid such problems: it looks at what the preceding indexes have decided to remove, checks if any of these rows belong to its group, and adjusts its calculation accordingly. In this example the index B will find out that the row with id=2 is already displaced by the index A. That leaves only 2 rows in the index B, so adding a new one will need no displacement. The resulting table contents will be

    id=1
    id=3
    id=2

However here the order of index types is important. If the table were to be defined as

    TableType
    +-IndexType "B" FifoIndex limit=3
    +-IndexType "A" HashIndex key=(id)

then the replacement policy of the index type B would run first, find that nothing has been displaced yet, and displace the row id=1. After that the replacement policy of the index type A will run, and being a hashed index, it doesn't have a choice, it has to replace the row id=2. And both rows end up displaced.

If the situations with automatic replacement of rows by the keyed indexes may arise, always make sure to put the keyed leaf index types before the FIFO leaf index types. However if you always diligently send a DELETE before the INSERT of the new version of the recond, then this problem won't occur and the order of index types will not matter.

index leaf
index non-leaf
RowHandle
index root
index group
table iteration
index default
table insert
table iteration
table find
table replacement

## Table and index type introspection

A lot of information about a table type and the index types in it can be read back from them.

    $result = $tabType->isInitialized();
    $result = $idxType->isInitialized();

return whether a table or index type has been initialized. The index type gets initialized when the table type where it belongs gets initialized. After a table or index type has been initialized, it can not be changed any more, and any methods that change it will return an error. When an index type becomes initialized, it becomes tied to a particular table type. This table type can be read with

    $tabType = $idxType->getTabtype() or confess "$!";

Even though an initialized index type can't be tied to another table, when you add it to another table or index type, a deep copy with all its sub-indexes will be made automatically, and that copy will be uninitialized. So it will be able to get initialized and tied to the new table. However if you want to add more sub-indexes to it, do a manual copy first:

    $idxTypeCopy = $idxType->copy();

The information about the nested indexes can be found with:

    $itSub = $tabType->findSubIndex("indexName") or confess "$!";
    @itSubs = $tabType->getSubIndexes();

    $itSub = $idxType->findSubIndex("indexName") or confess "$!";
    @itSubs = $idxType->getSubIndexes();

The

    findSubIndex()

has been already shown in &xrsp;. It allows to find the index types on the next level of nesting, starting down from the table, and going recursively into the sub-indexes.

    getSubIndexes()

returns the information about the index types of the next level at once, as the name => value pairs. The result array can be placed into a hash but that would lose the order of the sub-indexes, and the order is important for the logic.

This finds the index types step by step. An easier way to find an index type in a table type by the path of the index is with

    $idxType = $tabType->findIndexPath(\@idxNames);

The arguments in the array form a path of names in the index type tree. If the path is not found, the function would confess. An empty path is also illegal and would cause the same result. Yes, the argument is not an array but a reference to array. This array is used essentially as a path object. For example the index from the &xrsp; could be found as:

    $itLast2 = $ttWindow->findIndexPath([ "bySymbol", "last2" ]);

The key (the set of fields that uniquely identify the rows) of the index type can be found with

    @keys = $it->getKey();

It can be used on any kind of index types but actually returns the data only for the Hashed index types. On the other index types it returns an empty array, though a better support will be available for the Sorted and Ordered indexes in the future.

A fairly common need is to find an index by its name path, and also all the key fields that are used by all the indexes in this path. It's used for such purposes as joins, and it allows to treat a nested index pretty much as a composition of all the indexes in its path. The method

    ($idxType, @keys) = $tableType->findIndexKeyPath(\@path);

solves this problem and finds by path an index type that allows the direct look-up by key fields. It requires that every index type in the path returns a non-empty array of fields in

    getKey()

. In practice it means that every index in the path must be a Hashed index. Otherwise the method confesses. When the Sorted and maybe other index types will support

    getKey()

, they will be usable with this method too.

Besides checking that each index type in the path works by keys, this method builds and returns the list of all the key fields required for a look-up in this index. Note that

    @keys

is an actual array and not a reference to array. The return protocol of this method is a little weird: it returns an array of values, with the first value being the reference to the index type, and the rest of them the names of the key fields. If the table type were defined as

    $tt = Triceps::TableType->new($rt)
    	->addSubIndex("byCcy1",
    		Triceps::IndexType->newHashed(key => [ "ccy1" ])
    		->addSubIndex("byCcy12",
    			Triceps::IndexType->newHashed(key => [ "ccy2" ])
    		)
    	)
    	->addSubIndex("byCcy2",
    		Triceps::IndexType->newHashed(key => [ "ccy2" ])
    		->addSubIndex("grouping", Triceps::IndexType->newFifo())
    	)
    or confess "$!";

then

    $tt->findIndexKeyPath([ "byCcy1", "byCcy12" ])

would return

    ($ixtref, "ccy1", "ccy2")

, where

    $ixtref

is the reference to the index type. When assigned to

    ($ixt, @keys)

,

    $ixtref

would go into

    $ixt

, and

    ("ccy1", "ccy2")

would go into

    @keys

.

The key field names in the result go in the order they occurred in the definition, from the outermost to the innermost index. The key fields must not duplicate. It's possible to define the index types where the key fields duplicate in the path, say:

    $tt = Triceps::TableType->new($rt)
    	->addSubIndex("byCcy1",
    		Triceps::IndexType->newHashed(key => [ "ccy1" ])
    		->addSubIndex("byCcy12",
    			Triceps::IndexType->newHashed(key => [ "ccy2", "ccy1" ])
    		)
    	)
    or confess "$!";

And they would even work fine, with just a little extra overhead from duplication. But

    findIndexKeyPath()

will refuse such indexes and confess.

The kind of the index type is also known as the type id. It can be found for an index type with

    $id = $idxType->getIndexId();

It's an integer constant, matching one of the values:

-     &Triceps::IT_HASHED

-     &Triceps::IT_FIFO

-     &Triceps::IT_SORTED

There is no different id for the ordered index, because it's built on top of the sorted index, and would return

    &Triceps::IT_SORTED

.

The conversion between the strings and constants for index type ids is done with

    $intId = &Triceps::stringIndexId($stringId);
    $stringId = &Triceps::indexIdString($intId);

If an invalid value is supplied, the conversion functions will return

    undef

.

There is also a way to find the first index type of a particular kind. It's called somewhat confusingly

    $itSub = $idxType->findSubIndexById($indexTypeId) or confess "$!";

where

    $indexTypeId

is one of either of Triceps constants or the matching strings

    "IT_HASHED"

,

    "IT_FIFO"

,

    "IT_SORTED"

.

Technically, there is also IT_ROOT but it's of little use for this situation since it's the root of the index type tree hidden inside the table type, and would never be a sub-index type. It's possible to iterate through all the possible index type ids as

    for ($i = 0; $i < &Triceps::IT_LAST; $i++) { ... }

The first leaf sub-index type, that is the default for iteration, can be found explicitly as

    $itSub = $tabType->getFirstLeaf();
    $itSub = $idxType->getFirstLeaf();

If an index is already a leaf,

    getFirstLeaf()

on it will return itself. The leaf-ness of an index type can be found with:

    $result = $idxType->isLeaf();

The usual reference comparison methods are:

    $result = $tabType1->same($tabType2);
    $result = $tabType1->equals($tabType2);
    $result = $tabType1->match($tabType2);

    $result = $idxType1->same($idxType2);
    $result = $idxType1->equals($idxType2);
    $result = $idxType1->match($idxType2);

Two table types are considered equal when they have the equal row types, and exactly the same set of index types, with the same names.

Two table types are considered matching when they have the matching row types, and matching set of index types, although the names of the index types may be different.

Two index types are considered equal when they are of the same kind (type id), their type-specific parameters are equal, they have the same number of sub-indexes, with the same names, and equal pair-wise. They must also have the equal aggregators, which will be described in detail in the &xrsp;.

Two index types are considered matching when they are of the same kind, have matching type-specific parameters, they have the same number of sub-indexes, which are matching pair-wise, and the matching aggregators. The names of the sub-indexes may differ. As far as the type-specific parameters are concerned, it depends on the kind of the index type. The FIFO type considers any parameters matching. For a Hashed index the key fields must be the same. For a Sorted index the sorted condition must also be the same, and by extension this means the same condition for the Ordered index.

index copy
index path
error handling
index key
index type id
constants
index default
TableType equals
TableType match
IndexType equals
IndexType match

## The copy tray

The table methods

    insert()

,

    remove()

and

    deleteRow()

have an extra optional argument: the copy tray.

If used, it will put a copy of all the rowops produced during the operation (including the output of the aggregators, which will be described in &xrsp; ) into that tray. The idea here is to use it in cases if you don't want to connect the output labels of the table directly, but instead collect and process the rows from the tray manually afterwards. Like this:

    $ctr = $unit->makeTray();
    $table->insert($row, $ctr);
    foreach my $rop ($ctr->toArray()) {
    	...
    }

However in reality it didn't work out so well. The processing loop would have to have all the lengthy if-else sequences to branch first by the label (if there are any aggregators) and then by opcode. It looks too difficult. Well, it could work in the simple situations but not more than that.

In the future this feature will likely be deprecated unless it proves itself useful, and I already have a better idea. Because of this, I see no point in going into the more extended examples.

copy tray
tray

## Table wrap-up

Not all of the table's features have been shown yet. The table class is the cornerstone of Triceps, and everything is connected to it. The aggregators work with the tables and are a whole separate big subject with their own chapter. There also are many more options and small methods that haven't been touched upon yet. They are enumerated in the reference chapter, please refer there.