# Scheduling

## Overview of the scheduling

The scheduling determines, in which order the row operations are processed. If there are multiple operations available, which one should be processed first? The scheduler keeps a queue of the operations and selects, which one to execute next. This has a major effect on the logic of a CEP model.

There are multiple approaches to scheduling. Aleri essentially doesn't have any, except for the flow control between threads, because each its element is a separate thread. Coral8 has an intricate scheduling algorithm. Sybase R5 has the same logic as Coral8 inside each thread. StreamBase presumably also has some.

The scheduling logic in Triceps is different from the other CEP systems. The Coral8 logic looks at first like the only reasonable way to go, but could not be used in Triceps for three reasons: First, it's a trade secret, so it can't be simply reused. If I'd never seen it, that would not be an issue but I've worked on it and implemented its version for R5. Second, it relies on the properties that the compiler computes from the model graph analysis. Triceps has no compiler, and could not do this. Third, in reality it simply doesn't work that well. There are quite a few cases when the Coral8 scheduler comes up with a strange and troublesome execution order.

For a while I've hoped that Triceps would need no scheduler at all, and everything would be handled by the procedural calls. This has proved to have its own limitations, and thus the labels and their scheduling were born. The Triceps scheduling still has issues to resolve, but overall it still feels much better than the Coral8 one.

scheduling
model
Aleri
Coral8
Sybase
StreamBase

## No bundling

The most important principle of Triceps scheduling is: No Bundling. Every rowop is for itself. The bundling is what messes up the Coral8 scheduler the most.

What is a bundle? It's a set of records that go through the execution together. If you have a model consisting of two functional elements F1 and F2 connected in a sequential fashion

    F1->F2

and a few loose records R1, R2, R3, the normal execution order without bundling will be:

    F1(R1), F2(R1), F1(R2), F2(R2), F1(R3), F2(R3)

Each row goes through the whole model (a real simple one in this case) before the next one is touched. This allows F2 to take into accont the state of F1 exactly as it was right after processing the same record, without any interventions in between.

If the same records are placed in a bundle (R1, R2, R3), the execution order will be different:

    F1(R1), F1(R2), F1(R3), F2(R1), F2(R2), F2(R3)

The whole bundle goes through F1 before the rows go to F2.

That would not be a problem, and even could be occasionally useful, if the bundles were always created explicitly. In the reality of Coral8, every time a statement produces multiple record from a single one (think of a join that picks multiple records from another side), it creates a bundle and messes up all the logic after it. Some logic gets affected so badly that a few statements in CCL (like ON UPDATE) had to be designated as always ignoring the bundles, otherwise they would not work at all. At DB I wrote a CCL pattern for breaking up the bundles. It's rather heavyweight and thus could not be used all over the place but provides a generic solution for the most unpleasant cases.

Worse yet, the bundles may get created in Coral8 absolutely accidentally: if two records happen to have the same timestamp, for all practical purposes they would act as a bundle. In the models that were designed without the appropriate guards, this leads to the time-based bugs that are hard to catch and debug. Writing these guards correctly is hard, and testing them is even harder.

Another issue with bundles is that they make the large queries slower. Suppose you do a query from a window that returns a million records. All of them will be collected in a bundle, then the bundle will be sent to the interface gateway that would build one huge protocol packet, which will then be sent to the client, which will receive the whole packet and then finally iterate on the records in it. Assuming that nothing runs out of memory along the way, it will be a long time until the client sees the first record. Very, very annoying.

Aleri also has its own version of bundles, called transactions, but a more smart one. Aleri always relies on the primary keys. The condition for a transaction is that it must never contain multiple modification for the same primary key. Since there are no execution order guarantees between the functional elements, in this respect the transactions work in the same way as loose records, only with a more efficient communication between threads. Still, if the primary key changes in an element (say, an aggregator), the condition does not propagate through it. Such elements have to internally collapse the outgoing transactions along the new key, adding overhead.

bundling

## Basic scheduling in Triceps

In Triceps the scheduling is done by the execution unit, or simply unit as it's often referred to. It provides 3 basic ways of executing of a rowop:

Call:

  Execute the label right now, including all the nested calls. All of this will be completed after the call returns.

call
Fork:

  Execute the label after the current label returns but before anything else is done. Obviously, if multiple labels are forked, they will execute in order after the current label returns (but before its caller gets the control back). This method has looked promising at one point but has currently fallen out of favor and will likely be removed in the future.

fork
Schedule:

  Execute the label after everything else is done.

schedule

This is kind of intuitively clear but the details might sometimes be a bit surprising. So let us look in detail at how it works inside on an example of a fairly convoluted scheduling sequence.

A scheduler in the execution unit keeps a stack of queues. Each queue is essentially a stack frame, so I'll be using the terms

    queue

and

    frame

interchangeably. The stack always contains at least one queue, which is called the outermost stack frame.

When the new rowops come from the outside world, they are added with

    schedule()

to that stack frame. That's what

    schedule()

does: always adds rowops to the outermost stack frame. If rowops 1, 2 and 3 are added, the stack looks like this (the brackets denote a stack frame):

    [1, 2, 3]

The unit method

    drainFrame()

is then used to run the scheduler and process the rowops. It makes the unit call each rowop on the innermost frame (which is initially the same as outermost frame, since there is only one frame) in order.

First it calls the rowop 1. It's removed from the queue, then a new frame is pushed onto the stack:

    [ ] ~1
    [2, 3]

This new frame is the rowop 1's frame, which is marked on the diagram by ~1. The diagram shows the most recently pushed, innermost, frame on the top, and the oldest, outermost frame on the bottom. The concepts of innermost and outermost come from the nested calls: the most recent call is nested the deepest in the middle and is the innermost one.

Then the rowop 1 executes. If it calls rowop 4, another frame is pushed onto the stack for it:

    [ ] ~4
    [ ] ~1
    [2, 3]

Then the rowop 4 executes. The rowop 4 never gets onto any of the queues. The call just pushes a new frame and executes the rowop right away. The identity of rowop being processed is kept in the call context. A call also involves a direct &Cpp; call on the thread stack, and if any Perl code is involved, a Perl call too. Because of this, if you nest the calls too deeply, you may run out of the thread stack space and get it to crash.

After the rowop 4 is finished (not calling any other rowops), the innermost empty frame is popped before the execution of rowop 1 continues. The queue stack reverts to the previous state.

    [ ] ~1
    [2, 3]

Suppose then rowop 1 forks rowops 5 and 6. They are appended to the innermost frame in the order they are forked.

    [5, 6] ~1
    [2, 3]

If rowop 1 then calls rowop 7, again a frame is pushed onto the stack before it executes:

    [ ] ~7
    [5, 6] ~1
    [2, 3]

The rowops 5 and 6 still don't execute, they keep sitting on the queue until the rowop 1 would return. After the call of rowop 7 completes, the scheduler stack returns to the previous state.

Suppose now the execution of rowop 1 completes. But its stack frame can not be popped yet, because it is not empty. The scheduler calls

    drainFrame()

recursively, which picks the next rowop from the innermost queue (rowop 5), and calls it, pushing a new stack frame and executing the rowop 5 code:

    [ ] ~5
    [6] ~1*
    [2, 3]

The former rowop 1's frame is now marked with ~1* for the ease of tracking, even though it has completed.

If rowop 5 forks rowop 8, the stack becomes:

    [8] ~5
    [6] ~1*
    [2, 3]

When the execution of rowop 5 returns, its queue is also not empty. So the scheduler starts draining the innermost frame again, and calls rowop 8. During its execution the stack is:

    [ ] ~8
    [ ] ~5*
    [6] ~1*
    [2, 3]

Suppose the rowop 8 doesn't call or fork anything else and returns. Its innermost queue is empty, so the call completes and pops the stack frame:

    [ ] ~5*
    [6] ~1*
    [2, 3]

Now the queue of rowop 5 is also empty, so its draining completes and pops the drained frame:

    [6] ~1*
    [2, 3]

The draining of the rowop 1's frame continues by picking the rowop 6 from the queue and calling it:

    [ ] ~6
    [ ] ~1*
    [2, 3]

Suppose rowop 6 calls

    schedule()

of rowop 9. Rowop 9 is then added to the outermost queue:

    [ ] ~6
    [ ] ~1*
    [2, 3, 9]

Rowop 6 then returns, its queue is empty, so it's popped and its call completes.

    [ ] ~1*
    [2, 3, 9]

Now the queue of rowop 1 has become empty, so it's popped from the stack and the call of rowop 1 completes:

    [2, 3, 9]

The unit method

    drainFrame()

keeps running on the outermost frame, now taking the rowop 2 and executing it, and so on, until the outermost queue becomes empty, and

    drainFrame()

returns.

An interesting question is, what happens with the chained labels? Where do they fit in the order of execution? It turns out to be a bit of a mix between a

    call()

and a

    fork()

. They get checked after the original label completes its execution and has its frame drained but before that frame gets popped.

If any chained labels are found, they are called one by one. But they don't get a new frame created. They all reuse the frame left over from the parent label. The frame gets popped only after all the chained labels have completed.

unit
row operation
queue
frame
stack
draining
label chaining

## Loop scheduling

The easiest and most efficient way to schedule the loops is to do it procedurally, something like this:

    foreach my $row (@rowset) {
    	$unit->call($lbA->makeRowop(&Triceps::OP_INSERT, $row));
    }

However the labels topologically connected into a loop can come handy as well. Some logic may be easier to express this way. Suppose the model contains the labels connected in a loop, as in &xrsp;.

Labels forming a loop.

But if handled simple-mindedly, it can use a lot of stack space. Suppose some rowop X1 is scheduled for label X, and causes the loop to be executed twice, with rowops X1, A2, B3, C4, A5, B6, C7, Y8. If each operation is done as a

    call()

, the stack grows like this: It starts with X1 scheduled.

    [X1]

Which then gets executed, with its own execution frame (marked as such for clarity):

    [ ] ~X1
    [ ]

Which then calls A2:

    [ ] ~A2
    [ ] ~X1
    [ ]

By the time the execution comes to Y8, the stack looks like this:

    [ ] ~Y8
    [ ] ~C7
    [ ] ~B6
    [ ] ~A5
    [ ] ~C4
    [ ] ~B3
    [ ] ~A2
    [ ] ~X1
    [ ]

The loop has been converted into recursion, and the whole length of execution is the depth of the recursion. If the loop executes a million times, the stack will be three million levels deep. Worse yet, it's not just the Triceps scheduler stack that grows, it's also the process (&Cpp;) stack.

Which is why this kind of recursive calls are explicitly forbidden in Triceps. If you try to do it, on the first recursive call the execution will die with an error.

Would things be better with

    fork()

instead of

    call()

used throughout the loop? It starts the same way:

    [X1]

Then X1 executes, gets its own frame and forks A2:

    [A2] ~X1
    [ ]

Then A2 executes, gets its own frame and forks B3:

    [B3] ~A2
    [ ] ~X1*
    [ ]

Even though X1 has completed, its stack frame stays until all the rowops forked in it complete too. By the end of the loop the stack picture becomes exactly the same as with

    call()

. For a while I've thought that optimizing out the empty stack frames would solve the problem, but no, that doesn't work: the problem is that the &Cpp; process stack keeps growing no matter what. The jump back in the loop needs to be placed into an earlier stack frame to prevent the stack from growing.

One way to do it would be to use the

    schedule()

operation in C to jump back to A, placing the rowop A5 back onto the outermost frame. The scheduler stack at the end of C4 would look like:

    [ ] ~C4
    [ ] ~B3
    [ ] ~A2
    [ ] ~X1
    [A5]

Then the stack would unwind back to:

    [A5]

And the next iteration of the loop will start afresh. The problem here is that if X1 wanted to complete the loop and then do something, it can't. By the time the second iteration of the loop starts, X1 is completely gone. It would be better to be able to enqueue the next execution of the loop at the specific point of the stack.

Here the concept of the frame mark comes in. A frame mark is a token object, completely opaque to the program. It can be used only in two operations:

-     setMark()

  remembers the position in the frame stack, just outside the current frame.

-     loopAt()

  enqueues a rowop at the marked frame.

Then the loop wold have its mark object M. The label A will execute

    setMark(M)

, and the label C will execute

    loopAt(M, rowop(A))

. The rest of the execution can as well use

    call()

, as shown in &xrsp;.

Proper calls in a loop.

When A2 calls setMark(M), the stack will look like this:

    [ ] ~A2
    [ ] ~X1, mark M
    [ ]

The mark M remembers the frame one outer to the current one. The stack at the end of C4, after it has called

    loopAt(M, A5)

, is:

    [ ] ~C4
    [ ] ~B3
    [ ] ~A2
    [A5] ~X1, mark M
    [ ]

The stack then unwinds until A5 starts its execution:

    [ ] ~A5
    [ ] ~X1*, mark M
    [ ]

Each iteration starts with a fresh stack, and the stack depth is limited to one iteration. The nested loops can also be properly executed.

Now, why does the mark get placed on the frame that is one out from the current one? After all, this means that X1 can not wait for the loop to complete. It has to return before the second iteration of the loop can start. And then the rest of the loop will run before the control returns to X1's caller. At least the caller of X1 can wait for the loop to complete before continuing its execution. Why all this trouble? Its the result of a compromise. Suppose that it did remember the current frame. Then at the end of C4 the stack will be:

    [ ] ~C4
    [ ] ~B3
    [A5] ~A2, mark M
    [ ] ~X1
    [ ]

The stack will unwind until A5. Which would then have its own frame pushed onto the stack, and the code in the label A will call

    setMark(M)

again, moving the mark to A5's own frame because it's the topmost frame now:

    [ ] ~A5, mark M
    [ ] ~A2*
    [ ] ~X1
    [ ]

So on each iteration of the loop one extra frame will be pushed onto the stack, and the mark moved by one level. A loop executing a million times will push a million frames, which is bad. Marking the next outer frame prevents this. Another option would have been to put the mark operation in X, but that would mean that every loop must have a preceding label that just marks the frame (well, and potentially could do the other initializations too), which seems to be too annoying.

It's one problem or the other, and the lesser problem won. To further reduce the complexity, I've also added the methods

    makeLoopHead()

and

    makeLoopAround()

that take care of constructing the whole front part of the loop, including the setting of the mark. They will be described below in &xrsp;. This is still messy, and I'm still thinking about the ways to improve the situation.

What happens after the stack unwinds past the mark? The mark gets unset. When someone calls

    loopAt()

with an unset mark, the rowop is enqueued in the outermost frame, having the same effect as schedule

    ()

.

This handling of an unset mark comes handy in case if the loop execution takes a pause in the middle. Suppose the label B finds that it can't process the rowop B3 until some other data has arrived. What it can do then is remember B3 somewhere in the thread state and return. The loop has not completed but it can't progress either, so the call unrolls until it becomes empty. Since the frame of X1 is popped off the stack, the mark M gets unset. The knowledge that the loop needs to be continued stays remembered in the state.

After some time that awaited data arrives, as some other rowop. When that rowop gets processed, it finds that remembered state with B3 and makes it continue, maybe by calling

    call(B3)

again. So now the logic in B finds all the data it needs and continues with the loop, calling C4. C4 will do its job and call

    loopAt(M, A5)

. But the mark M has been unset a while ago! Scheduling A5 at the outermost frame seems to be a logical thing to do at this point. Then whatever current processing will complete and unwind, and the loop will continue after it.

What if

    setMark()

is called when there is only one frame on the stack? Then there is no second frame outer to it. The mark will simply be left unset.

But overall pausing and then restarting a loop like this is not such a good idea. The caller of the loop normally expects that it can wait for the loop to complete, and that when the loop returns, it's all done. If a loop may decide to bail out now and continue later, the caller has to be prepared for it.

And the sequence of execution when the loop continues might not be direct. In the normal execution one iteration of the loop follows directly after the previous one because of the orchestration by the label at the head of the loop (X in this example). When C calls

    loopAt()

, the rowop gets pushed onto the stack frame of X. It would execute immediately only if X is draining its frame, and only if there are no other rowops queued on that frame in front of this one. I've been seeing it as a feature: X can easily be careful and make sure that the whole loop executes in one go without any interruptions.

    makeLoopHead()

and

    makeLoopAround()

create such careful labels. But it may also decide to run multiple loops interleaved, with each one making one iteration at a time. To do that, all it needs is fork the rowops to start all these loops and then drain the frame (directly or by returning from its own code). This way you can for example make a batch of records run through the loop (or even through the different loops) with the whole batch going throgh one iteration before the another iteration starts, achieving a kind of bundling. Though, if fork gets removed, this would not be possible any more. Maybe is's a reason for fork to stay, maybe it's a useless feature.

However if a loop decides to pause and then continues on some other event, its following

    loopAt()

pushing the rowop onto the outermost frame, there is no caring parent to do the careful orchestration. There is no way to tell, which other rowops have been pushed onto the outermust frame by this time. The loop won't continue until these rowops execute. They may change the state of the model, so if the loop code expects it to stay the same, it will be mightily surprised.
scheduling loop
topological loop
scheduling recursion
recursion
frame mark
scheduling loop interleaving
batch
bundling
fork
schedule

## Execution unit

After discussing the principles of scheduling in Triceps, let's get down to the nuts and bolts.

A unit is created as:

    $myUnit = Triceps::Unit->new("name") or confess "$!";

The name argument is as usual used for later debugging, and by convention should be the same as the name of the unit variable (myUnit in this case). The name can also be changed later:

    $myUnit->setName("newName");

It returns no value. Though in practice there is no good reason for changing names, and this call will likely be removed in the future. The name can be read back:

    $name = $myUnit->getName();

Also, as usual, the variable $myName here contains a reference to the actual unit object, and two references can be compared for whether they refer to the same object:

    $result = $unit1->same($unit2);

A unit also keeps an empty row type (one with no fields), primarily for the creation of the clearing labels, but you can use it for any other purposes too. You can get it with the method:

    $rt = $unit->getEmptyRowType();

Each unit has its own instance of an empty row type. Its purely for the conveniece of memory management, they are all equivalent.

The rowops are enqueued with the calls:

    $unit->call($rowop, ...);
    $unit->fork($rowop, ...);
    $unit->schedule($rowop, ...);

Enqueued is an ugly word but since I've already used the word schedule for a specific purpose, I needed another word to name all these operations together. Hence enqueue.

The ... shows that multiple rowops may be passed as arguments. So the real signature of these methods is:

    $unit->call(@rowops);
    $unit->fork(@rowops);
    $unit->schedule(@rowops);

But this way it loos more confusing. Calling these functions with multiple arguments produces the same result as doing multiple calls with one argument at a time. Not only rowops but also trays (to be discussed later) of rowops can be used as arguments.

These methods are among those that use the new error handling, that makes the operation to confess on any fatal errors. So there is no need to check their results with

    or confess

.

Also there is a call that selects the enqueueing mode by argument:

    $unit->enqueque($mode, @rowops);

The calling rules are exactly the same for the other enqueueing methods, may have multiple rowops or trays as arguments, no need to check the result. The

    $mode

argument is one of:

-     &Triceps::EM_CALL

  or

      "EM_CALL"

-     &Triceps::EM_FORK

  or

      "EM_FORK"

-     &Triceps::EM_SCHEDULE

  or

      "EM_SCHEDULE"

As usual, there are calls to convert between the integer constant and string representations:

    $string = &Triceps::emString($value);
    $value = &Triceps::stringEm($string);

And as usual, if the value can not be translated, they return

    undef

.

The frame marks for looping are created as their own class:

    $mark = Triceps::FrameMark->new("name") or confess "$!";

The name can be obtained back from the mark:

    $name = $mark->getName();

Other than that, the frame marks are completely opaque, and can be used only for the loop scheduling. Not even the

    same()

method is supported for them at the moment, though it probably will be in the future. The mark gets set and used as:

    $unit->setMark($mark);
    $unit->loopAt($mark, @rowops);

The rowop arguments of the

    loopAt()

are the same as for the other enqueueing functions, and as for other functions they may happen to be trays. These methods also use the new error handling scheme, and will confess on errors. No need to check the results.

There also are the convenience methods that create the rowops from the field values and immediately enqueue them:

    $unit->makeHashCall($label, $opcode,
    	$fieldName => $fieldValue, ...);
    $unit->makeArrayCall($label, $opcode, @fieldValues);

    $unit->makeHashSchedule($label, $opcode,
    	$fieldName => $fieldValue, ...);
    $unit->makeArraySchedule($label, $opcode, @fieldValues);

    $unit->makeHashLoopAt($mark, $label, $opcode,
    	$fieldName => $fieldValue, ...);
    $unit->makeArrayLoopAt($mark, $label, $opcode, @fieldValues);

These are essentially the shorter ways to make the rowops and enqueue them without the three-deep calls. Only the methods for the most frequently used enqueueing modes are provided, not for all of them. All these methods also confess on errors.

The methods for creation of labels have been already discussed in &xrsp;. Here is their recap along with the similar methods for creation of tables and trays that will be discussed later:

    $label = $unit->makeDummyLabel($rowType, "name")
    	or confess "$!";

    $label = $unit->makeLabel($rowType, "name",
    	$clearSub, $execSub, @args) or confess "$!";

    $label = $unit->makeClearingLabel("name", @args);

    $table = $unit->makeTable($tableType, $enqMode, "name")
    	or confess "$!";

    $tray = $unit->makeTray(@rowops) or confess "$!";

Of them

    makeClearingLabel()

uses the new error handling convention, confessing by itself, and the rest return an undef on errors that has to be checked. It's actually real difficult to make

    makeClearingLabel()

fail, only by corrupting some of the Triceps internal variables, and it was a late additiion, so going straight with the new convention for it made sense.

A special thing about the labels is that when a unit creates a label, it keeps a reference to it, for clearing. A label keeps a pointer back to the unit but not a reference (if you call

    getUnit()

on a label, the returned value becomes a reference). For a table or a tray, the unit doesn't keep a reference to them. Instead, they keep a reference to the unit. With the tables, it can get pretty involved: A table has labels associated with it. When a table is created, it also creates these labels. The unit keeps references of these labels. The table also keeps references of these labels. The table keeps a reference of the unit. The labels (they are at the &Cpp; level, not Perl level) have pointers to the unit and the table but not references, to avoid the reference cycles.

See more on the memory management and label clearing in the &xrsp;.

The convenience methods to create the whole front part of the topological loop are:

    ($labelBegin, $labelNext, $frameMark) = $unit->makeLoopHead(
    	$rowType, "name", $clearSub, $execSub, @args);

    ($labelBegin, $labelNext, $frameMark) = $unit->makeLoopAround(
    	"name", $labelFirst);

You don't have to use them, you can create the loops manually. These methods merely make it more convenient. Remember also that a procedural loop is usually much easier to write and debug and read later than a topological one.

These methods use the new error handling convention, confessing on the errors. There is no need to check the result.

    makeLoopHead()

creates the front part of the loop that starts with a Perl label. It gets the arguments for that label and creates it among the other things.

    makeLoopAround()

creates the front part of the loop around an existing label that will be the first one executed in the loop.

    makeLoopHead()

is really redundant and can be replaced with a combination of

    makeLabel()

and

    makeLoopAround()

.

They both return the same results, a triplet:

- The label where you send a rowop to initiate the loop (remember that the loop consists of one row going through the loop at a time),

      $labelBegin

  .
- The label that you use at the end of the loop in the

      loopAt()

  to do the next iteration of the loop,

      $labelNext

  .
- The frame mark that you use in

      loopAt()

  ,

      $frameMark

  . You don't need to set the frame mark, it will be set for you in the wrapper logic.

The name is used to construct the names of the elements by adding a dotted suffix: name.begin, name.next for

    makeLoopHead()

or name.wrapnext for

    makeLoopAround()

, name.mark.

    makeLoopAround()

takes the row type for its created labels from the first label that is given to it as an argument.

The makeLoop methods may be easier to understand if you look at their source code:

    sub makeLoopHead # ($self, $rt, $name, $clearSub, $execSub, @args)
    {
    	my ($self, $rt, $name, $clear, $exec, @args) = @_;

    	my $mark = Triceps::FrameMark->new($name . ".mark") or confess "$!";

    	my $lbNext = $self->makeLabel($rt, $name . ".next", $clear, sub {
    		$self->setMark($mark);
    		&$exec(@_);
    	}, @args) or confess "$!";

    	my $lbBegin = $self->makeLabel($rt, $name . ".begin", undef, sub {
    		$self->call($lbNext->adopt($_[1]));
    	}) or confess "$!";

    	return ($lbBegin, $lbNext, $mark);
    }

    sub makeLoopAround # ($self, $name, $lbFirst)
    {
    	my ($self, $name, $lbFirst) = @_;
    	my $rt = $lbFirst->getRowType();

    	my $mark = Triceps::FrameMark->new($name . ".mark") or confess "$!";

    	my $lbWrapNext = $self->makeLabel($rt, $name . ".wrapnext", undef, sub {
    		$self->setMark($mark);
    	}) or confess "$!";
    	$lbWrapNext->chain($lbFirst) or confess "$!";

    	my $lbBegin = $self->makeLabel($rt, $name . ".begin", undef, sub {
    		$self->call($lbWrapNext->adopt($_[1]));
    	}) or confess "$!";

    	return ($lbBegin, $lbWrapNext, $mark);
    }

The label execution handlers in them use

    $_[1]

to get their rowop argument, without assigning it to a variable first. An extended example of them will also be shown in &xrsp;.

The unit can be checked for the emptiness of its queues:

    $result = $unit->empty();

Also the current depth of the call stack (the number of the stack frames on the queue) can be found with:

    $result = $unit->getStackDepth();

It isn't of any use for the model logic as such but comes handy for debugging, to check in the loops that you haven't accidentally created a stack growing with iterations. When the unit is not running, the stack depth is 1, since the outermost frame always stays on the stack. When a rowop is being executed, the stack depth is at least 2.

The functions for execution from the queues are:

    $unit->callNext();
    $unit->drainFrame();

    callNext()

takes one label from the top (innermost) stack frame queue and calls it. If the innermost frame happens to be empty, it does nothing.

    drainFrame()

calls the rowops from the top stack frame until it becomes empty. This includes any rowops that may be created and enqueued as part of the execution of the previous rowops. But it doesn't pop the frame from the stack. And of course the method

    call()

causes the argument rowops to be executed immediately, without even being technically enqueued.
unit
enqueue
error handling
tray
constants
frame mark
memory management
scheduling loop

## Error handling during the execution

When the labels execute, they may produce errors in one of two ways:

- The Perl code in the label might die.
- The call topology might violate the rules.

The rules are basically that you can't make the recursive calls. A label may not make calls directly or through other labels to itself. The idea is to catch the call sequences that are likely to go into the deep recursion and overflow the stack. It catches them early, on the first attempt of recursion. If you need to do the recursion, use

    schedule()

or

    loopAt()

. That way you avoid overrunning the stack.

Whichever way the error is detected, it causes the Triceps call stack to be unwound. The Perl error messages from

    die

or

    confess

and the &Cpp; tracing of rowop calls and label chainings get combined into a common stack trace. When the code gets back to Perl, the XS code triggers a

    confess

. If that happens to be in the handler of another rowop, it continues the Triceps hybrid stack unwinding. If not caught by

    eval

, it keeps going to the topmost

    call()

or

    drainFrame()

and causes the whole program to die. Which is a reasonable reaction most of the time.

Remember, the root cause is a serious error that is likely to leave the model in an inconsistent state, and it should usually be considered as fatal. If you want to catch the errors, nip them in the bud by wrapping your Perl code in

    eval

. Then you can handle the errors before thay have a chance to propagate.

An interesting question is, what happens to the rowops that were in the Triceps stack frames when the stack gets unwound? They get thrown away. The memory gets collected thanks to the reference counting, but the rowops and their sequence order get thrown away. The reason is basically that there may be no catching of the errors until unwinding to the

    drainFrame()

. The choice is to either throw away everything after the first error or keep trying to execute the following rowops, collecting the errors. And that might become a lot of errors. I've taken the choice of stopping as early as possible, because the state of the model will probably be corrupted anyway and nothing but garbage would be coming out (if anything would be coming at all and not be stuck in an endless loop).
error handling
XS
draining
stack unwinding

## The main loop

The execution unit doesn't magically process the data by itself. The data needs to be pushed into it, and the unit has to be told to process it. There has to be some internal code to drive it, that would continuously read the data, schedule, drain.

A typical way of processing the incoming rowops in a loop is:

    $stop = 0;
    while (!$stop) {
    	$rowop = &readRowop(); # some user-defined function
    	$unit->call($rowop);
    	$unit->drainFrame();
    }

The rowops coming from the outside get executed as they are received, and then any rows left over from them get handled by

    drainFrame()

before the next incoming rowop is read. Some of the executed rowops may set

    $stop

, and the main loop will exit.

There is also another version of the main loop that has been more historic:

    $stop = 0;
    while (!$stop) {
    	$rowop = &readRowop(); # some user-defined function
    	$unit->schedule($rowop);
    	$unit->drainFrame();
    }

It uses

    schedule()

instead of

    call()

for the rowop. As long as only one rowop is scheduled before draining the frame, both versions work equally well. But if you schedule multiple rowops before draining the frame, you can introduce a subtle unpredictability in the execution order. It is described in detail in &xrsp;. Actually, you can have the same problem if you don't drain the frame after each top-level

    call()

too. But mentally

    call()

kind of reminds better to feed the rowops one at a time, and also is slightly more efficient, so now I prefer the version with it.

Many of the examples in this manual use the main loop along the following lines (with variations, to fit the examples, and as the main loop was refined over time):

    while(&readLine) {
    	chomp;
    	my @data = split(/,/); # starts with a command, then string opcode
    	my $type = shift @data;
    	if ($type eq "lbCur") {
    		$unit->makeArrayCall($lbCur, @data);
    	} elsif ($type eq "lbPos") {
    		$unit->makeArrayCall($lbPos, @data);
    	}
    	$unit->drainFrame();
    }

It reads the CSV (Comma-Separated Values) data from stdin, with the label name in the first column, the opcode in the second, and the data fields in the rest. Then dispatches according to the label. Doing a

    call()

instead of

    schedule()

works just as well, and the following

    drainFrame()

takes care of any rowops scheduled from the call.

Many variations are possible. It can be generalized to look up the labels from the hash:

    while(&readLine) {
    	chomp;
    	my @data = split(/,/); # starts with a command, then string opcode
    	my $type = shift @data;
    	$unit->makeArrayCall($labels{$type}, @data);
    	$unit->drainFrame();
    }

Or call the procedural functions for some types:

    while(&readLine) {
    	chomp;
    	my @data = split(/,/); # starts with a command, then string opcode
    	my $type = shift @data;
    	if ($type eq "lbCur") {
    		$unit->makeArrayCall($lbCur, @data);
    	} elsif ($type eq "lbPos") {
    		$unit->makeArrayCall($lbPos, @data);
    	} elsif ($type eq "clear") { # clear the previous day
    		&clearByDate($tPosition, @data);
    	}
    	$unit->drainFrame();
    }

Though none of these small examples are production-ready. At the very least, their parsing of the CSV data is primitive. It can't handle the quoting properly and can't parse the data with commas in it.

A better ready way to parse the data will be provided in the future. For now, make your own.

main loop

## Main loop with a socket

A fairly typical situation is when a CEP model has to run in a daemon process, receiving and sending data through the network sockets. Here goes an example that does this. It's not production-ready either. It still has the issue with the parsing of the CSV data, its handling of the errors is not well-tested, and it makes a few simplifying assumptions about the buffering (more on this below). Other than that, it's a decent starting point. If you want to copy-and-paste this code for your experiments, it can be found in

    perl/Triceps/t/xQuery.t

.

    use Triceps;
    use Carp;
    use Errno qw(EINTR EAGAIN);
    use IO::Poll qw(POLLIN POLLOUT POLLHUP);
    use IO::Socket;
    use IO::Socket::INET;

    # the socket and buffering control for the main loop;
    # they are all indexed by a unique id
    our %clients; # client sockets
    our %inbufs; # input buffers, collecting the whole lines
    our %outbufs; # output buffers
    our $poll; # the poll object
    our $cur_cli; # the id of the current client being processed
    our $srv_exit; # exit when all the client connections are closed

    # writing to the output buffers
    sub outBuf # ($id, $string)
    {
    	my $id = shift;
    	my $line = shift;
    	$outbufs{$id} .= $line;
    	# If there is anything to write on a buffer, stop reading from it.
    	$poll->mask($clients{$id} => POLLOUT);
    }

    sub outCurBuf # ($string)
    {
    	outBuf($cur_cli, @_);
    }

    sub closeClient # ($id, $h)
    {
    	my $id = shift;
    	my $h = shift;
    	$poll->mask($h, 0);
    	$h->close();
    	delete $clients{$id}; # OK perl Perl manual even when iterating
    	delete $inbufs{$id};
    	delete $outbufs{$id};
    }

    # The server main loop. Runs with the specified server socket.
    # Uses the labels hash to send the incoming data to Triceps.
    sub mainLoop # ($srvsock, $%labels)
    {
    	my $srvsock = shift;
    	my $labels = shift;

    	my $client_id = 0; # unique strings
    	our $poll = IO::Poll->new();

    	$srvsock->blocking(0);
    	$poll->mask($srvsock => POLLIN);
    	$srv_exit = 0;

    	while(!$srv_exit || keys %clients != 0) {
    		my $r = $poll->poll();
    		confess "poll failed: $!" if ($r < 0 && ! $!{EAGAIN} && ! $!{EINTR});

    		if ($poll->events($srvsock)) {
    			while(1) {
    				my $client = $srvsock->accept();
    				if (defined $client) {
    					$client->blocking(0);
    					$clients{++$client_id} = $client;
    					# &send("Accepted client $client_id\n");
    					$poll->mask($client => (POLLIN|POLLHUP));
    				} elsif($!{EAGAIN} || $!{EINTR}) {
    					last;
    				} else {
    					confess "accept failed: $!";
    				}
    			}
    		}

    		my ($id, $h, $mask, $n, $s);
    		while (($id, $h) = each %clients) {
    			$cur_cli = $id;
    			$mask = $poll->events($h);
    			if (($mask & POLLHUP) && !defined $outbufs{$id}) {
    				# &send("Lost client $client_id\n");
    				closeClient($id, $h);
    				next;
    			}
    			if ($mask & POLLOUT) {
    				$s = $outbufs{$id};
    				$n = $h->syswrite($s);
    				if (defined $n) {
    					if ($n >= length($s)) {
    						delete $outbufs{$id};
    						# now can accept more input
    						$poll->mask($h => (POLLIN|POLLHUP));
    					} else {
    						substr($outbufs{$id}, 0, $n) = '';
    					}
    				} elsif(! $!{EAGAIN} && ! $!{EINTR}) {
    					warn "write to client $id failed: $!";
    					closeClient($id, $h);
    					next;
    				}
    			}
    			if ($mask & POLLIN) {
    				$n = $h->sysread($s, 10000);
    				if ($n == 0) {
    					# &send("Lost client $client_id\n");
    					closeClient($id, $h);
    					next;
    				} elsif ($n > 0) {
    					$inbufs{$id} .= $s;
    				} elsif(! $!{EAGAIN} && ! $!{EINTR}) {
    					warn "read from client $id failed: $!";
    					closeClient($id, $h);
    					next;
    				}
    			}
    			# The way this works, if there is no '\n' before EOF,
    			# the last line won't be processed.
    			# Also, the whole output for all the input will be buffered
    			# before it can be sent.
    			while($inbufs{$id} =~ s/^(.*)\n//) {
    				my $line = $1;
    				chomp $line;
    				local $/ = "\r"; # take care of a possible CR-LF
    				chomp $line;
    				my @data = split(/,/, $line);
    				my $lname = shift @data;
    				my $label = $labels->{$lname};
    				if (defined $label) {
    					my $unit = $label->getUnit();
    					confess "label '$lname' received from client $id has been cleared"
    						unless defined $unit;
    					eval {
    						$unit->makeArrayCall($label, @data);
    						$unit->drainFrame();
    					};
    					warn "input data error: $@\nfrom data: $line\n" if $@;
    				} else {
    					warn "unknown label '$lname' received from client $id: $line "
    				}
    			}
    		}
    	}
    }

The general outline follows the single-threaded multiplexing server described in .

    mainLoop()

gets the server socket and a dispatch table of labels as its arguments. It then proceeds with waiting for connections.

Once a connection is received, it gets added to the set of active connections, to get included in the waiting for the input data. The input data is read as simplified CSV (no commas in the middle of values, and no way to reprsent the NULL values othar than for those omitted at the end of the line). It's expected to have the format:

    name,opcode,data...

Such as:

    window,OP_INSERT,5,AAA,30,30
    window.query,OP_INSERT
    exit,OP_NOP

The name part is then used to find a label in the dispatch table. The rest of the data is used to create a rowop for that label and execute it.

The data is sent back to the client through buffering. To send some data to a client, use

    &outBuf($id, $text);

The

    $id

is the unique id of the client. How do you find, what is the id of the client you want to send the data to? When an input line is processed, the main loop knows, from what client it was received. It puts the id of that client in the global variable

    $cur_cli

. You can take it from there and remember. If you want to reply to the current client, you don't need to bother yourself with the id at all, just call

    &outCurBuf($text);

If you remember an id for the future use, you'd probably need to check that the client hasn't disconnected before sending data to it:

    if (exists $clients{$id}) {
    	&outBuf($id, $text);
    }

Otherwise your output attempts would be leaking memory in the output buffer. In any case, if a client has disconnected, the further processing of its requests shoudl usually be stopped. The client ids are not reused, so this check is always safe.

Once some output is buffered to send to a client, the further input from that client stops being accepted until the output buffer drains. But the processing in the Triceps unit scheduler keeps running until it runs out of things to do before it returns to the main loop. All this time the output buffer keeps collecting data without sending it to the client. Also, the input buffer might happen to already contain multiple lines. Then all these lines will be processed before the data from the output buffer starts being sent to the client. If a request produces a large amount of data, all this data will be buffered first. It's a simplification but really the commercial CEP systems aren't doing a whole lot better: when asked for the contents of a table/window/materliaized view, Coral8 and Aleri and Sybase (don't know about StreamBase but it might be not different either) would make a copy of it first before sending the data. In some cases the copy is more efficient because it references the rows rather than copying the whole byte data, but in the grand scheme of things it's all the same.

Internally the information about the client sockets and their buffers is kept in the global hashes

    %clients

,

    %inbufs

,

    %outbufs

. It could be done a a single hash of objects but this was simpler.

The loop exits when the global variable

    $srv_exit

gets set (synchronously, i.e. by one of the label handlers) to 1 and all the clients disconnect. The requirement for disconnection of all the clients makes sure that all the output buffers get flushed before exit, and that was the easiest way to achieve this goal.

    mainLoop()

relies on the listening socket being already created, bound and given to it as a parameter. Here is a function used in the examples to create this socket and run the server in a separate process:

    sub startServer # ($labels)
    {
    	my $labels = shift;

    	my $srvsock = IO::Socket::INET->new(
    		Proto => "tcp",
    		LocalPort => 0,
    		Listen => 10,
    	) or confess "socket failed: $!";
    	my $port = $srvsock->sockport() or confess "sockport failed: $!";
    	my $pid = fork();
    	confess "fork failed: $!" unless defined $pid;
    	if ($pid) {
    		# parent
    		$srvsock->close();
    	} else {
    		# child
    		&mainLoop($srvsock, $labels);
    		exit(0);
    	}
    	return ($port, $pid);
    }

It binds the socket to the port 0 to request that the OS bind it to a randum unused port. The port number is then read back with

    sockport()

. The pair of the port numer and the server's child process id is then returned as the result. The process where the server runs is in this case just a child process, it's not properly daemonized.

For a simple complete example, let's make an echo server that would print back the rows it receives:

    my $uEcho = Triceps::Unit->new("uEcho");
    my $lbEcho = $uEcho->makeLabel($rtTrade, "echo", undef, sub {
    	&outCurBuf($_[1]->printP() . "\n");
    });
    my $lbEcho2 = $uEcho->makeLabel($rtTrade, "echo2", undef, sub {
    	&outCurBuf(join(",", "echo", &Triceps::opcodeString($_[1]->getOpcode()),
    		$_[1]->getRow()->toArray()) . "\n");
    });
    my $lbExit = $uEcho->makeLabel($rtTrade, "exit", undef, sub {
    	$srv_exit = 1;
    });

    my %dispatch;
    $dispatch{"echo"} = $lbEcho;
    $dispatch{"echo2"} = $lbEcho2;
    $dispatch{"exit"} = $lbExit;

    my ($port, $pid) = &startServer(\%dispatch);
    print STDERR "port=$port pid=$pid\n";
    waitpid($pid, 0);
    exit(0);

It starts the server and waits for it to exit.

    waitpid()

is used here in a simplified way too, it should properly be done in a loop until it succeeds or an error other than

    EINTR

is returned.

    $rt

is the row type for the expected data. It's not particularly important here, so I didn't show its definition. Two labels, echo and echo2 differ in the way they print the data back: echo prints it in the symbolic form while echo2 prints in CSV. The label exit sets the exit flag.

The names in the dispatch table don't have to be the same as the names of the labels. It's often convenient to have them the same but not mandatory.

main loop
socket
dispatch table
Aleri
Coral8
Sybase
StreamBase
dispatch table

## Example of a topological loop

How to build the models with the topological loop is much easier to understand with an example. So let's make an example that computes the Fibonacci numbers. It's a real overcomplicated and perverse way of calculating the Fibonacci numbers. But it also is a great fit to the type of problems that get solved with the topological loop, one of a simple kind.

First, a quick reminder of what is a Fibonacci number. Historically it's a solution to the problem of breeding the spherical rabbits in vacuum. But in the mathematical reality it's the sequence of numbers where each number is a sum of the two previous ones. Two initial elements are defined to be equal to 1, and it goes from there:

Fi = Fi-1 + Fi-2

F1 = 1; F2 = 1

The Fibonacci numbers are often used as an example of recursive computations in the beginner's books on programming. The computation of the n-th Fibonacci number is usually shown computed like this:

    sub fib1 # ($n)
    {
    	my $n = shift;
    	if ($n <= 2) {
    		return 1;
    	} else {
    		return &fib1($n-1) + &fib1($n-2);
    	}
    }

However that's not a good way to compute in the real world. When a function calls itself recursively once, its complexity is linear, O(n). When a function calls itself twice or more, its complexity becomes exponential, O(en). At first you might think that it's only quadratic O(n 2) because it forks two ways on each step. But these two ways keep forking and forking on each step, and it compounds to exponential. Which is a real bad thing.

To think of it, it's a huge waste, since the (n-2)&emdash;th number is calculated anyway for the (n-1)&emdash;th number. Why calculate it separately the second time? We could as well have saved and reused it. The Lisp people have figured this out a long time ago, and the Lisp books (if you can read Finnish or Russian, is a classical one) are full of examples that do exactly that. However I'm too lazy to explain how they work, so we're going to skip it together with the conversion of a tail recursion into a loop and get directly to the loop version. I find the loop version more natural and easier to write than a recursion anyway.

    sub fibStep2 # ($prev, $preprev)
    {
    	return ($_[0] + $_[1], $_[0]);
    }

    sub fib2 # ($n)
    {
    	my $n = shift;
    	my @prev = (1, 0); # n and n-1

    	while ($n > 1) {
    		@prev = &fibStep2(@prev);
    		$n--;
    	}
    	return $prev[0];
    }

The split into two functions is not mandatory for the loop version, it just does the clean separation of the loop counter logic and of the computation of the next step of the function. (But for the recursion version if would be mandatory).

I'm going to take this procedural loop version and transform it into a topological loop. It actually happens to be a real good match for the topological loop. In a topological loop a record keeps traveling through it and being transformed until it satisfies the loop exit condition. Here

    @prev

is the record contents, and the iteration count will be added to them to keep track of the exit condition.

    $uFib = Triceps::Unit->new("uFib") or confess "$!";

    my $rtFib = Triceps::RowType->new(
    	iter => "int32", # iteration number
    	cur => "int64", # current number
    	prev => "int64", # previous number
    ) or confess "$!";

    my $lbPrint = $uFib->makeLabel($rtFib, "Print", undef, sub {
    	&send($_[1]->getRow()->get("cur"));
    });

    my $lbCompute; # will fill in later

    my ($lbBegin, $lbNext, $markFib) = $uFib->makeLoopHead(
    	$rtFib, "Fib", undef, sub {
    		my $iter = $_[1]->getRow()->get("iter");
    		if ($iter <= 1) {
    			$uFib->call($lbPrint->adopt($_[1]));
    		} else {
    			$uFib->call($lbCompute->adopt($_[1]));
    		}
    	}
    );

    $lbCompute = $uFib->makeLabel($rtFib, "Compute", undef, sub {
    	my $row = $_[1]->getRow();
    	my $cur = $row->get("cur");
    	$uFib->makeHashLoopAt($markFib, $lbNext, $_[1]->getOpcode(),
    		iter => $row->get("iter") - 1,
    		cur => $cur + $row->get("prev"),
    		prev => $cur,
    	);
    }) or confess "$!";

    my $lbMain = $uFib->makeLabel($rtFib, "Main", undef, sub {
    	my $row = $_[1]->getRow();
    	$uFib->makeHashCall($lbBegin, $_[1]->getOpcode(),
    		iter => $row->get("iter"),
    		cur => 1,
    		prev => 0,
    	);
    	&send(" is a Fibonacci number ", $row->get("iter"), "\n");
    }) or confess "$!";

    while(&readLine) {
    	chomp;
    	my @data = split(/,/);
    	$uFib->makeArrayCall($lbMain, @data);
    	$uFib->drainFrame(); # just in case, for completeness
    }

You can see that it has grown quite a bit. That's why the procedural loops are generally a better idea. However if the computation involves a lot of the SQLy logic, the topological loops are still beneficial. Also, the Triceps

    call()

is at the moment not a real call: you pass the arguments, you call the code but you don't directly get the results. You could pass them back through some static variables, or you could use the topological loop to pass them directly to the next iteration of the loop.

The main loop reads the CSV lines with opcodes (which aren't really used here, just passed through and then thrown away before printing) and calls

    $lbMain

. Here is an example of an input and output as they would intermix if the input was typed from the keyboard. As in the rest of this manual, the input lines are shown in bold. > OP_INSERT,1 1 is a Fibonacci number 1 > OP_DELETE,2 1 is a Fibonacci number 2 > OP_INSERT,5 5 is a Fibonacci number 5 > OP_INSERT,6 8 is a Fibonacci number 6

The input lines contain the values only for the field

    iter

, which intentionally happens to be the first field in the row type. The other fields will be reset anyway in

    $lbMain

, so they are left as NULL.

The point of

    $lbMain

is to call the loop begin label

    $lbBegin

and then print the message about which Fibonacci number was requested. The value of the computed number is printed at the end of the loop, so when the words is a Fibonacci number are printed after it, that demonstrates that the execution of

    $lbMain

continues only after the loop is completed.

The loop logic is split into two labels

    $lbNext

and

    $lbCompute

purely to show that it can be split like this.

    $lbNext

handles the loop termination condition, and

    $lbCompute

does essentially the work of

    fibStep2()

. After the loop terminates, it passes the result row to

    $lbPrint

for the priniting of the value.

When the code for

    $lbNext

is created, it contains the call of

    $lbCompute

. However the label

    $lbCompute

has not been created at this time yet! Not a problem, creating in advance an empty variable

    $lbCompute

is enough. The closure in

    $lbNext

will keep a reference to that variable, and the variable will be filled with the reference to the label later (but before the main loop executes).

And here is the version with

    makeLoopAround()

:

    my ($lbBegin, $lbNext, $markFib); # will fill in later

    $lbCompute = $uFib->makeLabel($rtFib, "Compute", undef, sub {
    	my $row = $_[1]->getRow();
    	my $cur = $row->get("cur");
    	my $iter = $row->get("iter");
    	if ($iter <= 1) {
    		$uFib->call($lbPrint->adopt($_[1]));
    	} else {
    		$uFib->makeHashLoopAt($markFib, $lbNext, $_[1]->getOpcode(),
    			iter => $row->get("iter") - 1,
    			cur => $cur + $row->get("prev"),
    			prev => $cur,
    		);
    	}
    }) or confess "$!";

    ($lbBegin, $lbNext, $markFib) = $uFib->makeLoopAround(
    	"Fib", $lbCompute
    );

The unit, row type,

    $lbPrint

,

    $lbMain

and the main loop have stayed the same, so they are omitted from this example. The whole loop logic, both the termination condition and the computation step, have been collected into one label

    $lbCompute

, to show that it can be done this way too. Then the loop head is created around

    $lbCompute

.
topological loop
scheduling loop

## Issues with the Triceps scheduling

As much as I like it, the Triceps scheduling is not perfect, and has some open issues at the moment. Some of them have been already mentioned in the description of the loop scheduling: it's a bit confusing that the frame mark is placed on the next outer scheduling stack frame and not on the current one. This leads to the interesting effects in execution order.

The other one has been mentioned in the main loop discussion: the

    schedule()

call, when used from inside the scheduled code, may introduce unpredictability in the execution order. It puts the rowop after the last rowop in the outermost stack frame. But the outermost stack frame may contain a whole queue of rowops that come from the outside. This means that the exact order of execution will depend on the timing of the rowops arriving from outside.

Let me demonstrate it with an example. Suppose the main loop tries to optimize by collecting and schedulng as many incoming rowops as it can before running them:

    $stop = 0;
    while (!$stop) {
    	&waitForIncomingData(); # some user-defined function
    	while ($rowop = &readRowop()) { # some user-defined function
    		$unit->schedule($rowop);
    	}
    	$unit->drainFrame();
    }

Suppose the rowops A, B, C, D are being received from the outside. When the rowop A executes, it schedules the rowop E. Then depending on the timing of the packets in the network, the call sequence may be

    schedule(A)
    drainFrame()
    schedule(B)
    drainFrame()
    schedule(C)
    drainFrame()
    schedule(D)
    drainFrame()

or

    schedule(A)
    schedule(B)
    drainFrame()
    schedule(C)
    schedule(D)
    drainFrame()

or

    schedule(A)
    schedule(B)
    schedule(C)
    schedule(D)
    drainFrame()

or a few other combinations. In the first case the actual execution order will be A, E, B, C, D. That's because when A schedules E, E will be picked up and executed by the first following frame drain. In the second case the execution order will be A, B, E, C, D. Here when E gets scheduled, B is already on the queue in front of it. In the third case the order will be A, B, C, D, E. And it will fluctuate at random between the runs.

If the repeatable execution order is important (and usually it is), the solution if to feed the rowops one by one and drain the frame right afterwards. Then the execution order will always be A, E, B, C, D. When feeding one by one,

    call()

can be used instead of

    schedule()

, and even slightly more efficient. Just don't forget to drain the frame after each call.

The same issue happens with the topological loops that have been temporarily stopped and then resumed on arrival of more data from outside. The mark of such a loop will be unset when the loop continues, and looping at this mark will be equivalent to

    schedule()

, having the same repeatability problem. The same solution works for this issue too.

The method

    fork()

is not exactly useful. It was created when I've thought that it's the solution to the problem of the loops. Which it has turned out to not solve, and another solution had to be devised. Now it really doesn't have much use, and will probably be removed in the future.

I have a few ideas for better solutions of these issues, but they will need a bit more experimentation. Just keep in mind that the scheduling will be refined in the future. It will still have the same general shape but differ in detail.

main loop
scheduling loop

## Trays, or yes bundling

Even though Triceps does no bundling in scheduling, there still is a need to store the sequences of row operations. This is an important distinction, since the stored sequences are to be scheduled somewhere in the future (or maybe not scheduled at all, but iterated through manually). If and when they get scheduled, they will be unbundled. The ordered storage only provides the order for that future scheduling or iteration.

The easiest way to store rowops is to put them into the Perl arrays, like:

    my @ops = ($rowop1, $rowop2);
    push @ops, $rowop3;

However the &Cpp; internals of Triceps do not know about the Perl arrays. And some of them can work directly with the sequences of rowops. So Triceps defines an internal sort-of-equivalent of Perl array for rowops, called a Tray.

The trays have first been used to catch the side effects of operations on the stateful elements, so the name tray came from the metaphor put a tray under it to catch the drippings.

The trays get created as:

    $tray = $unit->makeTray(@rowops) or confess "$!";

A tray always stores rowops for only one unit. It can be only used in one thread. A tray can be used in all the enqueueing methods, just like the direct rowops:

    $unit->call($tray);
    $unit->fork($tray);
    $unit->schedule($tray);
    $unit->enqueue($mode, $tray);
    $unit->loopAt($mark, $tray);

Moreover, multiple trays may be passed, and the loose rowops and trays can be mixed in the multiple arguments of these functions, for example:

    $unit->call($rowopStartPkg, $tray, $rowopEndPkg);

In this example nothing really stops you from placing the start and end rowops into the tray too. A tray may contain the rowops of any types mixed in any order. This is by design, and it's an important feature that allows to build the protocol blocks out of rowops and perform an orderly data exchange. This feature is an absolute necessity for proper inter-process and inter-thread communication.

The ability to send the rows of multiple types through the same channel in order is a must, and its lack makes the communication with some other CEP systems exceedingly difficult. Coral8 supports only one stream per connection. Aleri (and I believe Sybase R5) allows to send multiple streams through the same connection but has no guarantees of order between them. I don't know about the others, check yourself.

To iterate on a tray in the Perl code, it can be converted to a Perl array:

    @array = $tray->toArray();

The size of the tray (the count of rowops in it) can be read directly without a conversion, and the unit can be read back too:

    $size = $tray->size();
    $traysUnit = $tray->getUnit();

Another way to create a tray is by copying an existing one:

    $tray2 = $tray1->copy();

This copies the contents (which is the references to the rowops) and does not create any ties between the trays. The copying is really just a more efficient way to do

    $tray2 = $tray1->getUnit()->makeTray($tray1->toArray());

The tray references can be compared for whether they point to the same tray object:

    $result = $tray1->same($tray2);

The contents of a tray may be cleared. Which is more convenient and more efficient than discarding a tray and creating another one:

    $tray->clear();

The data may be added to the back of a tray:

    $tray->push(@rowops);

Multiple rowops can be pushed in a single call. There are no other Perl-like operations on a tray: it's either create from a set of rowops, push, or convert to a Perl array.

Note that the trays are mutable, unlike the rows and rowops. Multiple references to a tray will see the same contents. If a tray is changed through one reference, the others will see the changes too.

bundling
tray
protocol
Coral8
Aleri
Sybase

## Tracing the execution

When developing the CEP models, there always comes the question: WTF had just happened? How did it manage get this result? Followed by subscribing to many intermediate results and trying to piece together the execution order.

Triceps provides two solutions for this situation: First, the procedural approach should make the logic much easier to follow. Second, it has a ready way to trace the execution and then read the trace in one piece. It can also be used to analyze any variables on the fly, and possibly stop the execution and enter some manual mode.

The idea here is simple: provide the Unit with a method that will be called:

- before a label executes,
- after the label executes but before draining its frame,
- after the frame is drained but before the chained labels execute,
- after all the execution caused by the label is completed.

For the simple tracing, there is a small simple tracer provided. It actually executes directly as compiled in &Cpp; so it's quite efficient:

    $tracer = Triceps::UnitTracerStringName(option => $value, ...)
    	or confess "$!";

The arguments are specified as the option name-value pairs.

The only option supported is

    verbose

, which may be 0 (default) or non-0. If it's 0 (false), the tracer will record a message only before executing each label. If true, it will record a message after each stage. The class is named UnitTracerStringName because it records the execution trace in the string format, including the names of the labels. The tracer is set into the unit:

    $unit->setTracer($tracer);

The unit's current tracer can also be read back:

    $oldTracer = $unit->getTracer();

If no tracer was previously set,

    getTracer()

will return

    undef

. And

    undef

can also be used as an argument of

    setTracer()

, to cancel any previously set tracing.

    setTracer()

has the new-style error handling and confesses on errors.

The tracer references can be compared for whether they refer to the same underlying object:

    $result = $tracer1->same($tracer2);

There are multiple kinds of tracer objects, and

    same()

can be called safely for either kind of tracer, including mixing them together. Of course, the tracers of different kinds definitely would not be the same tracer object.

As the unit runs, the tracing information gets collected in the tracer object. It can be extracted back with:

    $data = $tracer->print();

This does not reset the trace. To reset it, use:

    $tracer->clearBuffer();

Here is a code sequence designed to produce a fairly involved trace:

    $sntr = Triceps::UnitTracerStringName->new(verbose => 1);
    $u1->setTracer($sntr);

    $c_lab1 = $u1->makeDummyLabel($rt1, "lab1")
    	or confess "$!";
    $c_lab2 = $u1->makeDummyLabel($rt1, "lab2")
    	or confess "$!";
    $c_lab3 = $u1->makeDummyLabel($rt1, "lab3")
    	or confess "$!";

    $c_op1 = $c_lab1->makeRowop(&Triceps::OP_INSERT, $row1)
    	or confess "$!";
    $c_op2 = $c_lab1->makeRowop(&Triceps::OP_DELETE, $row1)
    	or confess "$!";

    $v = $c_lab1->chain($c_lab2) or confess "$!";
    $v = $c_lab1->chain($c_lab3) or confess "$!";
    $v = $c_lab2->chain($c_lab3) or confess "$!";

    $u1->schedule($c_op1);
    $u1->schedule($c_op2);

    $u1->drainFrame();

The trace is:

unit 'u1' before label 'lab1' op OP_INSERT { unit 'u1' drain label 'lab1' op OP_INSERT unit 'u1' before-chained label 'lab1' op OP_INSERT unit 'u1' before label 'lab2' (chain 'lab1') op OP_INSERT { unit 'u1' drain label 'lab2' (chain 'lab1') op OP_INSERT unit 'u1' before-chained label 'lab2' (chain 'lab1') op OP_INSERT unit 'u1' before label 'lab3' (chain 'lab2') op OP_INSERT { unit 'u1' drain label 'lab3' (chain 'lab2') op OP_INSERT unit 'u1' after label 'lab3' (chain 'lab2') op OP_INSERT } unit 'u1' after label 'lab2' (chain 'lab1') op OP_INSERT } unit 'u1' before label 'lab3' (chain 'lab1') op OP_INSERT { unit 'u1' drain label 'lab3' (chain 'lab1') op OP_INSERT unit 'u1' after label 'lab3' (chain 'lab1') op OP_INSERT } unit 'u1' after label 'lab1' op OP_INSERT } unit 'u1' before label 'lab1' op OP_DELETE { unit 'u1' drain label 'lab1' op OP_DELETE unit 'u1' before-chained label 'lab1' op OP_DELETE unit 'u1' before label 'lab2' (chain 'lab1') op OP_DELETE { unit 'u1' drain label 'lab2' (chain 'lab1') op OP_DELETE unit 'u1' before-chained label 'lab2' (chain 'lab1') op OP_DELETE unit 'u1' before label 'lab3' (chain 'lab2') op OP_DELETE { unit 'u1' drain label 'lab3' (chain 'lab2') op OP_DELETE unit 'u1' after label 'lab3' (chain 'lab2') op OP_DELETE } unit 'u1' after label 'lab2' (chain 'lab1') op OP_DELETE } unit 'u1' before label 'lab3' (chain 'lab1') op OP_DELETE { unit 'u1' drain label 'lab3' (chain 'lab1') op OP_DELETE unit 'u1' after label 'lab3' (chain 'lab1') op OP_DELETE } unit 'u1' after label 'lab1' op OP_DELETE }

In non-verbose mode the same trace would be:

unit 'u1' before label 'lab1' op OP_INSERT unit 'u1' before label 'lab2' (chain 'lab1') op OP_INSERT unit 'u1' before label 'lab3' (chain 'lab2') op OP_INSERT unit 'u1' before label 'lab3' (chain 'lab1') op OP_INSERT unit 'u1' before label 'lab1' op OP_DELETE unit 'u1' before label 'lab2' (chain 'lab1') op OP_DELETE unit 'u1' before label 'lab3' (chain 'lab2') op OP_DELETE unit 'u1' before label 'lab3' (chain 'lab1') op OP_DELETE

The verbose trace has the before and after lines marked with the curly braces, so that when it gets loaded into an editor, it can be navigated relatively easily.

The actual contents of the rows is not printed in either case. This is basically because the tracer is implemented in &Cpp;, and I've been trying to keep the knowledge of the meaning of the simple data types out of the &Cpp; code as much as possible for now. But it can be implemented with a Perl tracer.

A Perl tracer is created with:

    $tracer = Triceps::UnitTracerPerl->new($sub, @args)
    	or confess "$!";

The arguments are a reference to a function, and optionally arguments for it. The resulting tracer can be used in the unit's

    setTracer()

as usual.

The function of the Perl tracer gets called as:

    &$sub($unit, $label, $fromLabel, $rowop, $when, @args)

The arguments are:

-     $unit

  is the usual unit reference.

-     $label

  is the current label being traced.

-     $fromLabel

  is the parent label in the chaining (would be

      undef

  if the current label is called directly, without chaining from anything).

-     $rowop

  is the current row operation.

-     $when

  is an integer constant showing the point when the tracer is being called. It's value may be one of

      &Triceps::TW_BEFORE

  ,

      &Triceps::TW_BEFORE_DRAIN

  ,

      &Triceps::TW_BEFORE_CHAINED

  ,

      &Triceps::TW_AFTER

  ; the prefix

      TW

  stands for tracer when.

-     @args

  are the extra arguments passed from the tracer creation.

The

    TW_*

constants can as usual be converted to and from strings with the calls

    $string = &Triceps::tracerWhenString($value);
    $value = &Triceps::stringTracerWhen($string);

There also are the conversion functions with strings more suitable for the human-readable messages: before, drain, before-chained, drain. These are actually the conversions used in the UnitTracerStringName. The functions for them are:

    $string = &Triceps::tracerWhenHumanString($value);
    $value = &Triceps::humanStringTracerWhen($string);

The Perl tracers allow to execute any arbitrary actions when tracing. They can act as breakpoints by looking for certain conditions and opening a debugging session when those are met.

For an example of a Perl tracer, let's start with a tracer function that works like UnitTracerStringName:

    sub tracerCb()
    {
    	my ($unit, $label, $from, $rop, $when, @extra) = @_;
    	our $history;

    	my $msg = "unit '" . $unit->getName() . "' "
    		. Triceps::tracerWhenHumanString($when) . " label '"
    		. $label->getName() . "' ";
    	if (defined $fromLabel) {
    		$msg .= "(chain '" . $fromLabel->getName() . "') ";
    	}
    	$msg .= "op " . Triceps::opcodeString($rop->getOpcode());
    	if ($when == &Triceps::TW_BEFORE) {
    		$msg .= " {";
    	} elsif ($when == &Triceps::TW_AFTER) {
    		$msg .= " }";
    	}
    	$msg .= "\n";
    	$history .= $msg;
    }

    undef $history;
    $ptr = Triceps::UnitTracerPerl->new(\&tracerCb);
    $u1->setTracer($ptr);

It's slightly different, in the way that it always produces the verbose trace, and that it collectes the trace in the global variable

    $history

. But the resulting text is the same as with UnitTracerStringName.

Now let's improve on it by printing the whole rowop contents too. In a proper way this advanced tracer would be defined as a class constructing the tracer objects. But to reduce the amount of code let's just make it a standalone function to be used with the Perl tracer constructor.

And for extra nicety let's make the result nicely indented, with two spaces per the indenting level. The indenting is actually not such a great idea: with the long sequences of the calls between the labels, the nesting levels would also be deep, and the output would be indented way off the right end of the screen. That's why it's not done in UnitTracerStringName (though it might be a good idea as an option). But for the small short examples it works well. The function would take 3 extra arguments:

- Verbosity, a boolean value.
- Reference to an array variable where to append the text of the trace. This is more flexible than the fixed

      $history

  . The array will contain the lines of the trace as its elements. And appending to an array should be more efficient than appending to the end of a potentially very long string.
- Reference to a scalar variable that would be used to keep the indenting level. The value of that variable will be updated as the tracing happens. Its initial value will determine the initial indenting level. Keeping the indenting is actually not easy because the indenting level can be changed for two reasons, the label chaining and the label calls. To get the logic working, one indenting level gets added before in advance, in front of the outermost trace lines. So, to make the outermost lines appear with no indenting, initialize this variable to -1 and not 0.

    sub traceStringRowop
    {
    	my ($unit, $label, $fromLabel, $rowop, $when,
    		$verbose, $rlog, $rnest) = @_;

    	if ($verbose) {
    		${$rnest}++ if ($when == &Triceps::TW_BEFORE);
    		${$rnest}-- if ($when == &Triceps::TW_AFTER);
    	} else {
    		return if ($when != &Triceps::TW_BEFORE);
    	}

    	my $msg =  "unit '" . $unit->getName() . "' "
    		. Triceps::tracerWhenHumanString($when) . " label '"
    		. $label->getName() . "' ";
    	if (defined $fromLabel) {
    		$msg .= "(chain '" . $fromLabel->getName() . "') ";
    	}
    	my $tail = "";
    	if ($when == &Triceps::TW_BEFORE) {
    		$tail = " {";
    	} elsif ($when == &Triceps::TW_AFTER) {
    		$tail = " }";
    	}
    	push (@{$rlog}, ("  " x ${$rnest}) . $msg . "op "
    		. $rowop->printP() . $tail);

    	if ($verbose) {
    		${$rnest}++ if ($when == &Triceps::TW_BEFORE);
    		${$rnest}-- if ($when == &Triceps::TW_AFTER);
    	}
    }

    undef @history;
    my $tnest =  -1; # keeps track of the tracing nesting level
    $ptr = Triceps::UnitTracerPerl->new(\&traceStringRowop, 1, \@history, \$tnest);
    $u1->setTracer($ptr);

For the same call sequence as before, the output will be as follows (I've tried to wrap the long lines in a logically consistent way but it still spoils the effect of indenting a bit):

unit 'u1' before label 'lab1' op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" { unit 'u1' drain label 'lab1' op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" unit 'u1' before-chained label 'lab1' op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" unit 'u1' before label 'lab2' (chain 'lab1') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" { unit 'u1' drain label 'lab2' (chain 'lab1') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" unit 'u1' before-chained label 'lab2' (chain 'lab1') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" unit 'u1' before label 'lab3' (chain 'lab2') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" { unit 'u1' drain label 'lab3' (chain 'lab2') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" unit 'u1' after label 'lab3' (chain 'lab2') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" } unit 'u1' after label 'lab2' (chain 'lab1') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" } unit 'u1' before label 'lab3' (chain 'lab1') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" { unit 'u1' drain label 'lab3' (chain 'lab1') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" unit 'u1' after label 'lab3' (chain 'lab1') op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" } unit 'u1' after label 'lab1' op lab1 OP_INSERT a="123" b="456" c="789" d="3.14" e="text" } unit 'u1' before label 'lab1' op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" { unit 'u1' drain label 'lab1' op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" unit 'u1' before-chained label 'lab1' op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" unit 'u1' before label 'lab2' (chain 'lab1') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" { unit 'u1' drain label 'lab2' (chain 'lab1') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" unit 'u1' before-chained label 'lab2' (chain 'lab1') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" unit 'u1' before label 'lab3' (chain 'lab2') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" { unit 'u1' drain label 'lab3' (chain 'lab2') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" unit 'u1' after label 'lab3' (chain 'lab2') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" } unit 'u1' after label 'lab2' (chain 'lab1') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" } unit 'u1' before label 'lab3' (chain 'lab1') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" { unit 'u1' drain label 'lab3' (chain 'lab1') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" unit 'u1' after label 'lab3' (chain 'lab1') op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" } unit 'u1' after label 'lab1' op lab1 OP_DELETE a="123" b="456" c="789" d="3.14" e="text" }

As mentioned before, each label produces two levels of indenting: one for everything after before, another one for the nested labels.

Eventually this tracing should become another standard class in Triceps.

tracing
unit
tracing
error handling
constants