# API Fundamentals

## Languages and layers

As mentioned before, at the moment Triceps provides the APIs in &Cpp; and Perl. They are similar but not quite the same, because the nature of the compiled and scripted languages is different. The &Cpp; API is more direct and expects discipline from the programmer: if some incorrect arguments are passed, everything might crash. The Perl API should never crash. It should detect any incorrect use and report an orderly error. Besides, the idioms of the scripted languages are different from the compiled languages, and different usages become convenient.

So far only the Perl API is documented in this manual. Its is considered the primary one for the end users, and also richer and easier to use. The &Cpp; API will be documented as well, just it didn't make the cut for the version 1.0. If you're interested in the &Cpp; API, read the Perl documentation first, to understand the ideas of Triceps, and then look in the source code. The &Cpp; classes have very extensive comments in the header files.

The Perl API is implemented in XS. Some people, may wonder, why not SWIG? SWIG would automatically export the API into many languages, not just Perl. The problem with SWIG is that it just maps the API one-to-one. And this doesn't work any good, it makes for some very ugly APIs with abilities to crash from the user code. Which then have to be wrapped into more scripting code before they become usable. So then why bother with SWIG, it's easier to just use the scripting language's native extension methods. Another benefit of the native XS support is the access to the correct memory management.

In general, I've tried to avoid the premature optimization. The idea is to get it working at all first, and then bother about working fast. Except for the cases when the need for optimization looked obvious, and the logic intertwined with the general design strongly ehough, that if done one way, would be difficult to change in the future. We'll see, if these obvious cases really turn out to be the obvious wins, or will they become a premature-optimization mess.

There is usually more than one way to do something in Triceps. It has been written in layers: There is the &Cpp; API layer on the bottom, then the Perl layer that closely parallels it, then more of the niceties built in Perl. There is more than one way to organize the manual, structuring it by features or by layers. Eventually I went in the order of the major features, discussing each one of them at various layers.

I've also tried to show, how these layers are built on top of each other and connected. Which might be too much detail for the first reading. If you feel that something is going over your head, just skim over it. It could be marked more clearly but I don't like this kind of marking. I hate the side-panels in the magazines. I like the text to flow smoothly and sequentially. I don't like the simplifications that distort the real meaning and add all kinds of confusion. I like having all the details I can get, and then I can skip over the ones that look too complicated (and read them again when they start making sense).

Also, a major goal of Triceps is the extendability. And the best way to learn how to extend it, is by looking up close at how it has already been extended.

C++
XS
SWIG

## Errors, deaths and confessions

In Perl, when a Triceps method detects an error, it has two ways of reporting it:

1. Set the error code in the special variable

       $!

   and return an

       undef

   . This is the traditional approach that is gradually replaced by the other one.
2. Set the error code in the special variable

       $@

   (and also

       $!

   , for compatibility with the other way) and tell the Perl interpreter to die. This is the new approach.

Currently most of the methods inplemented in &Cpp; through XS return the errors in the first way but some have been converted to the new second way. Changing all the code at once is a lot of work, so the code is being converted to the new way gradually. The methods implemented in Perl use the second way. The methods using the second way are marked in their documentation as such. I hesitate to give a whole list of them, because the list changes as more methods get converted to the new way of error handling.

When you use the methods of the first group, you need to check their return code. The common idiom is:

    my $table = $unit->makeTable($tabType, "EM_CALL", $name)
    	or confess "$!";

It checks the return value and dies with an error message.

    confess

is a nicer form of

    die

and comes from the module

    Carp

.

    Carp

is a standard part of modern Perl, so all you need to do is just say

    use Carp;

No need to download and install anything.

The problem with the simple

    die

is that it reports an error but prints only the location where it has been called, which may be ten layers deep inside a library, not a full stack trace. The functions in

    Carp

fix that.

    confess

is the most interesting one of them. It works just like

    die

but prints the whole stack trace.

The full description of Carp is available at . It has more functions, however I find the full stack trace the most helpful thing in any case.

The nice part abour writing

    or confess

is that the error message is fully controlled. If the error happens in a template, the template can produce a smarter message, telling what was wrong with it arguments on the high level, for example:

    my $table = $unit->makeTable($tabType, "EM_CALL", $name)
    	or confess "Query2: bad table type, table creation failed: $!";

But the problem is that you need to not forget writing

    or confess

after every call. Also, it's inconvenient when the result of one call is passed directly to another one, like:

    $self->{unit}->call(
    	$self->{resLabel}->makeRowop("OP_INSERT", $rh->getRow()));

Here

    call()

is a method with the second way of the error reporting, and it doesn't need

    or confess

. But

    makeRowop()

and

    getRow()

use the first way. Writing

    or confess

for each of them would work but would be quite tedious:

    $self->{unit}->call(
    	$self->{resLabel}->makeRowop(
    		"OP_INSERT", $rh->getRow() or confess "$!"
    	) or confess "$!"
    );

Because of this, the error checking in such nested calls usually gets skipped and the error manifests itself by the enveloping call dying with the XS error of bad argument type. Which doesn't print the stack trace and loses the information about the original error.

The new second way of error reporting is much better in this regard. It reports the error where and when it happens. It executes

    confess

directly, which includes the stack trace into the error message and then proceeds like classic

    die

. This includes the code implemented both in Perl and in &Cpp; XS. With some exceptions though: the errors detected by the code auto-generated in XS, such as bad call arguments, still report through the plain

    die

.

There are modules to make all the cases of

    die

work like

    confess

,

    Devel::SimpleTrace

and

    Carp::Always

. They work by intercepting the pseudo-signals

    __WARN__

and

    __DIE__

. The logic of

    Carp::Always

is pretty simple, see , so if you're not feeling like installing the module, you can easily do the same directly in your code.

If you want to intercept the error to add more information to the message, use

    eval

:

    eval { $self->{unit}->call($rowop) } or confess "Bad rowop argument:\n$@";

I have some better ideas about reporting the errors in the nested templated but they need to be implemented and tried out yet.

When the Perl code inside a label or tracer or aggregator or index sorting handler dies, the &Cpp; infrastructure around it catches the error. Here Triceps also has the old and the new way of dealing with it. The old way just prints the error on stderr and continues like nothing has happened. It's still used for the tracers, aggregators and index sorting. The new way unrolls the stack trace through the &Cpp; code and passes the

    die

request to the Perl code that called it. The labels have been converted to this new way. When one Perl label calls another Perl label that calls the third Perl label, the call sequence goes in layers of Perl&emdash;&Cpp;&emdash;Perl&emdash;&Cpp;&emdash;Perl&emdash;&Cpp;&emdash;Perl. If that last label has its Perl code die and there are no

    eval

s in between, the stack will be correctly unwound back through all these layers and reported in the error message. The &Cpp; code will include the reports of all the chained label calls as well. If one of the intermediate Perl layers wraps the call in

    eval

, it will receive the error message with the stack trace up to that point.
error handling
XS
die
confess
Carp

## Memory management fundamentals

The memory is managed in Triceps using the reference counters. Each Triceps object has a reference counter in it. In &Cpp; this is done explicitly, in Perl it gets mostly hidden behind the Perl memory management that also uses the reference counters. Mostly.

In &Cpp; the Autoref template is used to produce the reference objects. As the references are copied around between these objects, the reference counts in the target objects are automatically adjusted. When the reference count drops to 0, the target object gets destroyed. While there are live references, the object can't get destroyed from under them. All nice and well and simple, however still possible to get wrong.

The major problem with the reference counters is the reference cycles. If object A has a reference to object B, and object B has a reference (possibly, indirect) to object A, then neither of them will ever be destroyed. Many of these cases can be resolved by keeping a reference in one direction and a plain pointer in the other. This of course introduces the problem of hanging pointers, so extra care has to be taken to not reference them. There also are the unpleasant situations when there is absolutely no way around the reference cycles. For example, the Triceps label's method may keep a reference to the next label, where to send its processed results. If the labels are connected into a loop (a perfectly normal occurrence), this would cause a reference cycle. Here the way around is to know when all the labels are no longer used (before the thread exit), and explicitly tell them to clear their references to the other labels. This breaks up the cycle, and then bits and pieces can be collected by the reference count logic.

The reference cycle problem can be seen all the way up into the Perl level. However Triceps provides the ready solutions for its typical occurences. To explain it, more about Triceps operation has to be explained first, so it's described in detail later in &xrsp;.

The reference counting may be single-threaded or multi-threaded. If an object may only be used inside one thread, the references to it use the faster single-threaded counting. In &Cpp; it's real important to not access and not reference the single-threaded objects from multiple threads. In Perl, when a new thread is created, only the multithreaded objects from the parent thread become accessible for it, the rest become undefined, so the issue gets handled automatically (as of version 1.0 even the potentially multithreaded objects are still exported to Perl as single-threaded, with no connection between threads yet).

The &Cpp; objects are exported into Perl through wrappers. The wrappers perform the adaptation between Perl reference counting and Triceps reference counting, and sometimes more of the helper functions. Perl sees them as blessed objects, from which you can inherit and otherwise treat like normal objects.

When we say that a Perl variable

    $label

contains a Triceps label object, it really means that it contains a referece to a label object. When it gets copied like

    $label2 = $label

, this copies the reference and now both variables refer to the same label object (more exactly, even to the same wrapper object). Any changes to the object's state done through one reference will also be visible through the other reference.

When the Perl references are copied between the variables, this increases the Perl reference count to the same wrapper object. However if an object goes into the &Cpp; land, and then is extracted back (such as, create a Rowop from a Row, and then extract the Row from that Rowop), a brand new wrapper gets created. It's the same underlying &Cpp; object but with multiple wrappers. You can't tell that it's the same object by comparing the Perl references, because they may be pointing to the different wrappers. However Triceps provides the method

    same()

that compares the data inside the wrappers. It can be used as

    $row1->same($row2)

and if it returns true, then both

    $row1

and

    $row2

point to the same underlying row.

Note also that if you inherit from the Triceps objects and add some extra data to them, none of that data nor even your derived class'es identity will be preserved when a new wrapper is created from the underlying &Cpp; object.

memory management
Autoref
label
wrapper

## Triceps constants

Triceps has a number of symbolic constants that are grouped into essentially enums. The constants themselves will be introduced with the classes that use them, but here is the general description common to them all.

In Perl they all are placed into the same namespace. Each group of constants (that can be thought of as an enum) gets its name prefix. For example, the operation codes are all prefixed with OP_, the enqueueing modes with EM_, and so on.

The underlying constants are all integer. The way to give symbolic names to constants in Perl is to define a function without arguments that would return the value. Each constant has such a function defined for it. For example, the opcode for the insert operation is the result of function

    Triceps::OP_INSERT

.

Most methods that take constants as arguments are also smart enough to recognise the constant names as strings, and automatically convert them to integers. For example, the following calls are equivalent:

    $label->makeRowop(&Triceps::OP_INSERT, ...);
    $label->makeRowop("OP_INSERT", ...);

For a while I've thought that the version with

    Triceps::OP_INSERT

would be more efficient and might check for correctness of the name at compile time. But as it turns out, no, on both counts. The look-up of the function by name happens at run time, so there is no compile-time check. And that look-up happens to be a little slower than the one done by the Triceps &Cpp; code, so there is no win there either. The string version is not only shorter but also more efficient. The only win with the function is if you call it once, remember the result in a variable and then reuse. Unless you're chasing the last few percent of performance in a tight loop, it's not worth the trouble. Perhaps in the future the functions will be replaced with the module-level variables: that would be both faster and allow the compile-time checking with

    use strict

.

What if you need to print out a constant in a message? Triceps provides the conversion functions for each group of constants. They generally are named `Triceps::somethingString`. For example,

    print &Triceps::opcodeString(&Triceps::OP_INSERT);

would print OP_INSERT If the argument is out of range of the valid enums, it would return

    undef

(but not set any error message in

    $!

, since it's not a fatal error).

There also are functions to convert from strings to constant values. They generally are named `Triceps::stringSomething`. For example,

    &Triceps::stringOpcode("OP_INSERT")

would return the integer value of

    Triceps::OP_INSERT

. If the string name is not valid for this kind of constants, it would also return

    undef

.
constants
error handling

## Printing the object contents

When debugging the programs, it's important to find from the error messages, what is going on, what kinds of objects are getting involved. Because of this, many of the Triceps objects provide a way to print out their contents into a string. This is done with the method

    print()

. The simplest use is as follows:

    $message = "Error in object " . $object->print();

Most of the objects tend to have a pretty complicated internal structure and are printed on multiple lines. They look better when the components are appropriately indented. The default call prints as if the basic message is un-indented, and indents every extra level by 2 spaces.

This can be changed with extra arguments. The general format of

    print()

is:

    $object->print([$indent, [$subindent] ])

where $indent is the initial indentation, and $subindent is the additional indentation for every level. The default

    print()

is equivalent to

    print("", "  ")

.

A special case is

    $object->print(undef)

It prints the object in a single line, without line breaks.

Here is an example of how a row type object would get printed. The details of the row types will be described later, for now just assume that a row type is defined as:

    $rt1 = Triceps::RowType->new(
    	a => "uint8",
    	b => "int32",
    	c => "int64",
    	d => "float64",
    	e => "string",
    );

Then

    $rt1->print()

produces:

    row {
      uint8 a,
      int32 b,
      int64 c,
      float64 d,
      string e,
    }

With extra arguments

    $rt1->print("++", "--")

:

    row {
    ++--uint8 a,
    ++--int32 b,
    ++--int64 c,
    ++--float64 d,
    ++--string e,
    ++}

The first line doesn't have a ++ because the assumption is that the text gets appended to some other text already on this line, so any prefixes are used only for the following lines.

And finally with an undef argument

    $rt1->print(undef)

:

    row { uint8 a, int32 b, int64 c, float64 d, string e, }

The Rows and Rowops do not have the

    print()

method. That's largely because the &Cpp; code does not deal with printing the actual data, this is left to the Perl code. So instead they have the method

    printP()

that does a similar job. Only it's simpler and doesn't have any of the indenting niceties. It always prints the data in a single line. The P in printP stands for Perl. The name is also different because of this lack of indenting niceties. See more about it in the &xrsp;.
print
printP

## The Hungarian notation

The Hungarian notation is the idea that the name of each variable should be prefixed with some abbreviation of its type. It has probably become most widely known from the Microsoft operating systems.

Overall it's a complete abomination and brain damage. But I'm using it widely in the examples in this manual. Why? The problem is that there usually too many components for one logical purpose. For a table, there would be a row type, a table type, and the table itself. Rather than inventing separate names for them, it's easier to have a common name and an uniform prefix. Eventually something better would have to be done but for now I've fallen back on the Hungarian notation. One possibility is to just not give names to the intermediate entities. Say just have a named table, and then there would be the the type of the table and the row type of the table.

Among the CEP systems, Triceps is not unique in the Hungarian notation department. Coral8/Sybase CCL has this mess of lots of schemas, input streams, windows and output streams, with the same naming problems. The uniform naming prefixes or suffixes help making this mess more navigable. I haven't actually used StreamBase but from reading the documentation I get the feeling that the Hungarian notation is probably useful for its SQL as well.