# Rows

In Triceps the relational data is stored and passed around as rows (once in a while I call them records, which is the same thing here). Each row belongs to a certain type, that defines the types of the fields. Each field may belong to one of the simple types.

## Simple types

The simple values in Triceps belong to one of the simple types:

- uint8
- int32
- int64
- float64
- string

I like the explicit specification of the data size, so it's not some mysterious double but an explicit float64.

When the data is stored in the rows, it's stored in the strongly-typed binary format. When it's extracted from the rows for the Perl code to access, it gets converted into the Perl values. And the other way around, when stored into the rows, the conversion is done from the Perl values.

    uint8

is the type intended to represent the raw bytes. So, for example, when they are compared, they should be compared as raw bytes, not according to the locale. Since Perl stores the raw bytes in strings, and its

    pack()

and

    unpack()

functions operate on strings, The Perl side of Triceps extracts the uint8 values from records into Perl strings, and the other way around.

The string type is intended to represent a text string in whatever current locale (at some point it may become always UTF-8, this question is open for now).

Perl on the 32-bit machines has an issue with int64: it has no type to represent it directly. Because of that, when the int64 values are passed to Perl on the 32-bit machines, they are converted into the floating-point numbers. This gives only 54 bits (including sign) of precision, but that's close enough. Anyway, the 32-bit machines are obsolete by now, and Triceps it targeted towards the 64-bit machines.

On the 64-bit machines both int32 and int64 translate to the Perl 64-bit integers.

Note that there is no special type for timestamps. As of version 1.0 there is no time-based processing inside Triceps, but that does not prevent you from passing around timestamps as data and use them in your logic. Just store the timestamps as integers (or, if you prefer, as floating point numbers). When the time-based processing will be added to Perl, the plan is to still use the int64 to store the number of microseconds since the Unix epoch. My experience with the time types in the other CEP systems is that they cause nothing but confusion. In the meantime, the time-based processing is still possible by driving the notion of time explicitly. It's described in the &xrsp;.

type simple
time

## Row types

A row type is created from a sequence of (field-name, field-type) string pairs, for example:

    $rt1 = Triceps::RowType->new(
    	a => "uint8",
    	b => "int32",
    	c => "int64",
    	d => "float64",
    	e => "string",
    );

Even though the pairs look like a hash, don't use an actual hash to create row types! The order of pairs in a hash is unpredictable, while the order of fields in a row type usually matters.

In an actual row the field may have a value or be NULL. The NULLs are represented in Perl as

    undef

.

The real-world records tend to be pretty wide and contain repetitive data. Hundreds of fields are not unusual, and I know of a case when an Aleri customer wanted to have records of two thousand fields (and succeeded). This just begs for arrays. So the Triceps rows allow the array fields. They are specified by adding [] at the end of field type. The arrays may only be made up of fixed-width data, so no arrays of strings.

    $rt2 = Triceps::RowType->new(
    	a => "uint8[]",
    	b => "int32[]",
    	c => "int64[]",
    	d => "float64[]",
    	e => "string", # no arrays of strings!
    ) or confess "$!";

The arrays are of variable length, whatever array data passed when a row is created determines its length. The individual elements in the array may not be NULL (and if

    undef

s are passed in the array used to construct the row, they will be replaced with 0s). The whole array field may be NULL, and this situation is equivalent to an empty array.

The type uint8 is typically used in arrays, uint8[] is the Triceps way to define a blob field. In Perl the uint8[] is represented as a string value, same as a simple unit8.

The rest of array values are represented in Perl as references to Perl arrays, containing the actual values.

The row type objects provide a way for introspection:

    $rt->getdef()

returns back the array of pairs used to create this type. It can be used among other things for the schema inheritance. For example, the multi-part messages with daily unique ids can be defined as:

    $rtMsgKey = Triceps::RowType->new(
    	date => "string",
    	id => "int32",
    ) or confess "$!";

    $rtMsg = Triceps::RowType->new(
    	$rtMsgKey->getdef(),
    	from => "string",
    	to => "string",
    	subject => "string",
    ) or confess "$!";

    $rtMsgPart = Triceps::RowType->new(
    	$rtMsgKey->getdef(),
    	type => "string",
    	payload => "string",
    ) or confess "$!";

The meaning here is the same as in the CCL example:

    create schema rtMsgKey (
    	string date,
    	integer id
    );
    create schema rtMsg inherits from rtMsgKey (
    	string from,
    	string to,
    	string subject
    );
    create schema rtMsgPart inherits from rtMsgKey (
    	string type,
    	string payload
    );

The grand plan is to provide some better ways of defining the commonality of fields between row types. It should include the ability to rename fields, to avoid conflicts, and to remember this equivalence to be reused in the further joins without the need to write it over and over again. But it has not come to the implementation stage yet.

The other methods are:

    $rt->getFieldNames()

returns the array of field names only.

    $rt->getFieldTypes()

returns the array of field types only.

    $rt->getFieldMapping()

returns the array of pairs that map the field names to their indexes in the field definitions. It can be stored into a hash and used for name-to-index translation. It's used mostly in the templates, to generate code that accesses data in the rows by field index (which is more efficient than access by name). For example, for

    rtMsgKey

defined above it would return

    (date => 0, id => 1)

.
type row
RowType
type array
arrays
arrays empty

## Row types equivalence

The Triceps objects are usually strongly typed. A label handles rows of a certain type. A table stores rows of a certain type.

However there may be multiple ways to check whether a row fits for a certain type:

- It may be a row of the exact same type, created with the same RowType object.
- It may be a row of another type but one with the exact same definition.
- It may be a row of another type that has the same number of fields and field types but different field names. The field names (and everything else in Triceps) are case-sensitive.

case sensitivity

The row types may be compared for these conditions using the methods:

    $rt1->same($rt2)
    $rt1->equals($rt2)
    $rt1->match($rt2)

The comparisons are hierarchical: if two type references are the same, they would also be equal and matching; two equal types are also matching.

Most of the objects would accept the rows of any matching type (this may change or become adjustable in the future). However if the rows are not of the same type, this check involves a performance penalty. If the types are the same, the comparison is limited to comparing the pointers. But if not, then the whole type definition has to be compared. So every time a row of a different type is passed, it would involve the overhead of type comparison.

For example:

    my @schema = (
    	a => "int32",
    	b => "string"
    );

    my $rt1 = Triceps::RowType->new(@schema) or confess "$!";
    # $rt2 is equal to $rt1: same field names and field types
    my $rt2 = Triceps::RowType->new(@schema) or confess "$!";
    # $rt3  matches $rt1 and $rt2: same field types but different names
    my $rt3 = Triceps::RowType->new(
    	A => "int32",
    	B => "string"
    ) or confess "$!";

    my $lab = $unit->makeDummyLabel($rt1, "lab") or confess "$!";
    # same type, efficient
    my $rop1 = $lab->makeRowop(&Triceps::OP_INSERT,
    	$rt1->makeRowArray(1, "x")) or confess "$!";
    # different row type, involves a comparison overhead
    my $rop2 = $lab->makeRowop(&Triceps::OP_INSERT,
    	$rt2->makeRowArray(1, "x")) or confess "$!";
    # different row type, involves a comparison overhead
    my $rop3 = $lab->makeRowop(&Triceps::OP_INSERT,
    	$rt3->makeRowArray(1, "x")) or confess "$!";

A dummy label used here is a label that does nothing (its usefulness will be explained later).

Once the Rowop is constructed, no further penalty is involved: the row in the Rowop is re-typed to the type of the label from now on. It's physically still the same row with another reference to it, but when you get it back from the Rowop, it will have the label's type. It's all a part of the interesting interaction between &Cpp; and Perl. All the type checking is done in the Perl XS layer. The &Cpp; code just expects that the data is always right and doesn't carry the types around. When the Perl code wants to get the row back from the Rowop, it wants to know the type of the row. The only way to get it is to look, what is the label of this Rowop, and get the row type from the label. This is also the reason why the types have to be checked when the Rowop is constructed: if a wrong row is placed into the Rowop, there will be no later opportunity to check it for correctness, and bad data may cause a crash.

type match
type equals
RowType match
RowType equals
Row re-typing
XS

## Rows

The rows in Triceps always belong to some row type, and are always immutable. Once a row is created, it can not be changed. This allows it to be referenced from multiple places, instead of copying the whole row value. Naturally, a row may be passed and shared between multiple threads.

The row type provides the constructor methods for the rows:

    $row = $rowType->makeRowArray(@fieldValues);
    $row = $rowType->makeRowHash($fieldName => $fieldValue, ...);

Here

    $row

is a reference to the resulting row. As usual, in case of error it will be left as

    undef

, with the error message in

    $!

.

In the array form, the values for the fields go in the same order as they are specified in the row type (if there are too few values, the rest will be considered NULL, having too many values is an error).

The Perl value of

    undef

is treated as NULL.

In the hash form, the fields are specified as name-value pairs. If the same field is specified multiple times, the last value will overwrite all the previous ones. The unspecified fields will be left as NULL. Again, the arguments of the function actually are an array, but if you pass a hash, its contents will be converted to an array on the call stack.

If the performance is important, the array form is more efficient, since the hash form has to translate internally the field names to indexes.

The row itself and its type don't have any concept of keys in general and of the primary key in particular. So any fields may be left as NULL. There is no NOT NULL constraint.

Some examples:

    $row  = $rowType->makeRowArray(@fields) or die "$!";
    $row  = $rowType->makeRowArray($a, $b, $c) or die "$!";
    $row  = $rowType->makeRowHash(%fields) or die "$!";
    $row  = $rowType->makeRowHash(a => $a, b => $b) or die "$!";

The usual Perl conversions are applied to the values. So for example, if you pass an integer 1 for a string field, it will be converted to the string 1. Or if you pass a string for an integer field, it will be converted to 0.

If a field is an array (as always, except for

    uint8[]

which is represented as a Perl string), its value is a Perl array reference (or

    undef

). For example:

    $rt1 = Triceps::RowType->new(
    	a => "uint8[]",
    	b => "int32[]",
    ) or confess "$!";
    $row = $rt1->makeRowArray("abcd", [1, 2, 3]) or confess "$!";

An empty array will become a NULL value. So the following two are equivalent:

    $row = $rt1->makeRowArray("abcd", []) or die "$!";
    $row = $rt1->makeRowArray("abcd", undef) or die "$!";

Remember that an array field may not contain NULL values. Any

    undef

s in the array fields will be silently converted to zeroes (since arrays are supported only for the numeric types, a zero value would always be available for all of them). The following two are equivalent:

    $row = $rt1->makeRowArray("abcd", [undef, undef]) or die "$!";
    $row = $rt1->makeRowArray("abcd", [0, 0]) or die "$!";

The row also provides a way to copy itself, modifying the values of selected fields:

    $row2 = $row1->copymod($fieldName => $fieldValue, ...);

The fields that are not explicitly specified will be left unchanged. Since the rows are immutable, this is the closest thing to the field assignment.

    copymod()

is generally more efficient than extracting the row into an array or hash, replacing a few of them with new values and constructing a new row. It bypasses the binary-to-Perl-to-binary conversions for the unchanged fields.

The row knows its type, which can be obtained with

    $row->getType()

Note that this will create a new Perl wrapper to the underlying type object. So if you do:

    $rt1 = ...;
    $row = $rt1->makeRow...;
    $rt2 = $row->getType();

then

    $rt1

will not be equal to

    $rt2

by the direct Perl comparison (

    $rt1 != $rt2

). However both

    $rt1

and

    $rt2

will refer to the same row type object, so

    $rt1->same($rt2)

will be true.

The row references can also be compared for sameness:

    $row1->same($row2)

The row contents can be extracted back into Perl representation as

    @adata = $row->toArray();
    %hdata = $row->toHash();

Again, the NULL fields will become

    undef

s, and the array fields (unless they are NULL) will become Perl array references. Since the empty array fields are equivalent to NULL array fields, on extraction back they will be treated the same as NULL fields, and become

    undef

s.

There is also a convenience function to get one field from a row at a time by name:

    $value = $row->get("fieldName");

If you need to access only a few fields from a big row,

    get()

is more efficient (and easier to write) that extracting the whole row with

    toHash()

or even with

    toArray()

. But don't forget that every time you call

    get()

, it creates a new Perl value, which may be pretty involved if the value is an array. So the most efficient way then for the values that get reused many times is to call

    get()

, remember the result in a Perl variable, and then reuse that variable.

There is also a way to conveniently print a rows contents, usually for the debugging purposes:

    $result = $row->printP();

The name

    printP

is an artifact of implementation: it shows that this method is implemented in Perl and uses the default Perl conversions of values to strings. The

    uint8[]

arrays are printed directly as strings. The result is a sequence of

    name="value"

or

    name=["value", "value", "value"]

for all the non-NULL fields. The backslashes and double quotes inside the values are escaped by backslashes in Perl style. For example, reusing the row type above,

    $row = $rt1->makeRowArray('ab\ "cd"', [0, 0]) or die "$!";
    print $row->printP(), "\n";

will produce

    a="ab\\ \"cd\"" b=["0", "0"]

Finally, there is a deep debugging method:

    $result = $row->hexdump()

That dumps the raw bytes of the row's binary format, and is useful only to debug the more weird issues.

Row
print