=head1 NAME

ObjStore::Internals - a few notes on the implementation

=head1 SYNOPSIS

You don't have to understand anything about the technical
implementation.  Just know that:

=over 4

=item *

C<ObjectStore> is outrageously powerful; sophisticated; and even
over-engineered.

=item *

The perl interface is optimized to be fun and easy.  Since
C<ObjectStore> is also blindingly fast, you can happily leave
relational databases to collect dust on the bookshelf where they
belong.

=back

So basically, you don't have to understand anything to a greater
depth.  It's not necessary.  You've already arrived and you will be
successful.  On the other hand, some people want more.  They like to
turn things inside-out.  These are my kind of people.  In this case,
read on!

=head1 DESCRIPTION

=head2 Perl & C<C++> APIs: What's The Difference?

Most stuff should be roughly the same.  The few exceptions have
generally arisen because there was a perl way to make the interface
more programmer friendly.

=over 4

=item *

Transactions are perlified.

=item *

Some static methods sit directly under C<ObjStore::> instead of under
their own classes.  (Easier to import.)

=item *

Databases are always blessed according to your pleasure.

=item * 

C<lookup>, C<open>, C<is_open>, and C<lock_timeout> are augmented with
multi-color, pop-tart style interfaces.

=back

=head2 Why not just store perl data with the usual perl structures?

=over 4

=item * CHANGE CONTROL

As perl evolves, new data layouts are introduced.  These changes
must not cause database compatibility problems.

=item * BINARY COMPATIBILITY

Perl doesn't have to worry about binary compatibility between
platforms.  Databases do.  In addition, databases impose a number of
restrictions on persistent data layout that would be onerous and
sub-optimal if adopted by perl.

=item * MEMORY USAGE

Perl often trades memory for speed.  This is the wrong trade for a
database.  Memory usage is much more of a concern when data sets can
be as large or larger than ten million megabytes.  A few percent
difference in compactness can be quite noticable.

=back

=head2 Representation

All values take a minimum of 8 bytes (OSSV).  These 8 bytes are used
to store 16-bits of type information, a pointer, and a general purpose
16-bit value.

  value stored                   extra allocation (in addition to OSSV)
  ------------------------------ -------------------------------------
  undef                          none
  pointer                        none
  16-bit signed integers         none
  32-bit signed integers         4 byte block (OSPV_iv)
  double                         8 byte block (OSPV_nv)
  string                         length of string (char*)
  object (ref or container)      sizeof object (see subclasses of OSSVPV)
                                 additional references take no extra space
  bless                          .5-1k bytes per class (zero per object)

  %ObjStore::sizeof XXX

Since 32-bit integers and doubles are fairly common and should be
stored densely, a pool allocation algorithm is planned.

The ODI FAQ also states: I<In addition, there is an associated entry
in the info segment for the segment in question for each allocation of
the object. This is done in the tag table. The overhead is 16 bits
(i.e., 2 bytes) for each singleton (i.e., non-array) allocation, 32
bits for each character array allocation for character arrays E<lt>=
255 characters, and 48 bits for each character array allocation E<gt>
255 characters, or any array allocation of an object of another type.
Also, depending on the size of an object (i.e., if you allocate a
"huge" object - one that is E<gt>64Kb) there is other overhead caused
by alignment constraints.>

If this seems like a lot of overhead, consider that it is not really
possible to directly compare these numbers to RDBMS statistics.  (Part
of the problem is that RDBMS vendors can't even give you these
statistics.)  At least note that relational data can be stored with
much less duplication when moved into C<ObjectStore>.  This is
unquestionably true when you tailor your own C++ extensions to fit the
data and access patterns.

=head2 Hard-Coded Limits

=over 4

=item *

Reference counts are only 32 bits unsigned.  (The first person to hit
this limit in a real application will receive a check from me for $32.
Please submit a one page description of your application for judging.
:-)

=item *

Strings are limited to a length of 32767 bytes.  (This limit will be
relaxed in a future version.)

=back

=head2 Bless

If you are a suspicious person (like my mom) you might have suspected
that the ObjStore module installs its own version of C<bless>.
Natually it does!  The augmented C<bless> implements extra quality
assurance to insure that blessings are correctly stored persistently.
For example:

    package Scottie;
    use ObjStore;
    use base 'ObjStore::HV';
    $VERSION = '2.00';

    sub new {
        my ($class, $store) = @_;
        my $o = $class->SUPER::new($store, { fur => 'buffy' });
        $o;
    }

    package main;

    my Scottie $dog = Scottie->new($db);
    # once a Scottie, always a Scottie

Persistent C<bless> also does some extra work to make evolution
easier.  It stores the current C<@ISA> tree along with the C<$VERSION>
of every class in the C<@ISA> tree.  Furthermore, the C<isa> method is
tweaked such that it reports according to the moment of the C<bless>.
Similarly, the C<versionof> method lets you query the saved
C<$VERSION>s.  This is helpful when doing evolution, as you can
compare the old C<@ISA> and C<$VERSION>s to figure out what to change
(and how).  C<UNIVERSAL::can> is unmodified.

Technically speaking, C<bless> is re-implemented such that it can be
extended by the I<bless from> and the I<bless to> classes via the
C<BLESS> method.  Both the I<bless from> and I<bless to> operations
are funnelled through a single C<BLESS> method like this:

  sub BLESS {
      my ($r1,$r2);
      if (ref $r1) { warn "$r1 leaving ".ref $r1." for a new life in $r2\n"; }
      else         { warn "$r2 entering $r1\n"; }
      $r1->SUPER::BLESS($r2);
  }

=head2 UNLOADED

Generic tools such as C<posh> or C<ospeek> must C<bless> objects when
reading from an arbitrary database.  To C<bless>, there must be
support code.  To try to get support code, classes found in the
database are C<require>'d.  However, the C<require> may fail.  A
package must be faked-up and C<${"${package}::UNLOADED"}> is set to
true.  This signals that the @ISA tree should not be considered
authoritative for a particular package.

=head2 Go Extension Crazy

You cannot directly access persistent scalars from perl. They are
always immediately copied into transient scalars.  This is actually
faster than the alternatives in most cases.

While all persistent objects are blessed, they are not considered
I<blessed> in the database unless they are members of some non-default
class (not os_class).  C<NOREFS> is not invoked on I<non-blessed>
database objects.

$ObjStore::COMPILE_TIME XXX

C<ObjStore::File> will be the base class for large binary data.

Each subclass of C<ObjStore::UNIVERSAL::Container> has a C<%REP> hash.
The C<new> method decides on the best representation, calls the best
creation function from the C<%REP> hash, returning the newly minted
persistent object.

Add your own C<C++> representation.  New families of objects can
inherit from C<ObjStore::UNIVERSAL>.  Suppose you want highly
optimized, persistent bit vectors?  Or matrics?  No problem.

=head2 ObjStore::Index

Indices are extremely efficient because they do not copy their keys.
It is critical that another pointer to C<OSSV>s is not stored, since
C<OSSV>s can be relocated when arrays need to grow.  C<OSSVPV>s are
never relocated.  OTOH, C<OSSVPV>s should be able to change
representations.  Hm..

=head2 Notes On The Source Code

=over 4

=item *

Functions or methods starting with '_' are internal to the C<ObjStore>
extension.  They are subject to change (entirely) without notice.

=item *

Avoid C<const>, privacy & templates.  C<C++> sucks!  Long live C<C++>!

=item *

The relationship between references and cursors is strange.  It's
probably best not to think about it.

=back

=head1 BUGS

=over 4

=item * UNFORGEABLE NOTIFICATION SENDER INFORMATION

ODI feature request #13632: It would be very useful to know which
client sent a given notification.  While the client I<could> fill in
this information as part of the notification, the C<osserver> already
knows the sender's client# and could pass this information
transparently to subscribers.  The additional overhead be just 2 bytes
per received notification.

=item * CROSS DATABASE POINTERS

This feature is highly depreciated and will likely be discontinued,
but at the moment you can allow cross database B<pointers> with:

  $db->_allow_external_pointers;    #never do this!

But you should avoid this if at all possible.  Using real pointers
will affect refcnts, even between two different databases.  Your
refcnts will be wrong if you simply C<osrm> a random database.  This
will cause some of your data to become permenently un-deletable.
Currently, there is no way to safely delete un-deletable data.

Instead, you can use references or cursors to refer to data in other
databases.  References may use the C<os_reference_protected> class
which is designed precisely to address this problem.  Refcnts will not
be updated remotely, but you'll still be protected from accessing
deleted objects or removed databases.  (Imagine the freedom. :-)

=item * LEAKS TRANSIENT C<XPVRV>s

The problem is thoroughly understood.  Work-arounds or a real fix have
been discussed on the perl5-porters mailing list.  Well designed
mechanisms are being developed to solve this problem correctly.
[Fixed?]

=item * C<ObjStore::AVHV> EVOLUTION

Indexed records temporarily cannot be evolved due to const-ness.  For
now, it is recommended that records be removed, changed, and re-added
to the table when changing indexed fields.

=item * C<os_protected_reference>

Allocates persistent memory that cannot be reclaimed without
destroying the segment.  This makes it non-trival to determine whether
a segment is empty or not.  The needed change is listed as ODI feature
request C<#SE055496_O#>.

=item * WIN32

Threads will not work until perl has a well-tested API for an
externally created thread to call into the perl interpreter.  (The
beginnings of this are there in the form of new_struct_thread(), but
it still needs a bit work.)

=item * C<MOP>

This is not a general purpose C<ObjectStore> editor with complete
C<MOP> support.  Actually, I don't think this is a bug!

=item * HIGH VOLITILITY

Everything is subject to change without notice.  (But backward
compatibility will be preserved when possible. :-)

=item * POOR QUALITY DOCUMENTATION

I didn't get a Ph.D in English.  Sorry!

=back

=head1 SOURCE CODE AVAILABILITY

This section is not a legal contract.  This section is not a legal
statement in any way, shape, or form.


While there have been gains in software quality in the form of GNU,
Perl, Apache, Linux, Qt, and recently the Netscape browser
(potentially), we have been in the dark ages with respect to database
technology.  After avoiding relational databases for years, I sensed a
combination of ObjectStore and Perl could offer the same level of
quality and simplicity that I find invaluable in addressing the
hurdles I face as a software professional.

It might seem obvious in hindsight (doesn't it always?!), but it is my
conviction that it was something in particular (not just luck!) that
encouraged me imagine and implement this cutting-edge technology years
before it would be recognized and adopted.  Therefore, I would like to
say "thank you!" to all the wonderful teachers with whom I've had the
opportunity of studying.  My ideas are truly theirs.  While I predict
that this software will be considered a world-class inter-disciplinary
achievement, I do not see it as such.  Rather, I see it as a simple
mechanical exercise employed to bring me to a basic level of clear
understanding.  My subsequent hope is that people who use this
software will also be able to learn from its clarity.  But what are my
real aspirations?

I should keep quiet about that but I can say this much: One of my
friends had an email signature: "Life would be so much easier if we
could just look at the source code."  (Imagine, "computational
metaphysics" :-) In using this module, you might feel a hint of it?
That's my real interest.  You might also be interested!?  Perhaps even
interested in understanding more (even beyond the bounds of all
thought and limitations)?  "With our thoughts we make the world."
Perhaps it's not common knowledge, but some things don't require
belief to be true.  If you want to find out for yourself (first hand),
I would strongly encourage such an inquiry.  You must seriously
consider the hard, imporant questions in concert with a well regarded
teacher.  If you have never studied in this way, I highly recommend
it.  (It is an opportunity that is also priceless.)  And I guarantee
that if you look sincerely you will find what you seek.  (Or I'll give
you double your money back. :-) Trust me, it's worth it.

In fact, one of the main reasons I bothered to kill myself developing
this software (and foolishly giving it away for free!) is that I
wanted to emphasize this point as strongly as possible.  There is
nothing more important to me.  And yet on the other hand, I know that
other people may have different priorities.  That is okay.  You are
welcome to use this software even if you are interested exclusively in
satisfying your own greed (it will work wonders :-).  However, I have
an obligation to state the truth.  You can either be part of the
problem or part of the solution.  And what is at stake is much more
significant, much bigger than you think.  Designing software requires
a basic subtlety of awareness: to be successful, you must be able to
fluctuate fluidly between the place of reason and the place of silent
knowledge.  The place of concern is the forerunner to the place of
reason.  Similarly, the place of unbending intent is the forerunner to
the place of silent knowledge.  Yet, this technique, in my opinion, is
(worse than?) worthless unless it is properly directed at trying to
answer the hard, important questions.  If you continue to persist in
not caring very much, you will find out what is the result.  Your life
will be become the result!  But always I wish everyone well and the
best of luck.  :-) You have been warned, but will you take action?

=cut
