scratchpad for ideas
_________________________________

-testing this shit-
 The tests are in dire need of update and have to be sure
 there is full code coverage. *sigh* the unfun part.
 especially test the swapout and the _copy_record.

 Test the hell out of the silo first. That may be
 somewhat simpler to test.


-transactions-

 Transactions are provided to the programmer and the programmer
 must still manage them as they do not provide a locking mechanism
 of their own.

 A transaction is a list of pending updates to the
 master index of the record store. When a transaction commits,
 the index is updated for all the items in the list.
 The transaction record is then removed. If the process is killed
 before the transaction record is removed, the datastore is in a
 bad state and the transaction started but never finished. The
 transaction can be resumed and completed or it can be reverted.

 transaction directories
    TRANS
    TRANS/META
    TRANS/instances/1
    TRANS/instances/2
    TRANS/instances/...

 transactions share a metadata silo, it has
    id of the transaction
    process that started the transaction
    time the transaction started
    state of the transaction (integer)
          1) active
          2) in commit
          3) in rollback or cancel
          4) in cleaning up

 transaction each get a silo, it has
    action character 'S'tow, 'D'elete, 'R'ecycle
    record id
    from silo id
    from id inside silo
    to silo id
    to id inside silo

 transaction methods
x    list pending transactions
x    create a new transaction 

x    stow <-- exception if in a dead state
x    delete_record <-- exception if in a dead state
x    recycle <-- exception if in a dead state

x    commit a transaction    <-- state is then dead
x    roll back a transaction <-- state is then dead

x    get_update_time
x    get_process_id
x    get_state
x    get_id

-clean datastore silos-

 Datastore silos could gain garbage records over time.
 This happens when a new value for a record is created
 and the system is killed before it can remove the
 old value of the record.

 It is tempting to assume this is a rare enough event
 that it wouldn't make a difference, but prudence
 dictates that this case should have a solution.

 Cleaning out the silos is a recycling process involving
 the silo, the main record index, and a scratch store with
 as many entrees as the silo grouping has.

 A silo grouping is all the silos of one size for records.

 The pseudocode for cleaning all silo groupings :

  - create and empty out a directory location for scratch silo indexes

  - create a scratch index (using a silo) for each silo. This index
    has a boolean as the single field

  - iterate over the master record index

    - look at : silo id, index in that silo

    - write an entry in the scratch index corresponding
      to the silo id and write a 1 at the appropriate index.

  - iterate each scratch silo index from 0 to length-1

    - if the entry has a 1, do nothing

    - if the entry doesn't have a 1 :
       - copy the last item in the silo to this record entry
       - update the position in the master record of the copied item
       - truncate the silo by one

    

   
