Re: ADT Lib: Persistence

[Michael van Acken <mvacken@t-online.de>]

Thank you for your valuable input.  I was always wondering how
Oberon/F did its alien magic.

> It may be worth considering something similar to the Blackbox "Alien"
> mechanism. When Blackbox finds an object for which it has no associated
> module, it loads the object as type "Alien". Alien objects can be saved
> again but cannot be otherwise interpreted. 

Can pointers appear inside an alien object?  That is, if an alien
object is part of a file, do we need to take special precautions to
avoid breaking any of its internal pointers?

> There is a further issue that arises if we would like to be able to
> compress the type names. Blackbox only ever writes a type name once per
> store (the first time it is used). Thereafter it substitutes a more compact
> (persumably integer) identifier. This is fairly easy to do, and a procedure
> is outlined in the ETH report that you mention. Without compression the
> type names may well be much larger than actual objects which can waste time
> and space in encoding and decoding. However, the situation gets complicated
> if a type name _first_ appears inside an Alien object. You can't simply
> ignore the contents of an Alien object, since you might miss the
> type-name-to-identifier mapping that will be needed later. The implication
> is that the components of Aliens must be interpretable, even if the object
> itself is of unknown format. Again, this is not too hard to achieve: after
> each type header, you write the offset to the next type header (this
> effectively gives a linked-list of all the objects in a file). When
> encountering an Alien object, you actually need to load its components (you
> can't simply copy them) since the type-name-to-identifier mappings may be
> different next time the object is written. This would be a result _not_ of
> the object itself, but of other objects that may have previously been
> written during the whole writing process. Parts of the Alien that occur
> between its components can be copied verbatim.

When you say "components", do you mean pointers (references to other
objects) embedded in the alien?  If this is the case, this would
answer my question from above.  

Is it correct, that internalizing an alien object means to store in
memory a sequence of bytes, with pointers interspersed somewhere in
between?  And externalizing the alien means to store the byte
sequences "as is", but rewrite the interspersed pointers to fit the
newly written file?

Note that I am talking about copying the parts into memory, instead of
just keeping track of file intervals.  After all, a file may disappear
after it has been internalized.

> You can guess what Blackbox is doing from the following definition in
> module Stores:
> 
>   AlienComp = POINTER TO LIMITED RECORD
>     next-: AlienComp
>   END;
> 
>   AlienPiece = POINTER TO LIMITED RECORD (AlienComp)
>     pos-, len-: INTEGER
>   END;
> 
>   AlienPart = POINTER TO LIMITED RECORD (AlienComp)
>     store-: Store
>   END;

I don't know what a `Store' is for Blackbox, and how it maps to `Files
a la OOC'.

>   Alien = POINTER TO LIMITED RECORD (Store)
>     path-: TypePath;
>     cause-: INTEGER;
>     file-: Files.File;
>     comps-: AlienComp
>   END;
> 
> >3. The algorithm requires that every instance of `Object' has a field
> >`mark'.  Alternative: like reading, writing could use a object table
> >and do table lookups to determine the marker.  This would keep
> >`Object' small, but takes longer because of the table lookups.
> 
> Hmmm. I always wondered how this was supposed to work. Thanks for the
> reference to the ETH report. The problem as I saw it was that a mark field
> will always (?) be initialised to zero by the run-time when an object is
> created. Thus, you can write it once, but not a second time since you have
> already marked all the objects. The only way I could see of clearing the
> mark field was to do a "fake" WriteObject (with a dummy BinaryWriter) that
> clears the marks before doing the "real" WriteObject. 

This could be better done the other way round: after writing the whole
graph, the same algorithm that was used to set `mark' can be used to
clear it again.  This would establish the invariant, that `mark' is 0
during the entire lifetime of an object, except temporarily when
writing a data structure.

> There is no other formal way in which an object enumerates all its
> components. I guess the other alternative is to use a "virtual
> clock" as described in Josef Templ's article. 

I believe, the clock is kept in the type descriptor.  While I can
easily change OOC to have such a field in its run-time data, this
would make it impossible to port the persistence model to other Oberon
systems.

With Stewart's mail, I have all the necessary pieces to put together
an implementation of persistence that includes Blackbox's `Alien'
concept.  Only minor implementation details are still debatable.

One question remains: Should we also tackle `Libraries' a la Gadgets?
I have no experience with Gadgets whatsoever, do we have any Gadgets
users on this list who can provide some hard information?

I believe, a Gadget `Library' is a file containing a number of named,
persistent objects.  References can be made to objects of other
libraries by quoting the library name and the item name (similar to
qualified import in O2).  The differences between `Library' and the
persistence model discussed here, is primarily the ability to
distribute persistent objects over multiple files.  Our "simple"
persistence can only dump a whole data structure into a single file.

My feeling is, that `Library' is a too high-level concept to establish
it at the very root of the class hierarchy.  There may be a chance to
implement it later, either on top of the "simple" persistence model,
or by slightly extending it.  Therefore I suggest to ignore
`Libraries' at the moment.

-- mva