To get started with this blank [[TiddlyWiki]], you'll need to modify the following tiddlers:
* [[SiteTitle]] & [[SiteSubtitle]]: The title and subtitle of the site, as shown above (after saving, they will also appear in the browser title bar)
* [[MainMenu]]: The menu (usually on the left)
* [[DefaultTiddlers]]: Contains the names of the tiddlers that you want to appear when the TiddlyWiki is opened
You'll also need to enter your username for signing your edits: <<option txtUserName>>

/***
StyleSheet for use when a translation requires any css style changes.
This StyleSheet can be used directly by languages such as Chinese, Japanese and Korean which need larger font sizes.
***/
/*{{{*/
body {font-size:0.8em;}
#sidebarOptions {font-size:1.05em;}
#sidebarOptions a {font-style:normal;}
#sidebarOptions .sliderPanel {font-size:0.95em;}
.subtitle {font-size:0.8em;}
.viewer table.listView {font-size:0.95em;}
/*}}}*/

//Monads are of questionable usefullness//
Monads are a concept and theoretical framework from category theory.
But the use of Monads as a programming construct is much touted in the realm of functional programming -- which unfortunately is riddled with some kind of phobia towards ''State''.
And this in itself is ill-guided, since //not State is the problem,// ''Complexity'' is. Complexity arises from far reaching non local interdependencies and coupling. Some complexity is essential.
A telltale sign is that people constantly ask »What is a Monad?«
And they don't get an answer, rather they get a thousand answers.
The term //"Monad" fails to evoke an image// once mentioned.
What remains is a set of clever technicalities. Such can be applied and executed without understanding. The same technicality can be made to work on vastly distinct subjects. And a way to organise such technicalities can be a way to rule and control subjects. The exertion of control is not in need of invoking images, once its applicability is ensured. //That// state is indifferent towards complexity, at least.
When we care for complexity, we do so while we care for matters tangible to humans. Related to software, this is the need to maintain it, to adjust it, to adapt and remould it, to keep it relevant. At the bottom of all of these is the need to understand software. And this mandates to use terms and notions, even patterns, which evoke meaning -- even to the uninitiated. How can Monads even be helpful with that?
!To make sensible usage of Monads
Foremost, they should be kept as what they are: technicalities. For the understanding, they must be subordinate to a real concept or pattern. One with the power to reorient our view of matters at hand.
Thus we ask: what can be said about Monads?
!!!Formalism
Regarding the formalism, it should be mentioned
* that a Monad is a //type constructor// -- it takes a type parameter and generates a new instant of monadic kind.
* and assuming such an instance, there is a //constructor// <br>{{{
unit: A → M<A>
}}}
* and once we've optained a concrete entity of type {{{M<A>}}}, we can //bind a function// to transform it in another monadic entity<br/>{{{
M<A>::bind( A → M<B> ) → M<B>
}}}
At this point, also the ''Monad Axioms'' should be mentioned
;neutrality
:{{{unit}}} (the monad constructor) is neutral with respect to the {{{bind}}} operation
{{{
unit(a)::bind(f) ≡ f(a)
M::bind(unit) ≡ M
}}}
;composition
:we can define a //composition// of monadic functions,
:which is equivalent to consecutive binding
{{{
(M::bind(f) )::bind(g) ≡ M::bind( f ∘ g ) with f ∘ g defined as λ.x → { f(x)::bind(g) }
}}}
!!!Distinct properties
The obvious first observation is that a Monad must be some kind of //container// -- or //object,// for that.
The next observation to spring into mind is the fact that the {{{bind}}}-operation is notoriously hard to understand.
Why is this so? Because it //intermingles// the world of monads with the things from the value domain. You can not just write a monadic function for use with {{{bind}}}, without being aware of the world of monads. It is not possible to "lift" an operation from the world of values automatically into the world of monads. Please contrast this to the ''map operation'', which is easy to understand for that very reason: if you map an operation onto a container of values, the operation itself does not need to be aware of the existence of containers as such. It just operates on values.
This observation can be turned into a positive use-case for monadic structures -- whenever there is an ''additional cross-cutting concern'' within the world of values, to necessitate our constant concern all over the place. The Monad technique can then be used to re-shuffle this concern in such a way that it becomes possible to keep it //mentally separate.// Thus, while it is not possible to //remove// that concern, re-casting it as Monad creates a more regular structure, which is easier to cope with. A Monad can thus be seen as an ''augmented or "amplified" value''. A value outfitted with additional capabilities. This view might be illustrated by looking at some of the most prominent applications
;failure handling
:when we mark an expression as prone to failure by wrapping it into some //exception handling monad,//
:it becomes easy to combine several such unreliable operations into a single program unit with just one failure handler
;partially defined operations
:when an operation can only work on some part of the value domain, we can mark the result as //optional//
:and combining several tasks with optional result into one chain of operations can then be done completely generic
;multiple results
:when some tasks might possibly produce multiple results, we can mark those with a //container monad,//
:and collecting and combining of such multifold results can then be dealt with completely separate from the generating actions
;secondary state attributions
:when we have to observe, extend and communicate additional data beyond what is involved in the basic computation
:the collection and management of such additional attributions can be externalised with a //state monad//
;local technicalities
:when some algorithm or computation involves specific technicalities to get at the underlying data
:a monad can be used to encapsulate those details and expose only abstracted higher-level //operation primitives//
!!!Building with monads
As pointed out above, while all those disparate usages might share some abstract structural similarities, no overarching theme can be found to span them all. If we tend to distil an essence from all those usages, we're bound to end up with nothing. The reason is, //we do apply// monadic techniques while coping with the problem, but there is //nothing inherent "monadic" in the nature// of things at hand.
Yet to guide our thinking and doing, when we deal with matters, other concepts, notions and patterns are better suited to guide our actions, because they concur with the inherent nature of things:
;builder
:a builder allows to assemble a very specific structure from building blocks,
:when done, the structure is self-contained and can be operated and used devoid all intricacies involved in building.
:Monads are //exceptionally viable// for performing this process of assembly within the builder.
;domain specific language
:a DSL can found a higher level of abstraction, where local handling details are stashed away.
:Monads can be useful as supporting backbone below a DSL, especially in cases where a full language system and runtime engine would be overblown
;abstract composite value
:an abstract data type or complex value can be used to represent computations beyond simple numbers,
:prominent examples being colours with colour space or measurement values with numerical precision and unit of measurement.
:Monads can be used to build the basic combining operations on such custom types, and their interplay.
;engine
:an engine guides and enacts ongoing computations, thereby transitioning internal state
:Monads can be used as blueprint for managing these internal state transitions, and to prevent them from leaking out into the target of processing
;context
:a (possibly global) context can be used to represent all further concerns, which need to be there as well.
:Monads can be used to create a distinct, secondary system of communication (»wormhole«) and they can be used to capture and replay events.
:beyond that, for most practical purposes, simple plain mutable variables plus nested scopes //are a solution superior to the application of Monads.//

//pattern of collaboration for loosely coupled entities, to be used for various purposes within Proc...//
Expecting Advice and giving Advice &mdash; this collaboration ranges somewhere between messaging and dynamic properties, but cross-cutting the primary, often hierarchical relation of dependencies. Always happening at a certain //point of advice,// this exchange of data gains a distinct, static nature -- it is more than just a convention or a protocol. On the other hand, Advice is deliberately kept optional and received synchronously (albeit possibly within an continuation), this way allowing for loose coupling. This indirect, cross-cutting nature allows to build integration points into library facilities, without enforcing a strong dependency link.
!Specification
''Definition'': Advice is an optional, mediated collaboration between entities taking on the roles of advisor and advised, thereby passing a custom piece of advice data, managed by the advice support system. The possibility of advice is created by both of the collaborators entering the system, where the advised entity exposes a point of advice, while the advising entity provides an actual advice value.
[>img[Entities for Advice collaboration|uml/fig141445.png]]
!!Collaborators
* the ''advised'' entity
* the ''advisor''
* ''point of advice''
* ''advice system''
* the ''binding''
* the ''advice''
Usually, the ''advised'' entity opens the collaboration by requesting an advice. The ''advice'' itself is a piece of data of a custom type, which needs to be //copyable.// Obviously, both the advised and the advisor need to share knowledge about the meaning of this advice data. (in a more elaborate version we might allow the advisor to provide a subclass of the advice interface type). The actual advice collaboration happens at a ''point of advice'', which needs to be derived first. To this end, two prerequisites are to be fulfilled (without fixed sequence): The advised puts up an ''advice request'' by specifying his ''binding'', which is a pattern for matching. An entity about to give advice attaches a possible ''advice provision'', combined with an advisor binding, which similarly is a pattern. The ''advice system'' as mediator resolves both sides, by matching (which in the most general case could be an unification). This process creates an ''advice point solution'' &mdash; allowing the advisor to fed the piece of advice into a kind of communication channel (&raquo;advice channel&laquo;), causing the advice data to be placed into the point of advice. After passing a certain (implementation defined) break point, the advice leaves the influence of the advisor and gets exposed to the advised entities. Especially, this involves copying the advice data into a location managed by the advice system. In the standard case, the advised entity picks up the advice synchronously (and non-blocking). Of course, there could be a status flag to find out if there is new advice. Moreover, typically the advice data type is default constructible and thus there is always a basic form of advice available, thereby completely decoupling the advised entity from the timings related to this collaboration.
!!extensions
In a more elaborate scheme, the advised entity could provide a signal to be invoked either in the thread context of the advisor (being still blocked in the advice providing call), or in a completely separate thread. A third solution would be to allow the advised entity to block until receiving new advice. Both of these more elaborate schemes would also allow to create an advice queue &mdash; thereby developing the advice collaboration into a kind of messaging system. Following this route seems questionable though.
&rarr; AdviceSituations
&rarr; AdviceRequirements
&rarr; AdviceImplementation

[<img[Advice solution|uml/fig141573.png]]
The advice system is //templated on the advice type// &mdash; so basically any collaboration is limited to a distinct advice type. But currently (as of 5/2010), this typed context is kept on the interface level, while the implementation is built on top of a single lookup table (which might create contention problems in the future and thus may be changed without further notice). The advice system is a system wide singleton service, but it is never addressed directly by the participants. Rather, instances of ~AdviceProvision and ~AdviceRequest act as point of access. But these aren't completely symmetric; while the ~AdviceRequest is owned by the advised entity, the ~AdviceProvision is a value object, a uniform holder used to introduce new advice into the system. ~AdviceProvision is copied into an internal buffer and managed by the advice system, as is the actual advice item, which is copied alongside.
In order to find matches and provide advice solutions, the advice system maintains an index data structure called ''~Binding-Index''. The actual binding predicates are represented by value objects stored within this index table. The matching process is triggered whenever a new possibility for an advice solution enters the system, which could be a new request, a new provision or a change in the specified bindings. A successful match causes a pointer to be set within the ~AdviceRequest, pointing to the ~AdviceProvision acting as solution. Thus, when a solution exists, the advised entity can access the advice value object by dereferencing this pointer. A new advice solution just results in setting a different pointer, which is atomic and doesn't need to be protected by locking. But note, omitting the locking means there is no memory barrier; thus the advised entity might not see any changed advice solution, until the corresponding thread(s) refresh their CPU cache. This might or might not be acceptable, depending on the context, and thus is configurable as policy. Similarly, the handling of default advice is configurable. Usually, advice is a default constructible value object. In this case, when there isn't any advice solution (yet), a pseudo solution holding the default constructed advice value is used to satisfy any advice access by the client (advised entity). The same can be used when the actual ~AdviceProvision gets //retracted.// As an alternative, when this default solution approach doesn't work, we can provide a policy either to throw or to wait blocking &mdash; but this alternative policy is similarly implemented with a //null object// (a placeholder ~AdviceProvision). Anyway, this implementation technique causes the advice system to collect some advice provisions, bindings and advice objects over time. It should use a pooling custom allocator in the final version. As the number of advisors is expected to be rather small, the storage occupied by these elements, which is effectively blocked until application exit, isn't considered a problem.
!organising the advice solution
This is the tricky part of the whole advice system implementation. A naive implementation will quickly degenerate in performance, as costs are of order ~AdviceProvisions * ~AdviceRequests * (average number of binding terms). But contrary to the standard solutions for rules based systems (either forward or backward chaining), in this case here always complete binding sets are to be matched, which allows to reduce the effort.
!!!solution mechanics
The binding patterns are organised by //predicate symbol and the lists are normalised.// A simple normalisation could be lexicographic ordering of the predicate symbols. Then the resulting representation can be //hashed.// When all predicates are constant, a match can be detected by hashtable lookup, otherwise, in case some of the predicates contain variable arguments ({{red{planned extension}}}), the lookup is followed by an unification. For this to work, we'll have to include the arity into the predicate symbols used in the first matching stage. Moreover, we'll create a //matching closure// (functor object), internally holding the arguments for unification. This approach allows for //actual interpretation of the arguments.// It is conceivable that in special cases we'll get multiple instances of the same predicate, just with different arguments. The unification of these terms needs to consider each possible pairwise combination (cartesian product) &mdash; but working out the details of the implementation can safely be deferred until we'll actually hit such a special situation, thanks to the implementation by a functor.
Fortunately, the calculation of this normalised patterns can be separated completely from the actual matching. Indeed, we don't even need to store the binding patterns at all within the binding index &mdash; storing the hash value is sufficient (and in case of patterns with arguments we'll attach the matching closure functor). Yet still we need to store a marker for each successful match, together with back-links, in order to handle changing and retracting of advice.
!!!storage and registrations
We have to provide dedicated storage for the actual advice provisions and for the index entries. Mostly, these objects to be managed are attached through a single link &mdash; and moreover the advice system is considered performance critical, so it doesn't make sense to implement the management of these entries by smart-ptr. This rules out ~TypedAllocationManager and prompts to write a dedicated storage frontend, later to be backed by Lumiera's mpool facility.
* both the advice provision and the advice requests attach to the advice system after fulfilling some prerequisites; they need to detach automatically on destruction.
* in case of the provision, there is a cascaded relation: the externally maintained provision creates an internal provision record, which in turn attaches an index entry.
* both in case of the provision and the request, the relation of the index bears some multiplicity:
** a multitude of advice provisions can attach with the same binding (which could be the same binding pattern terms, but different variable arguments). Each of them could create a separate advice solution (at least when variable arguments are involved); it would be desirable to establish a defined LIFO order for any search for possibly matching advice.
** a multitude of advice requests can attach with the same binding, and each of them needs to be visited in case a match is detected.
* in both cases, any of these entries could be removed any time on de-registration of the corresponding external entity
* we need to track existing advice solutions, because we need to be able to overwrite with new advice and to remove all solutions bound to a given pattern about to leave the system. One provision could create a large number of solutions, while each registration always holds onto exactly one solution (which could be a default/placeholder solution though)
!!!!subtle variations in semantics
While generally advice has value semantics and there is no ownership or distinguishable identity, the actual implementation technique creates the possibility for some subtle semantic variations. At the time of this writing (5/2010) no external point of reference was available to decide upon the correct implementation variant. These variations get visible when advice is //retracted.// Ideally, a new advisor would re-attach to an existing provision and supersede the contained advice information with new data. Thus, after a chain of such new provisions all attaching with the identical binding, when finally the advice gets retracted, any advice provisions would be gone and we'd fall back onto the default solution. Thus, "retracting" would mean to void any advice given with this binding.
But there is another conceivable variation of semantics, which yields some benefits implementation-wise: Advice can be provided as "I don't care what was said, but here is new information". In this case, the mechanism of resolving and finding a match would be responsible to pick the latest addition, while the provisions would just be dumped into the system. In this case, "retracting" would mean just to cancel //one specific//&nbsp; piece of information and might cause in an older advice solution to be uncovered and revived. The default (empty) solution would be used in this case only after retracting all advice provisions.
!!!!implementation variants with respect to attachment and memory management
Aside from the index, handling of the advice provisions turns out to be tricky.
* management by ref-count was ruled out due to contention and locality considerations
* the most straight forward implementation would be for the ~AdviceProvision within the advisor to keep kind of an "inofficial" link to "its" provision, allowing to modify and retract it during the lifetime of the advisor. When going away without retracting (the default behaviour), the provision, as added into the system would remain there as a dangling entry. It is still reachable via the index, but not maintained in any further way. If memory usage turns out to be a problem, we'd need to enqueue these entries for clean-up.
* but as this simple solution contradicts the general advice semantics in a subtle way (see previous paragraph), we could insist on really re-capturing and retracting previous advice automatically on each new advice provision or modification. In this case, due to the requirement of thread safety, each addition, binding modification, placing of new advice or retraction would require to do an index search to find an existing provision with equivalent binding (same binding definition, not just a matching binding pattern). As a later provision could stomp upon an existing provision without the original advisor noticing this, we can't use the internal references anymore; we really need to search each time and also need a global lock during the modification transaction.
* an attempt to reduce this considerable overhead would be to use a back-link from the provision as added to the system to the original source (the ~AdviceProvision owned by the advisor). On modification, this original source would be notified and thus detached. Of course this is tricky to implement correctly, and also requires locking.
The decision for the initial implementation is to use the first variant and just accept the slightly imprecise semantics.
When copying a Provision, the hidden link to existing advice data is //not shared.//
!!!!de-allocation of advice data
It is desirable that the dtors of each piece of advice data be called eventually. But ensuring this reliably is tricky, because advice
data may be of various types and is added to the system to remain available, even after the original {{{advice::Provision}}} went out of scope. Moreover, the implementation decision was //not//&nbsp; to employ a vtable for the advice collaborators and data holders, so we're bound to invoke the dtor with the correct specific type.
There are some special cases when de-allocation happens while the original provision is still alive (new advice, changed binding, retracting). But in any other case, responsibility for de-allocation has to be taken by the ~AdviceSystem, which unfortunately can't handle the specific type information. Thus the original provision needs to provide a deleter function, and there is no way to avoid storing a function pointer to this deleter within the ~AdviceSystem, together with the advice data holder.
It seems reasonable to create this deleter right away and not to share the link to advice data, when copying a provision, to keep responsibilities straight. {{red{Question: does this even work?? }}} to be verified: does the address of the advice data buffer really determine alone what is found as "existing" provision?
!!!lifecycle considerations
Behind the scenes, hidden within the {{{advice.cpp}}} implementation file, the ~AdviceSystem is maintained as singleton. According to a general lifecycle policy within Lumiera, no significant logic is allowed to execute in the shutdown phase of the application, once the {{{main()}}} has exited. Thus, any advice related operations might throw {{{error::Logic}}} after that point. The {{{~AdviceSystem()}}} also is a good place to free any buffers holding incorporated advice data, after having freed the index datastructure referring to these buffer storage, of course.
!!!!handling of default advice
Basically, the behaviour when requesting non-existing advice may be configured by policy. But the default policy is to return ref to a default constructed instance of the advice type in that case. Just the (implementation related) problem is that we return advice by {{{const&}}}, not by value, so we're bound to create and manage this piece of default advice during the lifetime of the ~AdviceSystem. The way the ~AdviceSystem is accessed (only through the frontend of {{{advice::Request}}} and {{{advice::Provision}}} objects, in conjunction with the desire to control this behaviour by policy, creates a tricky implementation situation.
* regarding the lifecycle (and also from the logical viewpoint) it would be desirable to handle this "default" or "no solution" case similar to accessing an existing solution. But unfortunately doing so would require a fully typed context; thus basically on inserting a new request, when returning from the index search without a dedicated solution, we'd need to fabricate a fallback solution to insert it into the provision index, while still holding the index lock. At that point it is not determined if we ever need that fallback solution. Alternatively we could consider to fabricate this fallback solution on first unsuccessful advice fetch. But this seems sill worse, as it turns a (possibly even lock free) ptr access into an index operation. Having a very cheap advice access seems like an asset.
* on the other hand, using some separate kind of singleton bundle just for these default advice data (e.g. a templated version of //Meyer's Singleton...//), the fallback solution can be settled independent from the ~AdviceSystem, right in the {{{advice.hpp}}} and using static memory, but the downside is now the fallback solution might be destroyed prior to shutdown of the ~AdviceSystem, as it lives in another compilation unit.
Thus the second approach looks favourable, but we should //note the fact that it is hard to secure this possible access to an already destroyed solution,// unless we decline using the advice feature after the end of {{{main()}}}. Such a policy seems to be reasonable anyway, as the current implementation also has difficulties to prevent accessing an already destroyed {{{advice::Provision}}}, being incorporated in the ~AdviceSystem, but accessed through a direct pointer in the {{{advice::Request}}}.
!!!locking and exception safety
The advice system is (hopefully) written such as not to be corrupted in case an exception is thrown. Adding new requests, setting advice data on a provision and any binding change might fail due to exhausted memory. The advice system remains operational in this case, but the usual reaction would be //subsystem shutdown,// because the Advice facility typically is used in a very low-level manner, assuming it //just works.// As far as I can see, the other mutation operations can't throw.
The individual operations on the interface objects are //deliberately not thread-safe.// The general assumption is that {{{advice::Request}}} and {{{advice::Provision}}} will be used in a safe environment and not be accessed or modified concurrently. A notable exception to this rule is accessing Advice: as this just includes checking and dereferentiating a pointer, it might be done concurrently. But note, //the advice system does nothing to ensure visibility of the solution within a separate thread.// If this thread still has the old pointer value in his local cache, it won't pick up the new solution. In case the old solution got retracted, this even might cause access to already released objects. You have been warned. So it's probably a good idea to ensure a read barrier happens somewhere in the enclosing usage context prior to picking up a possibly changed advice solution concurrently.
''Note'': the underlying operations on the embedded global {{{advice::Index}}} obviously need to be protected by locking the whole index table on each mutation, which also ensures a memory barrier and thus propagates changed solutions. While this settles the problem for the moment, we might be forced into a more fine grained locking due to contention prolems later on...
!!!index datastructure
It is clear by now that the implementation datastructure has to serve as a kind of //reference count.// Within this datastructure, any constructed advice solution needs to be reflected somehow, to prevent us from discarding an advice provision still accessible. Allowing lock-free access to the advice solution (planned feature) adds a special twist, because in this case we can't even tell for sure if an overwritten old solution is actually gone (or if its still referred from some thread's cached memery). This could be addressed by employing a transactional approach (which might be good anyway) -- but I tend to leave this special concern aside for now.
To start with, any advice matching and solution will //always happen within matching buckets of a hash based pattern organisation.// The first stage of each access involves using the correct binding pattern, and this binding pattern can be represented within the index data structure by the binding pattern's hash value. Since the advice typing can be translated at interface level into a further predicate within the binding, the use of these binding hashes as first access step also limits access to advice with proper type. All further access or mutation logic takes place within a sub structure corresponding to this top-level hash value. The binding index thus relies on two hashtables (one for the requests and one for the provisions), but using specifically crafted datastructures as buckets. The individual entries within these bucket sub structures in both cases will be comprised of a binding matcher (to determine if a match actually happens) and a back-link to the registered entitiy (provision or request). Given the special pattern of the advice solutions, existing solutions could be tracked within the entries at the request side.
* Advice provisions are expected to happen only in small numbers; they will be searched stack-like, starting from the newes provisions, until a match is found.
* Each advised entity basically creates an advice request, so there could be a larger number of request entries. In the typical search triggered from the provision side, each request entry will be visited and checked for match, which, if successful, causes a pointer to be set within the ~AdviceRequest object (located outside the realm of the advice system). While -- obviously -- multiple requests with similar binding match could be folded into a sub-list, we need actual timing measurements to determine the weight of these two calculation steps of matching and storing, which together comprise the handling of an advice solution.
The above considerations don't fully solve the question how to represent a computed solution within the index data structure, candidates being to use the index within the provision list, or a direct pointer to the provision or even just to re-use the pointer stored into the ~AdviceRequest. My decision is to do the latter. Besides solutions found by matching, we need //fallback solutions// holding a default constructed piece of advice of the requested type. As these defaults aren't correlated at all to the involved bindings, but only to the advice type as such, it seems reasonable to keep them completely apart, like e.g. placing them into static memory managed by the ~AdviceProvision template instantiations.
!!!interactions to be served by the index
[>img[Advice solution|draw/adviceBindingIndex1.png]]
;add request
:check existing provisions starting from top until match; use default solution in case no match is found; publish solution into the new request; finally attach the new request entry
;remove request
:just remove the request entry
;modify request
:handle as if newly added
;add provision
:push new provision entry on top; traverse all request entries and check for match with this new provision entry, publish new solution for each match
;retract provision
:remove the provision entry; traverse all request entries to find those using this provision as advice solution, treat these as if they where newly added requests
;modify provision
:add a new (copy of the) provision, followed by retracting the old one; actually these two traversals of all requests can be combined, thus treating a request which used the old provision but doesn't match the new one is treated like a new request
<<<
__Invariant__: each request has a valid solution pointer set (maybe pointing to a default solution). Whenever such a solution points to a registered provision, there is a match between the index entries and this is the top-most possible match to any provision entry for this request entry
<<<
Clearly, retracting advice (and consequently also the modification) is expensive. After finishing these operations, the old/retracted provision can be discarded (or put aside in case of non-locking advice access). Other operations don't cause de-allocation, as provisions remain within the system, even if the original advising entity is gone.

From analysing a number of intended AdviceSituations, some requirements for an Advice collaboration and implementation can be extracted.
* the piece of advice is //not shared// between advisor and the advised entities; rather, it is copied into storage managed by the advice system
* the piece of advice can only be exposed {{{const}}}, as any created advice point solution might be shared
* the actual mode of advice needs to be configurable by policy &mdash; signals (callback functors) might be used on both sides transparently
* the client side (the advised entity) specifies initially, if a default answer is acceptable. If not, retrieving advice might block or fail
* on both sides, the collaboration is initiated specifying an advice binding, which is an conjunction of predicates, --optionally dynamic--^^no!^^
* there is a tension between matching performance and flexibility. The top level should be entirely static (advice type)
* the analysed usage situations provide no common denominator on the preferences regarding the match implementation.
* some cases require just a match out of a small number of tokens, while generally we might get even a double dispatch
* later, possible and partial solutions could be cached, similar to the rete algorithm. Dispatching a solution should work lock-free
* advice can be replaced by new advice, which causes all matching advice solutions to behave as being overwritten.
* when locking is left out, we can't give any guarantee as to when a given advice gets visible to the advised entity
* throughput doesn't seem to be an issue, but picking up existing advice should be as fast as possible
* we expect a small number of advisors collaborating with and a larger number of advised entities.
!!questions
;when does the advice collaboration actually happen?
:when there is both a client (advised) and a server (advisor) and their advice bindings match
;can there be multiple matches?
:within the system as a whole there can be multiple solutions
:but the individual partners never see more than one connection
:each point of advice has exactly one binding and can establish one advice channel
;but when an attempt is made to transfer more information?
:both sides don't behave symmetrically, and thus the consequences are different
:on the client side, advice is just //available.// When there is newer one, the previous advice is overwritten
:the server side doesn't //contain// advice &mdash; rather, it is placed into the system. After that, the advisor can go away
:thus, if an advisor places new advice into an existing advice provision, this effectively initiates a new collaboration
:if the new advice reaches the same destination, it overwrites; but it may as well reach a different destination this time
;can just one advice provision create multiplicity?
:yes, because of the matching process there could be multiple solutions. But neither the client nor the server is aware of that.
;can advice be changed?
:No. When inserted into the system, the advisor looses any direct connection to the piece of advice (it is copied)
:But an advisor can put up another piece of advice into the same advice provision, thereby effectively overwriting at the destination
;if advice is copied, what about ownership and identity?
:advice has //value semantics.// Thus it has no distinguishable identity beyond the binding used to attach it
:a provision does not "own" advice. It is a piece of information, and the latest information is what counts
;can the binding be modified dynamically?
:this is treated as if retracting the existing point of advice and opening a new one.
;what drives the matching?
:whenever a new point of advice is opened, search for a matching solution happens.
:thus, the actual collaboration can be initiated from both sides
:when a match happens, the corresponding advice point solution gets added into the system
;what about the lifetime of such a solution?
:it is tied to the //referral// &mdash; but there is an asymmetry between server and client
:referral is bound to the server sided / client sided point of advice being still in existence
:but the server sided point of advice is copied into the system, while the client sided is owned by the client
:thus, when an advisor goes away without explicitly //retracting//&nbsp; the advice, any actual solution remains valid
:on the client side there is an asymmetry: actually, a new advice request can be opened, with an exactly identical binding
:in this case, existing connections will be re-used. But any differences in the binding will require searching a new solution
;is the search for an advice point solution exhaustive?
:from the server side, when a new advice provision / binding is put up, //any// possible advice channel will be searched
:contrary to this, at the client side, the first match found wins and will establish an advice channel.
!decisions
After considering the implementation possibilities, some not completely determined requirements can be narrowed down.
* we //do// support the //retracting of advice.//
* there is always an implicit //default advice solution.//
* advice //is not an messaging system// &mdash; no advice queue
* signals (continuations) are acceptable as a extension to be provided later
* retracting advice means to retreat a specific solution. This might or might not uncover earlier solutions (undefined behaviour)
* we don't support any kind of dynamic re-evaluation of the binding match (this means not supporting the placement use case)
* the binding pattern is //interpreted strictly as a conjunction of logic predicates// &mdash; no partial match, but arguments are allowed
* we prepare for a later extension to //full unification of arguments,// and provide a way of accessing the created bindings as //advice parameters.//
Combining all these requirements and properties provides the foundation for the &rarr; AdviceImplementation

[[Advice]] is a pattern extracted from several otherwise unrelated constellations
!Proxy media in the engine
Without rebuilding the engine network, we need the ability to reconfigure some parts to adapt to low resolution place-holder media temporarily. The collaboration required to make this happen seems to ''cross-cut'' the normal processing logic. Indeed, the nature of the adjustments is highly context dependent &mdash; not every processing node needs to be adjusted. There is a dangerous interference with the ongoing render processes, prompting for the possibility to pick up this information synchronously.
* the addressing and delivery of the advice is based on a mix of static (type) and dynamic information
* it is concievable that the actual matching may even include a token present in the direct invocation context (but this possibility was ruled out by later decision)
* the attempt to recieve and pick up advice needs to be failsafe
* locking should be avoided by design
!Dependency injection for testing
While inversion of control is a guiding principle on all levels, the design of the Lumiera application deliberately stays just below the level of employing a dependency injection container. Instead, common services are accessible //by type// and the builder pattern is used more explicitly at places. Interestingly, the impact on writing unit tests was by far not so serious as one might expect, based on the usual reasoning of D.I. proponents. But there remain some situations, where sharing a common test fixture would come in handy
* here the test depending on a fixture puts up a hard requirement for the actual advice to be there.
* thus, the advice support system can be used to communicate a need for advice
* but it seems unreasonable to extend it actually to transmitt a control flow
!properties of placement
The placement concept plays a fundamental role within Lumiera's HighLevelModel. Besides just being a way of sticking objects together and defining the common properties of //temporal position and output destination,// we try to push this approach to enable a more general, open and generic use. "Placement" is understood as locating within a grid comprised of various degrees of freedom &mdash; where locating in a specific way might create additional dimensions to be included into the placement. The standard example is an output connection creating additional adjustable parameters controlling the way the connected object is embedded into a given presentation space (consider e.g. a sound object, which &mdash; just by connection, gains the ability of being //panned// by azimuth, elevation and distance)
* in this case, obviously the colaboration is n:m, while each partner preferrably should only see a single advice link.
* advice is used here to negotiate a direct colaboration, which is then handed off to another facility (wiring a control connection)
* the possibility of an advice colaboration in this case is initiated rather from the side of the advisor
* deriving an advice point solution includes some kind of negotioation or active re-evaluation
* the possible adivsors have to be queried according to their placement scope relations
* this queriying might even trigger a resolution process within the advising placement.
__Note__: after detailed analysis, this use case was deemed beyond the scope of the [[Advice]] core concept and idea.
//As a use case, it was dropped.// But we retain some of the properties discovered by considering this scenario, especially the n:m relation, the symmetry in terms of opening the collaboration, and the possibility to have a specially implemented predicate in the binding pattern.
&rarr; AdviceRequirements

Memory management facility for the low-level model (render nodes network). The model is organised into temporal segments, which are considered to be structurally constant and uniform. The objects within each segment are strongly interconnected, and thus each segment is being built in a single build process and is replaced or released as a whole. __~AllocationCluster__ implements memory management to support this usage pattern. He owns a number of object families of various types.[>img[draw/AllocationCluster.png]]
* [[processing nodes|ProcNode]] &mdash; probably with several subclasses (?)
* [[wiring descriptors|WiringDescriptor]]
* the input/output descriptor arrays used by the latter
To Each of those families we can expect an initially undetermined (but rather large) number of individual objects, which can be expected to be allocated within a short timespan and which are to be released cleanly on destruction of the AllocationCluster.
''Problem of calling the dtors''
Even if the low-level memory manager(s) may use raw storage, we require that the allocated object's destructors be called. This means keeping track at least of the number of objects allocated (without wasting too much memory for bookkeeping). Besides, as the objects are expected to be interconnected, it may be dangerous to destroy a given family of objects while another family of objects may rely on the former in its destructor. //If we happen do get into this situation,// we need to define a priority order on the types and assure the destruction sequence is respected.
&rarr; see MemoryManagement

Asset management is a subsystem on its own. Assets are "things" that can be loaded into a session, like Media, Clips, Effects, Transitions. It is the "bookkeeping view", while the Objects in the Session relate to the "manipulation and process view". Some Assets can be //loaded// and a collection of Assets is saved with each Session. Besides, there is a collection of basic Assets always available by default.
The Assets are important reference points holding the information needed to access external resources. For example, an Clip asset can reference a Media asset, which in turn holds the external filename from which to get the media stream. For Effects, the situation is similar. Assets thus serve two quite distinct purposes. One is to load, list, group search and browse them, and to provide an entry point to create new or get at existing MObject in the Session, while the other purpose is to provide attribute and property information to the inner parts of the engine, while at the same time isolating and decoupling them from environmental details.
We can distinguish several different Kinds of Assets, each one with specific properties. While all these Kinds of Assets implement the basic Asset interface, they in turn are the __key abstractions__ of the asset management view. Mostly, their interfaces will be used directly, because they are quite different in behaviour. Thus it is common to see asset related operations being templated on the Asset Kind.
&rarr; see also [[Creating and registering Assets|AssetCreation]]
[img[Asset Classess|uml/fig130309.png]]
!Media Asset
Some piece of Media Data accessible at some external Location and able to be processed by Lumiera. A Media File on Harddisk can be considered as the most basic form of Media Asset, with some important derived flavours, like a Placeholder for a currently unavailable Source, or Media available in different Resolutions or Formats.
* __outward interface operations__ include querying properties, creating an Clip MObject, controlling processing policy (low res proxy placeholders, interlacing and other generic pre- and postprocessing)
* __inward interface operations__ include querying filename, codec, offset and any other information necessary for creating a source render node, getting additional processing policy decisions (handling of interlacing, aspect ratio).
&rarr; MediaAsset
!Processing Asset
Some software component able to work on media data in the Lumiera Render engine Framework. This includes all sorts of loadable effects, as well as some of the standard, internal facilities (Mask, Projector). Note that Processing Assets typically provide some attachment Point or means of communication with GUI facilities.
* __outward interface operations__ include getting name and description, investigating the media types the processor is able to handle, cause the underlying module to be acutally loaded...
* __inward interface operations__ include resolving the actual processing function.
&rarr; ProcAsset
!Structural Asset
Some of the building blocks providing the framework for the objects placed into the current Session. Notable examples are [[processing pipes|Pipe]] within the high-level-model, Viewer attachment points, Sequences, Timelines etc.
* __outward interface operations__ include...
* __inward interface operations__ include...
&rarr; StructAsset {{red{still a bit vague...}}}
!Meta Asset
Any resources related to the //reflective recurse of the application on itself,// including parametrisation and customisation aspects and similar metadata, are categorised and tracked apart of the primary entities. Examples being types, scales and quantisation grids, decision rules, control data stores (automation data), annotations attached to labels, inventory entities, error items etc.
* __outward interface operations__ include...
* __inward interface operations__ include...
&rarr; MetaAsset {{red{just emerging as of 12/2010}}}
!!!!still to be worked out..
is how to implement the relationship between [[MObject]]s and Assets. Do we use direct pointers, or do we prefer an ID + central registry approach? And how to handle the removal of an Asset.
&rarr; see also [[analysis of mem management|ManagementAssetRelation]]
&rarr; see also [[Creating Objects|ObjectCreation]], especially [[Assets|AssetCreation]]
//9/07: currently implementing it as follows: use a refcounting-ptr from Clip-~MObject to asset::Media while maintaining a dependency network between Asset objects. We'll see if this approach is viable//
{{red{NOTE 8/2018}}} there seems to be a fuzziness surrounding the distinction between StructAsset and MetaAsset.
I am suspicious this is a distinction //merely derived from first principles...// &rarr; {{red{Ticket #1156}}}

Assets are created by a Factories returning smart pointers; the Asset creation is bound to specific use cases and //only available// for these specific situations. There is no generic Asset Factory.
For every Asset we generate a __Ident tuple__ and a long ID (hash) derived from this Ident tuple. The constructor of the abstract base class {{{Asset}}} takes care of this step and automatically registeres the new Asset object with the AssetManager. Typically, the factory methods for concrete Asset classes provide some shortcuts providing sensible default values for some of the Ident tuple data fields. They may take additional parameters &mdash; for example the factory method for creating {{{asset::Media}}} takes a filename (and may at some point in the future aply "magic" based on examination of the file &rarr; LoadingMedia)
Generally speaking, assets can be seen as the statical part or view of the session and model. They form a global scope and are tied to the [[model root|ModelRootMO]] &mdash; which means, they're going to be serialised and de-serialised alongside with this model root scope. Especially the de-serialisation triggers (re)-creation of all assets associated with the session to be loaded.
{{red{TODO:}}} //there will be a special factory mechanism for this case, details pending definition as of 2/2010 //

The Asset Manager provides an Interface to an internal Database holding all Assets in the current Session and System state. It may be a real Database at some point (and for the moment it's a Hashtable). Each [[Asset]] is registered automatically with the Asset Manager; it can be queried either by it's //identification tuple// or by it's unique ID.
Conceptually, assets belong to the [[global or root scope|ModelRootMO]] of the session data model. A mechanism for serialising and de-serialising all assets alongside with the session is planned as of 2/2010

Conceptually, Assets and ~MObjects represent different views onto the same entities. Assets focus on bookkeeping of the contents, while the media objects allow manipulation and EditingOperations. Usually, on the implementation side, such closely linked dual views require careful consideration.
!redundancy
Obviously there is the danger of getting each entity twice, as Asset and as ~MObject. While such dual entities could be OK in conjunction with much specialised processing, in the case of Lumiera's Proc-Layer most of the functionality is shifted to naming schemes, configuration and generic processing, leaving the actual objects almost empty and deprived of distinguishing properties. Thus, starting out from the required concepts, an attempt was made to join, reduce and straighten the design.
* type and channel configuration is concentrated to MediaAsset
* the accounting of structural elements in the model is done through StructAsset
* the object instance handling is done in a generic fashion by using placements and object references
* clips and labels appear as ~MObjects solely; on the asset side there is just an generic [[id tracking mechanism|TypedID]].
* tracks are completely deprived of processing functionality and become lightweight containers, also used as clip bins.
* timelines and sequences are implemented as façade to equivalent structures within the model
* this leaves us only with effects requiring both an object and asset implementation
[<img[Fundamental object relations used in the session|uml/fig138885.png]]

Placing an MObject relatively to another object such that it should be handled as //attached//&nbsp; to the latter results in several design and implementation challenges. Actually, such an attachment creates a cluster of objects. The typical use case is that of an effect attached to a clip or processing pipe.
* attachment is not a globally fixed relation between objects, rather, it typically exists only for some limited time span (e.g. the duration of the basic clip the effect is attached to)
* the order of attachment is important and the attached placement may create a fork in the signal flow, so we need a way for specifying reproducibly how the resulting wiring should be
* when building, we access the information in reversed direction: we have the target object and need to query for all attachments
The first step towards an solution is to isolate the problem; obviously we don't need to store the objects differently, we just need //information about attached objects//&nbsp; for some quite isolated tasks (namely for creating a GUI representation and for combining attached objects into a [[Pipe]] when building). Resorting to a query (function call) interface should turn the rest of the problem into an implementation detail. Thus
* for an __attachment head__ (= {{{Placement<MObject>}}} to which other objects have been attached) get the ordered list of attachments
* for an __attached placement__ (member of the cluster) get the placement of the corresponding attachment head
* retrieve and break the attachment when //deleting.//
!!Implementation notes
Attachment is managed within the participating placements, mostly by special [[locating pins|LocatingPin]]. Attachment doesn't necessarily nail down an attached object to a specific position, rather the behaviour depends on the type of the object and the locating pins actually involved, especially on their order and priority. For example, if an {{{Placement<Effect>}}} doesn't contain any locating pin defining a temporal position, then the attachment will result in the placement inheriting the temporal placement of the //attachment head// (i.e. the clip this effect has been attached to). But, if on the contrary the effect in question //does// have an additional locating pin, for example relative to another object or even to a fixed time position, this one will "win" and determine the start position of the effect &mdash; it may even move the effect out of the time interval covered by the clip, in which case the attachment has no effect on the clip's processing pipe.
The attachment relation is hierarchical and has a clearly defined //active// and //passive// side: The attachment head is the parent node in a tree, but plays the role of the passive partner, to which the child nodes attach. But note, this does not mean we are limited to a single attachment head. Actually, each placement has a list of locating pins and thus can attach to several other placements. For example, a transition attaches to at least two local pipes (clips). {{red{TODO: unresolved design problem; seems to contradict the PlacementScope}}}
!!!!Relation to memory management
Attachment on itself does //not// keep an object alive. Rather, it's implemented by an opaque ID entry (&rarr; PlacementRef), which can be resolved by the PlacementIndex. The existence of attachments should be taken into account when deleting an object, preferably removing any dangling attachments to prevent an exception to be thrown later on. On the other hand, contrary to the elements of the HighLevelModel, processing nodes in the render engine never depend on placements &mdash; they always refer directly to the MObject instance or even the underlying asset. In the case of MObject instances, the pointer from within the engine will //share ownership// with the placement (remember: both are derived from {{{boost::shared_ptr}}}).

Automation is treated as a function over time. It is always tied to a specific Parameter (which can thus be variable over the course of the timeline). All details //how// this function is defined are completely abstracted away. The Parameter uses a ParamProvider to get the value for a given Time (point). Typically, this will use linear or bezier interpolation over a set of keyframes internally. Parameters can be configured to have different value ranges and distribution types (on-off, stepped, continuous, bounded)
[img[how to implement Automation|uml/fig129669.png]]

While generally automation is treated as a function over time, defining and providing such a function requires some //Automation Data.// The actual layout and meaning of this data is deemed an implementation detail of the [[parameter provider|ParamProvider]] used, but nevertheless an automation data set has object characteristics within the session (high-level-model), allowing it to be attached, moved and [[placed|Placement]] by the user.

Starting out from the concepts of Objects, Placement to Tracks, render Pipes and connection properties (&rarr; see [[here|TrackPipeSequence]]) within the session, we can identify the elementary operations occuring within the Builder. Overall, the Builder is organized as application of //visiting tools// to a collection of objects, so finally we have to consider some object kind appearing in the working function of the given builder tool, which holds at this moment some //context//. The job now is to organize this context such as to create a predictable build process from this //event driven// approach.
&rarr;see also: BuilderPrimitives for the elementary situations used to cary out the building operations
!Builder working Situations
# any ''Clip'' (which at this point has been reduced already to a part of a simple elementary media stream &rarr; see [[Fixture]])
## yields a source reading node
## which needs to be augmented by the underlying media's [[processing pattern|ProcPatt]]
##* thus inserting codec(s) and source transformations
##* effectively this is an application of effects
## at this point we have to process (and maybe generate on-the-fly) the [[source port of this clip|ClipSourcePort]]
##* the output of the source reading and preprocessing defined thus far is delivered as input to this port, which is done by a ~WiringRequest (see below)
##* as every port, it is the entry point to a [[processing pipe|Pipe]], thus the source port has a processing pattern, typically inserting the camera (transformation effect) at this point
## followed by the application of effects
##* separately for every effect chain rooted (placed) directly onto the clip
##* and regarding the chaining order
## next we have to assess the [[pipes|Pipe]] to which the clip has been placed
## producing a [[wiring request|WiringRequest]] for every pair {{{(chainEndpoint, pipe)}}}
# [>img[draw/Proc.builder1.png]] attaching an ''Effect'' is actually always an //insertion operation// which is done by //prepending// to the previously built nodes. Effects may be placed as attached to clips and pipes, which causes them to be included in the processing chain at the given location. Effects may as well be placed at an absolute time, which means they are to be applied to every clip that happens to be at this time &mdash; but this usecase will be reolved when creating the Fixture, causing the effect to be attached to the clips in question. The same holds true for Effects put on tracks.
# treating an ''wiring request'' means
## detecting possible and impossible connections
## deriving additional possible "placement dimensions" generated by executing such an connection (e.g. connecting a mono source to a spatial sound system bus creates panning possibilities)
##* deriving parameter sources for this additional degrees of freedom
##* fire off insertion of the necessary effects to satisfy this connection request and implement the additional "placement dimensions" (pan, layer order, overlay mode, MIDI channel selection...)
# processing the effects and further placements ''attached to a Pipe'' is handled identical to the processing done with all attachments to individual clips.
# ''Transitions'' are to be handled differently according to their placement (&rarr; more on [[Transitions|TransitionsHandling]])
#* when placed normally to two (or N) clips, they are inserted at the exit node of the clip's complete effect chain.
#* otherwise, when placed to the source port(s) or when placed to some other pipes they are inserted at the exit side of those pipe's effect chains. (Note: this puts additional requirements on the transition processor, so not every transition can be placed this way)
After consuming all input objects and satisfying all wiring requests, the result is a set of [[exit nodes|ExitNode]] ready for pulling data. We call the network reachable from such an exit node a [[Processor]], together all processors of all segments and output data types comprise the render engine.
!!!dependencies
Pipes need to be there first, as everything else will be plugged (placed) to a pipe at some point. But, on the other hand, for the model as such, pipes are optional: We could create sequences with ~MObjects without configuring pipes (but won't be able then to build any render processor of course). Similarily, there is no direct relation between tracks and pipes. Each sequence is comprised of at least one root track, but this has no implications regarding any output pipe.
Effects can be attached only to already existing pipelines, starting out at some pipes entry port or the source port of some clip. Besides that, all further parts can be built in any order and independent of each other. This is made possible by using [[wiring requests|WiringRequest]], which can be resolved later on. So, as long as we start out with the tracks (to resolve any pipe they are placed to), and further, if we manage to get any effect placed to some clip-MO //after// setting up and treating the clip, we are fine and can do the building quasi event driven.
!!!building and resolving
Building the network for the individual objects thus creates a queue of wiring requests. Some of them may be immediately resolvable, but detecting this correctly can be nontrivial, and so it seems better to group all wiring requests based on the pipe and treat them groupwise. Because &mdash; in the most general case &mdash; connecting includes the use of transforming and joining nodes, which can create additional wiring requests (e.g. for automation parameter data connections). Finally, if the network is complete, we could perform [[optimisations|RenderNetworkOptimisation]]

Binding-~MObjects are used to associate two entities within the high-level model.
More specifically, such a binding serves
* to outfit any top-level [[Timeline]] with real content, which is contained within a [[Sequence]]
* to build a VirtualClip, that is to link a complete sequence into another sequence, where it appears like a new virtual media or clip.
!Properties of a Binding
Binding is a relation entity, maintaining a link between parts of the session. Actually this link is achieved somewhat indirect: The binding itself is an MObject, but it points to a [[sequence asset|Sequence]]. Moreover, in case of the (top-level) timelines, there is a timeline asset acting as a frontend for the ~BindingMO.
* the binding exposes special functions needed to implement the timeline {{red{planned as of 11/10}}}
* similarly, the binding exposes functions allowing to wrap up the bound sequence as VirtualMedia (when acting as VirtualClip).
* the Binding holds an OutputMapping -- allowing to specify, resolve and remember [[output designations|OutputDesignation]]
Note: there are other binding-like entities within the model, which are deliberately not subsumed below this specification, but rather implemented stand alone.
&rarr; see also SessionInterface
!Implementation
On the implementation side, we use a special kind of MObject, acting as an anchor and providing an unique identity. Like any ~MObject, actually a placement establishes the connection and the scope, and typically constitutes a nested scope (e.g. the scope of all objects //within// the sequence to be bound into a timeline)
Binding can be considered an implementation object, rarely to be created directly. Yet it is part of the high-level model
{{red{WIP 11/10}}}: it is likely that -- in case of creating a VirtualClip -- BindingMO will be hooked up behind another façade asset, acting as ''virtual media''
!!!channel / output mapping {{red{WIP 11/10}}}
The Binding-~MObject stores an OutputMapping. Basically this, together with OutputDesignation, implements the mapping behaviour
* during the build process, output designation(s) will be retrieved for each pipe.
* indirect and relative designations are to resolved; the relative ones are forwarded to the next enclosing binding
* in any case, the result is an direct WiringPlug, which can then be matched up and wired
* but in case of implementing a virtual clip, in addition to the direct wiring...
** a relative output designation (the N^^th^^ channel of this kind) is carried over to the target scope to be re-resolved there.
** any output designation specification yields a summation pipe at the binding, i.e. a position corresponding to the global pipes when using the same sequence as timeline.
** The output of these summation pipes is treated like a media channels
** but for each of those channels, an OutputDesignation is //carried over//&nbsp; into the target (virtual clip)
*** now, if a distinct output designation can be determined at this placement of the virtual clip, it will be used for the further routing
*** otherwise, we try to re-evaluate the original output designation carried over. <br/>Thus, routing will be done as if the original output designation was given at this placement of the virtual clip.

There is some flexibility in the HighLevelModel, allowing to attach the same [[Sequence]] onto multiple [[timelines|Timeline]] or even into a [[meta-clip|VirtualClip]]. Thus, while there is always an containment relation which can be used to define the current PlacementScope, we can't always establish an unique path from any given location up to the model root. In the most general case, we have to deal with a DAG, not a tree.
!solution idea
Transform the DAG into a tree by //focussing//&nbsp; on the current situation and context. Have a state containing the //current path.// &rarr; QueryFocus
Incidentally, this problem is quite similar to file system navigation involving ''symlinks''.
* under which circumstances shall discovery follow symlinks?
* where do you get by {{{cd ..}}} &mdash; especially when you went down following a symlink?
This leads us to building our solution here to match a similar behaviour pattern, according to the principle of least surprise. That is, disovery shall follow the special [[bindings|BindingMO]] only when requested explicitly, but the current "shell" (QueryFocus) should maintain a virtual/effective path to root scope.
!!detail decisions
!!!!how to represent scoping
We us a 2-layer approach: initially, containment is implemented through a registration in the PlacementIndex. Building on that, scope is layered on top as an abstraction, which uses the registered containment relation, but takes the current access path into account at the critical points (where a [[binding|BindingMO]] comes into play)
!!!!the basic containment tree
each Placement (with the exception of the root) gets registered as contained in yet another Placement. This registration is entirely a tree, and thus differs from the real scope nesting at the Sequence level: The scopes constituting Sequences and Timelines are registered as siblings, immediately below the root. This has some consequences
# Sequences as well as Timelines can be discovered as contents of the model root
# ScopePath digresses at Sequence level from the basic containment tree
!!!!locating a placement
constituting the effective logical position of a placement poses sort-of a chicken or egg problem: We need already a logical position to start with. In practice, this is done by recurring on the QueryFocus, which is a stack-like state and automatically follows the access or query operations on the session. //Locating a placement//&nbsp; is done by //navigating the current query focus.//
!!!!navigating a scope path location
As the current query focus stack top always holds a ScopePath, the ''navigating operation'' on ScopePath is the key for managing this logical view onto the "current" location.
* first, try to locate the new scope in the same sequence as the current scope, resulting in a common path prefix
* in case the new scope belongs to a different sequence, this sequence might be connected to the current one as a meta-clip, again resulting in a common prefix
* otherwise use the first possible binding according to the ordering of timelines as a prefix
* use the basic containment path as a fallback if no binding exists
!!{{red{WIP 9/10}}}Deficiencies
To buy us some time, analysing and implementing the gory details of scope path navigation and meta-clips was skipped for now.
Please note the shortcomings and logical contradictions in the solution currently in code:
* the {{{ScopePath::navigate()}}}-function was chosen as the location to implement the translation logic. //But actually this translation logic is missing.//
* so basically we're just using the raw paths of the basic containment tree; more specifically, the BindingMO (=Timeline) isn't part of the derived ScopePath
* this will result in problems even way before we implement meta-clips (because the Timeline is assumed to provide output routing information) to the Placements
* QueryFocus, with the help of ScopeLocator exposes the query services of the PlacementIndex. So actually it's up to the client code to pick the right functions. This might get confusing
* The current design rather places the implementation according to the roles of the involved entities, which causes some ping-pong on the implementation level. Especially the ScopeLocator singleton can be accessed multiple times. This is the usual clarity vs. performance tradeoff. Scope resolution is assumed rather to be //not performance critical.//

All rendering, transformations and output of media data requires using ''data buffers'' -- but the actual layout and handling of these buffers is closely related to the actual implementation of these operations. As we're relying heavily on external libraries and plug-ins for performing these operations, there is no hope getting away with just one single {{{Buffer}}} data type definition. Thus, we need to confine ourselves to a common denominator of basic operations regarding data buffers and abstract the access to these operations through a BufferProvider entity. Beyond these basic operations, mostly we just need to assure that //a buffer exists as an distinguishable element// -- which in practice boils down to pushing around {{{void*}}} variables.
Obviously, overloading a pointer with semantic meaning isn't exactly a brilliant idea -- and the usual answer is to embed this pointer into a smart handle, which also yields the nice side-effect of explaining this design to the reader. Thus a buffer handle
* can only be obtained from a BufferProvider
* can be used to identify a buffer
* can be dereferenced
* can be copied
!design quest: buffer type information
To perform anything useful with such a buffer handle, the client code needs some additional information, which can be generalised into a //type information:// Either, the client needs to know the size and kind of data to expect in the buffer, maybe just assume to get a specific buffer with suitably dimensions, or the client needs to know which buffer provider to contact for any management operations on that buffer (handle). And, at some point there needs to be a mechanism to verify the validity of a handle. But all of this doesn't mean that it's necessary to encode or embedd this information directly into the handle -- it might also be stored into a registration table (which has the downside of creating contention), or it might just be attached implicitly to the invocation context.
Just linking this type information to the context is certainly the most elegant solution, but also by far the most difficult to achieve -- not to mention the implicit dependency on a very specific invocation situation. So for now (9/2011) it seems best to stick to the simple and explicit implementation, just keeping that structural optimisation in mind. And the link to this buffer type information should be made explicit within the definition anyway, even if we choose to employ another design tradeoff later.
* thus the conclusion is: we introduce a ''descriptor object'', which will be stored within the handle
* each BufferProvider exposes a ''descriptor prototype''; it can be specialised and used by to [[organise implementation details|BufferMetadata]]
!sanity checks
there are only limited sanity checks, and they can be expected to be optimised away for production builds.
Basically the client is responsible for sane buffer access.

Buffers are used to hold the media data for processing and output. Within the Lumiera RenderEngine and [[Player]] subsystem, we use some common concepts to handle the access and allocation of working buffers. Yet this doesn't imply having only one central authority in charge of every buffer -- such an approach wouldn't be possible (due to collaboration with external systems) and wouldn't be desirable either. Rather, there are some common basic usage //patterns// -- and there are some core interfaces used throughout the organisation of the rendering process.
Mostly, the //client code,// i.e. code in need of using buffers, can access some BufferProvider, thereby delegating the actual buffer management. This binds the client to adhere to kind of a //buffer access protocol,// comprised of the ''announcing'', ''locking'', optionally ''attaching'' and finally the ''releasing'' steps. Here, the actual buffer management within the provider is a question of implementation and will be configured during build-up of the scope in question.
!usage situations
;rendering
:any calculations and transformations of media data typically require an input- and output buffer. To a large extent, these operations will be performed by specialised libraries, resulting in a call to some plain-C function receiving pointers to the required working buffers. Our invocation code has the liability to prepare and provide those pointers, relying on a BufferProvider in turn.
;output
:most any of the existing libraries for handling external output require the client to adhere to some specific protocol. Often, this involves some kind of callback invoked at the external library's discretion, thus forcing our engine to prepare data within an intermediary buffer. Alternatively, the output system might provide some mechanism to gain limited direct access to the output buffers, and such an access can again be exposed to our internal client code through the BufferProvider abstraction.
!primary implementations
;memory pool
:in all those situations, where we just need a working buffer for some time, we can rely on our internal custom memory allocator.
:{{red{~Not-Yet-Implemented as of 9/11}}} -- as a fallback we just rely on heap allocations through the language runtime
;frame cache
:whenever a calculated result may be of further interest, beyond the immediate need triggering the calculation, it might be eligible for caching.
:The Lumiera ''frame cache'' is a special BufferProvider, maintaining a larger pool of buffers which can be pinned and kept around for some time,
:accomodating limited resources and current demand for fresh result buffers.

the generic BufferProvider implementation exposes a service to attach and maintain additional metadata with individual buffers. Using this service is not mandatory -- a concrete buffer provider implementation may chose to maintain more specific metadata right on the implementation level, especially if more elaborate management is necessary within the implementation anyway (e.g. the frame index). We can expect most buffer provider implementations to utilise at least the generic buffer type id service though.
!buffer types and descriptors
Client code accesses buffer through [[smart buffer handles|BuffHandle]], including some kind of buffer type information, encoded into a type ID within the ''buffer descriptor''. These descriptors are used like prototypes, relating the type-~IDs hierarchically. Obviously, the most fundamental distinction is the BufferProvider in charge for that specific buffer. Below that, the next mandatory level of distinction is the ''buffer size''. In some cases, additional distinctions can be necessary. Each BufferProvider exposes a service to yield unique type ~IDs governed by such a hierarchical scheme.
!state and metadata for individual buffers
Beyond that, it can be necessary to associate at least a state flag with //individual buffers.// Doing so requires the buffer to be in //locked state,// otherwise it wouldn't be distinguishable as an separate entity (a client is able to access the buffer memory address only after "locking" this buffer). Especially when using a buffer provider in conjunction with an OutputSlot, these states and transitions are crucial for performing an orderly handover of generated data from the producer (render engine) to the consumer (external output sink).
__Note__: while the API to access this service is uniform, conceptually there is a difference between just using the (shared) type information and associating individual metadata, like the buffer state. Type-~IDs, once allocated, will never be discarded (within the lifetime of an Lumiera application instance -- buffer associations aren't persistent). To the contrary, individual metadata //will be discarded,// when releasing the corresponding buffer. According to the ''prototype pattern'', individual metadata is treated as a one-way-off specialisation.

It turns out that -- throughout the render engine implementation -- we never need direct access to the buffers holding actual media data. Buffers are just some entity to be //managed,// i.e. "allocated", "locked" and "released"; the //actual meaning of these operations can be left to the implementation.// The code within the render engine just pushes around ''smart-prt like handles''. These [[buffer handles|BuffHandle]] act as a front-end, being created by and linked to a buffer provider implementation. There is no need to manage the lifecycle of buffers automatically, because the use of buffers is embedded into the render calculation cycle, which follows a rather strict protocol anyway. Relying on the [[capabilities of the scheduler|SchedulerRequirements]], the sequence of individual jobs in the engine ensures...
* that the availability of a buffer was ensured prior to planning a job ("buffer allocation")
* that a buffer handle was obtained ("locked") prior to any operation requiring a buffer
* that buffers are marked as free ("released") after doing the actual calculations.
!operations
While BufferProvider is an interface meant to be backed by various different kinds of buffer and memory management approaches, there is a common set of operations to be supported by any of them
;announcing
:client code may announce beforehand that it expects to get a certain amount of buffers. Usually this causes some allocations to happen right away, or it might trigger similar mechanisms to ensure availability; the BufferProvider will then return the actual number of buffers guaranteed to be available. This announcing step is optional an can happen any time before or even after using the buffers and it can be repeated with different values to adjust to changing requirements. Thus the announced amount of buffers always denotes //additional buffers,// on top of what is actively used at the moment. This safety margin of available buffers usually is accounted separately for each distinct kind of buffer (buffer type). There is no tracking as to which specific client requested buffers, beyond the buffer type.
;locking
:this operation actually makes a buffer available for a specific client and returns a [[buffer handle|BuffHandle]]. The corresponding buffer is marked as used and can't be locked again unless released. If necessary, at that point the BufferProvider might allocate memory to accommodate (especially when the buffers weren't announced beforehand). The locking may fail and raise an exception. You may expect failure to be unlikely when buffers have been //announced beforehand.// To support additional sanity checks, the client may provide a token-ID with the lock-operation. This token may be retrieved later and it may be used to ensure the buffer is actually locked for //this token.//
;attaching
:optionally the client may attach an object to a locked buffer. This object is placement-constructed into the buffer and will be destroyed automatically when releasing the buffer. Alternatively, the client may provide a pair of constructor- / destructor-functors, to be invoked in a similar way. This allows e.g. to install descriptor structures within the buffer, as required by an external library.
;releasing
:buffers need to be released explicitly by the client code. This renders the corresponding BuffHandle invalid, (optionally) invokes a destructor function of an attached object and maybe reclaims the buffer memory
!!type metadata service
In addition to the basic operations, clients may associate BufferMetadata with individual buffers;
in the basic form, this means just maintaining a type tag describing the kind of buffer, while optionally this service might be extended to e.g. associating a state flag.
__see also__
&rarr; OutputSlot relying on a buffer provider to deal with frame output buffers
&rarr; more about BufferManagement within the RenderEngine and [[Player]] subsystem
&rarr; RenderMechanics for details on the buffer management within the node invocation for a single render step

The invocation of individual [[render nodes|ProcNode]] uses an ''buffer table'' internal helper data structure to encapsulate technical details of the allocation, use, re-use and feeing of data buffers for the media calculations. Here, the management of the physical data buffers is delegated through a BufferProvider, which typically is implemented relying on the ''frame cache'' in the backend. Yet some partially quite involved technical details need to be settled for each invocation: We need input buffers, maybe provided as external input, while in other cases to be filled by a recursive call. We need storage to prepare the (possibly automated) parameters, and finally we need a set of output buffers. All of these buffers and parameters need to be rearranged for invoking the (external) processing function, followed by releasing the input buffers and commiting the output buffers to be used as result.
Because there are several flavours of node wiring, the building blocks comprising such a node invocation will be combined depending on the circumstances. Performing all these various steps is indeed the core concern of the render node -- with the help of BufferTable to deal with the repetitive, tedious and technical details.
!requirements
The layout of the buffer table will be planned beforehand for each invocation, allongside with planning the individual invocation jobs for the scheduler. At that point, a generic JobTicket for the whole timeline segment is available, describing the necessary operations in an abstract way, as determined by the preceeding planning phase. Jobs are prepared chunk wise, some time in advance (but not all jobs of at once). Jobs will be executed concurrently. Thus, buffer tables need to be created repeatedly and placed into a memory block accessed and owned exclusively by the individual job.
* within the buffer table, we need a working area for the output handles, the input handles and the parameter descriptors
* actually, these can be seen as pools holding handle objects which might even be re-used, especially for a chain of effects calculated in-place.
* each of these pools is characterised by a common //buffer type,// represented as buffer descriptor
* we need some way to integrate with the StateProxy, because some of the buffers need to be marked especially, e.g. as result
* there should be convenience functions to release all pending buffers, forwarding the release operation to the individual handles

//Building the fixture is actually at the core of the [[builder's operation|Builder]]//
{{red{WIP as of 11/10}}} &rarr; see also the [[planning page|PlanningBuildFixture]]
;Resolving the DAG[>img[Steps towards creating a Segmentation|draw/SegmentationSteps1.png]]
Because of the possibility of binding a Sequence multiple times, and maybe even nested as virtual clip, the [[high-level model|HighLevelModel]] actually constitutes a DAG, not a tree. This leds to quite some tricky problems, which we try to resolve by //rectifying the DAG into N virtual trees.// (&rarr; BindingScopeProblem)
Relying on this transformation, each Timeline spans a sub-tree virtually separated from all other timelines; the BuildProcess is driven by [[visiting|VisitorUse]] all the //tangible// objects within this subtree. In the example shown to the right, Sequence-β is both bound as VirtualClip into Sequence-α, as well as bound independently as top-level sequence into Timeline-2. Thus it will be visited twice, but the QueryFocus mechanism ensures that each visitation »sees« the proper context.
;Explicit Placements
Each tangible object placement (relevant for rendering), which is encountered during that visitation, gets //resolved// into an [[explicit placement|ExplicitPlacement]]. If we see [[Placement]] as a positioning within a multi dimensional configuration space, then the resolution into an explicit placement is like the creation of an ''orthogonal base'': Within the explicit placement, each LocatingPin corresponds exactly to one degree of freedom and can be considered independently from all other locating pins. This resolution step removes any fancy dynamic behaviour and all scoping and indirect references. Indeed, an explicit placement is a mere //value object;// it isn't part of the session core (PlacementIndex), isn't typed and can't be referred indirectly.
;Segmentation of Time axis
This simple and explicit positioning thus allows to arrange all objects as time intervals on a single axis. Any change and especially any overlap is likely to create a different wiring configuration. Thus, for each such configuration change, we fork off a new //segment// and //copy over// all partially touched placements. The resulting seamless sequence of non-overlapping time intervals provides the backbone of the datastructure called [[Fixture]].
;Building the Network
From this backbone, the actual [[building mechanism|BuilderMechanics]] proceeds as a ongoing visitation and resolution, resulting in the gowth of a network of [[render nodes|ProcNode]] starting out from the source reading nodes and proceeding up through the local pipes, the transitions and the global pipes. When this build process is exhausted, besides the actual network, the result is a //residuum of nodes not connected any further.// Any of these [[exit nodes|ExitNode]] can be associated to a ~Pipe-ID in the high-level model. Within each segment, there should be one exit node per pipe-ID at max. These are the [[model ports|ModelPort]] resulting from the build process, keyed by their corresponding ~Pipe-ID.
&rarr; see [[Structure of the Fixture|Fixture]]

All decisions on //how // the RenderProcess has to be carried out are concentrated in this rather complicated Builder Subsystem. The benefit of this approach is, besides decoupling of subsystems, to keep the actual performance-intensive video processing code as simple and transparent as possible. The price, in terms of increased complexity &mdash; to pay in the Builder &mdash; can be handled by making the Build Process generic to a large degree. Using a Design By Contract approach we can decompose the various decisions into small decision modules without having to trace the actual workings of the Build Process as a whole.
[>img[Outline of the Build Process|uml/fig129413.png]]
The building itself will be broken down into several small tool application steps. Each of these steps has to be mapped to the MObjects found on the [[Timeline]]. Remember: the idea is that the so called "[[Fixture]]" contains only [[ExplicitPlacement]]s which in turn link to MObjects like Clips, Effects and [[Automation]]. So it is sufficient to traverse this list and map the build tools to the elements. Each of these build tools has its own state, which serves to build up the resulting Render Engine. So far I see two steps to be necessary:
* find the "Segments", i.e. the locations where the overall configuration changes
* for each segment: generate a ProcNode for each found MObject and wire them accordingly
Note, //we still have to work out how exactly building, rendering and playback work// together with the backend-design. The build process as such doesn't overly depend on these decisions. It is easy to reconfigure this process. For example, it would be possible as well to build for each frame separately (as Cinelerra2 does), or to build one segment covering the whole timeline (and handle everything via [[Automation]]
&rarr;see also: [[Builder Overview|Builder]]
&rarr;see also: BasicBuildingOperations
&rarr;see also: BuilderStructures
&rarr;see also: BuilderMechanics
&rarr;see also: PlanningBuildFixture
&rarr;see also: PlanningSegementationTool
&rarr;see also: PlanningNodeCreatorTool
[img[Colaborations in the Build Process|uml/fig128517.png]]

Actually setting up and wiring a [[processing node|ProcNode]] involves several issues and is carried out at the lowest level of the build process.
It is closely related to &rarr; [[the way nodes are operated|NodeOperationProtocol]] and the &rarr; [[mechanics of the render process|RenderMechanics]]
!!!object creation
The Nodes are small polymorphic objects, carrying configuration data, but no state. They are [[specially allocated|ManagementRenderNodes]], and the object creation is accessible by means of the NodeFactory solely. They //must not be deallocated manually.// The decision of what concrete node type to create depends on the actual build situation and is worked out by the combination of [[mould|BuilderMould]] and [[processing pattern|ProcPatt]] at the current OperationPoint, issuing a call to one of NodeFactory's {{{operator()}}}
!!!node, plugin and processing function
Its a good idea to distinguish clearly between those concepts. A plugin is a piece of (possibly external) code we use to carry out operations. We have to //discover its properties and capabilities.// We don't have to discover anything regarding nodes, because we (Lumiera builder and renderengine) are creating, configuring and wiring them to fit the specific purpose. Both are to be distinguished from processing functions, which do the actual calculations on the media data. Every node typically encompasses at least one processing function, which may be an internal function in the node object, a library function from Lumiera or GAVL, or external code loaded from a plugin.
!!!node interfaces
As a consequence of this distinctions, in conjunction with a processing node, we have to deal with three different interfaces
* the __build interface__ is used by the builder to set up and wire the nodes. It can be full blown C++ (including templates)
* the __operation interface__ is used to run the calculations, which happens in cooperation of Proc-Layer and Backend. So a function-style interface is preferable.
* the __inward interface__ is accessed by the processing function in the course of the calculations to get at the necessary context, including in/out buffers and param values.
!!!wiring data connections
A node //knows its predecessors, but not its successors.// When being //pulled//&nbsp; in operation, it can expect to get a frame provider for accessing the in/out buffer locations (some processing functions may be "in-place capable", but that's only a special case of the former). At this point, the ''pull principle'' comes into play: the node may request input frames from the frame provider, passing its predecessors as a ''continuation''.
With regard to the build process, the wiring of data connections translates into providing the node with its predecessors and preconfiguring the possible continuations. While in the common case, a node has just one input/output and pulls from its predecessor a frame for the same timeline position, the general case can be more contrived. A node may process N buffers in parallel and may require several different time positions for it's input, even at a differing framerate. So the actual source specification is (predNode,time,frameType). The objective of the wiring done in the build process is to factor out the parts known in advance, while in the render process only the variable part need to be filled in. Or to put it differently: wiring builds a higher order function (time)->(continuation), where continuation can be invoked to get the desired input frame.
!!!wiring control conections
In many cases, the parameter values provided by these connections aren't frame based data, rather, the processing function needs a call interface to get the current value (value for a given time), which is provided by the parameter object. Here, the wiring needs to link to the suitable parameter instance, which is located within the high-level model (!). As an additional complication, calculating the actual parameter value may require a context data frame (typically for caching purposes to speed up the interpolation). While these parameter context data frames are completely opaque for the render node, they have to be passed in and out similar to the state needed by the node itself, and the wiring has to prepare for accessing these frames too.

The Builder takes some MObject/[[Placement]] information (called Timeline) and generates out of this a Render Engine configuration able to render this Objects. It does all decisions and retrieves the current configuration of all objects and plugins, so the Render Engine can just process them stright forward.
The Builder is the central part of the [[Builder Pattern|http://en.wikipedia.org/wiki/Builder_pattern]]
<br/>
As the builder [[has to create a render node network|BuilderModelRelation]] implementing most of the features and wiring possible with the various MObject kinds and placement types, it is a rather complicated piece of software. In order to keep it manageable, it is broken down into several specialized sub components:
* clients access builder functionality via the BuilderFacade
* the [[Proc-Layer-Controller|Controller]] initiates the BuildProcess and does the overall coordination of scheduling edit operations, rebuilding the fixture and triggering the Builder
* to carry out the building, we use several primary tools (SegmentationTool, NodeCreatorTool,...), together with a BuilderToolKit to be supplied by the [[tool factory|BuilderToolFactory]]
* //operating the Builder// can be viewed at from two different angles, either emphasizing the [[basic building operations|BasicBuildingOperations]] employed to assemble the render node network, or focussing rather at the [[mechanics|BuilderMechanics]] of cooperating parts while processing.
* besides, we can identify a small set of elementary situations we call [[builder primitives|BuilderPrimitives]], to be covered by the mentioned BuilderToolKit; by virtue of [[processing patterns|ProcPatt]] they form an [[interface to the rule based configuration|BuilderRulesInterface]].
* the actual building (i.e. the application of tools to the timeline) is done by the [[Assembler|BuilderAssembler]], which is basically a collection of functions (but has a small amount of global configuration state)
* any non-trivial wiring of render nodes, forks, pipes and [[automation|Automation]] is done by the services of the [[connection manager|ConManager]]

The cooperation of several components creates a context of operation for the primary builder working tool, the [[node creator|PlanningNodeCreatorTool]]:
* the BuilderToolFactory acts as the "builder for the builder tools", i.e. we can assume to be able to retrive all needed primary tools and elementary tools from this factory, completely configured and ready to use.
* the [[Assembler|BuilderAssembler]] has the ability to consume objects from the high level model and feed them to the node creator (which translates into a dispatch of individual operations suited to the objects to be treated). This involves some sort of scheduling or ordering of the operaions, which is the only means to direct the overall process such as to create a sensible and usable result. //This is an fundamental design decision:// the actual working tools have no hard wired knowledge of the "right process", which makes the whole Builder highly configurable ("open").
* the [[connection manager|ConManager]] on the contrary is a passive service provider. Fed with [[wiring requests|WiringRequest]], he can determine if a desired connection is possible, and what steps to take to implement it; the latter recursively creates further building requests to satisfy by the assembler, and possibly new wiring requests.
!!pattern of operation
The working pattern of this builder mechanics can be described as triggering, enqueuing, priorizing, recursing and exhausting. Without the priorizing part, it would be a depth-first call graph without any context state, forcing us to have all cross reference information available at every node or element to be treated. We prefer to avoid this overhead by ordering the operations into several phases and within these phases into correlated entities with the help of a ''weighting function'' and scheduling with a ''priority queue''
!!call chain
After preparing the tools with the context state of this build process, the assembler drives the visitation process in the right order. The functions embedded within the visitor (NodeCreatorTool) for treating specific kinds of objects in turn use the toolkit (=the fully configured tool factory) to get the mould(s) for the individual steps they need to carry out. This involves preparing the mould (with the high-level object currently in-the-works, a suitable processing pattern and additional references), followed by operating the mould. The latter "plays" the processing pattern in the context of the mould, which, especially with the help of the operation point, carries out the actual building and/or connecting step. While doing so, the node factory will be invoked, which in turn invokes the wiring factory and thus pre-determines the node's prospective mode of operation when later called for rendering.

[>img[Builder creating the Model|uml/fig132868.png]]

The [[Builder]] uses different kinds of tools for creating a network of render nodes from a given high-level model. When breaking down this (necessarily complex) process into small manageable chunks, we arrive at [[elementary building situations|BuilderPrimitives]]. For each of these there is a specialized tool. We denote these tools as "moulds" because they are a rather passive holder for the objects to be attached and wired up. They are shaped according to the basic form the connections have to follow for each of these basic situations:
* attaching an effect to a pipe
* combining pipes via a transition
* starting out a pipe from a source reader
* general connections from the exit node of a pipe to the port of another pipe
In all those cases, the active part is provided by [[processing patterns|ProcPatt]] &mdash; sort of micro programs executed within the context of a given mould: the processing pattern defines the steps to take (in the standard/basic case this is just "attach"), while the mould holds and provides the location where these steps will operate. Actually, this location is represented as a OperationPoint, provided by the mould and abstracting the details of making multi-channel connections.

While assembling and building up the render engines node network, a small number of primitive building situations is encountered repeatedly. The BuilderToolKit provides a "[[mould|BuilderMould]]" for each of these situations, typically involving parametrisation and the application of a [[processing pattern|ProcPatt]].
The ''Lifecycle'' of such a mould starts out by arming it with the object references involved into the next building step. After conducting this building step, the resulting render nodes can be found &mdash; depending on the situation &mdash; attached either to the same mould, or to another kind of mould, but in any case ready to be included in the next building step. Thus, //effectively//&nbsp; the moulds are //used to handle the nodes being built,// due to the fact that the low-level model (nodes to be built) and the high-level model (objects directing what is to be built) are //never connected directly.//
!List of elementary building situations
!!!inserting an Effect or Plugin
[>img[draw/builder-primitives1.png]]
The __~PipeMould__ is used to chain up the effects attached to a clip (=local pipe) or global pipe (=bus)
* participating: a Pipe and an Effect
* point of reference: current exit node of the pipe
* result: Effect appended at the pipe's exit node
* returns: ~PipeMould holding onto the new exit node
@@clear(right):display(block):@@
!!!attaching a transition
[>img[draw/builder-primitives2.png]]
After having completed N pipe's node chains, a __~CombiningMould__ can be used to join them into a [[transition|TransitionsHandling]]
* participating: N pipe's exit nodes, transition
* point of reference: N exit nodes corresponding to (completed) pipes
* result: transition has been attached with the pipe's exit nodes, new wiring requests created attached to the transition's exit node(s)
* returns: ~WiringMould, connected with the created wiring request
Using this mould implicitly "closes" the involved pipes, which means that we give up any reference to the exit node and can't build any further effect attached to this pipes. Generally speaking, "exit node" isn't a special kind of node, rather it's a node we are currently holding on. Similarly, there is nothing directly correlated to a pipe within the render nodes network after we are done with building the part of the network corresponding to the pipe; the latter serves rather as a blueprint for building, but isn't an entity in the resulting low-level model.
Actually, there is {{red{planned}}} a more general (and complicated) kind of transition, which can be inserted into N data connections without joining them together into one single output, as the standard transitions do. The ~CombiningMould can handle this case too by just returning N wiring moulds as a result.
@@clear(right):display(block):@@
!!!building a source connection
[>img[draw/builder-primitives3.png]]
The __~SourceChainMould__ is used as a starting point for any further building, as it results in a local pipe (=clip) rooted at the clip source port. This reflects the fact that the source readers (=media access points) are the //leaf nodes// in the node graph we are about to build.
* participating: source port of a clip, media access point, [[processing pattern|ProcPatt]]
* point of reference: //none//
* result: processing pattern has been //executed//, resulting in a chain of nodes from the source reader to the clip source port
* returns: ~PipeMould holding onto the new exit node (of a yet-empty pipe)
@@clear(right):display(block):@@
!!!wiring a general connection
Any wiring (outside the chain of effects within a pipe) is always done from exit nodes to the port of another pipe, requiring an [[wiring request|WiringRequest]] already checked and deemed resolvable. Within the __~WiringMould__ the actual wiring is conducted, possibly adding a summation node (called "overlayer" in case of video) and typically a fader element (the specific setup to be used is subject to configuration by processing patterns)
* participating: already verified connection request, providing a Pipe and an exit node; a processing pattern and a Placement
* points of reference: exit node and (optionally) starting point of a pipe's chain (in case there are already other connections)
* result: summation node prepended to the port of the pipe, processing pattern has been //executed// for building the connection from the exit node to the pipe's port, ParamProvider has been setup in [[accordance|PlacementDerivedDimension]] to the Placement.
* returns: ~PipeMould holding onto the destination pipe's exit node, ~WiringMould holding onto the port side of the same pipe, i.e. the destination where further connections will insert summation nodes. {{red{TODO how to handle the //empty//-case?}}}
[>img[draw/builder-primitives4.png]]
@@clear(right):display(block):@@

* the MObjects implement //Buildable//
* each Buildable can "receive" a Tool object and apply it
* the different Tool objects are iterated/mapped onto the list of MObjects in the [[Timeline]]
* __Rationale__
** the MObject class hierarchy is rather fixed (it is unlikely the we will be adding much new MObject subclasses)
** so this design makes it easy to add new Tool subclasses, and within each Tool subclass, all operations on the different MObject classes are grouped together, so it is easy to see what is going on.
** a given Tool instance can carry state while being iterated, so we don't need any global (or object-global) variables to hold the result of the build process
This programming technique is often referred to as [["double dispatch" or "visitor"|VisitorUse]]. We use a specialized library implementation of this pattern &mdash; heavily inspired by the [[Loki library|http://loki-lib.sourceforge.net/]]. We use this approach not only for the builder, but also for carrying out operations on the objects in the session in a typesafe manner.
It is the low level foundation of the actual [[building operations|BasicBuildingOperations]] necessary to create render nodes starting from the given high level model.
[img[Entities cooperating in the Builder|uml/fig129285.png]]
!Colaborations
While building, the application of such a visiting tool (especially the [[NodeCreatorTool|PlanningNodeCreatorTool]]) is embedded into an execution context formed by the BuilderToolFactory providing our BuilderToolKit, the [[Assembler|BuilderAssembler]] and [[connection manager|ConManager]]. The colaboration of these parts can be seen as the [[mechanics of the builder|BuilderMechanics]] &mdash; sort of the //outward view//, contrary to the //invard aspects// visible when focussing on how the nodes are put together.
[img[Colaborations in the Build Process|uml/fig128517.png]]

Besides the primary working tool within the builder (namely the [[Node Creator Tool|PlanningNodeCreatorTool]]), on a lower level, we encounter several [[elementary building situations|BuilderPrimitives]] &mdash; and for each of these elementary situations we can retrieve a suitable "fitting tool" or [[mould|BuilderMould]]. The palette of these moulds is called the ''tool kit'' of the builder. It is subject to configuration by rules.
!!addressing a mould
All mould instances are owned and managed by the [[tool factory|BuilderToolFactory]], and can be referred to by their type (PipeMould, CombiningMould, SourceChainMould, WiringMould) and a concrete object instance (of suitable type). The returned mould (instance) acts as a handle to stick together the given object instance (from the high-level model) with the corresponding point in the low-level node network under construction. As consequence of this approach, the tool factory instance holds a snapshot of the current building state, including all the active spots in the build process. As the latter is driven by objects from the high-level model appearing (in a sensible order &rarr; see BuilderMechanics) within the NodeCreatorTool, new moulds will be created and fitted as necessary, and existing moulds will be exhausted when finished, until the render node network is complete.
!!configuring a mould
As each mould kind is different, it has a {{{prepare(...)}}} function with suitably typed parameters. The rest is intended to be self-configuring (for example, a ~CombiningMould will detect the actual kind of Transition and select the internal mode of operation), so that it's sufficient to just call {{{operate()}}}
!!sequence of operations
When {{{operate()}}} doesn't throw, the result is a list of //successor moulds// &mdash; you shouldn't use the original mould after triggering its operation, because it may have been retracted as a result and reused for another purpose by the tool factory. It is not necessary to store these resulting moulds either (as they can be retrieved as described above), but they can be used right away for the next building step if applicable. In the state they are returned from a successful building step (mould operation = execution of a contained [[processing pattern|ProcPatt]]), they are usually already holding a reference to the part of the network just created and need to be configured only with the next high-level object (effect, placement, pipe, processing pattern or similar, depending on the concrete situation) in order to carry out the next step.
!!single connection step
at the lowest level within the builder there is the step of building a //connection.// This step is executed by the processing pattern with the help of the mould. Actually, making such a connection is more complicated, because in the standard case it will connect N media streams simultaneously (N=2 for stereo sound or 3D video, N=6 for 5.1 Surround, N=9 for 2nd order Ambisonics). These details are encapsulated within the OperationPoint, which is provided by the mould and exhibits a common interface for the processing pattern to express the connecting operation.
&rarr;see also: BuilderPrimitives for the elementary working situations corresponding to each of these [[builder moulds|BuilderMould]]

''Bus-~MObjects'' create a scope and act as attachment point for building up [[global pipes|GlobalPipe]] within each timeline. While [[Sequence]] is a frontend -- actually implemented by attaching a [[Fork]]-root object (»root track«) -- for //each global pipe// a BusMO is attached as child scope of the [[binding object|BindingMO]], which in turn actualy implements either a timeline or a [[meta-clip|VirtualClip]].
* each global pipe corresponds to a bus object, which thus refers to the respective ~Pipe-ID
* bus objects may be nested, forming a //subgroup//
* the placement of a bus holds a WiringClaim, denoting that this bus //claims to be the corresponding pipe.//
* by default, a timeline is outfitted with one video and one sound master bus

Calculation stream is an organisational unit used at the interface level of the Lumiera engine.
Representing a //stream of calculations,// delivering generated data within //timing constraints,// it is used
*by the [[play process(es)|PlayProcess]] to define and control properties of the output generation
*at the engine backbone to feed the [[Scheduler]] with individual [[render jobs|RenderJob]] to implement this stream of calculations
Calculation stream objects are stateless, constant chunks of definition -- any altering of playback or rendering parameters just causes the respective descriptors to be superseeded. The presence of a CalcStream (being alive within the denoted time span) implies that using any of the associated jobs, dispatcher tables, node and wiring descriptors is safe
!lifecycle
Calculation stream descriptors can be default constructed, representing a //void calculation.// You can't do anything with these.
Any really interesting calculation stream needs to be retrieved from the EngineFaçade. Additionally, an existing calculation stream can be chained up or superseded, yielding a new CalcStream based on the parameters of the existing one, possibly with some alterations.
!purpose
When a calculation stream is retrieved from the EngineFaçade it is already registered and attached there and represents an ongoing activity. Under the hood, several further collaborators will hold a copy of that calculation stream descriptor. While, as such, a CalcStream has no explicit state, at any time it //represents a current state.// In case the running time span of that stream is limited, it becomes superseded automatically, just by the passing of time.
Each calculation stream refers a relevant [[frame dispatcher table|FrameDispatcher]]. Thus, for the engine (interface level), the calculation stream allows to produce the individual [[render jobs|RenderJob]] to enqueue with the [[Scheduler]]. This translation step is what links and relates nominal time with running wall clock time, thereby obeying the [[timing constraints|Timings]] given when initially defining the calculation stream.
Additionally, each calculation stream knows how to access a //render environment closure,// allowing to re-schedule and re-adjust the setup of this stream. Basically, this closure is comprised of several functors (callbacks), which could be invoked to perform management tasks later on. Amongst others, this allows the calculation stream to redefine, supersede or "cancel itself", without the need to access a central registration table at the engine interface level.
&rarr; NodeOperationProtocol

//A clip represents some segment of media, which is arranged to appear at some time point within the edit.//
Lumiera agrees to this common understanding (of most film editing and sound handling applications), to the degree that a clip within Lumiera is largely an abstract entity, avoiding implicit or explicit further assumptions. A clip has a //temporal extension,// (start point and a duration) and we assume it features some media content. Yet the underlying media need not be uniform, it might be structured, a compound of several sources (e.g. sound and image) -- it might even be //virtual,// part of another sequence, in which case we'll get a VirtualClip.
//For the user,// clips are the most relevant entities encountered when working with and building the edit. As such, it is represented in the UI as [[clip widget|GuiClipWidget]], which can appear as arranged within the [[fork ("tracks")|Fork]] or as item within a //media bin.// But for the internal processing, clips are not conceived as primary entities; rather, they are translated by the [[Builder]] into an arrangement of interconnected [[pipes|Pipe]], which in turn are wired into a data processing network. In the end, this translates into a [[stream of processing jobs|CalcStream]], which are [[scheduled|Scheduler]] for calculation "just in time".
&rarr; SessionInterface
&rarr; TrackHandling
&rarr; PipeHandling

//mediating entity used to guide and control the presentation of a clip in the UI.//
The clip representation in the UI links together two distinct realms and systems of concerns. For one, there are the properties and actions performed on the clip for sake of editing a movie. Everything which has a tangible effect on the resulting render. These information and operations are embodied into the HighLevelModel and can be manipulated script driven, without relying on an UI. But beyond that, there is also the local mechanics of the interface, everything which makes working on the edit and interacting with the model into a smooth experience and workflow. With respect to these concerns, a clip is a self-contained entity which owns its own state and behaviour. The ClipPresenter is the pivotal element to link together those two realms.
[>img[Clip presentation control|uml/Timeline-clip-display.png]]
Regarding the global angle, the ClipPresenter is an UI-Element connected to the UI-Bus, and it is added as child element to some parent entity, a TrackPresenter, which likewise serves as UI representation of a [[fork ("track")|Fork]], and controls widgets to render a track like working scope (in the header pane and in the timeline contents pane). Commands manipulating the clip can be sent via the embedded bus terminal, and status regarding clip properties and nested child elements (effects, transitions) is received as messages via the bus, which insofar plays the role of model and controller.
But regarding the local UI behaviour, the ClipPresenter acts autonomous. It controls an actual {{{ClipWidget}}} for presentation, negotiating the display strategy with some overarching presentation manager. Active parts of the widget are wired back to handling methods of the presenter.

Within Proc-Layer, a Command is the abstract representation of a single operation or a compound of operations mutating the HighLevelModel.
Thus, each command is a ''Functor'' and a ''Closure'' ([[command pattern|http://en.wikipedia.org/wiki/Command_pattern]]), allowing commands to be treated uniformly, enqueued in a [[dispatcher|ProcDispatcher]], logged to the SessionStorage and registered with the UndoManager.
Commands are //defined// using a [[fluent API|http://en.wikipedia.org/wiki/Fluent_interface]], just by providing apropriate functions. Additionally, the Closure necessary for executing a command is built by binding to a set of concrete parameters. After reaching this point, the state of the internal representation could be serialised by plain-C function calls, which is important for integration with the SessionStorage.
&rarr; see CommandDefinition
&rarr; see CommandHandling
&rarr; see CommandLifecycle
&rarr; see CommandUsage

Commands can be identified and accessed //by name// &mdash; consequently there needs to be an internal command registry, including a link to the actual implementing function, thus allowing to re-establish the connection between command and implementing functions when de-serialising a persisted command. To create a command, we need to provide the following information
* operation function actually implementing the command
* function to [[undo|UndoManager]] the effect of the command
* function to capture state to be used by UNDO.
* a set of actual parameters to bind into these functions (closure).
!Command definition object
The process of creating a command by providing these building blocks is governed by a ~CommandDef helper object. According to the [[fluent definition style|http://en.wikipedia.org/wiki/Fluent_interface]], the user is expected to invoke a chain of definition functions, finally leading to the internal registration of the completed command object, which then might be dispatched or persisted. For example
{{{
CommandDefinition ("test.command1")
.operation (command1::operate) // provide the function to be executed as command
.captureUndo (command1::capture) // provide the function capturing Undo state
.undoOperation (command1::undoIt) // provide the function which might undo the command
.bind (obj, val1,val2) // bind to the actual command parameters (stores command internally)
.executeSync(); // convenience call, forwarding the Command to dispatch.
}}}
!Operation parameters
While generally there is //no limitation// on the number and type of parameters, the set of implementing functions and the {{{bind(...)}}} call are required to match. Inconsistencies will be detected by the compiler. In addition to taking the //same parameters as the command operation,// the {{{captureUndo()}}} function is required to return (by value) a //memento//&nbsp; type, which, in case of invoking the {{{undo()}}}-function, will be provided as additional parameter. To summarise:
|!Function|>|!ret(params)|
| operation| void |(P1,..PN)|
| captureUndo| MEM |(P1,..PN)|
| undoOperation| void |(P1,..PN,MEM)|
| bind| void |(P1,..PN)|
Usually, parameters should be passed //by value// &mdash; with the exception of target object(s), which are typically bound as MObjectRef, causing them to be resolved at commad execution time (late binding).
!Actual command definition scripts
The actual scripts bound as functors into the aforementioned command definitions are located in translation units in {{{proc/cmd}}}
These definitions must be written in a way to ensure that just compiling those translation units causes registration of the corresponding command-~IDs
This is achieved by placing a series of CommandSetup helper instances into those command defining translation units.

To organise any ''mutating'' operation executable by the user (via GUI) by means of the [[command pattern|http://en.wikipedia.org/wiki/Command_pattern]] can be considered //state of the art//&nbsp; today. First of all, it allows to discern the specific implementation operations to be called on one or several objects within the HighLevelModel from the operation requested by the user, the latter being rather a concept. A command can be labeled clearly, executed under controlled circumstances, allowing transactional behaviour.
!Defining commands
[>img[Structure of Commands|uml/fig134021.png]] Basically, a command could contain arbitrary operations, but we'll assume that it causes a well defined mutation within the HighLevelModel, which can be ''undone''. Thus, when defining (&rarr;[[syntax|CommandDefinition]]) a command, we need to specify not only a function to perform the mutation, but also another function which might be called later to reverse the effect of the action. Besides, the action operates on a number of ''target'' objects and additionally may require a set of ''parameter'' values. These are to be stored explicitly within the command object, thus creating a ''closure'' -- the operation //must not//&nbsp; rely on other hidden parameters (with the exception of generic singleton system services).
Theoretically, providing an "undo" functionality might draw on two different approaches:
* to specify an //inverse operation,// known to cancel out the effect of the command
* capturing of a //state memento,// which can later be played back in order to restore the state found prior to command execution.
While obviously the first solution looks elegant and is much simpler to implement on behalf of the command framework, the second solution has distinct advantages, especially in the context of an editing application: there might be rounding or calculation errors, the inverse might be difficult to define correctly, the effect of the operation might depend on circumstances, be random, or might even trigger a query resolution operation to yield the final result. Thus for Lumiera the decision is to //favour state capturing// -- but in a modified, semi-manual and not completely exclusive way.
!Undo state
While the usual »Memento« implementation might automatically capture the whole model (resulting in a lot of data to be stored and some uncertainty about the scope of the model to be captured), in Lumiera we rely instead on the client code to provide a ''capture function''&nbsp;and a ''playback function'' alongside with the actual operation. To help with this task, we provide a set of standard handlers for common situations. This way, operations might capture very specific information, might provide an "intelligent undo" to restore a given semantic instead of just a fixed value -- and moreover the client code is free actually to employ the "inverse operation" model in special cases where it just makes more sense than capturing state.
!Handling of commands
A command may be [[defined|CommandDefinition]] completely from scratch, or it might just serve as a CommandPrototype with specific targets and parameters. The command could then be serialised and later be recovered and re-bound with the parameters, but usually it will be handed over to the ProcDispatcher, pending execution. When ''invoking'', the handling sequence is to [[log the command|SessionStorage]], then call the ''undo capture function'', followed from calling the actual ''operation function''. After success, the logging and [[undo registration|UndoManager]] is completed. In any case, finally the ''result signal'' (a functor previously stored within the command) is emitted. {{red{10/09 WIP: not clear if we indeed implement this concept}}}
By design, commands are single-serving value objects; executing an operation repeatedly requires creating a collection of command objects, one for each invocation. While nothing prevents you from invoking the command operation functor several times, each invocation will overwrite the undo state captured by the previous invocation. Thus, each command instance should be seen as the promise (or later the trace) of a single operation execution. In a similar vein, the undo capturing should be defined as to be self sufficient, so that invoking just the undo functor of a single command performs any necessary steps to restore the situation found before invoking the corresponding mutation functor -- of course only //with respect to the topic covered by this command.// So, while commands provide a lot of flexibility and allow to do a multitude of things, certainly there is an intended CommandLifecycle.
&rarr; command [[definition|CommandDefinition]] and [[-lifecycle|CommandLifecycle]]
&rarr; more on possible [[command usage scenarios|CommandUsage]]
&rarr; more details regarding [[command implementation|CommandImpl]]

Commands are separated in a handle (the {{{control::Command}}}-object), to be used by the client code, and an implementation level, which is managed transparently behind the stages. Client code is assumed to build a CommandDefinition at some point, and from then on to access the command ''by ID'', yielding the command handle.
Binding of arguments, invocation and UNDO all are accessible through this frontend.
!Infrastructure
To support this handling scheme, some infrastructure is in place:
* a command registry maintains the ID &harr; Command relation.
* indirectly, through a custom alloctaor, the registry is also involved into allocation of the command implementation frame
* this implementation frame combines
** an operation mutation and an undo mutation
** a closure, implemented through an argument holder
** an undo state capturing mechanism, based on a capturing function provided on definition
* performing the actual execution is delegated to a handling pattern object, accessed by name.
;~Command-ID
:this ID is the primary access key for stored command definitions within the registry. When a command is //activated,// the command implementation record is also tagged with that ID; this is done for diagnostic purposes, e.g. to find out what commands in the command log of the session can be undone.
;prototypes
:while, technically, any command record in the registry can be outfitted with arguments and executed right away, the standard usage pattern is to treat the //globally known, named entries// in this registry as prototype objects, from which the actual //instances for execution// are created by cloning. This is done to circumvent concurrency problems with argument binding.
;named and anonymous instances
:any command entry in the registry can be clone-copied. There are two flavours of this functionality: either, the new entry can be stored under a different name in the global registry, or alternatively just an unnamed copy can be created and returned. Since such an anonymous copy is not tracked in the registry, its lifetime is controlled solely by the ref-count of the handle returned from this {{{Command::newInstance()}}} call. Please note that the {{{CommandImpl}}} record managed by this handle still bears a copy of the original ~Command-ID, which helps with diagnostics when invoking such an instance. But the presence of this ID in the implementation record does not mean the command is known to the registry; to find out about that, use the {{{Command::isAnonymous()}}} predicate.
!Definition and usage
In addition to the technical specification regarding the command, memento and undo functors, some additional conventions are established
* Command scripts are defined in translation units in {{{proc/cmd}}}
* these reside in the corresponding namespace, which is typically aliased as {{{cmd}}}
* both command definition and usage include the common header {{{proc/cmd.hpp}}}
* the basic command-~IDs defined therein need to be known by the UI elements using them

//This page is a scrapbook to collect observations about command invocation in the UI//
{{red{2/2017}}} the goal is to shape some generic patterns of InteractionControl (&rarr; GuiCommandBinding, &rarr; GuiCommandCycle)
!Add Sequence
The intention is to add a new sequence //to the current session.//
As discussed in &rarr; ModelDependencies, there is quite some degree of magic involved into such a simple activity, depending on the circumstances
* a Session always has at least one Timeline; even a default-created Session has
* when the Session is just //closed,// the UI becomes mostly disabled, with the exception of {{{Help}}}, {{{Quit}}} and {{{New Session}}}...
* what this operation //actually means,// depends on the user's expectations. Here "the user" is an abstracted user we conceive, since there are no real-world users of Lumiera yet.
** when just invoking "add Sequence", the user wants a new playground
** when, on the other hand, the user right-clicks a scope-fork ("track" or "media bin"), she wants a new sequence made out of the contents of this fork
* in both cases, after creating the new Sequence, it should become visible in the UI -- <br/>which means we have to set up a new timeline too, taking a copy of the currently active timeline. (When the latter is just pristine, we want the new one to take its place)
At this point, it seems adequate to limit "add Sequence" to the first use case, and name the second one as "make Sequence".
The latter will probably be used within a context menu, but don't forget that actions can be bound to keys...
//add Sequence// thus means a global command, which can be issued against the current session with no further arguments...
* it will fabricate a new fork (appearing as track in this context)
* it will //root-attach// this fork, i.e. it will [[place|Placement]] it as child of the ModelRootMO scope, and this act automagically creates a new [[Sequence]] asset
* it will fabricate a new BindingMO as a prototype copy from the currently active one, and likewise root-attach it, which magically creates a new [[Timeline]] asset
* it will create a new ''focus goal'' ({{red{TODO new term coined 2/2017}}}), which should lead to activating the corresponding tab in the timeline pane, once this exists...
Now, while all of this means a lot of functionality and complexity down in Proc, regarding the UI this action is quite simple: it is offered as an operation on the InteractionDirector, which most conveniently also corresponds to "the session as such" and thus can fire off the corresponding command without much further ado. On a technical level we just somehow need to know the corresponding command ID in proc, which is not subject to configuration, but rather requires some kind of well behaved hard-wiring. And, additionally, we need to build something like the aforementioned //focus goal...// {{red{TODO}}}
!Add Track
Here the intention is to add a new scope //close to where we "are" currently.//
If the currently active element is something within a scope, we want the new scope as a sibling, otherwise we want it as a child, but close at hand.
So, for the purpose of this analysis, the "add Track" action serves as an example where we need to pick up the subject of the change from context...
* the fact there is always a timeline and a sequence, also implies there is always a fork root (track)
* so this operation basically adds to a //"current scope"// -- or next to it, as sibling
* this means, the UI logic has to provide a //current model element,// while the details of actually selecting a parent are decided elsewhere (in Proc-Layer, in rules)

[<img[Structure of Commands|uml/fig135173.png]]
While generally the command framework was designed to be flexible and allow a lot of different use cases, employ different execution paths and to serve various goals, there is an ''intended lifecycle'' &mdash; commands are expected to go through several distinct states.
The handling of a command starts out with a ''command ID'' provided by the client code. Command ~IDs are unique (human readable) identifiers and should be organised in a hierarchical fashion. When provided with an ID, the CommandRegistry tries to fetch an existing command definition. In case this fails, we enter the [[command definition stage|CommandDefinition]], which includes specifying functions to implement the operation, state capturing and UNDO. When all of this information is available, the entity is called a ''command definition''. Conceptually, it is comparable to a //class// or //meta object.//
By ''binding'' to specific operation arguments, the definition is //armed up//&nbsp; and becomes a real ''command''. This is similar to creating an instance from a class. Behind the scenes, storage is allocated to hold the argument values and any state captured to create the ability to UNDO the command's effect later on.
A command is operated or executed by passing it to an ''execution pattern'' &mdash; there is a multitude of possible execution patterns to choose from, depending on the situation.
{{red{WIP... details of ~ProcDispatcher not specified yet}}}
When a command has been executed (and maybe undone), it's best to leave it alone, because the UndoManager might hold a reference. Anyway, a ''clone of the command'' could be created, maybe bound with different arguments and treated separately from the original command.
!State predicates
* fetching an non-existent command raises an ~LUMIERA_ERROR_INVALID_COMMAND
* a command definition becomes //valid// ({{{bool true}}}) when all necessary functions are specified. Technically this coincides with the creation of a CommandImpl frame behind the scenes, which also causes the Command (frontend/handle object) to evaluate to {{{true}}} in bool context from then on.
* when, in addition to the above, the command arguments are bound, it becomes //executable.//
* after the (fist) execution, the command gets also //undo-able.//
State predicates are accessible through the Command (frontend); additionally there are static query functions in class {{{Command}}}

//Helper facility to ease the creation of actual command definitions.//
A [[Proc-Layer command|CommandHandling]] is a functor, which can be parametrised with actual arguments. It needs to be [[defined|CommandDefinition]] beforehand, which means to establish an unique name and to supply three functions, one for the actual command operation, one to capture state and one to [[UNDO]] the effect of the command invocation.
The helper class {{{CommandSetup}}} allows to create series of such definitions with minimal effort. Since any access and mutation from the UI into the Session data must be performed by invoking such commands, a huge amount of individual command definitions need to be written eventually. These are organised into a series of implementation translation units with location {{{poc/cmd/*-cmd.cpp}}}.
Each of these files is specialised to defining a set of thematically related commands, supplying the code for the actual command scripts. Each definition is introduced by a single line with the macro {{{COMMAND_DEFINITION(name)}}}, followed by a code block, which actually ends up as the body of a lambda function, and receives the bare [[CommandDef|CommandDefinition]] as single argument with name {{{cmd}}}. The {{{name}}} argument of the macro ends up both stringified as the value of the command-ID, and as an identifier holding a new {{{CommandSetup}}} instance. It is assumed that a header with //corresponding declarations// (the header {{{cmd.hpp}}}) is included by all UI elements actually to use, handle and invoke commands towards the SessionSubsystem

//for now (7/09) I'll use this page to collect ideas how commands might be used...//
* use a command for getting a log entry and an undo possibility automatically
* might define, bind and then execute a command at once
* might define it and bind it to a standard set of parameters, to be used as a prototype later.
* might just create the definition, leaving the argument binding to the actual call site
* execute it and check the success/failure result
* just enqueue it, without caring for successful execution
* place it into a command sequence bundle
* repeat the execution
!!!a command definition....
* can be created from scratch, by ID
* can be re-accessed, by ID
* can't be modified once defined (this is to prevent duplicate definitions with the same ID)
* but can be dropped, which doesn't influence already existing dependent command instances
* usually will be the starting point for creating an actual command by //binding//
!!!a command instance....
* normally emerges from a definition by binding arguments
* the first such binding will create a named command registration
* but subsequent accesses by ID or new bindings create an anonymous clone
* which then in turn might then be registered explicitly with a new ID
* anonymous command instances are managed by referral and ref-counting
!!!an execution pattern....
* can only be defined in code by a class definition, not at runtime
* subclasses the ~HandlingPattern interface and uses an predefined ID (enum).
* a singleton instance is created on demand, triggered by referring the pattern's ID
* is conceptually //stateless// &mdash; of course there can be common configuration values
* is always invoked providing a concrete command instance to execute
* is configured into the command instance, to implement the command's invocation
* returns a duck-typed //result// object
!command &harr; interaction
The User Interface does not just trigger commands -- rather it performs //user interactions.//
An interaction is formed like a sentence of spoken language, which means, there is a process of forming such a sentence, giving rise to [[interaction state|InteractionState]]. In many cases, the actual arguments of a command invocation are to be drawn from the current context. Thus, GuiCommandBinding is a way more elaborate topic, while it builds upon the fundamentals defined here...

The Connection Manager is a service for wiring connections and for querying information and deriving decisions regarding various aspects of data streams and the possibility of connections. The purpose of the Connection Manager is to isolate the [[Builder]], which is client of this information and decision services, from the often convoluted details of type information and organizing a connection.
!control connections
my intention was that it would be sufficient for the builder to pass an connection request, and the Connection Manager will handle the details of establishing a control/parameter link.
{{red{TODO: handling of parameter values, automation and control connections still need to be designed}}}
!data connections
Connecting data streams of differing type involves a StreamConversion. Mostly, this aspect is covered by the [[stream type system|StreamType]]. The intended implementation will rely partially on [[rules|ConfigRules]] to define automated conversions, while other parts need to be provided by hard wired logic. Thus, regarding data connections, the ConManager can be seen as a specialized Facade and will delegate to the &rarr; [[stream type manager|STypeManager]]
* retrieve information about capabilities of a stream type given by ID
* decide if a connection is possible
* retrieve a //strategy// for implementing a connection

This index refers to the conceptual, more abstract and formally specified aspects of the Proc-Layer and Lumiera in general.
More often than not, these emerge from immediate solutions, being percieved as especially expressive, when taken on, yielding guidance by themselves. Some others, [[Placements|Placement]] and [[Advice]] to mention here, immediately substantiate the original vision.

Configuration Queries are requests to the system to "create or retrieve an object with //this and that // capabilities". They are resolved by a rule based system ({{red{planned feature}}}) and the user can extend the used rules for each Session. Syntactically, they are stated in ''prolog'' syntax as a conjunction (=logical and) of ''predicates'', for example {{{stream(mpeg), pipe(myPipe)}}}. Queries are typed to the kind of expected result object: {{{Query<Pipe> ("stream(mpeg)")}}} requests a pipe excepting/delivering mpeg stream data &mdash; and it depends on the current configuration what "mpeg" means. If there is any stream data producing component in the system, which advertises to deliver {{{stream(mpeg)}}}, and a pipe can be configured or connected with this component, then the [[defaults manager|DefaultsManagement]] will create/deliver a [[Pipe|PipeHandling]] object configured accordingly.
&rarr; [[Configuration Rules system|ConfigRules]]
&rarr; [[accessing and integrating configuration queries|ConfigQueryIntegration]]

* planning to embed a YAP Prolog engine
* currently just integrated by a table driven mock
* the baseline is a bit more clear by now (4/08)
&rarr; see also ConfigRules
&rarr; see also DefaultsManagement
!Use cases
[<img[when to run config queries|uml/fig131717.png]]
The key idea is that there is a Rule Base &mdash; partly contained in the session (building on a stock of standard rules supplied with the application). Now, whenever there is the need to get a new object, for adding it to the session or for use associated with another object &mdash; then instead of creating it by a direct hard wired ctor call, we issue a ConfigQuery requesting an object of the given type with some //capabilities// defined by predicates. The same holds true when loading an existing session: some objects won't be loaded back blindly, rather they will be re-created by issuing the config queries again. Especially an important use case is (re)creating a [[processing pattern|ProcPatt]] to guide the wiring of a given media's processing pipeline.
At various places, instead of requiring a fixed set of capabilities, it is possible to request a "default configured" object instead, specifying just those capabilities we really need to be configured in a specific way. This is done by using the [[Defaults Manager|DefaultsManagement]] accessible on the [[Session]] interface. Such a default object query may either retrieve an already existing object instance, run further config queries, and finally result in the invocation of a factory for creating new objects &mdash; guarded by rules to suit current necessities.
@@clear(left):display(block):@@
!Components and Relations
[>img[participating classes|uml/fig131461.png]]
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
Access point is the interface {{{ConfigRules}}}, which allowes to resolve a ConfigQuery resulting in an object with properties configured such as to fulfill the query. This whole subsystem employes quite some generic programming, because actually we don't deal with "objects", but rather with similar instantiations of the same functionality for a collection of different object types. For the purpose of resolving these queries, the actual kind of object is not so much of importance, but on the caller side, of course we want to deal with the result of the queries in a typesafe manner.
Examples for //participating object kinds// are [[pipes|Pipe]], [[processing patterns|ProcPatt]], effect instances, [[tags|Tag]], [[labels|Label]], [[automation data sets|AutomationData]],...
@@clear(right):display(block):@@
For this to work, we need each of the //participating object types// to provide the implementation of a generic interface {{{TypeHandler}}}, which allows to access actual C/C++ implementations for the predicates usable on objects of this type within the Prolog rules. The implementation has to make sure that, alongside with each config query, there are additional //type constraints// to be regarded. For example, if the client code runs a {{{Query<Pipe>}}}, an additional //type guard// (implemented by a predicate {{{type(pipe)}}} has to be inserted, so only rules and facts in accordance with this type will be used for resolution.
!when querying for a [["default"|DefaultsManagement]] object
[<img[colaboration when issuing a defaults query|uml/fig131845.png]]
@@clear(left):display(block):@@

Many features can be implemented by specifically configuring and wiring some unspecific components. Rather than tie the client code in need of some given feature to these configuration internals, in Lumiera the client can //query // for some kind of object providing the //needed capabilities. // Right from start (summer 2007), Ichthyo had the intention to implement such a feature using sort of a ''declarative database'', e.g. by embedding a Prolog system. By adding rules to the basic session configuration, users should be able to customize the semi-automatic part of Lumiera's behaviour to great extent.
[[Configuration Queries|ConfigQuery]] are used at various places, when creating and adding new objects, as well when building or optimizing the render engine node network.
* Creating a [[pipe|PipeHandling]] queries for a default pipe or a pipe with a certain stream type
* Adding a new [[fork ("track")|TrackHandling]] queries for some default placement configuration, e.g. the pipe it will be plugged to.
* when processing a [[wiring request|WiringRequest]], connection possibilities have to be evaluated.
* actually building such a connection may create additional degrees of freedom, like panning for sound or layering for video.
!anatomy of a Configuration Query
The query is given as a number of logic predicates, which are required to be true. Syntactically, it is a string in prolog syntax, e.g. {{{stream(mpeg)}}}, where "stream" is the //predicate, // meaning here "the stream type is...?" and "mpeg" is a //term // denoting an actual property, object, thing, number etc &mdash; the actual kind of stream in the given example. Multible comma separated predicates are combined with logical "and". Terms may be //variable// at start, which is denoted syntactically by starting them with a uppercase letter. But, through the computation of a solution, any variable term needs to be //bound//&nbsp; to some known fixed value, otherwise the counts as failed. A failed query is treated as a local failure, which may cause some operation being aborted or just some other possibility being chosen.
Queries are represented by instantiations of the {{{Query<TYPE>}}} template, because their actual meaning is "retrieve or create an object of TYPE, configured such that...!". At the C++ side, this ensures type safety and fosters programming against interfaces, while being implemented rule-wise by silently prepending the query with the predicate {{{object(X, type)}}}, which reads as "X is an object with this type"
!!!querying for default
A special kind of configuration query includes the predicate {{{default(X)}}}, which is kind-of an existential quantisation of the variable {{{X}}}. Using this predicate states that a suitable //default-configured// object exists somehow. This behaviour could be used as an fallback within a config query, allowing us always to return a solution. The latter can be crucial when it comes to integrating the query subsystem into an existing piece of implementation logic, which requires us to give some guarantees.
&rarr; see DefaultsManagement on details how to access these captured defaults
But note, the exact relation of configuration queries and query-for-default still needs to be worked out, especially when to use which flavour.
! {{red{WIP 2012}}} front-end and integration
The overall architecture of this rules based configuration system remains to be worked out
* the generic front-end to unite all kinds of configuration queries
* how to configure the various [[query resolvers|QueryResolver]]
* how to pick and address a suitable resolver for a given query situation
* the taxonomy of queries:
** capability queries
** retrieval queries
** defaults queries
** discovery queries
** connection queries
** ...
{{red{WIP 10/2010 &rArr; see Ticket [[#705|http://issues.lumiera.org/ticket/705]]}}}
!executing a Configuration Query
Actually posing such an configuration query, for example to the [[Defaults Manager in the Sessison|DefaultsManagement]], may trigger several actions: First it is checked against internal object registries (depending on the target object type), which may cause the delivery of an already existing object (as reference, clone, or smart pointer). Otherwise, the system tries to figure out an viable configuration for a newly created object instance, possibly by issuing recursive queries. In the most general case this may silently impose additional decisions onto the //execution context // of the query &mdash; by default the session.
!Implementation
At start and for debugging/testing, there is an ''dummy'' implementation using a map with predefined queries and answers. But for the real system, the idea is to embed a ''YAP Prolog'' engine to run the queries. This includes the task of defining and loading a set of custom predicates, so the rule system can interact with the object oriented execution environment, for example by transforming some capability predicate into virtual calls to a corresponding object interface. We need a way for objects to declare some capability predicates, together with a functor that can be executed on an object instance (and further parameters) in the cause of the evaluation of some configuration query. Type safety and diagnostics play an important role here, because effectively the rule base is a pool of code open for arbitray additions from the user session.
&rarr; [[considerations for a Prolog based implementation|QueryImplProlog]]
&rarr; [[accessing and integrating configuration queries|ConfigQueryIntegration]]
&rarr; see {{{src/common/query/mockconfigrules.cpp}}} for the table with the hard wired (mock) answers
!!!fake implementation guidelines
The fake implementation should follow the general pattern planned for the Prolog implementation. That is, find, query, create, query. Thus, such a config query can be considered existentially quantised. The interplay with the factories is tricky. Because factories might issue config queries themselves, while a factory invocation driven from within this rule pattern //must not// re-invoke the factory. Besides, a factory invocation should create, not fetch an existing solution (?)
{{red{WARN}}} there is an interference with the (planned) Undo function: a totally correct implementation of "Undo" would need to remember and restore the internal state of the query system (similar to backtracking). But, more generally, such an correct implementation is not achievable, because we are never able to capture and control the whole state of a real world system doing such advanced things like video and sound processing. Seemingly we have to accept that after undoing an action, there is no guarantee we can re-do it the same way as it was first time.

Here, in the context of the Render Engine, the Controller component is responsible for triggering and coordinating the build process and for activating the backend and the Render Engine configuration created by the Builder to carry out the actual rendering. There is another Controller in the backend, the ~PlaybackController, which is in charge of the playback/rendering/display state of the application, and consequently will call this (Proc-Layer) Controller to get the necessary Render Engine.
!Facade
This is an very important external Interface, because it links together all three Layers of our current architecture. It can be used by the backend to initiate [[Render Processes (=StateProxy)|StateProxy]] and it will probably be used by the Dispatcher for GUI actions as well...

''special service connected to the UI-Bus to handle all messages touching core concerns''
Especially this service handles the {{{act}}} messages to deal with commands [[operating on the Session|CommandHandling]]. And it deals with "uplink" {{{note}}} messages to record ongoing PresentationState changes. For all the other UI-Element nodes connected to the bus, CoreService is just assumed to be there, yet it is not known by ID, nor can it be addressed directly, like regular ~UI-Elements can.
On an operational level, CoreService additionally serves as lifecycle manager for the whole UI backbone, insofar it maintains the central bus hub, the Nexus. CoreService is instantiated (as a PImpl) when the UI subsystem starts up, and it is destroyed when leaving the UI main loop. At this point, the automatic registration and deregistration mechanism for bus terminals and ~UI-Elements must have cleared all connections in the central routing table (within Nexus).

The question is where to put all the state-like information [[associated with the current session|SessionOverview]]. Because this is certainly "global", but may depend on the session or need to be configured differently when loading another session. At the moment (9/07) Ichthyo considers the following solution:
* represent all configuration as [[Asset]]s
* find a way {{red{TODO}}} how to reload the contents of the [[AssetManager]].
* completely hide the Session object behind a ''~PImpl'' smart pointer, so the session object can be switched when reloading.
* the [[Fixture]] acts as isolation layer, and all objects refered from the Fixture are refcounting smart pointers. So, even when the session gets switched, the old objects remain valid as long as needed.

A ''frame of data'' is the central low-level abstraction when dealing with media data and media processing.
Deliberately we avoid relying on any special knowledge regarding such data frames, beyond the fact
* that a frame resides within a ''memory buffer'' &rarr; [[buffer provider abstraction|BufferProvider]]
* that the frame has a ''frame number'', which can be related to a ''time span'' &rarr; [[time quantisation framework|TimeQuant]]
* that the frame belongs to an abstract media stream, which can be described through our &rarr; [[stream type system|StreamType]]
* and that some external ''media handling library'' knows to deal with the data in this frame

[[ProcLayer and Engine]]

As detailed in the [[definition|DefaultsManagement]], {{{default(Obj)}}} is sort of a Joker along the lines "give me a suitable Object and I don't care for further details". Actually, default objects are implemented by the {{{mobject::session::DefsManager}}}, which remembers and keeps track of anything labeled as "default". This defaults manager is a singleton and can be accessed via the [[Session]] interface, meaning that the memory trail regarding defaults is part of the session state. Accessing an object via the query for an default actually //tagges// this object (storing a weak ref in the ~DefsManager). Alongside with each object successfully queried via "default", the degree of constriction is remembered, i.e. the number of additional conditions contained in the query. This enables us to search for default objects starting with the most unspecific.
!Skeleton
# ''search'': using the predicate {{{default(X)}}} enumerates existing objects of suitable type
#* candidates are delivered starting with the least constrained default
#* the argument is unified
#** if the rest of the query succeeds we've found our //default object// and are happy.
#** otherwise, if all enumerated solutions are exhausted without success, we enter
# ''default creation'': try to get an object fulfilling the conditions and remember this situation
#* we issue an ConfigQuery with the query terms //minus// the {{{default(X)}}} predicate
#* it depends on the circumstances how this query is handled. Typically the query resolution first searches existing objects and then creates a new instance to match the required capabilities. Usually, this process succeeds, but there can be configurations leading to failure.
#** failing the ~ConfigQuery is considered an (non-critical) exception (throws), as defaults queries are supposed to succeed
#** otherwise, the newly created object is remembered (tagged) as new default, together with the degree of constriction
!!!Implementation details
Taken precisely, the "degree of constriction" yields only a partial ordering &mdash; but as the "default"-predicate is sort of a existential quantification anyway, its sole purpose is to avoid polluting the session with unnecessary default objects, and we don't need to care for absolute precision. A suitable approximation is to count the number of predicates terms in the query and use a (sorted) set (separate for each Type) to store weak refs to the the objects tagged as "default"
{{red{WARN}}} there is an interference with the (planned) Undo function. This is a general problem of the config queries; just ignoring this issue seems reasonable.
!!!Problems with the (preliminary) mock implementation
As we don't have a Prolog interpreter on board yet, we utilize a mock store with preconfigured answers. (see MockConfigQuery). As this preliminary solution is lacking the ability to create new objects, we need to resort to some trickery here (please look away). The overall logic is quite broken, not to say outright idiotic, because the system isn't capable to do any real resolution &mdash; if we ignore this fact, the rest of the algorithm can be implemented, tested and used right now.

For several components and properties there is an implicit default value or configuration; it is stored alongside with the session. The intention is that defaults never create an error, instead, they are to be extended silently on demand. Objects configured according to these defaults can be retrieved at the [[Session]] interface by a set of overloaded functions {{{Session::current->default(Query<TYPE> ("query string"))}}}, where the //query string // defines a capability query similar to what is employed for pipes, stream types, codecs etc. This query mechanism is implemented by [[configuration rules|ConfigRules]]
!!!!what is denoted by {{{default}}}?
{{{default(Obj)}}} is a predicate expressing that the object {{{Obj}}} can be considered the default setup under the given conditions. Using the //default// can be considered as a shortcut for actually finding an exact and unique solution. The latter would require to specify all sorts of detailed properties up to the point where only one single object can satisfy all conditions. On the other hand, leaving some properties unspecified would yield a set of solutions (and the user code issuing the query had to provide means for selecting one solution from this set). Just falling back on the //default// means that the user code actually doesn't care for any additional properties (as long as the properties he //does// care for are satisfied). Nothing is said specifically on //how//&nbsp; this default gets configured; actually there can be rules //somewhere,// and, additionally, anything encountered once while asking for a default can be re-used as default under similar circumstances.
&rarr; [[implementing defaults|DefaultsImplementation]]

//Access point to dependencies by-name.//
In the Lumiera code base, we refrain from building or using a full-blown Dependency Injection Container. Rather, we rely on a generic //Singleton Factory// -- which can be augmented into a //Dependency Factory// for those rare cases where we actually need more instance and lifecycle management beyond lazy initialisation. Client code indicates the dependence on some other service by planting an instance of that Dependency Factory (for Lumiera this is {{{lib::Depend<TY>}}}). The //essence of a "dependency"// of this kind is that we ''access a service //by name//''. And this service name or service ID is in our case a //type name.//
&rarr; see the Introductory Page about [[Dependencies|http://lumiera.org/documentation/technical/library/Dependencies.html]] in Lumiera online documentation.
!Requirements
Our DependencyFactory satisfies the following requirements
* client code is able to access some service //by-name// -- where the name is actually the //type name// of the service interface.
* client code remains agnostic with regard to the lifecycle or backing context of the service it relies on
* in the simplest (and most prominent case), //nothing// has to be done at all by anyone to manage that lifecycle.<br/>By default, the Dependency Factory creates a singleton instance lazily (heap allocated) on demand and ensures thread-safe initialisation and access.
* we establish a policy to ''disallow any significant functionality during application shutdown''. After leaving {{{main()}}}, only trivial dtors are invoked and possibly a few resource handles are dropped. No filesystem writes, no clean-up and reorganisation, not even any logging is allowed. For this reason, we established a [[Subsystem]] concept with explicit shutdown hooks, which are invoked beforehand.
* the Dependency Factory can be re-configured for individual services (type names) to refer to an explicitly installed service instance. In those cases, access while the service is not available will raise an exception. There is a simple one-shot mechanism to reconfigure Dependency Factory and create a link to an actual service implementation, including automatic deregistration.
!!!Configuration
The DependencyFactory and thus the behaviour of dependency injection can be reconfigured, ad hoc, at runtime.
Deliberately, we do not enforce global consistency statically (since that would lead to one central static configuration). However, a runtime sanity check is performed to ensure configuration actually happens prior to any use, which means any invocation to retrieve (and thus lazily create) the service instance. The following flavours can be configured
;default
:a singleton instance of the designated type is created lazily, on first access
:define an instance for access (preferably static) {{{Depend<Blah> theBla;}}}
:access the singleton instance as {{{theBla().doIt();}}}
;singleton subclass
:{{{DependInject<Blah>::useSingleton<SubBlah>() }}}
:causes the dependency factory {{{Depend<Bla>}}} to create a {{{SubBlah}}} singleton instance from now on
;attach to service
:{{{DependInject<Blah>::ServiceInstance<SubBlah> service{p1, p2, p3}}}}
:* build and manage an instance of {{{SubBlah}}} in heap memory immediately (not lazily)
:* configure the dependency factory to return a reference //to this instance//
:* the generated {{{ServiceInstance<SubBlah>}}} object itself acts as lifecycle handle (and managing smart-ptr)
:* when it is destroyed, the dependency factory is automatically cleared, and further access will trigger an error
;support for test mocking
:{{{DependInject<Blah>::Local<SubBlah> mock }}}
:temporarily shadows whatever configuration resides within the dependency factory
:the next access will create a (non singleton) {{{SubBlah}}} instance in heap memory and return a {{{Blah&}}}
:the generated object again acts as lifecycle handle and smart-ptr to access the {{{SubBlah}}} instance like {{{mock->doItSpecial()}}}
:when this handle goes out of scope, the original configuration of the dependency factory is restored
;custom constructors
:both the subclass singleton configuration and the test mock support optionally accept a functor or lambda argument with signature {{{SubBlah*()}}}.
:the contract is for this construction functor to return a heap allocated object, which will be owned and managed by the DependencyFactory.
:especially this enables use of subclasses with non default ctor and / or binding to some additional hidden context.
:please note //that this closure will be invoked later, on-demand.//
We consider the usage pattern of dependencies a question of architecture rather -- such can not be solved by any mechanism at implementation level.
For this reason, Lumiera's Dependency Factory prevents reconfiguration after use, but does nothing exceeding such basic sanity checks
!!!Performance considerations
To gain insight into the rough proportions of performance impact, in 2018 we conducted some micro benchmarks (using a 8 core AMD ~FX-8350 64bit CPU running Debian/Jessie and GCC 4.9 compiler)
The following table lists averaged results //in relative numbers,// in relation to a single threaded optimised direct non virtual member function invocation (≈ 0.3ns)
| !Access Technique |>| !development |>| !optimised |
|~| single threaded|multithreaded | single threaded|multithreaded |
|direct invoke on shared local object | 15.13| 16.30| ''1.00''| 1.59|
|invoke existing object through unique_ptr | 60.76| 63.20| 1.20| 1.64|
|lazy init unprotected (not threadsafe) | 27.29| 26.57| 2.37| 3.58|
|lazy init always mutex protected | 179.62| 10917.18| 86.40| 6661.23|
|Double Checked Locking with mutex | 27.37| 26.27| 2.04| 3.26|
|DCL with std::atomic and mutex for init | 44.06| 52.27| 2.79| 4.04|
Some observations:
* The numbers obtained pretty much confirm [[other people's measurments|http://www.modernescpp.com/index.php/thread-safe-initialization-of-a-singleton]]
* Synchronisation is indeed necessary; the unprotected lazy init crashed several times randomly during multithreaded tests.
* Contention on concurrent access is very tangible; even for unguarded access the cache and memory hardware has to perform additional work
* However, the concurrency situation in this example is rather extreme and deliberately provokes collisions; in practice we'd be closer to the single threaded case
* Double Checked Locking is a very effective implementation strategy and results in timings within the same order of magnitude as direct access
* Unprotected lazy initialisation performs spurious duplicate initialisations, which can be avoided by DCL
* Naïve Mutex locking is slow even with non-recursive Mutex without contention
* Optimisation achieves access times around ≈ 1ns

Along the way of working out various [[implementation details|ImplementationDetails]], decisions need to be made on how to understand the different facilities and entities and how to tackle some of the problems. This page is mainly a collection of keywords, summaries and links to further the discussion. And the various decisions should allways be read as proposals to solve some problem at hand...
''Everything is an object'' &mdash; yes of course, that's a //no-brainer,// todays. Rather, important is to note what is not "an object", meaning it can't be arranged arbitrarily
* we have one and only one global [[Session]] which directly contains a collection of multiple [[Timelines|Timeline]] and is associated with a globally managed collection of [[assets|Asset]]. We don't utilise scoped variables here (no "mandantisation"); if a media has been //opened,// it is just plain //globally known//&nbsp; as asset.
* the [[knowledge base|ConfigRules]] is just available globally. Obviously, the session gets a chance to install rules into this knowledge base, but we don't stress ownership here.
* we create an unique [[Fixture]] which acts as isolation layer towards the render engine and is (re)built automatically.
The high-level view of the tangible entities within the session is unified into a ''single tree'' -- with the notable exception of [[external outputs|OutputManagement]] and the [[assets|Asset]], which are understood as representing a //bookkeeping view// and kept separate from the //things to be manipulated// (MObjects).
We ''separate'' processing (rendering) and configuration (building). The [[Builder]] creates a network of [[render nodes|ProcNode]], to be processed by //pulling data // from some [[Pipe]]
''Objects are [[placed|Placement]] rather'' than assembled, connected, wired, attached. This is more of a rule-based approach and gives us one central metaphor and abstraction, allowing us to treat everything in an uniform manner. You can place it as you like, and the builder tries to make sense out of it, silently disabling what doesn't make sense.
An [[Sequence]] is just a collection of configured and placed objects (and has no additional, fixed structure). [["Tracks" (forks)|Fork]] form a mere organisational grid, they are grouping devices not first-class entities (a track doesn't "have" a pipe or "is" a video track and the like; it can be configured to behave in such manner by using placements though). [[Pipes|Pipe]] are hooks for making connections and are the only facility to build processing chains. We have global pipes, and each clip is built around a lokal [[source port|ClipSourcePort]] &mdash; and that's all. No special "media viewer" and "arranger", no special role for media sources, no commitment to some fixed media stream types (video and audio). All of this is sort of pushed down to be configuration, represented as asset of some kind. For example, we have [[processing pattern|ProcPatt]] assets to represent the way of building the source network for reading from some media file (including codecs treated like effect plugin nodes)
The model in Proc-Layer is rather an //internal model.// What is exposed globally, is a structural understanding of this model. In this structural understanding, there are Assets and ~MObjects, which both represent the flip side of the same coin: Assets relate to bookkeeping, while ~MObjects relate to building and manipulation of the model. In the actual data represntation within the HighLevelModel, we settled upon some internal reductions, preferring either the //Asset side// or the //~MObject side// to represent some relevant entities. See &rarr; AssetModelConnection.
Actual ''media data and handling'' is abstracted rigorously. Media is conceived as being stream-like data of distinct StreamType. When it comes to more low-level media handling, we build on the DataFrame abstraction. Media processing isn't the focus of Lumiera; we organise the processing but otherwise ''rely on media handling libraries.'' In a similar vein, multiplicity is understood as type variation. Consequently, we don't build an audio and video "section" and we don't even have audio tracks and video tracks. Lumiera uses tracks and clips, and clips build on media, but we're able to deal with [[multichannel|MultichannelMedia]] mixed-typed media natively.
Lumiera is not a connection manager, it is not an audio-visual real time performance instrument, and it doesn't aim at running presentations. It's an ''environment for assembling and building up'' something (an edit, a session, a piece of media work). This decision is visible at various levels and contexts, like a reserved attitude towards hardware acceleration (it //will// be supported, but reliable proxy editing has a higher priority), or the decision, not to incorporate system level ports and connections directly into the session model (they are mapped to [[output designations|OutputDesignation]] rather)
''State'' is rigorously ''externalized'' and operations are to be ''scheduled'', to simplify locking and error handling. State is either treated similar to media stream data (as addressable and cacheable data frame), or is represented as "parameter" to be served by some [[parameter provider|ParamProvider]]. Consequently, [[Automation]] is just another kind of parameter, i.e. a function &mdash; how this function is calculated is an encapsulated implementation detail (we don't have "bezier automation", and then maybe a "linear automation", a "mask automation" and yet another way to handle transitions)
Deliberately there is an limitaion on the flexibility of what can be added to the system via Plugins. We allow configuration and parametrisation to be extended and we allow processing and data handling to be extended, but we disallow extensions to the fundamental structure of the system by plugins. They may provide new implementations for already known subsystems, but they can't introduce new subsystems not envisioned in the general design of the application.
* these fixed assortments include: the possbile kinds of MObject and [[Asset]], the possible automation parameter data types, the supported kinds of plugin systems
* while plugins may extend: the supported kinds of media, the [[media handling libraries|MediaImplLib]] used to deal with those media, the session storage backends, the behaviour of placments
A strong emphaisis is placed on ''Separation of Concerns'' and especially on ''Open Closed'' design. We resist the temptation to build an //universal model.// Rather, we understand our whole unterdaking a ''transformation of metadata'' -- which, as a whole, remains conceptual and is only ever implemented partially within context. A [[language of diff exchanges|TreeDiffFundamentals]] serves as integrating backbone. It works against a rather symbolic representation of strucutred matters, conceived as an ExternalTreeDescription. In fact, this desctiption is only materialised in parts, if at all.

This __proc-Layer__ and ~Render-Engine implementation started out as a design-draft by [[Ichthyo|mailto:Ichthyostega@web.de]] in summer 2007. The key idea of this design-draft is to use the [[Builder Pattern|http://en.wikipedia.org/wiki/Builder_pattern]] for the Render Engine, thus separating completely the //building// of the Render Pipeline from //running,// i.e. doing the actual Render. The Nodes in this Pipeline should process Video/Audio and do nothing else. No more decisions, tests and conditional operations when running the Pipeline. Move all of this out into the configuration of the pipeline, which is done by the Builder.
!Why doesn't the current Cinelerra-2 Design succeed?
The design of Cinelerra-2 basically follows a similar design, but __fails because of two reasons__
# too much differentiation is put into the class hierarchy instead of configuring Instances differently.<br>This causes overly heavy use of virtual functions and -- in order to ameliorate this -- falling back to hard wired branching
# far too much coupling and back-coupling to the internals of the »EDL«, forcing a overly rigid structure on the latter
!Try to learn from the Problems of the current Cinelerra-2 codebase
* build up an [[Node Abstraction|ProcNode]] powerful enough to express //all necessary Operations// without the need to recure on the actual object type
* need to redesign the internals of the Session in a far more open manner. Everything is an [[M-Object|MObjects]] which is [[placed|Placement]] in some manner
* strive at a StrongSeparation between Session and Render Engine, encapsulate and allways implement against interfaces
!Design Goals
As always, the main goal is //to cut down complexity// by the usual approach to separate into small manageable chunks.
To achieve this, here we try to separate ''Configuration'' from ''Processing''. Further, in Configuration we try to separate the ''high level view'' (users view when editing) from the ''low level view'' (the actual configuration effective for the calculations). Finally, we try to factor out and encapsulate ''State'' in order to make State explicit.
The main tool used to implement this separation is the [[Builder Pattern|http://en.wikipedia.org/wiki/Builder_pattern]]. Here especially we move all decisions and parametrization into the BuildProcess. The Nodes in the render pipeline should process Video/Audio and do nothing else. All decisions, tests and conditional operations are factored out of the render process and handled as configuration of the pipeline, which is done by the Builder. The actual color model and number of ports is configured in by a pre-built wiring descriptor. All Nodes are of equal footing with each other, able to be connected freely within the limitations of the necessary input and output. OpenGL and renderfarm support can be configured in as an alternate implementation of some operations together with an alternate signal flow (usable only if the whole Pipeline can be built up to support this changed signal flow), thus factoring out all the complexities of managing the data flow between core and hardware accelerated rendering. We introduce separate control data connections for the [[automation data|Automation]], separating the case of true multi-channel-effects from the case where one node just gets remote controlled by another node (or the case of two nodes using the same automation data).
Another pertinent theme is to make the basic building blocks simpler, while on the other hand gaining much more flexibility for combining these building blocks. For example we try to unfold any "internal-multi" effects into separate instances (e.g. the possibility of having an arbitrary number of single masks at any point of the pipeline instead of having one special masking facility encompassing multiple sub-masks. Similarly, we treat the Objects in the Session in a more uniform manner and gain the possibility to [[place|Placement]] them in various ways.

//Currently (5/2011) this page is used to collect and build up a coherent design for the player subsystem of Lumiera..//
!Starting point
The intention is to start out with the design of the PlayerDummy and to //transform//&nbsp; it into the full player subsystem.
* the ~DisplayService in that dummy player design moves down into Proc-Layer and becomes the OutputManager
* likewise, the ~DisplayerSlot is transformed into the interface OutputSlot, with various implementations to be registered with the OutputManager
* the core idea of having a play controler act as the frontend and handle to an PlayProcess is retained.
!!Idea
Introduce a new kind of MObject, a GeneratorMO, to represent what the current dummy player does: generate some kind of test data. As a first step, we might retro-fit the existing dummy player to this structure, and then expand it to have a minimal render engine setup for tests and diagnostics. That is, there will be some kind of GeneratorNode to produce test data.
!!Stages of transition
Reworking the dummy player into the [[Player subsystem|PlayService]] is a larger undertaking, and best broken up into several //stages.//
# just moving the existing entities into the final location (typically down in the layer hierarchy)
# introduce a GeneratorMO and a GeneratorNode, while providing a shortcut to get the GeneratorNode without requiring the (not yet implemented) Builder.
# rework the ~ProcessImpl and the ~TickService to become the [[calculation stream|CalcStream]] and use the real Engine Interface, but still with an mock implementation hooked up behind
# extend and elaborate the handling of output, build the foundation of the real OutputManager
# switch to the real Scheduler and Engine implementation
!!!!new entities to build {{red{#805}}}
* the ~DisplayService needs to be generalised into an OutputManager interface
* we need to introduce a new kind of StructAsset, the ViewerAsset.
* then, of course, the PlayService interface needs to be created
* several interfaces need to be shaped: the ''engine façade'' and a ''scheduler frontend''.
* the realms of player and rendering are interconnected within the [[dispatch operation|FrameDispatcher]]
* as a temporary soltion, a ''~DummyPlayConnection'' will be created hard-wired, within the ~DummyPlayer service. This dumy container is a test setup, explicitly creating a OutputManager implementation, a GeneratorMO, a GeneratorNode (thus omitting the not-yet-implemented Builder) and the corresponding ModelPort. This way, the ~DummyPlayer service can be changed to use the real PlayService for creating the dummy generated output data. It will then pass back the resulting PlayController to the existing GUI setup.
The Method of pushing frames to the Viewer widget is changed fundamentally during that transition: while previously, in the PlayerDummy design study, the GUI opened a Display-façade interface -- now rather the GUI has to register an OutputSlot implementation with the global OutputManager. Such a registration requires an explicit deregistration on shutdown, and this can be made to block, until no further data will be delivered. Using this approach we hope to circumvent problems with occasional crashes on closing the application, due to dispatching frame output signals in the event thread while shutdown is already in progress.

//A self referential intermediary, used to collect and consolidate structural information about some element.//
This structural information can then be emanated as ''population diff'' and is thus the foundation to build a further representation of the same structure //"elsewhere".// This //further representation// could be a model entity within the session's HighLevelModel, or it could be a widget tree in the UI. And, to close the circle, this other representation resulting from that diff could in turn again be a DiffConstituent.

The Dispatcher Tables are hosted within the [[Segmentation]] and serve as a ''strategy'' governing the play process.
For each [[calculation stream|CalcStream]] there is a concrete implementation of the [[dispatcher|FrameDispatcher]] interface, based on a specific configuration of dispatcher tables.

LayerSeparationInterface provided by the GUI.
Access point especially for the playback. A render- or playback process uses the DisplayFacade to push media data up to the GUI for display within a viewer widget of full-screen display. This can be thought off as a callback mechanism. In order to use the DisplayFacade, client code needs a DisplayerSlot (handle), which needs to be set up by the UI first and will be provided when starting the render or playback process.
!Evolving the real implementation {{red{TODO 1/2012}}}
As it stands, the ~DisplayFacade is a placeholder for parts of the real &rarr; OutputManagement, to be implemented in conjunction with the [[player subsystem|Player]] and the render engine. As of 1/2012, the intention is to turn the DisplayService into an OutputSlot instance -- following this line of thought, the ~DisplayFacade might become some kind of OutputManager, possibly to be [[implemented within a generic Viewer element|ViewerPlayConnection]]

A service within the GUI to manage output of frames generated by the lower layers of the application.
*providing the actual implementation of the DisplayFacade
*creating and maintaining of [[displayer slots|DisplayerSlot]], connected to viewer widgets or similar
{{red{TODO 1/2012}}} to be generalised into a service exposing an OutputSlot for display.

An output port wired up to some display facility or viewer widget within the UI. For the client code, each slot is represented by a handle, which can be used to lock into this slot for an continuous output process. Managed by the DisplayService

''EDL'' is a short-hand for __E__dit __D__ecision __L__ist. The use of this term can be confusing; for the usual meaning see the definition in [[Wikipedia|http://en.wikipedia.org/wiki/Edit_decision_list]]
Cinelerra uses this term in a related manner but with a somewhat shifted focus: In Cinelerra the EDL is comprised of the whole set of clips and other media objects arranged onto the tracks by the user. It is the result of the user's //editing efforts.//
In this usage, the EDL in most cases will be almost synonymous to &raquo;the session&laquo;, just the latter emphasizes more the state aspect. While the Lumiera project started out using the same terminology, later on, when support for multiple "containers" within the session and for [[meta-clips|VirtualClip]] was determined to be of much importance, the new term &raquo;[[Sequence]]&laquo; was preferred.

These are the tools provided to any client of the Proc layer for handling and manipulating the entities in the Session. When defining such operations, //the goal should be to arrive at some uniformity in the way things are done.// Ideally, when writing client code, one should be able to guess how to achieve some desired result.
!guiding principle
The approach taken to define any operation is based primarily on the ''~OO-way of doing things'': entities operate themselfs. You don't advise some manager, session or other &raquo;god class&laquo; to manipulate them. And, secondly, the scope of each operation will be as large as possible, but not larger. This often means performing quite some elementary operations &mdash; sometimes a convenience shortcut provided by the higher levels of the application may come in handy &mdash; and basically this gives rise to several different paths of doing the same thing, all of which need to be equivalent.
!main tasks
* you ''create a clip'' either from a source media or from another clip (copy, maybe clone?). The new clip always reflects the full size (and other properties) of the source used for creating.
* you can request a clip to ''resize'', which always relates to its current dimensions.
* you can ''place or attach'' the clip to some point or other entity by creating a [[Placement]] from the clip. (&rarr; [[handling of Placements|PlacementHandling]])
* you can ''adjust'' a placement relative to some other placement, meta object (i.e. selection), label or media, and this may cause the placement to resize or even effectively remove the clip.
* you can perform ''adjustments'' on a Sequence as a whole.
All these operations propagate to directly dependant objects and may schedule global reconfigurations.
!state, locking, policies
While performing any manipulative task, the state of the object is considered inconsistent, but it is guaranteed to be consistent after any such operation, irrespective of the result. There is no automatic atomicity and, consequently each propagation is considered a separate operation.
!!parallelism
At the level of fine grained operations on individual entities, there is ''no support for parallelism''. Individual entities don't lock themselves. Tasks perform locking and are to be scheduled. Thus, we need an isolation layer towards all inherently multithreaded parts of the system, like the GUI or the renderengine.
This has some obvious and some subtle consequences. Of course, manipulating //tasks// will be queued and scheduled somewhere. But as we rely on object identity and reference semantics for the entities in the session, some value changes are visible to operations going on in parallel threads (e.g. rendering) and each operation on the session entities //has to be aware of this.//
Consequently, sometimes there needs to be done sort of a ''read-copy-update'', i.e. self-replacement by copy followed by manipulation of the new copy, while ongoing processes use the unaltered original object until they receive some sort of reset.
!!undo
Basically, each elementary operation has to record the information necessary to be undone. It does so by registering a Memento with some central UndoManager facility. This Memento object contains a functor pre-bound with the needed parameter values. (Besides, the UndoManager is free to implement a second level of security by taking independent state snapshots).
{{red{to be defined in more detail later...}}}

We have to deal with effects on various different levels. One thing is to decide if an effect is applicable, another question is to find out about variable and automatable parameters, find a module which can be used as an effect GUI and finally to access a processing function which can be executed on a buffer filled with suitable data.
The effect asset (which is a [[processing asset|ProcAsset]]) provides an unified interface for loading external processing modules and querying their properties. As usual with assets, this is the "bookkeeping view" to start with, but we can create a //processor// from such an asset, i.e. an instance of the processing facility which can be used and wired into the model. Such a processor is an MObject and can be placed into the session (high level model); moreover it holds a concrete wiring for the various control parameters and it has an active link to the effect GUI module. As every ~MObject, it could be placed multiple times to affect several pipes, but all those different Placements would behave as being linked together (with respect to the control parameter sources and the GUI)
When building the low-level model, the actual processing code is resolved and a processing function is installed into the processing node representing the effect. This step includes checking of actual [[media type information|StreamType]], which may result in selecting a specifically tailored implementation function. The parameter wiring on the other hand is //shared// between the effect ~MObject, the corresponding GUI module and all low-level processing nodes. Actually, parameters are more of a reference than actually being values: they provide a {{{getValue()}}} function, which also works with automation. This way, any parameter or automation changes are reflected immediately into the produced output.
Initially, only the parameter (descriptors) are present on the effect ~MObject, while the actual [[parameter providers|ParamProvider]] are created or wired (by the ConManager) on demand.

The primary interface used by the upper application layers to interact with the render engine, to create and manage ongoing [[calculation streams|CalcStream]].
Below this facade, there is a thin adaptadion and forwarding layer, mainly talking to
* the individual [[Calculation Streams|CalcStream]] created for each PlayProcess.
* the FrameDispatcher, which translates such streams into a series of RenderJob entries
* the [[Scheduler]], which is responsible to perform these jobs in a timely fashion
!Quality of Service
Within the Facade, there is the definition of the {{{EngineService::Quality}}} tag, alongside with several pre-defined quality settings.
Actually this interface is a strategy, allowing to define quite specific quality levels, in case we need that. Clients can usually just use
these ~QoS-tags like enum values (they are copyable), without caring for the engine implementation related details.

A general identification scheme, combining a human readable symbolic name, unique within a //specifically typed context,// and machine readable hash ID (LUID). ~Entry-IDs allow for asset-like position accounting and for type safe binding between configuration rules and model obects. They allow for creating an entry with symbolic id and distinct type, combined with an derived hash value, without the overhead in storage and instance management imposed by using a full-fledged Asset.
Similar to an Asset, an identification tuple is available (generated on the fly), as is an unique LUID and total ordering. The type information is attached as template parameter, but included into the hash calculation. All instantiations of the EntryID template share a common baseclass, usable for type erased common registration.
~Entry-IDs as such do not provide any automatic registration or ensure uniqueness of any kind, but they can be used to that end. Especially, they're used within the TypedID registration framework for addressing individual entries {{red{planned as of 12/2010}}}
&rarr; TypedLookup
&rarr; TypeHandler
&rarr; MetaAsset

The &raquo;Timeline&laquo; is a sequence of ~MObjects -- here clips -- together with an ExplicitPlacement, locating each clip at a given time and track. (Note: I simplified the time format and wrote frame numbers to make it more clear)
[img[Example1: Objects in the Session/Fixture|uml/fig128773.png]]
----
After beeing processed by the Builder, we get the following Render Engine configuration
{{red{note: please take this only as a "big picture", the implementation details got a lot more complicated as of 6/08}}}
[img[Example1: generated Render Engine|uml/fig129029.png]]

{{red{TODO: seemingly this example is slightly outdated, as the implementation of placements is now indirect via LocatingPin objects}}}
This Example showes the //high level// Sequence as well. This needs to be transformed into a Fixture by some facility still to be designed. Basically, each [[Placement]] needs to be queried for this to get the corresponding ExplicitPlacement. The difficult part is to handle possible Placement constraints, e.g. one clip can't be placed at a timespan covered by another clip on the same track. In the current Cinelerra2, all of this is done directly by the GUI actions.
The &raquo;Timeline&laquo; is a sequence of ~MObjects -- note: using the same Object instances -- but now with the calculated ExplicitPlacement, locating the clip at a given time and track. The effect is located absolutely in time as well, but because it is the same Instance, it has the pointer to the ~RelativePlacement, wich basically attaches the effect to the clip. This structure may look complicated, but is easy to process if we go "backward" and just rely on the information contained in the ExplicitPlacement.
[img[Example2: Clip with Effect and generated Fixture for this Sequence|uml/fig128901.png]]
----
After beeing processed by the Builder, we get a Render Engine configuration.<br>
It has to be segmented at least at every point with changes in the configuration, but some variations are possible, e.g. we could create a Render Engine for every Frame (as Cinelerra2 does) or we could optimize out some configurations (for example the effect extended beyond the end of the clip)
{{red{note: as of 6/08 this can be taken only as the "big picture". Implementation will differ in details, and is more complicated than showed here}}}
[img[Example2: generated Render Engine|uml/fig129157.png]]

!MObject assembly
To make the intended use of the classes more clear, consider the following two example Object graphs:
* a video clip and a audio clip placed (explicitly) on two tracks &rarr;[[Example1]]
* a video clip placed relatively, with an attached HUE effect &rarr;[[Example2]]

a special ProcNode which is used to pull the finished output of one Render Pipeline (Tree or Graph). This term is already used in the Cinelerra2 codebase. I am unsure at the moment if it is a distinct subclass or rahter a specially configured ProcNode (a general design rule tells us to err in favour of the latter if in doubt).
The render nodes network is always built separate for each [[timeline segment|Segmentation]], which is //constant in wiring configuration.// Thus, while exit node(s) are per segment, the corresponding exit nodes of consecutive segments together belong to a ModelPort, which in turn corresponds to a global pipe (master bus not connected any further). These relations guide the possible configuration for an exit node: It may still provide multiple channels -- but all those channels are bound to belong to a single logical stream -- same StreamPrototype, always handled as bundle, connected and routed in one step. For example, when there is an 5.1 Audio master bus with a single fader, then "5.1 Audio" would be a prototype and these 6 channels will always be handled together; in such a case it makes perfectly sense to access these 6 audio channels through a single exit node, which is keyed (identified) by the same PipeID as used at the corresponding ModelPort and the corresponding [[global pipe|GlobalPipe]] ("5.1 Audio master bus")

A special kind (subclass) of [[Placement]]. As such it is always linked to a //Subject//, i.e. a MObject. But contrary to the (standard) placements, which may exhibit all kinds of fancy dynamic and scope dependent behaviour, within an explicit placement all properties are resolved and materialised. While the (standard) placement may contain an arbitrary list of LocatingPin objects, the resolution into an explicit placement performs a kind of »orthogonalisation«: each remaining LocatingPin defines exactly one degree of freedom independent of all others. Most notably, the explicit placement always specifies a absolute time and [[output designation|OutputDesignation]] for for locating the Subject. Explicit placements are ''immutable''.
!!Implementation considerations
Explicit placements are just created and never mutated, but copying and storage might become a problem.
It would thus be desirable to have a fixed-sized allocation, able to hold the placement body as well as the (fixed) locating pins inline.
!!!Storage
Explicit placements are value objects and stored at the respective usage site, most notably the [[Segmentation]]. They are //not// attached to the placement index in the session, nor do they bear any referential or indexing semantics. The only dynamic side effect of an explicit placement is to keep the reference count of the corresponding MObject up and thus keep it alive and accessible.

//to symbolically represent hierarchically structured elements, without actually implementing them.//
The purpose of this »external« description is to remove the need of a central data model to work against. We consider such a foundation data model as a good starting point, yet harmful for the evolution of any larger structure to be built. According to the subsidiarity principle, we prefer to turn the working data representation into a local concern. Which leaves us with the issue of collaboration. Any collaboration requires, as an underlying, some kind of common understanding. Yet any formalised, mechanical collaboration requires to represent that common point of attachment -- at least as a symbolic representation. The ExternalTreeDescription is shaped to fulfil this need: //in theory,// the whole field could be represented, symbolically, as a network of hierarchically structured elements. Yet, //in practice,// all we need is to conceive the presence of such a representation, as a backdrop to work against. And we do so -- we work against that symbolic representation, by describing ''changes'' made to the structure and its elements. Thus, the description of changes, the ''diff language'', refers to and partially embodies such symbolically represented elements and relations.
&rarr; TreeDiffFundamentals
&rarr; TreeDiffModel and [[-Implementation|TreeDiffImplementation]]
!Elements, Nodes and Records
We have to deal with //entities and relationships.// Entities are considered the building blocks, the elements, which are related by directional links. Within the symbolic representation, elements are conceived as [[generic nodes|GenNode]], while the directed relations are impersonated as being attached or rooted at the originating side, so the target of a relation has no traces or knowledge of being part of that relation. Moreover, each of our nodes bears a //relatively clear-cut identity.// That is to say, within the relevant scope in question, this identity is unique. Together, these are the building blocks to represent any ''graph''.
For practical purposes, we have to introduce some distinctions and limitations.
* we have to differentiate the generic node to be either a mere data element, or an object-like ''Record''
* the former, a mere data element, is considered to be "just data", to be "right here" and without further meta information. You need to know what it is to deal with it.
* to the contrary, a Record has an associated, symbolic type-ID, plus it can potentially be associated with and thus relate to further elements, with the relation originating at the Record.
* and we distinguish //two different kinds of relations possibly originating from a Record://
** ''attributes'' are known by-name; they can be addressed through this name-ID as a key, while the value is again a generic node, possibly even another record.
** ''children'' to the contrary can only be enumerated; they are considered to be within (and form) the ''scope'' of the given Record (object).
And there is a further limitation: The domain of possible data is fixed, even hard wired. Implementation-wise, this turns the data within the generic node into a »Variant« (typesafe union). Basically, this opens two different ways to //access// the data within a given GenNode: either you know the type to expect ''beforehand'' (and the validity of this assumption is checked on each access -- please recall, all of this is meant for symbolic representation, not for implementation of high performance computing). Or we offer the ability for //generic access// through a ''variant visitor'' (double dispatch). The latter includes the option not to handle all possible content types (making the variant visitor a //partial function// -- as in any non exhaustive pattern match).
Basically, you can expect to encounter the following kinds of data
*{{{int}}}, {{{int64_t}}}, {{{short}}}, {{{char}}}
*{{{bool}}}
*{{{double}}}
*{{{std::string}}}
*{{{time::Time}}}, {{{time::Duration}}}, {{{time::TimeSpan}}}
*{{{hash::LuidH}}}
*{{{diff::Record<GenNode>}}}
The last option is what makes our representation recursive.
Regarding the implementation, all these data elements are embedded //inline,// as values.
With the exception of the record, which, like any {{{std::vector}}} implicitly uses heap allocations for the members of the collection.
!{{red{open questions 5/2015}}}
!!!the monadic nature of records
A //monad// is an entity which supports the following operations
*construction:{{{a:A -> M<A>(a)}}}
*flatMap:{{{(M<A>, f:A->M<B>) -> M<B>}}}
Operationally, a modad can be constructed to wrap-up and embody a given element. And to flat-map a monad generating function means to apply it to all "content elements" within the given source monad, and then to //join// the produced monads "flat" into a new, uniform monad of the new type.
Now, the interesting question is: //what does "join" mean?// --
*will it drill down?
*will it lift the contents of generated monads into the parent level, or attach them as new subtrees?
*will the function get to see {{{Record}}} elements, or will it immediately see the "contents", the attributes or the children?
!!!names, identity and typing
It was a design decision that the GenNode shall not embody a readable type field, just a type selector within the variant to hold the actual data elements.
This decision more or less limits the usefulness of simple values as children to those cases, where all children are of uniform type, or where we agree to deal with all children through variant visitation solely. Of course, we can still use simple values as //attributes,// since those are known and addressed by name. And we can use filtering by type to limit access to some children of type {{{Record}}}, since every record does indeed embody a symbolic type name, an attribute named {{{"type"}}}. It must be that way, since otherwise, records would be pretty much useless as representation for any object like entity.
The discriminating ID of any GenNode can serve as a name, and indeed will be used as the name of an attribute within a record.
Now, the interesting question is: what constitutes the full identity? Is it the ~ID-string? does it also include some kind of type information, so that two children with the same name, but different types would be considered different? And, moreover, we could even go as far as to make the path part of the identity, so that two otherwise identical elements would be different, when located at different positions within the graph. But since we did not rule out cyclic graphs, the latter idea would necessitate the notion of an //optimal path// --
A somewhat similar question is the ordering and uniqueness of children. While attributes -- due to the usage of the attribute node's ID as name-key -- are bound to be unique within a given Record, children within the scope of a record could be required to be unique too, making the scope a set. And, of course, children could be forcibly ordered, or just retain the initial ordering, or even be completely unordered. On a second thought, it seems wise not to impose any guarantees in that regard, beyond the simple notion of retaining an initial sequence order, the way a »stable« sorting algorithm does. All these more specific ordering properties can be considered the concern of some specific kinds of objects -- which then just happen to "supply" a list of children for symbolic representation as they see fit.

The use of factories separates object creation, configuration and lifecycle from the actual usage context. Hidden behind a factory function
* memory management can be delegated to a centralised facility
* complex internal details of service implementation can be turned into locally encapsulated problems
* the actual implementation can be changed, e.g. by substituting a test mock
!common usage pattern
Within Lumiera, factories are frequently placed as a static field right into the //service interface.// Common variable names for such an embedded factory are {{{instance}}} or {{{create}}}.
The actual fabrication function is defined as function operator -- this way, the use of the factory reads like a simple function call. At usage site, only the reference to the //interface// or //kind of service// is announced.
!flavours
* A special kind of factory used at various places is the ''singleton'' factory for access to application wide services and facade interfaces. All singleton factories for the same target type share a single instance of the target, which is created lazily on first usage. Actually, this is our very special version of [[dependency injection|DependencyFactory]]
* whenever there is a set of objects of some kind, which require registration and connection to a central entity, client code gets (smart) handles, which are emitted by a factory, which connects to the respective manager for registration automatically.
* to bridge the complexities of using an external (plug-in based) interface, proxy objects are created by a factory, wired to invoke the external calls to forward any client invoked operation.
* the ''configurable factory'' is used to define a family of production lines, which are addressed by the client by virtue of some type-ID. Runtime data and criteria are used to form this type-ID and thus pick the suitable factory function

A grouping device within an ongoing [[playback or render process|PlayProcess]].
Any feed corresponds to a specific ModelPort, which in turn typically corresponds to a given GlobalPipe.
When starting playback or render, a play process (with a PlayController front-end for client code) is established to coordinate the processing. This ongoing data production might encompass multiple media streams, i.e. multiple feeds pulled from several model ports and delivered into several [[output slots|OutputSlot]]. Each feed in turn might carry structured MultichannelMedia, and is thus further structured into individual [[streams of calculation|CalcStream]]. Since the latter are //stateless descriptors,// while the player and play process obviously is stateful, it's the feed's role to mediate between a state-based (procedural) and a stateless (functional and parallelised) organisation model -- ensuring a seamless data feed even during modification of the playback parameters.

a specially configured view -- joining together high-level and low-level model.
The Fixture acts as //isolation layer// between the two models, and as //backbone to attach the render nodes.//
* all MObjects have their position, length and configuration set up ready for rendering.
* any nested sequences (or other kinds of indirections) have been resolved.
* every MObject is attached by an ExplicitPlacement, which declares a fixed position (Time, [[Pipe|OutputDesignation]])
* these ~ExplicitPlacements are contained immediately within the Fixture, ordered by time
* besides, there is a collection of all effective, possibly externally visible [[model ports|ModelPortRegistry]]
As the builder and thus render engine //only consults the fixture,// while all editing operations finally propagate to the fixture as well, we get an isolation layer between the high level part of the Proc layer (editing, object manipulation) and the render engine. [[Creating the Fixture|BuildFixture]] is an important first step and sideeffect of running the [[Builder]] when createing the [[render engine network|LowLevelModel]].
''Note'': all of the especially managed storage of the LowLevelModel is hooked up behind the Fixture
&rarr; FixtureStorage
&rarr; FixtureDatastructure
!{{red{WIP}}} Structure of the fixture
[<img[Structure of the Fixture|draw/Fixture1.png]]
The fixture is like a grid, where one dimension is given by the [[model ports|ModelPortRegistry]], and the other dimension extends in time. Within the time dimension there is a grouping into [[segments|Segmentation]] of constant structure.
;Model Ports
:The model ports share a single uniform and global name space: actually they're keyed by ~Pipe-ID
:Model ports are derived as a result of the build process, as the //residuum// of all nodes not connected any further
:Each port belongs to a specific Timeline and is associated with the [[Segmentation]] of that timeline.
;Segmentation
:The segmentation partitiones the time axis of a single timeline into segments of constant (wiring) configuration
:Together, the segments form a seamless sequence of time intervals. They contain a copy of each (explicit) placement of a visible object touching that time interval. Besides that, segments are the top level grouping device of the render engine node graph; individual segments are immutable, they are built and discarded as a whole chunk.
:Segments (and even a different Segmentation) may be //hot swapped// into an ongoing render.
;Exit Nodes
:Each segment holds an ExitNode for each relevant ModelPort of the corresponding timeline.
:Thus the exit nodes are keyed by ~Pipe-ID as well (and consequently have a distinct [[stream type|StreamType]]) -- each model port coresponds to {{{<number_of_segments>}}} separate exit nodes, but of course an exit node may be //mute//&nbsp; in some segmehts.

Generally speaking, the datastructure to implement the ''Fixture'' (&rarr; see a more general description [[here|Fixture]]) is comprised of a ModelPortRegistry and a set of [[segmentations|Segmentation]] per Timeline.
This page focusses on the actual data structure and usage details on that level. See also &rarr; [[storage|FixtureStorage]] considerations.
!transactional switch
A key point to note is the fact that the fixture is frequently [[re-built|BuildFixture]] by the [[Builder]], while render processes may be going on in parallel. Thus, when a build process is finished, a transactional ''commit'' happens to ''hot swap'' the new parts of the model. This is complemented by a clean-up of tainted render processes; finally, storage can be reclaimed.
To support this usage pattern, the Fixture implementation makes use of the [[PImpl pattern|http://c2.com/cgi/wiki?PimplIdiom]]
!Collecting usage scenarios {{red{WIP 12/10}}}
* ModelPort access
** get the model port for a given ~Pipe-ID
** enumerate the model ports for a Timeline
* rendering frame dispatch
** get or create the frame dispatcher table
** dispatch a single frame to yield the corresponding ExitNode
* (re)building
** create a new implementation transaction
** create a new segmentation
** establish what segments actually need to be rebuilt
** dispatch a newly built segment into the transaction
** schedule superseded segments and tainted process for clean-up
** commit a transaction
!Conclusions about the structure {{red{WIP 12/10}}}
* the ~PImpl needs to be a single (language) pointer. This necessitates having a monolithic Fixture implementation holder
* moreover, this necessitates a tight integration down to implementation level, both with the clean-up and the render processes themselves

The Fixture &rarr; [[data structure|FixtureDatastructure]] acts as umbrella to hook up the elements of the render engine's processing nodes network (LowLevelModel).
Each segment within the [[Segmentation]] of any timeline serves as ''extent'' or unit of memory management: it is built up completely during the corresponding build process and becomes immutable thereafter, finally to be discarded as a whole when superseded by a modified version of that segment (new build process) -- but only after all related render processes (&rarr; CalcStream) are known to be terminated.
Each segment owns an AllocationCluster, which in turn manages all the numerous small-sized objects comprising the render network implementing this segment -- thus the central question is when to //release the segment.//
* for one, we easily detect the point when a segment is swapped out of the segmentation; at this point we also have to detect the //tainted calculation streams.//
* but those render processes (calc streams) terminate asynchronously, and that forces us to do some kind of registration and deregistration.
!!question: is ref-counting acceptable here?
//Not sure yet.// Of course it would be the simplest approach. KISS.
Basically the concern is that each new CalcStream had to access the shared counts of all segments it touches.
''Note'': {{{shared_ptr}}} is known to be implemented by a lock-free algorithm (yes it is, at least since boost 1.33. Don't believe what numerous FUD spreaders have written). Thus lock contention isn't a problem, but at least a memory barrier is involved (and if I&nbsp;judge GCC's internal documentation right, currently their barriers extend to //all// globally visible variables)
!!question: alternatives?
There are. As the builder is known to be run again and again, no one forces us to deallocate as soon as we could. That's the classical argument exploited by any garbage collector too. Thus we could just note the fact that a calculation stream is done and re-evaluate all those noted results on later occasion. Obviously, the [[Scheduler]] is in the best position for notifying the rest of the system when this and that [[job|RenderJob]] has terminated, because the Scheduler is the only facility required to touch each job reliably. Thus it seems favourable to add basic support for either termination callbacks or for guaranteed execution of some notification jobs to the [[Scheduler's requirements|SchedulerRequirements]].
!!exploiting the frame-dispatch step
Irrespective of the decision in favour or against ref-counting, it seems reasonable to make use of the //frame dispatch step,// which is necessary anyway. The idea is to give each render process (maybe even each CalcStream) a //copy//&nbsp; of a dispatcher table object -- basically just a list of breaking points in time and a pointer to the corresponding relevant exit node. If we keep track of those dispatcher tables, add some kind of back-link to identify the process and require the process in turn to deregister, we might get a tracking of tainted processes for free.
!!assessment {{red{WIP 12/10}}}
But the primary question here is to judge the impact of such an implementation. What would be the costs?
# creating individual dispatcher tables trades memory for simplified parallelism
# the per-frame lookup is efficient and negligible compared with just building the render context (StateProxy) for that frame
# when a process terminates, we need to take out that dispatcher and do deregistration stuff for each touched segment (?)
!!!Estimations
* number of actually concurrent render processes is at or below 30
* depending on the degree of cleverness on behalf of the scheduler, the throughput of processes might be multiplied (dull means few processes)
* the total number of segments within the Fixture could range into several thousand
* but esp. playback is focussed, which makes a number of rather several hundred tainted segments more likely
&rArr; we should try quickly to dispose of the working storage after render process termination and just retain a small notification record
&rArr; so the frame dispatcher table should be allocated //within//&nbsp; the process' working storage; moreover it should be tiled
&rArr; we should spend a second thought about how actually to find and process the segments to be discarded
!!!identifying tainted and disposable segments
Above estimation hints at the necessity of frequently finding some 30 to 100 segments to be disposed, out of thousands, assumed the original reason for triggering the build process was a typically local change in the high-level model. We can only discard when all related processes are finished, but there is a larger number of segments as candidate for eviction. These candidates are rather easy to pinpoint -- they will be uncovered during a linear comparison pass prior to committing the changed Fixture. Usually, the number of candidates will be low (localised changes), but global manipulations might invalidate thousands of segments.
* if we frequently pick the segments actually to be disposed, there is the danger of performance degeneration when the number of segments is high
* the other question is if we can afford just to keep all of those candidates around, as all of them are bound to get discardable eventually
* and of course there is also the question how to detect //when// they're due.
;Model A
:use a logarithmic datastructure, e.g. a priority queue. Possibly together with LRU ordering
:problem here is that the priorities change, which either means shared access or a lot of "obsoleted" entries in this queue
;Model B
:keep all superseded segments around and track the tainted processes instead
:problem here is how to get the tainted processes precisely and with low overhead
//as of 12/10, decision was taken to prefer Model B...//
Simply because the problems caused by Model A seem to be fundamental, while the problems related to Model B could be overcome with some additional cleverness.
But actually, at that point I'm struck here, because of the yet limited knowledge about those render processes....
* how do we //join// an aborted/changed rendering process to his successor, without creating a jerk in the output?
* is it even possible to continue a process when parts of the covered time-range are affected by a build?
If the latter question is answered with "No!", then the problem gets simple in solution, but maybe memory consuming: In that case, //all//&nbsp; the processes related to a timeline are affected and thus get tainted; we'd just dump them onto a pile and delay releasing all of the superseded segments until all of them are known to be terminated.
!!re-visited 1/13
the last conclusions drawn above where confirmed by the further development of the overall design. Yes, we do //supersede// frequently and liberally. This isn't much of a problem, since the preparation of new jobs, i.e. the [[frame dispatch step|FrameDispatcher]] is performed chunk wise. A //continuation job// is added at the end of each chunk, and this continuation will pick up the task of job planning in time.
At the 1/2013 developer's meeting, Cehteh and myself had a longer conversation regarding the topic of notifications and superseding of jobs within the scheduler. The conclusion was to give ''each play process a separate LUID'' and treat this as ''job group''. The scheduler interface will offer a call to supersede all jobs within a given group.
Some questions remain though
* what are those nebulous dispatcher tables?
* do they exist per CalcStream ?
* is it possible, to file these dedicated dispatch informations gradually?
* how to store and pass on the control information for NonLinearPlayback?
since the intention is to have dedicated dispatch tables, these would implement the {{{engine.Dispatcher}}} interface and incorporate some kind of strategy corresponding to the mode of playback. The chunk wise continuation of the job planning process would have to be reformulated in terms of //real wall clock time rather// -- since the relation of the playback process to nominal time can't be assumed to be a simple linear progression in all cases.
!liabilities of fixture storage {{red{WIP 1/14}}}
The net effect of all the facilities related to fixture storage is to keep ongoing memory allocations sane. The same effect could be achieved by using garbage collection -- but the latter solves a much wider area of problems and as such incurs quite some price in terms of lock contention or excessive memory usage. Since our problem here is confined to a very controlled setup, employing a specific hand made solution will be more effective. Anyway, the core conflict is not to hinder parallel execution of jobs and to avoid excessive use of memory.
The management of fixture storage has to deal with some distinct situations
;superseding a CalcStream
:as long as the respective calculations are still going on, the commonly used data structures need to stay put
:* the {{{CalcPlanContinuation}}} closure used by the job planning jobs
:* the {{{RenderEnvironmentClosure}}} shared by all ~CalcStreams for for the same output configuration
:* the -- likewise possibly shared -- specific strategy record to govern the playback mode
;superseding a [[Segment|Segmentation]]
:as long as any of the //tainted// ~CalcStreams is still alive, all of the data structures held by the AllocationCluster of that segment need to stay around
:* the DispatcherTables
:* the JobTicket structure
:* the [[processing nodes|ProcNode]] and accompanying WiringDescriptor records
!!!conclusions for the implementation
In the end, getting the memory management within Segmentation and Playback correct boils down into the following requirements
* the ability to identify ~CalcStreams touching a segment about to be obsoleted
* the ability to track such //tainted ~CalcStreams//
* the ability to react on reaching a pre defined //control point,// after which releasing of resources is safe
The building blocks for such a chain of triggers and reactions are provided by a helper facility, the &rarr; SequencePointManager
__3/2014__: The crucial point seems to be the impedance mismatch between segments and calculation streams. We have a really high number of segments, which change only occasionally. But we have a rather small number of calculation streams, which mutate rapidly. And, over time, any calculation stream might -- occasionally -- touch a large number of segments. Thus, care should be taken not to implement the dependency structure naively. We only need to care about the tainted calculation streams when it comes to discarding a segment.

Within Lumiera, tracks are just a structure used to organize the Media Objects within the Sequence. Tracks are associated allways to a specific Sequence and the Tracks of an Sequence form a //tree of tracks.// They can be considered to be an organizing grid, and besides that, they have no special meaning. They are grouping devices, not first-class entities. A track doesn't "have" a port or pipe or "is" a video track and the like; it can be configured to behave in such manner by using placements.
To underpin this design decision, Lumiera introduces the more generic concept of a ''Fork'' -- to subsume the "tracks" within the timeline, as well as the "media bins" in the asset management section
The ~Fork-IDs are assets on their own, but they can be found within a given sequence. So, several sequences can share a single track or each sequence can hold tracks with their own, separate identity. (the latter is the default)
* Like most ~MObjects, tracks have a asset view: you can find a track asset (actually just a fork ID) in the asset manager.
* and they have an object view: there is an ''Fork'' MObject which can be [[placed|Placement]], thus defining properties of this track within one sequence, e.g. the starting point in time
Of course, we can place other ~MObjects relative to some fork (that's the main reason why we want to have tracks). In this sense, the [[handling of Tracks|TrackHandling]] is somewhat special: the placements forming the fork ("tree of tracks") can be accessed directly through the sequence, and a fork acts as container, forming a scope to encompass all the objects "on" this track. Thus, the placement of a fork defines properties of "the track", which will be inherited (if necessary) by all ~MObjects placed within the scope of this fork. For example, if placing (=plugging) a fork to some global [[Pipe]], and if placing a clip to this fork, without placing the clip directly to another pipe, the associated-to-pipe information of the fork will be fetched by the builder when needed to make the output connection of the clip.
&rarr; [[Handling of Tracks|TrackHandling]]
&rarr; [[Handling of Pipes|PipeHandling]]
&rarr; [[Anatomy of the high-level model|HighLevelModel]]

The situation focussed by this concept is when an API needs to expose a sequence of results, values or objects, instead of just yielding a function result value. As the naive solution of passing an pointer or array creates coupling to internals, it was superseded by the ~GoF [[Iterator pattern|http://en.wikipedia.org/wiki/Iterator]]. Iteration can be implemented by convention, polymorphically or by generic programming; we use the latter approach.
!Lumiera Forward Iterator concept
''Definition'': An Iterator is a self-contained token value, representing the promise to pull a sequence of data
* rather then deriving from an specific interface, anything behaving appropriately //is a Lumiera Forward Iterator.//
* the client finds a typedef at a suitable, nearby location. Objects of this type can be created, copied and compared.
* any Lumiera forward iterator can be in //exhausted// &nbsp;(invalid) state, which can be checked by {{{bool}}} conversion.
* especially, default constructed iterators are fixed to that state. Non-exhausted iterators may only be obtained by API call.
* the exhausted state is final and can't be reset, meaning that any iterator is a disposable one-way-off object.
* when an iterator is //not//&nbsp; in the exhausted state, it may be //dereferenced// ({{{*i}}}), yielding the "current" value
* moreover, iterators may be incremented ({{{++i}}}) until exhaustion.
!!Discussion
The Lumiera Forward Iterator concept is a blend of the STL iterators and iterator concepts found in Java, C#, Python and Ruby. The chosen syntax should look familiar to C++ programmers and indeed is compatible to STL containers and ranges. To the contrary, while a STL iterator can be thought off as being just a disguised pointer, the semantics of Lumiera Forward Iterators is deliberately reduced to a single, one-way-off forward iteration, they can't be reset, manipulated by any arithmetic, and the result of assigning to an dereferenced iterator is unspecified, as is the meaning of post-increment and stored copies in general. You //should not think of an iterator as denoting a position// &mdash; just a one-way off promise to yield data.
Another notable difference to the STL iterators is the default ctor and the {{{bool}}} conversion. The latter allows using iterators painlessly within {{{for}}} and {{{while}}} loops; a default constructed iterator is equivalent to the STL container's {{{end()}}} value &mdash; indeed any //container-like// object exposing Lumiera Forward Iteration is encouraged to provide such an {{{end()}}}-function, additionally enabling iteration by {{{std::for_each}}} (or Lumiera's even more convenient {{{util::for_each()}}}).
!!!interoperation with the C++11 range-for construct
Lumiera Forward Iterators can be made to work together with the 'range for loop', as introduced with C++11. The preferred solution is to define the necessary free functions {{{begin}}} and {{{end}}} for ADL. This is best done on a per implementation base. We consider a generic solution not worth the effort. See {{{71167be9c9aaa}}}.
This is to say, these two concepts can be made to work together well. While, at a conceptual level, they are not really compatible, and build on a different understanding: The standard for-loop assumes //a container,// while our Forward Iterator Concept deals with //abstract data sources.//.
The user needs to understand the fine points of that conceptual difference:
* if you apply the range-`for` construct on a container, you can iterate as often as you like. Even if the iterators of that container are implemented in compliance with the Lumiera Forward Iterator concept.
* but if you apply the range-`for` construct on //a given iterator, // you can do so only once. There is no way to reset that iterator, other than obtaining a new one.
The bridge methods are usually defined so that the {{{end}}}-function just returns a default constructed iterator, which -- by concept -- is the marker for iteration end
!!Implementation notes
''iter-adapter.hpp'' provides some helper templates for building Lumiera Forward Iterators.
* __~IterAdapter__ is the most flexible variant, intended for use by custom facilities. An ~IterAdapter maintains an internal back-link to a facilitiy exposing an iteration control API, which is accessed through free functions as extension point. This iteration control API is similar to C#, allowing to advance to the next result and to check the current iteration state.
* __~RangeIter__ wraps two existing iterators &mdash; usually obtained from {{{begin()}}} and {{{end()}}} of an STL container embedded within the implementation. This allows for iterator chaining.
* __~PtrDerefIter__ works similar, but can be used on an STL container holding //pointers,// to be dereferenced automatically on access
Similar to the STL habits, Lumiera Forward Iterators should expose typedefs for {{{pointer}}}, {{{reference}}} and {{{value_type}}}.
Additionally, they may be used for resource management purposes by embedding a ref-counting facility, e.g. allowing to keep a snapshot or restult set around until it can't be accessed anymore.

This term has //two meanings, //so care has to be taken for not confusing them.
# in general use, a Frame means one full image of a video clip, i.e an array of rows of pixels. For interlaced footage, one Frame contains two halfimages, commonly called Fields. (Cinelerra2 confuses this terms)
# here in this design, we use Frame as an abstraction for a buffer of raw media data to be processed. If in doubt, we should label this "DataFrame".
#* one video Dataframe contains a single video frame
#* one audio Dataframe contains a block of raw audio samples
#* one OpenGL Dataframe could contain raw texture data (but I am lacking expertise for this topic)

An entity within the RenderEngine, responsible for translating a logical [[calculation stream|CalcStream]] (corresponding to a PlayProcess) into a sequence of individual RenderJob entries, which can then be handed over to the [[Scheduler]]. Performing this operation involves a special application of [[time quantisation|TimeQuant]]: after establishing a suitable starting point, a typically contiguous series of frame numbers need to be generated, together with the time coordinates for each of those frames.
The dispatcher works together with the job ticket(s) and the scheduler; actually these are the //core abstractions//&nbsp; the process of ''job planning'' relies on. While the actual scheduler implementation lives within the backend, the job tickets and the dispatcher are located within the [[Segmentation]], which is the backbone of the [[low-level model|LowLevelModel]]. More specifically, the dispatcher interface is //implemented//&nbsp; by a set of &rarr; [[dispatcher tables|DispatcherTables]] within the segmentation.
{{red{stalled since 2014}}} -- development on this (important) topic has been postponed. Moreover, some rough edges remain within the Design &rarr; see [[some notes...|AboutMonads]]
!defining the dispatcher interface
The purpose of this interface is to support the planning of new jobs, for a given CalcStream. From time to time, a chunk of new RenderJob entries will be prepared for the [[Scheduler]]. Each job knows his frame number and the actual ProcNode to operate. So, to start planning jobs, we need to translate time &rarr; frame number &rarr; segment &rarr; real exit node.
!!!Invocation situation
* our //current starting point//&nbsp; is given as ''time anchor'' closure
* we want to address a specific frame, offset by a given number of frames relative to the current position
* &rArr; the dispatcher returns the //fundamental coordinates// of that frame (''frame coordinates''), comprised of
** the absolute nominal time
** the absolute frame number
** a concrete ExitNode to pull for this frame
** focussed context information:
*** the relevant [[segment|Segmentation]] responsible for producing this frame
*** the corresponding JobTicket to use at this point
!!!the ~TimeAnchor
In fact, the whole process of playback or rendering is a continued series of exploration and evaluation. The outline of what needs to be calculated is determined continuously, proceeding in chunks of evaluation. The evaluation structure of the render engine is quite similar to the //fork-join//-pattern, just with the addition of timing constraints. This leads to an invocation pattern, where a partial evaluation happens from time to time. Each of those evaluations establishes a breaking point in time: everything //before// this point is settled and planned thus far. So, this point is an ''anchor'' or closure to root the next partial evaluation. More specifically, this ~TimeAnchor closure is the definitive binding between the abstract logical time of the session timeline, and the real wall-clock time forming the deadline for render evaluation.
!!!related timelines and the frame grid
The frame dispatch step joins and combines multiple time axes. Through the process of //scheduling,// the output generation is linked to real ''wall clock time'' and the dispatch step establishes the deadlines, taking the ''engine latency'' into account. As such, any render or playback process establishes an ''output time grid'', linking frame numbers to nominal output time or timecode, based on the ''output frame rate'' -- and both the framerate and the actual progression (speed) might be changed dynamically. But beyond all of this there is a third, basically independent temporal structure involved: the actual content to render, the ''effective session timeline''. While encoded in nominal, absolute, internal time values not necessarily grid aligned, in practice at least the //breaking points,// the temporal location where the content or structure of the pipeline changes, are aligned //to a grid used while creating the edit.// Yet this session timing structure need not be related in any way to the playback grid, nor is it necessarily the same as the ''source grid'' defined by the media data used to feed the pipeline.
These complex relationships are reflected in the invocation structure leading to an individual frame job. The [[calculation stream|CalcStream]] provides the [[render/playback timings|Timings]], while the actual implementation of the dispatcher, backed by the [[Fixture]] and thus linked to the session models, gets to relate the effective nominal time, the frame number, the exit node and the //processing function.//
!!!controlling the planning process
New render jobs are planned as an ongoing process, proceeding in chunks of evaluation. Typically, to calculate a single frame, several jobs are necessary -- to find out which and how, we'll have to investigate the model structures corresponding to this frame, resulting in a tree of prerequisites. Basically, the planning for each frame is seeded by establishing the nominal time position, in accordance to the current [[mode of playback|NonLinearPlayback]]. Conducted by the [[play controller|PlayController]], there is a strategy to define the precise way of spacing and sequence of frames to be calculated -- yet for the actual process of evaluating the prerequisites and planning the jobs, those details are irrelevant and hidden behind the dispatcher interface, as is most of the model and context information. The planning operation just produces a sequence of job definitions, which can then be associated with real time (wall clock) deadlines for delivery. The relation between the spacing and progression of the nominal frame time (as controlled by the playback mode) and the actual sequence of deadlines (which is more or less dictated by the output device) is rather loose and established anew for each planning chunk, relying on the ''time anchor''. The latter in turn uses the [[timings record|Timings]] of the [[calculation stream|CalcStream]] currently being planned, and these timings act as a strategy to represent the underlying timing grid and playback modalities.
While the sequence of frame jobs to be planned is possibly infinite, the actual evaluation is confined to the current planning chunk. When done with planning such a chunk of jobs, an additional ''continuation job'' is included to prepare a re-invocation of the planning function for preparation of the next chunk. Terminating playback is equivalent to not including or not invoking this continuation job. Please note that planning proceeds independently for each [[Feed]] -- in Lumiera the //current playback position//&nbsp; is just a conceptual projection of wall clock time to nominal time, yet there is no such thing like a synchronously proceeding "Playhead"
!!!producing actual jobs
The JobTicket is created on demand, specialised for a single [[segment|Segmentation]] of the timeline and a single [[feed|Feed]] of data frames to be pulled from a ModelPort. Consequently this means that all frames and all channels within that realm will rely on the same job ticket -- which is a //higher order function,// a function producing another function: when provided with the actual channel number and the specific frame coordinates, the job ticket produces a [[concrete job definition|RenderJob]], which itself is a function to be invoked by the [[scheduler|Scheduler]] just in time.

The ''Gmerlin Audio Video Library''. &rarr; see [[homepage|http://gmerlin.sourceforge.net/gavl.html]]
Used within Lumiera as a foundation for working with raw video and audio media data

__G__roup __of__ __P__ictures: several compressed video formats don't encode single frames. Normally, such formats are considered mere //delivery formates// but it was one of the key strenghts of Cinelrra from start to be able to do real non linear editing on such formats (like the ~MPEG2-ts unsed in HDV video). The problem of course is that the data backend needs to decode the whole GOP to be serve single raw video frames.
For this Lumiera design, we could consider making GOP just another raw media data frame type and integrate this decoding into the render pipeline, similar to an effect based on several source frames for every calculated output frame.
&rarr;see in [[Wikipedia|http://en.wikipedia.org/wiki/Group_of_pictures]]

//Abstract generic node element to build a ~DOM-like rendering of Lumiera's [[session model|HighLevelModel]].//
GenNode elements are values, yet behave polymorphic. They are rather light-weight, have an well established identity and can be compared. They are //generic// insofar they encompass several heterogeneous ordering systems, which in themselves can not be subsumed into a single ordering hierarchy. The //purpose// of these generic nodes is to build a symbolic representation, known as [[external tree description|ExternalTreeDescription]], existing somewhere "outside", at a level where the fine points of ordering system relations do not really matter. Largely, this external description is not represented or layed down as a whole. Rather, it is used as a conceptual reference frame to describe //differences and changes.// Obviously, this means that parts of the altered structures have to appear in the description of the modifications. So, practically speaking, the prime purpose of GenNode elements is to appear as bits of information within a ''diff language'' to exchange such information of changes.
To be more specific, within the actual model there are [[Placements|Placement]]. These refer to [[M-Objects|MObject]]. Which in turn rely on [[Assets|Asset]]. Moreover, we have some processing rules, and -- last but not least -- the "objects" encountered in the model have state, visible as attributes of atomic value type (integral, floating point, string, boolean, time, time ranges and [[quantised time entities|TimeQuant]]).
A generic node may //represent any of these kind// -- and it may have ~GenNode children, forming a tree. Effectively all of this together makes ~GenNode a ''Monad''.
GenNode elements are conceived as values, and can thus be treated as mutable or immutable; it is up to the user to express this intent through const correctness.
Especially the nested structures, i.e. a GenNode holding an embedded {{{diff::Record}}}, are by default immutable, but expose a object builder API for remoulding. This again places the actual decision about mutations into the usage context, since the remoulded Record has to be assigned explicitly. In fact, this is an underlying theme of the whole design laid out here: Data represented as GenNode graph is //structured, yet opaque.// It is always tied into an usage context, that "happens to know" what to expect. Moreover, the values to be integrated into such a structure are of limited type -- covering just the basic kinds of data plus a recursive structuring device:
* integral values: {{{int}}}, {{{int64_t}}}, {{{short}}}, {{{char}}}
* logic values: {{{bool}}}
* measurement data: {{{double}}}
* textual data: {{{std::string}}}
* time representation: {{{time::Time}}}, {{{time::Duration}}}, {{{time::TimeSpan}}}
* identity and reference: {{{hash::LuidH}}}
* object-like structure: {{{diff::Record<GenNode>}}}
* for cross-links and performance considerations, we also provide a {{{Record}}}-//reference element.//
!to reflect or not reflect
When dealing with this external model representation, indeed there are some rather global concerns which lend themselves to a generic programming style. Simply because, otherwise, we'd end up explicating and thereby duplicating the structure of the model all over the code. Frequently, such a situation is quoted as the reason to demand introspection facilities on any data structure. We doubt this is a valid conclusion. Since introspection allows to accept just //any element// -- followed by an open-ended //reaction on the received type// -- we might arrive at the impression that our code reaches a maximum of flexibility and "openness". Unfortunately, this turns out to be a self-deception, since code to do any meaningful operation needs pre-established knowledge about the meaning of the data to be processed. More so, when, as in any hierarchical data organisation, the relevant meaning is attached to the structure itself, so consequently this pre-established knowledge tends to be scattered over several, superficially "open" handler functions. What looks open and flexible at first sight is in fact littered with obscure and scattered, non obvious additional presumptions.
This observation from coding practice gets us to the conclusion, that we do not really want to support the full notion of data and type introspection. We //do want// some kind of passive matching on structure, where the receiver explicitly has to supply structural presuppositions. In a fully functional language with a correspondingly rich type system, a partial function (pattern match) would be the solution of choice. Under the given circumstances, we're able to emulate this pattern based on our variant visitor -- which basically calls a different virtual function for each of the types possibly to be encountered "within" a ~GenNode.
This is a rather challenging attitude, and in practice we're bound to allow for a bit of leeway, leave some loopholes to at least "peek" into data about to be handled:
* based on the form of a given node's ID element, we allow to distinguish //attributes and children.//
* the {{{Record<GenNode>}}}, which is used to represent object-like entities, exposes a //semantic type filed// to be used at the local scope's discretion.
* we explicitly enable receiving or handling code to "peek" into that type field, thereby silently absorbing the case when the GenNode in question in fact does not hold a Record.

Each [[Timeline]] has an associated set of global [[pipes|Pipe]] (global busses), similar to the subgroups of a sound mixing desk.
In the typical standard configuration, there is (at least) a video master and a sound master pipe. Like any pipe, ingoing connections attach to the input side, attached effects form a chain, where the last node acts as exit node. The PipeID of such a global bus can be used to route media streams, allowing the global pipe to act as a summation bus bar.
&rarr; discussion and design rationale of [[global pipes|GlobalPipeDesign]]
!Properties and decisions
* each timeline carries its own set of global pipes, as each timeline is an top-level element on its own
* like all [[pipes|Pipe]] the global ones are kept separated per stream (proto)type
* any global pipe //not// connected to another OutputDesignation automatically creates a ModelPort
* global pipes //do not appear automagically just by sending output to them// -- they need to be set up explicitly
* the top-level (BusMO) of the global pipes isn't itself a pipe. Thus the top-level of the pipes forms a list (typically a video and sound master)
* below, a tree-like structure //may// be created, building upon the same scope based routing technique as used for the tracks / forks

//This page serves to shape and document the design of the global pipes//
Many aspects regarding the global pipes turned out while clarifying other parts of ~Proc-Layer's design. For some time it wasn't even clear if we'd need global pipes -- common video editing applications get on without. Mostly it was due to the usefulness of the layout found on sound mixing desks, and a vague notion to separate time-dependant from global parts, which finally led me to favouring such a global facility. This decision then helped in separating the concerns of timeline and sequence, making the //former// a collection of non-temporal entities, while the latter concentrates on time varying aspects.
!Design problem with global Pipes
actually building up the implementation of global pipes seems to pose a rather subtle design problem: it is difficult to determine how to do it //right.//
To start with, we need the ability to attach effects to global pipes. There is already an obvious way how to attach effects to clips (=local pipes), and thus it's desirable to handle it the same way for global pipes. At least there should be a really good reason //not//&nbsp; to do it the same way. Thus, we're going to attach these effects by placement into the scope of another placed MObject. And, moreover, this other object should be part of the HighLevelModel's tree, to allow using the PlacementIndex as implementation. So this reasoning brings us to re-using or postulating some kind of object, while lacking a point or reference //outside this design considerations//&nbsp; to justify the existence of the corresponding class or shaping its properties on itself. Which means &mdash; from a design view angle &mdash; we're entering slippery ground.
!!!~Model-A: dedicated object per pipe
Just for the sake of symmetry, for each global pipe we'd attach some suitable ~MObject as child of the BindingMO representing the timeline. If no further specific properties or functionality is required, we could use Track objects, which are generally used as containers within the model. Individual effects would then be attached as children, while output routing could be specified within the attaching placement, the same way as it's done with clips or tracks in general. As the pipe asset itself already stores a StreamType reference, all we'd need is some kind of link to the pipe asset or pipe-ID, and maybe façade functions within the binding to support the handling.
!!!~Model-B: attaching to the container
Acknowledging the missing justification, we could instead use //just something to attach// &mdash; and actually handle the real association elsewhere. The obvious "something" in this case would be the BindingMO, which already acts as implementation of the timeline (which is a façade asset). Thus, for this approach, the bus-level effects would be attached as direct children of the {{{Placement<BindingMO>}}}, just for the sake of beeing attached and stored within the session, with an additional convention for the actual ordering and association to a specific pipe. The Builder then would rather query the ~BindingMO to discover and build up the implementation of the global pipes in terms of the render nodes.
!!!Comparison
While there might still be some compromises or combined solutions &mdash; to support the decision, the following table detailes the handling in each case
|>| !~Model-B|!~Model-A |
|Association: | by plug in placement|by scope |
|Output: | entry in routing table|by plug in placement |
|Ordering: |>| stored in placement |
|Building: | sort children by plug+order<br/>query output from routing|build like a clip |
|Extras: | ? |tree-like bus arrangement,<br/> multi-stream bus |
So through this detailed comparison ''~Model-A looks favourable'': while the other model requires us to invent a good deal of the handling specifically for the global pipes, the former can be combined from patterns and solutions already used in other parts of the model, plus it allows some interesting extensions.
On a second thought, the fact that the [[Bus-MObject|BusMO]] is rather void of any specific meaning doesn't weight so much: As the Builder is based on the visitor pattern, the individual objects can be seen as //algebraic data types.// Besides, there is at least one little bit of specific functionality: a Bus object actually needs to //claim//&nbsp; to be the OutputDesignation, by referring to the same ~Pipe-ID used in other parts of the model to request output routing to this Bus. Without this match on both ends, an ~OutputDesignation may be mentioned at will, but no connection whatsoever will happen.
!{{red{WIP}}} Structure of the global pipes
;creating global pipes automatically?
:defining the global bus configuration is considered a crucial part of each project setup. Lumiera isn't meant to support fiddling around thoughtlessly. The user should be able to rely on crucial aspects of the global setup never being changed without notice.
;isn't wiring and routing going to be painful then?
:routing is scope based and we employ a hierarchical structure, so subgroups are routed automatically. Moreover, wiring is done based on best match regarding the stream type. We might consider feeding all non-connected output designations to the GUI after the build process, to allow short-cuts for creating further buses.
;why not making buses just part of the fork ("track tree")?
:anything within the scope of some fork ("track") has a temporal extension and may vary -- while it's the very nature of the global pipes to be static anchor points.
;why not having one grand unified root, including the outputs?
:you might consider that a matter of taste (or better common-sense). Things different in nature should not be forced into uniformity
;should global pipes be arranged as list or tree?
:sound mixing desks use list style arrangement, and this has proven to be quite viable, when combined with the ability to //send over// output from one mixer stripe to the input of another, allowing to build arbitrary complex filter matrices. On the other hand, organising a mix in //subgroups// can be considered best practice. This leads to arranging the pipes //as wood:// by default and on top level as list, optionally expanding into a subtree with automatic rooting, augmented by the ability to route any output to any input (cycles being detected and flagged as error).

//some information how to achieve custom drawing with ~GTKmm...//
valuable reference documentation comes bundled with lib ~GTKmm, in the guide [["Programming with GTKmm"|https://developer.gnome.org/gtkmm-tutorial/stable/index.html.en]]
* the chapter detailing [[use of the Gtk::DrawingArea|https://developer.gnome.org/gtkmm-tutorial/stable/chapter-drawingarea.html.en]], including an introduction to [[Cairomm|https://www.cairographics.org/documentation/cairomm/reference/]]
* the chapter about [[constructing a custom widget|https://developer.gnome.org/gtkmm-tutorial/stable/sec-custom-widgets.html.en]]
Basically we have to handle the {{{signal_draw}}} events. Since we need to control the event processing, it is recommended to do this event handling by //overriding the {{{on_draw()}}} function,// not by connecting a slot directly to the signal. Two details are to be considered here: the //return value// controls if the event can be considered as fully handled. If we return {{{false}}}, enclosing (parent) widgets get also to handle this event. This is typically what we want in case of custom drawing. And, secondly, if we derive from any widget, it is a good idea to invoke the //parent implementation of {{{on_draw()}}} at the appropriate point.// This is especially relevant when our custom drawing involves the ''canvas widget'' [[Gtk::Layout|GtkLayoutWidget]], which has the ability to place several further widgets embedded onto the canvas area. Without invoking this parent event handler, those embedded widgets won't be shown.
Typically, when starting the draw operation, we retrieve our //allocation.// This is precisely the rectangle reserved for the current widget, //insofar it is visible.// Especially this means, when a larger canvas is partially shown with the help of scrollbars, the allocation is the actually visible rectangle, not the virtual extension of the canvas. Each scrollbar is associated with a {{{Gtk::Adjustment}}}, which is basically a bracketed value with preconfigured step increments. The //value// of the adjustment corresponds to the //coordinates of the viewport// within the larger area of the canvas. Since coordinates in Gtk and Cairo are oriented towards the right and downwards, the value properties of both adjustments (horizontal and vertical) together give us the coordinates of the upper left corner of the viewport. The maximum value possible within such an adjustment is such as to fulfil {{{max(value) + viewport-size == canvas-size}}}. By printing values from the {{{on_draw()}}} callback, it can be verified that Gtk indeed handles it precisely that way.
Thus, if we want to use absolute canvas coordinates for our drawing, we need to adjust the cairo context prior to any drawing operations: we translate it in the opposite direction of the values retrieved from the scrollbar {{{Gtk::Adjustment}}}s. This causes the //user coordinates// to coincede with the absolute canvas coordinates.

//This is the canvas widget of ~GTK-3//
It allows not only custom drawing, but also to embed other widgets at defined coordinates.
!to be investigated
In order to build a sensible plan for our timeline structure, we need to investigate and clarify some fundamental properties of the GtkLayoutWidget
* how are overlapping widgets handled?
* how is resizing of widgets handled, esp. when they need to grow due to content changes?
* how does the event dispatching deal with partially covered widgets?
* how can embedded widgets be integrated into a tabbing / focus order?
* how is custom drawing and widget drawing interwoven?
!test setup
we need a test setup for this investigation.
* easy to launch
* don't waste much time, it is disposable
* realistic: shall reflect the situation in our actual UI
As starting point, {{red{in winter 2016/17}}} the old (broken) timeline panel was moved aside and a new panel was attached for GTK experiments. These basically confirmed the envisioned approach; it is possible to place widgets freely onto the canvas; they are drawn in insert order, which allows for overlapped widgets (and mouse events are dispatched properly, as you'd expect). Moreover, it is also possible to draw directly onto the canvas, by overriding the {{{on_draw()}}}-function. However, some (rather trivial) adjustments need to be done to get a virtual canvas, which moves along with the placed widgets. That is, GtkLayoutWidget handles scrolling of embedded widgets automatically, but you need to adjust the Cairo drawing context manually to move along. The aforementioned experimental code shows how.
After that initial experiments, my focus shifted to the still unsatisfactory top-level UI structure, and I am working towards an initial integration with Proc-Layer since then.

//The representation of a [[media clip|Clip]] for manipulation by the user within the UI.//
Within Lumiera, a clip is conceived as a //chunk of media,// which can be handled in compound. Clip as such is an abstract concept, which is treated with minimal assumptions...
* we know that a clip has //media content,// which need not be uniform and can be inherently structured (e.g. several media, several channels)
* like anything within a timeline, a clip has a temporal extension (but not necessarily finite; it can //not be zero,// but it can be infinite)
* by virtue of [[Placement]], a clip acquires a time position.
Due to this unspecific nature, a clip might take on various //appearances// within the UI.
!clip appearances
To start with, a clip can be rendered in ''abridged form'', which means that the content is stylised and the temporal extension does not matter. In this form, the clip is reduced to an icon, an expand widget and a ID label. This is the standard representation encountered within the //media bins.// The intent of this representation is to save on screen area, especially to minimise vertical extension. As a derivative of this layout style, a clip may be shown in abridged form, but with proper representation of the temporal extension; to this end, the enclosing box is extended horizontally as needed, while the compound of icon, control and label is aligned such as to remain in sight.
The next step in a series of progressively more detailed clip representations is the ''compact form'', which still focuses on handling the clip as an unit, while at least indicating some of the inherent structuring. Essentially, the clip here is represented as a //strip of rendered preview content,// decorated with some overlays. One of these overlays is the //ID pane,// which resembles the arrangement known from the abridged form: The icon here is always the [[Placement]] icon, followed by the expand widget and the ID label. Again, this pane is aligned such as to remain in sight. Then, there is a pair of overlays, termed the //boundary panes,// which indicate the begin and the end of the clip respectively. Graphically, these overlays should be rendered in a more subtle way, just enough to be recognisable. The boundary panes are the attachment areas for //trimming gestures,// as opposed to moving and dragging the whole clip or shuffle editing of the content. Moreover, these boundary panes compensate for the alignment of the ID pane, which mostly keeps the latter in sight. As this might counterfeit the visual perception of scrolling, the boundary panes serve to give a clear visual clue when reaching the boundary of an extended clip. Optionally, another overlay is rendered at the upper side of the clip's area, to indicate attached effect(s). It is quite possible for these effect decorations not to cover the whole temporal span of the clip.
A yet more detailed display of the clip's internals is exposed in the ''expanded form.'' Here, the clip is displayed as a window pane holding nested clip displays, which in turn might again be abridged, compact or ({{red{maybe 11/16}}}) even expanded. This enclosing clip window pane should be rendered semi transparent, just to indicate the enclosing whole. The individual clip displays embedded therein serve to represent individual media parts or channels, or individual attached effects. Due to the recursive nature of Lumiera's HighLevelModel, each of these parts exposes essentially the same controls, allowing to control the respective aspects of the part in question. We may even consider ({{red{unclear as of 11/16}}}) to allow accessing the parts of a VirtualClip, this way turning the enclosing clip into a lightweight container ({{red{11/2016 give proper credit for this concept! Who proposed this initially in 2008? was this Thosten Wilms?}}}
Finally, there can be a situation where it is just not possible to render any of the aforementioned display styles properly, due to size constraints. Especially, this happens when zooming out such as to show a whole sequence or even timeline in overview. We need to come up with a scheme of ''graceful display degradation'' to deal with this situation -- just naively attempting to render any form might easily send our UI thread into a minute long blocking render state, for no good reason. Instead, in such cases display should fall back to a ''degraded form''
* showing just a placeholder rectangle, when the clip (or any other media element) will cover a temporal span relating to at least 1 pixel width (configurable trigger condition)
* even further collapsing several entities into a strike of elements,to indicate at least that some content is present in this part of the timeline.
Together this shows we have to decide on a ''display strategy'' //before we even consider to add// a specific widget to the enclosing GTK container....
!!!strategy decision how to show a clip
Depending on specific circumstances within the presentation, we get a fundamentally different code path, well beyond just a variation of display parametrisation. Even more, this decision happens above the level of individual widgets, and this decision might be changed dynamically, independently of adding or removing individual widgets. We may
* need to build a custom drawing container for a single widget, to produce the compact and expanded display; theoretically this display can be //reduced// to the abridged form.
* alternatively we may omit the overhead of a custom drawing container, if we know that we'll most likely stick to the abridged form, or reduce display to a placeholder block
* but in the extreme case, we do not even display anything for a given widget -- rather we need to create a placeholder to stand-in for //N clips.//
This is unfortunate, since it defeats a self-contained control structure, where each widget holds its own model and manages its own state. Rather, we have to distinguish between child elements as belonging to the model vs child widgets as far as the view is concerned. A given scope in the fork could have more clip child elements than child widgets for display. Which means, we have to separate between the view aspect and the modelling aspect. And this split happens already on a level global to the timeline.
This creates a tension related to the kind of architecture we want to build with the help of the UI-Bus. The intention is to split and separate between the application global concerns, like issuing an editing command or retrieving changed model contents, and the //local UI mechanics.// This separation can only work, if -- with respect to the global concerns -- the UI-Bus embodies (actually mediates) the roles of model //and// controler, while the UI widgets take on the role of a "mere view". While, for the local concerns, the same widgets are to act self-contained. But now, already the existence of aforementioned widgets becomes dependent from another, obviously local concern, which unfortunately cross-cuts the 1:1 relation between model entities and view counterpart.
Starting from the architecture as predetermined by the UI-Bus, it is clear that we have to add a UI-Element for every tangible child entity represented in the (abstracted) model. And the parent entity receiving a diff and adding such a child element is also in charge of that element, controlling its lifecycle. Yet this child element is not necessarily bound to be a widget. This is our chance here: what "adding" means is completely within the scope of the parent element and basically an implementation detail. So the parent may create and own a tracker or controller object to stand for the clip, and the latter may create and adjust the actual widget in negotiation with a display strategy. So effectively we're introducing a //mediator,// which represents the clip (as far as the modelling is concerned) and which at the same time communicates with a global display manager. We call this mediating entity a ClipPresenter
!!!how to carry out the clip appearances
Initially, one could think that we'd need to build several widgets to realise the wide variety of clip appearances. But in fact it turns out that we're able to reshape a single base widget to encompass all the necessary presentation styles. This base widget is a simple, one-element container, with {{{Gtk::Frame}}} being the most obvious pick. This gives us already a rectangular covered space, and the ability to add a label widget, optionally with controlled alignment of the label. All the more elaborate presentation styles can be achieved by adding a canvas widget into this frame and then placing additional stuff on top of that. The only tricky part arises in overview display, when just some clip rectangle can stand-in for a whole series of clips, which themselves remain hidden as UI elements.
A prime consideration regarding this whole clip presentation strategy is the performance concern. It is quite common for movie edits to encompass several hundred individual clips. Combined with several tracks and an elaborate audio edit, it may well happen that we end up with thousands of individual UI objects. If treated naively, this load might seriously degrade the responsiveness of the interface. Thus we need to care for the relevant infrastructure to enable optimisation of the display. For that reason, the memory footprint of the ClipPresenter and the basic widget has to be kept as small as possible. And moreover, since we do our own layout management in the timeline display, in theory it is possible //only to add// those widgets to the enclosing GTK container, which are //actually about to become visible.// (if we follow this approach, a problem yet to be solved is how to remove widgets falling out of sight, since removing N widgets easily turns into a quadratic operation).
!clip content rendering
In a typical editing application, the user can expect to get some visual clue regarding the media content of a clip. For example, sound clips can be visualised as waveform, while movie clips might feature a sequence of images taken from the video. Our intention is to ''use our own rendering engine'' to produce these thumbnails. In fact, our engine is perfectly suited for this task: it has precisely the necessary media decoding and rendering abilities, plus it offers an elaborate system of priorities and deadlines, allowing to throttle the load produced by thumbnail generation. In addition to all those qualities, our engine is planned to be complemented by an "intelligent" frame cache, which, given proper parametrisation, ensures the frequently requested thumbnails will be available for quick display. For this approach to work, we need to provide some infrastructure
* we need to configure and maintain a //preview rendering strategy.//
* the request for rendering has to be cast "just somewhere" as message, possibly via the UI-Bus
* actually rendered content will likewise arrive asynchronously as message via UI-Bus.
* we still need to work out how buffer management for this task will be handled; it should be a derivative of typical buffer management for display rendering.
* the clip widget needs to provide a simple placeholder drawing to mark the respective space in the interface, until the actual preview arrives.
To start with, mostly this means to avoid a naive approach, like having code in the UI to pull in some graphics from media files. We certainly won't just render every media channel blindly. Rather, we acknowledge that we'll have a //strategy,// depending on the media content and some further parameters of the clip. This might well just be a single ''pivot image'' chosen explicitly by the editor to represent a given take. Seemingly, the proper place to hose that display strategy is ''within the session model'', not within the UI. And the actual implementation of content preview rendering will largely be postponed until we get our rendering engine into a roughly working state.

//how to access proc layer commands from the UI and to talk to the command framework//
!Command access DSL
{{red{WIP 4/2017}}} first rough draft of a framework for dealing with proc layer commands from within UI code
{{{
Symbol ADD_CLIP = CmdAccess::to (cmd::scope_addClip, INTO_FORK);
prepareCommand (cmdAccess(ADD_CLIP).bind (scope(HERE), element(RECENT)))
issueCommand (cmdAccess(ADD_CLIP).execute());
}}}
* access to commands is prepared by defining an instanceID as a local Symbol
** here »instance« means a command instance, and the ID is formed by decorating the ID of the command definition
** such an instance is visible and accessible from within a context, which is indicated by the second part of the ID
** we use some standardised context designators
**;~INTO_PROJECT
**:operations working on the project, the project setup or configuration
**;~INTO_SCOPE
**:operations working on or adding something into whatever container is currently in focus
**;~INTO_FORK
**:operations especially working on tracks or media bins
**;~INTO_BIN
**:operations limited to asset management
** beyond that, individual UI elements are always free to add their own, local context symbols<br/>the result of such is a command instance known and accessibly only to the element defining this context
** we rely on a naming scheme for the command definitions.<br/>the expectation is for commands to be rather generic and adapt to the usage scope
* the task of supplying or binding the command arguments can be automated to some degree
** we use several standardised ''roles'' for the command arguments,
** so instead of explicit parameters, the binding can indicate //argument resolver expressions//
**;scope(HERE)
**:will be resolved to the scope right at or encompassing the [[Spot]] (&rarr; InteractionControl)
**;element(RECENT)
**:the most recently touched object which is applicable for this argument
**:* in the example this might be the last clip or media selected in the asset section
**:* but another clip, which was copied to clipboard more recently will take precedence
**:* but if the last action was a drag-n-drop, the binding will try to use the dropped element first
* overall, these invocations generate messages for use on the UI-Bus. It is up to the client to issue them through some {{{BusTerm}}}<br/>in the example given, the code obviously is within the scope of a {{{model::Tangible}}}, allowing to use the API {{{Tangible::prepareCommand()}}} and {{{Tangible::issueCommand()}}}
!Design Critique
The above design for command access looked as a good idea -- at first sight. Yet on further investigation, it seems to miss the most common use case. Basically, we can identify three styles of usage for command bindings
;fire and forget
:trigger some action, maybe with well determined additional arguments, and not in any further relation to the context.
:just do it and be done with it -- no framework whatever is required beyond the basic command ID
;widget-local
:some action is bound to incoming signals; everything of relevant is confined within that widget
:while the arguments are in this case picked up from widget fields or sub widgets, again no further framework is necessary
;context bound
:this is the //sole case// where all the above mentioned complexities might make sense.
:we should not dismiss this case, since in fact most known existing applications fall short in that very area
Another point of contention is the duality between a command access framework and the UI-Bus. It turns out this is rooted in a fundamental conflict. The command system and by extension also a command access framework represents the classic »command and control« style API (to use a term coined by Martin Fowler). This kind of interface shines when used on top of a shared data or object model. Which is exactly the kind of architecture we shun, on a global level. It is fine when //confined within some subsystem,// yet creates a corroding tendency towards high coupling, when used on a global scale. For this reason, we introduced a message driven connection, and for the same reason, we should refrain from using the command and control structure as a second channel, bypassing the bus. //This is entirely an architectural decision// -- on the level of tracing actual calls, any message based connection looks overengineered, when compared to "just invoke the f**cking function"
The ''conclusions'' drawn from this critique is to forego using the InvocationTrail, in favour of a plain command ID, and to simplify and automate the command instance management (&rarr; GuiCommandCycle)
!reworked DSL draft
{{red{WIP 4/2017 unimplemented for the time being}}}
{{{
CmdContext::of (cmd::scope_addClip, INTO_FORK)
.activate ([&](yes) { addClip.enabled (yes); });
...
this->invoke (cmd::scope_addClip, scope(HERE), element(RECENT))
}}}
* define a callback when the command becomes executable
* {{red{TODO 4/2017}}} not clear how the InteractionState is able to figure out this is the case.<br/>probably need either binding rules or need to pre-bind the arguments
* when actual invocation shall be triggered, use the //argument resolvers// to fill in the arguments<br/>these yield intermediary objects, which can be converted to LUID

The topic of command binding addresses the way to access, parametrise and issue [[»Proc-Layer Commands«|CommandHandling]] from within the UI structures.
Basically, commands are addressed by-name -- yet the fact that there is a huge number of commands, which moreover need to be provided with actual arguments, which are to be picked up from some kind of //current context// -- this all together turns this seemingly simply function invocation into a challenging task.
The organisation of the Lumiera UI calls for a separation between immediate low-level UI element reactions, and anything related to the user's actions when working with the elements in the [[Session]] or project. The immediate low-level UI mechanics is implemented directly within the widget code, whereas to //"work on elements in the session",// we'd need a collaboration spanning UI-Layer and Proc-Layer. Reactions within the UI mechanics (like e.g. dragging a clip) need to be interconnected and translated into "sentences of operation", which can be sent in the form of a fully parametrised command instance towards the ProcDispatcher
* questions of architecture related to command binding &rarr; GuiCommandBindingConcept
* study of pivotal action invocation situations &rarr; CommandInvocationAnalysis
* actual design of command invocation in the UI &rarr; GuiCommandCycle
* the way to set up the actual command definitions &rarr; CommandSetup
* access and use commands from UI code &rarr; GuiCommandAccess

The question //how to connect the notion of an ''interface action'' to the notion of a ''command'' issued towards the [[session model|HighLevelModel]].//
* actual design of command invocation in the UI &rarr; GuiCommandCycle
* study of pivotal action invocation situations &rarr; CommandInvocationAnalysis
!prerequisites for issuing a command
Within the Lumiera architecture, with the very distinct separation between [[Session]] and interface view, several steps have to be met before we're able to operate on the model.
* we need a pre-written script, which directly works on the entities reachable through the session interface &rarr; [[Command handling in Proc-Layer|CommandHandling]]
* we need to complement this script with a state capturing script and a script to undo the given action
* we need to combine these fixed snippets into a //command prototype.//
* we need to care for the supply of parameters
** indirectly this defines and limits how this command can be issued
** which in fact embeds the raw command into a context or a gesture of invocation
** and only through this explication the command-as-seen-from-session translates into something tangible within the UI
* next we have to consider conditions and circumstances. Not every command can be invoked any given time
** the focus and current selection is relevant
** the user interaction might supply context by pointing at something
** the proximity of tangible interface elements might be sufficient to figure out missing parts
** at times it might also be necessary to intersperse the invocation of a detail parameter dialogue prior to command execution
* and finally, after considering all these concerns, it is possible to wire a connection into the actual invocation point in UI
This observation might be surprising; even while a given command is well defined, we can not just invoke it right away. The prevalence of all these intermediary complexities is what opens the necessity and the room for InteractionControl, which is a concern specific to human-computer interfaces. Faced with such a necessity, there are two fundamentally different approaches.
!!!Binding close to the widget
This approach concentrates knowledge about the operation at that location, where it is conceived "to happen" -- that is, close to the concrete UI widget.
So the actual widget type implies knowledge about the contents and the meaning of the command scripts. At the point when the widget is actually triggered, it starts to collect the necessary parameters and to this end needs to acquire connections to other facilities within the UI. In fact, even //before// anything can be triggered, the widget has to administer the activation state and show some controls as enabled or disabled, and it needs to observe ongoing state changes to be able to do so.
The consequence of this approach is that the relations and whereabouts of entities involved into this decision tend to be explicated right into the widget code. Any overarching concerns end up being scattered over various implementation sites, need to be dealt with by convention, or rely on all invocation sites to use some helper facilities voluntarily.
!!!Abstracted binding definitions
This contrastive approach attempts to keep knowledge and definition clustered in a way according to the commands and actions to be performed -- even at the price of some abstractions and indirections. There is no natural and obvious place where to collect those information, and thus we need to create such a location deliberately. This new entity or location to be conceived will serve as a link between user interface and session elements, and it tends to rely on definitions from both realms.
* in addition to the command script, here we build a parameter accessor, which is some kind of functor or closure.
* we need to introduce a new abstraction, termed InteractionState. This is deliberately not a single entity, rather some distinct facility in charge for one specific kind of interaction, like gestures being formed by mouse, touch or pen input.
* from the command definition site, we need to send a combination of //rules// and parameter accessors, which together define an invocation path for one specific flavour of a command
* the InteractionState, driven by the state changes he observes, will evaluate those rules and determine the feasibility of specific command invocation paths
* he sends the //enablement of a command invocation trail// as a preconfigured binding to the actual //trigger sites,// which in turn allows them to react to local user interactions properly
* if finally some button is hit, the local event binding can issue the command right away, by accessing just any UI-Bus terminal at reach within that context
''Lumera decides to take the latter approach'' -- resulting in a separation between immediate low-level UI element reactions, and anything of relevance to the workings of the application as a whole. The widget code embodies the low-level UI element reactions and as such becomes more or less meaningless beyond local concerns of layout and presentation. If you want to find out about the //behaviour of the UI,// you need to know where to look, and you need to know how to read and understand those enablement rules. Another consequence is the build-up of dedicated yet rather abstract state tracking facilities, hooking like an octopus into various widgets and controllers, which might work counter to the intentions behind the design of common UI toolkit sets.
&rarr; GuiCommandCycle
&rarr; CommandSetup

//the process of issuing a session command from the UI//
Within the Lumiera UI, we distinguish between core concerns and the //local mechanics of the UI.// The latter is addressed in the usual way, based on a variation of the [[MVC-Pattern|http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller]]. The UI toolkit set, here the GTK, affords ample ways to express actions and reactions within this framework, where widgets in the presentation view are wired with the corresponding controllers vice versa (GTK terms these connections as //"signals"//, we rely on {{{libSigC++}}} for implementation).
A naive approach would extend these mature mechanisms to also cover the actual functionality of the application. This compelling solution allows quickly to get "something tangible" up and running, yet -- on the long run -- inevitably leads to core concerns being tangled into the presentation layer, which in turn becomes hard to maintain and loaded with "code behind". Since we are here "for the long run", we immediately draw the distinction between UI mechanics and core concerns. The latter are, by decree and axiom, required to perform without even an UI layer running. This decision gives rise to the challenge how to form and integrate the invocation of ''core commands'' into the presentation layer.
In a nutshell, we understand each such core command as a ''sentence'', with a //subject//, a //predication//, which is the command script in ~Proc-Layer and can be represented by an ID, and possibly additional arguments. And the key point to note is: //such an action sentence needs to be formed, before it can be issued.//
[>img[Command usage in UI|uml/Command-ui-usage.png]]
!use case analysis
To gain some understanding of the topic, we pose the question "who has to deal with core commands"?
* the developer of ~Proc-Layer, obviously. The result of that development process is a set of [[command definitions|CommandHandling]], which get installed during start-up of the SessionSubsystem
* global menu actions (and keybindings) want to issue a specific command, but possibly also need context information
* a widget, button or context-menu binding typically want to trigger a command on some [[tangible element|UI-Element]] (widget or controller), but also needs to prepare the arguments prior to invocation
* some [[interaction state manager|InteractionState]] observes contextual change and needs to mark possible consequences for invoking a given command
from these use cases, we can derive the //crucial activities for command handling...//
;instance management
:our commands are prototypes; we need to manage instances for binding concrete arguments
:there is a delay between issuing a command in the UI and dispatching it into the session (and dispatch happens in another thread)
:moreover, there is an "air-gap" when passing a command invocation via ~UI-Bus and Interface System
;forming and enrichment of invocation state
:interactions might happen in the form of ''gestures''
:consequently, interaction state is picked up from context, during an extended time span prior to the invocation
;access to the right ~InteractionState
:a widget just wants to invoke a command, yet it needs the help of "some" InteractionState for
:* creating the command instance, so arguments can be bound
:* fill in missing values for the arguments, depending on context
!command invocation protocol
* at start-up, command definitions are created in Proc, hard wired
* ~UI-Elements know the basic ~Command-IDs relevant to their functionality. These are &rarr; [[defined in some central header|CommandSetup]]
* command usage may happen either as simple direct invocation, or as part of an elaborate argument forming process within a context.<br/>thus we have to distinguish two usage patterns
*;fire and forget
*:a known command is triggered with likewise known arguments
*:* just the global ~Command-ID (ID of the prototype) is sent over the UI-Bus, together with the arguments
*:* the {{{CmdInstanceManager}}} in Proc creates an anonymous clone copy instance from the prototype
*:* arguments are bound and the instance is handed over into the ProcDispatcher, without any further registration
*;context bound
*:invocation of a command is formed within a context, typically through a //interaction gesture.//
*:most, if not all arguments of the command are picked up from the context, based on the current [[Spot]]
*:* on setup of such an invocation context, the responsible part in the UI queries the {{{CmdContext}}} for an InteractionState
*:* the latter in turn retrieves a new command instance ID from the {{{CmdInstanceManager}}} in Proc
*:* and the latter keeps a smart-ptr corresponding to this instance in its internal registration table
*:* within the UI, the InteractionState instance responsible for this context tracks state changes and keeps track of all command instances bound to this context
*:* ~UI-Elements use the services of {{{CmdContext}}} to act as observer of state changes &rarr; GuiCommandAccess
*:* when a command is completely parametrised, it can be invoked. The managing {{{InteractionState}}} knows about this
*:* on invocation, the ID of the instance and the fully resolved arguments are sent via UI-Bus to the {{{CmdInstanceManager}}}
*:* which in turn removes the instance handle from its registration table and hands it over into the ProcDispatcher
* in any case, only the {{{CmdInstanceManager}}} need to know about this actual command instance; there is no global registration
[<img[Access to Session Commands from UI|uml/Command-ui-access.png]]
An immediate consequence is that command instances may be formed //per instance// of InteractionState. Each distinct kind of control system has its own instances, which are kept around, until they are ready for invocation. Each invocation "burns" an instance -- on next access, a new instance ID will be allocated, and the next command invocation cycle starts...
Command instances are like prototypes -- thus each additional level of differentiation will create a clone copy and decorate the basic command ID. This is necessary, since the same command may be used and parametrised differently at various places in the UI. If necessary, the {{{CmdInstanceManager}}} internally maintains and tracks a prepared anonymous command instance within a local registration table. The //smart-handle//-nature of command instance is enough to keep concurrently existing instances apart; instances might be around for an extended period, because commands are enqueued with the ProcDispatcher.
''command definition'':
&rarr; Command scripts are defined in translation units in {{{proc/cmd}}}
&rarr; They reside in the corresponding namespace, which is typically aliased as {{{cmd}}}
&rarr; definitions and usage include the common header {{{proc/cmd.hpp}}}
see the description in &rarr; CommandSetup

//A view within the UI, featuring some component of relevance to »the model«.//
While any UI is comprised of numerous widgets acting as //view of something,// only some of those views play the prominent role to act as //building block component// of the user interface.
Such UI component views exhibit some substantial traits
* they conform to a built-in fixed list of view types, each of which is unique and dedicated to a very specific purpose: //''Timeline'', ''Viewer'', (Asset)''Bin'', ''Infobox'', ''Playcontrol'',...//
* each component view has a distinguishable identity and is connected to and addressable through the UI-Bus
* it can be hosted only at a dedicated location within one or several specific [[docking panels|GuiDockingPanel]].
* multiplicity (only one, one per window, many) depends on the type of view and needs to be managed.
* such a view is not just //created// -- rather it needs to be //allocated//
!Allocation of UI component views
Here, //Allocation// means
* to constitute the desired element's identity
* to consider multiplicity and possibly retrieve an existing instance
* to determine the hosting location
* possibly to instantiate and register a new instance
* and finally to configure that instance for the desired role
The classical example to verify this definition is the //allocation of viewers:// when starting playback of a new media item, we "need a viewer" to show it. But we can not just create yet another viewer window -- rather we're bound to allocate one of the possible "viewer slots". In fact this is a configurable property of the UI layout employed; sometimes some people need the limitation to one single viewer entity (which might even be external, routed to a beamer or monitor), while other ones request the classical editor layout with two viewer windows side by side, while yet different working styles might exploit a limited set of viewers allocated in stack- or round-robin style.
!View access
The global access point to component views is the ViewLocator within InteractionDirector, which exposes a generic access- and management API to
* get (possibly create) some view of given type
* get (possibly create) a view with specific identity
* destroy a specific view
For all these direct access operations, elements are designated by a private name-ID, which is actually more like a type-~IDs, and just serves to distinguish the element from its siblings. The same ~IDs are used for the components in [[UI coordinate specifications|UICoord]]; both usages are closely interconnected, because view access is accomplished by forming an UI coordinate path to the element, which is then in turn used to navigate the internal UI widget structure to reach out for the actual implementation element.
While these aforementioned access operations expose a strictly typed direct reference to the respective view component and thus allow to //manage them like child objects,// in many cases we are more interested in UI elements representing tangible elements from the session. In those cases, it is sufficient to address the desired component view just via the UI-Bus. This is possible since component ~IDs of such globally relevant elements are formed systematically and thus always predictable: it is the same ID as used within Proc-Layer, which basically is an {{{EntryID<TYPE>}}}, where {{{TYPE}}} denotes the corresponding model type in the [[Session model|HighLevelModel]].
!!!Configuration of view allocation
Since view allocation offers a choice amongst several complex patterns of behaviour, it seems adequate to offer at least some central configuration site with a DSL for readability. That being said -- it is conceivable that we'll have to open this topic altogether for general configuration by the user. For this reason, the configuration site and DSL are designed in a way to foster further evolution of possibilites...
* at definition site {{{id-scheme.hpp}}}, explicit specialisations are given for the relevant types of component view
* each of those general //view configurations//
** defines the multiplicity allowed for this kind of view
** defines how to locate this view
* and that //location definition// is given as a list of //alternatives in order of precedence.// That is, the system tries each pattern of location and uses the first one applicable
!!Standard examples
;Timeline
:add to group of timelines within the timelinePanel
{{{
alloc = unlimited
locate = perspective(edit).panel(timeline)
or panel(timeline)
or currentWindow().panel(timeline).create()
}}}
;Viewer
:here multiple alternatives are conceivable
:* allow only a single view instance in the whole application
{{{
alloc = onlyOne
locate = external(beamer)
or view(viewer)
or perspective(mediaView).panel(viewer)
or panel(viewer)
or firstWindow().panel(viewer).view(viewer).create()
}}}
:* allow two viewer panels (the standard layout of editing applications)
{{{
alloc = limitPerWindow(2)
locate = perspective(edit).panel(viewer)
or currentWindow().panel(viewer)
or panel(viewer)
or currentWindow().panel(viewer).create()
}}}
;(Asset)Bin
:within the dedicated asset panel, add to the appropriate group for the kind of asset
{{{
alloc = unlimited
locate = currentWindow().perspective(edit).tab(assetType())
or perspective(asset).view(asset)
or tab(assetType())
or view(asset).tab(assetType()).create()
or firstWindow().panel(asset).view(asset).create()
}}}
;~Error-Log
:use the current {{{InfoBoxPanel}}} if such exists, else fall back to using a single view on the primary window
{{{
alloc = limitPerWindow(1)
locate = currentWindow().panel(infobox)
or view(error)
or panel(infobox)
or firstWindow().panel(infobox).view(error).create()
}}}
;Playcontrol
://not determined yet what we'll need here....//
!!DSL structure
* the DSL assigns tokens to some expected Specs
* the tokens themselves are provided as constants
* in fact those tokens are functors
* and are opaquely bound into {{{ViewLocator}}}'s implementation
!!!Signatures
;locate
:is evaluated first and determines the UICoord of the target view
:is a disjunction of alternative specs, which are resolved to use the first applicable one
;alloc
:the //Allocator// is run on those coordinates and actually //allocates// the view
:returns coordinates of an (now) existing view
;allocator specs
:these are //builder functions// to yield a specifically parametrised allocator
:typical example is an allocator to create only a limited number of view instances per window
:the actual builder tokens are synthesised by partial function application on the given lambda
!!!Spec Tokens
;locate
:firstWindow
: currentWindow
: perspective(id)
: panel(id)
: assetType() {{red{WIP 2/18 not clear if necessary}}}
: create()
;alloc
: unlimited
: onlyOne
: limitPerWindow(cnt)
!!!Semantics of location
The given UICoord specs are matched one by one, using the first one applicable. The location indicated by this process describes the parent or scope where the desired view can be found or shall be created. The matching itself is based on the matching of UI coordinates against the existing UI topology, but enriched with some contextual information. By default, a given coordinate spec //is required to exist// -- which implies the semantics of //total coverage,// as defined by the coordinate resolver. When to the contrary the predicate {{{create()}}} is used, we only require that the given path //can be formed unambiguously// within the existing window topology -- the semantics of which is, again, defined by the coordinate resolver and basically implies that the path can be anchored and any missing parts can be interpolated without ambiguity, while possibly some extraneous suffix of components remains to be created by the instantiation process.
The coordinate specs as written within the DSL are slightly abridged, as the final element, the actual component to be created, can be omitted and will be supplied automatically. This is possible insofar the ID of the queried element is actually more like a type specifier, and thus drawn from a finite and well known collection of possible elements (Timeline, Asset Bin, Error Log, Media Viewer, etc.). The elements to be created must invoke existing code and must be able to interface with their actual environment after all.
However, the semantics of UI coordinate resolution and matching are applied against the coordinate specs //as given in the DSL.// Since, by default, what is given is required to exist, it makes quite a difference as to what kind of element is used as terminal element of a given spec. Since our UI coordinate system establishes a distinct depth for each kind of element, we can and even must reject any spec which does not yield a definite location for at least the parent element; we do not interpolate or even invent arbitrary elements, we only ever match against existing ones. Thus any DSL definition must encompass a sane default, a final alternative clause which always succeeds -- and the DSL is considered broken if it doesn't.
!!!!The problem with {{{assetType()}}}
There is a very special use case, where we'd might want to express some fine points regarding placement of a new sub-tab within the view requested. This need arises when the system requires to show some sub-element through a dedicated view. Such might be the case for a special kind of asset, or when a virtual clip or media is to be opened from a context //other than editing the timeline.// In such a situation, a preferable action would be to use or re-use an existing tab of a view of similar kind. Only if this fails, we'd want a new view to be created at a generic location (e.g. a window dedicated to asset management), and only as fall-back we'd want a completely new asset section to show up within the current window.
Unfortunately such surpasses the ability of the solution mechanism backing this location DSL, which basically relies on pattern matching without further expression evaluation or unification. To express the aforementioned special treatment, we'd need a placeholder within the rules, indicated above as {{{assetType()}}}, which has to be replaced by the actual typeID used in the concrete location query. Since our system does not support unification, either the matching mechanism has to be manipulated based on the context, or the specific rule must be preprocessed for token replacement.
It is not clear yet {{red{as of 2/2018}}}, if those additional complexities are justified...
&rarr; Ticket #1130
!!!Semantics of allocation
While the process of probing and matching the location specification finally yields an explicit UICoord path to the desired element, it is up to the allocation step actually to decide on the action to be taken. Some allocation operations impose some kind of limit, and are thus free to ignore the given spec and rather return an existing element in place. In the end, the purpose of this whole matching and allocation process is to get hold of a suitable UI component without knowing its precise coordinates in the UI topology. And this is the very property to enable flexible mapping of the strictly hierarchical session structures onto the UI in a fluid way.
&rarr; [[low level component access|UILowLevelAccess]]

All communication between Proc-Layer and GUI has to be routed through the respective LayerSeparationInterfaces. Following a fundamental design decision within Lumiera, these interface are //intended to be language agnostic// &mdash; forcing them to stick to the least common denominator. Which creates the additional problem of how to create a smooth integration without forcing the architecture into functional decomposition style. To solve this problem, we rely on ''messaging'' rather than on a //business facade// -- our facade interfaces are rather narrow and limited to lifecycle management. In addition, the UI exposes a [[notification facade|GuiNotificationFacade]] for pushing back status information created as result of the edit operations, the build process and the render tasks.
!anatomy of the Proc/GUI interface
* the GuiFacade is used as a general lifecycle facade to start up the GUI and to set up the LayerSeparationInterfaces.<br/>It is implemented by a class //in core// and loads the Lumiera ~GTK-UI as a plug-in.
* once the UI is running, it exposes the GuiNotificationFacade, to allow pushing state and structure updates up into the user interface.
* in the opposite direction, for initiating actions from the UI, the SessionSubsystem opens the SessionCommandFacade, which can be considered //"the" public session interface.//
!principles of UI / Proc interaction
By all means, we want to avoid a common shared data structure as foundation for any interaction. For a prominent example, have a look at [[Blender|https://developer.blender.org]] to see where this leads; //such is not bad,// but it limits to a very specific kind of evolution. //We are aiming for less and for more.// Fuelled by our command and UNDO system, and our rules based [[Builder]] with its asynchronous responses, we came to rely on a messaging system, known as the [[UI-Bus]].
The consequence is that both sides, "the core" and "the UI" remain autonomous within their realm. For some concerns, namely //the core concerns,// that is editing, arranging, processing, the core is in charge and has absolute authority. On the other hand, when it comes to user interaction, especially the //mechanics and materiality of interaction,// the UI is the authority; it is free to decide about what is exposed and in which way. The collaboration between both sides is based on a ''common structural understanding'', which is never fully, totally formed in concrete data structures.
Rather, the core sends ''diff messages'' up to the UI, indicating how it sees this virtual structure to be changing. The UI reflects these changes into //its own understanding and representation,// that is here a structure of display widgets. When the user interacts with these structures of the presentation layer, ''command messages'' are generated, using the element ~IDs to designate the arguments of the intended operation. This again causes reaction and change in the core, which is reflected back in the form of further diff messages. (&rarr; GuiCommandCycle)

//install a listener into the session to cause sending of population diff messages.//
The Lumiera application is not structured as an UI with internal functionality dangling below. Rather, the architecture calls for several self-contained subsystems, where all of the actual "application content" is modelled within the [[Session]] subsystem. The UI implements the //mechanics of user interaction// -- yet as far as content is concerned, it plays a passive role. Within the UI-Layer, there is a hierarchy of presentation entities, to mirror the structure of the session contents. These entities are built step by step, in reception of //population diff messages// sent upwards from the session.
To establish this interaction pattern, a listener gets installed into the session. In fact, the UI just issues a specific command to indicate it is ready for receiving content; this command, when executed down within the ProcDispatcher, causes the setup of aforementioned listener. This listener first has to //catch up// with all content already existing within the session -- it still needs to be defined {{red{as of 7/2018}}} in which form this existing content can be translated into a diff to reflect it within the UI. Since the session itself is {{red{planned (as of 2018)}}} to be persisted as a sequence of building instructions (CQRS, event sourcing), it might be possible just to replay the appropriate defining messages from the global log. After that point, the UI is attached and synchronised with the session contents and receives any altering messages in the order they emerge.
!Trigger
It is clear that content population can commence only when the GTK event loop is already running and the application frame is visible and active. For starters, this sequence avoids all kinds of nasty race conditions. And, in addition, it ensures a reactive UI; if populating content takes some time, the user may watch this process through the visible clues given just by changing the window contents and layout in live state.
And since we are talking about a generic facility, the framework of content population has to be established in the GuiTopLevel. Now, the top-level in turn //starts the event loop// -- thus we need to //schedule// the trigger for content population. The existing mechanisms are not of much help here, since in our case we //really need a fully operative application// once the results start bubbling up from Proc-Layer. The {{{Gio::Application}}} offers an "activation signal" -- yet in fact this is only necessary due to the internals of {{{Gio::Application}}}, with all this ~D-Bus registration stuff. Just showing a GTK window widget in itself does not require a running event loop (although one does not make much sense without the other). The mentioned {{{signal_activation()}}} is emitted from {{{g_application_run()}}} (actually the invocation of {{{g_application_activate()}}} is burried within {{{g_application_real_local_command_line()}}}, which means, the activation happens //immediately before// entering the event loop. Which pretty much rules out this approach in our case, since Lumiera doesn't use a {{{Gtk::Application}}}, and moreover the signal would still induce the (small) possibility of a race between the actual opening of the GuiNotificationFacade and the start of content population from the [[Proc-Layer thread|SessionSubsystem]].
The general plan to trigger content population thus boils down to
* have the InteractionDirector inject the population trigger with the help of {{{Glib::signal_timeout()}}}
* which will activate with a slight delay (100ms) after UI start, and after primary drawing activities ceases
* the trigger itself issues a command towards the session
* execution of aforementioned command activates sending of population / diff messages
* on shutdown of the top-level, send the corresponding deactivation command

Inevitably, the UI of an advanced application like Lumiera needs some parts beyond the scope of what can be achieved by combining standard widgets. Typically, an UI toolkit (GTK is no exception here) offers some extension mechanism to integrate such more elaborate, application specific behaviour. Within Lumiera, the Timeline is probably the most prominent place where we need to come up with our own handling solution -- which also means to rely on such extension mechanisms and to integrate well with the more conventional parts of the UI. Since the core concept of typical UI toolkit sets is that of a //widget,// we end up with writing some kind of customised or completely custom defined widget.
!two fundamental models
So it is clear that we need to write a customised widget to embody the specific behaviour we need. Two distinct approaches are conceivable
;custom arrangement
:define or derive our own widget class to arrange existing widgets in a customised way
;custom drawing
:get the allocated screen area for our widget and perform all drawing and event updating explicitly in our own code
!!!perils of custom drawing
While the second approach intuitively seems to be adequate (and in fact, Cinelerra, Ardour and our own GUI choose this approach), we should note several problematic aspects
* once we "take over", we are entirely responsible for "our area" and GTK steps out of our way
* we have to replicate and code up any behaviour we want and know from the standard GUI
* chances are that we are handling some aspects different than the default, without even noticing there is a default
* chances are that we are lacking the manpower to cope with all interdependencies of concrete presentation situation, custom styling and event state
Our custom made UI elements impend to turn into a tremendous time sink (For reference, Paul Davis reported for Ardour 2.x that he spent 80% of the developer time not with audio processing, but rather with bashing the UI into shape), while non the less delivering just a crappy, home-brew and misaligned user experience which stands out as an alien element not playing well with the rest of the desktop.
{{red{Well}}} //at least we are aware of the danger.//
!!!is custom drawing necessary?
There are some issues though, which more or less force us into custom drawing
* for our task, we're always suffering from lack of screen real estate. This forces us into conceiving elaborate controls way beyond the capabilities of existing standard widgets
* and for the feedback, we also need to tap into very precise event handling, which is hard to achieve with the huge amount of variability found with standard widgets
* our timeline display has to comply to a temporal metric grid, which is incompatible with the leeway present in standard widgets for the purpose of styling and skinning the UI
* the sheer amount of individual elements we need to get to screen is way beyond anything in a conventional GUI -- the UI toolkit set can not be expected to handle such load smoothly
!Remedy: A Canvas Control
All of the above is a serious concern. There is no easy way out, since, for the beginning, we need to get hands on with the display -- to get any tangible elements to work against. Yet there exists a solution to combine the strengths of both approaches: a ''Canvas Widget'' is for one a regular widget, and can thus be integrated into the UI, while it allows to //place child widgets freely onto a managed area,// termed as "the canvas". These child widgets are wired and managed pretty much like any other widget, they participate in theming, styling and accessibility technologies. Moreover, the canvas area can be clipped and scrolled, so to allow for arrangements way beyond the limits of the actual screen.
* in the past, this functionality was pioneered by several extension libraries, for example by the [[GooCanvas|https://developer.gnome.org/goocanvas/stable/GooCanvas.html]] library for ~GTK-2
* meanwhile, ~GTK-3 features several special layout managers, one of which is the [[GtkLayout|https://developer.gnome.org/gtk3/stable/GtkLayout.html]] widget, which incorporates this concept of //widgets placed on a canvas.//
!Investigation: ~GtkLayout {{red{WIP 10/2016}}}
In order to build a sensible plan for our timeline structure, we need to investigate and clarify some fundamental properties of the GtkLayoutWidget
* how are overlapping widgets handled?
* how is resizing of widgets handled, esp. when they need to grow due to content changes?
* how does the event dispatching deal with partially covered widgets?
* how can embedded widgets be integrated into a tabbing / focus order?
* how is custom drawing and widget drawing interwoven?
!!!Plan of investigation
# place some simple widgets (Buttons)
# learn how to draw
# place a huge number of widgets, to scrutinise scrolling and performance
# place widgets overlapping
# bind signals to those widgets, to verify event dispatching
# bind some further signal(s) to the ~GtkLayout container
# hide and re-show a partially and a totally overlapped widget
# find a way to move a widget
# expand an existing widget (text change)
# build a custom "''clip''" widget
# retrofit all preceding tests to use this "''clip''" widget
!!!Observations
* children need to be made visible, otherwise they are added, but remain hidden
* when in sight, children receive events and are fully functional, even when placed out of the scrollable area.
* the coordinate of children is their upper left corner, but they may extend beyond that and even beyond the scrollable area
;layering
:children added later are always on top.
;scrolling and size
:the {{{Gtk::Layout}}} widget has a canvas size, which is defined implicitly, by placing child elements
:* the size of the scrollable area is independent of the actual size extension of the canvas
:* the scrollable area determines when scrollbars are visible and what area can be reached through them
:* but children can well be placed beyond; they are fully functional then, just covered and out of sight
:* enlarging the enclosing window and thus enlarging the layout canvas will uncover such stray children.
:* //problematic//
:** children may be so close to the boundary, that it is not possible to click on them
:** when children close to the boundary receive an onClick event, the scrollable area might jump back slightly..
:** the management of visible viewport within {{{Gtk::ScrolledWindow}}} is not correct (see [[ticket #1037|http://issues.lumiera.org/ticket/1037]])<br/>it does not account properly for the area covered by the scrollbars themselves
;moving child widgets
:works as expected
;expanding widgets
:works as expected
;delete child widgets
:is possible by the {{{Container::remove(Widget*)}}} function
:removed child widgets will also removed from display (hidden)
:but the widget object needs to be deleted manually, because detached widgets are no longer managed by GTK
;iteration over child widgets
://problematic//
:* the signal based {{{foreach}}} does not work -- there seems to be a problem with the slot's signature causing a wrong address to be passed in
:* the interface {{{Gtk::Container}}} exposes a {{{get_children}}} function, but this returns a //copy// of the vector with pointers to all child widgets
;about GtkCustomDrawing
:need to derive from {{{Gtk::Layout}}} and override the {{{on_draw(cairocontext)}}} function
:* layering is controlled by the order of the cairo calls, plus the point when the inherited {{{Gtk::Layout::on_draw()}}} is invoked
:** when invoked //before// our custom drawing, we draw on top of the embedded widgets
:** when invoked //after// our custom drawing, the embedded widgets stay on top
:* the {{{Gtk::Allocation}}} is precisely the visible screen area reserved for the widget.<br/>It is //not// the extension of the virtual canvas.
:* ...consequently, if our drawing shall be stitched to the canvas, we need to care for translation and for clipping ourselves. &rarr; see [[here|GtkCustomDrawing]]
;determine canvas extension
:when the extension of the (virtual) canvas area depends on position and size of child widgets, //we need to calculate this extension manually.//
:* beware: the allocation for child widgets is not setup immediately, when adding or moving children
:* rather, we need to wait until in the {{{on_draw()}}} callback for the {{{Gtk::Layout}}}
:* at this point, we may use the //foreach// mechanism of the container
{{{
uint extH=20, extV=20;
Gtk::Container::ForeachSlot callback
= [&](Gtk::Widget& chld)
{
auto alloc = chld.get_allocation();
uint x = alloc.get_x();
uint y = alloc.get_y();
x += alloc.get_width();
y += alloc.get_height();
extH = max (extH, x);
extV = max (extV, y);
};
foreach(callback);
set_size (extH, extV);
}}}

//Management of dockable content panes//
Within each top level application window, the usable screen real estate can be split and arranged into a number of standard view building blocks. Each of this //panels// follows a specific preconfigured basic layout -- these are hard coded, yet customisable in detail. There is a finite list of such panel types available:
* the [[timeline display|GuiTimelineView]]
* the media viewer
* asset management
* information box
Within the preconfigured layout it is a panel's role to host and incorporate GuiComponentView elements.
Please note the distinction between UI component view and panel; in many cases there is one of each for a given kind, and these need to be distinguished: e.g. there is the [[timeline view|GuiTimelineView]] and there is the timeline panel, which houses several timelines (as tabs). Or there is the viewer component, which is located within a dedicated viewer panel each.
!Instantiation
A panel as such is //an identifiable, named, delineated space within the UI.//
* since it is just a space, instantiation of any panel //implies the creation of its content// -- at least in minimal or default form.
* the type of the panel determines, what the possible content of such a panel might be, and what need to be created to allow the panel to exist.
Consequently, there are two distinct ways to instantiate a panel
;by view allocation
:some process or action requires a specific GuiComponentView, and requests access or allocation through the ViewLocator.
:in this case, the panel holding this view is created on demand as a by-product
;by explicit setup
:initiated through user interaction, or caused by default wiring, a certain panel is made to exist somewhere in the UI
:in this case, the type of the panel determines what content view(s) need to be present to allow the panel to exist
!!!requirements and consequences for panel instantiation
Based on these considerations, and based on the fact that any GuiComponentView needs an ID and an UI-Bus connection, we can conclude that the actual panel creation need to be carried out by some core component directly related to the GuiTopLevel (to get UI-Bus access). And we need to ensure that the ID of any created or allocated view is //predictable,// since other parts need to be able to talk to this component via the bus, without any direct way to discover and retrieve the ID beforehand.
!Placing and addressing of embedded contents
A specific problem arises insofar other parts of the UI need to create, address and control some UI entities, which at the same time exist as part of a docking panel. This is a problem of crosscutting concerns: UI control and interaction structure should not be mingled with the generic concern to maintain a component as part of a screen layout. Unfortunately, the design of standard UI toolkit sets is incredibly naive when it comes to interaction design, and the only available alternative is to rely on frameworks, which come with a hard-wired philosophy.
As a result, we're forced to build our UI from components which fall short on the distinction between //ownership and control.//
!Hierarchy and organisation
Each top-level window //holds a single dock,// which in turn might hold several docking panels, in a layout arranged by the user. However, the library ''GDL'', which Lumiera uses to implement docking functionality, introduce the notion of a //dock master.// This is an invisible control component, which maintains a common layout of panels, possibly located within several separated docks. The role of the master can be re-assigned; typically it is automatically attained by the first dock created, and several docks will share the same master, if and only if they are created as dependent, which means to create the second dock by referring to the first one (there is a dedicated function in GDL to achieve that). Only docks managed by the same master may participate in drag-n-drop actions beyond the scope of a single dock. Moreover, all iconified panels of a given master are represented as icons within a single //dock bar.//
!!!Panel Locator
In accordance with this structure, we introduce a central component, the {{{PanelLocator}}} -- to keep track of all {{{DockArea}}} elements within the UI. The latter are responsible for managing the docking panels //within a specific// top-level {{{WorkspaceWindow}}}.

special LayerSeparationInterface which serves the main purpose to load the GuiStarterPlugin, thus bringing up the Lumiera GTK UI at application start.
It is of no further relevance beyond management of subsystem lifecycle -- which in itself is treated in Lumiera as a mere implementation concern and is not accessible by general application logic. Thus, the UI is largely independent and will be actively accessing the other parts of the application, while these in turn need to go through the public UI façades, esp. the GuiNotificationFacade for any active access to the UI and presentation layer.

Considering how to interface to and integrate with the GUI Layer. Running the GUI is //optional,// but it requires to be [[started up|GuiStart]], installing the necessary LayerSeparationInterfaces. Probably the most important aspect regarding the GUI integration is how to get [[access to and operate|GuiConnection]] on the [[Session|SessionInterface]].
More specifically, the integration is based on ''messaging''. To start with, the UI needs to be [[populated with updates|GuiModelUpdate]], and while in operation, it will send command messages over the [[UI-Bus]]. Effectively, the UI maintains its own [[model|GuiModel]], specifically tailored for display and translation into tangible UI entities.
----
In a preliminary attempt to establish an integration between the GUI and the lower layers, in 1/2009 we created an PlayerDummy, which "pulls" dummy frames from the (not yet existing) engine and displays them within an XV viewer widget. This highlighted the problems we're about to encounter and made us think about the more radically decoupled approach we followed thereafter...

Building a layered architecture is a challenge, since the lower layer //really// needs to be self-contained, while prepared for usage by the higher layer.
A major fraction of all desktop applications is written in a way where operational logic is built around the invocation from UI events -- what should be a shell turns into a backbone. One possible way to escape from this common anti pattern is to introduce a mediating entity, to translate between two partially incompatible demands and concerns: Sure, the "tangible stuff" is what matters, but you can not build any significant piece of technology if all you want is to "serve" the user.
Within the Lumiera GTK UI, we use a proxying model as a mediating entity. It is based upon the ''generic aspect'' of the SessionInterface, but packaged and conditioned in a way to allow a direct mapping of GUI entities on top. The widgets in the UI can be conceived as decorating this model. Callbacks can be wired back, so to transform user interface events into a stream of commands for the Proc-Layer sitting below.
The GUI model is largely comprised of immutable ID elements, which can be treated as values. A mutated model configuration in Proc-Layer is pushed upwards as a new structure and translated into a ''diff'' against the previous structure -- ready to be consumed by the GUI widgets; this diff can be broken down into parts and consumed recursively -- leaving it to the leaf widgets to adapt themselves to reflect the new situation.
&rarr; [[Building blocks of the GUI model|GuiModelElements]]
&rarr; [[GUI update mechanics|GuiModelUpdate]]
!{{red{WARNING 2/2017}}} more questionable than ever
The whole Idea to have a "UI model" appears more questionable than ever. It leads to tight coupling with the session and a lot of thread synchronisation headaches, without any clear benefit -- beyond just being the obvious no-brainer solution. During the last months, step by step, several presentation related structures emerged, which //indeed are structured to parallel the outline of the session.// But those structures are widgets and controllers, and it might well be that we do not need a model, beyond the data already present within the widget implementation. Rather it seems we'll get a nested structure of //presenters,// which are linked to the session with the help of the UI-Bus and the [[diff framework|TreeDiffModel]].
* {{red{as of 8/2018}}}, matters on a large scale converge towards a structure //without a classical UI model.// Seemingly we can do without...
* there is a working demonstration in {{{BusTerm_test}}}, which pushes mutation diffs against a mock UI-Element. The immediate response to receiving such a diff notification via the UI-Bus has now been coded; it incurs invoking a passed callback (functor), which performs within the originating context, but within the ~UI-Thread, and produces the actual diff //on-demand.// ("pull principle")
!!!building the model structures
A fundamental decision within the Lumiera UI is to build every model-like structure as immediate response to receiving a diff message pushed up into the UI.
* either this happens when some change occured, which is directly reflected into the UI by a local diff
* or a whole subtree of elements is built up step wise in response to a ''population diff''. This is an systematic description of a complete sub-structure in current shape, and is produced as emanation from a DiffConstituent.
!synchronisation guarantees
We acknowledge that the gui model is typically used from within the GUI event dispatch thread. This is //not// the thread where any session state is mutated. Thus it is the responsibility of this proxying model within the GUI to ensure that the retrieved structure is a coherent snapshot of the session state. Especially the {{{gui::model::SessionFacade}}} ensures that there was a read barrier between the state retrieval and any preceding mutation command. Actually, this is implemented down in Proc-Layer, with the help of the ProcDispatcher.
The forwarding of model changes to the GUI widgets is another concern, since notifications from session mutations arrive asynchronous after each [[Builder]] run. In this case, we send a notification to the widgets registered as listeners, but wait for //them// to call back and fetch the [[diffed state|TreeDiffModel]]. The notification will be dispatched into the GUI event thread (by the {{{GuiNotification}}} façade), which implies that also the callback embedded within the notification will be invoked by the widgets to perform within the GUI thread.
!generic model tree building blocks
According {{red{to the current plan (2018)}}}, the GuiModel will not be a dedicated and isolated data structure -- rather it will be interwoven with the actual structure of the widgets and controllers. Some controllers and also some widgets additionally implement the {{{gui::model::Tangible}}}-interface and thus act as "tangible" UI-Element, where each such "tangible element" directly corresponds to a component within the session model. The interface UI-Element requires such elements to provide an attachment point to receive mutations in the form of diff messages, while it remains a local implementation detail within each such element //how actually to respond and reflect// the changes indicated by the diff message. Typically the adding of child elements will result in creation of several GTK widgets, some of which are in turn also again UI-Element implementations, and only the latter count conceptually as "children".
While most actual details are abstracted away by this approach, it can be expected that the handling of typical diff changes is somewhat similar in most actual widgets. For this reason we provide a selection of generic adapters and building blocks to simplify the assembly of actual widget implementations.
!!!expanding and collapsing
Several UI elements offer the ability to be collapsed into a minimal representation to save screen real estate. The UI-Element protocol defines this to happen either by invoking a dedicated signal slot on the element, or by sending an appropriate message over the UI-Bus. The actual implementation is quite simple, but unfortunately requires knowledge regarding the specific widget configuration. A commonly used approach is to wrap the expandable/collapsible element into a {{{Gtk::Expandable}}}, but there are notable exceptions, where the widget is bound to handle the expanding or reducing of information display all by itself. We bridge this discrepancy by introducing an {{{Expander}}} interface to act as adapter.
* the default implementation holds an {{{Expander}}} functor. In default state, this functor as well as expanding / collapsing functionality remains disabled
* to enable it, two lambdas need to be provided, to configure
** how to find out about the expansion state of the widget
** how to change this expansion state (i.e. how to expand or collapse the widget)
!!!revealing an element
The UI-Element protocol also includes the ability to //reveal an element// -- which means actively to bring this element into sight, in case it is hidden, collapsed or obscured by scrolling away.
{{red{As of 8/2018 this is just an idea}}}, and many details still need to be considered. Yet at least one point is clear: implementing such a feature requires the help of the container widget actually holding the element to be revealed. It might even happen that also the collaboration of the container holding aforementioned immediate parent container is necessary -- indicating a recursive implementation scheme. The default implementation is based on a similar scheme as the expand/collapse functionality: here we embed a {{{Revealer}}} functor, which then needs to be outfitted with a lambda binding into the internals of the parent container to effect the necessary presentation changes.

''Building Blocks for the User Interface Model and Control structure''
The fundamental pattern for building graphical user interfaces is to segregate into the roles of __M__odel, __V__iew and __C__controler ([[MVC|http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller]]). This approach is so succesful, that it turned into a de-facto standard in commen UI toolkit sets. But larger, more elaborate and specialised applications introduce several cross cutting concerns, which create a tension within this MVC solution pattern.
[<img[UI-Bus and GUI model elements|uml/fig158213.png]]
The Lumiera GTK UI is built around a distinct backbone, separate from the structures required and provided by GTK.
While GTK -- especially in the object oriented incantation given by Gtkmm -- hooks up a hierarchy of widgets into a UI workspace, each of these widgets can and should incorporate the necessary control and data elements. But actually, these elements are local access points to our backbone structure, which we define as the UI-Bus. So, in fact, the local widgets and controllers wired into the interface are turned into ''Decorators'' of a backbone structure. This backbone is a ''messaging system'' (hence the name "Bus"). The terminal points of this messaging system allow for direct wiring of GTK signals. Operations triggered by UI interactions are transformed into [[Command]] invocations into the Proc-Layer, while the model data elements remain abstract and generic. The entities in our UI model are not directly connected to the actual model, but they are in correspondence to such actual model elements within the [[Session]]. Moreover, there is an uniform [[identification scheme|GenNode]].
;connections
:all connections are defined to be strictly //optional.//
:inactive connections render each element passive
;attributes
:the GuiModel supports a notion of generic attributes, treated as unordered unique children and referred by key name.
;local state
:recommendation is to have widget local state represented //within the UI toolkit (GTK).// On loosing bus connection, any element should disable itself or maybe even shut down.
;updates
:tangible UI elements are //passive.// User interaction just results in messages sent to the bus. Any update and mutation, on notification from the bus, is [[pulled from the model|GuiModelUpdate]]. The individual element is thus required to update itself (and its children recursively) into compliance with the provided state.
!building and updating the tree
The workspace starts out with a single element, corresponding to the »model root« in the ~Proc-Layer HighLevelModel. Initially, or on notification, an [[interface element|UI-Element]] requests a //status update// -- which conceptually implies there is some kind of conversation state. The backbone, as represented by the UI-Bus, might be aware of the knowledge state of its clients and just send an incremental update. Yet the authority or the backbone is absolute. It might, at its own discretion, send a full state update, to which the client elements are expected to comply. The status and update information is exposed in the form of a diff iterator. The client element, which can be a widget or a controller within the workspace, is expected to pull and extract this diff information, adjusting itself and destroying or creating children as applicable. This implies a recursive tree visitation, passing down the diff iterator alongside.
Speaking of implementation, this state and update mechanics relies on two crucial provisions: Lumiera's framework for [[tree diff representation|TreeDiffModel]] and the ExternalTreeDescription, which is an abstracted, ~DOM-like rendering of the relevant parts of the session; this model tree is comprised of [[generic node elements|GenNode]] acting as proxy for [[calls into|SessionInterface]] the [[session model|HighLevelModel]] proper.

Considerations regarding the [[structure of custom timeline widgets|GuiTimelineWidgetStructure]] highlight again the necessity of a clean separation of concerns and an "open closed design". For the purpose of updating the timeline(s) to reflect the HighLevelModel in Proc-Layer, several requirements can be identified
* we need incremental updates: we must not start redrawing each and everything on each tiny change
* we need recursive programming, since this is the only sane way to deal with tree like nested structures.
* we need specifically typed contexts, driven by the type demands on the consumer side. What doesn't make sense at a given scope needs to be silently ignored
* we need a separation of model-structure code and UI widgets. The GUI has to capture event and intent and trigger signals, nothing else.
* we need a naming and identification scheme. Proc must be able to "cast" callback state and information //somehow towards the GUI// -- without having to handle the specifics.
!the UI bus
Hereby we introduce a new in-layer abstraction: The UI-Bus.
* some events and wiring is strictly UI related. This can be handled the conventional way: Connect a ~SigC handler to the slot, in the ctor of your widget.
* but anything related to model interaction has to be targetted at the next applicable service point of the UI bus.
* the UI bus is implemented and covered by unit tests -- and //must not expose any GTK dependecies.// (maybe with the exception of {{{GString}}})
!!Decisions
* the UI bus is strictly single threaded.
* It performs in the GTK event thread.
* no synchronisation is necessary
* use constant values as far as possible
* the UI bus is offered by the GuiModel.
* it is //owned// by the GUI model.
* there is a global "kill switch". If toggled "off" any invocation is NOP.
* thus there is no need for any ownership or resource tracking
* we use simple language functors.
!initiating model updates
Model updates are always pushed up from Proc-Layer, coordinated by the ProcDispatcher. A model update can be requested by the GUI -- but the actual update will arrive asynchronously. The update information originate from within the [[build process|BuildFixture]]. {{red{TODO 10/2014 clarify the specifics}}}. When updates arrive, a ''diff is generated'' against the current GuiModel contents. The GuiModel is updated to reflect the differences and then notifications for any Receivers or Listeners are scheduled into the GUI event thread. On reception, it is their responsibility in turn to pull the targeted diff. When performing this update, the Listener thus actively retrieves and pulls the diffed information from within the GUI event thread. The GuiModel's object monitor is sufficient to coordinate this handover.
&rarr; representation of changes as a [[tree of diffs|TreeDiffModel]]
&rarr; properties and behaviour of [[generic interface elements|UI-Element]]
!!!timing and layering intricacies
A relevant question to be settled is as to where the core of each change is constituted. This is relevant due to the intricacies of multithreading: Since the change originates in the build process, but the effect of the change is //pulled// later from within the GUI event thread, it might well happen that at this point, meanwhile further changes entered the model. As such, this is not problematic, as long as taking the diff remains atomic. This leads to quite different solution approaches:
* we might, at the moment of performing the update, acquire a lock from the ProcDispatcher. The update process may then effectively query down into the session datastructure proper, even through the proxy of a diffing process. The obvious downside is that GUI response might block waiting on an extended operation in Proc, especially when a new build process was started meanwhile. A remedy might be to abort the update in such cases, since its effects will be obsoleted by the build process anyway.
* alternatively, we might incorporate a complete snapshot of all information relevant for the GUI into the GuiModel. Update messages from Proc must be complete and self contained in this case, since our goal is to avoid callbacks. Following this scheme, the first stage of any update would be a push from Proc to the GuiModel, followed by a callback pull from within the individual widgets receiving the notification later. This is the approach we choose for the Lumiera GUI.
!!!information to represent and to derive
The purpose of the GuiModel is to represent an anchor point for the structures //actually relevant for the UI.// To put that into context, the model in the session is not bound to represent matters exactly the way they are rendered within the GUI. All we can expect is for the //build process// -- upon completion -- to generate a view of the actually altered parts, detailing the information relevant for presentation. Thus we do retain an ExternalTreeDescription, holding all the information received this way within the GuiModel. Whenever a completed build process sends an updated state, we use the diff framework to determine the actually relevant differences -- both for triggering the corresponding UI widgets, and for forwarding focussed diff information to these widgets when they call back later from the UI event thread to pull actual changes.
!!!switch of typed sub-context
When dealing with structural (tree) diffing, there is a specific twist regarding child nodes of mixed type: In the general case, we can not assume that all children of a given node are of the same kind. The classical example is (X)HTML, where a node has //attributes,// various //nested tags// and //nested text content.// The //generic node// thus must be modelled as having several collections of children -- both ordered and un-ordered collections are possible -- and the content of each such sub-collection is itself polymorphic. This constitutes a challenge for the representation of data within the tree diff format. These difficulties can be overcome as follows
#anything, even nested "raw" content is represented //as node//
#all nodes can be addressed by an //generic identifier//
#the diff is in //prefix order,// i.e. it first only mentions the ordering, additions and deletions of nodes designated by these ~IDs
#we introduce a //bracketing construct// into the diff language, to enter a subnode within the diff representation
#the diff is produced and consumed //demand-driven (by pull)//
#whenever a node sees this bracketing construct, in invokes the respective child //recursively//
This treatment ensures each nested diff is consumed within a properly typed context, and without any need of switching types from the outside: the actual consumer of each part of the whole diff description just happens to know the actual meaning of those elements he processes itself, and passes control to others with adequate knowledge for the rest. Changes are broken down and transformed into //atomic changes.// For an internal data exchange, this is sufficient: in the end we're dealing with references to binary data anyway. But when it comes to an external, self-contained representation of diffs, we additionally need a way to attach raw chunks of data corresponding to the description of those atomic changes.
&rarr; this is the purpose of the {{{DataCap}}} within our [[generic node element|GenNode]]

LayerSeparationInterface provided by the GUI.
Access point for the lower layers to push information and state changes (asynchronously) to the GUI. Most operations within Lumiera are in fact initiated by the user through the GUI. In the course of such actions, the GUI uses the services of the lower layer and typically receives an immediate synchronous response to indicate the change was noted. Yet often, these operations may cause additional changes to happen asynchronously from the GUI's perspective. For example, an edit operation might trigger a re-build of the low-level model, which then detects an error. Any such consequences and notifications can be "cast" up into the UI, using the {{{NotificationService}}} described here.
Beyond that, a call to trigger shutdown of the UI layer is also exposed at this façade -- which becomes relevant when other [[sub-systems|Subsystem]] initiate general shutdown.
!Lifecycle and Threading concerns
The GuiNotificationFacade is installed as part of and managed by the ''UI Manager'', and connected to the UI-Bus, which happens while establishing the GuiTopLevel. Yet there is a specific twist, insofar GTK is ''not threadsafe by design''. Any event handling and presentation changes will happen from within a dedicated UI event loop, which needs to be started explicitly, after all of the primary windows and widgets are created and wired. Only after this point the UI becomes //life.//
A dedicated ''activation state'' is necessary for this reason -- which within the implementation translates into a queuing and dispatching facility to reschedule any calls into the UI event thread ''asynchronously''.
!Addressing of UI elements
Most calls in this interface require to specify a receiver or target, in the form of an element ID. It is assumed the caller effectively just knows these ~IDs, typically because the same ~IDs are also used as element ~IDs for the corresponding session entities. Even more so, since the whole UI representation of the session was at some point generated by //population diff messages,// which used the same ~IDs to indicate the creation of the corresponding UI representation elements.

While the HighLevelModel is self-contained and forms an autonomous »Universe«, the Lumiera GUI uses a well defined set of Metaphors, structural patterns and conventions to represent the user's view upon the model within the session.
The most fundamental principle is that of ''subsidiarity'': we understand both "the core" and "the UI" to be autonomous and responsible for their own concerns. The core has to deal with editing, arranging and processing, while the UI forms the materiality and mechanics of interaction. The [[link between both sides|GuiConnection]] is established through a communication system, the UI-Bus.
&rarr; the UI view [[is updated|GuiModelUpdate]] by ''diff messages''
&rarr; and in turn, commands are [[prepared and issued|GuiCommandBinding]] in the UI and sent as ''command messages''
Based on these foundations, we shape and form the core part of the interface, which is the [[timeline display|GuiTimelineView]]

Starting up the GUI is optional and is considered part of the Application start/stop and lifecycle.
* main and {{{lumiera::AppState}}} activate the lifecyle methods on the ~GuiSubsysDescriptor, accessible via the GuiFacade
* loading a GuiStarterPlugin actually
** loads the GUI (shared lib)
** creates instances of all the GUI services available through LayerSeparationInterfaces
*** GuiNotificationFacade
*** DisplayFacade
** Finally, the ~GTK-main() is invoked.
* bring up a network of framework entities known als the GuiTopLevel
* establish a connection to allow the GUI to be [[populated with content|GuiContentPopulation]]
!public services
The GUI provides a small number of public services, callable through LayerSeparationInterfaces. Besides that, the main purpose of the GUI of course is user interaction. Effectively the behaviour of the whole system is driven by GUI events to a large extent. These events are executed within the event handling thread (~GTK-main-Thread) and may in turn invoke services of the lower layers, again through the respective LayerSeparationInterfaces.
But the question of special interest here is how the //public services// of the GUI are implemented and made accessible for the lower Layers. Layer isolation is an issue here. If implemented in a rigorous manner, no facility within one layer may invoke implementation code of another layer directly. In practice, this tends to put quite some additional burden on the implementer, without any obvious benefit. Thus we decided to lower the barrier somewhat: while we still require that all service invoking calls are written against a public LayerSeparationInterface, actually the GUI (shared lib) is //linked// against the respective shared libs of the lower layers, thus especially enabling the exchange of iterators, closures and functor objects.
!!!Layer separation
Note that we retain strict isolation in the other direction: no part of the lower layers is allowed to call directly into the GUI. Thus it's especially interesting how access to some GUI public service from the lower layers works in detail.
* when the GUI plugin starts, instances of the Services implementing those public service interfaces are created.
* these service objects in turn hold an ~InstanceHandle, which cares to register and open the corresponding C Language Interface
* additionally this InstanceHandle is configured such as to create a "facade proxy" object, which is implemented within {{{liblumieracommon.so}}}
Now, when invoking an operation on some public interface, the code in the lower layers actually executes an implementation of this operation //on the facade proxy,// which in turn forwards the call through the CL interface into the GUI, where functionality is actually implemented by the corresponding service object instance.
!!!UI top level
Regarding the internal organisation of Lumiera's ~UI-Layer, there is a [[top level structure|GuiTopLevel]] to manage application lifecycle.
This top-level circle is established starting from the UI-Bus (''Nexus'') and the ''UI Manager'', which in turn creates the other dedicated control entities, especially the InteractionDirector. All these build-up steps are triggered right from the UI main() function, right before starting the ''UI event loop''. The remainder of the start-up process is driven by //contextual state,// as discovered by the top-level entities, delegating to the controllers and widgets.
!!!reaching the operative state
The UI is basically in operative state when the GTK event loop is running. Before this happens, the initial //workspace window// is explicitly created and made visible -- showing an empty workspace frame without content and detail views. However, from that point on, any user interaction with and UI control currently available is guaranteed to yield the desired effect, which is typically to issue and enqueue a command into the ProcDispatcher, or to show/hide some other UI element. Which also means that all backbone components of the UI have to be created and wired prior to entering operative state. This is ensured through the construction of the {{{UIManager}}}, which holds all the relevant core components either as directly managed "~PImpl" members, or as references. The GTK UI event loop is activated through a blocking call of {{{UIManager::performMainLoop()}}}, which also happens to open all external façade interfaces of the UI-Layer. In a similar vein, the //shutdown of the UI// can be effected through the call {{{UIManager::terminateUI()}}}, causing the GTK loop to terminate, and so the UI thread will leave the aforementioned {{{performMainLoop()}}} and commence to destroy the {{{UIManager}}}, which causes disposal of all core UI components.
!content population
In accordance with the Lumiera application architecture in general, the UI is not allowed to open and build its visible parts on its own behalf. Content and structure is defined by the [[Session]] while the UI takes on a passive role to receive and reflect the session's content. This is accomplished by issuing a //content request,// which in turn installs a listener within the session. This listener in turn causes a //population diff// to be sent upwards into the UI-Layer. Only in response to these content messages the UI will build and activate the visible structures for user interaction.
&rarr; GuiContentPopulation

A specially configured LumieraPlugin, which actually contains or loads the complete code of the (GTK)GUI, and additionally is linked dynamically against the application core lib. During the [[UI startup process|GuiStart]], loading of this Plugin is triggered from {{{main()}}}. Actually this causes spawning of the GTK event thread and execution of the GTK main loop.

There are various reasons why we might want to offer multiple equivalent UI representations of the same Timeline...
* the user might want to see several remote parts of the same timeline simultaneously, in focussed display
* we allow several indeptendent top-level windows (think several desktops), so it might just happen that the same timeline is selected in several windows
* we might want to introduce a focussed view on a nested sequence or virtual clip
Now, since we build our UI on the notion of mapping session contents via a messaging system onto suitable presenters in the UI, we get a conceptual mismatch. Basically we need to cut at some point and duplicate some connections. Either we need the ability within a timeline presentation entity to serve several sets of slave widgets, or we need the ability for those presentation entities to collaborate, where one of them becomes the leader and automatically forwards all notifications to the other members of the cluster. Or, alternatively, we could think of pushing that duplication down into the session, in which case we get a TimelineClone entity, and the Builder then needs to be aware of this situation and generate duplicated responses to be sent to the UI.
{{red{While reconsidering this topic in 10/2018}}}, it looks like I am leaning towards the most systematic option, which is to represent this duplication already within the session as TimelineClone. The rationale is
* splitting this way is likely to produce the least accidental complexity -- at that point we are forced to cross-cut only a small number of other concerns. Were we to cut and duplicate within the UI, we'd be forced to carry care for slave entities into a huge number of entirely unrelated UI concerns, like layout management or media display feedback.
* in the case of a focussed view on a nested sequence we are even forced to go that route, since doing otherwise would carry over core session responsibilities into the presentation layer. Consequently, any other solution scheme causes duplicating of functionality.
* however -- if we implement slave timelines already within the session, we still need to make the UI counterpart basically aware of the situation. Thus the {{{TimelineControler}}} needs the ability to delegate some behaviour to another primary controller. That being said -- I still confirm the decision to postpone that topic altogether...
In any case, this is an advanced topic, and nowhere near trivial. It seems reasonable to reject opening duplicate timeline presentations as a first step, and then address this topic way later, when we've gained sufficient knowledge regarding all the subtleties of timeline presentation and editing.

Within the Lumieara GUI, the [[Timeline]] structure(s) from the HighLevelModel are arranged and presented according to the following principles and conventions.
Several timeline views may be present at the same time -- and there is not necessarily a relation between them, since »a Timeline« is the top-level concept within the [[Session]]. Obviously, there can also be several //views// based on the same »Timeline« model element, and in this latter case, these //coupled views// behave according to a linked common state. An entity »Timeline« as represented through the GUI, emerges from the combination of several model elements
* a root level [[Binding|BindingMO]] acts as framework
* this binding in turn ties a [[Sequence]]
* and the sequence provides a [[Fork ("tree of tracks")|Fork]]
* within the scope of these tracks, there is content ([[clips|Clip]])
* and this content implies [[output designations|OutputDesignation]]
* which are resolved to the [[global Pipes|GlobalPipe]] belonging to //this specific Timeline.//
* after establishing a ViewerPlayConnection, a running PlayProcess exposes a PlayController
Session, Binding and Sequence are the mandatory ingredients.
!Basic layout
[>img[Clip presentation control|draw/UI-TimelineLayout-1.png]]The representation is split into a ''Header pane'' exposing structure and configuration, and a ''Content pane'' extending in time. The ''Time ruler'' running alongside the top of the content pane represents the //position in time.// Beyond this temporal dimension, the content area is conceived as a flexible working space. This working space //can// be structured hierarchically -- when interacting with the GUI, hierarchical nesting will be created and collapsed on demand. Contrast this with conventional editing applications which are built upon the rigid notion of "Tracks": Lumiera is based on //Pipes// and //Scopes// rather than Tracks.
In the temporal dimension, there is the usual scrolling and zooming of content, and possibly a selected time range, and after establishing a ViewerPlayConnection, there is an effective playback location featured as a "Playhead"
The workspace dimension (vertical layout) is more like a ''Fork'', which can be expanded recursively. More specifically, each strip or layer or "track" can be featured in //collapsed// or //expanded state.//
* the collapsed state features a condensed representation ("the tip of the iceberg"). It exposes just the topmost entity, and might show a rendered (pre)view. Elements might be stacked on top, but any element visible here //is still accessible.//
* when expanding, the content unfolds into...
** a ''scope ruler'' to represent the whole sub-scope. This is rendered as a small pane, extending horizontally, holding any locally attached labels and track-wide or temporally scoped effects
** the content stack, comprised of [[clip widgets|GuiClipWidget]], attached effects and transitions
** a stack of nested sub-scopes (recursive).
@@float: right;background-color: #e9edf8;width: 82ex;padding: 2ex;margin: 0px 4em 1em 2em;__Note in this example__
* on top level, there are two tracks, the second track has nested sub tracks
* Clip-2 has an effect attached, Clip-3 is expanded and also has an effect attached
* the second track has a global effect attached; it shows up in the scope ruler
@@
This collapsed, expanded and possibly nested workspace structure is always exactly paralleled in the header pane. In addition, it allows to configure specific placement properties for each nested scope, which especially means to display faders and some toggles, depending on what kind of placement was added. Of course, this placement configuration needs to be collapsible too. Effects and markers can appear at various different scopes, sometimes requiring an abridged display
!!!lifecycle and instances
A given instance of the {{{TimelineWidget}}} is always dedicated to render the contents of //one specific timeline.// We never switch the data model while retaining the UI entities. This also means, a given instance is tied to one conversation with the core; it is created when the core tells us about this timeline with an initial population diff, and it lives until either this timeline is discarded in the core model, or the whole session is shut down.
The dockable ''timeline pannel'' holds onto the existing {{{TimelineWidget}}} instances, allowing to make one of them visible for interaction. Yet the timeline itself is represented by the {{{TimelineControler}}}, which lives //within this widget,// and is attached to and managed by the InteractionDirector, who incorporates the role of representing the [[model root|ModelRootMO]]. Aside of the timeline display, there is also an ''asset pannel''; display of and interaction with those asset views is handled by dedicated widgets, backed by the {{{AssetControler}}} -- which is also a child of and managed by the InteractionDirector, since assets are modelled as global part of the [[Session]].
In case the UI starts with no session present in the core, an //empty timeline placeholder// will be displayed, which provides UI for creating a new session...
!!!slave Timelines
It is reasonable to expect the ability to have multiple [[slave timeline presentations|GuiTimelineSlave]] to access the same underlying timeline structure.
Currently {{red{as of 10/2018}}} there is a preference to deal with that problem on session level -- but for now we decide to postpone this topic &rarr; [[#1083|http://issues.lumiera.org/ticket/1083]]
!!!nesting
By principle, this workspace structure is //not a list of "Tracks"// -- it is a system of ''nested scopes''. The nesting emerges on demand.
In the most general case, there can be per-track content and nested content at the same point in time. The GUI is able to represent this state. But, due to the semantics of Lumiera's HighLevelModel, top-level content and nested content are siblings //within the same scope.// Thus, at a suitable point {{red{to be defined}}}, an equivalence transformation is applied to the GUI model, by prepending a new sibling track and moving top-level content there.
&rarr; important question: how to [[organise the widgets|GuiTimelineWidgetStructure]]

The Timeline is probably the most prominent place in the GUI where we need to come up with a custom UI design.
Instead of combining standard components in one of the well-known ways, here we need to come up with our own handling solution -- which also means to write one or several custom GTK widgets. Thus the question of layout and screen space division and organisation becomes a crucial design decision. The ~GTK-2 Gui, as implemented currently, did already take some steps along this route, yet this kind of decision should be cast and documented explicitly (be it after the fact).
Right away, this topic touches a tricky design and architectural challenge: the → question [[how to organise custom widgets|GuiCustomWidget]].
In a nutshell, ~GTKmm offers several degrees of customisation, namely to build a custom widget class, to build a custom container widget, and to use the [[Gtk::Layout "canvas widget"|GtkLayoutWidget]], possibly combined with //custom drawing.// In addition to assembling a timeline widget class by combining several nested panes, the timeline display needs to rely on the latter approach to allow for the necessary flexible arrangement of [[clip widgets|GuiClipWidget]] within the [[track fork|Fork]].
!the header pane problem
From general considerations we draw the conclusion to have a track header pane area always visible to the left. For one this means that our top level widget organisation in the timeline will be a horizontal split. And then this means that we get two distinct sub widgets, whose vertical layout needs to be kept in sync. And even more so, presumably the most adequate implementation technique is different in both parts: the header pane looks like a classical fit for the paradigm of nested boxes and grid layout within those boxes, while the right part -- the actual track contents -- impose very specific layout constraints not served by any of the pre-existing layout containers, which means we have to resort to custom drawing on a canvas widget. Following this line of thought, we need an overarching layout manager to coordinate these two disjoint technologies. Any viable alternatives?
!!!considering a table grid layout
The layout mechanics we try to establish here by explicit implementation would be more or less a given, if instead we'd build the whole timeline display from one base widget, which needs to be a table layout, i.e. {{{Gtk::Grid}}}. We'd use two columns, one for the header pane area, one for the timeline display, and we'd use N+1 rows, with the head row holding the time ruler and the additional rows holding individual tracks. But to get the specific UI mechanics desirable for a timeline display, we had to introduce some twists:
* //Scrolling is rather special.// We could use the default scrolling mechanisms in vertical direction only, while, as far as the Gtk Layout management is involved, we have to dress up things such as to make it appear as limited to the available horizontal space, so GTK never attempts to scroll the grid as a whole horizontally.
* rather, we have to integrate a free standing horizontal scrollbar somewhere //in the second column.// This scrollbar has to be tied to a scrolling/zooming function, which we apply by explicit coding synchronously to all cell content in all rows of the second column.
* and then obviously we'd have to add a canvas widget with custom drawing to each of those cells in the second column, each featuring the contents of a single track.
* the //nested scopes are difficult to represent in this layout.// This is, because we use a row for each track-like structure at most fine grained level. If we were to create additional visual clues to indicate nested structures, like indentation or a bracketing structure, we'd have to implement those by repetitively adding appropriate graphical structures to each row in the first column.
* it would be non trivial to place controls (e.g. a volume or fade control) acting on all sub-tracks within a group. We'd have to add those controls in a top row representing the whole group and then we'd have to indicate somehow graphically that those pertain to all nested tracks. In fact, this problem is not limited to a table grid implementation approach, since it is related to usage of screen real estate. But a table grid implementation takes away almost all remaining flexibility and thus makes a workable solution much harder: we'd have to allocate a lot of vertical space for those controls, space, which is wasted on the right side, within the track content display, since there all we need is a small overview ruler without much content demands in vertical direction.
* this problem is mitigated once we add a track with //automation data// linked to the mentioned controls. Yet still, this is not the default situation...
On the other hand, what would be the //obvious benefits...?//
* we just have to add stuff in both the left / right part of the display and get the vertical space management sorted out by framework code
While the special setup for scrolling doesn't really count (since it is necessary anyway), after this initial investigation it seems clear that a global grid layout doesn't yield enough benefit to justify all the quirks and limitations its use would impose.
!!!follow-up to the obvious choices
We came to this point of re-considering the overall organisation of widgets, after having to re-write the initial version of our timeline widget. This initial version was developed by Joel Holdsworth, and it followed a similar reasoning, involving a global timeline layout manager. The distinction between the two panes was not so clear though, and the access to the model code was awkward at places, so the necessity to re-write the timeline widget due to the transition to ~GTK-3 looks like a good opportunity to re-do the same basic reasoning a second time, verify decisions taken and improve matters turning out as difficult on first attempt.
So we get a timeline custom widget, which at top level establishes this two-part layout, provides the global scrollbars and integrates custom widget components for both parts. And this top-level timeline widget owns a layout manager, plus it exposes a common view management interface, to be used both from internal components (e.g. zoom widgets within the UI) and from external actors controlling the timeline display from a global level. Also at this global level, we get to define a layout control interface, the TimelineDisplayManager, which has to be implemented within the recursively structured parts of the timeline display, and which is used by the global layout manager to exert control over the layout as a whole. {{red{Note (11/2016)}}}: if this layout control interface works push or pull style is a decision yet to be worked out during the course of the implementation.
!dealing with nested structures
The handling of objects structured into nested scopes is a hallmark of the very specific approach taken by Lumiera when it comes to attaching, arranging and relating media objects. But here in the UI display of the timeline, this approach creates a special architectural challenge: the only sane way to deal with nested structures without exploding complexity is to find some way to exploit the ''recursive self similarity'' inherent in any tree structure. But the problematic consequence of this assessment is the tension, even contradiction it creates to the necessities of GUI programming, which forces us to come up with one single, definitive widget representation of what is going on eventually. The conclusion is that we need to come up with an interface such as to allow building and remoulding of the UI display through incremental steps -- where each of this incremental steps relies solely on relative, context based information. Because this is the only way we can deal with building a tree structure by recursive programming. We must not demand the individual step to know its arrangement within the tree, other than indicating a "current" or a "parent" reference point.
The structure of the display is extended or altered under two circumstances:
# some component receives a [[diff mutation message|MutationMessage]], prompting to add or remove a //child component.//
# the display (style) of some component is expanded or collapsed.
Here, the "component" relevant for such structural changes is always the UI representation of a track. Beyond that, the layout can also be changed //without changing the display structure,// when some embedded component, be it placement (in the track heads / the patchbay) or a clip, effect or transition, is expanded or collapsed. In such a case, a resizing challenge needs to be directed towards the next enclosing track container.
From these observations we can draw the conclusion, that we'll build a ''local structural model'', to reflect the logical relations between the parts comprising the timeline display. More precisely, these structuring components are not mere model objects, rather they are mediating entities used to guide and organise the actual view entities, which in turn are passive. They are more like a view model, while also bearing some local controller responsibilities. For this reason, we prefer to term these as ''presenters'' -- i.e. TrackPresenter and ClipPresenter. And each of these local representation components holds onto a ''display context'', which generally links it //into two different display widget stacks// within the two parts of the actual timeline display. Adding a child component thus becomes a rather tricky operation, involving to link possibly two child widgets into two disjoint parent widgets, thereby forming a similar display context for the child presenter. Overall, the guiding idea is that of self similarity: on each level, we have to reproduce the same relations and collaborations as present in the parent level.
!!!building the timeline representation structure
It is a fundamental decision within the Lumiera UI that any structure as to be injected as diff by the core. We do not hold a structure or data model within some UI entity and then "interpret" that model into a widget structure -- rather we build the widget structure itself in response to a diff message describing the structure. Especially in the case of timelines, the receiver of those messages is the InteractionDirector, which corresponds to the model root. On being prompted to set up a timeline, it will allocate a {{{TimelineWidget}}} within a suitable timeline docking panel and then attach to the {{{TimelineController}}} embedded within the widget.
{{red{Problem 10/2018}}} how can the InteractionDirector //manage// a timeline while the timeline widget physically resides within the panel? Can we exploit a simliar structure as we did for the error log?
* in part yes -- however we introduce an mediating entity, the {{{TimelineGui}}} proxy
* it uses a {{{WLink}}} to refer to the actual {{{TimelineWidget}}} -- meaning the widget need not exist, and will detach automatically when destroyed by its holder, which is the {{{TimelinePanel}}}
* {{red{TODO 11/2018}}} still need to care for removing a {{{TimelineWidget}}} when the logically managing parent, i.e. the InteractionDirector deletes the {{{TimelineGui}}} proxy
The diff describing and thus assembling the UI representation of a timeline is typically a ''population diff'' -- which means, it is a consolidated complete description of the whole sub-structure rooted below that timeline. Such a population diff is generated as emanation from the respective DiffConstituent.
!!!interplay with diff mutation
Applying a diff changes the structure, that is, the structure of the local model, not the structure of the display widgets. Because the latter are an entirely private concern of the UI and their structure is controlled by the model components in conjunction with the display manager. And since diff application effects the contents of the model such as to make the intended structural changes happen (indirectly), we are well advised to tie the display control and the widgets very closely to those local model elements, such as to //adjust the display automatically.//
* when a new model element is added, it has to inject something automatically into the display
* when a model element happens to be destructed, the corresponding display element has to be removed.
* such might be triggered indirectly, by clean-up of leftovers, since the {{{DiffApplicator}}} re-orders and deletes by leaving some data behind
* the diff also re-orders model elements, which does not have an immediate effect on the display, but needs to be interpreted separately.
Wrapping this together we get a fix up stage after model changes, where the display is re-adjusted to fit the new situation. This works in concert with the [[display manager|TimelineDisplayManager]] representing only those elements as actual widgets, which get a real chance to become visible. This way we can build on the assumption that the actual number of widgets to be managed any time remains so small as to get away with simple linear list processing. It remains to be seen how far this assumption can be pushed -- the problem is that the GTK container components don't support anything beyond such simple linear list processing; there isn't even a call to remove all child widgets of a container in a single pass.

To a large extent, the Lumiera user interface is built around a //backbone structure,// known as the UI-Bus.
But there are some dedicated top-level entities, collaborating to maintain a consistent application lifecycle
;Application
:the application object, {{{GtkLumiera}}} is what executes within the GuiStarterPlugin and thus within the Gtk event thread
:it is of no further relevance for any of the other UI entities, insofar it just creates and wires the top level constituents and encompasses their lifetime
;~UI-Bus
:the backbone of the user interface
:as central communication system, the UI-Bus has a star shaped topology with a central router and attached CoreService
;UI Manager
:maintain a coherent global interface and perform the GTK event loop
:responsible for all global framework concerns, resources and global application state
;Interaction Director
:establish the connection between global interaction state and global session state
:the InteractionDirector is the root controller and corresponds to the [[root object in session|ModelRootMO]].
;Window List
:organise and maintain the top level workspace windows
:which involves the interplay with [[docking panels|GuiDockingPanel]]
;Notification Façade
:attachment point for lower layers and anyone in need to "talk to the UI"
:the GuiNotificationFacade is a LayerSeparationInterface and integrated with Lumiera's interface system
Together, these entities form a cohesive circle of collaborating global managers, known as ''global UI context''; the interplay of these facilities is essentially an implementation detail, insofar there is not much necessity (and only a narrow API) for the rest of the UI to address those directly. Rather, each member of this circle serves a dedicated purpose and is visible to the rest of the application through some kind of service abstraction. For example, the InteractionDirector is mapped as a top-level model element into the logical model of the UI; typically, other parts of the application address this service through messages via the UI-Bus, while the Interaction Director itself is responsible to create a link between model and interaction state -- a service, which is fulfilled in a transparent way.
!Control structure
Within the UI-Layer, we distinguish between //core concerns and UI concerns,// the latter encompassing anything related to UI mechanics, presentation state, interaction state and InteractionControl. Core concerns are delegated and handled by the lower layers of the architecture, while the UI plays a passive role. This is a fundamental decision and leads to a dichotomy, where -- depending on the context -- a given part might operate as a slave related to core concerns, while taking a leading or controlling position when it comes to UI concerns. Deliberately, both sides are bridged by being interwoven into the same entities, and any entity of relevance to core concerns is also attached to the UI-Bus. Regarding the build-up of the UI, parts and elements are made visible and accessible through widgets, but widgets are created, installed and operated by controllers. The top-level circle defines global actions, which are passed through the relevant controller to come into effect.

While the low-level model holds the data used for carrying out the actual media data processing (=rendering), the high-level model is what the user works upon when performing edit operations through the GUI (or script driven in &raquo;headless mode&laquo;). Its building blocks and combination rules determine largely what structures can be created within the [[Session]].
On the whole, it is a collection of [[media objects|MObjects]] stuck together and arranged by [[placements|Placement]].
Basically, the structure of the high-level model is is a very open and flexible one &mdash; every valid connection of the underlying object types is allowed &mdash; but the transformation into a low-level node network for rendering follows certain patterns and only takes into account any objects reachable while processing the session data in accordance to these patterns. Taking into account the parameters and the structure of these objects visited when building, the low-level render node network is configured in detail. In a similar vein, the [[representation within the GUI|GuiPattern]] is based on distinct patterns and conventions -- any object not in line with these conventions remains //hidden// and is //silently ignored.//
The fundamental metaphor or structural pattern is to create processing ''pipes'', which are a linear chain of data processing modules, starting from an source port and providing an exit point. [[Pipes|Pipe]] are a //concept or pattern,// they don't exist as objects. Each pipe has an input side and an output side and is in itself something like a Bus treating a single [[media stream|StreamType]] (but this stream may still have an internal structure, e.g. several channels related to a spatial audio system). Other processing entities like effects and transitions can be placed (attached) at the pipe, resulting them to be appended to form this chain. Optionally, there may be a ''wiring plug'', requesting the exit point to be connected to another pipe. When omitted, the wiring will be figured out automatically.
Thus, when making an connection //to// a pipe, output data will be sent to the //source port// (input side) of the pipe, wheras when making a connection //from// a pipe, data from it's exit point will be routed to the destination. Incidentally, the low-level model and the render engine employ //pull-based processing,// but this is rather of no relevance for the high-level model.
[img[draw/high-level1.png]]
Normally, pipes are limited to a //strictly linear chain// of data processors ("''effects''") working on a single data stream type, and consequently there is a single ''exit point'' which may be wired to an destination. As an exception to this rule, you may insert wire tap nodes (probe points), which explicitly may send data to an arbitrary input port; they are never wired automatically. It is possible to create cyclic connections by such arbitrary wiring, which will be detected by the builder and flagged as an error.
While pipes have a rather rigid and limited structure, it is allowed to make several connections to and from any pipe &mdash; even connections requiring an stream type conversion. It is not even necessary to specify //any// output destination, because then the wiring will be figured out automatically by searching the context and finally using some general rule. Connecting multiple outputs to the input of another pipe automatically creates a ''mixing step'' (which optionally can be controlled by a fader). Several pipes may be joined together by a ''transition'', which in the general case simultaneously treats N media streams. Of course, the most common case is to combine two streams into one output, thereby also mixing them. Most available transition plugins belong to this category, but, as said, the model isn't limited to this simple case, and moreover it is possible to attach several overlapping transitions covering the same time interval.
Individual Media Objects are attached, located or joined together by ''Placements''. A [[Placement]] is a handle for a single MObject (implemented as a refcounting smart-ptr) and contains a list of placement specifications, called LocatingPin. Adding an placement to the session acts as if creating an //instance.// (it behaves like a clone in case of multiple placements of the same object). Besides absolute and relative placement, there is also the possibility of a placement to stick directly to another MObject's placement, e.g. for attaching an effect to a clip or to connect an automation data set to an effect. This //stick-to placement// creates sort of a loose clustering of objects: it will derive the position from the placement it is attached to. Note that while the length and the in/out points are a //property of the ~MObject,// it's actual location depends on how it is //placed// and thus can be maintained quite dynamically. Note further that effects can have an length on their own, thus by using these attachement mechaics, the wiring and configuration within the high-level model can be quite time dependant.
[>img[draw/high-level2.png]]
Actually a ''clip'' is handled as if it was comprised of local pipe(s). In the example shown here, a two-channel clip has three effects attached, plus a wiring plug. Each of those attachments is used only if applicable to the media stream type the respective pipe will process. As the clip has two channels (e.g. video and audio), it will have two ''source ports'' pulling from the underlying media. Thus, as showed in the drawing to the right, by chaining up any attached effect applicable to the respective stream type defined by the source port, effectively each channel (sub)clip gets its own specifically adapted processing pipe.
@@clear(right):display(block):@@
!!Example of an complete Session
[img[draw/high-level3.png]]
The Session contains several independent [[sequences|Sequence]] plus an output bus section (''global Pipes'') attached to the [[Timeline]]. Each sequence holds a collection of ~MObjects placed within a ''tree of tracks''.
Within Lumiera, "tracks" (actually implemented as [[forks|Fork]]) are a rather passive means for organizing media objects, but aren't involved into the data processing themselves. The possibility of nesting tracks allows for easy grouping. Like the other objects, tracks are connected together by placements: A track holds the list of placements of its child tracks. Each sequence holds a single placement pointing to the root track.
As placements have the ability to cooperate and derive any missing placement specifications, this creates a hierarchical structure throughout the session, where parts on any level behave similar if applicable. For example, when a fork ("track") is anchored to some external entity (label, sync point in sound, etc), all objects placed relatively to this track will adjust and follow automatically. This relation between the track tree and the individual objects is especially important for the wiring, which, if not defined locally within an ~MObject's placement, is derived by searching up this track tree and utilizing the wiring plug locating pins found there, if applicable. In the default configuration, the placement of an sequence's root track contains a wiring plug for video and another wiring plug for audio. This setup is sufficient for getting every object within this sequence wired up automatically to the correct global output pipe. Moreover, when adding another wiring plug to some sub track, we can intercept and reroute the connections of all objects creating output of this specific stream type within this track and on all child tracks.
Besides routing to a global pipe, wiring plugs can also connect to the source port of an ''meta-clip''. In this example session, the outputs of 'Seq-2' as defined by locating pins in it's root track's placement, are directed to the source ports of a [[meta-cllip|VirtualClip]] placed within 'Seq-1'. Thus, within 'Seq-1', the contents of 'Seq-2' appear like a pseudo-media, from which the (meta) clip has been taken. They can be adorned with effects and processed further completely similar to a real clip.
Finally, this example shows an ''automation'' data set controlling some parameter of an effect contained in one of the global pipes. From the effect's POV, the automation is simply a ParamProvider, i.e a function yielding a scalar value over time. The automation data set may be implemented as a bézier curve, or by a mathematical function (e.g. sine or fractal pseudo random) or by some captured and interpolated data values. Interestingly, in this example the automation data set has been placed relatively to the meta clip (albeit on another track), thus it will follow and adjust when the latter is moved.

This wiki page is the entry point to detail notes covering some technical decisions, details and problems encountered in the course of the years, while building the Lumiera application.
* [[Packages, Interfaces and Namespaces|InterfaceNamespaces]]
* [[Memory Management Issues|MemoryManagement]]
* [[Creating and registering Assets|AssetCreation]]
* [[Creating new Objects|ObjectCreation]]
* [[Multichannel Media|MultichannelMedia]]
* [[Editing Operations|EditingOperations]]
* [[Handling of the current Session|CurrentSession]]
* [[collecting Ideas for Implementation Guidelines|ImplementationGuidelines]]
* [[using the Visitor pattern?|VisitorUse]] -- resulting in [[»Visiting-Tool« library implementation|VisitingToolImpl]]
* [[Handling of Tracks and render Pipes in the session|TrackPipeSequence]]. [[Handling of Tracks|TrackHandling]] and [[Pipes|PipeHandling]]
* [[getting default configured|DefaultsManagement]] Objects relying on [[rule-based Configuration Queries|ConfigRules]]
* [[integrating the Config Query system|ConfigQueryIntegration]]
* [[identifying the basic Builder operations|BasicBuildingOperations]] and [[planning the Implementation|PlanningNodeCreatorTool]]
* [[how to handle »attached placement«|AttachedPlacementProblem]]
* working out the [[basic building situations|BuilderPrimitives]] and [[mechanics of rendering|RenderMechanics]]
* how to classify and [[describe media stream types|StreamType]] and how to [[use them|StreamTypeUse]]
* considerations regarding [[identity and equality|ModelObjectIdentity]] of objects in the HighLevelModel
* the [[identification of frames and nodes|NodeFrameNumbering]]
* the relation of [[Project, Timelines and Sequences|TimelineSequences]]
* how to [[start the GUI|GuiStart]] and how to [[connect|GuiConnection]] to the running UI.
* build the first LayerSeparationInterfaces
* create an uniform pattern for [[passing and accessing object collections|ForwardIterator]]
* decide on SessionInterface and create [[Session datastructure layout|SessionDataMem]]
* shaping the GUI/~Proc-Interface, based on MObjectRef and the [[Command frontend|CommandHandling]]
* defining PlacementScope in order to allow for [[discovering session contents|Query]]
* working out a [[Wiring concept|Wiring]] and the foundations of OutputManagement
* shaping the foundations of the [[player subsystem|Player]]
* detail considerations regarding [[time and time quantisation|TimeQuant]]
* [[Timecode]] -- especially the link of [[TC formats and quantisation|TimecodeFormat]]
* designing how to [[build|BuildFixture]] the [[Fixture]] (...{{red{WIP}}}...)
* from [[play process|PlayProcess]] to [[frame dispatching|FrameDispatcher]] and [[node invocation|NodeInvocation]]
* how to shape the GuiConnection: with the help of a mediating GuiModel, which acts as UI-Bus, exchanging TreeDiffModel messages for GuiModelUpdate
* shape the Backbone of the UI, the UI-Bus and the GuiTopLevel
* build a framework for InteractionControl in the User Interface
* establish a flexible layout structure for the GuiTimelineView

!Observations, Ideas, Proposals
''this page is a scrapbook for collecting ideas'' &mdash; please don't take anything noted here too literal. While writing code, I observe that I (ichthyo) follow certain informal guidelines, some of which I'd like to note down because they could evolve into general style guidelines for the Proc-Layer code.
* ''Inversion of Control'' is the leading design principle.
* but deliberately we stay just below the level of using Dependency Injection. Singletons and call-by-name are good enough. We're going to build //one// application, not any conceivable application.
* write error handling code only if the error situation can be actually //handled// at this place. Otherwise, be prepared for exceptions just passing by and thus handle any resources by "resource acquisition is initialisation" (RAII). Remember: error handling defeats decoupling and encapsulation.
* (almost) never {{{delete}}} an object directly, use {{{new}}} only when some smart pointer is at hand.
* clearly distinguish ''value objects'' from objects with ''reference semantics'', i.e. objects having a distict //object identity.//
* when user/client code is intended to create reference-semantics objects, make the ctor protected and provide a factory member called {{{create}}} instead, returning a smart pointer
* similarly, when we need just one instance of a given service, make the ctor protected and provide a factory member called {{{instance}}}, to be implemented by the [[dependency factory|DependencyFactory]].
* whenever possible, prefer this (lazy initialised [[dependency|DependencyFactory]]) approach and avoid static initialisation magic
* avoid doing anything non-local during the startup phase or shutdown phase of the application, especially avoid doing substantial work in any dtor.
* avoid asuming anything that can't be enforced by types, interfaces or signatures; this means: be prepared for open possibilities
* prefer {{{const}}} and initialisation code over assignment and active changes (inspired by functional programming)
* code is written for ''being read by humans''; code shall convey its meaning //even to the casual reader.//

/***
''InlineJavascriptPlugin for ~TiddlyWiki version 1.2.x and 2.0''
^^author: Eric Shulman - ELS Design Studios
source: http://www.TiddlyTools.com/#InlineJavascriptPlugin
license: [[Creative Commons Attribution-ShareAlike 2.5 License|http://creativecommons.org/licenses/by-sa/2.5/]]^^
Insert Javascript executable code directly into your tiddler content. Lets you ''call directly into TW core utility routines, define new functions, calculate values, add dynamically-generated TiddlyWiki-formatted output'' into tiddler content, or perform any other programmatic actions each time the tiddler is rendered.
!!!!!Usage
<<<
When installed, this plugin adds new wiki syntax for surrounding tiddler content with {{{<script>}}} and {{{</script>}}} markers, so that it can be treated as embedded javascript and executed each time the tiddler is rendered.
''Deferred execution from an 'onClick' link''
By including a label="..." parameter in the initial {{{<script>}}} marker, the plugin will create a link to an 'onclick' script that will only be executed when that specific link is clicked, rather than running the script each time the tiddler is rendered.
''External script source files:''
You can also load javascript from an external source URL, by including a src="..." parameter in the initial {{{<script>}}} marker (e.g., {{{<script src="demo.js"></script>}}}). This is particularly useful when incorporating third-party javascript libraries for use in custom extensions and plugins. The 'foreign' javascript code remains isolated in a separate file that can be easily replaced whenever an updated library file becomes available.
''Defining javascript functions and libraries:''
Although the external javascript file is loaded while the tiddler content is being rendered, any functions it defines will not be available for use until //after// the rendering has been completed. Thus, you cannot load a library and //immediately// use it's functions within the same tiddler. However, once that tiddler has been loaded, the library functions can be freely used in any tiddler (even the one in which it was initially loaded).
To ensure that your javascript functions are always available when needed, you should load the libraries from a tiddler that will be rendered as soon as your TiddlyWiki document is opened. For example, you could put your {{{<script src="..."></script>}}} syntax into a tiddler called LoadScripts, and then add {{{<<tiddler LoadScripts>>}}} in your MainMenu tiddler.
Since the MainMenu is always rendered immediately upon opening your document, the library will always be loaded before any other tiddlers that rely upon the functions it defines. Loading an external javascript library does not produce any direct output in the tiddler, so these definitions should have no impact on the appearance of your MainMenu.
''Creating dynamic tiddler content''
An important difference between this implementation of embedded scripting and conventional embedded javascript techniques for web pages is the method used to produce output that is dynamically inserted into the document:
* In a typical web document, you use the document.write() function to output text sequences (often containing HTML tags) that are then rendered when the entire document is first loaded into the browser window.
* However, in a ~TiddlyWiki document, tiddlers (and other DOM elements) are created, deleted, and rendered "on-the-fly", so writing directly to the global 'document' object does not produce the results you want (i.e., replacing the embedded script within the tiddler content), and completely replaces the entire ~TiddlyWiki document in your browser window.
* To allow these scripts to work unmodified, the plugin automatically converts all occurences of document.write() so that the output is inserted into the tiddler content instead of replacing the entire ~TiddlyWiki document.
If your script does not use document.write() to create dynamically embedded content within a tiddler, your javascript can, as an alternative, explicitly return a text value that the plugin can then pass through the wikify() rendering engine to insert into the tiddler display. For example, using {{{return "thistext"}}} will produce the same output as {{{document.write("thistext")}}}.
//Note: your script code is automatically 'wrapped' inside a function, {{{_out()}}}, so that any return value you provide can be correctly handled by the plugin and inserted into the tiddler. To avoid unpredictable results (and possibly fatal execution errors), this function should never be redefined or called from ''within'' your script code.//
''Accessing the ~TiddlyWiki DOM''
The plugin provides one pre-defined variable, 'place', that is passed in to your javascript code so that it can have direct access to the containing DOM element into which the tiddler output is currently being rendered.
Access to this DOM element allows you to create scripts that can:
* vary their actions based upon the specific location in which they are embedded
* access 'tiddler-relative' information (use findContainingTiddler(place))
* perform direct DOM manipulations (when returning wikified text is not enough)
<<<
!!!!!Examples
<<<
an "alert" message box:
{{{
<script>alert('InlineJavascriptPlugin: this is a demonstration message');</script>
}}}
<script>alert('InlineJavascriptPlugin: this is a demonstration message');</script>
dynamic output:
{{{
<script>return (new Date()).toString();</script>
}}}
<script>return (new Date()).toString();</script>
wikified dynamic output:
{{{
<script>return "link to current user: [["+config.options.txtUserName+"]]";</script>
}}}
<script>return "link to current user: [["+config.options.txtUserName+"]]";</script>
dynamic output using 'place' to get size information for current tiddler
{{{
<script>
if (!window.story) window.story=window;
var title=story.findContainingTiddler(place).id.substr(7);
return title+" is using "+store.getTiddlerText(title).length+" bytes";
</script>
}}}
<script>
if (!window.story) window.story=window;
var title=story.findContainingTiddler(place).id.substr(7);
return title+" is using "+store.getTiddlerText(title).length+" bytes";
</script>
creating an 'onclick' button/link that runs a script
{{{
<script label="click here">
if (!window.story) window.story=window;
alert("Hello World!\nlinktext='"+place.firstChild.data+"'\ntiddler='"+story.findContainingTiddler(place).id.substr(7)+"'");
</script>
}}}
<script label="click here">
if (!window.story) window.story=window;
alert("Hello World!\nlinktext='"+place.firstChild.data+"'\ntiddler='"+story.findContainingTiddler(place).id.substr(7)+"'");
</script>
loading a script from a source url
{{{
<script src="demo.js">return "loading demo.js..."</script>
<script label="click to execute demo() function">demo()</script>
}}}
where http://www.TiddlyTools.com/demo.js contains:
>function demo() { alert('this output is from demo(), defined in demo.js') }
>alert('InlineJavascriptPlugin: demo.js has been loaded');
<script src="demo.js">return "loading demo.js..."</script>
<script label="click to execute demo() function">demo()</script>
<<<
!!!!!Installation
<<<
import (or copy/paste) the following tiddlers into your document:
''InlineJavascriptPlugin'' (tagged with <<tag systemConfig>>)
<<<
!!!!!Revision History
<<<
''2006.01.05 [1.4.0]''
added support 'onclick' scripts. When label="..." param is present, a button/link is created using the indicated label text, and the script is only executed when the button/link is clicked. 'place' value is set to match the clicked button/link element.
''2005.12.13 [1.3.1]''
when catching eval error in IE, e.description contains the error text, instead of e.toString(). Fixed error reporting so IE shows the correct response text. Based on a suggestion by UdoBorkowski
''2005.11.09 [1.3.0]''
for 'inline' scripts (i.e., not scripts loaded with src="..."), automatically replace calls to 'document.write()' with 'place.innerHTML+=' so script output is directed into tiddler content
Based on a suggestion by BradleyMeck
''2005.11.08 [1.2.0]''
handle loading of javascript from an external URL via src="..." syntax
''2005.11.08 [1.1.0]''
pass 'place' param into scripts to provide direct DOM access
''2005.11.08 [1.0.0]''
initial release
<<<
!!!!!Credits
<<<
This feature was developed by EricShulman from [[ELS Design Studios|http:/www.elsdesign.com]]
<<<
!!!!!Code
***/
//{{{
version.extensions.inlineJavascript= {major: 1, minor: 4, revision: 0, date: new Date(2006,1,5)};
config.formatters.push( {
name: "inlineJavascript",
match: "\\<script",
lookahead: "\\<script(?: src=\\\"((?:.|\\n)*?)\\\")?(?: label=\\\"((?:.|\\n)*?)\\\")?\\>((?:.|\\n)*?)\\</script\\>",
handler: function(w) {
var lookaheadRegExp = new RegExp(this.lookahead,"mg");
lookaheadRegExp.lastIndex = w.matchStart;
var lookaheadMatch = lookaheadRegExp.exec(w.source)
if(lookaheadMatch && lookaheadMatch.index == w.matchStart) {
if (lookaheadMatch[1]) { // load a script library
// make script tag, set src, add to body to execute, then remove for cleanup
var script = document.createElement("script"); script.src = lookaheadMatch[1];
document.body.appendChild(script); document.body.removeChild(script);
}
if (lookaheadMatch[2] && lookaheadMatch[3]) { // create a link to an 'onclick' script
// add a link, define click handler, save code in link (pass 'place'), set link attributes
var link=createTiddlyElement(w.output,"a",null,"tiddlyLinkExisting",lookaheadMatch[2]);
link.onclick=function(){try{return(eval(this.code))}catch(e){alert(e.description?e.description:e.toString())}}
link.code="function _out(place){"+lookaheadMatch[3]+"};_out(this);"
link.setAttribute("href","javascript:;"); link.setAttribute("title",""); link.style.cursor="pointer";
}
else if (lookaheadMatch[3]) { // run inline script code
var code="function _out(place){"+lookaheadMatch[3]+"};_out(w.output);"
code=code.replace(/document.write\(/gi,'place.innerHTML+=(');
try { var out = eval(code); } catch(e) { out = e.description?e.description:e.toString(); }
if (out && out.length) wikify(out,w.output);
}
w.nextMatch = lookaheadMatch.index + lookaheadMatch[0].length;
}
}
} )
//}}}

An RAII class, used to manage a [[facade interface between layers|LayerSeparationInterface]].
The InstanceHandle is created by the service implementation and will automatically
* either register and open a ''C Language Interface'', using Lumiera's InterfaceSystem
* or load and open a LumieraPlugin, also by using the InterfaceSystem
* and optionally also create and manage a facade proxy to allow transparent access for client code
&rarr; see [[detailed description here|LayerSeparationInterfaces]]

This overarching topic is where the arrangement of our interface components meets considerations about interaction design.
The interface programming allows us to react on events and trigger behaviour, and it allows us to arrange building blocks within a layout framework. Beyond that, there needs to be some kind of coherency in the way matters are arranged -- this is the realm of conventions and guidelines. Yet in any more than trivial UI application, there is an intermediate and implicit level of understanding, where things just happen, which can not fully be derived from first principles. It is fine to have a convention to put the "OK" button right -- but how to we get at trimming a clip? How do we how we are to get at trimming a clip? if we work with the mouse? or the keyboard? or with a pen? or with a hardware controller we don't even know yet? We could deal with such on a case-by-case base (as the so called reasonable people do) or we could aim at an abstract intermediary space, with the ability to assimilate the practical situation yet to come.
;interface has a spatial quality
:the elements within an user interface are arranged in a way that parallels our experience when working in real world space. With the addition of some minor dose of //"hyper"// -- allowing for cross connections and shortcuts beyond spatial logic
;locality of work spaces
:but the arrangement of the interface interactions is not amorphous, rather it is segregated into cohesive clusters of closely interrelated actions. We move between these clusters of activity the same way as we move between several well confined rooms within a building.
;context and focus of activity
:most of what we could do //in theory,// is not relevant most of the time. But when the inner logic of what we're about to do coincides with the things at hand, then we feel enabled.
;shift of perspective
:and while we work, the focus moves along. Some things are closer, other things are remote and require us to move and re-orient and reshape our perspective, should we choose to turn towards them.
;the ability to arrange what is relevant
:we do the same stuff again and again, and this makes us observe and gradually understand matters. As we reveal the inner nature of what we're doing, we desire to arrange close at hand what belongs together, and to expunge the superficial and distracting.
&rarr; detailed [[analysis how commands are to be invoked|CommandInvocationAnalysis]]
!Foundation Concepts
The primary insight is //that we build upon a spatial metaphor// -- and thus we start out with defining various kinds of //locations.// We express interactions as //happening somewhere...//
;work site
:a distinct, coherent place where some ongoing work is done
:the WorkSite might move along with the work, but we also may leave it temporarily to visit some other work site
;the spot
:the [[Spot]] is //where we currently are// -- taken both in the sense of a location and a spotlight
:thus a spot is potentially at some work site, but it can be navigated to another one
;focus
:the concrete realisation of the spot within a given control system
;control system
:a practical technical realisation of an human-computer-interface, like keyboard input/navigation, mouse, pen, hardware controller, touch
;focus goal
:an order or instruction to bring something //into focus,// which also means to move the spot to the designated location.
;UI frame
:the overall interface is arranged into independent top-level segments of equal importance.
:practically speaking, we may have multiple top-level windows residing on multiple desktops...
;perspective
:a set of concrete configuration parameters defining the contents of one UI frame
:the perspective defines which views are opened and arranged at what position and within which docking panel
;focus path
:concrete coordinates to reach a specific work site
:the focus path specifies the UI frame (top-level window), the perspective, and then some canonical path to navigate down a hierarchy to reach the anchor point of the work site
;the spot locator
:navigating means to move the SpotLocator, in order to move the spot from work site to work site
:the spot locator is relocated by loading a new focus path leading to another [[work site|WorkSite]]
The concept of a //focus goal// has several ramifications: for one it implies that there is something akin the //"current cotrol system",// which also could be the //currently active control system(s).// Simply because focus, as the realisation of the abstract notion of the spot, is always tied to a control system able to implement it. And when we're able to define generic location coordinates and then //"move there",// with the help of the SpotLocator, we draw the conclusion that there must be a focus (implementation), somehow getting shifted towards that location. Like e.g. the desired entity to gain the keyboard focus. And, beyond that, the second thing we may conclude is that there need to be some degree of leeway in the way such a focus goal can be reached. Since the inner logic of control systems can be quite drastically different from each other, we are well advised to leave it to the actual control system //how actually to fulfil the focus goal.// To point out an obvious example: it is not a good idea to move the mouse pointer forcibly onto a screen element. Rather, we must use the established mechanisms of switching, scrolling and unfolding to bring the desired target element into the visible area, leaving the last step to the user, which must actively move the mouse onto the target. And we must give good visual clues as to what happened, and what we expect from the user (namely to direct her attention onto the element brought into focus).
!Building the framework
To create such a system is an ambitious goal for sure. We can not reach it in a single step, since it entails the formation of a whole intermediary layer, on top of the //usual UI mechanics,// yet below the concrete UI interactions. Especially, we'd need to clarify the meaning of //perspective,// we need to decide on the relation of top level frame, individual view, layout, focus and //current location within the UI.// On a second thought, building such a system implies we'll have to live with an intermediary state of evolution, where parts of the new framework are already in place without interfering with common conventional usage of the interface as-is.
!!!UI coordinates
Especially the focus navigation entails the use of some kind of ubiquitous [[coordinate system within the user interface|UICoord]]. In fact this is more of a topological navigation, since these coordinates describe the decisions and forks taken on navigation down the //focus path.//
* [optional] top-level Window (UI frame)
* [optional] Perspective
* Panel
* local path
** [optional] Group
** ~View-ID
** component.component.component...

//the top-level controller within the UI.//
In Lumiera, the structures of the model within the [[Session]] (the so called HighLevelModel) are mapped onto corresponding [[tangible UI entities|UI-Element]], which serve as a front-end to represent those entities towards the user. Within this model, there is a //conceptual root node// -- which logically corresponds to the session itself. This [[root element in model|ModelRootMO]] links together the actual top-level entities, which are the (multiple) timelines, together with the asset management and defaults and rules configuration within the session.
And the counterpart of this root element within the UI is the {{{InteractionDirector}}}, a top-level controller. As a controller, it responds to actions like opening a specific timeline, entering the asset management section, the request for help, and actions like saving, opening and closing of the session as a whole. Beyond that, the Interaction Director is the connection joint to that part of the UI which deals with global interaction state: this topic relates to questions about "the current element", "the focus", "where we are right now" (in what "location" or "room" within the UI) and also what tangible interface controller we're actually using (mouse, keyboard, graphics pen, hardware controller, touch screen).
Why do we need a connection joint between those parts?
Because issuing any actions on the model within the session -- i.e. any editing operation -- is like forming a sentence: we need to spell out //what we want to do// and we need to spell out the subject and the object of our activity. And any one of these can, and will in fact, be sometimes derived //from the context of the interaction.// Because, given the right context, it is almost clear what you want to do -- you just need to fill in that tiny little bit of information to actually make it happen. In Lumiera we strive at building a good UI, which is an UI well suited to this very human way of interacting with one's environment within a given context.
> to understand this, it is best &rarr; to [[look at some examples|CommandInvocationAnalysis]]
!Children
The InteractionDirector is part of the model, and thus we have to distinguish between model children and other UI components held as children.
;model
:anything related to //core concerns// and session global structures falls into that category
:* some parts might be related to ''Asset management''
:* some concerns might touch questions of ''configuration''
:* and interaction state is closely related to ''persistent UI state''
;interaction control
:anything related to forming and tracking of user interactions &rarr; InteractionControl
:* the SpotLocator is what is "moved" when the [[Spot]] of current activity moves
:* the FocusTracker is responsible for //detecting changes in current selection and focus.//
:* the [[Navigator]] is a special controller to handle moving the SpotLocator within the UI tree topology
:* the ViewLocator serves for any kind of high-level, abstracted access to [[component views|GuiComponentView]] and location resolution.
!Collaborations
* several global actions are exposed here, like opening the session and creating a new timeline.<br/>Such operations are typically bound into menu and action buttons, and in turn invoke [[commands|CommandHandling]] into the Session
* some further actions involve [[management of docking panels|GuiDockingPanel]], like e.g. instantiating a new [[timeline display|GuiTimelineView]] into a timeline pane.
* in fact it is the InteractionDirector's job to keep track of [[component views|GuiComponentView]], some of which
** can exist only once
** or can exist only once per top level window
** some might even be required once per window
** while others may be duplicated locally or globally
!!!cyclic dependencies
The InteractionDirector interconnects various aspects of UI management and thus can be expected to exhibit cyclic dependencies on several levels. Bootstraping the InteractionDirector is thus a tricky procedure and requires all participating services to operate demand-driven. In fact, these services represent some aspects of the same whole -- the UI. They are bounded by thematic considerations, not so much implementation concerns, and many of them rely on their siblings actually provide some essential part of their service. For example, the Navigator exposes the UI as topological tree structure, while the ViewLocator encapsulates the concrete access paths towards specific kinds of UI entities (views, panels). Obviously, the Navigator needs the ViewLocator to build its tree abstraction on top. But, thematically, //resolving a view// is part of accessing and / or creating some view, and thus the ViewLocator becomes indirectly dependent on the tree topology established by the Navigator.

A facility within the GUI to// track and manage one specific aspect of interaction state.//
In a more elaborate UI, as can be expected for such a task like editing, there are interactions beyond "point and shot". For a fluid and natural interaction it is vital to build and exploit an operation context, so to guide and specify the ongoing operations. Interaction events can not be treated in isolation, but rather in spatial and temporal clusters known as ''gestures''. A good example is the intention to trim or roll an edit. Here the user has some clips in mind, which happen to be located in immediate succession, and the kind of adjustment has to be determined from the way the user approaches the junction point. To deal with such an interaction pattern, we need to track a possible future interpretation of the user's actions as a hypothesis, to be confirmed and executed when all pieces fall into place.
An InteractionState is a global component, but often not addressed directly. To deal with context dependent activation, this tracking component attaches and taps into various information sources to observe some aspects of global state. Moreover, it is outfitted with a set of rules, leading to enablement of some [[command invocation trail|InvocationTrail]]. These enablements or disablements are forwarded to the actual trigger points, which are those UI elements to witness the completion of a gesture.
&rarr; CommandInvocationAnalysis
&rarr; InteractionControl
&rarr; GuiCommandBinding
&rarr; CommandUsage
! interaction state and the actual widgets
InteractionControl is conceived as an additional intermediary layer, distinct from the actual widgets. The whole idea is that we //do not want// intricate state managing logic to be scattered all over the concrete UI widget code -- doing so would defeat any higher level structuring and turn the UI code into highly tangled very technical implementation logic; ideally, UI code should mostly be specification, setup and wiring, yet void of procedural logic.
The actual widgets rely {{red{planned 4/2017}}} on the {{{CmdContext}}} to be notified when a specific command becomes executable &rarr; GuiCommandAccess.

Because we rely on strong decoupling and separation into self contained components, there is not much need for a common quasi-global namespace. Operations needing the cooperation of another subsystem will be delegated or even dispatched, consequently implementation code needs only the service acces points from "direct cooperation partner" subsystems. Hierarchical scopes besides classes are needed only when multiple subsystems share a set of common abstractions. Interface and Implementation use separate namespaces.
!common definitions
# surely there will be the need to use some macros (albeit code written in C++ can successfully avoid macros to some extent)
# there will be one global ''Application State'' representation (and some application services)
# we have some lib facilities, especially a common [[Time]] abstraction
For these we have one special (bilingual) header file __lumiera.h__, which places in its C++ part some declarations into __namespace lumiera__. These declarations should be pulled into the specific namespaces or (still better) into the implementation //on demand//. There is no nesting (we deliberately don't want an symbol appearing //automatically// in every part of the system).
!subsystem interface and facade
These large scale interfaces reside in special namespaces "~XXX_interface" (where XXX is the subsystem). The accompanning //definitions// depend on the implementation namespace and are placed in the top-level source folder of the corresponding subsystem.
&rarr; Example: [[Interfaces/Namespaces of the Session Subsystem(s)|InterfacesSession]]
!contract checks and test code
From experiences with other middle scale projects, I prefer having the test code in a separate tree (because test code easily doubles the number of source files). But of course it should be placed into the same namespace as the code being checked, or (better?) into a nested namespace "test". It is esp. desirable to have good coverage on the contracts of the subsystem interfaces and mayor components (while it is not always feasible or advisable to cover every implementation detail).
&rarr; see also [[testsuite documentation in the main wiki|index.html#TestSuite]]

* Subdir src/proc contains Interface within namespace proc_interface
* Subdir src/proc/mobject contains commonly used entities (namespace mobject)
** nested namespace controller
** nested namespace builder
** nested namespace session
* Subdir src/proc/engine (namespace engine) uses directly the (local) interface components StateProxy and ParamProvider; triggering of the render process is initiated by the controller and can be requested via the controller facade. Normally, the playback/render controller located in the backend will just use this interface and won't be aware of the build process at all.
[img[Example: Interfaces/Namespaces of the ~Session-Subsystems|uml/fig130053.png]]

//one specific way to prepare and issue a ~Proc-Layer-Command from the UI.//
The actual persistent operations on the session model are defined through DSL scripts acting on the session interface, and configured as a //command prototype.// Typically these need to be enriched with at least the actual subject to invoke this command on; many commands require additional parameters, e.g. some time or colour value. These actual invocation parameters need to be picked up from UI elements, sometimes even from the context of the triggering event. When all arguments are known, finally the command -- as identified by a command-ID -- can be issued on any bus terminal, i.e. on any [[tangible interface element|UI-Element]].
&rarr; CommandInvocationAnalysis
Thus an invocation trail represents one specific path leading to the invocation of a command. In the current state of the design, this is a concept; initially it was meant to exist as object, but this approach turned out to be unnecessarily complex. We can foresee that there will be the somewhat tricky situation, where a command is ''context-bound''. In those cases, we rely on the InteractionState helper, which is to track {{red{planned 4/2017}}} an enablement entry for each possible invocation trail. Basically this means that some commands need to be prepared and bound explicitly into some context (e.g. the tracks within a sequence), while enabling and parameter binding happens automatically, driven by interaction events.
&rarr; InteractionControl

The actual media data is rendered by [[individually scheduled render jobs|RenderJob]]. All these calculations together implement a [[stream of calculations|CalcStream]], as demanded and directed by the PlayProcess. During the preparation of playback, a ''node planning phase'' is performed, to arrange for [[dispatching|FrameDispatcher]] the individual calculations per frame. The goal of these //preparations//&nbsp; is to find out
* what channel(s) to pull
* what prerequisites to prepare
* what parameters to provide
The result of this planning phase is the {{{JobTicket}}}, a complete ''execution plan''.
This planning is uniform for each [[segment|Segmentation]] and treated for all channels together, resulting in a nested tree structure of sub job tickets, allocated and stored alongside with the processing nodes and wiring descriptors to form the segment's data and descriptor network. Job tickets are //higher order functions:// entering a concrete frame number and channel into a given job ticket will produce an actual job descriptor, which in itself is again a function, to be invoked through the scheduler when it's time to trigger the actual calculations.
!Structure of the Render Jobs created
To be more precise: in the general case, invoking the ~JobTicket with a given frame number will produce //multiple jobs// -- typically each frame rendering will require at least one further source media frame; and because Lumiera render jobs will //never block waiting on IO,// this source media access will be packaged as a separate [[resource retrieving job|ResourceJob]], to be treated specifically by the scheduler.
To support the generation of multiple dependent jobs, a ~JobTicket might refer to further ~JobTickets corresponding to the prerequisites. These prerequisite ~JobTickets are the result of a classical recursive descent call into the top level ProcNode, to perform the planning step. There is always an 1:1 relation between actual jobs generated and the corresponding tickets. More precisely, for each possible job to generate there is a suitable ''job closure'', representing exactly the context information necessary to get that job started. To stress that point: a ProcNode might be configured such as to perform a series of recursive invocations of other prerequisite nodes right away (especially when those recursive invocations correspond just to further CPU bound calculations) -- yet still there is only //one//&nbsp; ~JobTicket, because there will be only one Job to invoke the whole sequence. On the other hand, when there is a prerequisite requiring an operation to be scheduled separately, a corresponding separate {{{JobClosure}}} will be referred.

A major //business interface// &mdash; used by the layers for interfacing to each other; also to be invoked externally by scripts.
&rarr; [[overfiew and technical details|LayerSeparationInterfaces]]

Lumiera uses a 3-layered architecture. Separation between layers is crucial. Any communication between the layers regarding the normal operation of the application should //at least be initiated// through ''Layer abstraction interfaces'' (Facade Interfaces). This is a low-impact version of layering, because, aside from this triggering, direct cooperation of parts within the single Lumiera process is allowed, under the condition that is is implemented using additional abstractions (interfaces with implementation level granularity). We stick to the policy of //disallowing direct coupling of implementations located in different layers.//
[>img[Anatomy of a Layer Separation Interface|uml/fig132869.png]]
The goal is for the interface to remain fairly transparent for the client and to get an automatic lifecycle management.
To implement such a structure, each layer separation interface actually is comprised of several parts:
* an C Language Interface ("''CL Interface''") to be installed into the InterfaceSystem
* a ''facade interface'' defining the respective abstractions in terms of the client side impl. language (C or C++)
* a ''service implementation'' directly addressed by the implementation instantiated within the InterfaceSystem
* a ''facade proxy'' object on the client side, which usually is given inline alongside with the CL interface definition.
!opening and closing
Handling the lifecycle can be tricky, because client- and service-side need to carry out the opening and closing operations in sync and observing a specific call order to ensure calls get blocked already on the client side unless the whole interface compound is really up and running. To add to this complexity, plugins and built-in interfaces are handled differently regarding the question who is in charge of the lifecycle: interfaces are installed on the service side, whereas the loading of plugins is triggered from client side by requesting the plugin from the loader.
Anyway, interfaces are resources which best should be managed automatically. At least within the C++ part of the application we can ensure this by using the InstanceHandle template. This way the handling of plugins and interfaces can be unified and the opening and closing of the facade proxy happens automatically.
The general idea is, that each facade interface actually provides access to a specific service; there will always be a single implementation object somewhere, which can be thought of as acting as "the" service. This service-providing object will then contain the mentioned InstanceHandle; thus, its lifecycle becomes identical with the service lifecycle.
* when the service relies on a [[plugin|LumieraPlugin]], this service providing object (containing the InstanceHandle) needs to sit at the service accessing side (as the plugin may not yet be loaded and someone has to pull it up).
* otherwise, when the service is just an interface to an already loaded facility, the service providing object (containing the InstanceHandle) will live on the service providing side, not the client side. Then the ctor of the service providing object needs to be able to access an interface instance definition (&rarr; InterfaceSystem); the only difference to the plugin case is to create the InstanceHandle with a different ctor, passing the interface and the implementing instance.
!outline of the major interfaces
|!Layer|>|!Interface |
|GUI|GuiFacade|UI lifecycle &rarr; GuiStart|
|~|GuiNotificationFacade|status/error messages, asynchronous object status change notifications, trigger shutdown|
|~|DisplayFacade|pushing frames to a display/viewer|
|Proc|SessionFacade|session lifecycle|
|~|EditFacade|edit operations, object mutations|
|~|PlayerDummy|player mockup, maybe move to backend?|
|//Lumiera's major interfaces//|c

An implementation facility used by the session manager implementation to ensure a consistent lifecycle. A template-method-like skeleton of operations can be invoked by the session manager to go through the various lifecycle stages, while the actual implementation functionality is delegated to facilities within the session. The assumption is for the lifecycle advisor to be executed within a controlled environment, a single instance and single threaded.
---------------------
!Implementation notes
The goal on the implementation level is to get a consolidated view on the whole lifecycle.
Reading the source code should convey a complete picture about what is going on with respect to the session lifecycle.
!!entrance points
;close
: the shutdown sequence cleanly unwinds any contents and returns into //uninitialised state.//
;reset
: is comprised of two phases: shutdown and startup of an empty new session
: shutdown will be skipped automatically when uninitialised
: startup has to inject default session content
;load
: again, first an existing session is brought down cleanly
: the startup sequence behaves similarly to __reset__, but injects serialised content
Note that, while causing a short //freeze period,// __saving__ and __(re-)building__ aren't considered lifecycle operations
!!operation sequences
The ''pull up'' sequence performs basic initialisation of the session facilities and then, after the pImpl-switch, executes the various loading and startup phases. Any previous session switched away is assumed to unwind automatically, nothing is done especially to disconnect such an existing session, we simply require the session subsystem to be in a pristine state initially.
The ''shut down'' sequence does exactly that: halt processing and rendering, disconnect an existing session, if any, get back into initial state. It doesn't care for unwinding session contents, which is assumed to happen automatically when references to previous session contents go out of scope.

Opening and accessing media files on disk poses several problems, most of which belong to the domain of Lumiera's data backend. Here, we focus on the questions related to making media data available to the session and the render engine. Each media will be represented by an MediaAsset object, which indeed could be a compound object (in case of MultichannelMedia). Building this asset object thus includes getting information from the real file on disk. For delegating this to the backend, we use the following query interface:
* {{{queryFile(char* name)}}} requests accessing the file and yields some (opaque) handle when successful.
* {{{queryChannel(fHandle, int)}}} will then be issued in sequence with ascending index numbers, until it returns {{{NULL}}}.
* the returned struct (pointer) will provide the following information:
** some identifier which can be used to create a name for the corresponding media (channel) asset
** some identifier characterizing the access method (codec) needed to get at the media data. This should be rather a high level description of the media stream type, e.g. "H264"
** some (opaque) handle usable for accessing this specific stream. When the render engine later on pulls data for this channel, it will pass this handle down to the backend.
{{red{to be defined in more detail later...}}}
&rarr; see "~MediaAccessFacade" for a (preliminary) interface definitioin
&rarr; see "~MediaAccessMock" for a mock/test implementaion

Used to actually implement the various kinds of [[Placement]] of ~MObjects. ~LocatingPin is the root of a hierarchy of different kinds of placing, constraining and locating a Media Object. Basically, this is an instance of the ''state pattern'': The user sees one Placement object with value semantics, but when the properties of the Placement are changed, actually a ~LocatingPin object (or rather a chain of ~LocatingPins) is changed within the Placement. Subclasses of ~LocatingPin implement different placing/constraining behaviour:
* {{{FixedLocation}}} places a MObject to a fixed temporal position and track {{red{wrong! it is supposed to be an output designation}}}
* {{{RelativeLocation}}} is used to atach the MObject to some other anchor MObject
* //additional constraints, placement objectives, range restrictions, pattern rules will follow...//

All sorts of "things" to be placed and manipulated by the user in the session. This interface abstracts the details and just supposes
* the media object has a duration
* it is allways //placed// in some manner, i.e. it is allways accessed via a [[Placement]]
* {{red{and what else?}}}
&rarr; [[overview of the MObject hierarchy|MObjects]]

''The Problem of referring to an [[MObject]]'' stems from the object //as a concept// encompassing a wider scope then just the current implementation instance. If the object was just a runtime entity in memory, we could use a simple (language) reference or pointer. Actually, this isn't sufficient, as the object reference will pass LayerSeparationInterfaces, will be handed over to code not written in the same implementation language, will be included in an ''UNDO'' record for the UndoManager, and thus will need to be serialized and stored permanently within the SessionStorage.
Moreover [[MObject instances|MObject]] have a 2-level structure: the core object holds just the properties in a strict sense, i.e. the properties which the object //owns.// Any properties due to putting the object into a specific context, i.e. all relation properties are represented as [[Placement]] of the object. Thus, when viewed from the client side, a reference to a specific ~MObject //instance,// actually denotes a //specific//&nbsp; Placement of this object into the Session.
!Requirements
* just the reference allone is sufficient to access the placement //and//&nbsp; the core object.
* the reference needs to be valid even after internal restructuring in the object store.
* there must be a way to pass these references through serialisation/deserialisation
* we need a plain-C representation of the reference, which ideally should be incorporated into the complete implementation
* references should either be integrated into memory management, or at least it should be possible to detect dangling references
* it should be possible to make handling of the references completely transparent, if the implementation language supports doing so.
!Two models
For the implementation of the object references, as linked to the memory management in general, there seem to be the following approaches
* the handle is a simple table index. Consequently we'll need a garbage collection to deal with deleted objects. This model has several flavours
** using an generation ID to tag the index handle, keep an translation table on each GC, mapping old indices to new ones and raise an error when encountering an outdated handle
** use a specific datastructure allowing the handles to remain valid in case of cleanup/reorganisation. (Hashtable or similar)
** keep track of all index handles and rewrite them on GC
* handles are refcounting smart pointers. This solution is C++ specific, albeit more elegant and minimalistic. As all the main interfaces should be accessible via C bindings, we'd need to use a replacement mechanism on the LayerSeparationInterfaces, which could be mapped to or handled by an C++ smart-ptr. (We used a similar approach for the PlayerDummy design study)
Obviously, the second approach has quite some appeal &mdash; but, in order to use it, we'd have to mitigate its drawbacks: it bears the danger of creating a second separate code path for C language based clients, presumably receiving lesser care, maintenance and stress testing. The mentioned solution worked out earlier this year for the player mockup (1/2009) tries at least partially to integrate the C functionality, as the actual implementation is derived from the C struct used as handle, thus allowing to use the same pointers for both kinds of interface, and in turn by doing the de-allocation by a call through the C dtor function, which is installed as deleter function with the boost::smart_ptr used as base class of the handle. Conceptually, the interface on this handle is related to the actual implementation refered to by the handle (handle == smart_ptr) as if the latter was a subclass of the former. But on the level of the implementation, there is no inheritance relation between the handle and the referent, and especially this allows to define the handle's interface in terms of abstract interface types usable on the client side, while the referent's interface operates on the types of the service implementation. Thus, the drawback of using a C language interface is turned into the advantage of completely separating implementation and client.
!Implementation concept
Presumably, none of the both models is usable as-is; rather we try to reconstruct the viable properties of both, starting out with the more elegant second model. Thus, basically the ''reference is a smart-ptr'' referring to the core object. Additionally, it incorporates a ''systematic ID denoting the location of the placement''. This ID without the smart-ptr part is used for the C-implementation, making the full handle implementation a shortcut for an access sequence, which first querries the placement from the Session, followed by dereferencing the placement to get at the core object. Thus, the implementation builds upon another abstraction, the &rarr; PlacementRef, which in turn assumes for an index within the implementation of the [[session datastructure|SessionDataMem]] to track and retrieve the actual Placement.
[img[Structure of MObjectRef and PlacementRef|uml/fig136581.png]]
!using ~MObject references
~MObject references have a distinct lifecycle: usually, they are created //empty// (invalid ref, inactive state), followed by activating them by attachment to an existing placement within the session. Later on, the reference can be closed (detached, deactivated). Activation can be done either directly by a {{{Placement<MO>&}}}, or indirectly by any {{{Placement::ID}}} tag or even plain LUID denoting a placement known to the PlacementIndex embedded within the [[Session]]. Activation can fail, because the validity of the reference is checked. In this respect, they behave exactly like PlacementRef, and indeed are implemented on top of the latter. From this point onward, the referred ~MObject is kept alive by the reference &mdash; but note, this doesn't extend to the placement, which still may be modified or removed from the session without further notice. {{red{TODO investigate synchronisation guarantees or a locking mechanism}}} Thus, client code should never store direct references to the placement.
~MObject references have value semantics, i.e. they don't have an identity and can be copied freely. Dereferencing yields a direct (language) ref to the MObject, while the placement can be accessed by a separate function {{{getPlacement()}}}. Moreover, the MObjectRef instance provides a direct API to some of the most common query functions you could imagine to call on both the object and the placement (i.e. to find out about the start time, length, ....)

The ~MObjects Subsystem contains everything related to the [[Session]] and the various Media Objects placed within. It is complemented by the Asset Management (see &rarr; [[Asset]]). Examples for [[MObjects |MObject]](&rarr; def) being:
* audio/video clips
* [[effects and plugins|EffectHandling]]
* special facilities like mask and projector
* [[Automation]] sets
* labels and other (maybe functional) markup
This Design strives to achieve a StrongSeparation between the low-level Structures used to carry out the actual rendering and the high level Entities living in the session and being manipulated by the user. In this high level view, the Objects are grouped and located by [[Placements|Placement]], providing a flexible and open way to express different groupings, locations and ordering constraints between the Media Objects.
&rarr; EditingOperations
&rarr; PlacementHandling
&rarr; SessionOverview
[img[Classess related to the session|uml/fig128133.png]]

The HighLevelModel consists of MObjects, which are attached to one another through their [[Placement]]. While this is a generic scheme to arrange objects in a tree of [[scopes|PlacementScope]], some attachments are handled specifically and may trigger side-effects
{{red{drafted feature as of 6/2010}}}
* a [[binding|BindingMO]] attached to root is linked to a [[Timeline]]
* a [[Fork]] attached to root corresponds to a [[Sequence]]
&rarr; see ModelDependencies

Problem is: when removing an Asset, all corresponding MObjects need to disappear. This means, besides the obvious ~Ref-Link (MObject referring to an asset) we need backlinks or a sort of registry. And still worse: we need to remove the affected MObject from the object network in the session and rebuild the Fixture...
&rarr; for a general design discussion see [[Relation of Clip and Asset|RelationClipAsset]]
//Currently// Ichthyo considers the following approach:
* all references between MObjects and Assets are implemented as __refcounting__ boost::shared_ptr
* the opposite direction is also a __strong reference__, effectively keeping the clip-MO alive even if it is no longer in use in the session (note this is a cyclic dependency that needs to be actively broken on deletion). This design decision is based on logical considerations (&rarr; see "deletions, Model-2" [[here|RelationClipAsset]]). This back-link is implemented by a Placement which is stored internally within the asset::Clip, it is needed for clean deletion of Assets, for GUI search functions and for adding the Clip to the session (again after been removed, or multiple times as if cloned).
* MObjects and Assets implement an {{{unlink()}}} function releasing any internal links causing circular dependencies. It is always implemented such as to drop //optional// associations while retaining those associations mandatory for fulfilling the objects contract.
* Instead of a delete, we call this unlink() function and let the shared_ptr handle the actual deletion. Thus, even if the object is already unlinked, it is still valid and usable as long as some other entity holds a smart-ptr. An ongoing render process for example can still use a clip asset and the corresponding media asset linked as parent, but this media asset's link to the dependant clip has already been cleared (and the media is no longer registered with the AssetManager of course).
* so the back-link from dependant asset up to the parent asset is mandatory for the child asset to be functional, but preferably it should be {{{const}}} (only used for information retrieval)
* the whole hierarchy has to be crafted accordingly, but this isn't much of a limitation
* we don't use a registry, rather we model the real dependencies by individual dependency links. So a MediaAsset gets links to all Clips created from this Asset and by traversing this tree, we can handle the deletion
* after the deletion, the Fixture needs to be rebuilt.
* but any render processes still can have pointers to the Asset to be removed, and the shared_ptr will ensure, that the referred objects stay alive as long as needed.
{{red{let's see if this approach works...}}}

Contrary to the &rarr;[[Assets and MObjects|ManagementAssetRelation]], the usage pattern for [[render nodes|ProcNode]] is quite simple: All nodes are created together every time a new segment of the network is being build and will be used together until this segment is re-built, at which point they can be thrown away altogether. While it would be easy to handle the nodes automatically by smart-ptr (the creation is accessible only by use of the {{{NodeFactory}}} anyways), it //seems advisable to care for a bulk allocation/deallocation here.// The reason being not so much the amount of memory (which is expected to be moderate), but the fact the build process can be triggered repeatedly several times a second when tweaking the session, which could lead to fragmentation and memory pressure.
__10/2008__: the allocation mechanism can surely be improved later, but for now I am going for a simple implementation based on heap allocated objects owned by a vector or smart-ptrs. For each segment of the render nodes network, we have several families of objects, each of with will be maintained by a separate low-level memory manager (as said, for now implemented as vector of smart-ptrs). Together, they form an AllocationCluster; all objects contained in such a cluster will be destroyed together.

The Interface asset::Media is a //key abstraction// It ties together several concepts and enables to deal with them on the interfaces in a uniform manner. Besides, like every Asset kind, it belongs rather to the bookkeeping view: an asset::Media holds the specific properties and parametrisation of the media source it stands for. Regarding the __inward interface__ &mdash; as used from within the [[model|HighLevelModel]] or the [[render nodes|ProcNode]], it is irrelevant if any given asset::Media object stands for a complete media source, just a clip taken from this source or if a placeholder version of the real media source is used instead.
[img[Asset Classess|uml/fig130437.png]]
{{red{NOTE 3/2010:}}} Considering to change that significantly. Especially considering to collapse clip-asset and clip-MO into a single entity with multiple inheritance
&rarr; regarding MultichannelMedia (see the notes at bottom)
&rarr; see also LoadingMedia

The Proc-Layer is designed such as to avoid unnecessary assumptions regarding the properties of the media data and streams. Thus, for anything which is not completely generic, we rely on an abstract [[type description|StreamTypeDescriptor]], which provides a ''Facade'' to an actual library implementation. This way, the fundamental operations can be invoked, like allocating a buffer to hold media data.
In the context of Lumiera and especially in the Proc-Layer, __media implementation library__ means
* a subsystem which allows to work with media data of a specific kind
* such as to provide the minimal set of operations
** allocating a frame buffer
** describing the type of the data within such a buffer
* and this subsystem or external library has been integrated to be used through Lumiera by writing adaptation code for accessing these basic operations through the [[implementation facade interface|StreamTypeImplFacade]]
* such a link to an type implementation is registered and maintained by the [[stream type manager|STypeManager]]
!Problem of the implementation data types
Because we deliberately won't make any asumptions about the implementation library (besides the ones imposed indirectly by the facade interface), we can't integrate the data types of the library first class into the type system. All we can do is to use marker types and rely on the builder to have checked the compatibility of the actual data beforehand.
It would be possible to circumvent this problem by requiring all supported implementation libraries to be known at compile time, because then the actual media implementation type could be linked to a facade type by generic programming. Indeed, Lumiera follows this route with regards to the possible kinds of MObject or [[Asset]] &mdash; but to the contraty, for the problem in question here, being able to include support for a new media data type just by adding a plugin by far outweights the benefits of compile-time checked implementation type selection. So, as a consequence of this design decision we //note the possibility of the media file type discovery code to be misconfigured// and select the //wrong implementation library at runtime.// And thus the render engine needs to be prepared for the source reading node of any pipe to flounder completely, and protect the rest of the system accordingly

Of course: Cinelerra currently leaks memory and crashes regularilly. For the newly written code, besides retaining the same level of performance, a main goal is to use methods and techniques known to support the writing of quality code. So, besides the MultithreadConsiderations, a solid strategy for managing the ownership of allocated memory blocks is necessary right from start.
!Problems
# Memory management needs to work correct in a //fault tolerant environment//. That means that we need to be prepared to //handle on a non-local scale// some sorts of error conditions (without aborting the application). To be more precise: some error condition arises locally, which leads to a local abort and just the disabling/failing of some subsystem without affecting the application as a whole. This can happen on a regular base (e.g. rendering fails) and thus is __no excuse for leaking memory__
# Some (not all) parts of the core application are non-deterministic. That means, we can't tie the memory management to any assumptions on behalf of the execution path
!C++ solution
First of all -- this doesn't concern //every// allocation. It rather means there are certain //dangerous areas// which need to be identified. Anyhow, instead of carrying inherent complexities of the problem into the solution, we should rather look for common solution pattern(s) which help factoring out complexity.
For the case here in question this seems to be the __R__esource __A__llocation __I__s __I__nitialisation pattern (''RAII''). Which boils down to basically never using bare pointers when concerned with ownership. Client code allways gets to use a wrapper object, which cannot be obtained unless going through some well defined construction site. As an extension to the baisc RAII pattern, C++ allows us to build //smart wrapper objects//, thereby delegating any kind of de-registration or resource freeing automatically. This usage pattern doesn't necessarily imply (and in fact isn't limited to) just ref-counting.
!!usage scenarios
# __existence is being used__: Objects just live for being referred to in a object network. In this case, use refcounting smart-pointers for every ref. (note: problem with cyclic refs)
# __entity bound ownership__: Objects can be tied to some long living entity in the program, which holds the smart-pointer
#* if the existence of these ref-holding entity can be //guaranteed// (as if by contract), then the other users can build a object network with conventional pointers
#* otherwise, when the ref-holding entity //can disappear// in a regular program state, we need weak-refs and checking (because by our postulate the controlled resource needs to be destructed immediately, otherwise we would have the first case, existence == being used)
!!!dangerous uses
* the render nodes &rarr; [[detail analysis|ManagementRenderNodes]] {{red{TODO}}}
* the MObjects in the session &rarr; [[detail analysis|ManagementMObjects]] {{red{TODO}}}
* Asset - MObject relationship. &rarr; [[detail analysis|ManagementAssetRelation]] {{red{TODO}}}
!!!rather harmless
* Frames (buffers), because they belong to a given [[RenderProcess (=StateProxy)|StateProxy]] and are just passed in into the individual [[ProcNode]]s. This can be handled consistently with conventional methods.
* each StateProxy belongs to one top-level call to the [[Controller-Facade|Controller]]
* similar for the builder tools, which belong to a build process. Moreover, they are pooled and reused.
* the [[sequences|Sequence]] and the defined [[assets|Asset]] belong together to one [[Session]]. If the Session is closed, this means a internal shutdown of the whole ProcLayer, i.e. closing of all GUI representations and terminating all render processes. If these calles are implemented as blocking operations, we can assert that as long as any GUI representation or any render process is running, there is a valid session and model.
!using Factories
And, last but not least, doing large scale allocations is the job of the backend. Exceptions being long-lived objects, like the session or the sequences, which are created once and don't bear the danger of causing memory pressure. Generally speaking, client code shouldn't issue "new" and "delete" when it comes in handy. Questions of setup and lifecycle should allways be delegated, typically through the usage of some [[factory|Factories]], which might return the product conveniently wrapped into a RAII style handle. Memory allocation is crucial for performance, and needs to be adapted to the actual platform -- which is impossible unless abstracted and treated as a separate concern.

This category encompasses the various aspects of the way the application controls and manages its own behaviour. They are more related to the way the application behaves, as opposed to the way the edited data is structured and organised (which is the realm of [[structural assets|StructAsset]]) &rarr; {{red{Ticket #1156}}}
* StreamType &rarr; a type system for describing and relating media data streams
* ScaleGrid &rarr; to manage time scales and frame alignment
* {{{ErrorLog}}} &rarr; collect incident records
!accessing meta assets
It turns out that all meta assets follow a distinct usage pattern: //they aren't built as individual entities.// Either, they are introduced into the system as part of a larger scale configuration activity, or they are //derived from category.// The latter fits in with a prototype-like approach; initially, the individual entry just serves to keep track of a categorisation, while at some point, such a link into a describing category may evolve into a local differentiation of some settings.
Another distinct property of meta assets is to be just a generic front-end to some very specific data entry, which needs to be allocated and maintained and provided on demand. Consider for example configuration rules, which have both a textual and an AST representation and will be assembled and composed into an effective rule set, depending on usage scope. Another example would be the enormous amount of parameter data created by parameter automation in the session. While certainly the raw data needs to be stored and retrieved somehow, the purpose of the corresponding meta asset is to access and manipulate this data in a structured and specific fashion.
!!!self referential structure
These observation leads to a design relying on a self referential structure: each meta asset is a {{{meta::Descriptor}}}. In the most basic version -- as provided by the generic implementation by {{{asset::Meta}}}, this descriptor is just the link to another descriptor, which represents a category. Thus, meta assets are created or accessed by
* just an EntryID, which implicitly also establishes a type, the intent being "get me yet another of this kind"
* a descriptor and an EntryID, to get a new element with a more distinct characterisation.
!!!mutating meta assets
Meta assets are ''immutable'' -- but they can be //superseded.//
For each meta asset instance, initially a //builder// is created for setting up the properties; when done, the builder will "drop off" the new meta asset instance. The same procedure is used for augmenting or superseding an existing element.

Lumiera's Proc-Layer is built around //two interconnected models,// mediated by the [[Builder]]. Basically, the &rarr;[[Session]] is an external interface to the HighLevelModel, while the &rarr;RenderEngine operates the structures of the LowLevelModel.

Our design of the models (both [[high-level|HighLevelModel]] and [[low-level|LowLevelModel]]) relies partially on dependent objects being kept consistently in sync. Currently (2/2010), __ichthyo__'s assessment is to consider this topic not important and pervasive enough to justify building a dedicated solution, like e.g. a central tracking and registration service. An important point to consider with this assessment is the fact that the session implementation is deliberately kept single-threaded. While this simplifies reasoning, we also lack one central place to handle this issue, and thus care has to be taken to capture and treat all the relevant individual dependencies properly at the implementation level.
!known interdependencies
[>img[Fundamental object relations used in the session|uml/fig136453.png]]
* the session API relies on two kinds of facade like assets: [[Timeline]] and [[Sequence]], linked to the BindingMO and [[Fork]] ("track") objects within the model respectively.
* conceptually, the DefaultsManagement and the AssetManager count as being part of the [[global model scope|ModelRootMO]], but, due to their importance, these facilities are accessible through an singleton interface.
* currently as of 2/2010 the exact dependency of the automation calculation during the render process onto the automation definitions within the HighLevelModel remains to be specified.
!!Timelines and Sequences
While implemented as StructAsset, additionally we need to ensure every instance gets linked to the relevant parts of the model and registered with the session. Contrast this with other kinds of assets, which may just remain enlisted, but never actually used.
;the Session
:...is linked 1:1 with timelines and sequences. Registration and deregistration is directly tied to creation and destruction.
: __created__ &rArr; default timeline
: __destroy__ &rArr; discard all timelines, discard all sequences
;Timeline
:acts as facade and is implemented by an root-attached BindingMO. Can't exist in isolation.
: __created__ &rArr; create a dedicated new binding, either using an existing sequence, or a newly created empty sequence
: __destroy__ &rArr; remove binding, while the previously bound sequence remains in model.
;root-placed Binding
:while generally a Binding can just exist somewhere in the model, when attached to root, a Timeline will be created
: __created__ &rArr; an existing sequence might be given on creation, otherwise a default configured sequence is created
: __destroy__ &rArr; implemented by detaching from root (see below) prior to purging from the model.
: __attached__ to root &rArr; invoke Timeline creation
: __detached__ from root &rArr; will care to destroy the corresponding timeline
;Sequence
:is completely dependent on a root-scoped "track" (fork root), can optionally be bound, into one/multiple timelines/VirtualClip, or unbound
: __created__ &rArr; mandates specification of an track-MO, (necessarily) placed into root scope &mdash; {{red{TODO: what to do with non-root tracks?}}}
: __destroy__ &rArr; destroy any binding using this sequence, purge the corresponding track from model, if applicable, including all contents
: __querying__ &rArr; forwards to creating a root-placed track, unless the queried sequence exists already
;root-placed Track
:attachment of a track to root scope is detected magically and causes creation of a Sequence
: __attached__ to root &rArr; invoke sequence creation
: __detached__ from root &rArr; find and detach from corresponding sequence, which is then destroyed. //Note:// contents remain unaffected
: irrespective if the track exists, is empty, or gets purged entirely, only the connection to root scope counts; thus relocating a track by [[Placement]] might cause its scope with all nested contents to become a sequence of its own or become part of another sequence. As sequences aren't required to be bound into a timeline, they may be present in the model as invisible, passive container
!!Magic attachments
While generally the HighLevelModel allows all kinds of arrangements and attachments, certain connections are [[detected automatically|ScopeTrigger]] and may trigger special actions, like the creation of Timeline or Sequence façade objects as described above. The implementation of such [[magic attachments|MagicAttachment]] relies on the PlacementIndex.

When it comes to addressing and distinguishing object instances, there are two different models of treatment, and usually any class can be related to one of these: An object with ''value semantics'' is completely defined through this "value", and not distinguishable beyond that. Usually, value objects can be copied, handled and passed freely, without any ownership. To the contrary, an object with ''reference semantics'' has an unique identity, even if otherwise completely opaque. It is rather like a facility, "living" somewhere, often owned and managed by another object (or behaving special in some other way). Usually, client code deals with such objects through a reference token (which has value semantics). Care has to be taken with //mutable objects,// as any change might influence the object's identity. While this usually is acceptable for value objects, it is prohibited for objects with reference semantics. These are typically created by //factories// &mdash; and this fabrication is the only process to define the identity.
!Assets
Each [[Asset]] holds an identification tuple; the hash derived from this constant tuple is used as ~Asset-ID.
* the {{{Asset::Ident}}} tuple contains the following information
*# a __name-ID__, which is a human understandable but sanitised word
*# a tree-like classification of the asset's __category__, comprised of
*#* asset kind {{{{AUDIO, VIDEO, EFFECT, CODEC, STRUCT, META}}}}
*#* a path in a virtual classification tree. Some &raquo;folders&laquo; have magic meanings
*# an __origin-ID__ to denote the origin, authorship or organisation, acting like a namespace
*# a __version number__.
Of these, the tuple {{{(org,category,name)}}} defines the unique asset identity, and is hashed into the asset-ID. At most one asset with a given ident-tuple (including version) can exist in the whole system. Any higher version is supposed to be fully backwards compatible to all previous versions. Zero is reserved for internal purposes. {{red{1/10: shouldn't we extend the version to (major,minor), to match the plug-in versioning?}}} Thus, the version can be incremented without changing the identity, but the system won't allow co-existence of multiple versions.
* Assets are ''equality'' comparable (even ''ordered''), based on their //identity// &mdash; sans version.
* Categories are ''equality'' comparable value objects
!~MObjects
As of 1/10, MObjects are mostly placeholders or dummies, because the actual SessionLogic has still to be defined in detail.
The following properties can be considered as settled:
* reference semantics
* non-copyable, created by MObjectFactory, managed automatically
* each ~MObject is associated n:1 to an asset.
* besides that, it has an opaque instance identity, which is never made explicit.
* this identity springs from the way the object is created and manipulated. It is //persistent.//
* because of the ~MObject-identity's nature, there is //no point in comparing ~MObjects.//
* ~MObjects are always attached to the session by a placement, which creates kind-of an //instance//
* thus, because placements act as a subdivision of ~MObject identification, in practice always placements will be compared.
!Placements
[[Placements|Placement]] are somewhat special, as they mix value and reference semantics. First off, they are configuration values, copyable and smart-pointers, referring to a primary subject (clip, effect, fork, label, binding,....). But, //by adding a placement to the session,// we create an unique instance-identity. This is implemented by copying the placement into the internal session store and thereby creating a new hash-ID, which is then registered within the PlacementIndex. Thus, a ''placement into the model'' has a distict identity.
* Placements are ''equality'' comparable, based on this instance identity (hash-ID)
* besides, there is an equivalence relation regarding the "placement specification" contained in the [[locating pins|LocatingPin]] of the Placement.
** they can be compared for ''equivalent definition'': the contained definitions are the same and in the same order
** alternatively, they can be checked for ''effective equivalence'': both placements to be compared resolve to the same position
note: the placement equality relation carries over to ~PlacementRef and ~MObjectRef
!Commands
{{red{WIP}}} For now, commands are denoted by an unique, human-readable ID, which is hard-coded in the source. We might add an LUID and a version numbering scheme later on.
Commands are used as ''prototype object'' &mdash; thus we face special challenges regarding the identity, which haven't yet been addressed.
!References and Handles
These are used as token for dealing with other objects and have no identity of their own. PlacementRef tokens embody a copy of the referred placement's hash-ID. MObjectRef handles are built on top of the former, additionally holding a smart-ptr to the primary subject.
* these reference handles simply reflect the equality relation on placements, by virtue of comparing the hash-ID.
* besides, we could build upon the placement locating chain equivalence relations to define a semantic equivalence on ~MObjectRefs

Any point where output possibly might be produced. Model port entities are located within the [[Fixture]] &mdash; model port as a concept spans the high-level and low-level view. A model port can be associated both to a pipe in the HighLevelModel but at the same time denotes a set of corresponding [[exit nodes|ExitNode]] within the [[segments|Segmentation]] of the render nodes network.
A model port is rather derived than configured; it emerges when a pipe [[claims|WiringClaim]] an output destination, while some other entity at the same time actually //uses this designation as a target,// either directly or indirectly. This match of provision and usage is detected during the build process and produces an entry in the fixture's model port table. These model ports in the fixture are keyed by ~Pipe-ID, thus each model port has an associated StreamType.
Model ports are the effective, resulting outputs of each timeline; additional ports result from [[connecting a viewer|ViewConnection]]. Any render or display process happens at a model port.
!formal specification
Model port is a //conceptual entity,// denoting the possibility to pull generated data of a distinct (stream)type from a specific bus within the model -- any possible output produced or provided by Lumiera is bound to appear at a model port. The namespace of model ports is global, each being associated with a ~Pipe-ID.
Model ports are represented by small non-copyable descriptor objects with distinct identity, which are owned and managed by the [[Fixture]]. Clients are mandated to resolve a model port on each usage, as configuration changes within the model might cause ports to appear and decease. To stress this usage pattern, actually {{{ModelPort}}} instances are small copyable value objects (smart handles), which can be used to access data within an opaque registry. Each model port belongs to a specific Timeline and is aware of this association, as is the timeline, allowing to get the collection of all ports of a given timeline. Besides, within the Fixture each model port refers to a specific [[segmentation of the time axis|Segmentation]] (relevant for this special timeline actually). Thus, with the help of this segmentation, a model port can yield an ExitNode to pull frames for a given time.

Model ports are conceptual entities, denoting the points where output might possibly be produced &rarr; see [[definition|ModelPort]].
But there is an actual representation, a collection of small descriptor objects managed by the Fixture and organised within the model port table datastructure. Because model ports are discovered during the build process, we need the ability to (re)build this table dynamically, finally swapping in the modified configuration with a transactional switch. Only the builder is allowed to perform such mutations, while for client code the model ports are immutable.
!supported operations
* get the model port by ~Pipe-ID
* a collection of model ports per timeline
* sub-grouping by media kind, possibly even by StreamType
* possibility to enumerate model ports in distinct //order,// defined within the timeline
* with the additional possibility to filter by media kind or suitable stream type.
* a way to express the fact of //not having a model port.//
!!!mutating and rebuilding
Model ports are added once and never changed. The corresponding timeline and pipe is known at setup time, but the overall number of model ports is determined only as a result of completing the build process. At that point, the newly built configuration is swapped in transactionally to become the current configuration.
!Implementation considerations
The transactional switch creates a clear partitioning in the lifespan of the model port table. //Before// that point, entries are just added, but not accessed in any way. //After// that point, no further mutation occurs, but lookup is frequent and happens in a variety of different configurations and transient orderings.
This observation leads to the idea of using //model port references// as frontend to provide all kinds of access, searching and reordering. These encapsulate the actual access by silently assuming reference to "the" global current model port configuration. This way the actual model port descriptors could be bulk allocated in a similar manner as the processing nodes and wiring descriptors. Access to stale model ports could be detected by the port references, allowing also for a {{{bool}}} checkable "has no port" information.
A model port registry, maintained by the builder, is responsible for storing the discovered model ports within a model port table, which is then swapped in after completing the build process. The {{{builder::ModelPortRegistry}}} acts as management interface, while client code accesses just the {{{ModelPort}}} frontend. A link to the actual registry instance is hooked into that frontend when bringing up the builder subsystem.

A special kind of MObject, serving as a marker or entry point at the root of the HighLevelModel. As any ~MObject, it is attached to the model by a [[Placement]]. And in this special case, this placement froms the ''root scope'' of the model, thus containing any other PlacementScope (e.g. forks, clips with effects,...)
This special ''session root object'' provides a link between the model part and the &raquo;bookkeeping&laquo; part of the session, i.e. the [[assets|Asset]]. It is created and maintained by the session (implementation level) &mdash; allowing to store and load the asset definitions as contents of the model root element. Beyond that, this //root scope representation// serves another, closely related purpose: conceptually, structures of the HighLevelModel are mapped into the UI, by virtue of the [[diff framework|TreeDiffModel]]. Corresponding to each relevant entity in the model, there is a widget or a controller in the UI to serve as [[tangible front-end|UI-Element]] to expose this entity. And in accordance to this regime, the session root object is mapped onto the [[InteractionDirector|UI-InteractionDirector]] (top-level controller), which, in conjunction with {{{UiManager}}} and UI-Bus, forms the leading structure of the UI-Layer.
__Note__: nothing within the PlacementIndex requires the root object to be of a specific type; the index just assumes a {{{Placement<MObject>}}} (or subclass) to exist as root element. And indeed, for several unit tests we create an Index mock with a tree of dummy ~MObjects and temporarily shaddow the "real" PlacementIndex by this mock (&rarr; see SessionServices for the API allowing to access this //mock index//- functionality)

Based on practical experiences, Ichthyo tends to consider Multichannel Media as the base case, while counting media files providing just one single media stream as exotic corner cases. This may seem counter intuitive at first sight; you should think of it as an attempt to avoid right from start some of the common shortcomings found in many video editors, especially
* having to deal with keeping a "link" between audio and video clips
* silly limitations on the supported audio setups (e.g. "sound is mono, stereo or Dolby-5.1")
* unnecessary complexity when dealing with more realistic setups, esp. when working on dialogue scenes
* inability to edit stereoscopic (3D) video in a natural fashion
!Compound Media
Basically, each [[media asset|MediaAsset]] is considered to be a compound of several elementary media (tracks), possibly of various different media kinds. Adding support for placeholders (''proxy clips'') at some point in future will add still more complexity (because then there will be even dependencies between some of these elementary media). To handle, edit and render compound media, we need to impose some structural limitations. But anyhow, we try to configure as much as possible already at the "asset level" and make the rest of the proc layer behave just according to the configuration given with each asset.
{{red{Note 1/2015}}}: various details regarding the model representation of multichannel media aren't fully settled yet. There is a placeholder in the source, which can be considered more or less obsolete
!!Handling within the Model
* from a Media asset, we can get a [[Processing Pattern (ProcPatt)|ProcPatt]] describing how to build a render pipeline for this media
* we can create a Clip (MObject) from each Media, which will be linked back to the media asset internally.
* media can be compound internally, but will be handled as a single unit as long as possible. Even for compound media, we get a single clip.
* within the assets, we get a distinct ChannelConfig asset to represent the structural assembly of compound media based on elementary parts.
* sometimes its necessary to split and join the individual channels in processing, for technical reasons. Clearly the goal is to hide this kind of accidental complexity and treat it as an implementation detail. At HighLevelModel scope, conceptually there is just one "stream" for each distinct kind of media, and thus there is only one [[Pipe]]. But note, compound media can be structured beyond having several channels. The typical clip is a compound of image and sound. While still handled as one unit, this separation can't be concealed entirely; some effects can be applied to sound or image solely, and routing needs to separate sound and image at least when reaching the [[global pipes|GlobalPipe]] within the timeline.
* the Builder gets at the ProcPatt (descriptor) of the underlying media for each clip and uses this description as a template to build the render pipeline. That is, the ProcPatt specifies the codec asset and maybe some additional effect assets (deinterlace, scale) necessary for feeding media data corresponding to this clip/media into the render nodes network.
!!Handling within the engine
While initially it seems intuitive just to break down everything into elementary stream processing within the engine, unfortunately a more in-depth analysis reveals that this approach isn't viable. There are some crucial kinds of media processing, which absolutely require having //all channels available at the actual processing step.// Notable examples being sound panning, reverb and compressors. Same holds true for processing of 3D (stereoscopic) images. In some cases we can get away with replicating identical processor nodes over the multiple channels and applying the same parameters to all of them. The decision which approach to take here is a tricky one, and requires much in-depth knowledge about the material to be processed -- typically the quality and the image or sound fidelity depends on these kind of minute distinctions. Many -- otherwise fine -- existing software falls short on this domain. Making such fine points accessible through [[processing rules|Rules]] is one of the core goals of the Lumiera project.
As an immediate consequence of not being able to reduce processing to elementary stream processing, we need to introduce a [[media type system|StreamType]], allowing us to reason and translate between the conceptual unit at the model level, the compound of individual processing chains in the builder and scheduling level, and the still necessary individual render jobs to produce the actual data for each channel stream. Moreover it is clear that the channel configuration needs to be flexible, and not automatically bound to the existence of a single media with that configuration. And last but not least, through this approach, we also enable handling of nested sequences as virtual clips with virtual (multichannel) media.
&rArr; conclusions
* either the ~ClipMO referres directly to a ChannelConfig asset &mdash; or in case of the VirtualClip a BindingMO takes this role.
* as the BindingMO is also used to implement the top-level timelines, the treatment of global and local pipes is united
* every pipe (bus) should be able to carry multiple channels, //but with the limitation to only a single media StreamType//
* this "multichannel-of-same-kind" capability carries over to all entities within the build process, including ModelPort entries and even the OutputSlot elements
* in playback / rendering, within each "Feed" (e.g. for image and sound) we get [[calculation streams|CalcStream]] matching the individual channels
* thus, only as late as when allocating / "opening" an OutputSlot for actual rendering, we get multiple handles for plain single channels.
* the PlayProcess serves to join both views, providing a single PlayController front-end, while [[dispatching|FrameDispatcher]] to single channel processing.

//message on the UI-Bus to cause changes on the targeted UI-Element.//
The UI-Bus offers a dedicated API to direct ~MutationMessages towards {{{Tangible}}} elements, as designated by the given ID. Actually, such messages serve as //capsule to transport a [[diff-sequence|TreeDiffModel]]// -- since a diff sequence as such is always concrete and tied to a specific context, we can not represent it directly as an abstract type at interface level. The receiver of a diff sequence must offer the ability to be reshaped through diff messages, which is expressed through the interface {{{DiffMutable}}}.
In the case at hand, the basic building block of the Lumiera UI, the {{{Tangible}}} offers this interface and thus the ability to construct a concrete TreeMutator, which in turn is bound to the internals of the actual UI-Element in question. Together this allows for a generic implementation of MutationMessage handling, where the designated UI-Element is reshaped by applying a concrete diff sequence embedded in the message with the help of a {{{DiffApplicator<DiffMutable>}}}, based on the TreeMutator exposed.

//Service to navigate through the UI as generic structure.//
The Navigator is a component maintained by the InteractionDirector, and the actual implementation is backed by several facilities of the GuiTopLevel. It serves as foundation to treat the UI as a topological network of abstracted locations, represented as [[UI-Coordinates|UICoord]]. This design, together with the UI-Bus helps to reduce coupling within the UI implementation, since it enables to //get somewhere// and reach //some place// -- without the necessity to rely on concrete widget implementation structure.
!The problem of abstraction
This goal initially poses some challenge, since it aims beyond the conventional notion of UI programming, where it is sufficient just to wire some widgets and manipulate the receiver of an event notification. The usual UI widgets are just not meant to be treated in a more systematic, generic way -- and indeed, in most cases and for most purposes it is not a good idea to approach UI programming in a to much schematic way. User interfaces need to be tangible, something concrete, with lots of specifics and local traits. Yet this very nature of UI programming tends to turn some //cross-cutting concerns// into serious liabilities. So the deliberate ''decision to introduce a Navigator'' in avoidance of these future burdens and liabilities is a decision of priorities when it comes to shaping the Lumiera UI.
Which leaves us with the quest of mapping a generic location scheme onto a load of implementation defined structures not exposing any kind of genericness, and to accomplish this within an environment lacking meta information and support for self discovery beyond the most basic abstraction level. As a first step towards bridging this gap, we'll have to identify the actual //command-and-query operations// required to treat UI elements as a topological structure.
!!!Analysis of expected functionality
In order to build a navigation facility, we need to...
* follow a path
** which means to constitute a location
** and to discover child nodes at that location
* and we might want to extend (maybe also prune) the collection of children
!!!Use cases
In the current situation ({{red{10/2017}}}), before engaging into the actual implementation, we're able to identify two distinct use cases
;View [[specification|GuiComponentView]]
:locate a view based on a preconfigured placement
:* either to allocate a new view instance
:* or to get //just some instance// of a view identified by type
;WorkSite navication
:move the Spot to some other place in the UI known by its [[UI-Coordinates|UICoord]]
!!!{{red{Update 2/2018:}}} changed responsibilities
Elaboration on the topic of »View Allocation« caused some minor architectural rearrangements.
* Navigator became a pure information service (read-only), working on an abstracted view of the UI through [[UI coordinates|UICoord]]
* the ViewLocator became service point for any high-level access to GuiComponentView elements
!!!Requirements clarified
From these use cases we conclude that the actual requirements for a Navigator component are less than one might expect.
In fact it is sufficient to keep //the actual element// entirely opaque, so the Navigator works on [[UI coordinates|UICoord]] solely. The result -- some other UI coordinates -- can then be used to accomplish some tasks implemented elsewhere, like allocating a new view or actually moving [[the Spot|Spot]] (&rarr; InteractionControl)
!Challenges of the implementation
Some tricky problems remain to be solved, though: since the Navigator works on UI coordinates, the fundamental problem remains how to acquire the initial coordinates to start navigation. This is a problem of //reverse lookup:// given a concrete element of the UI, find it's UI coordinates. While we should note that it might not be necessary to "discover" coordinates, because in fact we may know them already -- either the element has to store them (on creation), or some lookup index table could be maintained to serve the same purpose. The actual access to low-level UI entities generates a host of further tecnicalities, which we attempt to stash away into a different low-level service, the [[gui:ctrl::ElmAccessDir|UILowLevelAccess]].
Moreover, the design of coordinate matching and resolving incurs a structure similar to [[render job planning|FrameDispatcher]] -- and the corresponding design problems remain to be solved in a satisfactory way &rarr; [[some notes...|AboutMonads]]

Various aspects of the individual [[render node|ProcNode]] are subject to configuration and may influence the output quality or the behaviour of the render process.
* the selection //what// actual implementation (plugin) to used for a formally defined &raquo;[[Effect|EffectHandling]]&laquo;
* the intermediary/common StreamType to use within a [[Pipe]]
* the render technology (CPU, hardware accelerated {{red{&rarr; Future}}})
* the ScheduleStrategy (possibly subdividing the calculation of a single frame)
* if this node becomes a possible CachePoint or DataMigrationPoint in RenderFarm mode
* details of picking a suitable [[operation mode|RenderImplDetails]] of the node (e.g. utilitsing "in-place" calculation)

~NodeCreatorTool is a [[visiting tool|VisitorUse]] used as second step in the [[Builder]]. Starting out from a [[Fixture]], the builder first [[divides the Timeline into segments|SegmentationTool]] and then processes each segment with the ~NodeCreatorTool to build a render nodes network (Render Engine) for this part of the timeline. While visiting individual Objects and Placements, the ~NodeCreatorTool creates and wires the necessary [[nodes|ProcNode]]

!Problem of Frame identification
!Problem of Node numbering
In the most general case the render network may be just a DAG (not just a tree). Especially, multiple exit points may lead down to the same node, and following each of this possible paths the node may be at a different depth on each. This rules out a simple counter starting from the exit level, leaving us with the possibility of either employing a rather convoluted addressing scheme or using arbitrary ID numbers.{{red{...which is what we do for now}}}

The [[nodes|ProcNode]] are wired to form a "Directed Acyclic Graph"; each node knows its predecessor(s), but not its successor(s). The RenderProcess is organized according to the ''pull principle'', thus we find an operation {{{pull()}}} at the core of this process. Meaning that there isn't a central entity to invoke nodes consecutively. Rather, the nodes themselves contain the detailed knowledge regarding prerequisites, so the calculation plan is worked out recursively. Yet still there are some prerequisite resources to be made available for any calculation to happen. So the actual calculation is broken down into atomic chunks of work, resulting in a 2-phase invocation whenever "pulling" a node. For this to work, we need the nodes to adhere to a specific protocol:
;planning phase
:when a node invocation is foreseeable to be required for getting a specific frame for a specific nominal and actual time, the engine has to find out the actual operations to happen
:# the planning is initiated by issuing an "get me output" request, finally resulting in a JobTicket
:# recursively, the node propagates "get me output" requests for its prerequisites
:# after retrieving the planning information for these prerequisites, the node encodes specifics of the actual invocation situation into a closure called StateAdapter <br/>{{red{TODO: why not just labeling this &raquo;~StateClosure&laquo;?}}}
:# finally, all this information is packaged into a JobTicket representing the planning results.
;pull phase
:now the actual node invocation is embedded within a job, activated through the scheduler to deliver //just in time.//
:# Node is pulled, with a StateProxy object as parameter (encapsulating BufferProvider for access to the required frames or buffers)
:# Node may now retrieve current parameter values, using the state accessible via the StateProxy
:# to prepare for the actual {{{process()}}} call, the node now has to retrieve the input prerequisites
:#* when the planning phase determined availability from the cache, then just these cached buffer(s) are now retrieved, dereferencing a BuffHandle
:#* alternatively the planning might have arranged for some other kind of input to be provided through a prerequisite Job. Again, the corresponding BuffHandle can now be dereferenced
:#* Nodes may be planned to have a nested structure, thus directly invoking {{{pull()}}} call(s) to prerequisite nodes without further scheduling
:# when input is ready prior to the {{{process()}}} call, output buffers will be allocated by locking the output [[buffer handles|BuffHandle]] prepared during the planning phase
:# since all buffers and prerequistes are available, the Node may now prepare a frame pointer array and finally invoke the external {{{process()}}} to kick off the actual calculations
:# finally, when the {{{pull()}}} call returns, "parent" state originating the pull holds onto the buffers containing the calculated output result.
{{red{WIP as of 9/11 -- many details here are still to be worked out and might change as we go}}}
{{red{Update 8/13 -- work on this part of the code base has stalled, but now the plain is to get back to this topic when coding down from the Player to the Engine interface and from there to the NodeInvocation. The design as outlined above was mostly coded in 2011, but never really tested or finished; you can expect some reworkings and simplifications, but basically this design looks OK}}}
some points to note:
* the WiringDescriptor is {{{const}}} and precalculated while building (remember another thread may call in parallel)
* when a node is "inplace-capable", input and output buffer may actually point to the same location
* but there is no guarantee for this to happen, because the cache may be involved (and we can't overwrite the contents of a cache frame)
* nodes in general may require N inputs and M output frames, which are expected to be processed in a single call
* some of the technical details of buffer management are encapsulated within the BufferTable of each invocation
&rarr; the [["mechanics" of the render process|RenderMechanics]]
&rarr; more fine grained [[implementation details|RenderImplDetails]]

The calculations for rendering and playback are designed with a base case in mind: calculating a linear sequence of frames consecutive in time.
But there are several important modes of playback, which violate that assumption...
* jump-to / skip
* looping
* pause
* reversed direction
* changed speed
** slow-motion
** fast-forward/backward shuffling
* scrubbing
* free-wheeling (as fast as possible)
!search for a suitable implementation approach {{red{WIP 1/2013}}}
The crucial point seems to be when we're picking a starting point for the planning related to a new frame. &rarr; {{{PlanningStepGenerator}}}
Closely related is the question when and how to terminate a planning chunk, what to dispose as a continuation, and when to cease planning altogether.
!requirement analysis
These non linear playback modes do pose some specific challenges on the overall control structure distributed over the various collaborators within the play and render subsystem.The following description treats each of the special modes within its relations to this engine context
;jumping
:creates a discontinuity in //nominal time,// while the progress of real wall clock time deadlines remains unaffected
:we need to distinguish two cases
:* a //pre planned jump// can be worked into the frame dispatch just like normal progression. It doesn't create any additional challenge on timely delivery
:* to the contrary, a //spontaneous re-adjustment of playback position// deprives the engine of any delivery headroom, forcing us to catch up anew.<br/>&rarr; we should introduce a configurable slippage offset, to be added on the real time deadlines in this case, to give the engine some head start
:since each skip might create an output discontinuity, the de-clicking facility in the output sink should be notified explicitly (implying that we need an API, reachable from within the JobClosure)
;looping
:looped playback is implemented as repeated skip at the loop boundary; as such, this case always counts as pre planned jump.
;pausing
:paused playback represents a special state of the engine, where we expect playback to be able to commence right away, with minimal latency
:&rarr;we need to take several coordinated measures to make this happen
:* when going to paused state, previously scheduled jobs aren't cancelled, rather rescheduled to background rendering, but in a way which effectively pins the first frames within cache
:* additionally, the OutputSlot needs to provide a special mode where output is //frozen// and any delivery is silently blocked. The reason is, we can't cancel already triggered output jobs
:* on return to normal playback, we need to ensure that the availability of cached results will reliably prevent superfluous prerequisite jobs from being scheduled at all &rarr; we need conditional prerequisites
:The availability of such a pausing mechanism has several ramifications. We could conceive an auto-paused mode, switching to playback after sufficient background pre rendering to ensure the necessary scheduling headroom. Since the link to wall clock time and thus the real time deadlines are established the moment actual playback starts, we might even transition through auto paused mode whenever playback starts from scratch into some play mode.
;single stepping
:this can be seen as application of paused mode: we'd schedule a single play-out job, as if resuming from paused state, but we re-enter paused state immediately
;reversed play direction
:while basically trivial to implement, the challenge lies in possible naive implementation decisions assuming monotonic ascending frame times. Additionally, media decoders might need some hinting...
:however, reversed (and speed adjusted) sound playback is surprisingly tricky -- even the most simplistic solution foces us to insert an effect processor into the calculation path.
;speed variations
:the relation between nominal time and real wall clock time needs to include a //speed factor.//
;fast cueing
:the purpose of cuing is to skip through a large amount of material to spot some specific parts. For this to work, the presented material needs to be still recognisable in some way. Typically this is done by presenting small continuous chunks of material interleaved with regular skips. For editing purposes, this method is often perceived as lacking, especially by experienced editors. The former, mechanical editing systems to the contrary had the ability to run with actually increased frame rate, without skipping any material.
:&rarr; for one, this is a very specific configuration of the loop play mode.
:&rarr; it is desirable to improve the editor's working experience here. We might consider actually increasing the frame rate, given the increased availability of high-framerate capable displays. Another approach would be to apply some kind of granular synthesis, dissolving several consecutive segments of material. The latter would prompt to include a specific buffering and processing device not present in the render path for normal playback. Since we do dedicated scheduling for each playback mode, we're indeed in a position to implement such facilities.
;scrubbing
:the actual scrubbing facility is an interactive device, typically even involving some kind of hardware control interface. But the engine needs to support scrubbing, which translates into chasing a playback target, which is re-adjusted regularly. The implementation facilities discussed thus far are sufficient to implement this feature, combined with the ability of life changes to the playback mode.
;free-wheeling
:at first sight, this feature is trivial to implement. All it takes is to ignore any real time scheduling targets, so it boils down to including a flag into the [[timing descriptor|Timings]]. But there is a catch. Since free-wheeling only makes sense for writing to a file like sink, actually the engine might be overrunning the consumer. In the end, we get to choose between the following alternatives: do we allow the output jobs to block in that case, or do we want to implement some kind of flow regulation?
!essential implementation level features
Drawing from this requirement analysis, we might identify some mandatory implementation elements necessary to incorporate these playback modes into the player and engine subsystem.
;for the __job planning stage__:
:we need a way to interact with a planning strategy, included when constituting the CalcStream
:* ability for planned discontinuities in the nominal time of the "next frame"
:* ability for placing such discontinuities repeatedly, as for looped playback
:* allow for interleaved skips and processed chunks, possibly with increased speed
:* ability to inject meta jobs under specific circumstances
:* placing callbacks into these meta jobs, e.g. for auto-paused mode
;for the __timings__:
:we need flexibility when establishing the deadlines
:* allow for an added offset when re-establishing the link between nominal and real time on replanning and superseding of planned jobs
:* flexible link between nominal and real time, allowing for reversed playback and changed speed
:* configurable slippage offset
;for the __play controller__:
:basically all changes regarding non linear playback modes are initiated and controlled here
:* a paused state
:* re-entering playback by callback
:* re-entering paused state by callback
:* a link to the existing feeds and calculation streams for superseding the current planning
:* use a strategy for fast-cueing (interleaved skips, increased speed, higher framerate, change model port on-the-fly to use a preconfigured granulator device)
;for the __scheduler interface__:
:we need some structural devices actually to implement those non-standard modes of operation
:* conditional prerequisites (prevent evaluation, re-evaluate later)
:* special "as available" delivery, both for free-wheeling and background
:* a special way of "cancelling" jobs, which effectively re-schedules them into background.
:* a way for hinting the cache to store background frames with decreasing priority, thus ensuring the foremost availability of the first frames when picking up playback again
;for the __output sinks__:
:on the receiver side, we need some support to generate smooth and error free output delivery
:* automated detection of timing glitches, leading to activation of the discontinuity handling (&raquo;de-click facility&laquo;)
:* low-level API for signalling discontinuity to the OutputSlot. This information pertains to the currently delivered frame -- this is necessary when this frame //is actually being delivered timely.//
:* high-level API to switch any ~OutputSlot into "frozen mode", disabling any further output, even in case of accidental delivery of further data by jobs currently in progression.
:* ability to detect and signal overload of the receiver, either through blocking or for flow-control

We have to consider carefully how to handle the Creation of new class instances. Because, when done naively, it can defeat all efforts of separating subsystems, or &mdash; the other extreme &mdash; lead to a //switch-on-typeID// programming style. We strive at a solution somewhere in the middle by utilizing __Abstract Factories__ on Interface or key abstraction classes, but providing specialized overloads for the different use cases. So in each use case we have to decide if we want to create a instance of some general concept (Interface), or if we have a direct collaboration and thus need the Factory to provide a more specific sub-Interface or even a concrete type.
!Object creation use cases
!![[Assets|Asset]]
|!Action|>|!creates |
|loading a media file|asset::Media, asset::Codec| |
|viewing media|asset::Sequence, session::Clip and Placement (on hold)| for the whole Media, if not already existent|
|mark selection as clip|session::Clip, Placement with unspec. LocatingPin| doesn't add to session|
|loading Plugin|asset::Effect| usually at program startup|
|create Session|asset::Sequence, asset::Timeline, asset::Pipe| |
&rarr; [[creating and registering Assets|AssetCreation]]
&rarr; [[loading media|LoadingMedia]]
!![[MObjects|MObject]]
|add media to sequence|session::Clip, Placement with unspecified LocatingPin| creating whole-media clip on-the-fly |
|add Clip to sequence|copy of Placement| creates intependent Placement of existing ~Clip-MO|
|attach Effect|session::Effect, Placement with RelativeLocation| |
|start using Automation|session::Auto, asset::Dataset, RelativeLocation Placement| |
!Invariants
when creating Objects, certain invariants have to be maintained. Because creating an Object can be considered an atomic operation and must not leave any related objects in an inconsistent state. Each of our interfaces implies some invariants:
* every Placement has a Subject it places
* MObjects are always created to be placed in some way or the other
* [[Assets|Asset]] manage a dependency graph. Creating a derived Object (e.g. a Clip from a Media) implies a new dependency. (&rarr; [[memory management|ManagementAssetRelation]] relies on this)

Cinelerra2 introduced OpenGL support for rendering previews. I must admit, I am very unhappy with this, because
* it just supports some hardware
* it makes building difficult
* it can't handle all color models Cinelerra is capable of
* it introduces a separate codepath including some complicated copying of video data into the textures (and back?)
* it can't be used for rendering
So my judgement would be: in contrary to a realtime/gaming application, for quality video editing it is not worth the effort implementing OpenGL support in all details and with all its complexity. I would accept ~OpenGL as an option, if it could be pushed down into a Library, so it can be handled and maintained transparently and doesnt bind our limited developer manpower.
But because I know the opinions on this topc are varying (users tend to be delighted if they hear "~OpenGL", because it seems to be likted to the notion of "speed" and "power" todays) &mdash; I try to integrate ~OpenGL as a possibility into this design of the Render Engine. But I insist on putting up the requirement that it //must not jeopardize the code structure.//
My proposed aproach is to treat OpenGL as a separate video raw data type, requiring separete and specialized [[Processing Nodes|ProcNode]] for all calculations. Thus the Builder could connect OpenGL nodes if it is possible to cover the render path in whole or partially or maybe even just for preview.

A low-level abstraction within the [[Builder]] &mdash; it serves to encapsulate the details of making multi-channel connections between the render nodes: In some cases, a node can handle N channels internally, while in other cases we need to replicate the node N times and wire each channel individually. As it stands, the OperationPoint marks the ''borderline between high-level and low-level model'': it is invoked in terms of ~MObjects and other entities of the high-level view, but internally it manages to create ProcNode and similar entities of the low-level model.
The operation point is provided by the current BuilderMould and used by the [[processing pattern|ProcPatt]] executing within this mould and conducting the current build step. The operation point's interface allows //to abstract//&nbsp; these details, as well as to //gain additional control//&nbsp; if necessary (e.g. addressing only one of the channels). The most prominent build instruction used within the processing patterns (which is the instruction {{{"attach"}}}) relies on the aforementioned //approach of abstracted handling,// letting the operation point determine automatically how to make the connection.
This is possible because the operation point has been provided (by the mould) with information about the media stream type to be wired, which, together with information accessible at the [[render node interface|ProcNode]] and from the [[referred processing assets|ProcAsset]], with the help of the [[connection manager|ConManager]] allows to figure out what's possible and how to do the desired connections. Additionally, in the course of deciding about possible connections, the PathManager is consulted to guide strategic decisions regarding the [[render node configuration|NodeConfiguration]], possible type conversions and the rendering technology to employ.

An ever recurring problem in the design of Luimiera's ~Proc-Layer is how to refer to output destinations, and how to organise them.
Wiring the flexible interconnections between the [[pipes|Pipe]] should take into account both the StreamType and the specific usage context ([[scope|PlacementScope]]) -- and the challenge is to avoid hard-linking of connections and tangling with the specifics of the target to be addressed and connected. This page, started __6/2010__ by collecting observations to work out the relations, arrives at defining a //key abstraction// of output management.
!Observations
* effectively each [[Timeline]] is known to expose a set of global Pipes
* when connecting a Sequence to a Timeline or a VirtualClip, we also establish a mapping
* this mapping translates possible media stream channels produced by the sequence into (real) output slots located in the enclosing entity
* uncertainty on who has to provide the global pipes, implementation wise &mdash;
** as Timeline is just a façade, BindingMO has to expose something which can be referred for attaching effects (to global pipes)
** when used as VirtualClip, there is somehow a channel configuration, either as asset, or exposed by the BindingMO
* Placements always resolve at least two dimensions: time and output. The latter means that a [[Placement]] can figure out to where to connect
* the resolution ability of Placements could help to overcome the problems in conjunction with a VirtualClip: missing output destination information could be inherited down....
* expanding on the basic concept of a Placement in N-dimensional configuration space, this //figuring out// would denote the ability to resolve the final output destination
* this resolution to a final destination is explicitly context dependent. We engage into quite some complexities to make this happen (&rarr; BindingScopeProblem)
* [[processing patterns|ProcPatt]] are used for creating nodes on the source network of a clip, and similarly for fader, overlay and mixing into a summation pipe
* in case the fork ("track tree") of a sequence doesn't contain specific routing advice, connections will be done directly to the global pipes in order and by matching StreamType (i.e. typically video to video master, audio to stereo audio master). When a monitor (viewer window) is attached to a timeline, similar output connections are made from the timeline's global pipes, i.e. the video display will take the contents of the first video (master) bus, and the first stereo audio pipe will be pulled and sent to system audio out.
* a mismatch between the system output possibilities and the stream type of a bus to be monitored should result in the same adaptation mechanism to kick in, as is used internally, when connecting an ~MObject to the next bus. Possibly we'll use separate rules in this case (allow 3D to flat, stereo to mono, render 5.1 into Ambisonics...)
!Conclusions
* there are //direct, indirect//&nbsp; and //relative//&nbsp; referrals to an output designation
* //indirect// means to derive the destination transitively (looking at the destination's output designation and so on)
* //relative// in this context means that we refer to "the N^^th^^ of this kind" (e.g. the second video out)
* we need to attach some metadata with an output; especially we need an associated StreamType
* the referral to an output designation can be observed on and is structured into several //levels://
** within the body of the model, mostly we address output destinations relatively
** at some point, we'll address a //subgroup// within the global pipes, which acts like a summation sink
** there might be //master pipe(s),// collecting the output of various subgroups
** finally, there is the hardware output or the distinct channel within the rendered result &mdash; never to be referenced explicitly
!!!relation to Pipes
in almost every case mentioned above, the output designation is identical with the starting point of a [[Pipe]]. This might be a global pipe or a ClipSourcePort. Thus it sounds reasonable to use pipe-~IDs directly as output designation. Pipes, as an accountable entity (=asset) just //leap into existence by being referred.// On the other hand, the //actual//&nbsp; pipe is a semantic concept, a specific structural arrangement of objects. Basically it means that some object acts as attachment point and thereby //claims//&nbsp; to be the entrance side of a pipe, while other processor objects chain up in sequence.
!!!system outputs
System level output connections seem to be an exception to the above rule. There is no pipe at an external port, and these externally visible connection points can appear and disappear, driven by external conditions. Yet the question is if system outputs shall be directly addressable at all as output designation? Generally speaking, Lumiera is not intended to be a system wide connection manager or a real time performance software. Thus it's advisable to refrain from direct referrals to system level connections from within the model. Rather, there should be a separate OutputManagement to handle external outputs and displays, both in windows or full screen. So these external outputs become rather a matter of application configuration &mdash; and for all the other purposes we're free to ''use pipe-~IDs as output designation''.
!!!consequences of mentioning an output designation
The immediate consequence is that a [[Pipe]] with the given ID exists as an accountable entity. Only if &mdash; additionally &mdash; a suitable object within the model //claims to root this pipe,// a connection to this designation is wired, using an appropriate [[processing pattern|ProcPatt]]. A further consequence then is for the mentioned output designation to become a possible candidate for a ModelPort, an exit node of the built render nodes network. By default, only those designations without further outgoing connections actually become active model ports (but an existing and wired pipe exit node can be promoted to a model port explicitly).
&rarr; OutputManagement
!!Challenge: mapping of output designations
An entire sequence can be embedded within another sequence as a VirtualClip. While for a simple clip there is a Clip-~MObject placed into the model, holding an reference to a media asset, in case of a virtual clip an BindingMO takes on the role of the clip object. Note that, within another context, BindingMO also acts as [[Timeline]] implementation &mdash; indeed even the same sequence might be bound simultaneously as a timeline and into a virtual clip. This flexibility creates a special twist, as the contents of this sequence have no way of finding out if they're used as timeline or embedded virtual clip. So parts of this sequence might mention a OutputDesignation which, when using the sequence as virtual clip, needs to be translated into a virtual media channel, which can be treated in the same manner like any channel (video, audio,...) found within real media. Especially, a new output designation has to be derived in a sensible way, which largely depends on how the original output designation has been specified.
&rarr; see OutputMapping regarding details and implementation of this mapping mechanism
!Output designation definition
OutputDesignation is a handle, denoting a target [[Pipe]] to connect.
It exposes a function to resolve to a Pipe, and to retrieve the StreamType of that resolved output. It can be ''defined'' either explicitly by ~Pipe-ID, or by an indirect or relative specification. The later cases are resolved on demand only (which may be later and might even change the meaning, depending on the context). It's done this way intentionally to gain flexibility and avoid hard wiring (=explicit ~Pipe-ID)
!!!Implementation notes
Thus the output designation needs to be a copyable value object, but opaque beyond that. Mandated by the various ways to specify an output designation, a hidden state arises regarding the partial resolution. The implementation solves that dilemma by relying on the [[State|http://en.wikipedia.org/wiki/State_pattern]] pattern in combination with an opaque in-place buffer.
!Use of output designations
While actually data frames are //pulled,// on a conceptual level data is assumed to "flow" through the pipes from source to output. This conceptual ("high level") model of data flow is comprised of the pipes (which are rather rigid), and flexible interconnections. The purpose of an output designation is to specify where the data should go, especially through these flexible interconnections. Thus, when reaching the exit point of a pipe, an output designation will be //queried.// Finding a suitable output designation involves two parts:
* first of all, we need to know //what to route// -- kind of the traits of the data. This is given by the //current pipe.//
* then we'll need to determine an output designation //suitable for this data.// This is given by a "Plug" (WiringRequest) in the placement, and may be derived.
* finally, this output designation will be //resolved// -- at least partially, resulting in a target pipe to be used for the wiring
As both of these specifications are given by [[Pipe]]-~IDs, the actual designation information may be reduced. Much can be infered from the circumstances, because any pipe includes a StreamType, and an output designation for an incompatible stream type is irrelevant. (e.g. and audio output when the pipe currently in question deals with video)

//writing down some thoughts//
* ruled out the system outputs as OutputDesignation.
* thus, any designation is a [[Pipe]]-ID.
* consequently, it is not obviously clear if such an designation is the final exit point
* please note the [[Engine interface proposal|http://lumiera.org/documentation/devel/rfc_pending/EngineInterfaceOverview.html]]
* this introduces the notion of a ModelPort: //a point in the (high level) model where output can be produced//
* thus obviously we need an OutputManager element to track the association of OutputDesignation to OutputSlot
Do we get a single [[Fixture]] &mdash; guess yes
From the implementation side, the only interesting exit nodes are the ones to be //actually pulled,// i.e. those immeditately corresponding to the final output.
* __rendering__ happens immediately at the output of a GlobalPipe (i.e. at a [[Timeline]], which is top-level)
* __playback__ always happens at a viewer element
!Attaching and mapping of exit nodes
Initially, [[Output designations|OutputDesignation]] are typically just local or relative references to another OutputDesignation; yet after some resolution steps, we'll arrive at an OutputDesignation //defined absolute.// Basically, these are equivalent to a [[Pipe]]-ID choosen as target for the connection and -- they become //real//&nbsp; by some object //claiming to root this pipe.// The applicability of this pattern is figured out dynamically while building the render network, resulting in a collection of [[model ports|ModelPort]] as part of the current [[Fixture]]. A RenderProcess can be started to pull from these -- and only from these -- active exit points of the model. Besides, when the timeline enclosing these model ports is [[connected to a viewer|ViewerPlayConnection]], an [[output network|OutputNetwork]] //is built to allow hooking exit points to the viewer component.// Both cases encompass a mapping of exit nodes to actual output channels. Usually, this mapping just relies on relative addressing of the output sinks, starting to allocate connections with the //first of each kind// (e.g. "connect to the first usable audio output destination").
We should note that in both cases this [[mapping operation|OutputMapping]] is controlled and driven and constrained by the output side of the connection: A viewer has fixed output capabilities, and rendering targets a specific container format -- again with fixed and pre-settled channel configuration ({{red{TODO 9/11}}} when configurting a render process, it might be necessary to pre-compute the //possible kinds of output streams,// so to provide a sensible pre-selection of possible output container formats for the user to select from). Thus, as a starting point, we'll create a default configured mapping, assigning channels in order. This mapping then should be exposed to modification and tweaking by the user. For rendering, this is part of the render options dialog, while in case of a viwer connection, a switch board is created to allow modifying the default mapping.
[>img[Output Management and Playback|uml/fig143877.png]]
!Connection to external outputs
External output destinations are never addressed directly from within the model. This is an design decision. Rather, model parts connect to an OutputDesignation, and these in turn may be [[connected to a viewer element|ViewerPlayConnection]]. At this point, related to the viewer element, there is a mapping to external destination(s): for images, a viewer typically has an implicit, natural destination (read: actually there is a corresponding viewer window or widget), while for sound we use an mapping rule, which could be overridden locally in the viewer.
Any external output sink is managed as a [[slot|DisplayerSlot]] in the ~OutputManager. Such a slot can be opened and allocated for a playback process, which allows the latter to push calculated data frames to the output. Depending on the kind of output, there might be various, often tight requirements on the timed delivery of output data, but any details are abstracted away &mdash; any slot implementation provides a way to handle time-outs gracefully, e.g. by just showing the last video frame delivered, or by looping and fading sound
&rarr; the OutputManager interface describes handling this mapping association
&rarr; see also the PlayService
!the global output manager
Within the model routing is done mostly just by referring to an OutputDesignation -- but at some point finally we need to map these abstract designations to real output capabilities. This happens at the //output managing elements.// This interface, OutputManager, exposes these mappings of logical to real outputs and allows to manage and control them. Several elements within the application, most notably the [[viewers|ViewerAsset]], provide an implementation of this interface -- yet there is one primary implementation, the ''global output manager'', known as OutputDirector. It can be accessed through the {{{Output}}} façade interface and is the final authority when it comes to allocating and mapping of real output possibilities. The OutputDirector tracks all the OutputSlot elements currently installed and available for output.
The relation between the central OutputDirector and the peripheral OutputManager implementations is hierarchical. Because output slots are usually registered rather at some peripheral output manager implementation, a direct mapping from OutputDesignation (i.e. global pipe) to these slots is created foremost at that peripheral level. Resolving a global pipe into an output slot is the core concern of any OutputManager implementation. Thus, when there is a locally preconfigured mapping, like e.g. for a viewer's video master pipe to the output slot installed by the corresponding GUI viewer element, then this mapping will be picked up foremost to resolve the video master output.
For a viewer widget in the GUI this yields exactly the expeted behaviour, but in other cases, e.g. for sound output, we need more general, more globally scoped output slots. In these cases, when a local mapping is absent, the query for output resolution is passed on up to the OutputDirector, drawing on the collection of globally available output slots for that specific kind of media.
{{red{open question 11/11: is it possible to retrieve a slot from another peripheral node?}}}
!!!output modes
Most output connections and drivers embody some kind of //operation mode:// Display is characterised by resolution and colour depth, sound by number of channels and sampling rate, amongst others. There might be a mismatch with the output expectations represented by [[output designations|OutputDesignation]] within the model. Nontheless we limit those actual operation modes strictly to the OutputManager realm. They should not leak out into the model within the session.
In practice, this decision might turn out to be rather rigid, but some additional mechanisms allow for more flexibility
* when [[connecting|ViewerPlayConnection]] timeline to viewer and output, stream type conversions may be added automatically or manually
* since resolution of an output designation into an OutputSlot is initiated by querying an output manager, this query might include additional constraints, which //some// (not all) concrete output implementations might evaluate to provide an more suitably configured output slot variant.

The term &raquo;''Output Manager''&laquo; might denote two things: first, there is an {{{proc::play::OutputManager}}} interface, which can be exposed by several components within the application, most notably the [[viewer elements|ViewerAsset]]. And then, there is "the" global output manager, the OutputDirector, which finally tracks all registered OutputSlot elements and thus is the gatekeeper for any output leaving the application.
&rarr; see [[output management overview|OutputManagement]]
&rarr; see OutputSlot
&rarr; see ViewerPlayConnection
!Role of an output manager
The output manager interface describes an entity handling two distinct concerns, tied together within a local //scope.//
* a ''mapping'' to resolve a ModelPort (given as ~Pipe-ID) into an OutputSlot
* the ''registration'' and management of possible output slots, thereby creating a preferred local mapping for future connections
Note that an OutputSlot acts as a unit for registration and also for allocating / "opening" an output, while generally there might still be multiple physical outputs grouped into a single slot. This is relevant especially for sound output. A single slot is just the ability to allocate output ports up to a given limit (e.g. 2 for a stereo device, or 6 for a 5.1 device). These multiple channels are allways connected following a natural channel ordering. Thus the mapping is a simple 1:1 association from pipe to slot, assuming that the media types are compatible (and this has been checked already on a higher level).
The //registration//&nbsp; of an output slot installs a functor or association rule, which later on allows to claim and connect up to a preconfigured number of channels. This allocation or usage of a slot is exclusive (i.e. only a single client at a time can allocate a slot, even if not using all the possible channels). Each output manager instance may or may not be configured with a //fall-back rule:// when no association or mapping can be established locally, the connection request might be passed down to the global OutputDirector. Again, we can expect this to be the standard behaviour for sound, while video likely will rather be handled locally, e.g. within a GUI widget (but we're not bound to configure it exactly this way)

An output mapping serves to //resolve//&nbsp; [[output designations|OutputDesignation]].
!Mapping situations
;external outputs
:external outputs aren't addressed directly. Rather we set up a default (channel) mapping, which then can be overridden by local rules.
:Thus, in this case we query with an internal OutputDesignation as parameter and expect an OutputSlot
;viewer connections
:any Timeline produces a number of [[model ports|ModelPort]]. On the other hand, any viewer exposes a small number of output designations, representing the image and sound output(s).
:Thus, in this case we resolve similar to a bus connection, possibly overridden by already pre-existing or predefined connections.
;switch board
:a viewer might receive multiple outputs and overlays, necessitating a user operated control to select what's actually to be displayed
:Thus, in this case we need a backwards resolution at the lower end of the output network, to connect to the model port as selected through the viewer's SwitchBoard
;global pipes or virtual media
:when binding a Sequence as Timeline or VirtualClip, a mapping from output designations used within the Sequence to virtual channels or global pipes is required
:Thus, in this case we need to associate output designations with ~Pipe-IDs encountered in the context according to some rules &mdash; again maybe overridden by pre-existing connections
!Conclusions
All these mapping steps are listed here, because they exhibit a common pattern.
* the immediately triggering input key is a [[Pipe]]-ID (and thus provides a stream type); additional connection hints may be given.
* the mapped result draws from a limited selection of elements, which again can be keyed by ~Pipe-IDs
* the mapping is initialised based on default mapping rules acting as fallback
* the mapping may be extended by explicitly setting some associations
* the mapping itself is a stateful value object
* there is an //unconnected//&nbsp; state.
!Implementation notes
Thus the mapping is a copyable value object, using an associative array. It may be attached to a model object and persisted alongside. The mapping is assumed to run a defaults query when necessary. To allow for that, it should be configured with a query template (string). Frequently, special //default pipe// markers will be used at places where no distinct pipe-ID is specified explicitly. Besides that, invocations might supply additional predicates (e.g. {{{ord(2)}}} to point at "the second stream of this kind") thereby hinting the defaults resolution. Moreover, the mapping needs a way to retrieve the set of possible results, allowing to filter the results of the rules based default. Mappings might be defined explicitly. Instead of storing a //bottom value,// an {{{isDefined()}}} predicate might be preferable.
First and foremost, mapping can be seen as a //functional abstraction.// As it's used at implementation level, encapsulation of detail types in't the primary concern, so it's a candidate for generic programming: For each of those use cases outlined above, a distinct mapping type is created by instantiating the {{{OutputMapping<DEF>}}} template with a specifically tailored definition context ({{{DEF}}}), which takes on the role of a strategy. Individual instances of this concrete mapping type may be default created and copied freely. This instantiation process includes picking up the concrete result type and building a functor object for resolving on the fly. Thus, in the way typical for generic programming, the more involved special details are moved out of sight, while being still in scope for the purpose of inlining. But there //is// a concern better to be encapsulated and concealed at the usage site, namely accessing the rules system. Thus mapping leads itself to the frequently used implementation pattern where there is a generic frontend as header, calling into opaque functions embedded within a separate compilation unit.

Within the Lumiera player and output subsystem, actually sending data to an external output requires to allocate an ''output slot''
This is the central metaphor for the organisation of actual (system level) outputs; using this concept allows to separate and abstract the data calculation and the organisation of playback and rendering from the specifics of the actual output sink. Actual output possibilities (video in GUI window, video fullscreen, sound, Jack, rendering to file) can be added and removed dynamically from various components (backend, GUI), all using the same resolution and mapping mechanisms (&rarr; OutputManagement)
!Properties of an output slot
Each OutputSlot is an unique and distinguishable entity. It corresponds explicitly to an external output, or a group of such outputs (e.g. left and right soundcard output channels), or an output file or similar capability accepting media content. First off, an output slot needs to be provided, configured and registered, using an implementation for the kind of media data to be output (sound, video) and the special circumstances of the output capability (render a file, display video in a GUI widget, send video to a full screen display, establish a Jack port, just use some kind of "sound out"). An output slot is always limited to a single kind of media, and to a single connection unit, but this connection may still be comprised of multiple channels (stereoscopic video, multichannel sound).
In order to be usable as //output sink,// an output slot needs to be //allocated,// i.e. tied to and locked for a specific client. At any time, there may be only a single client using a given output slot this way. To stress this point: output slots don't provide any kind of inherent mixing capability; any adaptation, mixing, overlaying and sharing needs to be done within the nodes network producing the output data fed to the slot. (in special cases, some external output capabilities -- e.g. the Jack audio connection system -- may still provide additional mixing capabilities, but that's beyond the scope of the Lumiera application)
[>img[Outputslot implementation structures|uml/fig151685.png]]
Once allocated, the output slot returns a set of concrete ''sink handles'' (one for each physical channel expecting data). The calculating process feeds its results into those handles. Size and other characteristics of the data frames are assumed to be suitable, which typically won't be verified at that level anymore (but the sink handle provides a hook for assertions). Besides that, the allocation of an output slot reveals detailed ''timing expectations''. The client is required to comply to these [[timings|Timings]] when ''emitting'' data -- he's even required to provide a //current time specification,// alongside with the data. Based on this information, the output slot has the ability to handle timing failures gracefully; the concrete output slot implementation is expected to provide some kind of de-click or de-flicker facility, which kicks in automatically when a timing failure is detected.
!!!usage and implementation
Clients retrieve just a reference to an output slot by asking a suitable OutputManager for an output possibility supporting a specific format. Usually, they just "claim" this slot by invoking {{{allocate()}}}, which behind the scenes causes building of the actual output connections and mechanisms. For each such connection -- corresponding to a single channel within the media format handled by this ~OutputSlot -- the client gets a smart-handle {{{DataSink}}}. The concrete ~OutputSlot implementation performs operations quite specific to the kind of output and external interface in question. All tese specific handling is embodied within the concrete connection implementation used by the concrete ~OutputSlot
!!!timing expectations
Besides the sink handles, allocation of an output slot defines some timing constraints, which are binding for the client. These [[Timings]] are detailed and explicit, including a grid of deadlines for each frame to deliver, plus a fixed //latency.// Within this context, &raquo;latency&laquo; means the requirement to be ahead of the nominal time by a certain amount, to compensate for the processing time necessary to propagate the media to the physical output pin. The output slot implementation itself is bound by external constraints to deliver data at a fixed framerate and aligned to an externally defined timing grid, plus the data needs to be handed over ahead of these time points by an time amount given by the latency. Depending on the data exchange model, there is an additional time window limiting the buffer management.
The assumption is for the client to have elaborate timing capabilities at his disposal. More specifically, the client is assumed to be a job running within the engine scheduler and thus can be configured to run //after// another job has finished, and to run within certain time limits. Thus the client is able to provide a //current nominal time// -- which is suitably close to the actual wall clock time. The output slot implementation can be written such as to work out from this time specification if the call is timely or overdue -- and react accordingly.
!!!output modes
some concrete output connections and drivers embody a specific operation mode (e.g. sample rate or number of channels). The decision and setup of these operational configuration is initiated together with the [[resolution|OutputMapping]] of an OutputDesignation within the OutputManager, finally leading to an output slot (reference), which can be assumed to be suitably configured, before the client allocates this slot for active use. Moreover, an individual output sink (corresponding to a single channel) may just remain unused -- until there is an {{{emit()}}} call and successful data handover, this channel will just feature silence or remain black. (More flexible system, e.g. Jack, allow to generate an arbitrary number of output pins -- Lumiera will support this by allowing to set up additional output slots and attach this information to the current session &rarr; SessionConfigurationAttachment)
!!!Lifecycle and storage
The concrete OutputSlot implementation is owned and managed by the facility actually providing the output possibility in question. For example, the GUI provides viewer widgets, while some sound output backend provides sound ports. The associated OutputSlot implementation object is required to stay alive as long as it's registered with some OutputManager. It needs to be de-registered explicitly prior to destruction -- and this deregistration may block until all clients using this slot did terminate. Beyond that, an output slot implementation is expected to handle all kinds of failures gracefully -- preferably just emitting a signal (callback functor).
{{red{TODO 7/11: Deregistration is an unsolved problem....}}}
!!!unified data exchange cycle
The planned delivery time of a frame is used as an ID throughout that cycle
# within a defined time window prior to delivery, the client can ''allocate and retrieve the buffer'' from the BufferProvider.
# the client has to ''emit'' within a (short) time window prior to deadline
# now the slot gets exclusive access to the buffer for output, signalling the buffer release to the buffer provider when done.
!!!lapses
This data exchange protocol operates on a rather low level; there is only limited protection against timing glitches
| !step|!problem ||!consequences | !protection |
| (1)|out of time window ||interference with previous/later use of the buffer | prevent in scheduler! |
|~|does not happen ||harmless as such | emit ignored |
|~|buffer unavailable ||inhibits further operation | ↯ |
| (2)|out of time window ||harmless as such | emit ignored |
|~|out of order ||allowed, unless out of time | -- |
|~|does not happen ||frame treated as glitch | -- |
|~|buffer unallocated ||frame treated as glitch | emit ignored |
| (3)|emit missing ||frame treated as glitch | -- |
|~|fail to release buffer ||unable to use buffer further | mark unavailable |
|~|buffer unavailable ||serious malfunction of playback | request playback stop |
Thus there are two serious problem situations
* allocating the buffer out of time window bears the danger of output data corruption; but the general assumption is for the scheduler to ensure each job start time remains within the defined window and all prerequisite jobs have terminated successfully. To handle clean-up, we additionally need special jobs running always, in order, and be notified of prerequisite failures.
* failure to release a buffer in a timely fashion blocks any further use of that buffer, any further jobs in need of that buffer will die immediately. This situation can only be caused by a serious problem //within the slot, related to the output mechanism.// Thus there should be some kind of trigger (e.g. when this happens 2 times consecutively) to request aborting the playback or render as a whole.
&rarr; SchedulerRequirements
&rarr; OutputSlotDesign
&rarr; OutputSlotImpl

The OutputSlot interface describes a point where generated media data can actually be sent to the external world. It is expected to be implemented by adapters and bridges to existing drivers or external interface libraries, like a viewer widget in the GUI, ALSA or Jack sound output, rendering to file, using an external media format library. The design of this core facility was rather difficult and stretched out over quite some time span -- this page documents the considerations and the decisions taken.
!intention
OutpotSlot is a metaphor to unify the organisation of actual (system level) outputs; using this concept allows to separate and abstract the data calculation and the organisation of playback and rendering from the specifics of the actual output sink. Actual output possibilities (video in GUI window, video fullscreen, sound, Jack, rendering to file) can be added and removed dynamically from various components (backend, GUI), all using the same resolution and mapping mechanisms (&rarr; OutputManagement)
!design possibilities
!!properties as a starting point
* each OutputSlot is an unique and distinguishable entity. It corresponds explicitly to an external output
* an output slot needs to be provided, configured and registered, using an implementation specifically tailored for the kind of media data
* an output slot is always limited to a single kind of media, and to a single connection unit, but this connection may still be comprised of multiple channels.
* in order to be usable as //output sink,// an output slot needs to be //allocated,// i.e. tied to and locked for a specific client.
* this allocation is exclusive: at any time, there may be only a single client using a given output slot.
* the calculating process feeds its results into //sink handles//&nbsp; provided by the allocated output slot.
* allocation of an output slot leads to very specific [[timing expectations|Timings]]
* the client is required to comply to these timings and operate according to a strictly defined protocol.
* timing glitches will be detected due to this protocol; the output slot provides mechanisms for failing gracefully in these cases
!!data exchange models
Data is handed over by the client invoking an {{{emit(time,...)}}} function on the sink handle. Theoretically there are two different models how this data hand-over might be performed. This corresponds to the fact, that in some cases our own code manages the output and the buffers, while in other situations we intend to use existing library solutions or even external server applications to handle output
;buffer handover model
:the client owns the data buffer and cares for allocation and de-allocation. The {{{emit()}}}-call just propagates a pointer to the buffer holding the data ready for output. The output slot implementation in turn has the liability to copy or otherwise use this data within a given time limit.
;shared buffer model
:here the output mechanism owns the buffer. Within a certain time window prior to the expected time of the {{{emit()}}}-call, the client may obtain this buffer (pointer) to fill in the data. The slot implementation won't touch this buffer until the {{{emit()}}} handover, which in this case just provides the time and signalles the client is done with that buffer. If the data emitting handshake doesn't happen at all, it counts as late and superseded by the next handshake.
!!relation to timing
Besides the sink handles, allocation of an output slot defines some timing constraints, which are binding for the client. These include a grid of deadlines for each frame to deliver, plus a fixed //latency.// The output slot implementation itself is bound by external constraints to deliver data at a fixed framerate and aligned to an externally defined timing grid, plus the data needs to be handed over ahead of these time points by an time amount given by the latency. Depending on the data exchange model, there is an additional time window limiting the buffer management.
The assumption is for the client to have elaborate timing capabilities at his disposal. More specifically, the client is assumed to be a job running within the engine scheduler and thus can be configured to run //after// another job has finished, and to run within certain time limits. Thus the client is able to provide a //current nominal time// -- which is suitably close to the actual wall clock time. The output slot implementation can be written such as to work out from this time specification if the call is timely or overdue -- and react accordingly.
{{red{TODO 6/11}}}in this spec, both data exchange models exhibit a weakness regarding the releasing of buffers. At which time is it safe to release a buffer, when the handover didn't happen? Do we need an explicit callback, and how could this callback be triggered? This is similar to the problem of closing a network connection, i.e. the problem is generally unsolvable, but can be handled pragmatically within certain limits.
{{red{WIP 11/11}}}meanwhile I've worked out the BufferProvider interface in deail. There's now a deatiled buffer handover protocol defined, which is supported by a little state engine tracking BufferMetadata. This mechanism provides a transition to {{{BLOCKED}}} state when order or timing constraints are being violated, which practically solves this problem. How to detect and resolve such a floundered state from the engine point of view still remains to be addressed.
!!!Lifecycle and storage
The concrete OutputSlot implementation is owned and managed by the facility actually providing the output possibility in question. For example, the GUI provides viewer widgets, while some sound output backend provides sound ports. The associated OutputSlot implementation object is required to stay alive as long as it's registered with some OutputManager. It needs to be de-registered explicitly prior to destruction -- and this deregistration may block until all clients using this slot did terminate. Beyond that, an output slot implementation is expected to handle all kinds of failures gracefully -- preferably just emitting a signal (callback functor).
{{red{TODO 7/11: Deregistration is an unsolved problem....}}}
-----
!Implementation / design problems
How to handle the selection of those two data exchange models!
-- Problem is, typically this choice isn't up to the client; rather, the concrete OutputSlot implementation dictates what model to use. But, as it stands, the client needs to cooperate and behave differently to a certain degree. Unless we manage to factor out an common denominator of both models.
Thus: Client gets an {{{OutputSlot&}}}, without knowing the exact implementation type &rArr; how can the client pick up the right strategy?
Solving this problem through //generic programming// -- i.e coding both cases effectively different, but similar -- would require to provide both implementation options already at //compile time!//
{{red{currently}}} I see two possible, yet quite different approaches...
;generic
:when creating individual jobs, we utilise a //factory obtained from the output slot.//
;unified
:extend and adapt the protocol such to make both models similar; concentrate all differences //within a separate buffer provider.//
!!!discussion
the generic approach looks as it's becoming rather convoluted in practice. We'd need to hand over additional parameters to the factory, which passes them through to the actual job implementation created. And there would be a coupling between slot and job (the slot is aware it's going to be used by a job, and even provides the implementation). Obviously, a benefit is that the actual code path executed within the job is without indirections, and all written down in a single location. Another benefit is the possibility to extend this approach to cover further buffer handling models -- it doesn't pose any requirements on the structure of the buffer handling --
On the other hand, if we accept to retrieve the buffer(s) via an indirection, which we kind of do anyway //within the render node implementation// -- the unified model looks more like a clean solution. It's more like doing away with some local optimisations possible if we handle the models explicitly, so it's not much of a loss, given that the majority of the processing time will be spent within the inner pixel calculation loops for frame processing anyway. When following this approach, the BufferProvider becomes a third, independent partner, and the slot cooperates tightly with this buffer provider, while the client (processing node) still just talks to the slot. Basically, this unified solution works like extending the shared buffer model to both cases.
&rArr; __conclusion__: go for the unified approach!
!!!unified data exchange cycle
The planned delivery time of a frame is used as an ID throughout that cycle
# within a defined time window prior to delivery, the client can ''allocate and retrieve the buffer'' from the BufferProvider.
# the client has to ''emit'' within a (short) time window prior to deadline
# now the slot gets exclusive access to the buffer for output, signalling the buffer release to the buffer provider when done.
&rarr; OutputSlotImpl

OutputSlot is an abstraction, allowing unified treatment of various physical output connections from within the render jobs. The actual output slot is a subclass object, created and managed from the "driver code" for a specific output connection. Moreover, each output slot will be outfitted with a concrete BufferProvider to reflect the actual buffer handling policy applicable for this specific output connection. Some output connections might e.g. require delivery of the media data into a buffer residing on external hardware, while others work just fine when pointed to some arbitrary memory block holding generated data.
!operation steps
[>img[Sequenz of output data exchange steps|uml/fig145157.png]]The OutputSlot class defines some standard operations as protected virtual functions. These represent the actual steps in the data exchange protocol corresponding to this output slot.
;lock()
:claims the specified buffer for exclusive use.
:Client may now dispose output data
;transfer()
:transfers control from the client to the actual output connection for feeding data.
:Triggered by the client invoking {{{emit()}}}
;pushout()
:request the actual implementation to push the data to output.
:Usually invoked from the {{{transfer()}}} implementation, alternatively from a separate thread.
!buffer states
While the BufferProvider abstracts away the actual access to the output buffers and just hands out a ''buffer handle'', the server side (here the concrete output slot) is allowed to associate and maintain a ''state flag'' with each buffer. The general assumption is that writing this state flag is atomic, and that other facilities will care for the necessary memory barriers (that is: the output slot and the buffer provider will just access this state flag without much ado). The generic part of the output slot implementation utilises this buffer state flag to implement a state machine, which -- together with the timing constraints established with the [[help of the scheduler|SchedulerRequirements]] -- ensures sane access to the buffer without collisions.
| !state||>| !lock() |>| !transfer() |>| !pushout() |
| {{{NIL}}}||↯| · |↯| | ↯ | |
| {{{FREE}}}||✔|↷ locked |✔|· (ignore) | · | · |
| {{{LOCKED}}}||↯|↷ emitted |✔|↷ emitted | · |↷ free |
| {{{EMITTED}}}||↯|↷ blocked |↯|↷ blocked | · |↷ free |
| {{{BLOCKED}}}||↯| ✗ |↯| ✗ | ∅ |↷ free |
where · means no operation, ✔ marks the standard cases (OK response to caller), ↯ will throw and thus kill the calling job, ∅ will treat this frame as //glitch,// ✗ will request playback stop.
The rationale is for all states out-of-order to transition into the {{{BLOCKED}}}-state eventually, which, when hit by the next operation, will request playback stop.

The Lumiera Processing Layer is comprised of various subsystems and can be separated into a low-level and a high-level part. At the low-level end is the [[Render Engine|OverviewRenderEngine]] which basically is a network of render nodes cooperating closely with the Backend Layer in order to carry out the actual playback and media transforming calculations. Whereas on the high-level side we find several different [[Media Objects|MObjects]] that can be placed into the session, edited and manipulated. This is complemented by the [[Asset Management|Asset]], which is the "bookkeeping view" of all the different "things" within each [[Session|SessionOverview]].
There is rather strong separation between these two levels, and &mdash; <br/>correspondingly you'll encounter the data held within the Processing Layer organized in two different views, the ''high-level-model'' and the ''low-level-model''
* from users (and GUI) perspective, you'll see a [[Session|SessionOverview]] with a timeline-like structure, where various [[Media Objects|MObjects]] are arranged and [[placed|Placement]]. By looking closer, you'll find that there are data connections and all processing is organized around processing chains or [[pipes|Pipe]], which can be either global (in the Session) or local (in real or [[virtual|VirtualClip]] clips)
* when dealing with the actual calculations in the Engine (&rarr; see OverviewRenderEngine), you won't find any Tracks, Media Objects or Pipes &mdash; rather you'll find a network of interconnected [[render nodes|ProcNode]] forming the low level model. Each structurally constant segment of the timeline corresponds to a separate node network providing an ExitNode corresponding to each of the global pipes; pulling frames from them means running the engine.
* it is the job of the [[Builder]] create and wire up this render nodes network when provided with a given hig-level-model. So, actually the builder (together with the so called [[Fixture]]) form an isolation layer in the middle, separating the //editing part&nbsp;// from the //processing part.//
[img[Block Diagram|uml/fig128005.png]]

Render Engine, [[Builder]] and [[Controller]] are closely related Subsystems. Actually, the [[Builder]] //creates// a newly configured Render Engine //for every// RenderProcess. Before doing so, it queries from the Session (or, to be more precise, from the [[Fixture]] within the current Session) all necessary Media Object Placement information. The [[Builder]] then derives from this information the actual assembly of [[Processing Nodes|ProcNode]] comprising the Render Engine. Thus:
* the source of the build process is a sequence of absolute (explicit) [[Placements|Placement]] called the [[Playlist]]
* the [[build process|BuildProcess]] is driven, configured and controlled by the [[Controller]] subsystem component. It encompasses the actual playback configuration and State of the System.
* the resulting Render Engine is a list of [[Processors]], each configured to calculate a segment of the timeline with uniform properties. Each of these Processors in turn is a graph of interconnected ProcNode.s.
see also: RenderEntities, [[two Examples (Object diagrams)|Examples]]
{{red{TODO: adjust terminology in this drawing: "Playlist" &rarr; "Fixture" and "Graph" &rarr; "Segment"}}}
[img[Overview: Components of the Renderengine|uml/fig128261.png]]

A ParamProvider is the counterpart for (one or many) [[Parameter]] instances. It implements the value access function made available by the Parameter object to its clients.
To give a concrete example:
* a Fade Plugin needs the actual fade value for Frame t=xxx
* the Plugin has a Parameter Object (from which we could query the information of this parameter being a continuous float function)
* this Parameter Object provides a getValue() function, which is internally linked (i.e. by configuration) to a //Parameter Provider//
* the actual object implementing the ParamProvider Interface could be a Automation MObject located somewhere in the session and would do bezier interpolation on a given keyframe set.
* Param providers are created on demand; while building the Render Engine configuration actually at work, the Builder would have to setup a link between the Plugin Parameter Object and the ParamProvider; he can do so, because he sees the link between the Automation MObject and the corresponding Effect MObject
!!ParamProvider ownership and lifecycle
Actually, ParamProvider is just an interface which is implemented either by a constant or an [[Automation]] function. In both cases, access is via direct reference, while the link to the ParamProvider is maintained by a smart-ptr, which &mdash; in the case of automation may share ownership with the [[Placement]] of the automation data set.
&rarr; see the class diagram for [[Automation]]
&rarr; see EffectHandling

Parameters are all possibly variable control values used within the Render Engine. Contrast this with configuration values, which are considered to be fixed and need an internal reset of the application (or session) state to take effect.
A ''Parameter Object'' provides a descriptor of the kind of parameter, together with a function used to pull the //actual value// of this parameter. Here, //actual// has a two-fold meaning:
* if called without a time specification, it is either a global (but variable) system or session parameter or a default value for automated Parameters. (the intention is to treat this cases uniformly)
* if called with a time specification, it is the query for an &mdash; probably interpolated &mdash; [[Automation]] value at this absolute time. The corresponding ParamProvider should fall back transparently to a default or session value if no time varying data is available
{{red{TODO: define how Automation works}}}

/***
|<html><a name="Top"/></html>''Name:''|PartTiddlerPlugin|
|''Version:''|1.0.6 (2006-11-07)|
|''Source:''|http://tiddlywiki.abego-software.de/#PartTiddlerPlugin|
|''Author:''|UdoBorkowski (ub [at] abego-software [dot] de)|
|''Licence:''|[[BSD open source license]]|
|''TiddlyWiki:''|2.0|
|''Browser:''|Firefox 1.0.4+; InternetExplorer 6.0|
!Table of Content<html><a name="TOC"/></html>
* <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Description',null, event)">Description, Syntax</a></html>
* <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Applications',null, event)">Applications</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('LongTiddler',null, event)">Refering to Paragraphs of a Longer Tiddler</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Citation',null, event)">Citation Index</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('TableCells',null, event)">Creating "multi-line" Table Cells</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Tabs',null, event)">Creating Tabs</a></html>
** <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Sliders',null, event)">Using Sliders</a></html>
* <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Revisions',null, event)">Revision History</a></html>
* <html><a href="javascript:;" onclick="window.scrollAnchorVisible('Code',null, event)">Code</a></html>
!Description<html><a name="Description"/></html>
With the {{{<part aPartName> ... </part>}}} feature you can structure your tiddler text into separate (named) parts.
Each part can be referenced as a "normal" tiddler, using the "//tiddlerName//''/''//partName//" syntax (e.g. "About/Features"). E.g. you may create links to the parts, use it in {{{<<tiddler...>>}}} or {{{<<tabs...>>}}} macros etc.
''Syntax:''
|>|''<part'' //partName// [''hidden''] ''>'' //any tiddler content// ''</part>''|
|//partName//|The name of the part. You may reference a part tiddler with the combined tiddler name "//nameOfContainerTidder//''/''//partName//.|
|''hidden''|When defined the content of the part is not displayed in the container tiddler. But when the part is explicitly referenced (e.g. in a {{{<<tiddler...>>}}} macro or in a link) the part's content is displayed.|
|<html><i>any&nbsp;tiddler&nbsp;content</i></html>|<html>The content of the part.<br>A part can have any content that a "normal" tiddler may have, e.g. you may use all the formattings and macros defined.</html>|
|>|~~Syntax formatting: Keywords in ''bold'', optional parts in [...]. 'or' means that exactly one of the two alternatives must exist.~~|
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!Applications<html><a name="Applications"/></html>
!!Refering to Paragraphs of a Longer Tiddler<html><a name="LongTiddler"/></html>
Assume you have written a long description in a tiddler and now you want to refer to the content of a certain paragraph in that tiddler (e.g. some definition.) Just wrap the text with a ''part'' block, give it a nice name, create a "pretty link" (like {{{[[Discussion Groups|Introduction/DiscussionGroups]]}}}) and you are done.
Notice this complements the approach to first writing a lot of small tiddlers and combine these tiddlers to one larger tiddler in a second step (e.g. using the {{{<<tiddler...>>}}} macro). Using the ''part'' feature you can first write a "classic" (longer) text that can be read "from top to bottom" and later "reuse" parts of this text for some more "non-linear" reading.
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!!Citation Index<html><a name="Citation"/></html>
Create a tiddler "Citations" that contains your "citations".
Wrap every citation with a part and a proper name.
''Example''
{{{
<part BAX98>Baxter, Ira D. et al: //Clone Detection Using Abstract Syntax Trees.//
in //Proc. ICSM//, 1998.</part>
<part BEL02>Bellon, Stefan: //Vergleich von Techniken zur Erkennung duplizierten Quellcodes.//
Thesis, Uni Stuttgart, 2002.</part>
<part DUC99>Ducasse, Stéfane et al: //A Language Independent Approach for Detecting Duplicated Code.//
in //Proc. ICSM//, 1999.</part>
}}}
You may now "cite" them just by using a pretty link like {{{[[Citations/BAX98]]}}} or even more pretty, like this {{{[[BAX98|Citations/BAX98]]}}}.
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!!Creating "multi-line" Table Cells<html><a name="TableCells"/></html>
You may have noticed that it is hard to create table cells with "multi-line" content. E.g. if you want to create a bullet list inside a table cell you cannot just write the bullet list
{{{
* Item 1
* Item 2
* Item 3
}}}
into a table cell (i.e. between the | ... | bars) because every bullet item must start in a new line but all cells of a table row must be in one line.
Using the ''part'' feature this problem can be solved. Just create a hidden part that contains the cells content and use a {{{<<tiddler >>}}} macro to include its content in the table's cell.
''Example''
{{{
|!Subject|!Items|
|subject1|<<tiddler ./Cell1>>|
|subject2|<<tiddler ./Cell2>>|
<part Cell1 hidden>
* Item 1
* Item 2
* Item 3
</part>
...
}}}
Notice that inside the {{{<<tiddler ...>>}}} macro you may refer to the "current tiddler" using the ".".
BTW: The same approach can be used to create bullet lists with items that contain more than one line.
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!!Creating Tabs<html><a name="Tabs"/></html>
The build-in {{{<<tabs ...>>}}} macro requires that you defined an additional tiddler for every tab it displays. When you want to have "nested" tabs you need to define a tiddler for the "main tab" and one for every tab it contains. I.e. the definition of a set of tabs that is visually displayed at one place is distributed across multiple tiddlers.
With the ''part'' feature you can put the complete definition in one tiddler, making it easier to keep an overview and maintain the tab sets.
''Example''
The standard tabs at the sidebar are defined by the following eight tiddlers:
* SideBarTabs
* TabAll
* TabMore
* TabMoreMissing
* TabMoreOrphans
* TabMoreShadowed
* TabTags
* TabTimeline
Instead of these eight tiddlers one could define the following SideBarTabs tiddler that uses the ''part'' feature:
{{{
<<tabs txtMainTab
Timeline Timeline SideBarTabs/Timeline
All 'All tiddlers' SideBarTabs/All
Tags 'All tags' SideBarTabs/Tags
More 'More lists' SideBarTabs/More>>
<part Timeline hidden><<timeline>></part>
<part All hidden><<list all>></part>
<part Tags hidden><<allTags>></part>
<part More hidden><<tabs txtMoreTab
Missing 'Missing tiddlers' SideBarTabs/Missing
Orphans 'Orphaned tiddlers' SideBarTabs/Orphans
Shadowed 'Shadowed tiddlers' SideBarTabs/Shadowed>></part>
<part Missing hidden><<list missing>></part>
<part Orphans hidden><<list orphans>></part>
<part Shadowed hidden><<list shadowed>></part>
}}}
Notice that you can easily "overwrite" individual parts in separate tiddlers that have the full name of the part.
E.g. if you don't like the classic timeline tab but only want to see the 100 most recent tiddlers you could create a tiddler "~SideBarTabs/Timeline" with the following content:
{{{
<<forEachTiddler
sortBy 'tiddler.modified' descending
write '(index < 100) ? "* [["+tiddler.title+"]]\n":""'>>
}}}
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!!Using Sliders<html><a name="Sliders"/></html>
Very similar to the build-in {{{<<tabs ...>>}}} macro (see above) the {{{<<slider ...>>}}} macro requires that you defined an additional tiddler that holds the content "to be slid". You can avoid creating this extra tiddler by using the ''part'' feature
''Example''
In a tiddler "About" we may use the slider to show some details that are documented in the tiddler's "Details" part.
{{{
...
<<slider chkAboutDetails About/Details details "Click here to see more details">>
<part Details hidden>
To give you a better overview ...
</part>
...
}}}
Notice that putting the content of the slider into the slider's tiddler also has an extra benefit: When you decide you need to edit the content of the slider you can just doubleclick the content, the tiddler opens for editing and you can directly start editing the content (in the part section). In the "old" approach you would doubleclick the tiddler, see that the slider is using tiddler X, have to look for the tiddler X and can finally open it for editing. So using the ''part'' approach results in a much short workflow.
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!Revision history<html><a name="Revisions"/></html>
* v1.0.6 (2006-11-07)
** Bugfix: cannot edit tiddler when UploadPlugin by Bidix is installed. Thanks to José Luis González Castro for reporting the bug.
* v1.0.5 (2006-03-02)
** Bugfix: Example with multi-line table cells does not work in IE6. Thanks to Paulo Soares for reporting the bug.
* v1.0.4 (2006-02-28)
** Bugfix: Shadow tiddlers cannot be edited (in TW 2.0.6). Thanks to Torsten Vanek for reporting the bug.
* v1.0.3 (2006-02-26)
** Adapt code to newly introduced Tiddler.prototype.isReadOnly() function (in TW 2.0.6). Thanks to Paulo Soares for reporting the problem.
* v1.0.2 (2006-02-05)
** Also allow other macros than the "tiddler" macro use the "." in the part reference (to refer to "this" tiddler)
* v1.0.1 (2006-01-27)
** Added Table of Content for plugin documentation. Thanks to RichCarrillo for suggesting.
** Bugfix: newReminder plugin does not work when PartTiddler is installed. Thanks to PauloSoares for reporting.
* v1.0.0 (2006-01-25)
** initial version
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!Code<html><a name="Code"/></html>
<html><sub><a href="javascript:;" onclick="window.scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
***/
//{{{
//============================================================================
// PartTiddlerPlugin
// Ensure that the PartTiddler Plugin is only installed once.
//
if (!version.extensions.PartTiddlerPlugin) {
version.extensions.PartTiddlerPlugin = {
major: 1, minor: 0, revision: 6,
date: new Date(2006, 10, 7),
type: 'plugin',
source: "http://tiddlywiki.abego-software.de/#PartTiddlerPlugin"
};
if (!window.abego) window.abego = {};
if (version.major < 2) alertAndThrow("PartTiddlerPlugin requires TiddlyWiki 2.0 or newer.");
//============================================================================
// Common Helpers
// Looks for the next newline, starting at the index-th char of text.
//
// If there are only whitespaces between index and the newline
// the index behind the newline is returned,
// otherwise (or when no newline is found) index is returned.
//
var skipEmptyEndOfLine = function(text, index) {
var re = /(\n|[^\s])/g;
re.lastIndex = index;
var result = re.exec(text);
return (result && text.charAt(result.index) == '\n')
? result.index+1
: index;
}
//============================================================================
// Constants
var partEndOrStartTagRE = /(<\/part>)|(<part(?:\s+)((?:[^>])+)>)/mg;
var partEndTagREString = "<\\/part>";
var partEndTagString = "</part>";
//============================================================================
// Plugin Specific Helpers
// Parse the parameters inside a <part ...> tag and return the result.
//
// @return [may be null] {partName: ..., isHidden: ...}
//
var parseStartTagParams = function(paramText) {
var params = paramText.readMacroParams();
if (params.length == 0 || params[0].length == 0) return null;
var name = params[0];
var paramsIndex = 1;
var hidden = false;
if (paramsIndex < params.length) {
hidden = params[paramsIndex] == "hidden";
paramsIndex++;
}
return {
partName: name,
isHidden: hidden
};
}
// Returns the match to the next (end or start) part tag in the text,
// starting the search at startIndex.
//
// When no such tag is found null is returned, otherwise a "Match" is returned:
// [0]: full match
// [1]: matched "end" tag (or null when no end tag match)
// [2]: matched "start" tag (or null when no start tag match)
// [3]: content of start tag (or null if no start tag match)
//
var findNextPartEndOrStartTagMatch = function(text, startIndex) {
var re = new RegExp(partEndOrStartTagRE);
re.lastIndex = startIndex;
var match = re.exec(text);
return match;
}
//============================================================================
// Formatter
// Process the <part ...> ... </part> starting at (w.source, w.matchStart) for formatting.
//
// @return true if a complete part section (including the end tag) could be processed, false otherwise.
//
var handlePartSection = function(w) {
var tagMatch = findNextPartEndOrStartTagMatch(w.source, w.matchStart);
if (!tagMatch) return false;
if (tagMatch.index != w.matchStart || !tagMatch[2]) return false;
// Parse the start tag parameters
var arguments = parseStartTagParams(tagMatch[3]);
if (!arguments) return false;
// Continue processing
var startTagEndIndex = skipEmptyEndOfLine(w.source, tagMatch.index + tagMatch[0].length);
var endMatch = findNextPartEndOrStartTagMatch(w.source, startTagEndIndex);
if (endMatch && endMatch[1]) {
if (!arguments.isHidden) {
w.nextMatch = startTagEndIndex;
w.subWikify(w.output,partEndTagREString);
}
w.nextMatch = skipEmptyEndOfLine(w.source, endMatch.index + endMatch[0].length);
return true;
}
return false;
}
config.formatters.push( {
name: "part",
match: "<part\\s+[^>]+>",
handler: function(w) {
if (!handlePartSection(w)) {
w.outputText(w.output,w.matchStart,w.matchStart+w.matchLength);
}
}
} )
//============================================================================
// Extend "fetchTiddler" functionality to also recognize "part"s of tiddlers
// as tiddlers.
var currentParent = null; // used for the "." parent (e.g. in the "tiddler" macro)
// Return the match to the first <part ...> tag of the text that has the
// requrest partName.
//
// @return [may be null]
//
var findPartStartTagByName = function(text, partName) {
var i = 0;
while (true) {
var tagMatch = findNextPartEndOrStartTagMatch(text, i);
if (!tagMatch) return null;
if (tagMatch[2]) {
// Is start tag
// Check the name
var arguments = parseStartTagParams(tagMatch[3]);
if (arguments && arguments.partName == partName) {
return tagMatch;
}
}
i += tagMatch[0].length;
}
}
// Return the part "partName" of the given parentTiddler as a "readOnly" Tiddler
// object, using fullName as the Tiddler's title.
//
// All remaining properties of the new Tiddler (tags etc.) are inherited from
// the parentTiddler.
//
// @return [may be null]
//
var getPart = function(parentTiddler, partName, fullName) {
var text = parentTiddler.text;
var startTag = findPartStartTagByName(text, partName);
if (!startTag) return null;
var endIndexOfStartTag = skipEmptyEndOfLine(text, startTag.index+startTag[0].length);
var indexOfEndTag = text.indexOf(partEndTagString, endIndexOfStartTag);
if (indexOfEndTag >= 0) {
var partTiddlerText = text.substring(endIndexOfStartTag,indexOfEndTag);
var partTiddler = new Tiddler();
partTiddler.set(
fullName,
partTiddlerText,
parentTiddler.modifier,
parentTiddler.modified,
parentTiddler.tags,
parentTiddler.created);
partTiddler.abegoIsPartTiddler = true;
return partTiddler;
}
return null;
}
// Hijack the store.fetchTiddler to recognize the "part" addresses.
//
var oldFetchTiddler = store.fetchTiddler ;
store.fetchTiddler = function(title) {
var result = oldFetchTiddler.apply(this, arguments);
if (!result && title) {
var i = title.lastIndexOf('/');
if (i > 0) {
var parentName = title.substring(0, i);
var partName = title.substring(i+1);
var parent = (parentName == ".")
? currentParent
: oldFetchTiddler.apply(this, [parentName]);
if (parent) {
return getPart(parent, partName, parent.title+"/"+partName);
}
}
}
return result;
};
// The user must not edit a readOnly/partTiddler
//
config.commands.editTiddler.oldIsReadOnlyFunction = Tiddler.prototype.isReadOnly;
Tiddler.prototype.isReadOnly = function() {
// Tiddler.isReadOnly was introduced with TW 2.0.6.
// For older version we explicitly check the global readOnly flag
if (config.commands.editTiddler.oldIsReadOnlyFunction) {
if (config.commands.editTiddler.oldIsReadOnlyFunction.apply(this, arguments)) return true;
} else {
if (readOnly) return true;
}
return this.abegoIsPartTiddler;
}
config.commands.editTiddler.handler = function(event,src,title)
{
var t = store.getTiddler(title);
// Edit the tiddler if it either is not a tiddler (but a shadowTiddler)
// or the tiddler is not readOnly
if(!t || !t.abegoIsPartTiddler)
{
clearMessage();
story.displayTiddler(null,title,DEFAULT_EDIT_TEMPLATE);
story.focusTiddler(title,"text");
return false;
}
}
// To allow the "./partName" syntax in macros we need to hijack
// the invokeMacro to define the "currentParent" while it is running.
//
var oldInvokeMacro = window.invokeMacro;
function myInvokeMacro(place,macro,params,wikifier,tiddler) {
var oldCurrentParent = currentParent;
if (tiddler) currentParent = tiddler;
try {
oldInvokeMacro.apply(this, arguments);
} finally {
currentParent = oldCurrentParent;
}
}
window.invokeMacro = myInvokeMacro;
// Scroll the anchor anchorName in the viewer of the given tiddler visible.
// When no tiddler is defined use the tiddler of the target given event is used.
window.scrollAnchorVisible = function(anchorName, tiddler, evt) {
var tiddlerElem = null;
if (tiddler) {
tiddlerElem = document.getElementById(story.idPrefix + tiddler);
}
if (!tiddlerElem && evt) {
var target = resolveTarget(evt);
tiddlerElem = story.findContainingTiddler(target);
}
if (!tiddlerElem) return;
var children = tiddlerElem.getElementsByTagName("a");
for (var i = 0; i < children.length; i++) {
var child = children[i];
var name = child.getAttribute("name");
if (name == anchorName) {
var y = findPosY(child);
window.scrollTo(0,y);
return;
}
}
}
} // of "install only once"
//}}}
/***
<html><sub><a href="javascript:;" onclick="scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
!Licence and Copyright
Copyright (c) abego Software ~GmbH, 2006 ([[www.abego-software.de|http://www.abego-software.de]])
Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or other
materials provided with the distribution.
Neither the name of abego Software nor the names of its contributors may be
used to endorse or promote products derived from this software without specific
prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT
SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED
TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR
BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN
ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH
DAMAGE.
<html><sub><a href="javascript:;" onclick="scrollAnchorVisible('Top',null, event)">[Top]</sub></a></html>
***/

Facility guiding decisions regarding the strategy to employ for rendering or wiring up connections. The PathManager is querried through the OperationPoint, when executing the connection steps within the Build process.

Pipes play an central role within the Proc Layer, because for everything placed and handled within the session, the final goal is to get it transformed into data which can be retrieved at some pipe's exit port. Pipes are special facilities, rather like inventory, separate and not treated like all the other objects.
We don't distinguish between "input" and "output" ports &mdash; rather, pipes are thought to be ''hooks for making connections to''. By following this line of thought, each pipe has an input side and an output side and is in itself something like a ''Bus'' or ''processing chain''. Other processing entities like effects and transitions can be placed (attached) at the pipe, resulting them to be appended to form this chain. Likewise, we can place [[wiring requests|WiringRequest]] to the pipe, meaning we want it connected so to send it's output to another destination pipe. The [[Builder]] may generate further wiring requests to fulfil the placement of other entities.
Thus //Pipes are the basic building blocks// of the whole render network. We distinguish ''global available'' Pipes, which are like the sum groups of a mixing console, and the ''lokal pipe'' or [[source ports|ClipSourcePort]] of the individual clips, which exist only within the duration of the corresponding clip. The design //limits the possible kinds of pipes // to these two types &mdash; thus we can build local processing chains at clips and global processing chains at the global pipes of the session and that's all we can do. (because of the flexibility which comes with the concept of [[placements|Placement]], this is no real limitation)
Pipes are denoted and addressed by [[pipe IDs|PipeID]], implemented as ID of a pipe asset. Besides explicitly named pipes, there are some generic placeholder ~IDs for a standard configured pipe of a given type. This is done to avoid creating a separate ~Pipe-ID for each and every clip to build.
While pipes are rather rigid building blocks -- and especially are limited to a single StreamType without conversions, the interconnections or ''routing'' links to the contrary are more flexible. They are specified and controled through the use of an OutputDesignation, which, when fully resolved, should again yield a target ~Pipe-ID to connect. The mentioned resolution of an output designation typically involves an OutputMapping.
Similar structures and mechanisms are extended beyond the core model: The GUI can connect the viewer(s) to some pipe (and moreover can use [[probe points|ProbePoint]] placed like effects and connected to some pipe), and likewise, when starting a ''render'', we get the opportunity to specify the ModelPort (exit point of a GlobalPipe) to pull the data from. Pulling data from some pipe is the (only) way to activate the render nodes network reachable from this pipe.
&rarr; [[Handling of Tracks|TrackHandling]]
&rarr; [[Handling of Pipes|PipeHandling]]

!Identification
Pipes are distinct objects and can be identified by their asset ~IDs. Besides, as for all [[structural assets|StructAsset]] there are extended query capabilities, including a symbolic pipe-id and a media (stream) type id. Any pipe can accept and deliver exactly one media stream kind (which may be inherently structured though, e.g. spatial sound systems or stereoscopic video)
!creating pipes
Pipe assets are created automatically by being used and referred. Each [[Timeline]] holds a collection of global pipes, attached to the BindingMO, which is the representation or attachement point of the Timeline within the HighLevelModel ([[Session]]) ({{red{todo: implementation missing as of 11/09}}}), and further pipes can be created by using a new pipe reference in some placement. Moreover, every clip has an (implicit) [[source port|ClipSourcePort]], which will appear as pipe asset when first used (referred) while [[building|BuildProcess]]. Note that creating a new pipe implies using a [[processing pattern|ProcPatt]], which will be queried from the [[Defaults Manager|DefaultsManagement]] (resulting in the use of some preconfigured pattern or maybe the creation of a new ProcPatt object if necessary)
!removal
Deleting a Pipe is an advanced operation, because it includes finding and "detaching" all references, otherwise the pipe will leap back into existence immediately. Thus, global pipe entries in the Session and pipe references in [[locating pins|LocatingPin]] within any placement have to be removed, while clips using a given source port will be disabled. {{red{todo: implementation deferred}}}
!using Pipes
there is not much you can do directly with a pipe asset. It is an point of reference, after all. Any connection or routing to a target pipe is done by a placement with a suitable WiringPlug in some part of the timeline (&rarr; OutputDesignation), so it isn't stored with the pipe. You can edit the (user visible) description an you can globally disable a pipe asset. The pipe's ID and media stream type of course are fixed, because any connection and referral (via the asset ID) is based on them. Later on, we should provide a {{{rewire(oldPipe, newPipe)}}} to search any ref to the {{{oldPipe}}} and try to rewrite it to use the {{{newPipe}}}, possibly with a new media stream type.
Pipes are integrated with the [[management of defaults|DefaultsManagement]]. For example, any pipe implicitly uses some [[processing pattern|ProcPatt]] &mdash; it may default to the empty pattern. This way, any kind of standard wiring might be applied to the pipes (e.g a fader for audio, similar to the classic mixing consoles). This //is // a global property of the pipe, but &mdash; contrary to the stream type &mdash; this pattern may be switched {{red{really? -- lots of open questions here as of 11/10}}}

A straight processing chain or ''Pipe'' is a core concept in Lumiera's HighLevelModel. Like most of the other building blocks in that model, pipes are represented in the asset view, which leads to having unique pipe-asset ~IDs for all pipes encountered within the model. Now, because of the special building pattern used within that model, which is comprised of pipes and flexible interconnections, the routing and wiring can be done in terms of pipe-~IDs, using them as a short representation of an OutputDesignation. As a consequence, by these relations, the pipe-~IDs are promoted to an universal key for wiring, routing and output targets, usable from the high-level objects down to the low level nodes within the engine.
* pipe-~IDs are used to label the global busses
* pipe-~IDs are used to denote output designations for routing
* pipe-~IDs are attached to the real output possibilities, available when resolving such a designation
* pipe-~IDs serve to denote the ExitNode corresponding to a global pipe within a single segment of the low-level model
* pipe-~IDs consequently can be used to address a ModelPort, corresponding both to such a global bus and the corresponding exit nodes

A Placement represents a //relation:// it is always linked to a //Subject// (this being a [[Media Object|MObject]]) and has the meaning to //place// this Subject in some manner, either relatively to other Media Objects, by some Constraint or simply absolute at (time, output). The latter case is especially important for the build process and thus represented by a special [[Sub-Interface ExplicitPlacement|ExplicitPlacement]]. Besides this simple cases, Placements can also express more specific kinds of "locating" an object, like placing a sound source at a pan position or placing a video clip at a given layer (above or below another video clip)
So basically placements represent a query interface: you can allways ask the placement to find out about the position of the related object in terms of (time, output), and &mdash; depending on the specific object and situation &mdash; also about these additional [[placement derived dimensions|PlacementDerivedDimension]] like sound pan or layer order or similar things which also fit into the general concept of "placing" an object.
The fact of being placed in the [[Session|SessionOverview]] is constitutive for all sorts of [[MObject]]s, without Placement they make no sense. Thus &mdash; technically &mdash; Placements act as ''smart pointers''. Of course, there are several kinds of Placements and they are templated on the type of MObject they are refering to. Placements can be //aggregated// to increasingly constrain the resulting "location" of the refered ~MObject. See &rarr; [[handling of Placements|PlacementHandling]] for more details
!Placements as instance
Effectively, the placement of a given MObject into the Session acts as setting up an concrete instance of this object. This way, placements exhibit a dual nature. When viewed on themselves, like any reference or smart-pointer they behave like values. But, by adding a placement to the session, we again create a unique distinguishable entity with reference semantics: there could be multiple placements of the same object but with varying placement properties. Such a placement-bound-into-the-session is denoted by an generic placement-ID or (as we call it) &rarr; PlacementRef; behind the scenes there is a PlacementIndex keeping track of those "instances" &mdash; allowing us to hand out the PlacementRef (which is just an opaque id) to client code outside the Proc-Layer and generally use it as an shorthand, behaving as if it was an MObject instance

For any [[media object|MObject]] within the session, we can allways at least query the time (reference/start) point and the output destination from the [[Placement]], by which the object is being handled. But the simple act of placing an object in some way, can &mdash; depending on the context &mdash; create additional degrees of freedom. To list some important examples:
* placing a video clip overlapping with other clips on other tracks creates the possibility for the clip to be above another clip or to be combined in various other ways with the other clips at the same time position
* placing a mono sound object plugged to a stereophoic output destination creates the freedom to define the pan position
The Placement interface allows to query for these additional //parameter values derived from the fact of being placed.//
!defining additional dimensions
probably a LocatingPin but... {{red{TODO any details are yet unknown as of 5/08}}}
!querying additional dimensions
basically you resolve the Placement, yielding an ExplicitPlacement... {{red{TODO but any details of how additional dimensions are resolved is still undefined as of 5/08}}}

[[Placement]]s are at the very core of all [[editing operations|EditingOperations]], because they act as handles (smart pointers) to access the [[media objects|MObject]] to be manipulated. Placements themselves are lightweight and can be handled with //value semantics//. But, when adding a Placement to the [[Session]], it gains an distinguishable identity and should be treated by reference from then on: changing the location properties of this placement has a tangible effect on the way the placed object appears in the context of the session. Besides the direct (language) references, there is a special PlacementRef type which builds on this registration of the placement within the session, can be represented as POD and thus passed over external interfaces. Many editing tasks include finding some Placement in the session and reference as parameter. By acting //on the Placement object,// we can change parameters of the way the media object is placed (e.g. adjust an offset), while by //dereferencing//&nbsp; the Placement object, we access the "real" media object (e.g. for trimming its length). Placements are ''templated'' on the type of the actual ~MObject they refer to, thus defining the interface/methods usable on this object.
Actually, the way each Placement ties and locates its subject is implemented by one or several small LocatingPin objects, where subclasses of LocatingPin implement the various different methods of placing and resolving the final location. Notably, we can give a ~FixedLocation or we can atach to another ~MObject to get a ~RelativeLocation, etc. In the typical use case, these ~LocatingPins are added to the Placement, but never retrieved directly. Rather the Placement acts as a ''query interface'' for determining the location of the related object. Here, "location" can be thought of as encompassing multiple dimenstions at the same time. An object can be
* located at a specific point in time
* related to and plugged into a specific output or global bus
* defined to have a position within some [[context-dependant additional dimensions|PlacementDerivedDimension]] like
** the pan position, either on the stereophoic base, or within a fully periphoic (spatial) sound system
** the layer order and overlay mode for video (normal, additive, subtractive, masking)
** the stereoscopic window position (depth parameter) for 3D video
** channel and parameter selection for MIDI data
Placements have //value semantics,// i.e. we don't stress the identity of a placement object (~MObjects on the other hand //do have// a distinguishable identity): initially, you create a Placement parametrized to some specific kind by adding [[LocatingPin]]s (fixed, relative,...) and possibliy you use a subclass of {{{Placement<MObject>}}} to encode additional type information, say {{{Placement<Clip>}}}, but later on, you treat the placement polymorphically and don't care about its kind. The sole purpose of the placement's kind is to select some virtual function implementing the desired behaviour. There is no limitation to one single Placement per ~MObject, indeed we can have several different Placements of the same MObject (from a users point of view, these behave like being clones). Besides, we can ''aggregate'' additional [[LocatingPin]]s to one Placements, resulting in their properties and constraints being combined to yield the actual position of the referred ~MObject.
!design decisions
* the actual way of placing is implemented similar to the ''State Pattern'' by small embedded LocatingPin objects.
* these LocatingPin objects form a ''decorator'' like chain
* resolving into an ExplicitPlacement traverses this chain
* //overconstraining// a placement is not an error, we just stop traversing the chain (ignoring the remaining additional Placements) at the moment the position is completely defined.
* placements can be treated like values, but incorporate an identity tag for the purpose of registering with the session.
* we provide subclasses to be able to form collections of e.g. {{{Placement<Effect>}}}, but we don't stress polymorphism here. &rarr; PlacementType
* Why was the question how to access a ~MObject subinterface a Problem?
*# we want the Session/Fixture to be a collection of Placements. This means, either we store pointers, or Placement needs to be //one// unique type!
*# but if Placement is //a single type//, then we can get only MObjects from a Placement.
*# then either we had to do everything by a visitor (which gets the concrete subtype dynamically), or we'd end up switching on type.

An implementation facility used to keep track of individual Placements and their relations.
Especially, the [[Session]] maintains such an index, allowing to use the (opaque) PlacementRef tags for referring to a specific "instance" of an MObject, //placed// in a unique way into the current session. And, moreover, this index allows for one placement referring to another placement, so to implement a //relative// placement mode. Because there is an index behind the scenes, it is possible to actually access such a referral in the reverse direction, which is necessary for implementing the desired placement behaviour (if an object instance used as anchor is moved, all objects placed relatively to it have to move accordingly, which necessitates finding those other objects).
Besides searching, [[placement instances|Placement]] can be added to the index, thereby creating a copy managed by the backing data structure. Thus, the session's PlacementIndex is the backbone of the session data structure, and the session's contents are actually contained within it.
!rationale of the choosen implementation
What implementation approach to take for the index largely depends on the usage pattern. Generally speaking, the Lumiera [[Session]] is a collection of MObjects attached by [[Placement]]; any relations between these objects are established on a logical level, implemented as markers within the respective placements. For [[building the render nodes graph|BuildProcess]], a consolidated view of the session's effective contents is created ("[[Fixture]]"), then to be traversed while emitting the LowLevelModel. The GUI is expected to query the index to discover parts of the structure; the [[object reference tags|MObjectRef]] returned by these queries will be used as arguments to any mutating operation on the objects within the session.
Using a ''flat hashtable'' allows to access a Placement denoted by ID in O(1). This way we get the Placement, but nothing more. So, additionally we'd have to set up an data record holding additional information:
* the [[scope|PlacementScope]] containing this placement
* allowing to create a path "up" from this scope, which is used for resolving any queries
* (maybe/planned) relations to other placements
Alternatively, we could try to use a ''structure based index'', thereby avoiding the mentioned description record by folding any of the contained information into the surrounding data structure:
* the scope would be obvious from the index, resp. from the path used to resolve this index
* any other information, especially the relations, would be folded into the placement
* this way, the "index" could be reduced to being the session data structure itself.
//does a placement need to know it's own ID?//&nbsp; Obviously, there needs to be a way to find out the ID for a given placement, especially in the following situations:
* for most of the operations above, when querying additional information from index for a given placement
* to create a PlacementRef (this is a variant of the "shared ptr from this"-problem)
On second sight, this problem turns out to be more involved, because either we have to keep a second index table for the reverse lookup (memory address -> ID), or have to tie the placement by a back-link when adding it to the index/session data structure, or (alternatively) it forces us to store a copy of the ID //within// the placement itself. The last possibility seems to create the least impact; but implementing it this way effectively gears the implementation towards a hashtable based approach.
!supported operations
The placement index is utilized by editing operations and by executing the build process. Besides these core operations it allows for resolving PlacementRef objects. This latter functionality is used by all kinds of relative placements and for dealing with them while building, but it is also used to resolve [[object reference tags|MObjectRef]], which possibly may have been handed out via an external API or may have crossed layer boundaries. From these use cases we derive the following main operations to support:
* find the actual [[Placement]] object for a given ID
* find the //scope//&nbsp; a given placement resides in. More specifically, find the [[placement defining this scope|PlacementScope]]
* find (any/all) other placements referring to a given placement ("within this scope")
* add a new placement to a scope given as parameter
* remove a placement from index
* (planned) move a placement to a different scope within the session
!!!Handling of Subtypes
While usually type relations don't carry over to smart-poitner like types, in case of Placement I used a special definition pattern to artificially create such type relations ({{{Placement<session::Clip>}}} is subclass of {{{Placement<MObject>}}}). Now, as we're going to copy and maintain Placements within the storage backing the index, the question is: do we actually store subtypes (all having the same size btw) or do we use a vtable based mechanism to recover the type information on access?
Actually, the handling of placement types quite flexible; the actual hierarchy of Placement types can be determined in the //usage context// &mdash; it is not really stored within the placement, and there is no point in storing it within the index. Only the type of the //pointee//&nbsp; can be checked with the help of Placement's vtable.
Thus, things just fall into place here, without the need of any additional implementation logic. The index stores {{{Placement<MObject>}}} instances. The usage context will provide a suitable meaning for more specifically typed placements, and as long as this is in line with the type relations on the pointee(s), as checked by the {{{Placement::isCompatible<TY>()}}} call, the placement relations will just work out right by the the cast happening automatically on results retrieval.
&rarr; see PlacementType
!implementation
Consequently, we incorporate a random hash (implemented as {{{LUID}}}) into the individual placement, this way creating an distinguishable //placement identity,// which is //not retained on copying.// The actual ID tag is complemented by a compile time type (template parameter), thus allowing to pass on additional context information through API calls. Placements themselves use a vtable (and thus RTTI), allowing to re-discover the exact type at runtime. Any further relation information is contained within the placement's [[locating pins|LocatingPin]], thus, any further description records can be avoided by storing the placements immediately //within the index.// To summarise, the implementation is comprised of
* a main table resolving hash-ID to storage location
* information about the enclosing scope for each placement, stored within the main entry
* a reverse lookup table to find all placements contained within a given scope
* an instance holding and managing facility based on pooled allocation

A generic reference mechanism for Placements, as added to the current session.
While this reference itself is not tied to the actual memory layout (meaning it's //not// a disguised pointer), the implementation relies on a [[placement index facility|PlacementIndex]] for tracking and retrieving the actual Placement implementation object. As a plus, this approach allows to create active interconnections between placements. We utilise this possibility to create a system of [[nested scopes|PlacementScope]]. The index facility allows to //reverse// the relation denoted by such a reference, inasmuch it is possible to retrieve all other placements referring to a given target placement. But for an (external) user, this link to an index implementation is kept transparent and implicit.
!implementation considerations
From the usage context it is clear that the PlacementRef needs to incorporate a simple ID as the only actual data in memory, so it can be downcasted to a POD and passed as such via LayerSeparationInterfaces. And, of course, this ID tag should be the one used by PlacementIndex for organising the Placement index entries, thus enabling the PlacementRef to be passed immediately to the underlying index for resolution. Thus, this design decision is interconnected with the implementation technique used for the index (&rarr; PlacementIndex). From the requirement of the ID tag to be contained in a fixed sized storage, and also from the expected kinds of queries Ichthyo (5/09) choose to incorporate a {{{LUID}}} as a random hash immediately into the placement and build the ID tag on top of it.
!using placement references
Placement references can be created directly from a given placement, or just from an {{{Placement::ID}}} tag. Creation and dereferencing can fail, because the validity of the reference is checked with the index. This implies accessing the //current session// behind the scenes. Placement references have value semantics. Dereferencing searches the denoted Placement via index, yielding a direct (language) ref.
Placement references mimic the behaviour of a real placement, i.e. they proxy the placement API (actually the passive, query-related part of placement's API functions), while directly forwarding calls to the pointee (~MObejct or subclass) when using {{{operator->()}}}. They can be copied and especially allow a lot of assignments (from placement, placement-ID or even plain LUID), even including a dynamic downcast on the pointee. Bottom line is that a placement ref can pretty much be used in place of a language ref to a real placement, which is crucial for implementing MObjectRef.

MObjects are attached into the [[Session]] by adding a [[Placement]]. Because this especially includes the possibility of //grouping or container objects,// e.g. [[sequences|Sequence]] or [[forks ("tracks")|Fork]] or [[meta-clips|VirtualClip]], any placement may optionally define and root a scope, and every placement is at least contained in one encompassing scope &mdash; of course with the exception of the absolute top level, which can be thought off as being contained in a scope of handling rules.
Thus, while the [[sequences|Sequence]] act as generic container holding a pile of placments, actually there is a more fine grained structure based on the nesting of the tracks, which especially in Lumiera's HighLevelModel belong to the sequence (they aren't a property of the top level timeline as one might expect). Building upon these observations, we actually require each addition of a placement to specify a scope. Consequently, for each Placement at hand it is possible to determine an //containing scope,// which in turn is associated with some Placement of a top-level ~MObject for this scope. The latter is called the ''scope top''. An example would be the {{{Placement<Track>}}} acting as scope of all the clips placed onto this track. The //implementation//&nbsp; of this tie-to-scope is provided by the same mechanism as utilised for relative placements, i.e. an directional placement relation. Actually, this relation is implemented by the PlacementIndex within the current [[Session]].
[>img[Structure of Placment Scopes|draw/scopeStructure1.png]]
!Kinds of scopes
There is only a limited number of situations constituting a scope
* conceptually, the very top level is a scope of general rules.
* the next level is the link of [[binding|BindingMO]] of a [[Sequence]] into either a (top-level) [[Timeline]] or as virtual media into a VirtualClip. It is implemented through a {{{Placement<Binding>}}}.
* each sequence has at least one (manadtory) top-level placement holding its root track
* tracks may contain nested sub tracks.
* clips and (track-level) effects likewise are associated with an enclosing track.
* an important special case of relative placement is when an object is [[attached|AttachedPlacementProblem]] to another leading object, like e.g. an effect modifying a clip
__note__: attaching a Sequence in multiple ways &rarr; [[causes scoping problems|BindingScopeProblem]]
!Purpose of Placement scoping
Similar to the common mechanisms of object visibility in programming languages, placement scopes guide the search for and resolution of properties of placement. Any such property //not defined locally// within the placement is queried ascending through the sequence of nested scopes. Thus, global definitions can be shadowed by local ones.

Placement is a smart-ptr. As such, usually smart-pointers are templated on the pointee type, but a type relation between different target types doesn't carry over into a type relation on the corresponding smart-pointers. Now, as a [[Placement]] or a PlacementRef often is used to designate a specific "instance" of an MObject placed into the current session, the type parametrisation plays a crucial role when it comes to processing the objects contained within the session. Because the session deliberately has not much additional structure, besides the structure created by [[scopes and aggregations|PlacementScope]] within the session's contents.
To this end, we're using a special definition pattern for Placements, so
* a placement can refer to a specific sub-Interface like Fork ("track"), Clip, Effect
* a specialised placement can stand-in for the more generic type.
!generic handling
Thus, ~MObject and Placement<~MObject> relate to the "everything is an object" view of affairs. More specific placements are registered, searched and retrieved within the session through this generic interface. In a similar vein, ~PlacementRef<~MObject> and MObjectRef is used on LayerSeparationInterfaces. This works, because it is possible to re-discover the more fine-grained target type.
* ''active type rediscovery'' happens when a [[using visitors|VisitorUse]], which requires support by the pointee types (~MObject subclasses), so the visitor implementation is able to build a trampoline table to dispatch into a specifically typed context.
* ''passive type rediscovery'' is possible whenever the usage context //is already specifically typed.// Because in this case we can check the type (by RTTI) and filter out any placement not convertible to the type requested within the given context.
!downcasting and slicing
Deliberately, all Placements have the same runtime size. Handling them value-like under certain circumstances is intended and acceptable. Of course then slicing on the level of the Placement will happen. But because the Placement actually is a smart-pointer, the pointee remains unaffected, and can be used later to re-gain the fully typed context.
On the other hand, care has to be taken when ''downcasting'' a placement. When possible, this should be preceded by a {{{Placement::isCompatible<TY>()}}}-call, which checks based on the pointee's RTTI. Client code is encouraged to avoid explicit casting and rather rely on the provided facilities:
* invoking one of the templated access functions of the PlacementIndex
* using the QueryFocus to issue a specifically typed {{{ScopeQuery<TY>}}} (which yields an suitable iterator)
* create an specifically typed MObjectRef and bind it to some reference source (~Placement-ID, LUID, Placement instance within the session)
* implementing a visitor (~BuilderTool)

//This page is a scrapbook for working out the implementation of how to (re)build the [[Fixture]]//
Structurally, (re)building the Fixture rather belongs to [[Session]], but it is implemented very similar to the render engine build process: by treating all ~MObjects found in the various [[sequences|Sequence]] with a common [[visiting tool|VisitorUse]], this tool collects a simplified view with everyting made explicit, which can be pulled of as Fixture, i.e. (special kind of sequence list) afterwards.
* there is a //gathering phase// and a //solving phase//, the gathering is done by visiting.
* during the gathering phase, there ''need to be a lock'' preventing any other edit operation.
* the solving is delegated to the individual ~Placements. It is effectively a {{{const}}} operation creating a ExplicitPlacement (copy)
* thus the Fixture contains these newly created ~ExplicitPlacements, refering to ~MObjects shared with the original Placements within the sequences
!!!prerequisites
* Session and sequences exist.
* Pipes exist and are configured
!!!postconditions
* the Fixture contains one sorted timeline of ExplicitPlacement instances
* Anything in this list is actually to be rendered
* {{red{TODO: how to store and group the effects?}}}
* any meta-clips or other funny things have been resolved to normal clips with placement
* any multichannel clips has been broken down to elementary clips {{red{TODO: what is "elementary". e.g. stereo sound streams?}}}
* any globally or otherwise strangely placed effects have been attached either to a clip or to some pipe
* we have one unified list of tracks
<<tasksum start>>
<<taskadder below>>
<<task >> work out how to get the processing of effects chained to some clip right
<<task >> work out how to handle multichannel audio (and stereo video)
!gathering phase
!!preparing
<<task>>what data collections to build?
!!treating a Track
<<task>>work out how to refer to pipes and do other config
<<task>>get some uniqe identifier and get relevant properties
!!treating a {{{Placement<Clip>}}}
<<task>>check the direct enablement status
<<task>>asses the compound status, maybe process recursively
!!treating an {{{Placement<Effect>}}}
<<task>>find out the application point {{red{really?}}}
!solving phase
<<task>>trigger solving on all placements
<<task>>sort the resulting ~ExplicitPlacements
<<tasksum end>>

//This page is a scrapbook for working out the implementation of the builder//
* NodeCreatorTool is a [[visiting tool|VisitorUse]]
* the render engine to be built is contained as state within this tool object while it is passed around
!!!prerequisites
* Session and sequences exist.
* Pipes exist and are configured
* Fixture contains ExplicitPlacement for every MObject to be rendered, and nothing else
<<tasksum start>>
<<taskadder>>
!!preparing
We need a way of addressing existing [[pipes|Pipe]]. Besides, as the Pipes and Tracks are referred by the Placements we are processing, they are guaranteed to exist.
!!treating a Pipe
<<task>>get the [[processing pattern|ProcPatt]] of the pipe by accessing the underlying pipe asset.
<<task>>process this ProcPatt recursively
!!treating a processing pattern
<<task>>{{red{finally go ahead and define what a ProcPatt need to be...}}}
!!treating a {{{Placement<Clip>}}}
<<task>>get the ProcPatt of the underlying media (asset)
<<task>>process the ProcPatt recursively
<<task>>access the ClipSourcePort (which may be created on-the-fly)
<<task>>enqueue an WiringRequest for connecting the source pipeline to the source port
<<task>>process the clip's render pipe recursively (thus adding the camera etc.)
<<task>>enqueue an WiringRequest for any placement to some pipe for this clip.
* __note__: we suppose
** all wiring requests will be done after the processing of entities
** all effects placed to this clip will be processed after this clip (but before the wiring requests)
!!treating an {{{Placement<Effect>}}}
<<task>>{{red{how to assure that effecs are processed after clips/pipes??}}}
<<task>>find out the application point
<<task>>build a transforming node for the effect and insert it there
!!postprocessing
<<task>>sort and group the assembled list of [[wiring requests|WiringRequest]] by pipes
<<tasksum end>>

With //play process//&nbsp; we denote an ongoing effort to calculate a stream of frames for playback or rendering.
The play process is an conceptual entity linking together several activities in the [[Backend]] and the RenderEngine. Creating a play process is the central service provided by the [[player subsystem|Player]]: it maintains a registration entry for the process to keep track of associated entities, resources allocated and calls [[planned|FrameDispatcher]] and [[invoked|RenderJob]] as a consequence, and it wires and exposes a PlayController to serve as an interface and information hub.
''Note'': the player is in no way engaged in any of the actual calculation and management tasks necessary to make this [[stream of calculations|CalcStream]] happen. The play process code contained within the player subsystem is largely comprised of organisational concerns and not especially performance critical.
* the [[engine backbone|RenderBackbone]] is responsible for [[dispatching|FrameDispatcher]] the [[calculation stream|CalcStream]] and preparing individual calculation jobs
* the [[Scheduler]] at the [[engine core|RenderEngine]] has the ability to trigger individual frame calculation carry out individual [[frame calculation jobs|RenderJob]].
* the OutputSlot exposed by the [[output manager|OutputManagement]] is responsible for accepting timed frame delivery
[>img[Anatomy of a Play Process|uml/fig144005.png]]
!Anatomy of a Play Process
The Controller is exposed to the client and acts as frontend handle, while the play process body groups and manages all the various parts cooperating to generate output. For each of the participating global pipes we get a [[feed|Feed]] to drive that pipeline to deliver media of a specific kind.
Right within the play process, there is a separation into two realms, relying on different programming paradigms. Obviously the play controller is a state machine, and similarily the body object (play process) has a distinct operation state. Moreover, the current collection of individual objects hooked up at any given instance is a stateful variable. To the contrary, when we enter the realm of actual processing, operations are carried out in parallel, relying on stateless descriptor objects, wired into individual calculation jobs, to be scheduled as non-blocking units of operation. For each series of consecutive frames to be calculated, there is a descriptor object, the CalcStream, which also links to a specificaly tailored dispatcher table, allowing to schedule the individual frame jobs. Whenever the controller determines a change in the playback plan (speed change, skip, scrubbing, looping, ...), a new CalcStream is created, while the existing one is just used to mark any not-yet processed job as superseded.
&rarr; for overview see also OutputManagement

The [[Player]] is an independent [[Subsystem]] within Lumiera, located at Proc-Layer level. A more precise term would be "rendering and playback coordination subsystem". It provides the capability to generate media data, based on a high-level model object, and send this generated data to an OutputDesignation, creating an continuous and timing controlled output stream. Clients may utilise these functionality through the ''play service'' interface.
!subject of performance
Every play or render process will perfrom a part of the session. This part can be specified in varios ways, but in the end, every playback or render boils down to //performing some model ports.// While the individual model port as such is just an identifier (actually implemented as ''pipe-ID''), it serves as a common identifier used at various levels and tied into several related contexts. For one, by querying the [[Fixture]], the ModelPort leads to the actual ExitNode -- the stuff actually producing data when being pulled. Besides that, the OutputManager used for establishing the play process is able to resolve onto a real OutputSlot -- which, as a side effect, also yields the final data format and data implementation type to use for rendering or playback.
!provided services
* creating a PlayProcess
* managing existing play processes
* convenience short-cuts for //performing//&nbsp; several kinds of high-level model objects
!creating a play process
This is the core service provided by the player subsystem. The purpose is to create a collaboration between several entities, creating media output
;data producers
:a set of ModelPort elements to ''pull'' for generating output. They can either be handed in direcly, or resolved from a set of OutputDesignation elements
;data sinks
:to be able to create output, the PlayProcess needs to cooperate with [[output slots|OutputSlot]]
:physical outputs are never handled directly, rather, the playback or rendering needs an OutputManager to resolve the output designations into output slots
;controller
:when provided with these two prerequisites, the play service is able to build a PlayProcess.
:for clients, this process can be accessed and maintained through a PlayController, which acts as (copyable) handle and front-end.
;engine
:the actual processing is done by the RenderEngine, which in itself is a compound of several services within [[Backend]] and Proc-Layer
:any details of this processing remain opaque for the clients; even the player subsystem just accesses the EngineFaçade

Within Lumiera, &raquo;Player&laquo; is the name for a [[Subsystem]] responsible for organising and tracking //ongoing playback and render processes.// &rarr; [[PlayProcess]]
The player subsystem does not perform or even manage any render operations, nor does it handle the outputs directly.
Yet it addresses some central concerns:
;uniformity
:all playback and render processes are on equal footing, handled in a similar way.
;integration
:the player cares for the necessary integration with the other subsystems
:it consults the OutputManagement, retrieves the necessary information from the [[Session]] and coordinates the forwarding of [[Backend]] calls.
;time quantisation
:the player translates continuous time values into discrete frame counts.
:to perform this [[quantisation|TimeQuant]], the help of the session for building a TimeGrid for each output channel is required.
!{{red{WIP 5/2011}}} under construction
The player subsystem is currently about to be designed and built up; some time ago, __Joel Holdsworth__ and __Ichthyo__ did a design study with a PlayerDummy, which is currently hooked up with the TransportControl in the Lumiera GUI. Starting from these experiences, and the general requirements of an NLE, the [[design of the Player subsystem|DesignPlayerSubsystem]] is being worked out.

__Joelholdsworth__ and __Ichthyo__ created this player mockup in 1/2009 to find out about the implementation details regarding integration and colaboration between the layers. There is no working render engine yet, thus we use a ~DummyImageGenerator for creating faked yuv frames to display. Within the GUI, there is a ~PlaybackController hooked up with the transport controls on the timeline pane.
# first everything was contained within ~PlaybackController, which spawns a thread for periodically creating those dummy frames
# then, a ~PlayerService was factored out, now implemented within ~Proc-Layer (later to delegate to the emerging real render engine implementation).<br/>A new LayerSeparationInterface called ''~DummyPlayer'' was created and set up as a [[Subsystem]] within main().
# the next step was to support multiple playback processes going on in parallel. Now, the ~PlaybackController holds an smart-handle to the ~PlayProcess currently generating output for this viewer, and invokes the transport control functions and the pull frame call on this handle.
# then, also the tick generation (and thus the handling of the thread which pulls the frames) was factored out and pushed down into the mentioned ~PlayProcess. For this to work, the ~PlaybackController now makes a display slot available on the public GUI DisplayFacade interface, so the ~PlayProcessImpl can push up the frames for display within the GUI
[img[Overview to the dummy player operation|draw/playerArch1.png]]
!when playing...
As a prerequisite, a viewer has to be prepared within the GUI. A XV video display widget is wired up to a sigc++ signal slot, using the Glib::Dispatcher to forward calls from the play process thread to the GTK main event loop thread. All of this wiring actually is encapsulated as a DisplayerSlot, created and registered with the DisplayService.
When starting playback, the display slot handle created by these preparations is used to create a ~PlayProcess on the ~DummyPlayer interface. Actually
* a ~PlayProcessImpl object is created down within the player implementation
* this uses the provided slot handle to actually //allocate// the display slot via the Display facade. Here, //allocating// means registering and preparing it for output by //one single// ~PlayProcess. For the latter, this allocation yields an actually opened display handle.
* moreover, the ~PlayProcessImpl aquires an TickService instance, which is still trotteling (not calling the periodic callback)
* probably, a real player at this point would initiate a rendering process, so he can fetch the actual output frames periodically.
* on the "upper" side of the ~DummyPlayer facade, a lib::Handle object is created to track and manage this ~PlayProcesImpl instance
The mentioned handle is returned to the ~PlaybackController within the GUI, which uses this handle for all further interactions with the Player. The handle is ref counting and has value semantics, so it can be stored away, passed as parameter and so on. All such handles corresponding to one ~PlayProcess form a family; when the last goes out of scope, the ~PlayProcess terminates and deallocates any resources. Conceptually, this corresponds to pushing the "stop" button. Handles can be deliberately disconnected by calling {{{handle.close()}}} &mdash; this has the same effect as deleting a handle (when all are closed or deleted the process ends).
All the other play control operations are simply forwarded via the handle and the ~PlayProcessImpl. For example, "pause" corresponds to setting the tick frequency to 0 (thus temporarily discontinuing the tick callbacks). When allocating the display slot in the course of creating the ~PlayProcessImpl, the latter only accesses the Display facade. It can't access the display or viewer directly, because the GUI lives within an plugin; lower layers aren't allowed to call GUI implementation functions directly. Thus, within the Display facade a functor (proxy) is created to represent the output sink. This (proxy) Displayer can be used within the implementation of the perodic callback function. As usual, the implementation of the (proxy) Displayer can be inlined and doesn't create runtime overhead. Thus, each frame output call has to pass though two indirections: the function pointer in the Display facade interface, and the Glib::Dispatcher.
!rationale
There can be multiple viewer widgets, to be connected dynamically to multiple play-controllers. (the latter are associated with the timeline(s)). Any playback can require multiple playback processes to work in parallel. The playback controller(s) should not be concerned with managing the play processes, which in turn should neither care for the actual rendering, nor manage the display frequency and synchronisation issues. Moreover, the mentioned parts live in different layers and especially the GUI needs to remain separated from the core. And finally, in case of a problem within one play process, it should be able to unwind automatically, without interfering with other ongoing play processes.

Playlist is a sequence of individual Render Engine Processors able to render a segment of the timeline. So, together these Processors are able to render the whole timeline (or part of the timeline if only a part has to be rendered).
//Note, we have yet to specify how exactly the building and rendering will work together with the backend. There are several possibilities how to structure the Playlist//

Within Lumiera, we distinguish between //model state// and //presentation state.// Any conceivable UI is stateful and reshapes itself through interaction -- but the common UI toolkits just give us this state as //transient state,// maybe with some means to restore state. For simple CRUD applications this might be sufficient, as long as the data is self contained and the meaning of data is self evident. But a work environment, like the NLE we're building here, layers additional requirements on top of mere data access. To be able to work, not only you need tools, you need //enablement.// Which, in a nutshell, means that things-at-hand need to be at hand. Sounds simple, yet is a challenge still not adequately fulfilled by contemporary computer based interfaces and environments.
A fundamental design decision in Lumiera is to distinguish between engine, model and working environment. The //model// is oriented towards the practicalities of film making, not towards the practicalities of implementing video processing. Also, the model needs to be self contained to the degree that //anything with a tangible influence on the final rendered result// needs to be represented within the model. So the task of editing a film is the task of building such a model. This is an ongoing effort over an extended period of time, which in fact turns this effort into a project. Yet, as such, any project also pertains to the way you are working within that project. These might be fundamental decisions about what material to use, about formats and technologies, but also some set of conventions emerging under way. Beyond that, any project develops habits of handling matters, which, over time, turn into rules. And even beyond that, over time, you expect to find common things at common places. The latter is what presentation state is all about.
!dealing with presentation state
The backbone of the Lumiera UI is arranged such as to produce a data feed with state transition notifications relevant for presentation state. This is not the raw data feed of UI signals, like a click on this or that button (such is kept confined within the realm of the UI toolkit, here GTK). Rather, it is a custom made, pre-filtered stream of messages on the UI-Bus, organised and structured and outfitted with meaning by the [[tangible interfeace elements|UI-Element]]. Technically speaking, these messages are known as ''state mark messages'', and they are handled through a specific set of API functions on the bus interface. It is a bidirectional data exchange protocol, encompassed by
;state notifications
:whenever an UI-Element deems a state transition ''relevant for persistent presentation state'', a state mark message will be emitted
:* the originating element, as designated by its ID, is evident from the API call
:* the message itself is a GenNode
:** the ~ID-symbol of this message element was chosen by the emitting entity; it has distinctive meaning for this entity, e.g. »expand«, »reOrder«,...
:** the payload of the message is a data element, sufficient for the originator to restore itself to precisely the state as represented through this state mark.
;state marking
:whenever the [[state manager|PresentationStateManager]] deems some state eligible to be restored, it casts the relevant state mark back at its originator
:* so the originator needs to be discernibly by its ID, and this ID needs to be fabricated in a way such as to be reproducible within a later editing session
:* the UI-Bus allows just to cast some state mark message at "someone" -- this is also used by the lower layers for notifications and error results
:* the receiver is assumed to "understand" the actual meaning of that message and reshape itself to comply

//The PresentationStateManager creates the ability to build persistent PresentationState//
Run as part of the UiCoreServices, it is attached to the UI-Bus and listens to all ''state mark messages'' to distil the notion of relevant current state.
On a basic level, the task is to group and store those messages in a way as to overwrite previous messages with new updates on the level of individual properties. Beyond that, there is the sensitivity to context, presentation perspective and work site, which means to impose an additional structure on this basic ''state snapshot'', to extract and replicate structured sets of state information.

Open issues, Things to be worked out, Problems still to be solved...
!!Parameter Handling
The requirements are not quite clear; obviously Parameters are the foundation for getting automation right and for providing effect editing interfaces, so it seems to me we need some sort of introspection, i.e. Parameters need to be discovered, enumerated and described at runtime. (&rarr; see [[tag:automation|automation]])
''Automation Type'': Directly connected is the problem of handling the //type// of parameters sensible, including the value type of automation data. My first (somewhat naive) approach was to "make everything a double". But this soon leads into quite some of the same problems haunting the automation solution implemented in the current Cinelerra codebase. What makes the issue difficult is the fact we both need static diversity as well as dynamic flexibility. Usually, when combining hierarchies and templates, one has to be very careful; so I just note the problem down at the moment and will revisit it later, when I have a more clear understanding of the demands put onto the [[ProcNode]]s
!!Treatment of Time (points) and Intervals
At the moment we have no clear picture what is needed and what problems we may face in that domain.
From experience, mainly with other applications, we can draw the following conclusions
* drift and rounding errors are dangerous, because time in our context usually is understood as a fixed grid (Frames, samples...)
* fine grained time values easily get very large
* Cinelerra currently uses the approach of simply counting natural values for each media type separately. In an environment mixing several different media types freely, this seems a bit too simplistic (because it actually brings in the danger of rounding errors, just think at drop frame TC)
!!Organizing of Output Channels
How to handle the simultaneous rendering of several output streams (video, audio channels). Shall we treat the session as one entity containing different output channels, or should it rather be seen as a composite of several sub-sessions, each for only one output channel? This decision will be reflected in the overall structure of the network of render nodes: We could have a list of channel-output generating pipelines in each processor (for every segment), or we could have independently segmented lists of Processors for every output channel/type. The problem is, //it is not clear what approach to prefer at the moment// because we are just guessing.
!!Tracks, Channels, Layers
Closely related to this is the not-so-obvious problem how to understand the common global structures found in most audio and video editing applications. Mostly, they stem from imitating hardware recording and editing solutions, thus easing the transition for professionals grown up with analogue hardware based media. But as digital media are the de-facto standard nowadays, we could rethink some of this accidental complexity introduced by sticking to the hardware tool metaphor.
* is it really necessary to have fixed global tracks?
* is it really helpful to feed "source tracks" into global processing busses/channels?
Users accustomed with modern GUI applications typically expect that //everything is a object// and can be pulled around and manipulated individually. This seems natural at start, but raises the problem of providing a efficient workflow for handling larger projects and editing tasks. So, if we don't have a hard wired multitrack+bus architecture, we need some sort of templating to get the standard editing use case done efficiently.
!!Compound and Multiplicity
Simple relations can be hard wired. But, on the contrary, it would be as naive to define a Clip as having a Video track and two audio tracks, as it would be naive to overlook the problem of holding video and corresponding audio together. And, moreover, the default case has to be processed in a //straight forward// fashion, with as few tests and decisions as possible. So, basically each component participating in getting the core processing done has to mirror the structure pattern of the other parts, so that processing can be done without testing and forking. But this leaves us with the problem where to put the initial knowledge about the structural pattern used for building up the compound structures and &mdash; especially &mdash; the problem how to treat different kinds of structural patterns, how to detect the pattern to be applied and how to treat multiple instances of the same structural pattern.
One example of this problem is the [[handling of multichannel media|MultichannelMedia]]. Following the above reasoning, we end with having a [["structural processing pattern"|ProcPatt]], typically one video stream with MPEG decoder and a pair of audio streams which need either to be routed to some "left" and "right" output pipes, or have to be passed through a panning filter accordingly. Now the problem is: //create a new instance of this structure for each new media, or detect which media to subsume under a existing pattern instance.//
!!Parallelism
We need to work out guidelines for dealing with operations going on simultaneously. Certainly, this will divide the application in several different regions. As always, the primary goal is to avoid multithread problems altogether. Typically, this can be achieved by making matters explicit: externalizing state, make the processing subsystems stateless, queue and schedule tasks, use isolation layers.
* the StateProxy is a key for the individual render processes state, which is managed in separate [[StateFrame]]s in the backend. The [[processing network|ProcNode]] is stateless.
* the [[Fixture]] provides an isolation layer between the render engine and the Session / high-level model
* all EditingOperations are not threadsafe intentionally, because they are [[scheduled|ProcLayerScheduler]]
!!the perils of data representation
In software development, there is a natural inclination to cast "reality" into data, the structure of which has to be nailed down first. Then, everyone might "access reality" and work on it. Doing so might sounds rational, natural, even self-evident and sound, yet as compelling as it might be, this approach is fundamentally flawed. It is known to work well only for small, "handsome" projects, where you clearly know up-front what you're up to: namely to get away, after being paid, before anyone realises the fact you've built something that looks nice but does not fit.
So the challenge of any major undertaking in software construction is //not to build an universal model of truth.// Rather, we want to arrive at something that can be made to fit.
Which can be remoulded, over and over again, without breaking down.
More specifically, we start building something, and while under way, our understanding sharpens, and we learn that actually we want something entirely different. Yet still we know what we need and we don't want just something arbitrary. There is a constant core in what we're headed at, and we need the ability to //settle matters.// We need a backbone to work against, a skeleton to support us with its firmness, while also offering joints and links, to be bent and remoulded without breakage. The distinctive idea to make such possible is the principle of ''Subsidiarity''. The links and joints between autonomous centres can be shaped to be in fact an exchange, a handover based on common understanding of the //specific matters to deal with,// at that given joint.

The architecture of the Lumiera application separates functionality into three Layers: __GUI__, __Proc__ and __Backend__.
While the Backend is responsible for Data access and management and for carrying out the computation intensive media opteratons, the middle Layer or ~Proc-Layer contains [[assets|Asset]] and [[Session]], i.e. the user-visible data model and provides configuration and behaviour for these entities. Besides, he is responsible for [[building and configuring|Builder]] the [[render engine|RenderEngine]] based on the current Session state.
&rarr; UI-Layer

All Assets of kind asset::Proc represent //processing algorithms// in the bookkeeping view. They enable loading, browsing and maybe even parametrizing all the Effects, Plugins and Codecs available for use within the Lumiera Session.
Besides, they provide an __inward interface__ for the [[ProcNode]]s, enabling them to dispatch the actual processing call while rendering. Actually, this interface is always accessed via an ~Effect-MObject; mostly it is investigated and queried in the build process when creating the corresponding processor nodes. &rarr; see EffectHandling for details
{{red{todo: the naming scheme??}}}
[img[Asset Classess|uml/fig131077.png]]
{{red{Note 3/2010}}} it is very unlikely we'll organise the processing nodes as a class hierarchy. Rather it looks like we'll get several submodules/special capabilities configured in within the Builder

//The guard and coordinator of any operation within the session subsystem.//
The session and related components work effectively single threaded. Any tangible operation on the session data structure has to be enqueued as [[command|CommandHandling]] into the dispatcher. Moreover, the [[Builder]] is triggered from the ProcDispatcher; and while the Builder is running, any command processing is halted. The Builder in turn creates or reshapes the processing nodes network, and the changed network is brought into operation with a //transactional switch// -- while render processes on this processing network operate unaffected and essentially multi-threaded.
Enqueueing commands through the SessionCommandFacade into the ProcDispatcher is the official way to cause changes to the session. And the running state of the ProcDispatcher is equivalent with the running state of the //session subsystem as a whole.//
!Requirements
To function properly as action coordinator of the session subsystem, the dispatcher has to fulfil multiple demands
;enqueue
:accept and enqueue command messages concurrently, any time, without blocking the caller
:*FIFO for //regular commands//
:*LIFO for //priority requests// {{red{unimplemented 1/17}}}
;process
:dequeue and process entries sequentially
;sleep
:work continuously until queue is empty, then enter wait state
;check point
:arrive at a well defined check point reliably non blocking ("ensure to make progress")
:* necessary to know when internal state is consistent
:* when?
:** after each command
:** after builder run
:** after wake-up
;manage
:care for rectifying entries in the queue
:* ensure they //match// current session, discard obsoleted requests
:* //aggregate// similar requests
:* //supersede// by newer commands of a certain kind
!Operational semantics
The ProcDispatcher is a component with //running state.// There is some kind of working loop, which possibly enters a sleep state when idle. In fact, this loop is executed ''exclusively in the session thread''. This is the very essence of treating the session entirely single threaded, thus evading all the complexities of parallelism. Consequently, the session thread will either
* execute a command on the session
* perform the [[Builder]]
* evaluate loop control logic in the ProcDispatcher
* block waiting in the ProcDispatcher
Initially the command queue is empty and the ProcDispatcher can be considered idle. Whenever more commands are available in the queue, the dispatcher will handle them one after another, without delay, until the queue is emptied. Yet the Builder run need to be kept in mind. Essentially, the builder models a //dirty state:// whenever a command has touched the session, the corresponding LowLevelModel must be considered out of sync, possibly not reflecting the intended semantics of the session anymore. From a strictly logical view angle, we'd need to trigger the builder after each and every session command -- but it was a very fundamental design decision in Lumiera to allow for a longer running build process, more akin to running a compiler. This decision opens all the possibilities of integrating a knowledge based system and resolution activities to find a solution to match the intended session semantics. For this reason, we decouple the UI actions from session and render engine consistency, and we enqueue session commands, to throttle down the number of builder runs.
So the logic to trigger builder runs has to take some leeway into account. Due to the typical interactive working style of an editing application, session commands might be trickling in in strikes of similar commands, intermingled with tiny pauses. For this reason, the ProcDispatcher implements some //hysteresis,// as far as triggering the builder runs is concerned. The builder is fired in idle state, but only after passing some //latency period.// On the other hand, massive UI activities (especially during a builder run) may have flooded the queue, thus sending the session into an extended period of command processing. From the user's view angle, the application looks non responsive in such a case, albeit not frozen, since the UI can still enqueue further commands and thus retains the ability to react locally on user interaction. To mitigate this problem, the builder should be started anyway after some extended period of command processing, even if the queue is not yet emptied. Each builder run produces a structural diff message sent towards the UI and thus causes user visible changes within the session's UI representation. This somewhat stuttering response conveys to the user a tangible sensation of ongoing activity, while communicating at the same time, at least subconsciously some degree of operational overload. {{red{note 12/2016 builder is not implemented, so consider this planning}}}
Any change to the circumstances determining the ProcDispatcher's behaviour needs to be imparted actively through the public interface -- the dispatcher is not designed to be a state listener or observer. Any such state change notifications are synchronised and cause a wakeup notification to the session thread. For this purpose, enqueuing of further commands counts as state change and is lock protected. Beyond that, any other activities, like //processing// of commands or builder runs, are performed within the session thread without blocking other threads; the locking on the ProcDispatcher is only ever short term to ensure consistent internal state. Clients need to be prepared for the effect of actions to appear asynchronously and with some delay. Especially this means that session switch or shutdown has to await completion of any session command or builder run currently in progress.
When the session is closed or dismantled, further processing in the ProcDispatcher will be disabled, after completing the current command or builder run. This disabled state can be reversed when a new session instance becomes operative. And while the dispatcher will then continue to empty the command queue, most commands in queue will probably be obsoleted and dropped, because of referring to a deceased session instance. Moreover, the lifecycle of the session instances has to be distinguished from the lifecycle of the SessionSubsystem as such. When the latter is terminated, be it by a fatal error in some builder run, or be it due to general shutdown of the application, the ProcDispatcher will be asked to terminate the session thread after completing the current activity in progress. Such an event will also discard any further commands waiting in the dispatcher's queue.

The middle Layer of our current Architecture plan, i.e. the layer managing all processing and manipulation, while the actual data handling is done in the backend and the user interaction belongs to the GUI Layer.
&rarr; see the [[Overview]]

The Render Engine is the part of the application doing the actual video calculations. Relying on system level services and retrieving raw audio and video data through [[Lumiera's Backend|Backend]], its operations are guided by the objects and parameters edited by the user in [[the session|Session]]. The middle layer of the Lumiera architecture, known as the Proc-Layer, spans the area between these two exteremes, providing the the (abstract) edit operations available to the user, the representation of [["editable things"|MObjects]] and the translation of those into structures and facilities allowing to [[drive the rendering|Rendering]].
!About this wiki page
|background-color:#e3f3f1;width:96ex;padding:2ex; This TiddlyWiki is the central location for design, planning and documentation of the Lumiera Proc-Layer. Some parts are used as //extended brain// &mdash; collecting ideas, considerations and conclusions &mdash; while other tiddlers contain the decisions and document the planned or implemented facilities. The intention is to move over the more mature parts into the emerging technical documentation section on the [[Lumiera website|http://www.lumiera.org]] eventually. <br/><br/>Besides cross-references, content is largely organised through [[Tags|TabTags]], most notably <br/><<tag overview>> &middot; <<tag def>> &middot; <<tag decision>> &middot; <<tag Concepts>> &middot; <<tag GuiPattern>> <br/> <<tag Model>> &middot; <<tag SessionLogic>> &middot; <<tag GuiIntegration>> &middot; <<tag Builder>> &middot; <<tag Rendering>> &middot; <<tag Player>> &middot; <<tag Rules>> &middot; <<tag Types>> |
!~Proc-Layer Summary
When editing, the user operates several kinds of //things,// organized as [[assets|Asset]] in the AssetManager, like media, clips, effects, codecs, configuration templates. Within the context of the [[Project or Session|Session]], we can use these as &raquo;[[Media Objects|MObjects]]&laquo; &mdash; especially, we can [[place|Placement]] them in various kinds within the session and relative to one another.
Now, from any given configuration within the session, we create sort or a frozen- and tied-down snapshot, here called &raquo;[[Fixture|Fixture]]&laquo;, containing all currently active ~MObjects, broken down to elementary parts and made explicit if necessary. This Fixture acts as a isolation layer towards the Render Engine. We will hand it over to the [[Builder]], which in turn will transform it into a network of connected [[render nodes|ProcNode]]. This network //implements//&nbsp; the [[Render Engine|OverviewRenderEngine]].
The system is ''open'' inasmuch every part mirrors the structure of corresponding parts in adjacent subsystems, and the transformation of any given structure from one subsystem (e.g. Asset) to another (e.g. Render Engine) is done with minimal "magic". So the whole system should be able to handle completely new structures mostly by adding new configurations and components, without much need of rewriting basic workings.
!!see also
&rarr; [[Overview]] of Subsystems and Components, and DesignGoals
&rarr; [[An Introduction|WalkThrough]] discussing the central points of this design
&rarr; [[Overview Session (high level model)|SessionOverview]]
&rarr; [[Overview Render Engine (low level model)|OverviewRenderEngine]]
&rarr; BuildProcess and RenderProcess
&rarr; [[Two Examples|Examples]] (Object diagrams)
&rarr; how [[Automation]] works
&rarr; [[Problems|ProblemsTodo]] to be solved and notable [[design decisions|DesignDecisions]]
&rarr; [[Concepts, Abstractions and Formalities|Concepts]]
&rarr; [[Implementation Details|ImplementationDetails]] {{red{WIP}}}

A data processing node within the Render Engine. Its key feature is the possibility to pull from it one (freely addressable) [[Frame]] of calculated data. Further, each ~ProcNode has the ability to be wired with other nodes and [[Parameter Providers|ParamProvider]]
!! {{red{open questions}}}
* how to address a node
* how to type them
* how to discover the number and type of the ports
* how to discover the possible parameter ports
* how to define and query for additional capabilities
&rarr; see also the [[open design process draft|http://www.pipapo.org/pipawiki/Lumiera/DesignProcess/DesignRenderNodesInterface]]
&rarr; see [[mem management|ManagementRenderNodes]]
&rarr; see RenderProcess

This special type of [[structural Asset|StructAsset]] represents information how to build some part of the render engine's processing nodes network. Processing patterns can be thought of as a blueprint or micro program for construction. Most notably, they are used for creating nodes reading, decoding and delivering source media material to the render network, and they are used for building the output connection via faders, summation or overlay nodes to the global pipes (busses). Each [[media Asset|MediaAsset]] has associated processing patterns describing the codecs and other transformations needed to get at the media data of this asset. (and because media assets are typically compound objects, the referred ~ProcPatt will be compound too). Similarily, for each stream kind, we can retrieve a processing pattern for making output connections. Obviously, the possibilities opened by using processing patterns go far beyond.
Technically, a processing pattern is a list of building instructions, which will be //executed// by the [[Builder]] on the render node network under construction. This implies the possibility to define further instruction kinds when needed in future; at the moment the relevant sorts of instructions are
* attach the given sequence of nodes to the specified point
* recursively execute a nested ~ProcPatt
More specifically, a sequence of nodes is given by a sequence of prototypical effect and codec assets (and from each of them we can create the corresponding render node). And the point to attach these nodes is given by an identifier &mdash; in most cases just "{{{current}}}", denoting the point the builder is currently working at, when treating some placed ~MObject which in turn yielded this processing pattern in question.
Like all [[structural assets|StructAsset]], ~ProcPatt employs a special naming scheme within the asset name field, which directly mirrors its purpose and allows to bind to existing processing pattern instances when needed. {{red{TODO: that's just the general idea, but really it will rather use some sort of tags. Yet undefined as of 5/08}}} The idea is letting all assets in need of a similar processing pattern refer to one shared ~ProcPatt instance. For example, within a MPEG video media asset, at some point there will be a ~ProcPatt labeled "{{{stream(mpeg)}}}". In consequence, all MPEG video will use the same pattern of node wiring. And, of course, this pattern could be changed, either globally, or by binding a single clip to some other processing pattern (for making a punctual exception from the general rule)
!!defining Processing Patterns
The basic working set of processing patterns can be expected to be just there (hard wired or default configuration, mechanism for creating an sensible fallback). Besides, the idea is that new processing patterns can be added via rules in the session and then referred to by other rules controlling the build process. Any processing pattern is assembled by adding individual build instructions, or by including another (nested) processing pattern.
!!retrieving a suitable Processing Pattern
For a given situation, the necessary ProcPatt can be retrieved by issuing a [[configuration query|ConfigQuery]]. This query should include the needed capabilities in predicate form (technically this query is a Prolog goal), but it can leave out some pieces of information by just requesting "the default" &rarr; see DefaultsManagement
!!how does this actually work?
Any processing pattern needs the help of a passive holder tool suited for a specific [[building situation|BuilderPrimitives]]; we call these holder tools [[building moulds|BuilderMould]]. Depending on the situation, the mould has been armed up by the builder with the involved objects to be connected and extended. So, just by issuing the //location ID// defined within the individual build instruction (in most cases simply {{{"current"}}}), the processing pattern can retrieve the actual render object to use for building from the mould it is executed in.
!!errors and misconfiguration
Viewed as a micro program, the processing patterns are ''weak typed'' &mdash; thus providing the necessary flexibility within an otherwise strong typed system. Consequently, the builder assumes they are configured //the right way// &mdash; and will just bail out when this isn't the case, marking the related part of the high-level model as erroneous.
&rarr; see BuilderErrorHandling for details

a given Render Engine configuration is a list of Processors. Each Processor in turn contains a Graph of ProcNode.s to do the acutal data processing. In order to cary out any calculations, the Processor needs to be called with a StateProxy containing the state information for this RenderProcess

The Quantiser implementation works by determining the grid interval containing a given raw time.
These grid intervals are denoted by ordinal numbers (frame numbers), with interval #0 starting at the grid's origin and negative ordinals allowed.
!frame quantisation convention
Within Lumiera, there is a fixed convention how these frame intervals are to be defined (&rArr; [[time handling RfC|http://lumiera.org/documentation/devel/rfc/TimeHandling.html]])
[img[Lumiera's frame quantisation convention|draw/framePositions1.png]]
Especially, this convention is agnostic of the actual zero-point of the scale and allows direct length calculations and seamless sequences of intervals.
The //nominal coordinate// of an interval is also the starting point -- for automation keys frames we'll utilise special provisions.
!range limitation problems
because times are represented as 64bit integers, the time points addressable within a given scale grid can be limited, compared with time points addressable through raw (internal) time values. As an extreme example, consider a time scale with origin at {{{Time::MAX}}} -- such a scale is unable to represent any of the original scale's value above zero, because the resulting coordinates would exceed the range of the 64bit integer. Did I mention that 64bit micro ticks can represent about 300000 years?
Now the actual problem is that using 64bit integers already means pushing to the limit. There is no easy escape hatch, like using a larger scale data type for intermediaries -- it //is// the largest built-in type. Basically we're touching the topic of ''safe integer arithmetics'' here, which is frequently discussed as a security concern. The situation is as follows:
* every programmer promises "I'll do the checks when necessary" -- just to forget doing so in practice then.
* for C programming, the situation is hopeless. Calling functions for simple arithmetics is outright impractical -- that won't happen volountarily
* it is possible to build a ~SafeInt datatype in C++ though. While theoretically fine, in practice this also creates a host of problems:
** actually detecting all cases and coding the checks is surprisingly hard and intricate.
** the attempt to create a smooth integration with the built-in data types drives us right into one of the most problematic areas of C++
** the performance hit is considerable (factor 2 - 4) -- again luring people into "being clever".
There is an existing [[SafeInt class by David LeBlanc|http://safeint.codeplex.com/]], provided by Microsoft through the ~CodePlex platform. It is part of the libraries shipped with ~VisualStudio and extensively used in Office 2010 and Windows, but also provided under a somewhat liberal license (you may use it but any derived work has to reproduce copyright and usage terms). Likely this situation also hindered the further [[development|http://thread.gmane.org/gmane.comp.lib.boost.devel/191010]] of a comparable library in boost (&rarr; [[vault|https://svn.boost.org/trac/boost/wiki/LibrariesUnderConstruction#Boost.SafeInt]]).
!!!solution possibilities
;abort operation
:an incriminating calculation need to be detected somehow (see below).
:the whole usage context gets aborted by excaption in case of an alarm, similar to an out of memory...
:thus immediate corruption is avoided, but the user has to realise and avoid the general situation.
;limit values
:provide a limiter to kick in after detecting an alarm (see below).
:time values will just stick to the maximum/minimum boundary value....
:the rest of the application needs to be prepared for timing calculations to return degenerate results
;~SafeInt
:base ~TimeValue on a ~SafeInt<gavl_time_t>
:this way we could easily guarantee for detecting any //situation.//
:of course the price is the performance hit on all timing calculations.
:Moreover -- if we want to limit values instead of raising an exception, we'd need to write our own ~SaveInt.
;check some operations
:time values are mostly immutable, thus it would likely be sufficient only to check some strategically important points
:#parsing or de-serialising
:#calculations in ~TimeVar
:#quantisation
:#build Time from framecount
;limit time range
:because the available time range is so huge, it wouldn't hurt to reduce it by one or two decimals
:this way both the implementation of the checks could be simplified and the probability of an overflow reduced
;ignore the problem
:again, because of the huge time range, the problem could generally be deemed irrelevant
:we could apply a limiter to enforce a reduced time range just in the quantisation and evaluation of timecodes
A combination of the last approaches seems to be most appropriate here:
Limit the officially allowed time range and perform simple checks when quantising a time value and when applying a scale factor.
We limit {{{Time::MAX}}} by factor 1/30 and define the minimum //symmetrically to this.// This still leaves us with ''±9700 years allowed timerange''.

{{red{WIP as of 10/09}}}...//brainstorming about the first ideas towards a query subsystem//
!use case: discovering the contents of a container in the HighLevelModel
In the course of shaping the session API, __joel__ and __ichthyo__ realised that we're moving towards some sort of discovery or introspection. This gives rise to the quest for a //generic// pattern how to issue and run these discovery operations. The idea is to understand such a discovery as running a query &mdash; using this specific problem to shape the foundation of a query subsystem to come.
* a ''query'' is a polymorphic, noncopyable, non-singleton type; a query instance corresponds to one distinctly issued query
* issuing a query yields a result set, which is hidden within the concrete query implementation.
* the transactional behaviour needs still to be defined: how to deal with concurrent modifications? COW?
* the query instance remains property of the entity exposing the query capability.
* client code gets a result iterator, which can be explored //only once until exhaustion.//
* the handed out result iterator is used to manage the allocation for the query result set by sideeffect (smart handle). &rarr; Ticket #353
For decoupling the query invocation from the facility actually processing the query, we need to come up with common pattern. In 10/09, there is an immediate demand for such a solution pattern for implementing the QueryFocus and PlacementScope framework, which is crucial for contents discovery in general on the session interface. &rarr; QueryResolver was shaped to deal with this situation, but has the potential to evolve into a general solution for issuing queries.
!use case: retrieving object to fulfil a condition
The requirement is to retrieve one (or multiple) objects of a specific kind, located within a scope, and fulfilling some additional condition. For example: find a sub-track with {{{id(blubb)}}}.
This is a special case of the general discovery (described in the previous use case), but it is also a common situation in a general ConfigQuery (&rArr; an object of a specific type and with additional capabilities...). The tricky question seems to be how to specify and resolve these additional conditions or capabilities.
* in a generic query handled by a resolution engine, we might represent these capabilities as a nested //goal.//
* if we approach the problem as a filter pipeline, then the condition becomes a functor (or closure). &rarr; QueryResolver
On second consideration, these two approaches don't contradict each other, because they live in different contexts and levels of abstraction. Performance-wise, both are bad and degenerate on large models, because both effectively cause a full table scan. Only specialised search functions for hardcoded individual properties could improve that situation, by backing them with an additional sub-index.
__Conclusion__: no objection against providing the functor/filter solution right now, even on the QueryFocus API &mdash;
notwithstanding the fact we need a better solution later.
----
See also the notes on
&nbsp; &rarr; QueryImplProlog
&nbsp; &rarr; QueryRegistration

While there are various //specialised queries// to be issued and resolved efficiently, as a common denominator we use a common ''syntactic representation'' of queries based on predicate logic. This allows for standardised processing when applicable, since a //generic query// can be used as a substitute of any given specialised kind of query. Moreover, using the predicate logic format as common definition format allows for programmatic extension, reshaping, combining and generic meta processing of queries within the system.
!the textual syntactic form
{{red{WIP 12/2012}}} for the moment we don't use any kind of "real" resolution engine, thus the syntactic representation is kind of a design draft.
The plan is to use a syntax which can be fed directly to a Prolog interpreter or similar existing rules based systems.
!the necessity of using an internal representation
Since the intention is to use queries pervasively as a means of orchestrating the interplay of the primary controlling facilities, the internal representation of this syntactic exchange format might become a performance bottleneck. The typical usage pattern is just to create and issue a query to retrieve some result value -- thus, in practice, we rarely need the syntactic representation, while being bound to create this representation for sake of consistency.
!textual vs parsed-AST representation
For the initial version of the implementation, just storing a string in Prolog syntax is enough to get us going. But on the long run, for the real system, a pre-parsed representation in the style of an __A__bstract __S__yntax __T__ree seems more appropriate: through the use of some kind of symbol table, actual queries can be tagged with the common syntactic representation just by attaching some symbol numbers, without the overhead of creating, attaching and parsing a string representation in most cases

There is a common denominator amongst all queries: they describe a pattern of relations, which can be //satisfied// by providing a //solution.// But queries can be used in a wide variety of situations, each binding them to a more specific meaning and each opening the possibility to use a specialised resolution mechanism. For example, a query //might// actually mean to fetch and filter the contents of some sub-section of the session model. Or it might cause the fabrication an registration of some new model content object. In order to tie those disjoint kinds of queries together, we need a mechanism to exchange one kind of query with another one, or to derive one from another one. This ''exchange of queries'' actually is a way of ''remoulding'' and rebuilding a query -- which is based on a [[common syntactic representation|QueryDefinition]] of all queries, cast into terms of predicate logic. Besides, there is a common, unspecific, generic resolution mechanism ({{red{planned 11/12}}}) , which can delegate to more specialised resolvers when applicable.
!handling of queries
Queries are are always exposed or revealed at some point or facility allowing to pose queries. This facility remains the owner of the query instances and knows its concrete flavour (type). But queries can be referred to and exchanged through the {{{Query<TY>}}}-abstraction.

When querying contents of the session or sub-containers within the session, the QueryFocus follows the current point-of-query. As such queries can be issued to explore the content of container-like objects holding other MObjects, the focus is always attached to a container, which also acts as [[scope|PlacementScope]] for the contained objects. QueryFocus is an implicit state (the current point of interrest). This sate especially remembers the path down from the root of the HighLevelModel, which was used to access the current scope. Because this path constitutes a hierarchy of scopes, it can be relevant for querying and resolving placement properties. (&rarr; SessionStructureQuery)
!provided operations
* shift to a given scope-like object. Causes the current focus to //navigate//
* open a new focus, thereby pushing the existing focus onto a [[focus stack|QueryFocusStack]]
* return (pop) to the previous focus
* get the current scope, represented by the "top" Placement of this scope
* get the current ScopePath from root (session globals) down to the current scope
* (typed) content discovery query on the current scope
[>img[Scope Locating|uml/fig136325.png]]
&rarr; [[more|SessionContentsQuery]] regarding generic scope queries
!!!relation to Scope
There is a tight integration with PlacementScope through the ScopeLocator, which establishes the //current focus.// But while the [[scope|PlacementScope]] just decorates the placement defining a scope (called //&raquo;scope top&laquo;//), QueryFocus is more of a //binding// &mdash; it links or focusses the current state into a specific scope with a ScopePath in turn depending on this current state. Thus, while Scope is just a passive container allowing to locate and navigate, QueryFocus by virtue of this binding allows to [[Query]] at this current location.
!implementation notes
we provide a static access API, meaning that there is a singleton (the ScopeLocator) behind the scenes, which holds the mentioned scope stack. The current focus stack top, i.e. the current ScopePath is managed through an ref-counting handle embedded into each QueryFocus instance. Thus, effectively QueryFocus is an frontend object for accessing this state. Moreover, embedded into ScopeLocator, there is an link to the current session. But this link is kept opaque; it works by the current session exposing an [[query service|QueryResolver]], while QueryFocus doesn't rely on knowledge about the session, allowing the focus to be unit tested.
The stack of scopes must not be confused with the ScopePath. Each single frame on the stack can be seen and accessed as a QueryFocus and as such relates to a current ScopePath. The purpose of the stack is to make the scope handling mostly transparent; especially this stack allows to write dedicated query functions directed at a given object: they work by pushing and then navigating to the object to use as starting point for the query, i.e. the //current scope.//
!!!simplifications
The full implementation of this scope navigation is tricky, especially when it comes to determining the relation of two positions. It should be ''postponed'' and replaced by a ''dummy'' (no-op) implementation for the first integration round.

The ScopeLocator uses a special stack of ScopePath &raquo;frames&laquo; to maintain the //current focus.//
What is the ''current'' QueryFocus and why is it necessary? There is a state-dependent part involved, inasmuch the effective ScopePath depends on how the invoking client has navigated the //current location// down into the HighLevelModel structures. Especially, when a VirtualClip is involved, there can be discrepancies between the paths resulting when descending down through different paths. (See &rarr; BindingScopeProblem).
Thus, doing something with the current location, and especially descending or querying adjacent scopes can modify this current path state. Thus we need a means of invoking a query in a way not interfering with the current path state, otherwise we wouldn't be able to provide side-effect free query operations accessible on individual objects within the model.
!maintaining the current QueryFocus
As long as client code is just interested to use the current query location, we can provide a handle referring to it. But when a query needs to be run without side effect on the current location, we //push//&nbsp; it aside and start using a new QueryFocus on top, which starts out at a new initial location. Client code again gets a handle (smart-ptr) to this location, and additionally may access the new //current location.// When all references are out of scope and gone, we'll drop back to the focus put aside previously.
!implementation of ref-counting and clean-up
Actually, client code should use QueryFocus instances as frontend to access this &raquo;current focus&laquo;. Each ~QueryFocus instance incorporates a smart-ptr. But as in this case we're not managing objects allocated somewhere, we use an {{{boost::intrusive_ptr}}} and maintain the ref-count immediately within the target objects to be managed. These target objects are ScopePath instances and are living within the QueryFocusStack, which in turn is managed by the ScopeLocator singleton (see the UML diagram &rarr;[[here|QueryFocus]]). We use an (hand-written) stack implementation to ensure the memory locations of these ScopePath &raquo;frames&laquo; remain valid (and also to help with providing strong exception guarantees). The stack is aware of these ref-count and takes it into account on performing the {{{pop_unused()}}} operation: any unused frame on top will be evicted, stopping at the first frame still in use (which may be even just the old top). This cleanup also happens automatically when accessing the current top, re-initialising an potentially empty stack with a default-constructed new frame if necessary. This way, just accessing the stack top always yields the ''current focus location'', which thereby is //defined as the most recently used focus location still referred.//
!concurrency
This concept deliberately ignores parallelism. But, as the current path state is already encapsulated (and ref-counting is in place), the only central access point is to reach the current scope. Instead of using a plain-flat singleton here, this access can easily be routed through thread local storage.
{{red{As of 10/09 it is not clear if there will be any concurrent access to this discovery API}}} &mdash; but it seems not unlikely to happen...

//obviously, getting this one to work requires quite a lot of technical details to be planned and implemented.// This said...
The intention is to get much more readable ("declarative") and changeable configuration as by programming the decision logic literately within the implementation of some object.
!Draft
As an example, specifying how a Track can be configured for connecting automatically to some "mpeg" bus (=pipe)
{{{
resolve(O, Cap) :- find(O), capabilities(Cap).
resolve(O, Cap) :- make(O), capabilities(Cap).
capabilities(Q) :- call(Q).
stream(T, mpeg) :- type(T, track), type(P, pipe), resolve(P, stream(P,mpeg)), placed_to(P, T).
}}}
Then, running the goal {{{:-resolve(T, stream(T,mpeg)).}}} would search a Track object, try to retrieve a pipe object with stream-type=mpeg and associate the track with this pipe. This relies on a predicate "stream(P,mpeg)" implemented (natively) for the pipe object. So, "Cap" is the query issued from calling code &mdash; here {{{stream(T,mpeg)}}}, the type guard {{{type(T, track)}}} will probably be handled or inserted automatically, while the predicate implementations for find/1, make/1, stream/2, and placed_to/2 are to be provided by the target types.
* __The supporting system__ had to combine several code snippets into one rule system to be used for running queries, with some global base rules, rules injected by each individual participating object kind and finally user provided rules added by the current session. The actual query is bound to "Cap" (and consequently run as a goal by {{{call(Q)}}}). The implementation needs to provide a symbol table associating variable terms (like "T" or "P") to C/C++ object types, enabling the participating object kinds to register their specific predicate implementations. This is crucial, because there can be no general scheme of object-provided predicates (for each object kind different predicates make sense, e.g. [[pipes|PipeHandling]] have other possibilities than [[wiring requests|WiringRequest]]). Basically, a query issues a Prolog goal, which in turn evaluates domain specific predicates provided by the participating objects and thus calls back into C/C++ code. The supporting system maintains the internal connection (via the "type" predicate) such that from Prolog viewpoint it looks as if we were binding Variables directly to object instances. (there are some nasty technical details because of the backtracking nature of Prolog evaluations which need to be hidden away)
* Any __participating object kind__ needs a way to declare domain specific predicates, thus triggering the registration of the necessary hooks within the supporting system. Moreover, it should be able to inject further prolog code (as shown in the example above with the {{{strem(T, mpeg)}}} predicate. For each of these new domain specific predicates, there needs to be a functor which can be invoked when the C implementation of the predicate is called from Prolog (in some cases even later, when the final solution is "executed", e.g. a new instance has been created and now some properties need to be set).
!!a note on Plugins
In the design of the Lumiera Proc Layer done thus far, we provide //no possibility to introduce a new object kind// into the system via plugin interface. The system uses a fixed collection of classes intended to cover all needs (Clip, Effect, Track, Pipe, Label, Automation, ~Macro-Clips). Thus, plugins will only be able to provide new parametrisations of existing classes. This should not be any real limitation, because the whole system is designed to achieve most of its functionality by freely combining rather basic object kinds. As a plus, it plays nicely with any plain-C based plugin interface. For example, we will have C++ adapter classes for the most common sorts of effect plugin (pull system and synchronous frame-by-frame push with buffering) with a thin C adaptation layer for the specific external plugin systems used. Everything beyond this point can be considered "configuration data" (including the actual plugin implementation to be loaded)

//Querying for some suitable element,// instead of relying on hard wired dependencies, is considered a core pattern within Lumiera.
But we certainly can't expect to subsume every query situation to one single umbrella interface, so we need some kind of ''query dispatch'' and a registration mechanism to support this indirection. This could lead to a situation, where the term &raquo;query&laquo; is just a vaguely defined umbrella, holding together several disjoint subsistems. Yet, while in fact this describes the situation in the code base as of this writing, {{red{11/2012}}}, actually all kinds of queries are intended to share a //common semantic space.// So, in the end, we need a mechanism to [[exchange queries|QueryExchange]] with one another, and this mechanism ought to be based on the common //syntactic representation through terms of predicate logic.//
Common definition forma &rarr; QueryDefinition
Closely related is the &rarr; TypedQueryProblem
!Plans and preliminary implementation
As of 6/10, the intention is to be able just to //pose queries eventually.// Behind the scenes, a suitable QueryResolver should then be picked to process the query and yield a resultset. Thus the {{{Goal}}} and {{{Query<TY>}}} interfaces are to become the access point to a generic dispatching service and a bundle of specialised resolution mechanisms.
But implementing this gets a bit involved, for several reasons
* we don't know the kinds of queries, their frequency and performance requirements
* we don't know the exact usage pattern with respect to memory management of the resultsets.
* we can't asses the relevance of //lock contention,// created by using a central dispatcher facility.
We might end up with a full blown subsystem, and possibly with a hierarchy of dispatchers.
But for now the decision is to proceed with isolated and specialised QueryResolver subclasses, and to pass a basically suitable resolver to the query explicitly when it comes to retrieving results. This resolver should be obtained by some system service suitable for the concrete usage situation, like e.g. the ~SessionServiceExploreScope, which exposes a resolver to query session contents through the PlacementIndex. Nonetheless, the actual dispatch mechanism is already implemented (by using an ~MultiFact instance), and each concrete resolution mechanism is required to do an registration within the ctor, by calling the inherited {{{QueryResolver::installResolutionCase(..)}}}.

Within the Lumiera Proc-Layer, there is a general preference for issuing [[queries|Query]] over hard wired configuration (or even mere table based configuration). This leads to the demand of exposing a //possibility to issue queries// &mdash; without actually disclosing much details of the facility implementing this service. For example, for shaping the general session interface (in 10/09), we need a means of exposing a hook to discover HighLevelModel contents, without disclosing how the model is actually organised internally (namely by using an PlacementIndex).
!Analysis of the problem
The situation can be decomposed as follows.[>img[QueryResolver|uml/fig137733.png]]
* first off, we need a way to state //what kind of query we want to run.// This includes stipulations on the type of the expected result set contents
* as the requirement is to keep the facility actually implementing the query service hidden behind an interface, we're forced to erase specific type information and pass on an encapsulated version of the query
* providing an iterator for exploring the results poses the additional constraint of having an fairly generic iterator type, while still being able to communicate with the actual query implementation behind the interface.
!!!Difficulties
*the usage pattern is not clear &mdash; mostly it's just //planned//
*# client might create a specific {{{Query<TY>}}} and demand resolution
*# client might create just a goal, which is then translated into a specific query mechanism behind the invocation interface
*# client issues a query and expect it just to be handled by //some//&nbsp; suitable resolver
* thus it's difficult to determine, //what// part of the issued query needs automatic management. More specifically, is it possible for the client to dispose the query after issuing it, but keeping and exploring the iterator obtained as result of the query?
* and then there is the notorious problem of re-gaining the specifically typed context //behind//&nbsp; the invocation interface. Especially, the facility processing the query needs to know both the expected result type and details about the concrete query and its parametrisation. <br/>&rarr; TypedQueryProblem
!!!Entities and Operations
The //client// &nbsp;(code using query-resolver.hpp) either wants a ''goal'' or ''query'' to be resolved; the former is just implicitly typed and usually given in predicate logic from ({{red{planned as of 11/09}}}), while the latter may be a specialised subclass templated to yield objects of a specific type as results. A ''query resolver'' is an (abstracted) entity capable of //resolving//&nbsp; such a goal. Actually, behind the scenes there is somehow a registration of the concrete resolving facilities, which are asumed to decide about their ability of handling a given goal. Issuing a goal or query yields a ''resolution'' &mdash; practically speaking, a set of indivitual solutions. These individual solution ''results'' can be explored by ''iteration'', thereby moving an embedded ''cursor'' through the ''result set''. Any result can be retrieved at most once &mdash; after that, the resolution is ''exhausted'' and will be released automatically when the expolration iterator goes out of scope.
!!!Decisions
* while, in the use case currently at hand, the query instance is created by the client on the stack, the possibility of managing the queries internally is deliberately kept open. Because otherwise, we had to commit to a specific way of obtaining results, for example by assuming always to use an embedded STL iterator.
* we endorse that uttermost performance is less important than clean separation an extensibility. Thus we accept accessing the current position pointer through reference and we use a ref-counting mechanism alongside with the iterator to be handed out to the client
* the result set is not tied to the query &mdash; at least not by design. The query can be discarded while further exploring the result set.
* for dealing with the TypedQueryProblem, we require the concrete resolving facilities to register with a system startup hook, to build a dispatcher table on the implementation side. This allows us to downcast to the concrete Cursor type on iteration and results retrieval.
* the intention is to employ a mix of generic processing (through a common generic [[syntactic query representation|QueryDefinition]]) and optimised processing of specialised queries relying on concrete query subtypes. The key for achieving this goal is the registration menchanism, which could evolve into a generic query dispatch system -- but right now the exact balance of this two approaches remains a matter of speculation...

What is the Role of the asset::Clip and how exactly are Assets and (Clip)-MObjects related?
First of all: ~MObjects are the dynamic/editing/manipulation view, while Assets are the static/bookkeeping/searching/information view of the same entities. Thus, the asset::Clip contains the general configuration, the ref to the media and descriptive properties, while all parameters being "manipulated" belong to the session::Clip (MObject). Besides that, the practical purpose of asset::Clip is that you can save and remember some selection as a Clip (Asset), maybe even attach some information or markup to it, and later be able to (re)create a editable representation in the Session (the GUI could implement this by allowing to drag from the asset::Clip GUI representation to the timeline window)
!!dependencies
The session::Clip (frequently called "clip-MO", i.e. the MObject) //depends on the Asset.// It can't exist without the Asset, because the Asset is needed for rendering. The other direction is different: the asset::Clip knows that there is a dependant clip-MO, there could be //at most one// such clip-MO depending on the Asset, but the Asset can exist without the clip-MO (it gives the possibility to re-create the clip-MO).
!!deletions
When the Asset or the corresponding asset::Media is deleted, the dependant clip-MO has to disappear. And the opposite direction?
* Model-1: asset::Clip has a weak ref to the clip-MO. Consequently, the clip-MO can go out of scope and disappear, so the asset::Clip has to maintain the information of the clip's dimensions (source position and length) somewhere. Because of MultichannelMedia, this is not so simple as it may look at first sight.
* Model-2: asset::Clip holds a smart ptr to the clip-MO, thus effectively keeping it alive. __obviously the better choice__
In either case, we have to solve the ''problem of clip asset proliferation''
!!multiplicity and const-ness
The link between ~MObject and Asset should be {{{const}}}, so the clip can't change the media parameters. Because of separation of concerns, it would be desirable that the Asset can't //edit// the clip either (meaning {{{const}}} in the opposite direction as well). But unfortunately the asset::Clip is in power to delete the clip-MO and, moreover, handles out a smart ptr ([[Placement]]) referring to the clip-MO, which can (and should) be used to place the clip-MO within the session and to manipulate it consequently...
At first sight the link between asset and clip-MO is a simple logical relation between entities, but it is not strictly 1:1 because typical media are [[multichannel|MultichannelMedia]]. Even if the media is compound, there is //only one asset::Clip//, because in the logical view we have only one "clip-thing". On the other hand, in the session, we have a compound clip ~MObject comprised of several elementary clip objects, each of which will refer to its own sub-media (channel) within the compound media (and don't forget, this structure can be tree-like)
{{red{open question:}}} do the clip-MO's of the individual channels refer directly to asset::Media? does this mean the relation is different from the top level, where we have a relation to a asset::Clip??
{{red{Note 1/2015}}} several aspects regarding the relation of clips and single/multichannel media are not yet settled. There is a preliminary implementation in the code base, but it is not sure yet how multichnnel media will actually be modelled. Currently, we tend to treat the channel multiplicity rather as a property of the involved media, i.e we have //one// clip object.

Conceptually, the Render Engine is the core of the application. But &mdash; surprisingly &mdash; we don't even have a distinct »~RenderEngine« component in our design. Rather, the engine is formed by the cooperation of several components spread out over two layers (Backend and Proc-Layer): The [[Builder]] creates a network of [[render nodes|ProcNode]], the [[Scheduler]] triggers individual [[calculation jobs|RenderJob]], which in turn pull data from the render nodes, thereby relying on the [[Backend's services|Backend]] for data access and using plug-ins for the actual media calculations.
&rarr; OverviewRenderEngine
&rarr; EngineFaçade

The [[Render Engine|Rendering]] only carries out the low-level and performance critical tasks. All configuration and decision concerns are to be handled by [[Builder]] and [[Controller]]. While the actual connection of the Render Nodes can be highly complex, basically each Segment of the Timeline with uniform characteristics is handled by one Processor, which is a graph of [[Processing Nodes|ProcNode]] discharging into a ExitNode. The Render Engine Components as such are //stateless// themselves; for the actual calculations they are combined with a StateProxy object generated by and connected internally to the [[Controller]], while at the same time holding the Data Buffers (Frames) for the actual calculations.
{{red{Warning: obsolete as of 9/11}}}
Currently the Render/Playback is beeing targetted for implementation; almost everything in this diagram will be implemented in a slightly differently way....
[img[Entities comprising the Render Engine|uml/fig128389.png]]

Below are some notes regarding details of the actual implementation of the render process and processing node operation. In the description of the [[render node operation protocol|NodeOperationProtocol]] and the [[mechanics of the render process|RenderMechanics]], these details were left out deliberately.
{{red{WIP as of 9/11 -- need to mention the planning phase more explicitly}}}
!Layered structure of State
State can be seen as structured like an onion. All the [[StateAdapter]]s in one call stack are supposed to be within one layer: they all know of a "current state", which in turn is a StateProxy (and thus may refer yet to another state, maybe accros the network or in the backend or whatever). The actual {{{process()}}} function "within" the individual nodes just sees a single StateAdapter and thus can be thought to be a layer below.
!Buffer identification
For the purpose of node operation, Buffers are identified by a [[buffer-handle|BuffHandle]], which contains both the actual buffer pointer and an internal indes and classification of the source providing the buffer; the latter information is used for deallocation. Especially for calling the {{{process()}}} function (which is supposed to be plain C) the node invocation needs to prepare and provide an array containing just the output and input buffer pointers. Typically, this //frame pointer array//&nbsp; is allocated on the call stack.
!Problem of multi-channel nodes
Some data processors simply require to work on multiple channels simultanously, while others work just on a single channel and will be replicated by the builder for each channel invoved. Thus, we are struck with the nasty situation that the node graph may go through some nodes spanning the chain of several channels. Now the decision is //not to care for this complexity within a single chain calculating a single channel.// We rely solely on the cache to avoid duplicated calculations. When a given node happens to produce multiple output buffers, we are bound to allocate them for the purpose of this node's {{{process()}}} call, but afterwards we'r just "letting go", releasing the buffers not needed immediately for the channel acutally to be processed. For this to work, it is supposed that the builder has wired in a caching, and that the cache will hit when we touch the same node again for the other channels.
Closely related to this is the problem how to number and identify nodes and thus to be able to find calculated frames in cache (&rarr; [[here|NodeFrameNumbering]])
!Configuration of the processing nodes
[>img[uml/fig132357.png]]
Every node is actually decomposed into three parts
* an interface container of a ProcNode subclass
* an {{{const}}} WiringDescriptor, which is actually parametrized to a subtype encoding details of how to carry out the intended operation
* the Invocation state created on the stack for each {{{pull()}}} call. It is comprised of references to an StateAdapter object and the current overall process state, the WiringDescriptor, and finally a table of suitable buffer handles
Thus, the outer container can be changed polymorphically to support the different kinds of nodes (large-scale view). The actual wiring of the nodes is contained in the WiringDescriptor, including the {{{process()}}} function pointer. Additionally, this WiringDescriptor knows the actual type of the operation Strategy, and this actual type has been chosen by the builder such as to select details of the desired operation of this node, for example caching / no caching or maybe ~OpenGL rendering or the special case of a node pulling directly from a source reader. Most of this configuration is done by selecting the right template specialisation within the builder; thus in the critical path most of the calls can be inlined
!!!! composing the actual operation Strategy
As shown in the class diagram to the right, the actual implementation is assembled by chaining together the various policy classes governing parts of the node operation, like Caching, in-Place calculation capability, etc. (&rarr; see [[here|WiringDescriptor]] for details). The rationale is that the variable part of the Invocation data is allocated at runtime directly on the stack, while a precisely tailored call sequence for "calculating the predecessor nodes" can be defined out of a bunch of simple building blocks. This helps avoiding "spaghetti code", which would be especially dangerous and difficult to get right because of the large number of different execution paths. Additionally, a nice side effect of this implementation technique is that a good deal of the implementation is eligible to inlining.
We //do employ//&nbsp; some virtual calls for the buffer management in order to avoid coupling the policy classes to the actual number of in/out buffers. (As of 6/2008, this is mainly a precaution to be able to control the number of generated template instances. If we ever get in the region of several hundred individual specialisations, we'd need to separate out further variable parts to be invoked through virtual functions.)
!Rules for buffer allocation and freeing
* only output buffers are allocated. It is //never necessary//&nbsp; to allocate input buffers!
* buffers are to be allocated as late as possible, typically just before invoking {{{process()}}}
* buffers are allways allocated by activating a [[buffer handle|BuffHandle]], preconfigured already during the planning phase
* {{{pull()}}} returns a handle at least for the single output requested by this call, allowing the caller to retrieve the result data
* any other buffers filled with results during the same {{{process()}}} call can be released immediately before returning from {{{pull()}}}
* similar, any input buffers are to be released immediately after the {{{process()}}} call, but before returing from this {{{pull()}}}
* while any handle contains the necessary information for releasing or "committing" this buffer, this has to be triggered explicitly.
@@clear(right):display(block):@@

An unit of operation, to be [[scheduled|Scheduler]] for calculating media frame data just in time.
Within each CalcStream, render jobs are produced by the associated FrameDispatcher, based on the corresponding JobTicket used as blue print (execution plan).
!Anatomy of a render job
Basically, each render job is a //closure// -- hiding all the prepared, extended execution context and allowing the scheduler to trigger the job as a simple function.
When activated, by virtue of this closure, the concrete ''node invocation'' is constructed, which is a private and safe execution environment for the actual frame data calculations. This (mostly stack based) environment embodies a StateProxy, acting as communication hub for accessing anything possibly stateful within the larger scope of the currently ongoing render process. The node invocation sequence is what actually implements the ''pulling of data'': on exit, all cacluated data is expected to be available in the output buffers. Typically (but not necessarily) each node embodies a ''calculation function'', holding the actual data processing algorithm.
!{{red{open questions 2/12}}}
* what are the job's actual parameters?
* how is prerequisite data passed? &rarr; maybe by an //invocation key?//
!Input and closure
each job gets only the bare minimum information required to trigger the execution: the really variable part of the node's invocation. The job uses this pieces of information to re-activate a pre-calculated closure, representing a wider scope of environment information. Yet the key point is for this wider scope information to be //quasi static.// It is shared by a whole [[segment|Segmentation]] of the timeline in question, and it will be used and re-used, possibly concurrently. From the render job's point of view, the engine framework just ensures the availability and accessibility of all this wider scope information.
Prerequisite data for the media calculations can be considered just part of that static environment, as far as the node is concerned. Actually, this prerequisite data is dropped off by other nodes, and the engine framework and the builder ensure the availability of this data just in time.
!observations
* the job's scope represents a strictly local view
* the job doesn't need to know about its output
* the job doesn't need to know anything about the frame grid or frame number
* all it needs to know is the ''effective nominal time'' and an ''invocation instance ID''

While the render process, with respect to the dependencies, the builder and the processing function is sufficiently characterized by referring to the ''pull principle'' and by defining a [[protocol|NodeOperationProtocol]] each node has to adhere to &mdash; for actually get it coded we have to care for some important details, especially //how to manage the buffers.// It may well be that the length of the code path necessary to invoke the individual processing functions is finally not so important, compared with the time spent at the inner pixel loop within these functions. But my guess is (as of 5/08), that the overall number of data moving and copying operations //will be//&nbsp; of importance.
{{red{WIP as of 9/11 -- need to mention the planning phase more explicitly}}}
!requirements
* operations should be "in place" as much as possible
* because caching necessitates a copy, the points where this happens should be controllable.
* buffers should accommodate automatically to provide the necessary space without clipping the image.
* the type of the media data can change while passing through the network, and so does the type of the buffers.
On the other hand, the processing function within the individual node needs to be shielded from these complexities. It can expect to get just //N~~I~~// input buffers and //N~~O~~// output buffers of required type. And, moreover, as the decision how to organize the buffers certainly depends on non-local circumstances, it should be preconfigured while building.
!data flow
[>img[uml/fig131973.png]]
Not everything can be preconfigured though. The pull principle opens the possibility for the node to decide on a per call base what predecessor(s) to pull (if any). This decision may rely on automation parameters, which thus need to be accessible prior to requesting the buffer(s). Additionally, in a later version we plan to have the node network calculate some control values for adjusting the cache and backend timings &mdash; and of course at some point we'll want to utilize the GPU, resulting in the need to feed data from our processing buffers into some texture representation.
!buffer management
{{red{NOTE 9/11: the following is partially obsolete and needs to be rewritten}}} &rarr; see the BufferTable for details regarding new buffer management...
Besides the StateProxy representing the actual render process and holding a couple of buffer (refs), we employ a lightweight adapter object in between. It is used //for a single {{{pull()}}}-call// &mdash; mapping the actual buffers to the input and output port numbers of the processing node and for dealing with the cache calls. While the StateProxy manages a pool of frame buffers, this interspersed adapter allows us to either use a buffer retrieved from the cache as an input, possibly use a new buffer located within the cache as output, or (in case no caching happens) to just use the same buffer as input and output for "in-place"-processing. The idea is that most of the configuration of this adapter object is prepared in the wiring step while building the node network.
The usage patern of the buffers can be stack-like when processing nodes require multiple input buffers. In the standard case, which also is the simplest case, a pair of buffers (or a single buffer for "in-place" capable nodes) suffices to calculate a whole chain of nodes. But &mdash; as the recursive descent means depth-first processing &mdash; in case multiple input buffers are needed, we may encounter a situation where some of these input buffers already contain processed data, while we have to descend into yet another predecessor node chain to pull the data for the remaining buffers. Care has to be taken //to allocate the buffers as late as possible,// otherwise we could end up holding onto a buffer almost for each node in the network. Effectively this translates into the rule to allocate output buffers only after all input buffers are ready and filled with data; thus we shouldn't allocate buffers when //entering// the recursive call to the predecessor(s), rather we have to wait until we are about to return from the downcall chain.
Besides, these considerations also show we need a means of passing on the current buffer usage pattern while calling down. This usage pattern not only includes a record of what buffers are occupied, but also the intended use of these occupied buffers, especially if they can be modified in-place, and at which point they may be released and reused.
__note__: this process outlined here and below is still an simplification. The actual implementation has some additional [[details to care for|RenderImplDetails]]
!!Example: calculating a 3 node chain
# Caller invokes calculation by pulling from exit node, providing the top-level StateProxy
# node1 (exit node) builds StateAdapter and calls retrieve() on it to get the desired output result
# this StateAdapter (ad1) knows he could get the result from Cache, so he tries, but it's a miss
# thus he pulls from the predecessor node2 according to the [[input descriptor|ProcNodeInputDescriptor]] of node1
# node2 builds its StateAdapter and calls retrieve()
# but because StateAdapter (ad2) is configured to directly forward the call down (no caching), it pulls from node3
# node3 builds its StateAdapter and calls retrieve()
# this StateAdapter (ad3) is configured to look into the Cache...
# this time producing a Cache hit
# now StateAdapter ad2 has input data, but needs a output buffer location, which re requests from its //parent state// (ad1)
# and, because ad1 is configured for Caching and is "in-place" capable, it's clear that this output buffer will be located within the cache
# thus the allocation request is forwarded to the cache, which provides a new "slot"
# now node2 has both a valid input and a usable output buffer, thus the process function can be invoked
# and after the result has been rendered into the output buffer, the input is no longer needed
# and can be "unlocked" in the Cache
# now the input data for node1 is available, and as node1 is in-place-capable, no further buffer allocation is necessary prior to calculating
# the finished result is now in the buffer (which happens to be also the input buffer and is actually located within the Cache)
# thus it can be marked as ready for the Cache, which may now provide it to other processes (but isn't allowed to overwrite it)
# finally, when the caller is done with the data, it signalles this to the top-level State object
# which forwards this information to the cache, which in turn may now do with the released Buffer as he sees fit.
[img[uml/fig132229.png]]
@@clear(right):display(block):@@
__see also__
&rarr; the [[Entities involved in Rendering|RenderEntities]]
&rarr; additional [[implementation details|RenderImplDetails]]
&rarr; [[Memory management for render nodes|ManagementRenderNodes]]
&rarr; the protocol [[how to operate the nodes|NodeOperationProtocol]]

For each segment (of the effective timeline), there is a Processor holding the exit node(s) of a processing network, which is a "Directed Acyclic Graph" of small, preconfigured, stateless [[processing nodes|ProcNode]]. This network is operated according to the ''pull principle'', meaning that the rendering is just initiated by "pulling" output from the exit node, causing a cascade of recursive downcalls or prerequisite calculations to be scheduled as individual [[jobs|RenderJob]]. Each node knows its predecessor(s), thus the necessary input can be pulled from there. Consequently, there is no centralized "engine object" which may invoke nodes iteratively or table driven &mdash; rather, the rendering can be seen as a passive service provided for the backend, which may pull from the exit nodes at any time, in any order (?), and possibly multithreaded.
All State necessary for a given calculation process is encapsulated and accessible by a StateProxy object, which can be seen as the representation of "the process". At the same time, this proxy provides the buffers holding data to be processed and acts as a gateway to the backend to handle the communication with the Cache. In addition to this //top-level State,// each calculation step includes a small [[state adapter object|StateAdapter]] (stack allocated), which is pre-configured by the builder and serves the purpose to isolate the processing function from the detals of buffer management.
__see also__
&rarr; the [[Entities involved in Rendering|RenderEntities]]
&rarr; the [[mechanics of rendering and buffer management|RenderMechanics]]
&rarr; the protocol [[how to operate the nodes|NodeOperationProtocol]]

The rendering of input sources to the desired output ports happens within the &raquo;''Render Engine''&laquo;, which can be seen as a collaboration of Proc-Layer, Backend together with external/library code for the actual data manipulation. In preparation of the RenderProcess, the [[Builder]] as wired up a network of [[processing nodes|ProcNode]] called the ''low-level model'' (in contrast to the high-level model of objects placed within the session). Generally, this network is a "Directed Acyclic Graph" starting at the //exit nodes// (output ports) and pointing down to the //source readers.// In Lumiera, rendering is organized according to the ''pull principle'': when a specific frame of rendered data is requested from an exit node, a recursive calldown happens, as each node asks his predecessor(s) for the necessary input frame(s). This may include pulling frames from various input sources and for several time points, thus pull rendering is more powerful (but also more difficult to understand) than push rendering, where the process would start out with a given source frame.
Rendering can be seen as a passive service available to the Backend, which remains in charge what to render and when. Render processes may be running in parallel without any limitations. All of the storage and data management falls into the realm of the Backend. The render nodes themselves are ''completely stateless'' &mdash; if some state is necessary for carrying out the calculations, the backend will provide a //state frame// in addition to the data frames.

a special kind of [[render job|RenderJob]], used to retrieve input data relying on external IO.
Since a primary goal for the design of Lumiera's render engine is to use the available resources effectively, we try to avoid the situation where a working thread gets blocked waiting for external IO to deliver data. Thus we create special marker jobs to keep track of any prerequisites and start the actual render calculations only if all these prerequisites are already fulfilled and all the required input data is available in RAM.

A distinct property of the Lumiera application is to rely on a rules based approach rather then on hard wired logic. When it comes to deciding and branching, a [[Query]] is issued, resulting either immediately in a {{{bool}}} result, or creating a //binding// for the variables used within the query. Commonly, there is more than one solution for a given query, allowing the result set to be enumerated.
!current state {{red{WIP as of 10/09}}}
We are still fighting to get the outline of the application settled down.
For now, the above remains in the status of a general concept and typical solution pattern: ''create query points instead of hard wiring things''.
Later on we expect a distinct __query subsystem__ to emerge, presumably embedding a YAP Prolog interpreter.

A facility allowing the Proc-Layer to work with abstracted [[media stream types|StreamType]], linking (abstract or opaque) [[type tags|StreamTypeDescriptor]] to an [[library|MediaImplLib]], which provides functionality for acutally dealing with data of this media stream type. Thus, the stream type manager is a kind of registry of all the external libraries which can be bridged and accessed by Lumiera (for working with media data, that is). The most basic set of libraries is instelled here automatically at application start, most notably the [[GAVL]] library for working with uncompressed video and audio data. //Later on, when plugins will introduce further external libraries, these need to be registered here too.//

A scale grid controls the way of measuring and aligining a quantity the application has to deal with. The most prominent example is the way to handle time in fixed atomic chunks (''frames'') addressed through a fixed format (''timecode''): while internally the application uses time values of sufficiently fine grained resolution, the acutally visible timing coordinates of objects within the session are ''quantised'' to some predefined and fixed time grid.
&rarr; QuantiserImpl

The [[Scheduler]] is responsible for geting the individual [[render jobs|RenderJob]] to run. The basic idea is that individual render jobs //should never block// -- and thus the calculation of a single frame might be split into several jobs, including resource fetching. This, together with the data exchange protocol defined for the OutputSlot, and the requirements of storage management (especially releasing of superseded render nodes &rarr; FixtureStorage), leads to certain requirements to be ensured by the scheduler:
;ordering of jobs
:the scheduler has to ensure all prerequisites of a given job are met
;job time window
:when it's not possible to run a job within the defined target time window, it must be marked as failure
;failure propagation
:when a job fails, either due to an job internal error, or by timing glitch, any dependent jobs need to receive that failure state
;guaranteed execution
:some jobs are marked as "ensure run". These need to run reliable, even when prerequisite jobs fail -- and this failure state needs to be propagated
!detecting termination
The way other parts of the system are built, requires us to obtain a guaranteed knowledge of some job's termination. It is possible to obtain that knowledge with some limited delay, but it nees to be absoultely reliable (violations leading to segfault). The requirements stated above assume this can be achieved through //jobs with guaranteed execution.// Alternatively we could consider installing specific callbacks -- in this case the scheduler itself has to guarantee the invocation of these callbacks, even if the corresponding job fails or is never invoked. It doesn't seem there is any other option.

A link to relate a compound of [[nested placement scopes|PlacementScope]] to the //current// session and the //current//&nbsp; [[focus for querying|QueryFocus]] and exploring the structure. ScopeLocator is a singleton service, allowing to ''explore'' a [[Placement]] as a scope, i.e. discover any other placements within this scope, and allowing to locate the position of this scope by navigating up the ScopePath finally to reach the root scope of the HighLevelModel.
In the general case, this user visible high-level-model of the [[objects|MObject]] within the session allows for more than tree-like associations, as a given [[Sequence]] might be bound into multiple [[timelines|Timeline]]. Effectively, this makes the ScopePath context dependent. The ScopeLocator is the point where the strictly tree-like hierarchy of placements is connected to this more elaborate scope and path structure. To this end, ScopeLocator maintaines a QueryFocusStack, to keep track of the current location in focus, in cooperation with the QueryFocus objects used by client code.
&rarr; see BindingScopeProblem
&rarr; see TimelineSequences
!!a note about concurrency
While there //is// a "current state" involved, the effect of concurrent access deliberately remains unspecified, because access is expected to be serialised on a higher level. If this assumption were to break, then probably the ScopeLocator would involve some thread local state.

The sequence of nested [[placement scopes|PlacementScope]] leading from the root (global) scope down to a specific [[Placement]] is called ''scope path''. Ascending this path yields all the scopes to search or query in proper order to be used when resolving some attribute of placement. Placements use visibility rules comparable to visibility of scoped definitions in common programming languages or in cascading style sheets, where a local definition can shadow a global one. In a similar way, properties not defined locally may be resolved by querying up the sequence of nested scopes.
A scope path is a sequence of scopes, where each scope is implemented by a PlacementRef pointing to the &raquo;scope top&laquo;, i.e. the placement in the session //constituting this scope.// Each Placement is registered with the session as belonging to a scope, and each placement can contain other placements and thus form a scope. Thus, the ''leaf'' of this path can be considered the current scope. In addition to some search and query functions, a scope path has the ability to ''navigate'' to a given target scope, which must be reachable by ascending and descending into the branches of the overall tree or DAG. Navigating changes the current path. ({{red{WIP 11/09}}} navigation to scopes outside the current path and the immediate children of the current leaf is left out for now. We'll need it later, when actually implementing meta-clips. &rarr; see BindingScopeProblem)
!access path and session structure
ScopePath represents an ''effective scoping location'' within the model &mdash; it is not necessarily identical to the storage structure (&rarr; PlacementIndex) used to organise the session. While similar in most cases, binding a sequence into multiple timelines or meta-clips will cause the internal and the logical (effective) structure to digress (&rarr; BindingScopeProblem). An internal singleton service, the ScopeLocator is used to constitute the logical (effective) position for a given placement (which in itself defines a position in the session datastructure). This translation involves a [[current focus|QueryFocus]] remembering the access path used to reach the placement in question. Actually, this translation is built on top of the //navigation//-Operation of ScopePath, which thus forms the foundation to provide such a logical view on the "current" location.
!Operations
* the default scope path contains just the root (of the implicit PlacementIndex, i.e. usually the root of the model in the session)
* a scope path can be created starting from a given scope. This convenience shortcut uses the ScopeLocator to establish the position of the given start scope. This way, effectively the PlacementIndex within the current session is queried for parentship relations until reaching the root of the HighLevelModel.
* paths are ''copyable value objects'' without identity on their own
* there is a special //invalid//&nbsp; path token, {{{ScopePath::INVALID}}}
* length, validity and empty check
* paths are equality comparable
* relations
** if a scope in question is contained in the path
** if a scope in question is at the leaf position of the path
** if a path in question is prefix (contained) in the given path
** if two paths share a common prefix
** if two paths are disjoint (only connected at root)
* navigation
** move up one step
** move up to the root
** navigate to a given scope
** clear a path (reset to default)

An implementation mechanism used within the PlacementIndex to detect some specific kinds of object connections(&raquo;MagicAttachment&laquo;), which then need to trigger a special handling.
{{red{planned feature as of 6/2010)}}}
//It is not clear yet, if that's just an implementation facility, or something which is exposed through the ConfigRules.//
!preliminary requirements
We need to detect attaching and detaching of
* root &harr; BindingMO
* root &harr; [[Fork]]

//Segmentation of timeline// denotes a data structure and a step in the BuildProcess.
When [[building the fixture|BuildFixture]], ~MObjects -- as handled by their Placements -- are grouped below each timeline using them; Placements are then to be resolved into [[explicit Placements|ExplicitPlacement]], resulting in a single well defined time interval for each object. This allows to cut this effective timeline into slices of constant wiring structure, which are represented through the ''Segmentation Datastructure'', a time axis with segments holding object placements and [[exit nodes|ExitNode]]. &nbsp;&rarr; see [[structure of the Fixture|Fixture]]
* for each Timeline we get a Segmentation
** which in turn is a list of non-overlapping segments
*** each holding
**** an ExplicitPlacement for each object touching that time interval
**** an ExitNode for each ModelPort of the corresponding timeline
!Usage pattern
;(1) build process
:&rarr; a tree walk yields the placements per timeline, which then get //resolved//
:&rarr; after //sorting,// the segmentation can be established, thereby copying placements spanning multiple segments
:&rarr; only //after running the complete build process for each segment,// the list of model ports and exit nodes can be established
;(2) commit stage
: -- after the build process(es) are completed, the new fixture gets ''committed'', thus becoming the officially valid state to be rendered. As render processes might be going on in parallel, some kind of locking or barrier is required. It seems advisable to make the change into a single atomic hot-swap. Meaning we'd get a single access point to be protected. But there is another twist: We need to find out which render processes to cancel and restart, to pick up the changes introduced by this build process -- which might include adding and deleting of timelines as a whole, and any conceivable change to the segmentation grid. Because of the highly dynamic nature of the placements, on the other hand it isn't viable to expect the high-level model to provide this information. Thus we need to find out about a ''change coverage'' at this point. We might expand on that idea to //prune any new segments which aren't changed.// This way, only a write barrier would be necessary on switching the actually changed segments, and any render processes touching these would be //tainted.// Old allocations could be released after all tainted processes are known to be terminated.
;(3) rendering use
:Each play/render process employs a ''frame dispatch step'' to get the right exit node for pulling a given frame (&rarr; [[Dispatcher|FrameDispatcher]]). From there on, the process proceeds into the [[processing nodes|ProcNode]], interleaved with backend/scheduler actions due to splitting into individually scheduled jobs. The storage of these processing nodes and accompanying wiring descriptors is hooked up behind the individual segments, by sharing a common {{{AllocationCluster}}}. Yet the calculation of individual frames also depends on ''parameters'' and especially ''automation'' linked with objects in the high-level model. It is likely that there might be some sharing or some kind of additional communication interface, as the intention was to allow ''live changes'' to automated values. <br/>{{red{WIP 12/2010}}} details need to be worked out. &rarr; [[parameter wiring concept|Wiring]]
!!!observations
* Storage and initialisation for explicit placements is an issue. We should strive at making that inline as much as possible.
* the overall segmentation emerges from a sorting of time points, which are start points of explicit placements
* after the segmentation has been built, the usage pattern changes entirely into a lookup of segment by time
* the individual segments act as umbrella for a lot of further objects hooked up behind.
* we need the ability to exchange or swap-in whole segments
* each segment controls an AllocationCluster
* we need to track processes for tainting
* access happens per ModelPort
!!!conclusions
The Fixture is mostly comprised of the Segementation datastructure, but some other facilities are involved too
# at top level, access is structured by groups of model ports, actually grouped by timeline. This first access level is handled by the Fixture
# during the build process, there is a collecting and ordering of placements; these intermediaries as well as the initial collection can be discarded afterwards
# the backbone of the segmentation is closely linked to an ordering by time. Initially it should support sorting, access by time interval search later on.
# discarding a segment (or failing to do so) has an high impact on the whole application. We should employ a reliable mechanism for that.
# the frame dispatch and the tracking of processes can be combined; data duplication is a virtue when it comes to parallel processes
# the process of comparing and tainting is broken out into a separate data structure to be used just once
Largely the storage of the render nodes network is hooked up behind the Fixture &rarr; [[storage considerations|FixtureStorage]]

A sequence is a collection of media objects, arranged onto a fork ("track tree"). Sequences are the building blocks within the session. To be visible and editable, a session needs to be bound into a top-level [[Timeline]]. Alternatively, it may be used as a VirtualClip nested within another sequence.
The sequences within the session establish a //logical grouping//, allowing for lots of flexibility. Actually, we can have several sequences within one session, and these sequences can be linked together or not, they may be arranged in temporal order or may constitute a logical grouping of clips used simultaneously in compositional work etc. The data structure comprising a sequence is always a sub-tree of tracks, attached allways directly below root (Sequences at sub-nodes are deliberately disallowed). Through the sequence as frontend, this track tree might be used at various places in the model simultaneously. Tracks in turn are only an organisational (grouping) device, like folders &mdash; so this structure of sequences and track trees referred through them allows to use the contents of such a track or folder at various places within the model. But at any time, we have exactly one [[Fixture]], derived automatically from all sequences and containing the content actually to be rendered.
&rarr; see considerations about [[the role of Tracks and Pipes in conjunction with the sequences|TrackPipeSequence]]
!!Implementation and lifecycle
Actually, sequences are façade objects to quite some extent, delegating the implementation of the exposed functionality to the relevant placements and ~MObjects within the model. But they're not completely shallow; each sequence has an distinguishable identity and may hold additional meta-data. Thus, stressing this static aspect, sequences are implemented as StructAsset, attached to the [[model root|ModelRootMO]] through the AssetManager, simultaneously registered with the session, then accessed and owned by ref-counting smart-ptr.
A sequence is always tied to a root-placed track, it can't exist without such. When moving this track by changing it's [[Placement]], thus disconnecting it from the root scope, the corresponding sequence will be automatically removed from the session and discarded. On the other hand, sequences aren't required to be //bound// &mdash; a sequence might just exist in the model (attached by its track placed into root scope) and thereby remain passive and invisible. Such an unbound sequence can't be edited, displayed in the GUI or rendered, it is only accessible as asset. Of course it can be re-activated by linking it to a new or existing timeline or VirtualClip.
&rarr; see detailed [[discussion of dependent objects' behaviour|ModelDependencies]]

A helper to implement a specific memory management scheme for playback and rendering control data structures.
In this context, model and management data is structured into [[Segments|Segmentation]] of similar configuration within the project timeline. Beyond logical reasoning, these segments also serve as ''extents'' for memory allocation. Which leads to the necessity of [[segment related memory management|FixtureStorage]]. The handling of actual media data buffers is outside the realm of this topic; these are managed by the frame cache within the backend.
When addressing this task, we're facing several closely related concerns.
;throughput
:playback operations are ongoing and incur a continuous memory consumption.Thus, we need to keep up a minimal clean-up rate
;availability
:the playback system operates time bound. "Stop the World" for clean-up isn't an option
;contention
:playback and rendering operations are essentially concurrent. We need reliable yet decentralised bookkeeping
!sequence points
A ''sequence point'' is a conceptual entity for the purpose of organisation of dependent operations. Any sequence point can be ''declared'', ''referred'', ''fulfilled'' and ''triggered''. The referral to sequence points creates an ordering -- another sequence point can be defined as being prerequisite or dependent. But the specific twist is: any of these operations can happen //in any thread.// Triggering a sequence point means to invoke an action (functor) tied to that point. This is allowed only when we're able to prove that this sequence point has been fulfilled, which means that all of its prerequisites have been fulfilled and that optionally an additional fulfilment signal was detected. After the triggering, a sequence point ceases to exist.
!solution idea
The solution idea is inspired by the pattern of operation within a garbage collector: The key point to note with this pattern is the ability to detect the usage status by external reasoning, without explicit help from //within// the actual context of usage. In a garbage collector, we're reasoning about reachability, and we're doing so at our own discretion, at some arbitrary point in time, later, when the opportunity for collecting garbage is exploited.
For the specific problem for handling sequence points as outlined above, a similar structure can be established by introducing a concept of ''water level''. When we're able to prove a certain water level, any sequence points below that level must have been fulfilled. And for modern computing architectures the important point is that we're able to do this reasoning for each thread separately, based just on local information. Once a given thread has proven a certain water level, this conclusion is published in a lock free manner -- meaning that this information will be available in any other thread //eventually, after some time.// After that, any triggers below water level can be performed in correct dependency order, any time and in any thread, just as we see fit.
!!!complications
The concurrent nature of the problem is what makes this simple task somewhat more involved. Also note that the "water level" cannot be a global property, since the graph of dependencies is not necessarily globally connected. In the general case, it's not a tree, but a wood.
* information about fulfilling a sequence point may appear in various threads
* referrals to already known sequence points might be added later, and also from different threads ({{red{WIP 3/14}}} not sure if we need to expand on this observation -- see CalcStream &hArr; Segment)
This structure looks like requiring a message passing approach: we can arrange for determining the actual dependencies and fulfillment state late, within a single consumer, which is responsible for invoking the triggers. The other clients (threads) just pass messages into a possibly lock-free messageing channel.

The Session contains all information, state and objects to be edited by the User. From a users view, the Session is synonymous to the //current Project//. It can be [[saved and loaded|SessionLifecycle]]. The individual Objects within the Session, i.e. Clips, Media, Effects, are contained in one (or several) collections within the Session, which we call [[Sequence]].
&rarr; [[Session design overview|SessionOverview]]
&rarr; Structure of the SessionInterface
!Session structure
The Session object is a singleton &mdash; actually it is a »~PImpl«-Facade object (because the actual implementation object can be swapped for (re)loading Sessions).<br/>The Session is the access point to the HighLevelModel; it is comprised of
* a number of (independent) top-level [[time-lines|Timeline]]
* some [[sequences|Sequence]] to be used within these timelines
* a [[scope structure|PlacementScope]] backed by an index, and a current QueryFocus
* a set of ConfigRules to guide default behaviour {{red{planned as of 10/09}}}
* the ''Fixture'' with a possibility to [[(re)build it|PlanningBuildFixture]] {{red{just partially designed as of 01/09}}}
* the [[Asset subsystem|AssetManager]] is tightly integrated; besides, there are some SessionServices for internal use
&rarr; see [[relation of timeline, sequences and objects|TimelineSequences]]
&rarr; see //clarification of some fine points// regarding [[relation of Assets and MObjects|AssetModelConnection]]
!Session lifecycle
The session lifecycle need to be distinguished from the state of the [[session subsystem|SessionSubsystem]]. The latter is one of the major components of Lumiera, and when it is brought up, the {{{SessionCommandFacade}}} is opened and the ProcDispatcher started. On the other hand, the session as such is a data structure and pulled up on demand, by the {{{SessionManager}}}. Whenever the session is fully populated and configured, the ProcDispatcher is instructed to //actually allow dispatching of commands towards the session.// This command dispatching mechanism is the actual access point to the session for clients outside Proc-Layer; when dispatching is halted, commands can be enqueued non the less, which allows for a reactive UI.

LayerSeparationInterface, provided by the Proc-Layer.
The {{{SessionCommand}}} façade and the corresponding {{{proc::control::SessionCommandService}}} can be considered //the public interface to the session://
They allow to send [[commands|CommandHandling]] to work on the session data structure. All these commands, as well as the [[Builder]], are performed in a dedicated thread, the »session loop thread«, which is operated by the ProcDispatcher. As a direct consequence, all mutations of the session data, as well as all logical consequences determined by the builder, are performed single-threaded, without the need to care for synchronisation issues. Another consequence of this design is the fact that running the builder disables session command processing, causing further commands to be queued up in the ProcDispatcher. Any structural changes resulting from builder runs will finally be pushed back up into the UI, asynchronously.

While the core of the persistent session state corresponds just to the HighLevelModel, there is additionaly attached state, annotations and specific bindings, which allow to connect the session model to the local application configuration on each system. A typical example would be the actual output channels, connections and drivers to use on a specific system. In a Studio setup, these setup and wiring might be quite complex, it may be specific to just a single project, and the user might want to work on the same project on different systems. This explains why we can't just embody these configuration information right into the actual model.

Querying and retrieving objects within the session model is always bound to a [[scope|PlacementScope]]. When using the //dedicated API,// this scope is immediately defined by the object used to issue the query, like e.g. when searching the contents of a fork ("track" or "media bin"). But when using the //generic API,// this scope is rather implicit, because in this case a (stateful) QueryFocus object is used to invoke the queries. Somewhat in-between, the top-level session API itself exposes dedicated query functions working on the whole-session scope (model root).
Based on the PlacementIndex, the treelike scope structure can be explored efficiently; each Placement attached to the session knows its parent scope. But any additional filtering currently is implemented on top of this basic scope exploration, which obviously may degenerate when searching large scopes and models. Filtering may happen implicitly; all scope queries are parametrised to a specific kind of MObject, while the PlacementIndex deals with all kinds of {{{Placement<MObject>}}} uniformly. Thus, more specifically typed queries automatically have to apply a type filter based on the RTTI of the discovered placements. The plan is later to add specialised sub-indices and corresponding specific query functions to speed up the most frequently used kinds of queries.
!Ways to query
;contents query
:running a ~ContentsQuery<TY> means to traverse the given start scope depth-first, the order of the retrieved elements is implementation defined (hashtable).
;exploration
:this special parametrisation of the ~ContentsQuery is confined to retrieving the immediate children of the start scope, and intended for exploring and processing the model contents in specific ways. The implementation is based on the bucket iterators of the reverse index (aka scope table).
;parent path
:this query proceeds in the other direction, ascending from the given start scope up to the model root
;special contents
:a ~ContentsQuery<TY> may be combined with a predicate supplied by the client code; this predicate is applied after the general type filter, allowing to use the special interface of the subtype. In a large model this may degenerate considerably.
;picking
:to pick by query means to retrieve the first object returned by the underlying query. As the oder of retrieval is implementation defined, the results may be unpredictable. Typically, this is used under the assumption that the given query can only yield a single result. Beware, the {{{pick(predicate)}}}-functions exposed on the QueryFocus and top-level session API moreover assume there will be a result, and throw when this assumption is broken. The type filter in this case is derived automatically (at compile time) from the provided predicate.

The [[Session]] is interconnected with the GUI, the SessionStorage, [[Builder]] and the CommandHandling. The HighLevelModel is an conceptual view of the session. All these dependencies are isolated from the actual data layout in memory, but the latter is shaped by the intended interactions.
{{red{WIP...}}}Currently as of 3/10, this is an ongoing implementation and planning effort
!Objects, Placements, References
Media objects are attached to the session by [[placements|Placement]]. A Placement within the session gets an distinguishable identity (&rarr; ModelObjectIdentity) and behaves like being an instance of the attached object. Client code usually interacts with the compound of placement + ~MObject. In order to decouple this interaction from the actual implementation within the session, client code rather deals with //references.// These are implemented like a smart-ptr, but based on an opaque hash value, which is equivalent to the //object instance identity.//
&rarr; MObjectRef
&rarr; PlacementRef
!Index of placements attached to the session
For implementation, the HighLevelModel can be reduced to a compound of interconnected placements. These placement instances are owned and managed by the session; attaching a placement means copying it into the session, thereby creating a new placement-ID. A lookup mechanism for placements and placement relations (PlacementIndex) thus is the actual core of the session data structure; the actual object instances are maintained by a pooling custom allocator ({{red{planned as of 1/10}}}).
!Lifecycle
MObject lifetime is managed by reference counting; all placements and client side references to an MObject share ownership. The placement instances attached to the session are maintained by the index; thus, as long as an placement exists, the corresponding object automatically stays alive. Similarly, assets, as managed by shared-ptrs, stay alive when directly referenced, even after being dropped from the AssetManager. A bare PlacementRef on the other hand doesn't guarantee anything about the referred placement; when dereferencing this reference token, the index is accessed to re-establish a connection to the object, if possible. The full-fledged MObjectRef is built on top of such a reference token and additionally incorporates a smart-ptr. For the client code this means, that holding a ref ensures existence of the object, while the //placement// of this object still can get removed from the session.
!Updates and dependent objects
The session and the models rely on dependent objects beeing kept updated and consistent. This problem can't be solved in a generic fashion &mdash; at least not without using a far more elaborate scheme (e.g. a transaction manager), which is beyond the scope here. Thus, care has to be taken on the implementation level, especially in conjunction with error handling and threading considerations
&rarr; see [[details here...|ModelDependencies]]

"Session Interface" has several meanings, depending on the context.
;application global
:the session is a data structure, which can be saved and loaded, and manipulated by [[sending commands|CommandHandling]]
;within ~Proc-Layer
:here »the session« can be seen as a compound of several interfaces and facilities,
:together forming the primary access point to the user visible contents and state of the editing project.
:* the API of the session class
:* the accompanying management interface (SessionManager API)
:* the primary public ~APIs exposed on the objects to be [[queried and retrieved|SessionStructureQuery]] via the session class API
:** [[Timeline]]
:** [[Sequence]]
:** [[Placement]]
:** [[Clip]]
:** [[Fork]]
:** Effect
:** Automation
:* the [[command handling framework|CommandHandling]], including the [[UNDO|UndoManager]] facility
__Note__: the SessionInterface as such is //not a [[external public interface|LayerSeparationInterfaces]].// Clients from outside Proc-Layer can talk to the session by issuing commands through the {{{SessionCommandFacade}}}. Processing of commands is coordinated by the ProcDispatcher, which also is responsible for starting the [[Builder]].
!generic and explicit API
The HighLevelModel exposes two kinds of interfaces (which are interconnected and rely on each other): A generic, but somewhat low-level API, which is good for processing &mdash; like e.g. for the builder or de-serialiser &mdash; and a more explicit API providing access to some meaningful entities within the model. Indeed, the latter (explicit top level entities) can be seen as a ''façade interface'' to the generic structures:
* the [[Session]] object itself corresponds to the ModelRootMO
* the one (or multiple) [[Timeline]] objects correspond to the BindingMO instances attached immediately below the model root
* the [[sequences|Sequence]] bound into these timelines (by the ~BindingMOs) correspond to the top level [[Fork]]-~MObjects within each of these sequences.
[<img[Object relations on the session façade|draw/sessionFacade1.png]]
Thus, there is a convenient and meaningful access path through these façade objects, which of course actually is implemented by forwarding to the actual model elements (root, bindings, forks)
Following this access path down from the session means using the ''dedicated'' API on the objects retrieved.
To the contrary, the ''generic'' API is related to a //current location (state),// the QueryFocus.
!purpose of these ~APIs
* to discover
** by ID
** by type (filter)
** all contained
* to add
* to destroy
!!relation of [[Assets|Asset]] and MObjects
{{red{WARNING -- still under construction 10/2018}}}. Basically these segments represent the flip sides of the same coin. //Assets relate to the bookkeeping view.// However, we build a data model, and thus use representations for the involved entities. This creates some redundancy at times; we made an effort to reduce this redundancy and minimise the necessary data model representation. This means, some things are rather handled and represented as Assets, while other stuff is primarily dealt with as ~MObject.
&rarr; see //discussion of some fine points// regarding [[relation of Assets and MObjects|AssetModelConnection]]
!!exploring session contents
Typically, the list of timelines serves as starting point for exploring the model. Basically, any kind of object could be attached everywhere, but both the GUI and the Builder rely on assumptions regarding the [[overall model structure|HighLevelModel]] &mdash; silently ignoring content not in line. This corresponds to the //dedicated API functions// on specific kinds of objects, which allow to retrieve content according to this assumed structure conveniently and with strong typing. From the timeline and the associated sequence you'll get the root track, and from there the sub-tracks and the clips located on them, which in turn may have attachments (effects, transitions, labels).
On the other hand, arbitrary structures can be retrieved using the //generic API:// Contents can be discovered on the QueryFocus, which automatically follows the //point of mutation,// but can be moved to any point using the {{{QueryFocus::attach(obj)}}} function.
!!queries and defaults
Queries can retrieve the immediate children, or the complete contents of a scope (depth-first). The results can be filtered by type. Additionally, the results of such a scoped query can be filtered by a client-provided predicate, which allows to pick objects with specific properties. The intention is to extend this later to arbitrary logical [[queries|Query]], using some kind of resolution engine. Besides these queries, [[default configured objects|DefaultsManagement]] can be retrieved or defined through the defaults manager, which is accessible as a self-contained component on the public session API. Defaults can be used to establish a standard way of doing things on a per-project base.
&rarr; SessionContentsQuery
{{red{WIP ... just emerging}}}
!!discovery and mutations
The discovery functions available on these ~APIs are wired such as to return suitably typed MObjectRef instances always. These are small value objects and can be used to invoke operations (both query and mutating) on the underlying object within the session. Raw placement (language)references aren't exposed on these outward interfaces.
While this protects against accessing dangling references, it can't prevent clients from invoking any mutating operation directly on these references. It would be conceivable, by using proxies, to create and invoke commands automatically. But we rather don't want to go this route, because
* Lumiera is an application focussed on very specific usage, not a general purpose library or framework
* regarding CommandHandling, the //design decision was to require a dedicated (and hand written) undo functor.//
!!!!protection against accidental mutation
{{red{WIP}}}As of 2/10, I am considering to add a protection against invoking an raw mutation operation accidentally, and especially bypassing the command frontend and the ProcDispatcher. This would not only be annoying (no UNDO), but potentially dangerous, because all of the session internals are not threadsafe by design.
The considered solution would be to treat this situation as if an authorisation is required; this authorisation for mutation could be checked by a &raquo;wormhole&laquo;-like context access (&rarr; aspect oriented programming). Of course, in our case we're not dealing with real access restrictions, just a safeguard: While command execution creates such an authorisation token automatically, a client actually wanting to invoke an mutation operations bypassing the command frontend, would need to set up such a token explicitly and manually.
!!adding and destroying
Objects can be added and destroyed directly on the top level session API. The actual memory management of the object instances works automatically, based on reference counts. (Even after purging an object from the session, it may still be indirectly in use by an ongoing render process).
When adding an object, a [[scope|PlacementScope]] needs to be specified. Thus it makes sense to provide {{{add()}}}-operations on the dedicated API of individual session parts, while the generic {{{attach()}}}-call on the top-level session API relies on the current QueryFocus to determine the location where to add an object. Besides, as we're always adding the [[Placement]] of an object into the session, this placement may specify an additional constraint or tie to a specific scope; resolving the placement thus may cause the object to move to another location
!!!{{red{Questions}}}
* what exactly is the relation of discovery and [[mutations|SessionMutation]]?
* what's the point of locating them on the same conceptual level?
* how to observe the requirement of ''dispatching'' mutations ([[Command]])?
* how to integrate the two possible search depths (children and all)?
* how is all of this related to the LayerSeparationInterfaces, here SessionFacade und EditFacade?
<<<
__preliminary notes__: {{red{3/2010}}} Discovery functions accessible from the session API are always written such as to return ~MObjectRefs. These expose generic functions for modifying the structure: {{{attach(MObjectRef)}}} and {{{purge()}}}. The session API exposes variations of these functions. Actually, all these functions do dispatch the respective commands automatically. {{red{Note 1/2015 not implemented, not sure if thats a good idea}}} To the contrary, the raw functions for adding and removing placements are located on the PlacementIndex; they are accessible as SessionServices &mdash; which are intended for Proc-Layer's internal use solely. This separation isn't meant to be airtight, just an reminder for proper use.
Currently, I'm planning to modify MObjectRef to return only a const ref to the underlying facilities by default. Then, there would be a subclass which is //mutation enabled.// But this subclass will check for the presence of a mutation-permission token &mdash; which is exposed via thread local storage, but //only within a command dispatch.// Again, no attempt is made to make this barrier airtight. Indeed, for tests, the mutation-permission token can just be created in the local scope. After all, this is not conceived as an authorisation scheme, rather as a automatic sanity check. It's the liability of the client code to ensure any mutation is dispatched.
<<<

The current [[Session]] is the root of any state found within Proc-Layer. Thus, events defining the session's lifecycle influence and synchronise the cooperative behaviour of the entities within the model, the ProcDispatcher, [[Fixture]] and any facility below.
* when ''starting'', on first access an empty session is created, which puts any related facility into a defined initial state.
* when ''closing'' the session, any dependent facilities are disabled, disconnected, halted or closed
* ''loading'' an existing session &mdash; after closing the previous session &mdash; sets up an empty (default) session and populates it with de-serialised content.
* when encountering a ''mutation point'', [[command processing|ProcDispatcher]] is temporarily halted to trigger off an BuildProcess.
!Role of the session manager
The SessionManager is responsible for conducting the session lifecycle. Accessible through the static interface {{{Session::current}}}, it exposes the actual session as a ~PImpl. Both session manager and session are indeed interfaces, backed by implementation classes belonging to ~Proc-Layer's internals. Loading, saving, resetting and closing are the primary public operations of the session manager, each causing the respective lifecycle event.
Beyond that, client code usually doesn't interact much with the lifecycle, which mostly is a pattern of events to happen in a well-defined sequence. So the //implementation// of the session management operations has to comply to this lifecycle, and does so by relying on a self-contained implementation service, the LifecycleAdvisor. But (contrary to an application framework) the lifecycle of the Lumiera session is rather fixed, the only possibility for configuration or extension being the [[lifecycle hooks|LifecycleEvent]], where other parts of the system (and even plug-ins) may install some callback methods.
!Synchronising access to session's implementation facilities
Some other parts and subsystems within the ~Proc-Layer need specialised access to implementation facilities within the session. Information about some conditions and configurations might be retrieved through [[querrying the session|Query]], and especially default configurations for many objects are [[bound to the session|DefaultsImplementation]]. The [[discovery of session contents|SessionStructureQuery]] relies on an [[index facility|PlacementIndex]] embedded within the session implementation. Moreover, some "properties" of the [[media objects|MObject]] are actually due to the respective object being [[placed|Placement]] in some way into the session; consequently, there might be an dependency on the actual [[location as visible to the placement|PlacementScope]], which in turn is constituted by [[querying the index|QueryFocus]].
Each of these facilities relies on a separate access point to session services, corresponding to distinct service interfaces. But &mdash; on the implementation side &mdash; all these services are provided by a (compound) SessionServices implementation object. This approach allows to switch the actual implementation of all these services simply by swapping the ~PImpl maintained by the session manager. A new implementation level service can thus be added to the ~SessionImpl just by hooking it into the ~SessionServices compound object. But note, this mechanism as such is ''not thread safe'', unless the //implementation// of the invoked functions is synchronised in some way to prevent switching to a new session implementation while another thread is still executing session implementation code.
Currently, the understanding is for some global mechanism to hold any command execution, script running and further object access by the GUI //prior//&nbsp; to invoking any of the session management operations (loading, closing, resetting). An alternative would be to change the top-level access to the session ~PImpl to go through an accessor value object, to acquire some lock automatically before any access can happen. C++ ensures the lifespan of any temporaries to surpass evaluation of the enclosing expression, which would be sufficient to prevent another thread to pull away the session during that timespan. Of course, any value returned from such an session API call isn't covered by this protection. Usually, objects are handed out as MObjectRef, which in turn means to resolve them (automatically) on dereferentiation by another session API access. But while it seems we could get locking to work automatically this way, still such a technique seems risky and involved; a plain flat lock at top level seems to be more appropriate.
!Interface and lifecycle hooks
As detailed above, {{{Session::current}}} exposes the management / lifecycle API, and at the same time can be dereferenced into the primary [[session API|SessionInterface]]. An default configured ~SessionImpl instance will be built automatically, in case no session implementation instance exists on executing this dereferentiation. At various stages within the lifecycle, specific LifecycleEvent hooks are activated, which serve as an extension point for other parts of the system to install callback functions to execute additional behaviour at the right moment.
!!!building (or loading) a session
# as a preparation step, a new implementation instance is created, alongside with any supporting facilities (especially the PlacementIndex)
# the basic default configuration is loaded into this new session instance
# when the new session is (technically) complete and usable, the switch on the ~PImpl happens
# the {{{ON_SESSION_START}}} LifecycleEvent is emitted
# content is loaded into the session, including hard wired content and any de-serialised data from persistent storage
# the {{{ON_SESSION_INIT}}} event is emitted
# additional initialisation, wiring and registration takes place; basically anything to make the session fully functional
# the session LayerSeparationInterface is opened and any further top-level blocking is released
# the {{{ON_SESSION_READY}}} event is emitted
!!!closing the session
# top-level facilities accessing the session (GUI, command processing, scripts) are blocked and the LayerSeparationInterface is closed
# any render processes are ensured to be terminated (or //disconnected// &mdash; so they can't do any harm)
# the {{{ON_SESSION_END}}} event is emitted
# the command processing log is tagged
# the command queue(s) are emptied, discarding any commands not yet executed
# the PlacementIndex is cleared, effectively releasing any object "instances"
# the [[asset registry|AssetManager]] is cleared, thereby releasing any remaining external resource references
# destruction of session implementation instances
{{red{none of the above is implemented as of 11/09}}}

The Session contains all information, state and objects to be edited by the User (&rarr;[[def|Session]]).
As such, the SessionInterface is the main entrance point to Proc-Layer functionality, both for the primary EditingOperations and for playback/rendering processes. Proc-Layer state is rooted within the session and guided by the [[session's lifecycle events|SessionLifecycle]].
Implementation facilities within the Proc-Layer may access a somewhat richer [[session service API|SessionServices]].
Currently (as of 3/10), Ichthyo is working on getting a preliminary implementation of the [[Session in Memory|SessionDataMem]] settled.
!Session, Model and Engine
The session is a [[Subsystem]] and acts as a frontend to most of the Proc-Layer. But it doesn't contain much operational logic; its primary contents are the [[model|Model]], which is closely [[interconnected to the assets|AssetModelConnection]].
!Design and handling of Objects within the Session
Objects are attached and manipulated by [[placements|Placement]]; thus the organisation of these placements is part of the session data layout. Effectively, such a placement within the session behaves like an //instances// of a given object, and at the same time it defines the "non-substantial" properties of the object, e.g. its positions and relations. [[References|MObjectRef]] to these placement entries are handed out as parameters, both down to the [[Builder]] and from there to the render processes within the engine, but also to external parts within the GUI and in plugins. The actual implementation of these object references is built on top of the PlacementRef tags, thus relying on the PlacementIndex the session maintains to keep track of all placements and their relations. While &mdash; using these references &mdash; an external client can access the objects and structures within the session, any actual ''mutations'' should be done based on the CommandHandling: a single operation of a sequence of operations is defined as [[Command]], to be [[dispatched|ProcDispatcher]] as [[mutation operation|SessionMutation]]. Following this policy ensures integration with the&nbsp;SessionStorage and provides (unlimited) [[UNDO|UndoManager]].
On the implementation level, there are some interdependencies to consider between the [[data layout|SessionDataMem]], keeping ModelDependencies updated and integrating with the BuildProcess. While the internals of the session are deliberately kept single-threaded, we can't make much assumptions regarding the ongoing render processes.

The session manager is responsible for maintaining session state as a whole and for conducting the session [[lifecycle|SessionLifecycle]]. The session manager API allows for saving, loading, closing and resetting the session. Accessible through the static interface {{{Session::current}}}, it exposes the actual session as a ~PImpl. Actually, both session and session manager are interfaces.

//Any modification of the session will pass through the [[command system|CommandHandling]].//
Thus any possible mutation comes in two flavours: a raw operation invoked directly on an object instance attached to the model, and a command taking an MObjectRef as parameter. The latter approach &mdash; invoking any mutation through a command &mdash; will pass the mutations trough the ProcDispatcher to ensure the're logged for [[UNDO|UndoManager]] and executed sequentially, which is important, because the session's internals are //not threadsafe by design.// Thus we're kind of enforcing the use of Commands: mutating operations include a check for a &raquo;permission to mutate&laquo;, which is automatically available within a command execution {{red{TODO as of 2/10}}}. Moreover, the session API and the corresponding LayerSeparationInterfaces expose MObjectRef instances, not raw (language) refs.
!!Questions to solve
* how to get from the raw mutation to the command?
* how to organise the naming and parametrisation of commands?
* should we provide the additional flexibility of a binding here?
* how to keep [[dependencies within the model|ModelDependencies]] up-to date?
!!who defines commands?
The answer is simple: the one who needs to know about their existence. Because basically commands are invoked //by-name// -- someone needs to be aware of that name and its meaning. Thus, for any given mutation, there is a place where it's meaningful to specify the details //and//&nbsp; to subsume them under a meaningful term. An entity responsible for this place could then act as the provider of the command in question.
Interestingly, there seems to be an alternative answer to this question. We could locate the setup and definition of all commands into a central administrative facility. Everyone in need of a command then ought to know the name and retrieve this command. Sounds like bureaucracy.

<<<
{{red{WARNING: Naming was discussed (11/08) and decided to be changed....}}}
* the term [[EDL]] was phased out in favour of ''Sequence''
* [[Session]] is largely synonymous to ''Project''
* there seems to be a new entity called [[Timeline]] which holds the global Pipes
<<<
The [[Session]] (sometimes also called //Project// ) contains all information and objects to be edited by the User. Any state within the Proc-Layer is directly or indirectly rooted in the session. It can be saved and loaded. The individual Objects within the Session, i.e. Clips, Media, Effects, are contained in one or multiple collections within the Session, which we call [[sequence(s)|Sequence]]. Moreover, the sesion contains references to all the Media files used, and it contains various default or user defined configuration, all being represented as [[Asset]]. At any given time, there is //only one current session// opened within the application. The [[lifecycle events|SessionLifecycle]] of the session define the lifecycle of ~Proc-Layer as a whole.
The Session is close to what is visible in the GUI. From a user's perspective, you'll find a [[Timeline]]-like structure, containing an [[Sequence]], where various Media Objects are arranged and placed. The available building blocks and the rules how they can be combined together form Lumiera's [[high-level data model|HighLevelModel]]. Basically, besides the [[media objects|MObjects]] there are data connections and all processing is organized around processing chains or [[pipes|Pipe]], which can be either global (in the Session) or local (in real or virtual clips).
!!!larger projects
For larger editing projects the simple structure of a session containing "the" timeline is not sufficient. Rather
* we may have several [[sequences|Sequence]], e.g. one for each scene. These sequences can be even layered or nested (compositional work).
* within one project, there may be multiple, //independant Timelines// &mdash; each of which may have an associated Viewer or Monitor
Usually, when working with this stucture, you'll drill down starting from a timeline, trough a (top-level) sequence, down into a fork ("track"), a clip, maybe even a embedded Sequence (VirtualClip), and from there even more down into a single attached effect. This constitutes a set of [[nested scopes|PlacementScope]]. Operations are to be [[dispatched|ProcDispatcher]] through a [[command system|CommandHandling]], including the target object [[by reference|MObjectRef]]. [[Timelines|Timeline]] on the other hand are always top-level objects and can't be combined further. You can render a single given timeline to output.
&rarr; see [[Relation of Project, Timelines and Sequences|TimelineSequences]]
!!!the definitive state
With all the structural complexities possible within such a session, we need an isolation layer to provide __one__ definitive state where all configuration has been made explicit. Thus the session manages a special consolidated view (object list), called [[the Fixture|Fixture]], which can be seen as all currently active objects placed onto a single timeline.
!!!organisational devices
The possibility of having multiple Sequences helps organizing larger projects. Each [[Sequence]] is just a logical grouping; because all effective properties of any MObject within this sequence are defined by the ~MObject itself and the [[Placement]], by which the object is anchored to some time point, some fork, can be connected to some pipe, or linked to another object. In a similar manner, [[Forks ("tracks")|Fork]] are just another organisational aid for grouping objects, disabling them and defining common output pipes.
!!!global pipes
[>img[draw/Proc.builder1.png]] Any session should contain a number of global [[(destination) pipes|Pipe]], typically video out and audio out. The goal is, to get any content producing or transforming object in some way connected to one of these outputs, either //by [[placing|Placement]] it directly// to some pipe, or by //placing it to a track// and having the track refer to some pipe. Besides the global destination pipes, we can use internal pipes to form busses or subgroups, either on a global (session) level, or by using the processing pipe within a [[virtual clip|VirtualClip]], which can be placed freely within the sequence(s). Normally, pipes just gather and mix data, but of course any pipe can have an attached effect chain.
&rarr; [[more on Tracks and Pipes within the Sequence|TrackPipeSequence]]
!!!default configuration
While all these possibilities may seem daunting, there is a simple default configuration loaded into any pristine new session:
It will contain a global video and audio out pipe, just one timeline holding a single sequence with a single track; this track will be configured with a fading device, to send any video and audio data encountered on enclosed objects to the global (master) pipes. So, by adding a clip with a simple absolute placement to this track and to some time position, the clip gets connected and rendered, after [[(re)building|PlanningBuildFixture]] the [[Fixture]] and passing the result to the [[Builder]] &mdash; and using the resulting render nodes network (Render Engine).
&rarr; [[anatomy of the high-level model|HighLevelModel]]
&rarr; considerations regarding [[Tracks and Pipes within the session|TrackPipeSequence]]
&rarr; see [[Relation of Project, Timelines and Sequences|TimelineSequences]]

Within Lumiera's Proc-Layer, there are some implementation facilities and subsystems needing more specialised access to implementation services provided by the session. Thus, besides the public SessionInterface and the [[lifecycle and state management API|SessionManager]], there are some additional service interfaces exposed by the session through a special access mechanism. This mechanism needs to be special in order to assure clean transactional behaviour when the session is opened, closed, cleared or loaded. Of course, there is the additional requirement to avoid direct dependencies of the mentioned Proc internals on session implementation details.
!Accessing session services
For each of these services, there is an access interface, usually through an class with only static methods. Basically this means access //by name.//
On the //implementation side//&nbsp; of this access interface class (i.e. within a {{{*.cpp}}} file separate from the client code), there is a (down-casting) access through the top-level session-~PImpl pointer, allowing to invoke functions on the ~SessionServices instance. Actually, this ~SessionServices instance is configured (statically) to stack up implementations for all the exposed service interfaces on top of the basic ~SessionImpl class. Thus, each of the individual service implementations is able to use the basic ~SessinImpl (becaus it inherits it) and the implementaion of the access functions (to the session service we're discussing here) is able to use this forwarding mechanism to get the actual implementation basically by one-liners. The upside of this (admittedly convoluted) technique is that we've gotten at runtime only a single indirection, which moreover is through the top-level session-~PImpl. The downside is that, due to the separation in {{{*.h}}} and {{{*.c}}} files, we can't use any specifically typed generic operations, which forces us to use type erasure in case we need such (an example being the content discovery queries utilised by all high-level model objects).

The frontside interface of the session allows to query for contained objects; it is used to discover the structure and contents of the currently opened session/project. Access point is the public API of the Session class, which, besides exposing those queries, also provides functionality for adding and removing session contents.
!discovering structure
The session can be seen as an agglomeration of nested and typed containers.
Thus, at any point, we can explore the structure by asking for //contained objects of a specific type.// For example, at top level, it may be of interest to enumerate the [[timelines within this session|Timeline]] and to ennumerate the [[sequences|Sequence]]. And in turn, on a given Sequence, it would be of interest to explore the forks or tracks, and also maybe to iterate over all clips within this sequence.
So, clearly, there are two flavours of such an contents exploration query: it could either be issued as an dedicated member function on the public API of the respective container object, e.g. {{{Track::getClips()}}} &mdash; or it could be exposed as generic query function, relying on the implicit knowledge of the //current location//&nbsp; rather.
!problem of context and access path
The (planned) session structure of Lumiera allows for quite some flexibility, which, of course comes at a price tag. Especially, as there can be multiple independent top level timelines, where a given sequence can be used simultaneously within multiple timelines and even as virtual media within a [[meta-clip|VirtualClip]], and moreover, as properties of any Placement are rather queried and discovered within the PlacementScope of this object &mdash; consequently the discovered values may depend on //how you look at this object.// More specifically, it depends on the ''access path'' used to discover this object, because this path constitutes the actual scope visible to this object.
To give an example, let's assume a clip within a sequence, and this sequence is both linked to the top-level timeline, but also used within a meta-clip. (see the drawing &rarr; [[here|PlacementScope]])
In this case, the sequence has an 1:n [[binding|BindingMO]]. A binding is (by definition) also a PlacementScope, and, incidentally, in the case of binding the sequence into a timeline, the binding also translates //logical// output designations into global pipes found within the timeline, while otherwise they get mapped onto "channels" of the virtual media used by the virtual clip. Thus, the absolute time position as well as the output connection of a given clip within this sequence //depends on how we look at this clip.// Does this clip apear as part of the global timeline, or did we discover it as contained within the meta-clip? &rarr; see [[discussion of this scope-binding problem|BindingScopeProblem]]
!!solution requirements
The baseline of any solution to this problem is clear: at the point where the query is issued, a context information is necessary; this context yields an access path from top level down to the object to be queried, and this access path constitutes the effective scope this object can utilise for resolving the query.
!!introducing a QueryFocus
A secondary goal of the design here is to ease the use of the session query interface. Thus the proposal is to treat this context and access path as part of the current state. To do so, we can introduce a QueryFocus following the queries and remembering the access path; this focus should be maintained mostly automatically. It allows for stack-like organisation, to allow sub-queries without affecting the current focus, where the handling of such a temporary sub-focus is handled automatically by a scoped local (RAII) object. The focus follows the issued queries and re-binds as necessary.
!!using QueryFocus as generic query interface
Solving the problem this way has the nice side effect, that we get a quite natural location where to put an unspecific query interface: Just let the current QueryFocus expose the basic set of query API functions. Such an generic query interface can be seen as a complement to the query functions exposed on specific objects ("get me the clips within this track") according to the structure. Because a generic interface especially allows for writing simple diagnostics and discovery code, with only a weak link to the actual session structure.
!Implementation strategy
The solution is being built starting from the generic part, as the actual session structure isn't hard coded into the session implementation, but rather created by convention.
The query API on specific objects (i.e. Session, Timeline, Sequence, Track and Clip) is then easily coded up on top, mostly with inlined one-liners of the kind
{{{
ITER getClips() { return QueryFocus::push(this).query<Clip>(); }
}}}
To make this work, QueryFocus exposes an static API (and especially the focus stack is a singleton). The //current// QueryFocus object can easily be re-bound to another point of interest (and ajusts the contained path info automatically), and the {{{push()}}}-function creates a local scoped object (which pops automatically).
And last but not least: the difficult part of this whole concept is encapsulated and ''can be left out for now''. Because, according to Lumiera's [[roadmap|http://issues.lumiera.org/roadmap]], meta-clips are to be postponed until well into beta! Thus, we can start with a trivial (no-op) implementation, but immediately gain the benefit of making the relevant parts of the session implementation aware of the problem.
{{red{WIP ... draft}}}

//A subsystem within Proc-Layer, responsible for lifecycle and access to the editing [[Session]].//
[img[Structure of the Session Subsystem|uml/Session-subsystem.png]]
!Structure
The ProcDispatcher is at the heart of the //Session Subsystem.// Because the official interface for working on the session, the [[SessionCommand façade|SessionCommandFacade]], is expressed in terms of sending command messages to invoke predefined [[commands|CommandHandling]] to operate on the SessionInterface, the actual implementation of such a {{{SessionCommandService}}} needs a component actually to enqueue and dispatch those commands -- which is the {{{DispatcherLoop}}} within the ProcDispatcher. As usual, the ''lifecycle'' is controlled by a subsystem descriptor, which starts and stops the whole subsystem; and this starting and stopping in turn translates into starting/stopping of the dispatcher loop. On the other hand, //activation of the dispatcher,// which means actively to dispatch commands, is controlled by the lifecycle of the session proper. The latter is just a data structure, and can be loaded / saved and rebuilt through the ''session manager''.
!Lifecycle
As far as lifecycle is concerned, the »session subsystem« has to be distinguished from the //session proper,// which is just a data structure with its own, separate lifecycle considerations. Accessing the session data only makes sense when this data structure is fully loaded, while the //session subsystem,// deals with performing commands on the session and with triggering the builder runs.
!!!start-up
The session subsystem lifecycle translates into method invocations on the {{{ProcDispatcher}}}, which in turn manages the parts actually implementing the session command processing and builder operations. This relation is expressed by holding onto the implementation as a //~PImpl.// As long as the {{{DispatcherLoop}}} object exists, the session subsystem can be considered in //running state.// This is equivalent to the following
* the ''session loop thread'' is spawned. This thread performs all of the session and builder operations (single-threaded).
* the {{{SessionCommandService}}} is started and connected as implementation of the {{{SessionCommand}}} façade.
!!!shutdown
Shutdown is initiated by sending a message to the dispatcher loop. This causes the internal loop control to wake up and leave the loop, possibly after finishing a command or builder run currently in progress. When leaving the loop, the {{{sigTerm}}} of the SessionSubsystem is invoked, which then in turn causes the {{{DispatcherLoop}}} object to be deleted and the ProcDispatcher thus returned into halted state.

A small (in terms of storage) and specifically configured StateProxy object which is created on the stack {{red{Really on the stack? 9/11}}} for each individual {{{pull()}}} call. It is part of the invocation state of such a call and participates in the buffer management. Thus, in a calldown sequence of {{{pull()}}} calls we get a corresponding sequence of "parent" states. At each level, the &rarr; WiringDescriptor of the respective node defines a Strategy how the call is passed on.

An Object representing a //Render Process// and containing associated state information.
* it is created in the Player subsystem while initiating the RenderProcess
* it is passed on to the generated Render Engine, which in turn passes it down to the individual Processors
* moreover, it contains methods to communicate with other state relevant parts of the system, thereby shielding the rendering code from any complexities of state synchronisation and management if necessary. (thus the name Proxy)
* in a future version, it may also encapsulate the communication in a distributed render farm

Conversion of a media stream into a stream of another type is done by a processor module (plugin). The problem of finding such a module is closely related to the StreamType and especially [[problems of querying|StreamTypeQuery]] for such. (The builder uses a special Facade, the ConManager, to access this functionality). There can be different kinds of conversions, and the existance or non-existance of such an conversion can influence the stream type classification.
* different //kinds of media// can be ''transformed'' into each other
* stream types //subsumed// by a given prototype should be ''lossless convertible'' and thus can be considered //equivalent.//
* besides, between different stream //implementation types,// there can be a ''rendering'' (lossy conversion) &mdash; or no conversion at all.

The stream Prototype is part of the specification of a media stream's type. It is a semantic (or problem domain oriented) concept and should be distinguished from the actual implementation type of the media stream. The latter is provided by an [[library implementation|StreamTypeImplFacade]]. While there are some common predefined prototypes, mostly, they are defined within the concrete [[Session]] according to the user's needs.
Prototypes form an open (extensible) collection, though each prototype belongs to a specific media kind ({{{VIDEO, IMAGE, AUDIO, MIDI,...}}}).
The ''distinguishing property'' of a stream prototype is that any [[Pipe]] can process //streams of a specific prototype only.// Thus, two streams with different prototype can be considered "something quite different" from the users point of view, while two streams belonging to the same prototype can be considered equivalent (and will be converted automatically when their implementation types differ). Note this definition is //deliberately fuzzy,// because it depends on the actual situation of the project in question.
Consequently, as we can't get away with an fixed Enum of all stream prototypes, the implementation must rely on a query interface. The intention is to provide a basic set of rules for deciding queries about the most common stream prototypes; besides, a specific session may inject additional rules or utilize a completely different knowledge base. Thus, for a given StreamTypeDescriptor specifying a prototype
* we can get a [[default|DefaultsManagement]] implementation type
* we can get a default prototype to a given implementation type by a similar query
* we can query if a implementation type in question can be //subsumed// by this prototype
* we can determine if another prototype is //convertible//
!!Examples
In practice, several things might be considered "quite different" and thus be distinguished by protorype: NTSC and PAL video, video versus digitized film, HD video versus SD video, 3D versus flat video, cinemascope versus 4:3, stereophonic versus monaural, periphonic versus panoramic sound, Ambisonics versus 5.1, data reduced ~MP3 versus full quality linear PCM...

//how to classify and describe media streams//
Media data is supposed to appear structured as stream(s) over time. While there may be an inherent internal structuring, at a given perspective ''any stream is a unit and homogeneous''. In the context of digital media data processing, streams are always ''quantized'', which means they appear as a temporal sequence of data chunks called ''frames''.
! Terminology
* __Media__ is comprised of a set of streams or channels
* __Stream__ denotes a homogeneous flow of media data of a single kind
* __Channel__ denotes a elementary stream, which can't be further separated in the given context
* all of these are delivered and processed in a smallest unit called __Frame__. Each frame corresponds to a //time interval.//
* a __Buffer__ is a data structure capable of holding a Frame of media data.
* the __~Stream-Type__ describes the kind of media data contained in the stream
! Problem of Stream Type Description
Media types vary largely and exhibit a large number of different properties, which can't be subsumed under a single classification scheme. On the other hand we want to deal with media objects in a uniform and generic manner, because generally all kinds of media behave somewhat similar. But the twist is, these similarities disappear when describing media with logical precision. Thus we are forced into specialized handling and operations for each kind of media, while we want to implement a generic handling concept.
! Lumiera Stream Type handling
!! Identification
A stream type is denoted by a StreamTypeID, which is an identifier, acting as an unique key for accessing information related to the stream type. It corresponds to an StreamTypeDescriptor record, containing an &mdash; //not necessarily complete// &mdash; specification of the stream type, according to the classification detailed below.
!! Classification
Within the Proc-Layer, media streams are treated largely in a similar manner. But, looking closer, not everything can be connected together, while on the other hand there may be some classes of media streams which can be considered //equivalent// in most respects. Thus separating the distinction between various media streams into several levels seems reasonable...
* Each media belongs to a fundamental ''kind'' of media, examples being __Video__, __Image__, __Audio__, __MIDI__, __Text__,... <br/>Media streams of different kind can be considered somewhat "completely separate" &mdash; just the handling of each of those media kinds follows a common //generic pattern// augmented with specialisations. Basically, it is //impossible to connect// media streams of different kind. Under some circumstances there may be the possibility of a //transformation// though. For example, a still image can be incorporated into video, sound may be visualized, MIDI may control a sound synthesizer.
* Below the level of distinct kinds of media streams, within every kind we have an open ended collection of ''prototypes'', which, when compared directly, may each be quite distinct and different, but which may be //rendered//&nbsp; into each other. For example, we have stereoscopic (3D) video and we have the common flat video lacking depth information, we have several spatial audio systems (Ambisonics, Wave Field Synthesis), we have panorama simulating sound systems (5.1, 7.1,...), we have common stereophonic and monaural audio. It is considered important to retain some openness and configurability within this level of distinction, which means this classification should better be done by rules then by setting up a fixed property table. For example, it may be desirable for some production to distinguish between digitized film and video NTSC and PAL, while in another production everything is just "video" and can be converted automatically. The most noticeable consequence of such a distinction is that any Bus or [[Pipe]] is always limited to a media stream of a single prototype. (&rarr; [[more|StreamPrototype]])
* Besides the distinction by prototypes, there are the various media ''implementation types''. This classification is not necessarily hierarchically related to the prototype classification, while in practice commonly there will be some sort of dependency. For example, both stereophonic and monaural audio may be implemented as 96kHz 24bit PCM with just a different number of channel streams, but we may as well get a dedicated stereo audio stream with two channels multiplexed into a single stream. For dealing with media streams of various implementation type, we need //library// routines, which also yield a //type classification system.// Most notably, for raw sound and video data we use the [[GAVL]] library, which defines a classification system for buffers and streams.
* Besides the type classification detailed thus far, we introduce an ''intention tag''. This is a synthetic classification owned by Lumiera and used for internal wiring decisions. Currently (8/08), we recognize the following intention tags: __Source__, __Raw__, __Intermediary__ and __Target__. Only media streams tagged as __Raw__ can be processed.
!! Media handling requirements involving stream type classification
* set up a buffer and be able to create/retrieve frames of media data.
* determine if a given media data source and sink can be connected, and how.
* determine and enumerate the internal structure of a stream.
* discover processing facilities
&rarr; see StreamTypeUse
&rarr; [[querying types|StreamTypeQuery]]

A description and classification record usable to find out about the properties of a media stream. The stream type descriptor can be accessed using an unique StreamTypeID. The information contained in this descriptor record can intentionally be //incomplete,// in which case the descriptor captures a class of matching media stream types. The following information is maintained:
* fundamental ''kind'' of media: {{{VIDEO, IMAGE, AUDIO, MIDI,...}}}
* stream ''prototype'': this is the abstract high level media type, like NTSC, PAL, Film, 3D, Ambisonics, 5.1, monaural,...
* stream ''implementation type'' accessible by virtue of an StreamTypeImplFacade
* the ''intended usage category'' of this stream: {{{SOURCE, RAW, INTERMEDIARY, TARGET}}}.
&rarr; see &raquo;[[Stream Type|StreamType]]&laquo; detailed specification
&rarr; notes about [[using stream types|StreamTypeUse]]
&rarr; more [[about prototypes|StreamPrototype]]

This ID is an symbolic key linked to a StreamTypeDescriptor. The predicate {{{stream(ID)}}} specifies a media stream with the StreamType as detailed by the corresponding descriptor (which may contain complete or partial data defining the type).

A special kind of media stream [[implementation type|StreamTypeImplFacade]], which is not fully specified. As such, it is supposed there //actually is// an concrete implementation type, while only caring for some part or detail of this implementation to exhibit a specific property. For example, using an type constraint we can express the requirement of the actual implementation of a video stream to be based on ~RGB-float, or to enforce a fixed frame size in pixels.
An implementation constraint can //stand-in// for a completely specified implementation type (meaning it's a sub interface of the latter). But actually using it in this way may cause a call to the [[defaults manager|DefaultsImplementation]] to fill in any missing information. An example would be to call {{{createFrame()}}} on the type constraint object, which means being able to allocate memory to hold a data frame, with properties in compliance with the given type constraint. Of cousre, then we need to know all the properties of this stream type, which is where the defaults manager is queried. This allows session customisation to kick in, but may fail under certain cicumstances.

Common interface for dealing with the implementation of media stream data. From a high level perspective, the various kinds of media ({{{VIDEO, IMAGE, AUDIO, MIDI,...}}}) exhibit similar behaviour, while on the implementation level not even the common classification can be settled down to a complete general and useful scheme. Thus, we need separate library implementations for deailing with the various sorts of media data, all providing at least a set of basic operations:
* set up a buffer
* create or accept a frame
* get an tag describing the precise implementation type
* ...?
&rarr; see also &raquo;[[Stream Type|StreamType]]&laquo;
//Note:// there is a sort-of "degraded" variant just requiring some &rarr; [[implementation constraint|StreamTypeImplConstraint]] to hold

Querying for media stream type information comes in various flavours
* you may want to find a structural object (pipe, output, processing patten) associated with / able to deal with a certain stream type
* you may need a StreamTypeDescriptor for an existing stream given as implementation data
* you may want to build or complete type information from partial specification.
Mostly, those queries involve the ConfigRules system in some way or the other. The [[prototype-|StreamPrototype]] and [[implementation type|StreamTypeImplFacade]]-interfaces themselves are mostly a facade for issuing appropriate queries. Some objects (especially [[pipes|Pipe]]) are tied to a certain stream type and thus store a direct link to type information. Others are just associated with a type by virtue of the DefaultsManagement.
The //problem// with this pivotal role of the config rules is that &mdash; from a design perspective &mdash; not much can be said specifically, besides //"you may be able to find out...", "...depends on the defaults and the session configuration".// This way, a good deal of crucial behaviour is pushed out of the core implementation (and it's quite intentionally being done this way). What can be done regarding the design of the core is mostly to setup a framework for the rules and determine possible ''query situations''.
!the kind of media
the information of the fundamental media kind (video, audio, text, MIDI,...) is assiciated with the prototype, for technical reasons. Prototype information is mandatory for each StreamType, and the impl facade provides a query function (because some implementation libraries, e.g. [[GAVL]], support multiple kinds of media).
!query for a prototype
__Situation__: given an implementation type, find a prototype to subsume it.
Required only for building a complete ~StreamType which isn't known at this point.
The general case of this query is //quite hairy,// because the solution is not necessary clear and unique. And, worse still, it is related to the semantics, requiring semantic information and tagging to be maintained somewhere. For example, while the computer can't "know" what stereopohinc audio is (only a human can, by listening to a stereophoic playback and deciding if it actually does convey a spatical sound image), in most cases we can overcome this problem by using the //heuristical rule// of assuming the prototype "stereophonic" when given two identically typed audio channels. This example also shows the necessity of ordering heuristic rules to be able to pick a best fit.
We can inject two different kinds of fallback solutions for this kind of query:
* we can always build a "catch-all" prototype just based on the kind of media (e.g. {{{prototype(video).}}}). This should match with lowest priority
* we can search for existing ~StreamTypes with the same impl type, or an impl type which is //equivalent convertible// (see &rarr; StreamConversion).
The latter case can yield multiple solutions, which isn't any problem, because the match is limited to classes of equivalent stream implementation, which would be subsumed under the same prototype anyway. Even if the registry holds different prototypes linked to the same implementation type, they would be convertible and thus could //stand-in// for one another. Together this results in the implementation
# try to get a direct match to an existing impl type which has an associated (complete) ~StreamType, thus bypassing the ConfigRules system altogether
# run a {{{Query<Prototype>}}} for the given implementation type
# do the search within equivalence class as described above
# fall back to the media kind.
{{red{TODO: how to deal with the problem of hijacking a prototype?}}} &rarr; see [[here|StreamTypeUse]]
!query for an implementation
__Situation 1__: given an partially specified ~StreamType (just an [[constraint|StreamTypeImplConstraint]])
__Situation 2__: find an implementation for a given prototype (without any further impl type guidlines)
Both cases have to go though the [[defaults manager|DefaultsManagement]] in some way, in order to give any default configuration a chance to kick in. This is //one of the most important use cases// of the defaults system: the ability to configure a default fromat for all streams with certain semantic classification. {{{prototype(video)}}} by default is RGBA 24bit non-interlaced for example.
But after having queried the defaults system, there remains the problem to build a new solution (which will then automatically become default for this case). To be more precise: invoking the defaults system (as implemented in Lumiera) means first searching through existing objects encountered as default, and then issuing an general query with the capabilities in question. This general query in turn is conducted by the query type handler and usually consists of first searching existing objects and then creating a new object to match the capabilities. But, as said, the details depend on the type (and are defined by the query handler installed for this type). Translated to our problem here in question, this means //we have to define the basic operations from which a type query handler can be built.// Thus, to start with, it's completely sufficient to wire a call to the DefaultsManagement and assume the current session configuration contains some rules to cover it. Plus being prepared for the query to fail (throw, that is).
Later on this could be augmented by providing some search mechanisms:
* search through existing stream type implementations (or a suitable pre filtered selection) and narrow down the possible result(s) by using the constraint as a filter. Obviously this requires support by the MediaImplLib facade for the implementation in question. (This covers __Situation 1__)
* relate a protoype in question to the other existing prototypes and use the convertibility / subsumption as a filter mechanism. Finally pick an existing impl type which is linked to one of the prototypes found thus far.
Essentially, we need a search mechanism for impltypes and prototypes. This search mechanism is best defined by rules itself, but needs some primitive operations on types, like ennumerating all registered types, filter those selections and match against a constraint.
!query for an (complete) StreamType
All situations discussed thus far can also occur wrapped into and triggered by a query for a complete type. Depending on what part is known, the missing bits will be queried.
Independent from these is __another Situation__ where we query for a type ''by ID''.
* a simple symbolic ID can be found by searching through all existing stream types (Operation supported by the type registry within STypeManager)
* a special ''classificating'' ID can be parsed into the components (media kind, prototype, impltype), resulting in sub searches for these.
{{red{not sure if we want to support queries by symboic ID}}}...problem is the impl type, because probably the library needs to support describing any implementation type by a string. Seemingly GAVL does, but requiring it for every lib?

Questions regarding the use of StreamType within the Proc-Layer.
* what is the relation between Buffer and Frame?
* how to get the required size of a Buffer?
* who does buffer allocations and how?
Mostly, stream types are used for querying, either to decide if they can be connected, or to find usable processing modules.
Even building a stream type from partial information involves some sort of query.
&rarr; more on [[media stream type queries|StreamTypeQuery]]
!creating stream types
seemingly stream types are created based on an already existing media stream (or a Frame of media data?). {{red{really?}}}
The other use case seems to be that of an //incomplete// stream type based on a [[Prototype|StreamPrototype]]
!Prototype
According to my current understanding, a prototype is merely a classification entity. But then &mdash; how to bootstrap a Prototype?
And how to do the classification of an existing implementation type.
Besides, there is the problem of //hijacking a prototype:// when a specific implementation type gets tied to a rather generic protoype, like {{{protoype(video)}}}, how to comply to the rule of prototypes subsuming a class of equivalent implementations?
!Defaults and partial specification
A StreamType need not be defined completely. It is sufficient to specify the media kind and the Prototype. The implementation type may be just given as a constraint, thus defining some properties and leaving out others. When creating a frame buffer based upon such an //incomplete type,// [[defaults|DefaultsManagement]] are queried to fill in the missing parts.
Constraints are objects provided by the Lumiera core, but specialized to the internals of the actual implementation library.
For example there might be a constraint implementation to force a specific {{{gavl_pixelformat_t}}}.
!the ID problem
Basically I'd prefer the ~IDs to be real identifiers. So they can be used directly within rules. At least the Prototypes //can// have such a textual identifier. But the implementation type is problematic, and consequently the ID of the StreamType as well. Because the actual implementation should not be nailed down to a fixed set of possibilities. And, generally, we can't expect an implementation library to yield textual identifiers for each implementation type. //Is this really a problem? {{red{what are the use cases?}}}//
As far as I can see, in most cases this is no problem, as the type can be retrieved or derived from an existing media object. Thus, the only problematic case is when we need to persist the type information without being able to guarantee the persistence of the media object this type was derived from. For example this might be a problem when working with proxy media. But at least we should be able to create a constraint (partial type specification) to cover the important part of the type information, i.e. the part which is needed to re-create the model even when the original media isn't there any longer.
Thus, //constraints may be viewed as type constructing functors.//
--------------
!use cases
* pulling data from a media file
* connecting pipes and similar wiring problems
* describing the properties of an processor plugin
!! pulling data from a media file
To open the file, we need //type discovery code,// resulting in a handle to some library module for accessing the contents, which is in compliance with the Lumiera application. Thus, we can determine the possible return values of this type discovery code and provide code which wires up a corresponding StreamTypeImplFacade. Further, the {{{control::STypeManager}}} has the ability to build a complete or partial StreamType from
* an ~ImplFacade
* a Prototype
* maybe even from some generic textual ~IDs?
Together this allows to associate a StreamType to each media source, and thus to derive the Prototype governing the immediately connected [[Pipe]]
A pipe can by design handle data of one Prototype solely.
!! wiring problems
When deciding if a connection can be made, we can build up the type information starting out from the source. (this requires some work, but it's //possible,// generally speaking.). Thus, we can allways get an ~ImplType for the "lower end" of the connection, and at least a Prototype for the "output side" &mdash; which should be enough to use the query functions provided by the stream type interfaces
!! describing properties
{{red{currently difficult to define}}} as of 9/2008, because the property description of plugins is not planned yet.
My Idea was to use [[type implementation constraints|StreamTypeImplConstraint]] for this, which are a special kind of ~ImplType

This design lays great emphasis on separating all those components and subsystems, which are considered not to have a //natural link// of their underlying concepts. This often means putting some additional constraints on the implementation, so basically we need to rely on the actual implementation to live up to this goal. In many cases it may seem to be more natural to "just access the necessary information". But on the long run this coupling of not-directly related components makes the whole codebase monolithic and introduces lots of //accidental complexity.//
Instead, we should try to just connect the various subsystems via Interfaces and &mdash; instead of just using some information, rather use some service to be located on an Interface to query other components for this information. The best approach of course is always to avoid the dependency altogether.
!Examples
* There is a separation between the __high level [[Session]] view__ and the [[Fixture]]: the latter only accesses the MObjects and the Placement Interfaces.
* same holds true for the Builder: it just uses the same Interfaces. The actual coupling is done rather //by type//, i.e. the Builder relies on several types of MObjects to exist and treats them via overloaded methods. He doesn't rely on a actual object structure layout in the session besides the requirement of having a [[Playlist]]
* the Builder itself is a separation layer. Neither do the Objects in the sessionL access directly [[Render Nodes|ProcNode]], nor do the latter call back into the session. Both connections seem to be necessary at first sight, but both can be avoided by using the Builder Pattern
* another separation exists between the Render Engine and the individual Nodes: The Render Engine doesn't need to know the details of the data types processed by the Nodes. It relies on the Builder having done the correct connections and just pulls out the calculated results. If there needs to be additional control information to be passed, then I would prefer to do a direct wiring of separate control connections to specialized components, which in turn could instruct the controller to change the rendering process.
* to shield the rendering code of all complexities of thread communication and synchronization, we use the StateProxy

Structural Assets are intended mainly for internal use, but the user should be able to see and query them. They are not "loaded" or "created" directly, rather they //leap into existence // by creating or extending some other structures in the session, hence the name. Some of the structural Asset parametrisation can be modified to exert control on some aspects of the Proc Layer's (default) behaviour.
* [[Processing Patterns|ProcPatt]] encode information how to set up some parts of the render network to be created automatically: for example, when building a clip, we use the processing pattern how to decode and pre-process the actual media data.
* [[Forks ("tracks")|Fork]] are one of the dimensions used for organizing the session data. They serve as an Anchor to attach parametrisation of output pipe, overlay mode etc. By [[placing|Placement]] to a track, a media object inherits placement properties from this track.
* [[Pipes|Pipe]] form &mdash; at least as visible to the user &mdash; the basic building block of the render network, because the latter appears to be a collection of interconnected processing pipelines. This is the //outward view; // in fact the render network consists of [[nodes|ProcNode]] and is [[built|Builder]] from the Pipes, clips, effects...[>img[Asset Classess|uml/fig131205.png]]<br/>Yet these //inner workings// of the render proces are implementation detail we tend to conceal.
* [[Sequence]] assets act as a façade to the fundamental compound building blocks within the model, a sequence being a collection of clips placed onto a tree of tracks. Sequences, as well as the top-level tracks enclosed will be created automatically on demand. Of course you may create them deliberately. Without binding it to a timeline or meta-clip, a sequence remains invisible.
* [[Timeline]] assets are the top level structures to access the model; similar to the sequences, they act as façade to relevant parts of the model (BindingMO) and will be created on demand, alongside with a new session if necessary, bound to the new timeline. Likewise, they can be referred by their name-ID
* [[Viewer|ViewerAsset]] assets correspond to the available viewer elements in the GUI. By [[connecting to a viewer|ViewConnection]], session elements derive a concrete (physical) output and gain the ability to be [[played|PlayService]].
!querying
Structural assets can be queried by specifying the specific type (Pipe, Track, ProcPatt) and a query goal. For example, you can {{{Query<Pipe> ("stream(mpeg)")}}}, yielding the first pipe found which declares to have stream type "mpeg". The access point for this querying facility is on the ~StructFactory, which (as usual within Lumiera) can be invoked as static member {{{Struct::retrieve(Query<TY> ...}}}. Given such a query, first an attempt is made to satisfy it by retrieving an existing object of type TY (which might bind variables as a side effect). On failure, a new structural asset of the requested type will be created to satisfy the given goal. In case you want to bypass the resolution step and create a new asset right away, use {{{Struct::retrieve.newInstance<TY>}}}
{{red{Note:}}} in the current implementation no real resolution engine is used (as of 2/2010); rather, we're just able to retrieve a hard-wired answer or create a new asset, simply pattern matching on parts of the query.
&rarr; [[Assets in general|Asset]]

The term ''Subsystem'' denotes a coherent and tightly coupled and integrated part of the application's functionality.
A Subsystem exists as an entity at runtime -- it has state and a lifecycle and might depend on other subsystems.
The arrangement of subsystems is part of the architecture; separation into subsystems is a more fine grained structuring and somewhat orthogonal to the arrangement of functionality into ''Layers'', which are more of conceptual nature.
!Layers and Subsystems
;GUI
:the actual user interface is loaded and started as a plug-in. It is typically monolithic and thus counts as //one// subsystem (but there might be several alternative interfaces)
;~Proc-Layer
:this is the metadata and organisational layer and serves to accomodate the user oriented view to the technical necessities of rendering and playback
:* SessionSubsystem, comprised of the [[Session(datastructure)|Session]] and the [[Builder]]
:* [[Player]]
:* RenderEngine
;Backend
:here the goal is to provide system level services for the upper layers. The structuring is not so clear yet {{red {1/2014}}}
:* probably the [[Scheduler]] gets the ability to be started explicitly
:* the other services are started in conjunction and form a common subsystem

Viewers for displaying output are connected to the timelines or other elements to be displayed through a ViewConnection. This creates an additional mapping step (&rarr; BindingMO), similar to the attachment of a sequence or virtual clip to the [[global busses|GlobalBus]]. Yet the viewers don't provide such an elaborate mixing desk like view as the timelines -- rather there is only a simplified version of the global busses, exposed to the user as ''switch board''.
The corresponding GUI control will allow to choose among possible data sources (esp. ProbePoint) for any given StreamType (more precisely, for every //prototype,// e.g. image, sound). Moreover, there are some (limited) overlay capabilities (split image, mix sound). By default, this additional control will likely be hidden, as there is a default 1:1 association between the master busses of the timeline and the usable output destinations.
!Model and Interfaces
Regarding the internal wiring of components, the Viewer with a switch board behaves similar as a timeline: At the output side, there are a small number of standard OutputDesignation elements for the selection of primary kinds of media handled within this session (typically Video and Audio). But contrary to a timeline, a Viewer element also exposes an OutputManager interface, which is backed by a specific OutputMapping element, which internally connects to the main OutputManager for the whole Application. This way, a Viewer has the capability to get a PlayController

The Lumiera development is //test driven.//
This does not mean, that we're using a formalised method, nor does it mean that we're aiming at a high coverage rate as a value //per se.// But indeed it does mean that we always design the usage situation before we go into the details of the implementation. Every major design work in the application starts out as a usage sketch in the form of a ''test narrative''. But our tests are limited in a characteristic way: We do not write unit tests to cover obvious details of the implementation -- whenever we test, we test abstractions. Which means, we have to bring in abstractions into even the fine grained detail level of implementation. Another consequence is that we don't use any mocking framework, nor do we rely on a dependency injection framework to build our application. Rather, we tend to build in testability and test rigging and diagnostics facilities into every major component of the application.
Over time, a selection of techniques and usage patterns for test support has emerged
!diagnositc output
As a rule, any component of significance has an overloaded {{{operator string()}}}. The {{{<iostream>}}} subsystem from the standard library will use these operators automatically. Moreover, we use a wrapper on top of {{{boost::format}}}, which does so as well, while providing exception safety and reducing code bloat by confining the actual {{{boost::format}}} instantiation to a single compilation unit with preselected specialisations. Thus, tests changing the relevant, tangible state of some application component will change the rendered diagnostic representation. While building those tests, effects can be seen immediately or in the debugger. Later the expected output of running such a test can be easily verified.
!equality and ordering
We go into great lengths of defining identity, equality and ordering operators when appropriate. This can be seen as an investment into the future. While certainly this doesn't pay off all the time, together with the diagnostic output it allows to write tests on the level of symbolic representation.
!test adapters
several core services offer a docking point to attach a test adapter. This special {{{friend class}}} is deeply integrated into the implementation level of said facility and may activate, when established, an extensive and costly additional layer of diagnostic reporting. For example, it may store an audit trail of every invoked state change. This approach allows us to write tests almost like formal specifications. We expect these test adapters to evolve into a diagnostic framework eventually -- Lumiera can be expected to become a challenge for diagnostics and user support, given the extensive use of rules based programming.
!rigged diagnositc setup
We build a collection of hard wired test configurations of the application. These can be activated by use of predefined name-~IDs. Depending on the level in use, a well defined set of processing rules, low-level wiring, mocked diagnostic output, faked input media and specific session contents will be injected. These can be used to write test scenarios against a known setup. Again we expect these provision to evolve into a general diagnostic facility, likely to be accessible for more advanced users too.

A service generating //periodic ticks// &mdash; repetedly invoking a callback with a given frequency.
* defined 1/2009 as part of the PlayerDummy (design sketch)
* probably to be implemented later on based on Posix timers
* will probably be later on integrated into a synchronisation framework (display sync, audio sync, MTC master/slave...)

The Name of the Software driving this Wiki. Is is written completely in ~JavaScript and contained in one single HTML page.
Thus no server and no network connection is needed. Simply open the file in your browser and save changes locally. As the [[Proc-Layer wiki|ProcLayer and Engine]] HTML is located in the Lumiera source tree, all changes will be managed and distributed via GIT. While doing so, you sometimes will have to merge conflicing changes manually in the HTML source.
* see GettingStarted
* see [[Homepage|http://tiddlywiki.org]], [[Wiki-Markup|http://tiddlywiki.org/#Markup]], [[CSS-formatting|http://tiddlywiki.org/#%5B%5BCSS%20Formatting%5D%5D]]

Simple time points are just like values and thus easy to change. The difficulties arise when time values are to be //quantised to an existing time grid.// The first relevant point to note is that for quantised time values, the effect of a change can be disproportional to the cause. Small changes below the threshold might be accumulated, and a tiny change might trigger a jump to the next grid point. While this might be annoying, the yet more complex questions arise when we acknowledge that the amount of the change itself might be related to a time grid.
The problem with modification of quantised values highlights an inner contradiction or conflicting goals within the design
* the whole system should fit in naturally and just feel like using raw time values
* quantisation should be added dynamically and //late// -- like a view
* there should be a guidance towards the intended proper use
!!!general assumptions in Lumiera
At this point, we should recall some general assumptions within Lumiera
* there are no hard-coded defaults regarding the specific nature of the manipulated objects (format, number of channels)
* there is not "the" timeline, but we have multiple timelines
* sequences might be nesetd and thus be brought into a different context
* framerate is a property of the stream type of a channel; it depends on the output. There is no global format or framerate.
!Usage situations
The complexity arises from the mixture of several concerns, which often blend into a single usage situation.
;Time span is an object prototype
:Within the context of an NLE, a time interval located at a given time point can be considered as the least common denominator of all entities to be manipulated by the user. //Manipulating// such objects is the whole point of using such an application.
:* an object or boundary point can be //dragged//
:* we want to //nudge// by defined amounts
:* we want to put it "somewhere"
;dragging
:as far as the GUI is concerned, dragging some element creates an //offset in display coordinates.//
:Now the task is to tranform this into a change on the object, which typically is grid-aligned
;nudging
:the primary interaction in this case just sends an numeric offset of ±N steps
:but the further processing might involve a global or maybe even local //nudge amount.//
:the recieving object is typically grid-aligned, thus nudge-grid and object-grid need to cascade, i.e. be applied in sequence
;placing
:an object or boundary point gets relocated to some predefined time position, which in turn can be quantised (grid aligned).
:now either we treat this target position as a value, to be applied, possibly with re-quantisation
:or we might treat it as a //source for time values// -- in which case the object just gets attached and consumes time coordinates produced elsewhere.
Considering the analysis this far, there seems to be a common denominator in all these operations: chaining of multiple grids.
A mutation or an increment gets applied to a grid-aligned variable, which then (maybe after transformation into a different scale) in turn is used again as a mutation to be applied to another grid-aligned variable. Now, because at this point we don't need to solve the whole bunch of wide scale design problems, it is sufficient to extract the following operations
::value &rArr; value
::value &rArr; quantised
::increment &rArr; quantised
This results in the following combinations to be implemented:
| !mutation|!receiver ||!result |
| ~TimeValue|Time || -- |
|~|Duration || ↯ |
|~|~TimeSpan ||set start |
|~|quantised ||set raw value |
| Duration|~TimeSpan ||set length |
|~|Duration ||set length |
|~|quantised || ↯ |
| Offset|Time || -- |
|~|Duration ||adjust length |
|~|~TimeSpan ||adjust start |
|~|quantised ||adjust raw value |
| ~QuTime|any raw ||materialise, then set raw |
|~|~QuTime ||same &rarr; chanined |
| increment|~QuTime ||snap to grid point |
As rationale, consider the following
* immutable values are bliss.
* by materialising a quantised change prior to applying it, we get the "chaining effect"
----------
this leads to the following solutions approach
* time values are immutable (as far as possible)
* for simple time calculations the ~TimeVar is sufficient
* but we allow to fed an abstract //modification object.//
* ''nudge by unit(s)'' is subsumed here as a special case
As the actual change logic to apply depends both on the target value and the kind of the change, we end up with //double dispatch.//
-- can we avoid accepting mutations for Time (and thus only mutate ~TimeSpan)?
Rationale: allowing mutations for Time bears the danger of making ~TimeVar obsolete
* //4/11 yes, implementing it now this way....//
* we have ~TimeVar, so the only thing that //needs// to be mutable is a whole object &rArr; ~TimeSpan
* err, because MObject will include a Duration separate from the start time in the Placement, Duration needs to be mutable too
!!!usage considerations
{{red{Question 5/11}}} when moving a clip taken from 50fps media, the new position might be quantised to the 50fps grid established by the media, even when the target timeline runs with 25fps, allowing for finer adjustments based on the intermediate frames present in the source material.
!!!design draft
The special focus of this problem seems to lead itself to a __visitor pattern__ based implementation. Because the basic hierarchy of applicable types is fixed, but the behaviour is open ended (and not yet fully determined). A conventional implementation would scatter this behaviour over all the time entities, thus making it hard to understand and reason about. The classical ~GoF cyclic visitor solution to the contrary allows us to arrange closely related behaviour into thematically grouped visitor classes. As a plus, the concrete action can be bound dynamically, allowing for more flexibility when it comes to dealing with the intricate situations when a quantised time span (= a clip) recieves a quantised mutation (= is re-alligend to a possibly different frame grid)
Thus the possibly mutalble time entities get an {{{accept(time::Mutation&)}}} function, which is defined to reflect back into the (virtual) visitation functions of the Mutation Interface. Additionally, we provide some static convenience shortcuts right in the Mutation interface, performing the trivial mutations.
!!!life value changes and monitoring
Based on this time::Mutation design, we provide a specialised element for dealing with //running time values:// When attached to a target time entity, a "life" connection is established. From then on, continuous changes and mutations can be fed to the target by invoking a functor interface. Besides, a change notification signal (callback) can be installed, which will be invoked on each change. This {{{time::Control}}} element is the foundation for implementing all kinds of running time display widgets, spin buttons, timeline selections, playheads, loop playback and similar.

The term &raquo;Time&laquo; spans a variety of vastly different entities. Within a NLE we get to deal with various //flavours of time values.//
;continuous time
:without any additional assumptions, ''points in time'' can be specified with arbitrary precision.
:the time values are just numbers; the point of reference and the meaning is implicit.
:within Lumiera, time is encoded as integral number of //micro ticks,// practically continuous
;time distance
:a range of time, a ''distance'' on the time axis, measured with the same arbitrary precision as time points.
:distances can be determined by //subtracting// two time points, consequently they are //signed numbers.//
;offset
:a distance can be used to adjust (offset) a time or a duration: this means applying a relative change.
:the //target// of an offset operation is a time or duration, while it's //argument// is a distance (synonymous to offset).
;duration
:the length of a time range yields a ''time metric''.
:the duration can be defined as the //absolute value//&nbsp; of the offset between start and endpoint of the time range.
:a duration always abstracts from the time //when// this duration or distance happens, the relation to any time scale remains implicit
;time span
:contrary to a mere duration, a ''time interval'' or time span is actually //anchored// at a specific point in time.
:it can be seen as a //special kind of duration,// which explicitly states the information //when// this time span takes place.
;changing time
:Time values are //immutable,// like numbers. Only a ''time variable'' can be changed.
:yet some of the special time entities can recieve [[mutation messages|TimeMutation]], allowing e.g. for adjustments to a time interval selection from the GUI
;internal time
:While the basic continuous time values don't imply any commitment regarding the time scale and origin to be used, actually, within the implementation of the application, the meaning of time values is uniform and free of contradictions. Thus effectively there is an ''implementation time scale'' -- but its scope of validity is //strictly limited to the implementation level of a single application instance.// It is never exposed and never persisted. It might not be reproducible over multiple instantiations of the application. The implementation reserves the right to recalibrate this internal scale. Later, when Lumiera gains the capability to run within a network of render nodes, these instance connections will include a negotiation about the internal time scale, which remains completely opaque to the outer world. This explains, why {{{lumiera::Time}}} instances lack the ability to show their time value beyond debugging purposes. This is to avoid confusion and to stress their opaque nature.
;wall clock and system time
:The core property of any external real world time is that it is //running// -- we have to synchronise to an external time source.
:This implies the presence of a //running synchronisation process,// with the authority to adjust the time base;
:contrast this to the internal time, which is static and unconnected --
;quantised time
:The ''act of quantisation'' transforms a continuous property into a ''discrete'' structure. Prominent examples can be found in the domain of micro physics and with digital information processing. In a broader sense, any measurement or //quantification// also encompasses a quantisation. Regarding time and time measurement, quantisation means alignment to a predefined ''time grid''. Quantisation necessarily is an //irreversible process// -- possible additional information is discarded.
:Note that quantisation introduces a ''time origin'' and a ''reference scale''
;frame count
:within the context of film and media editing, the specification of a ''frame number'' is an especially important instance of quantisation.
:all the properties of quantisation apply indeed to this special case: it is a time measurement or specification, where the values are aligned to a grid, and there is a reference time point where the counting starts (origin) and a reference scale (frames per second). Handling of quantised time values in Lumiera is defined such as to ensure the presence of all those bits of information. Without such precautions, operating with bare frame numbers leads itself to all kinds of confusions, mismatches, quantisation errors and unnecessary limitations of functionality.
;timecode
:Quantisation also is the foundation of all kinds of formalised time specifications
:actually even a frame count is some kind of (informal) timecode -- other timecodes employ a standardised format.
://every// presentation of time values and every persistent storage and exchange of such values is based on time codes.
:yet quantisation and time code aren't identical: a given quantised time value typically can be cast into multiple timecode formats.
!Patterns for handling quantised time
When it comes to actually handling quantised time values, several patterns are conceivable for dealing with the quantisation operation and representing quantised data. As guideline for judging these patterns, the general properties of time quantisation, as detailed above, should be taken into account. Quantising a time value means both //discarding information,// while at the same time //adding explicit information// pertaining the assumptions of the context.
__casual handling__: this is rather an frequently encountered ''anti pattern''. When reading such code, most striking is the sense of general unawareness of the problem, which is then "discovered" on a per case base, which leads to numerous repetitions of the same basic undertakings, but done with individual treatment of each instance (not so much copy-n-paste). Typical code smells:
* the rounding, modulo and subtract-base operations pertinent with scale handling are seemingly inserted as bugfix
* local code path forks to circumvent or compensate for otherwise hard wired calculations based on specific ways to invoke a function
* playing strikingly clever tricks or employing heuristics to "figure out" the missing scale information from accessible context after the fact
* advertising support for some of the conceivable cases as special feature, or adding it as plugin or extension module with limited scope
* linking parts of the necessary additional information to completely unrelated other structures, thus causing code tangling and scattering
* result or behaviour of calculations depends on the way things are set up in a seemingly contingent way, forcing users to stick to very specific procedures and ordered steps.
[>img[Time and Time Quantisation in Lumiera|uml/fig142725.png]]
__static typing__: an analysis of the cases to be expected establishes common patterns and some base cases, which are then represented by distinct types with well established conversions. This can be combined with generic programming for the common parts. Close to the data input, a factory establishes these statically typed values.
__tagged values__: quantised values are explicitly created out of continuous values by a quantiser entity. These quantised data values contain a copy of the original data, adjusted to be exactly aligned with respect to the underlying time grid. In addition, they carry a tag or ID to denote the respective scale, grid or timecode system. This tag can be used later on to assess compatibility or to recast values into another timecode system.
__delayed quantisation__: with this approach, the information loss is delayed as long as possible. Quantised time values are rather treated as promise for quantisation, while the actual time data remains unaltered. Additionally, they carry a tag, or even a direct link to the responsible quantiser instance. Effectively, these are specialised time values, instances of a sub-concept, able to stand-in for general time values, but exposing additional accessors to get a quantised value.
!!!discussion
For Lumiera, the static typing approach is of limited value -- it excels when values belonging to different scales are actually treated differently. There are such cases, but rather on the data handling level, e.g. sound samples are always handled block wise. But regarding time values, the unifying aspect is more important, which leads to prefering a dynamic (run time typed) approach, while //erasing// the special differences most of the time. Yet the dynamic and open nature of the Lumiera high-level model favours the delayed quantisation pattern; the same values may require different quantisation depending on the larger model context an object is encountered in. This solution might be too general and heavy weight at times though. Thus, for important special cases, the accessors should return tagged values, preferably even with differing static type. Time codes can be integrated this way, but most notably the ''frame numbers'' used for addressing throughout the backend, can be implemented as such specifically typed tagged values; the tag here denotes the quantiser and thus the underlying grid -- it should be implemented as hash-ID for smooth integration with code written in plain C.
At the level of individual timecode formats, we're lacking a common denominator; thus it is preferrable to work with different concrete timecode classes through //generic programming.// This way, each timecode format can expose operations specific only to the given format. Especially, different timecode formats expose different //component fields,// modelled by the generic ''Digxel'' concept. There is a common baseclass ~TCode though, which can be used as marker or for //type erasure.//
&rarr; more on [[usage situations|TimeUsage]]
&rarr; Timecode [[format and quantisation|TimecodeFormat]]
&rarr; Quantiser [[implementation details|QuantiserImpl]]

the following collection of usage situations helps to shape the details of the time values and time quantisation design. &rarr; see also [[time quantisation|TimeQuant]]
;time position of an object
:indeed the term "time position" encompasses two quite different questions
:* a time or timing specification within the object
:* determining the time point in reference to an existing scale
;time and length of an object
:basically the same situation, but length could be treated in two ways for quantisation
:* having a precise specification and then quantise the start and endpoint
:* quantise the start position and then establish an (independently quantised length)
;moving and resizing an object
:this can in itself be done in two different ways, and each of them can be applied in a quantised flavour
:which sums up to 8 possible combinations, considering that position and length are 2 degrees of freedom.
:* a variable can be //changed// by an offset
:* a variable can be //defined// to a new value
:another (hidden) degree of freedom lies in how to apply an quantised offset to an unquantised value (and reversed)
:because this operation might be done both in the quantised or non-quantised domain, and also the result might be (un)quantised
;updating the playback position
:this can be seen as a practical application of the above; basically we can choose to show the wall clock time or we can advance the playback position in frame increments, thus denoting the frame currently in display. For video, these distinctions may look moot, but they are indeed relevant for precise audio editing, especially when combined with loop playback (recall that audio is processed block wise, but the individual sample frames and thus the possible loop positions are way finer than the processing block size)
;dispatching individual frames for calculation
:when a [[render or playback process|PlayProcess]] is created, at some point we need to translate this logical unit ("calculation stream") into individual frame job entries. This requires to break continuous time into individual frames, and then ennumerating these frames.
;displaying time intervals
:for display, time intervals get //re-quantised// into display array coordinates.
:While evidently the display coordinates are themselves quantised and we obviously don't want to cancel out the effect of an quantisation of the values or intervals to be displayed (which means, we get two quantisations chained up after each other), there remains the question if the display array coordinates should be aligned to the grid of the //elements to be displayed,// and especially if the allowed zoom factors should be limited. This decision isn't an easy one, as it has an immediate and tangible effect on what can be showed, how reversible and reproducible a view is and (especially note this!) on the actual values which can be set and changed through the GUI.
;time value arithmetic
:Client code as well as the internal implementation of time handling needs to do arithmetic operations with time values. Time values are additive and totally ordered. Distance, as calculated by subtraction, can be made into a metric. Another and quite different question is to what extent a quantised variant of this arithmetics is required.
;relative placement
:because of the divergence between quantised and unquantised values, the question arises, if placement relative to another object refers to the raw position or the already quantised position. Basically all the variations discussed for //time and length of an object// also do apply here.
!notable issues
''Direct quantisation of length is not possible''. This is due to the non-linear nature of all but the most trivial time grids: Already such a simple addition like a start offset destroys linearity, and this still the more is true within a compound grid where the grid spacing changes at some point. Thus, the length has to be re-established at the target position of an time interval after each change involving quantisation. Regarding the //strategy// to apply when re-establishing the length, it seems more appropriate to treat the object as an entity which is moved, which means to do quantisation in two steps, first the position, then the endpoint (the second option in the description above). But it seems advisable not to hard wire that strategy -- better put it into the quantiser.
We should note, that the problems regarding quantised durations also carry over to //offsets:// it is difficult to ''define the semantics of a quantised offset''. Seemingly the only viable approach is to have a //intended offset,// and then to apply a re-quantisation to the target after applying the (raw) offset.
''When to materialise a quantisation''. Because of the basic intention to retain information, we delay actually applying the quantisation to the stored values as much as possible. But not materialising immediately at quantisation has the downside of possibly accumulating off-grid values without that being evident. Most notably, if we apply the raw offsets produced by GUI interactions, the object's positions and lengthes are bound to accumulate spurious information never intended by the user.
Thus, an especially important instance of that problem is ''how to deal with updates in a quantised environment''. If we handle quantisation stictly as a view employed on output, we run into the problems with accumulating spurious information. On the other hand, allowing for quantised changes inevitably pulls in all the complexity of mixing quantised and non-quantised values. It would be desirable somehow to move these distinctions out of the scope of this design and offload them onto the client (code using these time classes).
Another closely related problem is ''when to allow [[mutations|TimeMutation]]'', if at all. We can't completely do away with mutations, simply because we don't have a pure functional language at our disposal. The whole concept of //reference semantics// doesn't play so well with immutable objects. The Lumiera high-level (session) model certainly relies on objects intended to be //manipulated.// Thus we need a re-settable length field in {{{MObject}}} and we need a time variable for position calculations. Yet we could make any //derived objects// into immutable descriptor records, which certainly helps with parallelism.
The ''problem with playback position'' is -- that indeed it's an attempt to conceptualise a non-existing entity. There is no such thing like "the" playback position. Yet most applications I'm aware off //do// employ this concept. Likely they got trapped by the metaphor of the tape head, again. We should do away with that. On playback, we should show a //projection of wall-clock time onto the expected playback range// -- not more, not less. It should be acknowledged that there is //no direct link to the ongoing playback processes,// besides the fact that they're assumed to sync to wall-clock time as well. Recall, typically there are multiple playback processes going on in compound, and each might run on a different update rate. If we really want a //visual out-of-sync indicator,// we should treat that as a separate reporting facility and display it apart of the playback cursor.
An interesting point to note for the ''frame dispatch step'' is the fact, that in this case quantised values and quantisation are approached in the reverse direction, compared with the other uses. Here, after establishing a start point on the time scale, we proceed with ennumerating distinct frames and later on need to access the corresponding raw time, especially to find out about the [[timeline segment|Segmentation]] to address, or for retrieving parameter automation. &rarr; FrameDispatcher.
Note that the ''display window might be treated as just an independent instance of quantisation''. This is similar to the approach taken above for modifying quantised time span values. We should provide a special kind of time grid, the display coordinates. The origin of these is always defined to the left (lower) side of the interval to be displayed, and they are gauged in screen units (pixels or similar, as used by the GUI toolkit set). The rest is handled by the general quantisation mechanisms. The problem of aligning the display should be transformed into a general facility to align grids, and solved for the general case. Doing so solves the remaining problems with quantised value changes and with ''specifying relative placements'' as well: If we choose to represent them as quantised values, we might (or might not) also choose to apply this //grid-alignment function.//
!the complete time value usage cycle
The way time value and quantisation handling is designed in Lumiera creates a typical usage path, which actually is a one-way route. We might start out with a textual representation according to a specific ''timecode'' format. Assumed we know the implicit underlying ''time grid'' (coordinate system, framerate), this timecode string may be parsed. This brings us (back) to the very origin, which is a raw ~TimeValue (''internal time'' value). Now, this time value might be manipulated, compared to other values, combined into a ''time span'' (time point and duration -- the most basic notion of an //object// to be manipulated in the Session). Anyway, at some point these time values need to be related to some ''time scale'' again, leading to ''quantised'' time values, which -- finally -- can be cast into a timecode format for external representation again, thus closing the circle.
!substantial problems to be solved
* how to [[align multiple grids|TimeGridAlignment]]
* how to integrate [[modifications of quantised values|TimeMutation]].
* how to isolate the Time/Quantisation part from the grid MetaAsset in the session &rarr; we use the [[Advice]] system
* how to design the relation of Timecode, Timecode formatting and Quantisation &rarr; [[more here|TimecodeFormat]]

The handling of [[Timecode]] is closely related to [[time representation and quantisation|TimeQuant]]. In fact, these two topics blend into one another. Time will be quantised into a //grid,// but this grid only makes sense when linked to a externally relevant meaning and representation, which is the Timecode. But a timecode value also denotes a specific point in time -- performing operations on a timecode is equivalent to manipulating a quantised time value.
!Problem of dependencies
The general design of time and time quantisation in Lumiera requires us to do the actual quantisation as late as possible. The relation of the time-like entities creates a //one way route:// starting from the ''internal time'', which is contiguous and definite, but completely opaque for the client, the usage proceeds to a ''quantised time value'', requiring the specification of a concrete ''time grid''. Now, the possible ''timecode formats'' can be determined, and commiting to one specific timecode format finally allows to get an explicit, number-like value, e.g. the frame count, or the seconds part of a SMPTE Timecode.
In this handling sequence, the ~QuTime with the embedded link to a ''quantiser'' plays the role of an coordinating hub.
* a quantised time element depends on a concrete quantiser, which might be fetched by symbolic ID.
* the quantiser draws upon a configured collection of concrete timecode ''formats''.
* only the concrete format knows how to build the components of a value representation in that format, but in doing so, has to rely on the primitives exposed by the quantiser.
* the concrete individual ''timecode value'' is comprised of several value-like components called ''Digxel'' (generalised digits)
* the timecode value can be //(re)-built,// assumed there is a concrete quantised time and a format and a quantiser
!Usage and operations
Timecode //is a value.// It has an explicit and distinct structure, and the component parts can be accessed as plain numeric values, or in a formatted string representation. To this end, these values need to be (re)-built. This operation -- building the value(s) -- is what effectively //materialises// the quantisation: At this point, the actual rounding and truncating operation is performed.
But timecode values can also be //mutated.// In this respect they differ from a plain ~TimeValue, which is immutable like a number.
* the individual components (digxels) can be assigned, causing the rebuilding of the timecode
* the value as a whole can be incremented, decremented or be offset.
And last but not least, it is possible to get a new ~TimeValue, reflecting the current (quantised) time of this timecode. This closes the circle of time handling and quantisation; actually this is the usual way to enter a new time value into the system, by starting out from an explicitly specified grid aligned timecode value.
!!{{red{WIP 1/11}}}design tasks
* @@color:green;✔@@ find out about the connection to the MetaAsset &rarr; [[Advice]]
* @@color:green;✔@@ determine the primitives which need to be on the //effective quantiser API.//
* @@color:green;✔@@ find out how a format can address the individual components it's comprised of &rarr; direct access by concrete type
* @@color:green;✔@@ decide how a concrete TC value can refer to his quantiser &rarr; always using smart-ptr
* @@color:green;✔@@ decide on how to handle wrap-around and negative values &rarr; by strategy
!negative values and wrap-around
Many timecode formats were defined in the era of analogue media -- typically the key point of a timecode format was the way it can be encoded into some nits and bits within the available channel bandwidth. Often this causes the use of fixed field length representations, imposing hard limits on the number range. Adhering strictly to such ancient specifications in the context of a general purpose computer program usually results in lots of additional complexity without a fundamental reason.
When in doubt, for the design of Lumiera we tend to err for the basic mathematical definition. For time values this means to use an practically unlimited time axis with an arbitrary time origin. Under this assumptions, negative time values are just natural and will be handled without case distinctions. This way, at least the core is kept simple and straight forward. Which leaves us with some of the aforementioned additional complexities showing up when it comes to interfacing with an existing timecode format. Especially, the following points need to be clarified:
* wrap-around: when a timecode component (e.g. the seconds) exceeds the defined range, this creates a propagation (e.g. to the minutes) and a remainder (wrapped seconds value).
* range limitation: what happens when an internal time value exceeds the range of possible timecode values? (e.g. what is 23:59:59 + 70 frames?)
* how to extend the definition to negative values, if applicable.
Especially ''SMPTE Timecode'' is limited to the range from 0:0:0:0 to 23:59:59:##.
But because for Lumiera the SMPTE format is just a presentation rule applied to a more orthogonal internal repesentation, we could think of extending the allowed values...
# by just letting the hours increase arbitrarily
# by letting the timecode wrap from 23:59:59:## to 0:0:0:0 and treat this junction as continuous (effectively a "modulo 1 day").
# by making the hours-field signed, but otherwise contain the same wrapping scheme (0:0:0:0 - 1sec = -1:59:59:0)
# by a representation symmetrical to the zero point (0:0:0:0 - 1sec = -0:0:1:0) -- but beware: //the underlying frame quantisation won't flip//
Because this decision is considered arbitrary and any reasoning will be context dependant, the decision is to provide a hook for a //strategy// --
Currently (1/11), the strategy is implemented according to (1) and (4) above, leaving the actual configuration of a strategy for later.
!!{{red{WIP 1/11}}}BUG
Implementation of this strategy is still broken: it doesn't work properly when actually the change passing over the zero point happens by propagation from lower digits. Because then -- given the way the mutators are implemented -- the //new value of the wrapping digit hasn't been stored.// It seems the only sensible solution is to change the definition of the functors, so that any value will be changed by side-effect {{red{Question 4/11 -- isn't this done and fixed by now??}}}

Timeline is the top level element within the [[Session (Project)|Session]]. It is visible within a [[timeline view in the GUI|GuiTimelineView]] and represents the effective (resulting) arrangement of media objects, to be rendered for output or viewed in a Monitor (viewer window). A timeline is comprised of:
* a time axis in abolute time ({{red{WIP 1/10}}}: not clear if this is an entity or just a conceptual definition)
* a list of [[global Pipes|GlobalPipe]] representing the possible outputs (master busses)
* //exactly one// top-level [[Sequence]], which in turn may contain further nested Sequences.
* when used for Playback, a ViewConnection is necessary, allowing to get or connect to a PlayController
Please note especially that following this design //a timeline doesn't define tracks.// [[Tracks form a Tree|Fork]] and are part of the individual sequences, together with the media objects placed to this //track fork.// Thus sequences are independent entities which may exist stand-alone within the model, while a timeline is //always bound to hold a sequence.// &rarr; see ModelDependencies
[>img[Fundamental object relations used in the session|uml/fig136453.png]]
Within the Project, there may be ''multiple timelines'', to be viewed and rendered independently. But, being the top-level entities, multiple timelines may not be combined further. You can always just render (or view) one specific timeline. A given sequence may be referred directly or indirectly from multiple timelines though. A given timeline is represented within the GUI according to [[distinct principles and conventions|GuiTimelineView]]
''Note'': in early drafts of the design (2007) there was an entity called "Timeline" within the [[Fixture]]. This entity seems superfluous and has been dropped. It never got any relevance in existing code and at most was used in some code comments.
!Façade and implementation
Actually, Timeline is both an interface and acts as façade. Its an interface, because we'll need "timeline views" ({{red{really? is that a reason to create a hierarchy right here, or shouldn't that be rather conceptual?}}}. It is a façade to the raw structures in the model, in this case a {{{Placement<BindingMO>}}} attached immediately below the [[root scope|ModelRootMO]]. The implementation of the timeline(s) is maintained as StructAsset within the AssetManager, managed by shared-ptr. It always depends on a [[Sequence]], which might be created automatically as empty container when referring to a timeline.
Besides building on the asset management, implementing Timeline (and Sequence) as StructAsset yields another benefit: ~StructAssets can be retrieved by query, allowing to specify more details of the configuration immediately on creation. //But on the short term, this approach causes problems:// there is no real inference engine integrated into Lumiera yet (as of 2/2010 the plan is to get an early alpha working end to end first). For now we're bound to use the {{{fake-configrules}}} and to rely on a hard wired simulation of the intended behaviour of a real query resolution. Just some special magic queries will work for now, but that's enough to get ahead.

As [[detailed here|GuiTimelineSlave]], there are various good reasons to provide several UI views onto the same timeline. Yet taking an architectural viewpoint, we prefer representing such a slave display attachment as first-class citizen, right within the session model. Some further problems remain to be settled
* the primary Timeline entity need to be aware of the clone's presence, since any effects of a Builder run must be propagated through the latter
* we need a way to duplicate the diff feed, so the UI element representing the clone gets notified appropriately
* and even when following this design approach with a //materialised duplication,// we still need some awareness and collaboration among the UI elements involved
This topic is {{red{postponed as of 10/2018}}} &rarr; [[#1083|http://issues.lumiera.org/ticket/1083]]

//guide and control the concrete display properties of the various sub components (tracks, clips) comprising a timeline display.//
The TimelineDisplayManager actually is an abstraction, a control interface, revolving around the guidance individual components need in order to settle on a proper display. Those components are represented as mediating entities, the TrackPresenter and the ClipPresenter, each of which controls and manages a mostly passive GTK widget. To this end, the presenters need to know at which virtual coordinates their corresponding widgets would show up, and they need to know if these widgets need to be actually present at the moment. Also, especially the ClipPresenter needs to know which ''clip appearance style'' to choose for the corresponding slave widget.
!display evaluation pass
[>img[Clip presentation control|uml/Timeline-display-evaluation.png]]Since the timeline display is formed by several nested collections of elements, a proper display layout must incorporate information from all those parts. A naive approach likely would just iterate over them and reach in to extract data -- or even worse, build a separate display data store, which then needs to be kept in sync with the component hierarchy. Any of this would lead to high coupling and turns any necessary adjustment and changes due to evolution of requirements into a liability. Such a dangerous situation can be avoided altogether by admitting the collaborative nature of the task at hand. To get there, we need to distil the abstraction shared between the global and the local concerns, and we need to shape it such as to build onto some kind of invariant, allowing that abstraction to re-emerge at several levels of locality.

There is a three-level hierarchy: [[Project|Session]], [[Timeline]], [[Sequence]]. Each project can contain ''multiple timelines'', to be viewed and rendered independently. But, being the top-level entities, these timelines may not be combined further. You can always just render (or view) one specific timeline. Each of those timelines refers to a Sequence, which is a bunch of [[media objects|MObject]] placed to a [[fork ("tree of tracks")|Fork]]. Of course it is possible to use ~sub-sequences within the top-level sequence within a timeline to organize a movie into several scenes or chapters.
[>img[Relation of Timelines, Sequences and MObjects within the Project|uml/fig132741.png]]
As stated in the [[definition|Timeline]], a timeline refers to exactly one sequence, and the latter defines a [[tree of tracks|Fork]] and a bunch of media objects placed to these tracks. A Sequence may optionally also contain nested sequences as [[meta-clips|VirtualClip]]. Moreover, obviously several timelines (top-level entities) may refer to the same Sequence without problems.
This is because the top-level entities (Timelines) are not permitted to be combined further. You may play or render a given timeline, you may even play several timelines simultaneously in different monitor windows, and these different timelines may incorporate the same sequence in a different way. The Sequence just defines the relations between some objects and may be placed relatively to another object (clip, label,...) or similar reference point, or even anchored at an absolute time if desired. In a similar open fashion, within the track-tree of a sequence, we may define a specific signal routing, or we may just fall back to automatic output wiring.
!Attaching output
The Timeline owns a list of global [[pipes (busses)|Pipe]] which are used to collect output. If the track tree of a sequence doesn't contain specific routing advice, then connections will be done directly to these global pipes in order and by matching StreamType (i.e. typically video to video master, audio to stereo audio master). When a monitor (viewer window) is attached to this timeline, similar output connections are made from those global pipes, i.e. the video display will take the contents of the first video (master) bus, and the first stereo audio pipe will be pulled and sent to system audio out. The timeline owns a ''play control'' shared by all attached viewers and coordinating the rendering-for-viewing. Similarly, a render task may be attached to the timeline to pull the pipes needed for a given kind of generated output. The actual implementation of the play controller and the coordination of render tasks is located in the Backend, which uses the service of the Proc-Layer to pull the respective exit nodes of the render engine network.
!Timeline versus Timeline View
Actually, what the [[GUI creates and uses|GuiTimelineView]] is the //view// of a given timeline. This makes no difference to start with, as the view is modelled to be a sub-concept of "timeline" and thus can stand-in. All different views of the //same// timeline also share one single play control instance, i.e. they all have one single playhead position. Doing it this way should be the default, because it's the least confusing. Anyway, it's also possible to create multiple //independent timelines// &mdash; in an extreme case even so when referring to the same top-level sequence. This configuration gives the ability to play the same arrangement in parallel with multiple independent play controllers (and thus independent playhead positions)
To complement this possibilities, I'd propose to give the //timeline view// the possibility to be re-linked to a sub-sequence. This way, it would stay connected to the main play control, but at the same time show a sub-sequence //in the way it will be treated as embedded// within the top-level sequence. This would be the default operation mode when a meta-clip is opened (and showed in a separate tab with such a linked timeline view). The reason for this proposed handling is again to give the user the least surprising behaviour. Because, when &mdash; on the contrary &mdash; the sub-sequence would be opened as //separate timeline,// a different absolute time position and a different signal routing may result; doing such should be reserved for advanced use, e.g. when multiple editors cooperate on a single project and a sequence has to be prepared in isolation prior to being integrated in the global sequence (featuring the whole movie).

//Timing constraints of a render or playback process.//
!constituents
The Timings record is used at various places at the engine interface level, since it links together several pieces of information crucial for playback control.
* a reference to the underlying frame grid.
** the frame grid defines time origin and framerate
** additionally the grid translates nominal time into frame numbers
* the Timings record provides information about latency -- both output latency and engine latency are exposed here
* indication of the service class or playback urgency is included, e.g. {{{ASAP, NICE, TIMEBOUND}}}
* from the usage view the Timings record represents a //constraint// -- with the ability to specify additional constraining conditions
!!!!role of the ~TimeAnchor
While the timings define the nature of a specific playback feed, they omit the actual link to wall clock time. The latter is established and maintained during the planning process, which preceds in chunks of evaluation: each planning chunk is pinned to a specific real time deadline through the ''time anchor'' used as closure to define this planning chunk. Whenever a chunk of frames has been planned, a new time anchor is established and scheduled as a follow-up job to pick up the frame planning work.
note: still {{red{WIP as of 1/2013}}}
!where are the timings definied?
Timing constraint records are used at various places within the engine. Basically we need such a timings record for starting any kind of playback or render.
The effective timings are established when //allocating an OutputSlot// -- based on the timings defined for the ModelPort to be //performed,// i.e. the global bus to render.

What //exactly&nbsp;// is denoted by &raquo;Track&laquo; &mdash; //basically&nbsp;// a working area to group media objects placed at this track at various time positions &mdash; varies depending on context:
* viewed as [[structural asset|StructAsset]], tracks are nothing but global identifiers (possibly with attached tags and description)
* regarding the structure //within each [[Sequence]],// tracks form a tree-like grid, the individual track being attached to this tree by a [[Placement]], thus setting up properties of placement (time reference origin, output connection, layer, pan) which will be inherited down to any objects located on this track and on child tracks, if not overridden more locally.
* with respect to //object identity,// a given track-ID can have an incarnation or manifestation as real track-object within several [[sequences|Sequence]] (meaning you could select, disable or choose for rendering all objects in any sequence placed onto this track). Moreover, the track-//object// and the //placement&nbsp;// of this track within the tree of tracks of a given sequence are two distinguishable entities (meaning a given track &mdash; with a couple of objects located on it &mdash; could be placed differently several times within the same sequence {{red{really??}}}, for example with different start offset or with different layering, output mode or pan position)
{{red{3/2010: no!}}} &mdash; it doesn't work this way. Relative placements attach to other placements, not ~MObjects. Thuse the clips in this example are within the scope of //one// placement. We could create //another// placement of the same track and attach it to to a differenc sequence, but this other placement wouldn't "inherit" the clips attached to the first one!
!Identification
Tracks thus represent a blend of several concepts, but depending on the context it is allways clear which aspect is meant. Seen as [[assets|Asset]], tracks are known by a unique track-ID, which can be either [[queried|ConfigQuery]], or directly refered to by use of the asset-ID (which is a globally known hash). Usually, all referrals are via track-ID, including when you [[place|Placement]] an object onto a track.
Under some cincumstances though, especially from within the [[Builder]], we refer to a {{{Placement<Track>}}} rather, denoting a specific instantiation located at a distinct node within the tree of tracks of a given sequence. These latter referrals are always done by direct object reference, e.g. while traversing the track tree (generally there is no way to refer to a placement by name {{red{Note 3/2010 meanwhile we have placementIDs and we have ~MObjectRef. TODO better wording}}}).
!creating tracks
Similar to [[pipes|Pipe]] and [[processing patterns|ProcPatt]], track-assets need not be created, but rather leap into existence on first referral. On the contrary, you need to explicitly create the {{{Placement<Fork>}}} for attaching it to some node within the tree of tracks of an sequence. The public access point for creating such a placement is {{{MObject::create(forkID}}} (i.e. the ~MObjectFactory. Here, the {{{forkID}}} denotes the track-asset. This placement, as returned from the ~MObjectFactory isn't attached to the session yet; {{{Session::current->attach(trackPlacement)}}} performs this step by creating a copy managed by the PlacementIndex and attaching it at the current QueryFocus, which was assumed to point to a track previously. Usually, client code would use one of the provided convenience shortcuts for this rather involved call sequence:
* the interface of the fork-~MObjects exposes a function for adding new child forks.
* the session API contains a function to attach child tracks. In both cases, existing forks can be referred by plain textual ID.
* any MObjectRef to an object within the session allows to attach another placement or ~MObjectRef
!removal
Deleting a Fork is an operation with drastic consequences, as it will cause the removal of all child forks and the deletion of //all object placements to this fork,// which could cause the resepctive objects to go out of scope (being deleted automatically by the placements or other smart pointer classes in charge of them). If the removed fork was the fork root ("root track") of a sequence, this sequence and any timeline or VirtualClip binding to it will be killed as well. Deleting of objects can be achieved by {{{MObjectRef::purge()}}} or {{{Session::purge(MObjectRef)}}}
!using Tracks
The '''Track Asset''' is a rather static object with limited capabilities. It's main purpose is to be a point of referral. Track assets have a description field and you may assign a list of [[tags|Tag]] to them (which could be used for binding ConfigRules). Note that track assets are globally known within the session, they can't be limited to just one [[Sequence]] (but you are allways free not to refer to some track from a given sequence). By virtue of this global nature, you can utilize the track assets to enable/disable a bunch of objects irrespective of what sequence they are located in, and probably it's a good idea to allow the selection of specific tracks for rendering.
Matters are quite different for the placement of a Track within the tree of tracks of a given sequence, and for placing some media object onto a given track. The track placement defines properties which will be inherited to all objects on this track and on all child tracks and thus plays a key role for wiring the objects up to some output pipe. Typically, the top level track of each sequence has a placement-to "the" video and "the" audio master pipe.
!!!!details to note
* Tracks are global, but the placement of a track is local within one sequence
* when objects are placed onto a track, this is done by referal to the global fork asset ID. But because this placement of some media object is allways inherently contained within one sequence, the //meaning&nbsp;// of such a placement is to connect to the properties of any track-placement of this given track //within this sequence.//
* thus tracks-as-ID appear as something global, but tracks-as-propperty-carrier appear to the user as something local and object-like.
* in an extreme case, you'll add two different placements of a track at different points within the track tree of an sequence. And because the objects placed onto a track refer to the global track-ID, every object "on" this track //within this sequence&nbsp;// will show up two times independently and possibly with different inherited properties (output pipe, layering mode, pan, temporal position)
* an interesting configuration results from the fact that you can use an sequence as a [["meta clip" or "virtual clip"|VirtualClip]] nested within another sequence. In this case, you'll probably configure the tracks of the "inner" sequence such as to send their output not to a global pipe but rather to the [[source ports|ClipSourcePort]] of the virtual clip (which are effectively local pipes). Thus, within the "outer" sequence, you could attach effects to the virutal clip, combine it with transitions and place it onto another track, and any missing properties of this latter placement are to be resolved within the "outer" sequence <br/>(it would be perfectly legal to construct a contrieved example when using the same track-ID within "inner" and the "outer" sequence. Because the Placement of this track will probably be different in the both sequences, the behaviour of this placement could be quite different in the "inner" and the "outer" sequence. All of this may seem weird when discussed here in a textual and logical manner, but when viewed within the context and meaning of the various entities of the application, it's rather the way you'd expect it to be: you work locally and things behave as defined locally)
* note further, the root of the tree of tracks within each sequence //is itself again a //{{{Placement<Fork>}}}. There is no necessitiy for doing it this way, but it seemed more stright forward and logical to Ichthyo, as it allowes for an easy way of configuring some things (like ouput connections) as a default within one sequence. As every track can have a list of child tracks, you'll get the "list of tracks" you'd expect.
* a nice consequence of the latter is: if you create a new sequence, it automatically gets one top-level track to start with, and this track will get a default configured placement (according to what is defined as [[default|DefaultsManagement]] within the current ConfigRules) &mdash; typically starting at t=0 and being plugged into the master video and master audio pipe
* nothing prevents us from putting several objects at the same temporal location within one track. If the builder can't derive any additional layering information (which could be provided by some other configuration rules), then //there is no layering precedence// &mdash; simply the object encountered first (or last) wins.
* obviously, one wants the __edit function__ used to create such an overlapping placement&nbsp; also to create an [[transition|TransitionsHandling]] between the overlapping objects. Meaning this edit function will automatically create an transition processor object and provide it with a placement such as to attach it to the region of overlap.

''towards a definition of »Track«''. We don't want to tie ourself to some naive and overly simplistic definition, just because it is convenient. For classical (analogue) media, tracks are physical entities dictated by the nature of the process by which the media works. Especially, Tape machines have read/writing heads, which creates fixed tracks to which to route the signals. This is a practical geometric necessity. For digital media, there is no such necessity. We are bound primarily by the editor's habits of working.
!!!Assessment of Properties
Media are used as Clips (contiguous chunks), they are a compound of several elementary streams, and they have a temporal extension (length). Indeed, the temporal dimension is the only fundamental property that can't be changed. Orthogonal to this dimension, we find one or more organisational dimensions forming a grid:
* a media stream may be sent to one of several possible output destinations (image or sound, subgroup busses, MIDI instruments)
* for any given output destination there may be variations in the //way of connecting// (overlay mode and layer, pan position, MIDI channel)
* besides, we often just want to stash away some clip without using it, e.g. as an alternative or for later referral
This is to say we have //several degrees of freedom// within this organisational grid. Just because some sound is located on this track doesn't mean he will be sent to a given output, rather the clip is located on this track //and// is connected to that output //and// &mdash; supposed we have full-periphonic surround sound &mdash; it is located 60° to the right and with 30° elevation. Combined with the (always contiguous) temporal dimension, this discrete grid is thus extruded to form something like discrete Tracks.
!!!do we really need Tracks?
Starting with the assumption "everything is just connected processing nodes", Tracks may seem superfluous. The problem with this approach is: it doesn't scale well. While it is fine to be able to connect clips and effects as we see fit (indeed, we want to build such a system), it is clearly not feasible to wire every clip manually to the output ports or add a panner effect to each and every audio sample. Because while editing, most of the time things are done in a fairly regular and systematic manner. Preferably we use the tracks as //preconfigured group setup// and just //place media onto them;// any such [[Placement]] can do the necessary wiring semi-automatic (rule-based).
!!!the constant part
there seems to be some non time-varying part in each sequence, that doesn't fit well with the simple model "objects on a timeline". Tracks seen as an organisational grid fall into this category: they are a global property of the given sequence. They could be associated to the Session as a whole, but effectively this would subvert the concept of having [[several sequences|SessionOverview]]. On the other hand,
[[pipes|Pipe]] for Video and Sound output are obviously a global property of the Session. There can be several global pipes forming a matrix of subgroup busses. We could add ports or pipes to tracks by default as well, but we don't do this, because, again, this would run counter to our attempt of treating tracks as merely organisational entities. We have special [[source ports|ClipSourcePort]] on individual clips though, and we will have ports on [[virtual clips|VirtualClip]] too.
!Design
[[Forks ("tracks")|Fork]] are just a structure used to organize the Media Objects within the session. They form a grid, and besides that, they have no special meaning. It seems convenient to make the tracks not just a list, but allow grouping (tree structure) right from start. __~MObjects__ are ''placed'' rather than wired. The wiring is derived from the __Placement__. Placing can happen in several dimensions:
* placing in time will define when to activate and show the object.
* placing onto a track associates the ~MObject with this track; the GUI will show it on this track and the track may be used to resolve other properties of the object.
* placing to a __Pipe__ brings the object in conjunction with this pipe for the build process. It will be considered when building the render network for this pipe. Source-like objects (clips and exit nodes of effect chains) will be connected to the pipe, while transforming objects (effects) are inserted at the pipe. (you may read "placed to pipe X" as "plug into pipe X")
* depending on the nature of the pipe and the source, placing to some pipe may create additional degrees of freedom, demanding the object to be placed in this new, additional dimensions: Connecting to video out e.g. creates an overlay mode and a layer position which need to be specified, while connecting to a spatial sound system creates the necessity of a pan position. On the other hand, placing a mono clip onto a mono Pipe creates no additional degrees of freedom.
Placements are __resolved__ resulting in an ExplicitPlacement. In most cases this is just a gathering of properties, but as Placements can be incomplete and relative, there is room for real solving. The resolving mechanism tries to __derive missing properties__ from the __context__: When a clip isn't placed to some pipe but to a Track, than the Track and its parents will be inspected. If some of them has been placed to a pipe, the object will be connected to this pipe. Similar for layers and pan position. This is done by [[Placement]] and LocatingPin; as the [[Builder]] uses ~ExplicitPlacements, he isn't concerned with this resolving and uses just the data they deliver to drive the [[basic building operations|BasicBuildingOperations]]
&rarr; [[Definition|Fork]] and [[handling of Tracks|TrackHandling]]
&rarr; [[Definition|Pipe]] and [[handling of Pipes|PipeHandling]]

//mediating entity used to guide and control the track-like nested working space in the timeline display of the UI.//
Similar to the ClipPresenter, from a global angle this element fulfils a model like role, while also guiding and controlling a mostly passive view component implemented as GTK widget. Here, the authority of the presenter over the widget must be total, since display management //needs to work automatically,// due to model updates and mutations arriving as [[diff messages|MutationMessage]]. In addition, this structure is prerequisite for (possibly) implementing UI rendering optimisations, since it allows us to leave out widgets entirely, when it is clear they won't become visible: A ''display evaluation pass'', which is effectively a //tree walk,// consecutively visits each part of the timeline structure, to negotiate its concrete display properties in collaboration with a global TimelineDisplayManager. As a result, the presenter knows where to show its corresponding view, and it knows if to show it at all, allowing to either adjust, create or destroy actual GTK widgets within its local reference frame.
A special twist arises from the fact that track display has to happen aligned and in sync within the two display panes of the timeline at the same time. This means that each TrackPresenter has to hold and manage //two slave display elements,// each of which is inserted within a disjoint hierarchy of display elements. For one, there is the {{{timeline::TrackHeadWidget}}} which in turn renders a {{{PatchbayWidget}}}, and on the other side there is a {{{TrackBody}}} element, which is not strictly a widget of itself, but inserted into our custom drawing {{{timeline::BodyCanvasWidget}}} to manage the custom drawing of the respective track working area.
To deal with this typical problem of recursive programming, we introduce a binding element, the {{{DisplayFrame}}}
* the TrackPresenter holds a {{{DisplayFrame}}} member, which in turn houses the two head and body widgets. So basically the widgets are allocated within the presenter.
* however, after constructing the {{{DisplayFrame}}}, it invokes an anchor functor, which is passed in from the recursive outer scope, and which does the actual work of installing those widgets into the apropriate parent widgets
* but the {{{DisplayFrame}}} itself also exposes such an anchor functor, which can be passed down when (recursively) creating a TrackPresenter for a sub-Track
* obviously, at top-level, we need to digress from that general pattern -- here it is the {{{timeline::LayoutManager}}} responsible for the display structure of the whole timeline which exposes the necessary anchor function to attach the root track for a timeline.

Transitions combine the data from at least two processing chains and do this combining in a time varying fashion. So, any transition has
* N input connections
* either one or N output connections
* temporal coordinates (time, length)
* some control data connection to a ParamProvider, because in the most general case the controling curves are treated similar to [[automation data|AutomationData]]
!!!how much output ports?
The standard case of a transition is sort of mixing together two input streams, like e.g. a simple dissolve. For this to be of any use, this input streams need to be connected to the same ouput destination before and after the transition (with regards to the timeline), i.e. the inputs and the transition share placement to the same output pipe. In this case, when the transition starts, the direct connections can be suspended and the transition will switch in seamlessly.
Using transitions is a very basic task and thus needs viable support by the GUI. Handling of transitions need to be very convienient, because it is so important. Because of this, it is compelling to subsume a more complicated situation and treat this more complicated case similar. This is the case, when two (or N) elements have to be combined in the way of a transition, but their further processing in the processing chain //after// the transition needs to be different, maybe they belong to differnent subgroups, have to appear on different layers or with different pan positions. Of courese, the workaround is simple, at least "simple" from the programmers point of view. It is //not// simple from the editor's point of view the moment the editor has to do this simple thing (changing the wiring and add manualy synced fade curves to the individual parts) a dozend or even a hundred times in some larger project.
Because of this experience, ichthyo wants to support a more general case of transitions, which have N output connections, behave similar to their "simple" counterpart, but leave out the mixing step. As a plus, such transitions can be inserted at the source ports of N clips or between any intermediary or final output pipes as well. Any transition processor capable of handling this situation should provide some flag, in order to decide if he can be placed in such a manner. (wichin the builder, encountering a inconsistently placed transition is just an [[building error|BuildingError]])

At various points within the implementation we encounter problems of structural difference computation, like finding the effective changes in a tree data structure, or using a diff represetation as notification message format. It might be in place to point out some observations, especially relating our approach(es) to the problem as generally known .
!classical diff problem
There is very wide spread use of a very specific flavour of diff calculations: the classical diff of textual data. Here, the treated data has a completely uniform linear structure -- it is either a sequence of lines or a sequence of bytes ("binary diff"). The individual data cells are built from a finite alphabet of comparable values (characters). Since a given character or word may be present several times in the same document, we need to establish a //matching// as foundation of any difference calculation. We need to establish the identical parts to find the differences. This matching is not unique, and largely determines the differences found, so it is hard to determine if a given diff is //correct.// Thus, the core idea to attack and solve this classical diff problem was to look for an ''optimal diff''. Some variations include
* longest common substrings
* largest common subsequence
* minimal edit script to perform the diff
!challenge of the diff problem
While, from a mathematical point of view, the above optimisation problem can be considered as solved, in practice the available solutions are far from perfect. The fundamental assumption of a linear sequence to base the diff turns out as an oversimplification -- real world data carries meaning and can be judged by imposing additional structure. There can be structure violating diffs and there can be nonsensical and misguiding diffs. In the light of this notion, every diff is in fact a structural diff.
Unfortunately, attempts to amend the beautiful mathematical solution by incorporating this additional structure turns out to be incredible hard. Either, the general case problem can be proven to be ~NP-hard, or, at least, exploiting some special structural properties successfully renders the meaning of an "optimal" diff solution more or less arbitrary. The quest for a generic diff problem and an universal solution turns out to be a dead end.
!fundamentals of diff handling
The most fundamental distinction is the difference between //finding// a diff and //representing// a diff. The former is concerned with uncovering structural relations, while the latter deals with knowledge about structural relations and thus is more general. It is possible to capture structural relations while they emerge -- this way describing a process of transformation. A likewise fundamental distinction is between //reordering// elements and //mutating// them. This is related to the notion of //identity,// which in turn implies an underlying model of the elements and entities to be considered for diffing. If, as an example, we model the tokens, functions and classes of program code as mere characters, we will be happily matching curly braces and can not expect meaningful diffs. So first we need a theory about what can possibly happen, and we use this as a foundation to establish the representation of such possible processes. The choices taken here can make all the difference towards efficient and usable methods. The //matching problem// should be viewed from here: a matching is actually a hypothesis about possible processes of transformation -- this observation explains why the matching strategy is at the core of any diff algorithm. If we use some arbitrary notation of optimality, we have to consider endless combinations and end up with a lot of accidental complexity. Yet if we manage to base the calculation on a representation well aligned with the nature of the entities in question, the matching can be as simple as retrieving a given entry by ID. The actual work is reduced to extracting the raw changes in data.
Before we can consider a diffing technique, we need to clarify the primitive operations used as foundation. These primitives form //a language.// This incurs a problem and choice of context. We might base the representation language on the situation where the diff is applied, we may care for the conciseness of the representation, or the effort and space required to apply it. We should consider if the focus is on reading and understanding the diff, if we intend to derive something directly from the diff representation -- say, if and what and when something is changed -- and we should consider how the context where the diff is established relates to the context where it is applied.
&rarr; [[Implementation considerations|TreeDiffImplementation]]
!diff as data representation
We haven't said anything regarding the purpose of dif representation yet. There might be several. A human reader, for instance, may use a diff description to spot and judge the temporal evolution of data and structures. Another approach is to use diff representation as ''architecture for data exchange''. Connection through //diff messages// allows collaboration //without the need for shared data.// Going this route helps with focus and parallelisation. A self contained subsystem based on a private data representation is requires less resources for maintenance and inception. It incurs less overhead for coupling and dependencies with other subsystems, both in terms of human understanding and machine execution power.
Used this way, diff representation helps to separate structure and raw data in exchange. Any changes are broken down into mutations of the structure and chunks of atomic data updates. The data still embedded into the diff description is reduced to //elementary data.// The intricacies of typing and type based special treatment of data is stripped out by this usage pattern. Types aren't going away, yet they are subsumed to the structure and pushed to the inside. //Elementary data// is data form a small selection of fundamental elementary types, all of which can be represented textually: strings, integers, floats, date, time and binary byte sequences. This data is complemented by a small number of //structuring primitives:// the ordered sequence, the unordered set and the associative array of (key, value) pairs. A complete diff representation language needs provisions for any of these fundamentals.
;internal diff
:this kind of exchange language is usable on the level of implementation. It confines raw data to the small set of basic types outlined above.
:These are still passed on in binary form, either as value copies, by an ID based lookup mechanism, or by reference, supported by a synchronisation protocol
;external diff
:going full circle, this representation abstracts completely and removes the dependency on any binary format.
:Chunks of raw data are attached inline to the structural diff, assuming that each element implicitly knows the kind of data to expect

//This page details decisions taken for implementation of Lumiera's diff handling framework//
This topic is rather abstract, since diff handling is multi purpose within Lumiera: Diff representation is seen as a meta language and abstraction mechanism; it enables tight collaboration without the need to tie and tangle the involved implementation data structures. Used this way, diff representation reduces coupling and helps to cut down overall complexity -- so to justify the considerable amount of complexity seen within the diff framework implementation.
!outline of the diff handling framework
Provided as a loosely coupled collection of tools in the namespace {{{lib::diff}}}, the diff framework revolves around generic representation and handling of structural differences. Beyond some rather general assumptions, to avoid stipulating the usage of specific data elements or containers, the framework is cast in terms of ''elements'', ''sequences'' and ''strategies'' for access, indexing and traversal.
;elements
:the atomic units treated in diff detection, representation and application are considered to be
:*lightweight copyable values
:*equality comparable
:*bearing distinct identity
:*unique //as far as relevant//
;data sequence
:any collection of data is delivered in the form of a sequence, which is //stable// (retains a given sequence order), and allows for defined traversal in that order.
;diff language
:a diff represents the changes necessary to transform an input sequence ("old sequence") into a target sequence ("new sequence")
:*differences are spelled out in ''linearised form'': as a sequence of constant-size diff actions (called &raquo;diff verbs&laquo;)
:*these are conceived as operations, which, when applied consuming the input sequence, will produce the target sequence of the diff.
!building a list diff detector
transforming the algorithm sketch (&rarr; see [[technical documentation|http://lumiera.org/documentation/technical/library/DiffFramework.html]]) into working code incurs some challenges at the level of technical details.
!!!interface of the index component
Obviously we want the helper indices to be an internal component abstraction, so the outline of the algorithm remains legible. Additionally this allows to introduce a strategy for obtaining the index. This is important, since in case of performance problems we might consider to integrate the indexing efforts into the external data structure, because this might enable to exploit external structural knowledge.
So the challenge is to come up with an API not too high-level and not too low-level
!!!how to implement re-ordering
A first attempt was made to come up with a rather clever swapping and rotating scheme. But this turned out to be rather complex, due to the fact that we //do not want index numbers in the diff representation.// It turns out that a presumably optimal solution could be built on top of ''cycle sort'' -- but we're lacking any point of reference to determine if such an elaborate solution is worth the effort. Thus the decision to KISS and stick to plain flat insertion sort (which is known to be the best choice for small data sets). With clever usage of the indices, this approach allows us to emit the diff description right away, without any need to build a meta table or even to touch the given input sequences.
!!!goal of a generic implementation
The diff handling framework we intend to build here is meant to be //generic// -- the actual element data type, as well as the underlying data structure and the index access shall be supplied by strategy and specialisation. This has the downside (maybe this is even a benefit?) that most efficiency considerations are moot at this level; we need to look at actual use cases and investigate the composite performance in practice later.
!building the tree diff handling
Our tree diff handling framework is conceived as a direct specialisation of and extension to list diffing. We rely on a very specific element type for the tree nodes: the GenNode. This specific node element type is inherently generic, and the tree diff handling explicitly relies on some part of this generic nature to represent object features. Following the gist of the linearised list diff representation, also tree handling relies on an implicit ''tree representation protocol'':
* every GenNode is a distinct element and considered unique within context.
* nodes are inherently typed, but this typing is outside of the diff representation scope
* nodes and subtrees are spelled out in prefix order; a node optionally is followed by a list of children.
* nodes representing //objects// may have attribute children, which are always mentioned before the sub node children.
* when a node is //opened for diffing,// we first spell out any structural alterations (added, deleted or re-ordered children)
* the recursive descent into children happens postfix depth-first, each enclosed into a bracketing construct.
Consequently, when an element appears in the diff description sequence, first of all, its type is assumed to be known implicitly, so the receiver can use an appropriate handler for this kind of element. Moreover, if the element is of atomic type (an attribute), its value is part of the element itself and thus is known just by spelling out the element. Any structural changes can be dealt with on a completely generic level, without further knowledge about the object's nature. And finally, any internal alterations of the object and its children happen after the generic part and clearly delineated in the sequence of diff tokens -- a sub handler can be invoked recursively
!!!problem of the typed context
This is a problem every introspective framework has to face. When individual elements in the data structure require specific type information for proper handling, this information must be readily available, once we start doing anything of significance with the generic data representation. A naive implementation would attach the type identification to a descriptor record exposed alongside with the data representation, thereby forcing the client back into the worst conceivable programming style, suffocating any beneficial effect the use of types and contracts might have on understandability and maintainability of the code: Either, you'll just assume everything goes well, or you end up being prepared for anything conceivable, or -- even worse -- a combination of both evils.
There are two known remedies for this dilemma
* a visitor (double dispatch scheme), which has its own maintenance problems
* a //pull// approach, where the consuming client implicitly knows the full context, since it works from within this context
Both approaches incur some complexity on behalf of the involved parties, and a runtime indirection overhead, which might be considered the residual cost for every //reflective data handling.//
This design prefers the //pull// approach, with a special twist: we provide a completely generic, fixed implementation of the pull handling, yet offer the client to install a closure to receive any actual changes. Obviously, such a closure embodies the full typed context, and the validity of this typing can be checked at least dynamically, on //installation of the closure.// This is not as much type safety as is achievable through a visitor, but also less involved and ceremonial -- at least we can ensure the validity of the connection the moment we actually hook it up and remove the risk of running into fundamental mismatch problems in the middle of processing. The latter is the recurring plague haunting most "dynamic" systems.
!!!representation of objects
It should be noted, that the purpose of this whole architecture is to deal with »remote« stuff -- things we somehow need to refer and deal with, but nothing we can influence immediately, right here: every actual manipulation has to be turned into a message and sent //elsewhere.// This is the only context, where some, maybe even partial, generic and introspective object representation makes sense.
__Questions for Design (6/15)__
* do we need to //alter// object contents -- or do we just replace? &larr; provide a ''Mutator''
* to what degree is the distinction between attributes and children even relevant -- beyond the ability to address attributes by-name?
* how do we describe an object from scratch? &larr;''object builder''
* how do we represent the break between attributes and children in this linearised description?
** using a separator element?
** by convention through the element names? &larr; ''This''
** as additional metadata information sent beforehand?
* we need an object-reference element, since we do not want to copy whole subtrees while processing a diff
!!!Mapping a Diff Language to Object structures
"Objects" can be spelled out literally in code. We care to make the respective ctor syntax expressive enough. For nested objects, i.e. values of type {{{diff::Record}}}, a dedicated object builder notation is provided, because this is the point, where the syntax gets convoluted. Yet the interesting questions arise when it comes to spelling out a diff language description against an existing object tree. While a conventional list diff implicitly relies on the structural properties of a list, in our case, the //actual, concrete object// tree serves as structural backdrop and interpretation context of the description in diff language. Effectively this makes the language self contained: it is possible to unfold a new structure from scratch, and use this new structure as implicit context for further manipulations henceforth.
Within this framework, we represent //object-like// entities through a special flavour of the GenNode: Basically, an object is a flat collection of children, yet given in accordance to a distinct ''object protocol''. The relevant ''meta'' information is spelled out first, followed by the ''attributes'' and finally the ''children''. The distinction between these lies in the mode of handling. Meta information is something we need to know before we're able to deal with the actual stuff. Prominent example is the type of the object. Attributes are considered unordered, and will typically be addressed by-name. Children are an ordered collection of recursive instances of the same data structure. (Incidentally, we do not rule out the possibility that also an attribute holds a recursive subtree; only the mode of access is what makes the distinction).
Here the question arises as to what extent the //language// needs to know about these object semantics. While a commitment for precision might lead us towards strict language definition, in fact, languages usable in practice need largely not be defined at all, since they are applied against a context. And since the use of the language itself might guide us from one context to another, the possibility of multiple levels of language arises. We use this observation as guideline and hint to keep our diff language open. Basically, it is just a sequence of verbs, which needs an actual interpreter implementation, which in turn naturally leads itself to attachment to some working context. At this point, we get a //binding// between sequences of language terms and the operational semantics, which in turn defines the limits of legal language constructs. As long as both sides agree upon the same structural conventions, the exchange works without strict codification. We should better strive at defining our object semantics precisely though. Any leeway can be allowed, as long as it conforms with the general layout and as long as it doesn't open a path to confusion later.
Based on these considerations we establish the "two lists" schematics:
* we make our objects look like lists of attributes and children
* we define our protocol rules as
** attributes first
** metadata given by //magic attributes// (just a {{{"type"}}} attribute for now)
** occurrence of the first child switches from attribute zone to child scope
** children are recognisable by the form of their ID
Relying on these rules, we're able to systematically build and derive a sensible binding, while most of the implementation is just a specialisation of list diffing.
!!!handling of actual mutation
This question is closely linked to the semantics of equality. In a simple list diff, this matter doesn't pose any problems; when an element is different, it is a different element, and this change can be encoded as a deletion and insertion of a new element. Not so in tree diff handling. We do not want to delete and re-build whole subtrees, because some tiny bit is altered somewhere down. Thus, a recursive sub-structure can be considered //the same entity,// yet still //mutated.// Our diff handling framework deals with the identity first, followed by an recourse into investigating //inner changes.// This recursive investigation is spelled out as a bracketed construct, which can be processed by recursive invocation. In the end, at the level of the tree leaves, handling those inner mutations boils down to invoking the //mutation closure,// as mentioned above. The knowledge of type context is thus confined to the receiving client, as long as every GenNode implementation offers support to detect an inner mutation and allows to install and invoke such a specifically typed closure to deal with the mutation. The twist to note is the point, //where// this closure is installed: it certainly doesn't make sense to install it on the generating side.
!!!conceptual mismatch
In the attempt to represent changes to a data structure in the form of //abstracted diff(erences),// we're facing a conceptual mismatch on two levels
* even a generic, [[DOM-like structure|ExternalTreeDescription]] has more inherent rigidity than is compatible with the notion of describing differences. //We'll have to reject some diffs//
* any real language-object implementation bears rich shades of semantics, which are impossible to represent symbolically. //We'll have to paraphrase and omit some aspects//
!!!use cases of tree diff application
Within the context of GuiModelUpdate, we discern two distinct situations necessitating an update driven by diff:
* a GenNode representing an object pulls diff information provided by Proc. This information contains mutations of attributes, and //some// of these attributes are relevant for the GUI and thus represented within the GuiModel
* a widget was notified of pending changes in the GuiModel and calls back to pull a diff. While interpreting the attributes mentioned in the diff, it has to determine which widget state corresponds to the mentioned attributes, if applicable. Detected changes need to be interpreted and pushed into the corresponding widget and GTK elements.
the second case is what poses the real challenge in terms of writing well organised code. Since in that case, the receiver side has to translate generic diff verbs into operations on hard wired language level data structures -- structures, we can not control, predict or limit beforhand. We deal with this situation by introducing a specific intermediary, the &rarr; TreeMutator.
!!!apply a diff onto implementation data
Thus we need to carry the idea one step further. The goal is to allow a loose cooperation of components, without the need of a common data structure set in stone. Up to now, with our symbolic diff language and tree representation, we're able to render objects in some kind of DOM notation, and we're able to apply changes onto them. This is fine for maintaining an intermediary GuiModel, but to really achieve the desired loose coupling of cooperating components, we need //the ability to apply changes to implementation data structures.// To do so, we need a ''binding'' from the diff language to »''implemetation data''« -- without imposing restrictions onto the latter: We do not intend to invent yet another introspection framework, working on yet another meta object type. Thus, at some point, finally we need the help of the implementation context to //make sense of the diff verbs.//
As said, this turns out to be a tricky challenge: implementing the application of diff messages is inevitably an involved and technical task; we should avoid imposing these technicalities onto client code. But, at minimum, we need to know some basic traits regarding the target data structures, in order to handle receiving change messages properly. We solve this problem through implicit conventions plus an indirection layer
* we impose some very vague requirements onto the structure of target data: it is required to comply to our conceptual notion of an "object"; target data //needs to be congruent to [[ETD|ExternalTreeDescription]].//
* we introduce an adapter interface, the TreeMutator. Diff application can then be implemented in terms of this new abstraction
* we offer pre configured building blocks and a DSL, for client code to assemble a suitable {{{TreeMutator}}} implementation, privately tied into the opaque implementation data structure
* client code has to fill in some necessary information in the form of ''closures'' (lambdas)
** how to construct a new data element
** how to match a //diff verb// against a given data element
** how to determine applicability of a verb, allowing for multi layered binding
** how to assign a new value, given the corresponding diff verb
** how to open a nested scope for recursive mutation

for the purpose of handling updates in the GUI timeline display efficiently, we need to determine and represent //structural differences//
This leads to what could be considered the very opposite of data-centric programming. Instead of embody »the truth« into a central data model with predefined layout, we base our achitecture on a set of actors and their collaboration. In the mentioned example this would be the high-level view in the Session, the Builder, the UI-Bus and the presentation elements within the timeline view. Underlying to each such collaboration is a shared conception of data. There is no need to //actually represent that data// -- it can be conceived to exist in a more descriptive, declarative [[external tree description (ETD)|ExternalTreeDescription]]. In fact, what we //do represent// is a ''diff'' against such an external rendering.
We build a slightly abstracted representation of tree changes and use this to propagate //change notifications// to the actual widgets. To keep the whole process space efficient, a demand-driven, stateless implementation approach is chosen. This reduces the problem into several layered stages.
* our model is a heterogeneous tree &rArr; use demand-driven recursion
* the nodes are heterogeneous collections &rArr; use filtering by type tag
* find changes in ordered collections of children &rArr; symbolic list diffing algorithm
* problems with identity and state &rArr; encapsulate state and have a airtight object identity scheme
Doubtless we're dealing with a highly specific application here.
&rarr; see [[discussion of diffing solutions|TreeDiffFundamentals]]
&rarr; see [[diff handling implementation technique|TreeDiffImplementation]]
!list diffing algorithm
| !source data|!|!desired result |
|(a~~1~~, a~~2~~, a~~3~~, a~~4~~, a~~5~~) |&hArr;| {{{delete}}}(a~~1~~, a~~2~~)<br/>{{{permutate}}}(a~~3~~, a~~5~~, a~~4~~)<br/>{{{insert}}}(//before a~~3~~//, b~~1~~)<br/>{{{insert}}}(//before a~~4~~//, b~~2~~, b~~3~~)<br/>{{{append}}}(b~~4~~)|
|(b~~1~~, a~~3~~, a~~5~~, b~~2~~, b~~3~~, a~~4~~, b~~4~~)|~|~|
to cover reordering, we need to determine the deletes and (possible) updates in one set operation.
After reordering the remaining updates to the target order, the inserts are determined in a final merging pass.
Here all the fuzz about our {{{LUID}}} and identity management in the PlacementIndex definitively pays off: A standard multiset implementation should do.
!typed ordered views
The consumer -- in our case the GUI widgets -- impose a preconfigured order of things: elements not expected in a given part of the session will not be rendered and exposed. Thus the expectations at the consumer side constitute a typed context. So all we need to do is to intersperse a filter and then let the diffing algorithm work on these views filtered by type. All of this sounds horribly expensive, but it isn't -- functional programming to the rescue! We are dealing with lightweight symbolic value representations; everything can be implemented as a filtering and transforming pipeline. Thus we do not need any memory management, rather we (ab)use the storage of the client pulling the representation.
!structural differences
The tricky part with changes in a tree like structure is that they might involve rearrangements of whole sub-trees. So the question we need to pose is: to what extend do we need, and want to capture and represent those non local changes? In this respect, our situation here is significantly different than what is relevant for version management systems; we are not interested in //constructing a history of changes.// A widget moved into a completely different part or the model likely needs to be rebuilt from scratch anyway, so it doesn't hurt if we represent this change as deletion and insert of a new sub-tree. But it would be beneficial if we're able to move a sequence of clips in a fork ("track"), or even a whole fork at the current level. As a corner case, we might even consider representing a &raquo;fold-down/up&laquo; operation, where a sequence of elements is wrapped into a new sub-node, or extracted up from there -- but this is likely the most far-reaching structural change still worth to be represented first class.
!diff representation
Thus, for our specific usage scenario, the foremost relevant question is //how to represent the differences,// since our aim is to propagate complex structural changes through a narrow data mutation API as communication channel. The desired representation -- call it ''linearised diff representation'' -- can be constructed systematically from the predicate like notation used above to show the list differences. The principle is to break the representation down into atomic terms, and then to //push back// any term repreatedly, until we come accross a term which can be //consumed right-away// at the current top of our "old state" list. This way we consume the incoming change messages and our existing data simultaneously, while dropping off the mutated structure in a single pass. Applying this technique, the above example becomes
|!Message|!| !Result List|!remaining old |
| |!| ()|(a~~1~~, a~~2~~, a~~3~~, a~~4~~, a~~5~~) |
|{{{del}}}(a~~1~~) |!| ()|(a~~2~~, a~~3~~, a~~4~~, a~~5~~) |
|{{{del}}}(a~~2~~) |!| ()|(a~~3~~, a~~4~~, a~~5~~) |
|{{{ins}}}(b~~1~~) |!| (b~~1~~)|(a~~3~~, a~~4~~, a~~5~~) |
|{{{pick}}}(a~~3~~) |!| (b~~1~~, a~~3~~)|(a~~4~~, a~~5~~) |
|{{{find}}}(a~~5~~) |!| (b~~1~~, a~~3~~)|(a~~5~~, a~~4~~) |
|{{{pick}}}(a~~5~~) |!| (b~~1~~, a~~3~~, a~~5~~)|(a~~4~~) |
|{{{ins}}}(b~~2~~) |!| (b~~1~~, a~~3~~, a~~5~~, b~~2~~)|(a~~4~~) |
|{{{ins}}}(b~~3~~) |!| (b~~1~~, a~~3~~, a~~5~~, b~~2~~, b~~3~~)|(a~~4~~) |
|{{{pick}}}(a~~4~~) |!| (b~~1~~, a~~3~~, a~~5~~, b~~2~~, b~~3~~, a~~4~~)|() |
|{{{ins}}}(b~~4~~) |!| (b~~1~~, a~~3~~, a~~5~~, b~~2~~, b~~3~~, a~~4~~, b~~4~~)|() |
__Implementation note__:The representation chosen here uses terms of constant size for the individual diff steps; in most cases, the argument is redundant and can be used for verification when applying the diff -- with the exception of the {{{ins}}} term, where it actually encodes additional information. Especially the {{{find}}}-representation is a compromise, since we encode as "search for the term a~~5~~ and insert it at curent position". The more obvious rendering -- "push term a~~4~~ back by +1 steps" -- requires an additional integer argument not neccesary for any of the other diff verbs, defeating a fixed size value implementation.
!!!extension to tree changes
Diff description and diff handling can be applied to tree-like data structures as well. Some usages of textual comparison (e.g. diffing of programming language texts) are effectively working on tree structures -- yet they do not build on the structure of the diffed data explicitly. But if we represent the data structures symbolically, the change form text diffing to data structure diffing is marginal. The only relevant change is to handle embedded recursive diff descriptions of the child nodes. As it stands, each node or "object" can be represented as a list of properties plus the attachment of child nodes. This list can be treated with the methods developed for a stream of text tokens.
Basically the transition from text diffing to changes on data structures is achieved by exchanging the //type of the tokens.// Instead of words, or lines of text, we now use //data elements.// To do so, we introduce a symbolic ExternalTreeDescription of tree-like core data structures. The elementary token element used in this tree diff, the GenNode, embodies either simple plain data elements (numbers, strings, booleans, id-hashes, time values) -- or it describes a //recursive data element,// given as {{{Record<GenNode>}}}. Such a recursive data element describes object-like entities as a sequence of metadata, named attributes and ordered child-nodes -- it is handled in two phases: the first step is to treat the presence and ordering of child data elements, insertions and deletes. The second phase opens for each structurally changed child data element a recursive bracketing construct, as indicated by explicit (and slightly redundant) //bracketing tokens://
*{{{mut}}}(node-ID) : recurse into the designated node, which must be present already as result of the preceding changes. The following diff tokens describe //mutations// of the child
*{{{emu}}}(node-ID) : close the current node context and return one step up; the node-ID is given for verification, but can be used to restore the working position at parent level
*{{{set}}}(node) : shortcut notation for simple value elements; assign the new payload value to the designated element, retaining identity
In addition, in a future extension, we might consider to introduce up/down folding primitives
*{{{fold}}}(node-ID) : pick the following elements and fold them down into a new child with given node-ID. The down folding continues until the next {{{emu}}} token
*{{{lift}}}(node-ID) : remove the next child node, which must be node-ID, and insert its children at current position
Since the bracketing construct for mutation of child structures bears the ID of the parent, a certain degree of leeway is introduced. In theory, we could always open such a bracketing construct right after the {{{pick}}} token accepting the parent -- yet, while minimal, such a strictly depth-first representation would be hard to read -- so we allow to group the recursive treatement of children //post-fix,// after the messages for the current node. In a similar vein, we introduce another token to describe a //short-cut://
*{{{after}}}(node-ID) : fast-forward through the sequence of elements at current level until the position after the designated element.
To complement this language construct, we define some special, magical (meta) element-~IDs
*{{{_CHILD_}}} : marks an //unnamed// ID. The implementation exploits this specific marker to distinguish between nodes which are (named) attributes of an object, and real children.
*{{{_ATTRIBS_}}} : can be used to jump {{{after(_ATTRIBS_)}}} when mutating the contents of an object. So the following diff verbs will immediately start working on the children
*{{{_END_}}} : likewise can be used to jump {{{after(_END_)}}} to start appending new elements without caring for the existing current content.
All these additional language constructs aren't strictly necessary, but widen the usability of the language, also to cover the description of incomplete or fuzzy diffs.
!!!representation of objects
While we are talking about //structured data,// in fact what are about to handle are objects, understood in the standard flavour of object orientation, where an object is the instance of a type and offers a contract. Incidentally, this is not the original, "pure" meaning of object orientation, but the one that became prolific in brining our daily practice closer to the inherent structuring of modern human organisation. And in dealing with this kind of object, we sometimes get into conflict with the essentially open and permissive nature of structured data. So we need to establish a mapping rule, which translates into additional conventions about how to spell out matters in the diff language.
We choose to leave this largely on the level of stylistic rules, thus stressing the language nature of the diff. Even when this bears the danger to produce an exception very late, when it comes to applying the diff to a target data structure. The intention behind this whole diff approach is to transform tight coupling with strict rules into a loose collaboration based on a common understanding. So generally we'll assume the diff is just right, and if not, we'll get what we deserve.
The ''protocol to describe an object'' is as follows
* the ID is the objects identity and is once given, never changed
* we spell out any metadata (esp. a type information) first, followed by all attributes, and then followed by contents of the object's scope (children)
* attributes are to be given in a way not in contradiction to the more stringent semantics of an object field or property
** never attempt to re-order or delete such attributes, since their presence is fixed in the class definition
** when a field is mandatory //by its nature,// it shall be required in construction, and the corresponding data is to be given with the {{{ins}}} verb causing the constructor call
** on the other hand, the data for an optional field, when present, shall be spelled out by {{{ins}}} verb after construction, with the first //population diff.//
** we do not support attribute map semantics (or extended "object properties" of any kind). If necessary, treat them as nested entity with map semantics
!!!deriving conventional representations
On receiving a stream of tokens of this "diff language", it is possible to generate the well known and more conventional diff representations,
i.e. a ''unified diff'' or the ''predicate notation'' used above to describe the list diffing algorithm, just by accumulating changes.

The TreeMutator is an intermediary to translate a generic structure pattern into heterogeneous local invocation sequences.
within the [[diff framework|TreeDiffModel]], this is a crucial joint, since here the abstract, generic, ~DOM-like ExternalTreeDescription meeds opaque, local and undisclosed data structures.
&rarr; see TreeMutatorEvolution for an extensive discussion about the principles and architecture, which gradually leads to the final design of the ~TreeMutator, as realised now in code.
!Motivation
on a very large scale within the Lumiera application, disjoint subsystems are collaborating based on the notion of a //shared conceptual structure.//
This means, we do not share a data model -- rather we exchange ''diff messages'' with respect to this assumed, common, conceptual structure. In practice, this kind of very loose coupling is realised with the help of a ~DOM-like notation based on a small selection of primitive base types, plus an object-like //record// made from such base elements. Based on these elements represented as [[generic nodes|GenNode]], we can build up a symbolic representation of shared structures, which we term the ExternalTreeDescription. In fact, this is not meant and not used as actual implementation data structure -- rather it is used to derive and validate the diff messages handled within our [[diff framework|TreeDiffModel]].
//But --// if this symbolic representation is not meant for implementation -- how are we then able to relate it in any useful way to actual implementation data structures? The answer is two fold. For one, and most importantly, we do not collaborate on a shared data model, which means that the implementation data structures are always an implementation detail confined within some subsystem or even part of an subsystem. And this implementation layer can then -- secondly -- expose an adapter or intermediary, which offers a "generic side" for the diff framework to attach to. This adapter is the TreeMutator.
!Definition
;interface
:the tree mutator is an interface providing a set of virtual functions, the ''mutation primitives''
;adapter
:the implementation of this interface acts as adapter, to translate ''verbs of the diff language'' into invocations on opaque, private data structures
;binding
:this adapter is //created// by binding those opaque, private data structures into some generic implementation patterns for the TreeMutator
;builder interface
:the TreeMutator offers a builder interface to build up those bindings, with the help of closures provided by the client, which is the private, opaque data structure
So, from the usage side, some kind of implementation data structure will build a TreeMutator and tie itself into its binding Then, by exposing this {{{TreeMutator}}} implementation, this client data structure gets the ability to receive diff messages, which means this local and undisclosed data structure can be attached to and collaborate with some other subsystem; it can be altered, extended and reshaped remotely, without the need to share implementation details.
Incidentally, the ''diff language'' itself is meant to be readable and expressive. On the //sending side,// typically diff messages will be constructed directly in code. To this end, the GenNode offers a constructor and mutator syntax specifically crafted to allow writing clear and compact diff messages. Since the tree diff language embeds a simple list diff language as a subset, it is possible to generate structural changes by observing changes on data collections. Such //detected diffs// may be embedded within explicitly written diff message sequences.
The canonical example are changes to the editing session. After maybe running some resoluti