Rtosc Development

One software project that I’ve been working on for a while at this point is librtosc, a realtime safe implementation of the open sound control messaging protocol. If this was simply an implementation of a library to handle the serialized format this would be a done and sealed project a while ago, but the project also includes one aspect which is quite difficult to nail down properly, dispatching. Once a message is received it generally should trigger something as these messages canonically represent events.

This routing problem can be a problem in terms of how much code is dedicated to it, how hard it is to write said code, and what sort of runtime issues may arise in its execution. While this library may very well have general purpose applications, the primary target is ZynAddSubFX, a sizeable software synth, which unfortunately has some major architectural flaws currently. Given this application some architectural optimization needs to be considered which would normally be premature optimization. The OSC messages are normally dispatched according to a typed set of paths similar to what you would see in any url. If a naïve representation was taken for these paths, then there would be somewhere on the order of 2,000,000 paths stored in memory (16 parts * 16 kit instances * 8 voices * 2 oscillators * 256 phase/amplitude controls ⇒ 1048576 paths) and each path with reasonable names could look like "/part1/kit1/adsynth/voice1/oscilator1/phase1", resulting in over 11 MB of static data just for the paths. Considering the current binary size is \~6.5 MB, that is quite a bit of extra weight. As per the execution constraints it would not be surprising to see a system run with perhaps two milliseconds of latency and you likely don’t want to add more than 5% extra time with messages, so you only have 0.1 ms to deal with whatever incoming messages there are. While these remarks are not at all impassible, they do serve as reminders that some caution is needed in building up this system.

Now that the problem domain is specified, the problem can be restated to be: How do you define a massive number of simple but varied callbacks for a tree of objects in a manner that reduces the runtime resources? This sounds like a lovely time to make use of some sort of domain specific language, though things are somewhat limited if this information is included within the C++ source directly. The current approach defines a hierarchy of ports, which specify wildcards to handle duplications, and argument constraints to make things roughly typed; Each port has some form of a callback, some metadata, and possibly an associated subtree.

Port
   name      :String
   types     :Set{Types[]}
   properties:Map{String,String}
   callback  :Function(Arguments[], SupplimentalData)
   subtree?  :Ports[]

Ideally such a thing would translate into simple code such as

#Define the scope of all ports
@class Oscillator

#Define a linear parameter that controls the volume member of the oscillator
#that ranges between the floating point values 1.0 and 10.0 in dB with a
#linear mapper and the doc string of "Base Volume"
@linfpar volume 1..10dB Base Volume

#A midi parameter that controls Pphase, calls prepare() when done and has the
#doc string "phase of the nth harmonic"
@midipar#128 Pphase prepare() phase of the nth harmonic

Oh, did I forget to mention that MIDI mappings are tagging along with the ports metadata. Adding system wide metadata seems like a sane enough time to add enough metadata to reflect on the available parameters. While the above example looks fine enough, things get barrier and much uglier when real legacy code needs to be dealt with. Whenever some chunk of data does not quite match up that is when the abstraction leaks quite heavily and there needs to be a fallback for this case. Unfortunately to emulate the above behavior you are generally stuck with preprocessor macros, which do work, but they are simply unwieldy. Given that we are dealing with data that only needs to be initialized once per run of the application, one of a few options is available:

The current approach taken is the first one. This certainly works well for the current application, but there may be problems with pure static definitions for any system that does not know some information until runtime (like a plugin).

Now with that backdrop laid out, the issue with implementing such a system elegantly is that there are enough edge cases that any abstraction composed of simple macros will be broken at least a few times. That implies that there is only so much of an advantage in trying to create a clean interface. This means that the user will be exposed to some of the annoying bit twiddling that is involved in generating the dispatch tables that other libs hide away. Part of me thinks that the ``good enough'' system will work out, but there is always the question of how good can a tool be?

This brings me to one of the minuscule details that started this mind dump. Doubly null terminated strings. If you are dealing with parsing anything you generally end up rewriting all sorts of odds and ends of ad-hoc utilities to deal with different terminators and collections of buffers start piling up left and right (don’t forget the escape induced madness). Long macros to translate stuff into this format, like RTOSC_PARAMETER() just seem too verbose, but then again the terse side of things is not much better. Then there is the question of is the non-standard doubly null string idea better than a proper data structure? On the one hand it, like the previous ':' delimited string, can easily be transmitted via OSC (via a blob), it may take up more memory, be slower to iterate through, and just be odd. The interesting thing about these questions to me is that I simply do not have a good answer at the moment. Anyhow, back to trying to code this thing some more to see where the bad ideas lie. Perhaps the next post will be more coherent :p