mb dynamics

i used some sick time to write about mb dynamics. there was some interest about it in herwig's open letter post some weeks ago (hi catman). it's a long text with lots of images, so i've uploaded it here. much of it is descriptive, but i tried to explain some design choices and discarded options. you can discuss any of them -there's a lot of guesswork. i don't know how useful this could be to anyone, but at least it made me go revise all the little things that i always put off. salut!

 

updated to reflect v1.8 -bye bleps, hello 2nd order attacks. it's a facepalm update, that i'd have done way before if i thought in lines instead of circles
Catman Dude (just in case you didn't see it before)

 

Hi Laureano!
Great to hear both about the update and the thread with explanation. I have downloaded the new version and look forward to using soon, and studying the Core code while reading about how it works.
I have been inside the Process macro before, and it is HUGE, a rain forest of wires.
So having your explanation as a guide is fantastic. I guess it makes sense with such a large project that you could have started this a while back. Otherwise, if you hadn't been thinking mb_dynamics over for at least some months, if it had sprung from your head fully grown as it were, I would have felt that I might as well give up -- I'd never make a good Core programmer.
(I just found this thread, because I've been working on a new block of my own. It's nothing much, but at least I learned something about how important Logic is for anything to work right in Core.)
Many thanks for the new mb_dyn!
Will follow up soon.
EDIT: Holy crow, your documentation is awesome! Beautiful too. Many bows in your direction.
Will read very carefully.
David

 

yep, it's been a slow work of some years. i've revised it much more than any other thing i've done, because i use it a lot for mixing, so every time i'm between projects i fiddle with it again. what's weird is that it actually gets simpler every time, even though i'm adding functions -as i "get" things, i see the needless complications and untwist them. i don't consider myself a good programmer -it's still hard for me to read other people's code, both when it's too messy or when it shows a strong programming background. programmers tend to name and organize things differently, following patterns that i've forgotten or never learned.

btw, feel free to ask any questions you want.

 

Code gets simpler the more you know -- you realize there's a straight line between 2 points you connected originally with multiple arcs and jagged lines. But it also gets bigger, as the scope of what you can imagine your project doing grows with use and ambition.
So you may not consider yourself a good programmer (though humility is always a positive character trait), but the evidence is piling up on the other side of the argument.
If you (not you in particular, rather anyone) don't know what you're doing, you not only can't create a project this large or comprehensive (successfully), but you unwittingly build into it so many bugs and wrong methods that no one else would think of using it, and even the programmer comes around to learning he 'bit off more than he could chew'.

(I had hoped earlier today to start educating myself on your Splitting Tree, when this morning I got an email from a violinist who will be recording, with 2 other players, a string trio of mine in less than 2 weeks, asking questions about bugs I planted in the score (he didn't call them that), so I spent most of the day fixing them and uploading the repaired score to dropbox. So tomorrow I'll be able to ask some questions...)

 

those sound like great news! i've been fixing a score too, but just for completeness' sake -no one's playing it any time soon, but its bugs bugged me

i think many people can and actually do things in a different, more intuitive, maybe more relaxed way, and the results may be less efficient but not less effective. also, being on the other side of neurosis (the ocd side), i tend to think those people come more easily to weirder, further away places. then i realize i'm just seeing the grass greener on the other side, and we all just come to different places, differently weird. still, there are lots of things in the u/l that i can just stare at in awe, with no idea of what they're doing, or even what to do with them. i consider it a fault on my side, one that i'm not worried to fix.

thanks for the kind words!

 

Please don't take this the wrong way - it's not a criticism of your compressor, but I think there is an important issue here that often gets ignored.

Programming is a creative endeavour. The better a language is, the more easy it is to express what you are doing in a personal way - everyone will do it different unless they are working withing company style rules. In my experience, a good programmer with a 'strong programming background' will write the easiest to follow code - because they understand how important that is. So when they go back to code they haven't looked at for five years, they can get back up to speed quickly. Ideal code is self documenting - names of functions (or in Reaktor, macros) are enough to provide an understanding of the intention of the programmer, if this doesn't work, then adding comments and notes is another (less useful) option. The worst code is undocumented and doesn't provide meaningful names for functions and variables.

I had a look at your code, and it is extremely difficult to follow. The main reason for this is you don't name any of the macros. This is a really bad idea!

Here is a pic from your structure:


The only things with names here have cryptic abbreviated identifiers, and the macros are unnamed - pretty much throughout the whole structure.

Here's an example from some old code of mine. This was written many years ago, and I would probably write the application slightly differently now, but when I look at this I can get back up to speed with it very quickly even though it is extremely complex - because almost everything has a meaningful name:

This isn't always easy to achieve. Often you need to find different ways to break up processes so that its' easier to assign meaningful names to individual macros. It's also extra work - e.g. in the above example, there are a bunch of state variables that I have put into individual macros so that I can more easily document them. Short term, this takes extra time, but long term, it saves time because it is makes the project easier to manage as it grows. It's also difficult with Reaktor compared to more traditional programming languages, but there are lots of ways to do it, so there is no excuse.

Here's another example that uses more recent Reaktor features:

In the top level macro (unnamed, because it's just the main/only container at that level), I use distribution modules to assign meaningful names to 'magic number' signifiers that can then be use throughout the structure. So instead of 'if tonality == 0' which is meaningless, I have 'if tonality == major' which is meaningful. Throughout this particular project, it's is reasonably clear what is going on - so the macro that extracts the root and fifth from the currently active triad is called 'get root & fifth', and likewise, the macro that uses that root and fifth to generate a third, is called 'get third'. The code inside these is somewhat obscure, but I know what it is doing, so it's much easier at a later date to work out how it's doing it. If the macro was named '...' and it's internals used abbreviated quickbus names like ft, td and rt instead of fifthID, thirdID and rootID, I would be lost - there would be no way on earth to work out what is going on, withough reverse engineering the whole application.

Of course, there might be times when you want to obfuscate code on purpose, but that's a whole different discussion

 

yep, i've thought a lot about that when writing, this and other things. it's weird because when i write in code, i tend to use long names, but in reaktor i got used to this kind of... abstract naming that yes, doesn't make reading any easier. there are, i think, a pair of reasons for that. one is that when i'm working on something, it's much, and i really mean much more difficult for me to follow it if things are big, misaligned, too criss-crossed and even when there's a lot of text -this might be not the same two or three years after i've not touched the code, but it does happen when i'm working on it. i solve things much faster and better when i can look at the shape of it and see the flow. the other thing is that i really don't know how to name some macros, though i went overboard with the "..." thing. i do try to name them, but there are many times when the simplest and most functional solution, the one that lets me see the issues and simplify further, implies a grouping where the parts are not essentially anything else than containers. in the example you show i could name that macro "gain calculation, output detection and average levels" but it would be way more obscure for me to navigate the code with such phrases everywhere, and there's really not any other particular abstraction for that macro -it's not "band processing", because band processing is split in very separate parts not for "logical" reasons but because it made the code way simpler and reduced duplications (very relevant here because there's some intense copypasting). actually when i coded an earlier version of this in c++ i organized things differently, in a way that allowed more meaningful names, but when i try to do this in reaktor things get much harder to manage. i don't say i may not solve it better of course, and you are right anyway -it's just the way that's made the work easier for me, not for coding itself but for actually thinking the thing.

 

this is a particularly obscure "passage". there's that "g" up there and "2t" below, because some filters are sent filter gains (an awful abstraction -it may come from a frequency, a time constant, whatever) and others are sent twice the time constant -i've thought a lot about all the different meanings that g or even the whole word gain has in different places. there's c for clock(s) and wt for weighting and that m for mode, but i highlighted the "x" because it shows the kind of naming nightmare i get into. those x control the mix of static (above) and auto (below) out gain, and come from a point at which there was a simple xfade. they don't mean the same now. both out gains are disabled for get mode, and the lower x includes the auto mode itself (m: -1..1) to solve both in one multiply. so the upper one could be "static out gain scale for normal mode, zero otherwise" and the lower "auto out gain scale and mode for normal mode, zero otherwise". i know it may be better documentation, but i can't work with that

 

That's kind of odd, because it's possible to make different shapes out of the same piece of Reaktor code - the shape is pretty much irrelevant!, there is no meaning stored in the shape!

There is meaning in the connections, but when they are wires, the meaning is implicit, it just tells us that there is some meaning here - to ascertain the meaning, you have to look along the wire to see the port names that it is connected to. Fortunately in core, we can switch from a wire that tells us that there is some undefined but meaningful connection, to using quick-busses that let us explicitly state something about that meaning. That's a powerful tool! It also means we don't have to mentally untangle the wires to understand what's going on - just use a logical and consistent naming system, and everything should get easier.
Wires are useful visually to describe the processing order, but I find it's best to be sparing with wired connections, so the main structure of the process is defined with wires where appropriate, and it doesn't get lost in spaghetti...

Translating concepts from the problem space to the solution space is difficult. The general rule (often broken by me) is that if a function can't be named by a single meaningful name that completely describes its job, then it shouldn't be a function - either it should be multiple separate functions, or it is part of some other function. Factoring code down into a set of atomic functions that all fit into this rule is difficult, but when you can do it, it makes life so much easier. It also tells you that you actually understand the problem and it's a good sign that your solution might be good...
One issue I have in Reaktor is where to draw the line. You always get to a level of granularity where encapsulating into more macros will make things harder to work with , not easier, but usually I find that if it's correct, the low level stuff will always fit onscreen and be processable by my brain... otherwise it needs more parcelling up. Sometimes I go super low level just to make things easier to read...

If everything above that level was named in a meaningful way, it would be clear to someone reading the code that they had arrived at the level where it doesn't make sense to do that any more - which in itself helps to understand what's going on.

 

I shouldn't be jumping in here, as I'm still riding around with training wheels, so take my comments with liberal salting. But I think these posts do a good job of demonstrating what Laureano was getting at above, which I encapsulate with a quote from his prior post:
"i think many people can and actually do things in a different, more intuitive, maybe more relaxed way, and the results may be less efficient but not less effective."
I will say here that I agree with this sentiment. In the world of music (my comfort zone) one should never forget this rule of thumb. It doesn't much matter how you get there, if where you get to expresses some intense human emotive-cognitive imperative.

Programming is somewhat different, though I agree there is a significant creative component to writing good code.

I do think the 3 stretches of Core code demonstrate significantly different 'styles' of programming.
And, Colin, I totally agree with your principle of self-documenting code, plan-ahead naming and the like.
Yes, if I were writing Laureano's code, I would name the macros. But his code is relatively simple in your chosen example, it's very easy on the eye: all the modules are related. The structures themselves are small and coherent in themselves.
And he has given short-abbreviations whose meanings I will happily guess: avg = average; thr = threshold; wt = weight; g' = gain. Of course I could be wrong, but if I'm not, then you have a novice reading his code and getting the general drift!
If English is not his first language, as is quite possible, yet it appears that all these abbreviations are good ones for English words that most programmers would understand.
I contrast this relative simplicity to your first example's complexity. It's beautifully constructed, and impeccably self-documenting. Bravo! It is also very busy with modules and quick-buses and while eminently readable if one focuses on small subsets of the structure, it's still a little hard to take in from the 'high angle', with all its buzzing activity.
I take it for granted that the complexity of your project, which looks like a parser of keyboard input (in part) for note input involving portamento, etc., is necessary and couldn't easily be packaged differently, without loss of something essential. Under those circumstances, it's an impressive accomplishment, and I'd love to see the project!
Your second example is perhaps showing signs of tiring of such strenuous self-documenting, though not necessarily to bad effect.
Inputs: 'P' 'L' 'R' 'S' 'N' 'X' 'H' ???
I'm sure you know at a glance to what these refer. 'Pitch'... hmmm ... I won't go further because single-letters are notoriously difficult to infer meaning from.
Nonetheless, I do understand the general drift of what is happening, because of all the other self-documenting aspects.
I might say that your own stylistic evolution is well demonstrated in these 2 pretty different examples.
I find something beneficial to my education in both styles of programming. Thanks for putting this into such coherent form!

 

Was going to ask why you didn't include the breadcrumb trail in your pic so I could find it, but there are four different '...' macros at that level anyway, so it wouldn't have helped much

 

Exactly - you could be wrong - what about 'au' is that 'audio' - makes sense? nope, it's 'auto'... hmmm.

Well I'm not a novice, and I've written a few compressors, and I couldn't get the general drift. For me it's too difficult to connect the high level to the mid level to the low level, because too little of the meaning has been defined. I gave up!

Yes, it's very complex, because it's doing a complex task. I left it at a level that was easiest for me to follow, any more packaging into more layers of macros would have meant too many layers.
FWIW, its from digit8. It's part of the parser that tokenises input (reads individual characters and turns them into single internal tokens (e.g. >> is two characters, but has a single meaning 'right shift'), and converts it from infix notation into reverse polish notation. Reverse polish is much more efficient to process so can be done in real time in the audio stream using a stack.

'P' 'L' 'R' 'S' 'N' 'X' 'H' are specific to the problem domain in this case - it's a 'Tonnetz' processor, and the Papers I've read on this, and the hardware module that inpired it use (some of ) these single letter names for the various harmonic transitions. P stands for 'Parallel', R for 'relative', and L for 'leading tone', but in general, the capitalized abbreviations are used - see tonnetz sequent.

This works well here anyway because anywhere there is a single capital letter, I know it's definitely related to that particular transition form the problem space - because all the internal labels are verbose

 

Why not call the module 'gain control' or 'get gain', or 'calc gain'...
And internally, have two(or more) modules, one for each mode, and use routing to select them. You might find this to be more efficient, and you definitely shouldn't assume it wont be.
Looking a bit deeper at all the places where mode issued, it looks like mode came after the main structure was designed and was shoehorned in there?

Even if that's not true, it looks to me like there are some places where using mode to select between separate higher level modules might be a better solution than trying to use it at a very low level and trying to keep that efficient by tricky use of bitwise logic and multiplies to cancel processing branches... Not that that would be better, just that it might make more sense structurally, which would automatically mean easier naming for macros - which is a good sign...

It's really difficult to second guess what will be most efficient in Reaktor, and also what will help or hinder the compiler. Usually best to stick with well structured code, and only change it if there is a problem rather than try to write 'efficient' code by using various tricks. Premature optimization being the root of all evil... and all that jazz

 

well, that's in the eye of the beholder no but, seriously, i do think a lot by shape when i code in reaktor. if i were to solve everything with named ports, i could very well go back to supercollider, of which i got tired quickly. it's not that i'm so much of a visual person, but it's the visual thing what made me comfortable with reaktor compared with other languages. in fact i usually try the opposite: to wire as much as possible into something i can get without reading. if an arrangement needs too much quickbussing, i try again.

yap to all that. but... i tend to group things differently in reaktor than in say c++. less conceptually and more... "topologically". like, things belonging to a place in the least convoluted way. it's the shape thing

right! g' are "live" gains, as in dynamic+static band gains. the ' separates them from static gains g in contexts where both are involved. i know, too short! i know.

well, there's nothing in the whole ensemble explicitly called "audio" anything, and auto out gain is a main feature.

you seem mad though. sorry for that. just in case, i don't claim anything. i coded this for myself, to solve some specific concrete need i had in my job. i don't code for a living. i just upload things i think may be useful to someone.


this is the kind of thing i mean when i say patterns. there's obviously a space of discourse in which {} [] ! ? $ # mean very specific things that some people understand right away. i get some of them now, but it's still hard for me to follow this kind of stuff. and then there's also this:

i don't know how many different versions of a latch i've seen. at least five i'm sure. some seem to come from old versions of the library or something. some are called "Lc", some use weird icons...

 

but i've not second-guessed. i've spent literally hours testing different options not to prove anything, but because i'm mixing on an APU and i can use seven instances of this in the same project. those little "tricks" actually lowered cpu usage by more than a half over time, even when i kept adding functions. i can't make an elegant code that won't run. i do mix for a living. anyway, auto mode is not a selector, it's an inverter -the mode changes the sign of the out gain envelope.

 

Heh, I didn't mean it to come over in that way - just passionate about programming...
I was interested to see what differences there were between your compressor and various others I've looked at, but found the code to be difficult to follow - and left it there. But then you wrote about how you find it difficult to follow other peoples code.... so I figured that was a good opportunity for a discussion/debate on the topic of code style and whatnot...

 

we're both passionate, it's alright

i think there's quite some overlap with salamander anagram's pressure and multipressure -we read the same paper. gain computers may be almost the same, i'm just adding a range and a second knee. my envelope detection is quite different though. it's done before and after gain computers, the filters are zdf and one of them is 2nd order. then of course there's the whole relative-level thing, which is the whole reason i made this (and mainly why envelope detection needs to be different). also i think he didn't use crest factor to control the speed of envelopes -that was in other paper by the same people, and i adapted it a bit for what i needed.

 

Yes, fully agree on the partials framework symbol stuff - terrible design decision IMO. I get that some folk speak English and others speak German or French etc..., but why force everyone to learn something new that is unintuitive to the point that it seems intentionally cryptic.

The latch thing is also a bit annoying, but as long as it's consistent within that project, maybe not so bad.

 

i'll add something about optimization, premature or not. i would agree completely in other contexts, but i've not been able to come up with something moderately big and usable without going a bit cryptic, in audio. my polite, object-oriented and templated c++ test used as much cpu as the reaktor version (which was less efficient back then) without even a gui. when i say usable i mean usable by me -i may be able to buy a 9th gen i7, but i can't sustain my little business with that standard of investment. an APU has to make it and it does everyday. now, in this particular ensemble, routing was a hog. a tremendous hog. there's a lot of audio rate routing already, after reducing it to the minimum without cutting features i needed. every time i add some routing at this point, especially if it connects early to the main line, either the compiler or the runtime explodes. it doesn't matter if it happens on gui parameters completely asynchronous (and latched) to the audio lines. the compiler is not making that analysis -if i take that out to another core cell, it goes back to normal. so i did avoid routing a lot, for a reason. but i also used it a lot -each inactive crossover brings down cpu load quite a bit.

(edit) that's why some time ago i wondered if there's a compiler specification somewhere buried in ni's computers. i mean of course it would help to just be a better programmer, i dig into digit8 or emscher and i know i'm a bit precarious, but when something that's not actually running raises cpu load by 10% relative, someone is being weird and it's not me