Bogging core compiler challenge

Here's the challenge:

Find a way to bog down the core compiler that is either simpler, or significantly different from the attached example!

===============================================================
------------------------some background for anyone who is bored or interested-----------------------------

After a couple of recent threads about the core compiler getting bogged down and causing spinning wheel of death problems, I've been trying to isolate the causes. I decided to try and make the simplest module I could that would cause the compiler to bog down. I used my hypothesis that a cause of this problem is the number (and dependencies) of event sources the compiler has to trace and came up with this solution.

I timed the compile time with different numbers of sections attached in my macro chain.
26 = 12 secs
27 = 27 secs
28 = 42 secs
29 = 92 secs
30 = 175 secs
These results suggest that the compiler doesn't do much in the way clever data management in order to reduce the issue. Adding an extra section roughly doubles the compile time.... exponential explosion indeed.

cheers

Colin

 

impressive - perhaps it is a good idea to send it to developers team?

... and interesting is how to prevent such a bog down (using 128 branches):
View attachment bogdown core compiler test1.ens.zip

important is to use event merge module!

ciao herw

 

It's not difficult to prevent the bof down, as long as you understand the cause... you don't need event merge, standard merge will do. In fact, you don't even need to change the original topology of the macro. All you need is a way to stop the compiler tracing further, a +0 will do that!.
But hey, the challenge is to find another way to bog down... not to fix this one!

 

Interesting, with the version using the + module to 'help' the compiler, it compiles nice and quick with 192 modules in the chain. When I attach another 32, Reaktor doesn't bog down, it just crashes and quits (on mac osx 10.6.8)!

Here's the module, connect the final macro in the chain to the output and see if Reaktor crashes.

 

yes i know - nevertheless i think it is important to know how to prevent, f.i. your mention.

ciao herw

 

hmm i can't find it now, but the easiest one i made was trying to re-write the bitmusic ensemble to take place entirely in core.

just two nodes could bog down the compiler - each read 0-2 values from an array, and performed one of 12 or so math operations on the the two values, then stored the value back at a constant location.

 

That sounds intriguing. It would be really cool if you could find it or reproduce it.
These issues have caused me so much trouble in the past that I've given up on whole projects. I stopped using Reaktor for months at one point as a result. But I feel like I'm starting to get a grip on it.

 

i think the level of possible options was like 1 of 32 values from an array, twice, each travelling to one of 12 or 13 math operations, then merging into a single output being written to a constant place within the same array.

i was going to have 32 such structures so you could create very complex bit music equations, but reaktor started to act weird with just 2.

seems reasonable to me (and works fine if you wrap primary around it, but this causes sync problems).

 

Here's another slightly different example.

Note particularly the multiplies on the right hand side. If the outputs of the 8x merges are not interleaved like this, the problem is gone (or reduced). This can happen in real world situations when you are editing. e.g. you are shuffling connections around in your structure while refactoring, and you move an output so that is is both before and after a node with an output from another complex module. This 'inbetween' stage in an edit can catch you out. If the merges are bigger - say 16 instead of 8, then a single interleave would be enough.. ie A before and after B...

EDIT: added an interesting twist to this in version 3
EDIT2: updated bogdown3 to make structure clearer

col

 

Very interesting! Thanks alot, ColB and Herw, for your solutions!

But changing the simple + module of ColB´s solution to a latched operation (x + a) is really evil :angry:

greets, mark

 

sorry i haven't had time to follow this closely since it's an irritating subject and practicing making your computer run slowly is also a PITA since the more you succeed, the more annoying it is!

but it is an important subject for study so thanks for that.

so, is the idea to work around this to merge after a branch or directly before it? does this give us substantial loading speed improvements?

 

There seems to be a bug in the compiler where certain combinations of conditional branches and multiple merges (few big parallel, or longer sequence of singles) causes the compiler to get stuck. The bug(s?) can also causes the compiler to generate inefficient code.
The workarounds seem arbitrary, and don't all work in all situations, however if we can reduce the problems to very concise examples, it may help the developers to fix them, and it will definitely help us to avoid them and to find core hacks to fix them when they can't be avoided.

 

it seems, that undefined parts of the structure could cause this compiling problems. Undefined means, they´re not initialisated by an init event.

I found another solution with OBC:


In this example, the long worm of branches is parted into single pieces by the OBC connection.

 

can somebody please explain why this happens? i am having this problem with my ensemble. i tried the +0 trick but it doesnt work in my case.
i dont understand why this happens and why the fixes you have mentioned work.

thanks

 

Hi,

this is a long-time known issue. Unfortunately, it is quite complicated and until now we could not find a good solution for it. But we regularly return to it, to see if we have any new ideas.

The reason for the issue is the optimization in the compiler, where it tries to analyse the relationships between the events happening on different wires. This works very well, if you follow the "split then merge" or "split off then merge in" approaches. However if you do "completely random" routing of the events, the analysis algorithm may choke up.

The issue can be worked around by "breaking the event dependency chain" at some points. E.g. if you're progressively merging a large number of events, which themselves occur on rather complicated conditions, it might be worth reclocking the result of the merge at one or more points using an event with a "simpler history", like e.g. SR.C. You could also build custom merge modules with custom clocks by connecting a chain of Writes to a custom-clocked Read etc.

So basically the rule is, try to avoid too much of random merging of the events which have complicated runtime conditions (like the ones going through a large number of Routers). If you really have to do a lot of random merging, use the mentioned above workarounds.

As for the efficiency of the generated code, to our best knowledge, the issue is not causing any inefficiencies of the kind.

Best regards,
Vadim

 

you pretty much described my structure when you said lots of "random" merges with events with complicated conditions.

i dont fully understand the suggested solution though.
by reclocking using SR.C do you mean simply run the wire into a latch macro using SR.C to "trigger" the latch (SR.C connected to clock input and signal connected to other input)?
also, does this technique have the same effect as using a +0? i tried the +0 trick with a simplified version of my structure and it worked, so my guess is that i would just need to add more of these workarounds within my structure, or use them in better positions.

my structure needs all the merging and it is pretty much unavoidable to the best of my knowledge.
i think this would explain SR.C clocked latches that i have seen in other ensembles that i couldnt figure out the function for.

thank you for your help

 

Correct

Not exactly the same. +0 is probably less general, as this will affect only the analysis of the event relationships during the initialization event. Basically this ensures that the output event (of the Add module) is always coming during the initialization and thus has a trivial condition of occurence. If it's exactly the initialization event analysis, which is causing the problem in your particular structure, then it's going to help, otherwise not.

 

... and what about send-receive connection? The compiler is running and running if there are many receives and when inserting new send-modules. Would it have an effect to add an audiocore cell with latch module before send and/or after receive?

ciao herw

 

thank you so much. i will try this out in my structure when i get a chance.

 

i just tried this in my structure and it worked beautifully!
thank you so much!
the ensemble even seems to save quicker now but that might be in my head.

i think i went a little overboard with the SR clocks because i noticed the improvement as i was getting towards the end of each chain of events, but i will run over it and see what is actually necessary when i get more time, the point is that i know it can work.

thank you very much for explaining this and helping me with a solution.