Programming the Guide Lights :-)

Discussion in 'KOMPLETE KONTROL SERIES' started by Anykey, Sep 21, 2017.

  1. D-One

    D-One Moderator Moderator

    Messages:
    5,987
    Maybe this might help with that: https://github.com/hansfbaier/open-maschine


    For Maschine in MIDI mode but research done by others could be helpfull
     
  2. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    Thanks a lot for this hint.

    It seems like if the HID header is completely different on Kontrol and Machine. But it seems like the image data is also in XBM format.

    Comparing the Machine example and the USB-data sent to Komplete, it looks like if the Machine, the generic HID sub-interface is used which is also used to set up buttons and knobs and to control the lights. HID commands there are E0 (first display) and E1, second display. On the Komplete Kontrol MK2, a different subdevice (bInterfacenumber=3) to the common HID-interface (bInterfacenumber=2) is used. The data sent to this subdevice is also HID data, but it does not announce itself as a HID sub-device but as unknown sub-device. The command which is sent to there to change the screen content is 0x84 for both displays followed by a few description bytes (which include the display number, 0x00=left, 0x01=right) and then the image data.

    The biggest barrier to make some tests is the case that the HID library does not support the selection of a specific USB-subinterface. So, opening the USB device with it always opens a communication channel to subinterface 2 and not 3. This may make it necessary to communicate directly with the libUSB and to softcode the HID support there.
     
    Last edited: Aug 10, 2018
  3. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    I found out something interesting:

    if you replace the command identifier 0x81 with 0x80, you will set the colors of anything else instead of the key LEDs. And it is amazing what is able to light in different colors on the Komplete Kontrol MK2. For example, you will set custom colors on:

    - any point of the light strip
    - the "M" and "S"-buttons
    - any of the 8 MIDI CC buttons
    - any of the four LEDs around the right knob
     
  4. a1vv

    a1vv New Member

    Messages:
    1
    I made a script for my KK S88 MK1, based on the stuff you all have been doing, to have the keys light up only when the corresponding key gets pressed. Since MK1 and MK2 versions of the keyboard seem to handle colors differently this will most likely only work for MK1 keyboards. If you want to use it yourself you should only need to change two things:
    • numkeys to the number of keys on your keyboard
    • pid to your keyboard's ID.
    I've tested it with ableton, and it doesn't interfere with any other buttons or playback controls. If you start the script when ableton or komplete kontrol is already running, the displays under the knobs will go dark, although they turn on again instantly when you touch one of the knobs.

    This has only been tested on windows, since I don't own any other OS. I don't know how well this will work on any other platform

    https://github.com/a1vv/KompleteKontrolLightGuide

    EDIT: I also modified ojacques script on the synthesia forum to work with MK1 keyboards, which supports multiple colors.
     
    Last edited: Aug 12, 2018
  5. moss

    moss NI Product Owner

    Messages:
    170
    The button IDs are easy to find out, see https://github.com/git-moss/DrivenB...b/mkii/controller/Kontrol2ControlSurface.java
     
  6. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
  7. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    I had success on sending bitmap data to the screens under Linux using libusb.

    For testing, I coded a quick and dirty fork of the qkontrol software and the "bulk transfer" example from this thread:

    https://www.dreamincode.net/forums/topic/148707-introduction-to-using-libusb-10/


    Just paste any bulk image data into the text field and play around, beginning with:

    84000060 (first display)
    84000160 (second display)

    hexadecimal data without any spaces and without a leading 0x


    It works to paste any data here the Komplete Kontrol software submits but really no idea how the image data is encoded. It is completely different to that what is transmitted to the Maschine.
     

    Attached Files:

    Last edited: Aug 15, 2018
  8. moss

    moss NI Product Owner

    Messages:
    170
    That's great news! Could you please upload some examples of the dumps you paste in there?
     
  9. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    If you want to get examples, just run Wireshark under OSX or Windows to examine the USB interface, start "Komplete Kontrol" and monitor the data going to the endpoint 3. Export there the data with out the header there! - up from the third row in wireshark beginning with 84. Right-click to export.

    The data amount is huge. For example, 15 kilobytes just for the text "busy..."
     
  10. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    One example: the number 2

    Code:
    8400006000000000000000f9007a0017020000fb00000003000031a65aeb632c4a4908610200003900000005000039e7dedbfffffffffffffffff7be84100000020000380000000539c7ffdffffff7bebdf7bdd7f79effffffff84100200003800000006bdd7ffffef5d21040000000018c3dedbfffff79e084100000200003700000006f79effff8c710000000000000000738effffffff294500000200003700000006a534a5344a490000000000000000738effffffff294500000200003900000004000000000020d6bafffff79e00200000020000390000000300000841ad75ffffffff73ae0200003a000000034228defbffffffffa514000002000039000000031082a534fffffffff79e6b4d020000390000000400002945dedbffffffffad7518c3000002000039000000041082e71cffffef7d528a00000000000002000039000000028c51ffffffff4a690100000318e318e300000001000000000200003700000001defbffff01000004ffffffff00000001108200000200003700000001ffdfffff01000004ffffffff0000000110820000020001250300000040000000
    
    First two bytes always seem to be 8400.
    Third byte if 00 if data is send to the first display or 01 if it goes to the second one.
    The byte arrays always continues with 600000000000
    next byte (there: 00) is the position on the X-axis where the bitmap begins
    next byte (there: 00) moves the position somehow diagonal when raised
    next byte (there: f9 ) is the position on the y-axis
    next two bytes (there: 00 7a) the pixel width of the placed image, "01 e0" = whole display
    next two bytes (there: 00 17) the pixel height of the placed image (max value unknown yet)

    From there, line fragment data follows with placement describtions and pixel by pixel bitmap data. It is an indexed palette of 65536 colors. 0000=black, ffff=white, other values=something. For example: the number 2 in green by replacing some bytes with 6666:

    Code:
    8400016000000000000000f9007a0017020000fb00000003000031a65aeb632c4a4908610200003900000005000039e66666666666666666666666666b10000002000038000000053966666666666666666666666666666666666610020000380000000666666666666621040000000018c3d6666666666e0841000002000037000000066666666666660000000000000000666666666666294500000200003700000006a534a5344a490000000000000000738e666666662945000002000039000000040000000000206666666666660020000002000039000000030000066666666666666663ae0200003a00000003422666666666666ba514000002000039000000031082a534f666666666666666620000390000000400002666666666666666666b18c3000002000039000000041082e71c66666666666600000000000002000039000000026666666666664a690100000318e318e300000001000000000200003700000001def6666b0100000466666666000000011082000002000037000000016666666601000004666666660000000110820000020001250300000040000000
    
     
    Last edited: Aug 18, 2018
  11. moss

    moss NI Product Owner

    Messages:
    170
    Wow, great! I hope to find some time during the weekend to look into this (have to finish another project first).
     
  12. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    I think that these bytes which make the image appear twisted or distorted describe the width and the height in reality and let the given image appear so because pixels appear shifted if you change the width without adding one new pixel per row.

    After a little bit more research, I found out that the bitmap data is 16 bit and not 8 bit. Each two bytes describe two pixels. Furthermore it is not just pixel data which follows the header. The image format seems to be a "fragment-wise pixel-row placement set".

    Beside the bitmap data, there are frequently small headers following to describe anyhow where the next (following) few pixels are placed. By splitting the data, you get:

    Code:
    8400006000000000000000f9007a0017
    020000fb000000
    03000031a65aeb632c4a490861
    02000039000000
    05000039e7dedbfffffffffffffffff7be84100000
    02000038000000
    0539c7ffdffffff7bebdf7bdd7f79effffffff8410
    02000038000000
    06bdd7ffffef5d21040000000018c3dedbfffff79e08410000
    02000037000000
    06f79effff8c710000000000000000738effffffff29450000
    02000037000000
    06a534a5344a490000000000000000738effffffff29450000
    02000039000000
    04000000000020d6bafffff79e00200000
    02000039000000
    0300000841ad75ffffffff73ae
    0200003a000000
    034228defbffffffffa5140000
    02000039000000
    031082a534fffffffff79e6b4d
    02000039000000
    0400002945dedbffffffffad7518c30000
    02000039000000
    041082e71cffffef7d528a000000000000
    02000039000000
    028c51ffffffff4a690100000318e318e30000000100000000
    02000037000000
    01defbffff01000004ffffffff0000000110820000
    02000037000000
    01ffdfffff01000004ffffffff0000000110820000
    02000125030000
    0040000000
    
    each "02" line describes the offset of the following pixels in the next part. If you set there "02000000000000", next pixels start directly after the last. It is not always neccessary to care about line breaks because the pixels are continued in the next line if the defined maximum with is exceeded.

    The number in the middle of the "02"-parts (here mostly 37,39, 3a) are just enough offset to let the next pixels appear in the next line.
     
    Last edited: Aug 18, 2018
  13. moss

    moss NI Product Owner

    Messages:
    170
    Ah, good catch! After wasting several hours now with this stuff I noticed that this is actually not an image format!
    It is a drawing language similar to Logo! Each of the lines you identified is a drawing command. The first byte is the ID of the command.
    From your snapshot it seems that he following functions are available:

    00 = 4 bytes Finish / Commit the drawing
    01 = 20 bytes ???
    02 = 6 bytes Position the drawing cursor
    03 = 12 bytes Draw something ???
    04 = 16 bytes Draw something ???
    05 = 20 bytes Draw something ???
    06 = 24 bytes Draw something ???

    Could be that the commands 01, 03, 04, 05, 06 only draw lines but with different lengths.
     
  14. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    commands 01 and 02 seem both to be special cases.
    The command beginning with 028c seems to draw a big line instead of positioning the cursor (it makes the bottom line of the 2). Mostly, the Command 01 is used to draw 4 bytes (2 pixels) but there are some cases where more than 4 bytes follow the command. I don't know why and how this is possible yet.

    At the end, the 02 command is used to set the cursor position to the last position of the image before the data is commited. It does not matter if you really do that, but the 02 command after the last drawing is obligatory, beeing a little different by setting 03 after the position.

    02000000030000


    So, I added to the bitmaptest tool another quick and dirty function to load the tux image from the open machine project above to convert it into Komplete-friendly data.

    It works but with the failure that 8 clones of a shrinked version of the image appears. And I have no idea what is wrong with the data.

    Try out the attached tool which has been enhanced with a "draw tux" button.
     

    Attached Files:

    Last edited: Aug 19, 2018
  15. moss

    moss NI Product Owner

    Messages:
    170
    I guess its the values in the header, which I guess describe the size. Did you figure them out?
    Do you know what the resolution of the displays are?
     
  16. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    yes, I figured them out. Last 8 bytes are width (16 bit) and height (16 bit). Display is 01e0 = 480 pixels broad. This value is also used on the product banner images the komplete kontrol software transmits. Bigger values than 01e0 are also valid but senseless. You may create a virtual screen that is bigger than the real screen. If so, only the top-left part is visualized.

    In the case of the example program, for any reason the image becomes minimized and cloned 8 times. After searching a lot, I found out that the reason is just a mis-interpretation of the XBM.

    I modified my test program to draw all possible colors in a for loop and it fits the whole size - without having any shrinked clones. And regarding the illustration, the pixels may be pure RGB16.
     
    Last edited: Aug 20, 2018
  17. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    These commands 03 and above always draw one pixel for each two following two bytes (65536 colors, 16 bit). You can continue the list (07=28 bytes, 08=32 bytes and so on). Works.

    One special case is the 01 command. It is commonly used followed by 8 bytes describing two pixels. But there are also some cases where lots of data follow the 01 command to do anything different. But may be they don't matter.

    This drawing language may be easier to understand if this isn't a creation of Native Instrument but used to control common displays available as electronic spare parts. But that is not easy to research. I didn't find any advice that there are other displays which are instructed by the same way.
     
    Last edited: Aug 20, 2018
  18. j0tt

    j0tt New Member

    Messages:
    12
    I did some initial work some time ago partly based on Drachenkaetzchens Maschine MK3 sketches building some working parser and generator code.
    In my understanding the protocol is quite simple (even simpler than Drachenkaetzchens interpretation):
    You have the "Command Code" (16-bit), like 0x0000 "transmit pixels" or 0x8400 "start of data" always followed by and 16-bit argument and an optional variable data payload in 32-bit chunks (so it is always 32-bit aligned). Everything is in big endian byte order!

    So far I derived these "Command Codes":
    Code:
    enum class CommandCode : quint16
        {
            COMMAND_TRANSMIT_PIXEL = 0x0000,
            COMMAND_REPEAT_PIXEL = 0x0100,
            COMMAND_SKIP_PIXEL = 0x0200,
            COMMAND_BLIT = 0x0300,
            COMMAND_START_OF_DATA = 0x8400,
            COMMAND_END_OF_DATA = 0x4000
        };
    
    Now you can just build a sequence of commands like COMMAND_START_OF_DATA, COMMAND_TRANSMIT_PIXEL, COMMAND_BLIT, COMMAND_END_OF_DATA and send it to the proper USB device. It works reasonably well just pushing full updates this way to the screen but you can also optimize the process (like the Kontrol software does) by only blitting the changed parts (even multiple regions using the skip pixel command) or do fills with the "repeat pixel" command.

    So very basic implementation I hacked together could look like this:

    Code:
    class KompleteKontrolImageConverter
    {
    public:
        KompleteKontrolImageConverter(const QImage& image, quint8 display = 0, quint8 x = 0, quint8 y = 0)
        {
            buffer.setBuffer(&byteArray);
            buffer.open(QIODevice::WriteOnly);
            dataStream.setDevice(&buffer);
            startOfData(display, x, y, image.width(), image.height());
            writeImage(image);
            blit();
            endOfData(display);
        }
    
        QByteArray data() const
        {
            return byteArray;
        }
    
    private:
        enum class CommandCode : quint16
        {
            COMMAND_TRANSMIT_PIXEL = 0x0000,
            COMMAND_REPEAT_PIXEL = 0x0100,
            COMMAND_SKIP_PIXEL = 0x0200,
            COMMAND_BLIT = 0x0300,
            COMMAND_START_OF_DATA = 0x8400,
            COMMAND_END_OF_DATA = 0x4000
        };
    
    
        void startOfData(quint8 display, quint8 x, quint8 y, quint16 width, quint16 height)
        {
            dataStream << (quint16)CommandCode::COMMAND_START_OF_DATA;
            dataStream << display << (quint8)0x60;
            dataStream << (quint16)0 << (quint16)0 << (quint16)0 << x << y << width << height;
        }
    
        void writeImage(QImage image)
        {
            dataStream << (quint16)CommandCode::COMMAND_TRANSMIT_PIXEL;
            dataStream << (quint16)(image.byteCount()/4);
            ushort *swappedData = reinterpret_cast<ushort *>(image.bits());
            for (int i = 0; i < image.byteCount() / 2; i++)
            {
                swappedData[i] = _byteswap_ushort(swappedData[i]);
            }
            buffer.write(reinterpret_cast<const char*>(swappedData), image.byteCount());
        }
    
        void blit()
        {
            dataStream << (quint16)CommandCode::COMMAND_BLIT;
            dataStream << (quint16)0;
    
        }
    
    
        void endOfData(quint8 display)
        {
            dataStream << (quint16)CommandCode::COMMAND_END_OF_DATA;
            dataStream << display << (quint8)0;
        }
    
        QByteArray byteArray;
        QBuffer buffer;
        QDataStream dataStream;
    };
    
    So you can take an QImage image and convert it to Format_RGB16 and pass it to the converter like...
    Code:
    QByteArray data=KompleteKontrolImageConverter(image.convertToFormat(QImage::Format_RGB16)).data();
    sendDataToDevice(data);
    
    As you see the initial COMMAND_START_OF_DATA takes the display number followed by "0x60" (what ever this means) as argument. The "payload" is then a sequence of zeros, the x, y position as bytes as well as the size (again in total 32-bit aligned).
    For the COMMAND_TRANSMIT_PIXELS it's similar, you have the number of 32-bit-words that follow as argument and then the image data (in big endian, as 565-16-bit image in 32-bit-words of data chunks).
    Keep in mind that when "optimizing" (by skipping pixel or filling with a color) the final image must have the exact dimensions that it was created with.
    The other commands work also like this, so "skip pixel" has just the number of pixels to skip as argument, and "repeat pixels" the number of pixel-pairs to repeat followed by the colors (to make it 32-bit aligned again).

    Hope this helps ....
     
  19. GoaSkin

    GoaSkin NI Product Owner

    Messages:
    81
    Thanks a lot.

    Interesting code. I will try out how to make use of it. Yesterday I tried to load image files using the QImage class function but I hat to realize that it is a pain to get 16-bit RGB data from a QImage.
     
  20. j0tt

    j0tt New Member

    Messages:
    12
    Well, as you probably noticed above QImage::bits() is what you are looking for. Unfortunately there does not seem to be a 565-image format that is in big endian so you have to swap the bytes. For increased performance you should either only operate on an QImage with the closest image format (so QImage::Format_RGB16 seems to be the closest you can get with regular Qt so you only need to swap bytes and do not need to convert it on each update) or do the conversion/dithering your self with the correct endianess in mind.

    Anyway, this general approach makes it easy to do some offscreen rendering of a QML pipeline and draw the corresponding image on the device. You can achieve quite fluent animations with it. This is actually what the Ableton Push does for displaying its contents...

    I am on the Mac/Windows side of things so unfortunately I haven't found a way yet to tell the NIHardwareService to hand over control of the display so every "external" update will interfere with my custom renderer. Maybe someone here has an idea how to prevent this. I wonder what the Maschine software does as it'll have more dedicated views in addition to the browser/instrument/midi view (I don't own a Maschine yet unfortunately).