Sending long SYSEX message to hardware

[Yot]

Hi folks,

I would like to send a long SYSEX message to a USB controller.
The array is 451 characters long. If I send it in one shot, I obviously get a
"usb host midi output ringbuffer overflow"

Do you know a simple way to "split" it in some way.

I'm not a big C coder so I just tested by splitting the entire array manually with multiples arrays of 20 with a 10 ms pause between each send. It works but it's too long to code.

Is there a way I can code a pause every 20 characters of the array basically?

Thanks for your help.

[tele_player]

What have you got so far?

[Yot]

What do you mean? Here's my very "dirty" piece of code ...

[ivofx]

I was thinking about this problem yesterday, as I want to program my Novation Zero SL MK2.
Calling sleep is likely to cause audio dropouts.
I think what we need is some kind of ringbuffer that stores the individual bytes and in the control-rate call you would only send byte-chunks of a fixed size at a fixed rate (probably on every 50th out of the 3000 times per second or so).
I did not implement any of this yet and am wondering whether we can simply reuse an existing ring-buffer code from some existing object, like a delay.
The method could also be applied to regular midi messages or NRPNs (as when controlling many LEDs)

[jaffasplaffa]

I had success sending shorter strings to Blofeld, like parameter systes strings.

But when I tried to send sound dumos, which is longer strings, it failed on me. So very interested in seeing that you have something working for longer strings

[tele_player]

Was that code sending to a USB controller? Maybe I’m mistaken, but isn’t sdWrite for serial device? Or also for USB?

[Yot]

It works also for a USB controller.

[tele_player]

There isn't any need to break the data into multiple arrays.
You use variables to store your position in the array, and the number of bytes remaining. That's simple.

What isn't clear , does Axoloti firmware provide a mechanism for querying the amount of space in the send buffer? Without that, you'll only be quessing how many bytes to send, and when, which wil be different for DIN MIDI, and USB MIDI.

Added later:
This might be useful:
int MidiGetOutputBufferPending(midi_device_t dev);
And this, but it's only implemented for MIDI_DEVICE_USB_HOST:
int MidiGetOutputBufferAvailable

For more details, look in firmware/midi.c (after downloading the Axoloti source code)

[ivofx]

I was reading about ring-buffers and made a python prototype that could be translated to c/c++ (disclaimer, the implementation might be naive/messy).
I saw that the Axoloti delay objects brilliantly use bit-masking to make the indexing wrap around sizes multiples of two and adapted that. So if you want the size to be 4kb you would set it to 1 << 12 for example.
If / when I create an axo object, I would allocate an sdram-table for the storage inside the object.
Then, in every 50th control-rate call or so I would read say 32 bytes and send them as a sysex message. I assume sending a single sysex message in multiple chunks is possible (?)

# Python 2.7
class ByteRingBuffer(object):
    head = 0
    tail = 0
    size = 1 << 12
    mask = size-1
    length = 0
    full = False

    def __init__(self):
        self.buffer = [0] * self.size

    def write(self, value):
        if self.length > self.mask:
            self.full = True
            print "FULL!"
            return
        self.buffer[self.tail] = value
        self.tail = (self.tail + 1) & self.mask
        self.length += 1

    def read(self):
        if self.empty():
            return -1
        result = self.buffer[self.head]
        self.head = (self.head + 1) & self.mask
        self.length = max(self.length-1, 0)
        self.full = False
        return result

    def empty(self):
        return self.length <= 0

    def full(self):
        return self.full

[ivofx]

I turned it into an work-in-progress axo object. The only problem is defining the sysex strings conveniently in the UI.
Because strings are null-terminated and you cannot enter hex-values in the GUI widgets.

ivogrig/axoloti-contrib

You load the sysexbuffer object, then you can right-click and select "Help" to load the help-patch.

At the moment you need to define the sysex-data inside the code.

I have a vision to make defining the data more user-friendly.
You would save each sysex-string as a file in a certain directory and prefix the file with a number.
The object would load all files from the directory into a table and keep track of the start-offsets.
You would have an inlet to choose the sysex-string by index and an inlet to trigger the transmission.
On top of that one could setup a substitution-syntax to replace certain bytes with a variable value.
However, I have different plans for the weekend so it will take some more time.

[tele_player]

A ring buffer doesn't solve the problem of output buffer overrun, that is solved by checking the amount of space in the ring buffer before doing a send.

Agreed, for a long sysex message, you'll need to do it over several control-rate calls to your object. BUT, while you have a sysex message in progress, you also need to stop sending any other MIDI. For instance, if you send a NOTE ON or NOTE OFF, while your sysex is only partially complete, the sysex message will be corrupted.

[ivofx]

True, I did not think of these aspects
The code is not keeping track of the individual message lengths as of yet.
I seems quite tricky to ensure no other midi is transmitted when sending sysex in split chunks, I have no idea for that.
Still, a ringbuffer seems handy for storing and keeping track of the chunks.

Theoretically speaking, if it would be possible to grab data from MidiGetOutputBufferPending end erase the pending buffer somehow to resend it later, that would help.

Actually, the buffer appears to be exposed in the _MIDI_Process struct (looking at the USB one)

  uint8_t buff_in[USBH_MIDI_EPS_IN_SIZE];
  uint8_t buff_out[USBH_MIDI_EPS_OUT_SIZE];
  uint8_t buff_out_len;

Still, one does not know where in the order of objects the own call happens, probably in the middle, so no certainty gained.

[tele_player]

Assuming MidiGetOutputBufferPending() returns the number of bytes currently in the buffer, and goes down as bytes actually go out on the wire, your wouldn't want to erase pending bytes. Just let them fly.

Yes, up at the object level, blocking MIDI sends while a SYSEX message is in progress would require cooperation between objects used for sending normal MIDI and objects used for sending SYSEX. Sending long messages should really be handled in the firmware.

A ringbuffer is useful for queuing multiple messages. Not needed when your code is handling one message at a time. That's why the output buffer is implemented as a ringbuffer. At the level of your objects, you can send MIDI events asynchronously, without concern about other messages 'in flight'.

All of this stuff is very common in data communications software and file system software.

[ivofx]

OK I give up then. Hopefully the USB buffer size will be large enough for my needs, I think I will be fine.
I made the ring buffer because I want to allow my code to queue multiple messages at once and not be limited to one at a time.

[ivofx]

The output-buffer size USBH_MIDI_EPS_OUT_SIZE is defined as 64, but calling MidiGetOutputBufferAvailable will return 63 if nothing has been sent yet.

However, I can send more than that without getting the usb host midi output ringbuffer overflow error.

From observation, the maximum of sysex bytes that I was able to send without getting the error and nothing else going on was 185. 186 would error.

int available = MidiGetOutputBufferAvailable(MIDI_DEVICE_USB_HOST);
int pending = MidiGetOutputBufferPending(MIDI_DEVICE_USB_HOST);

EDIT:
Oh, I get it. The buffer might be for 3-byte messages. If we subtract one, we get 62*3=186 (?)
In fact, if I send a note-on and then call MidiGetOutputBufferAvailable, it returns 62.

[tele_player]

(I wrote something here, which I now think is wrong, but a correct answer needs more study than I can do right now)

[Yot]

Thanks ivofx. That's good to know.

[ivofx]

I wonder whether we can simply change USBH_MIDI_EPS_OUT_SIZE in the firmware, what would the consequences be. Is it a hardware limitation?
It's a bit of uncharted territory for me.

My own first goal ist just to write something like a patch editor where you only send short messages for indivdual synth-parameters, for that the current size limit is not a problem.
Sending longer patches would be a nice cherry to have on top.
Given that even receiving sysex is not easily done either, being only able to send long patches without receiving them seems not even that useful to me.
It would be cool to have the Axoloti be a sysex librarian though, I have several devices that couldbenefit from it.