MIDI in 1.0.10 more unreliable (Launchpad)

#1

In the last days i tried to transfer my current launchpad sequencer object from 1.0.9 to 1.0.10.

In the description of the test release i found this note

but in my case i found it more unreliable as before.

Now i am getting a lot of midi buffer overflows and the axoloti crashing when interacting with the launchpad very fast or sometimes the axoloti stops reacting to the input of the launchpad while the output to it still works fine (without any error messages). Both bugs i can not reproduce with 1.0.9. with the same objects.

So are there any major changes in version 1.0.10 which could cause this behavior?
Something i have to consider for the transfer process?

#2

you may want to check the output port on the launchpad, this may have changed with 1.0.10... BUT if this was the cause, then no output would work at all. (otherwise this is probably related to the 'dma' changes)

midi buffer overflows are caused by:

a) connected device not reading (or fast enough)
b) the midi send poll not being serviced

@johannes the change made regarding the 'polling' loop, you may want to check that there was no 'side effect' on the send not being called as regularly, output send frequency should not be tied to input obviously. (Ive not had a chance to review the changes)

what seems 'odd to me' , is that with output, Ive found in the past it tends to either work or not...
(assuming your not increasing the number of messages to output significantly)
to test, try running a test which has an LFO sending messages (CC say) over midi, and see at what rate it starts to fail. (note, there is a buffer, so leave it a few seconds, before assuming there is not an issue)

on midi input side, the test i did on midi devices had no issues, and early reports are that generally 1.0.10 has improved things. does this only fail, after output as failed? or its independent?
does it initial work, and then stop? is it dropping occasional messages? or one its fails, it stops working entirely?

Sorry, I don't have any time at the moment to do any debugging of this (or anything for that matter)

I think your best bet is to do as many tests as possible to narrow down the issues, also test with other devices to see if its just the launchpad... the more information and tests you can supply to @johannes the easier he will find it to narrow down the issue.

EDIT: I tested midi output with my Virus TI, sending notes and CC 1, at 200 msg/sec using the following, and no issue.

Screen Shot 2016-05-21 at 14.33.42.png820x306 18.4 KB

try the same with your launchpad, see if it works... and as I say, check using different host ports.

(one thing, I have noticed is we should throttle the ring buffer message to only report at a max of 1/sec or something so as not to kill the UI by flooding it with log messages ... that or consider an anti flood mode on the LogTextMessage as a more general improvement)

#3

@thetechnobear thank you for your quick answer! So your test setup worked just fine for the launchpad. but i am not sure if it is representative for my case.

in your setup you are sending max two midi messages in the same k-rate cycle, one midi note message and one midi cc. the only difference is the time interval in between. so you are testing how quick midi messages can be sent after each other (but in different k-rate cycles).

for the sequencer i have to be able to change "pages" displayed by the launchpad. in the worst case each pixel has to be resent to the launchpad to output the correct "information". so there are max 64 midi messages which have to be sent to the launchpad to update the whole button-matrix.

this approach led to midi buffer overflow under 1.0.9. so i decided to take advantage of the rapid update feature of the launchpad. instead of sending each pixel independently. two pixels can be updated by sending one midi message. this approach just works if the whole matrix is updated. so now i am sending constantly ca. 32 midi messages. to reduce the midi traffic i just update the matrix each 150 k-rate cycles. this worked just fine under 1.0.9.

its really strange that the output keeps working while no more input can be received. i tested the described setup in addition with an "midi/in/keyb" object and the keyb also stopped receiving midi from launchpad. after that i "un-lived" the patch and then pressed "live" again. the sequencer correctly displayed the data over the LP but still no messages could be received. after physically reconnecting the axoloti the patch worked again (input and output).

on code side i made sure that the matrix updating process can not be interrupted. but it does not any difference.

#4

Sorry, Ive not seen any input issues... so can't comment on that.

(generally, input and output are only 'partly' related, so its not a big surprise if input works, but output doesn't and vice versa)

yes, I know, my test was about sustained throughput,and as I don't have my Push 1 anymore, I don't really have a way of testing bulk messages easily. (and this, actually comes down to how you are throttling your output, which is necessary) ... unfortunately this means I can't comment on if the throughput has deteriorated in 1.0.10 or not. merely, its seems to be work ok as far as i can tell.

axoloti has a ring buffer of 64 messages for output, which is a reasonable trade off of 'allocated memory' vs throughput. this means you should not be sending more than 64 messages at a time

so yes throttling output, is expected of the client code, and an 'available' count is supplied for you to check.( I did this on the Push , but keep track of state, and only sending new state with there were no pending messages) ... the available count is important, you should not assume that you can write at N msgs/sec.

its obviously important when writing these kind of objects, to ensure you minimise the number of messages sent, and like audio processing, its better to spread the 'load' over as many cycles as possible whilst ensuring a decent user experience in terms of response.

#5

I'd like to replicate your problem, is the Launchpad Mini MK2 compatible with your object? I'm considering getting one now...

#6

@johannes that would be awesome . i am not familiar with the newer launchpad products, but on a quick research i found the statement that all the features of the older products are supported. so i think its worth a try (and i am curious to see if it works).

this is the way you should setup the object:

the object (voice 4.axo (5.3 KB)) is not finished yet. but the main interaction with the launchpad is already finished ("midi_channel", "root_note" spinners have no implemented functionality).

to reproduce the bug you have to press chaoticly and quick the matrix buttons. sometimes when it didn't crashed during this procedure, it crashed while turning off all the selected steps.
(both things i couldn't reproduce in 1.0.9.)

thank you for your efforts! i really appreciate this direct interest and communication.

#7

have you tried using MidiGetOutputBufferPending(), to throttle the updates, so you know you are not overrunning the buffer?

I had a quick look at your objects, and both appear to just 'assume' that they can write, this is an incorrect assumption.

and if you

that I would imagine mean thats lots of midi messages are sent, so you could overrun the buffer.

it may be that 1.0.10 has an issue but still your code should be handling the ring buffer correctly.

Im very interested if you dont overrun the buffer, if the crashing stops...
I suspect it might, and i suspect you will also find the throughput it ok.

a point of note: I put in the ring buffer overflow message to alert the user to the fact they are not handling the buffer correctly... i.e. a warning to alter your code, since once overrun you have no idea what messages have been sent and which not, you are in a completely unknown state.

btw, when you say it 'crashes' what do you mean by this? does the ui disconnect? does the core board continue to process anything?
one 'known' issue, is its possible to get so many ring buffer overflow messages you will break the link to the UI, and possibly hang the UI. this is a common problem with our log messages, that if you write too many it causes issues for the UI.

#8

@thetechnobear to be more precise: after the described procedure the axoloti seems to ignore further input from the launchpad but keeps on running the patch, i.e. the audio keeps running (sometimes with glitches) and the sequencer is correctly displayed on the launchpad (lights keep moving in sync, UI keeps working, ...). sometimes the axoloti says there is an midi-buffer overflow but most of the time it says there is no overflow...

no i don't. because i originally developed the object under 1.0.9 and i eliminated all overflows and other bugs there. that's why i wondering... otherwise i would assume that i made some kind of mistake...
i will try this command and see if it will get me some new insights to this problem. by the way is there some kind of API are other listing of axoloti specific functions?

in the object i posted above i made sure that not more than 40 messages are sent in one k-rate cycle every 150 cycles. so i don't think i can overrun the output buffer?
is there a buffer for incoming messages too, which could cause the overrun?

#9

no there is (currently) no queue on the input, its directly dispatched to the objects.

there really is not a lot of code shared between input and output, except for the underlying device.
(its interesting you see output is still working)

if you get the overflow error, then for some reason you are writing faster than the launchpad is processing the messages. (we can only write a new message when we get the OK back from the usb layer). we have the ring buffer to 'smooth' out the possible jitter, but it can only do so much (64 messages backlog)
... again, its interesting that your seeing its still updating the display after the overflow, this indicates message are still going out.
(the way I would probably debug this, is to send the messages to a PC/Mac instead, and log them, you might see that in some circumstances you are sending more than you expect)

can you check to see if the launchpad input stops working when your not doing any output...
it could be the input, is just failing on its own. this would be extremely useful to know.
(so we can rule out some kind of interaction between input and output being the cause)

#10

@thetechnobear i integrated the MidiGetOutputBufferPending() in the code like this:

int m = MidiGetOutputBufferPending(MIDI_DEVICE_USB_HOST);
if (m==0) { /*update launchpad leds*/ }
else /*log m*/;

but it doesn't help.

also i can reproduce the bug with these factory objects inside a poly patcher set to 16 voices.

again it's not possible to recreate it under 1.0.9

maybe it's a wrong conclusion, but it seems as if the problem is connected to version 1.0.10 and not to custom code inside some objects?

i hope that the last experiment helps to delimit the source of the bug?
thank you for all your time and your investigations.

#11

The problem is most likely related to my tweak of the usb midi host code


which was solving dropped midi data in some cases.
In detail, when DMA is enabled for the USB host, it was found that, after receiving a packet, it is not needed to wait until a timer expires before calling USBH_BulkReceiveData() again. Without DMA this would fail very often, with lost midi reception as a result - exactly as you're reporting. With DMA there may still be a condition where the usb host is not ready to initiate reception again, will look into this...

#12

OK, I can reproduce this issue with a launchpad mini, stay tuned...

#13

I have similiar issues with my LaunchcontrolXL here. Sometimes Axoloti doesn't take input from the controller anymore but the midi feedback into the controller works fine.
This was not an issue with previous versions.

#14

I'm working on a fix...

#15

Hey @johannes

I just updated to 1.0.10 and I am looking at getting a Novation Launch Control and thought I'd search the community and stumbled on this thread. How is this going?

Thanks

#16

I'm preparing a release, which contains a fix for this.

USB Overflow using Novation Launchkey Mini MK2
#17

Super great! I all ready have it in the shopping basket in Thomann. Checkout is next :slight_smile:

Thanks.

Ps. Sounds great about new release :wink: