Creating new objects - Guidlines for operations on large buffers?

Joren · 2015-08-11 11:34:59 UTC

Hi,

I am porting a monophonic pitch detection algorithm to an Axoloti object. The object does more ore less what it needs to do: it detects pitch in a buffer of 1024 samples. There are however a few things unclear to me:

1) What is the best way to execute operations on large buffers? I need 1024 samples, to I currently create an array and fill it with the incoming samples. After 1024/16=64 blocks of audio I do some pitch estimation calculation. Since audio starts to stutter i think this takes longer than 1/3000th of a second. I'm not sure how to handle this: do I create a thread as in the ifft object? Is that a good pattern to follow? chThdCreateStatic(waThreadX, sizeof(waThreadX), NORMALPRIO, ThreadX, (void *)this);

2) To organize the code I have introduced some methods in the declaration part of the object. Since I had not seen this in any other place I'm not sure if this is a good idea. Are there any provisions to create methods?

3) I want to return a fractional midi number that is compatible with the rest of the axoloti objects. So ranging from -64 to 63. I do not really understand the conventions that use fixed point math as fractional numbers as in the axoloti_math.h functions. Any guidlines there? How do i return a float? For the moment I this as outlet: <frac32 name="out" description="out"/>.

4) I used floating point math in my code. This has a performance penalty. Any idea how much there is to gain when converting the code to fixed point?

5) Is there a repository where i can publish objects that are perhaps of interest to others but too experimental/unstable/underperforming to include in the main distribution?

yin_test.axp (2.0 KB) yin.axo (9.3 KB)

thetechnobear · 2015-08-11 14:09:32 UTC

this could turn into an interesting thread
1) large buffers - I'll leave that for @johannes

2) there is an includes tag, which allows you to include header files, I usually use that to included a header, which then includes a c file, with the methods. (in case we later support separate source compiling)

3) have a look at the midi/in objects they show it reasonably clearly.

4) interesting topic as F4 has an FPU, and mutable instruments implied floats are actually better than used fixed ints BUT it will really depend on your code. (go check the MI forum search Axoloti to see their comments on this)

5) if you fork the axoloti repo, then you can issue pull requests. currently we have weeded out experimental stuff, as it just confuses users ... what does experimental mean? it works or not? what are the limitations? but perhaps we can add it back again (we have some objects in archive, that fall into this category)
theres also some idea of having a 'contributors' archive, as this might help with ownership issues.

johannes · 2015-08-11 17:09:14 UTC

Interesting, pitch detection is still a uncovered in the object set.
A quick reply:
1) check the spectral/rfft 128 object for an example
2) you can declare local methods in sLocalData.
3) float f = 12.345; outlet_out = (int)(f*(1<<21));
4) comparing float versus integer dsp performance, is a complex equation, really depending on the nature of the algorithm. Sometimes float32 is not accurate enough where int32 is more accurate. Float32 allows extremely wide dynamics, over 1500dB, this is far beyond useful, some bits would have been better "spend" on precision.
Float64 is accurate enough for "everything" but has a big performance impact.

Joren · 2015-08-14 11:47:21 UTC

Hi Johannes,

I have tried a few things but returning a float does not yield the expected results. If I understand correctly I need to convert the float to a fixed point int. I declare an output as follows:

<frac32 name="out" description="out"/>

And the only thing in my krate code is this:

float f = 12.345; outlet_out = (int)(f*(1<4));

if I display this with a disp/dial p (or scope or bar or..) it shows zero. What do I need to do to show the expected 12.345?

johannes · 2015-08-14 11:52:54 UTC

sorry there went a mistake in there
try

outlet_out = (int)(f*(1<<21));

Joren · 2015-08-14 12:14:33 UTC

Ok,

That seems to work! An update version with working pitch detection: yin.axo (9.3 KB)

Next up: threading to play nice with the audio pipeline.

Joren · 2015-08-14 15:12:15 UTC

An update to enable choosing the buffer size yin.axo (10.4 KB) via attributes. The buffer size determines the delay, computational load and the minimum frequency that can be detected. I was hoping that it could prevent audio glitches with lower buffer sizes, but it doesn't.

I'll really need to look into creating threads.

The object also returns a periodicity value. A number between 0 and 64 to determine the certainty of the estimation, the periodicity of the input signal. An example patch can be seen below or downloaded here yin_test.axp (2.7 KB) :

Joren · 2015-08-17 10:37:36 UTC

I have updated the yin.axo (10.9 KB) object to use threads. It also has correct hashes, so the patcher does not complain about that any more.

Two things are still unclear to me. One: it uses the following as working area: WORKING_AREA(waThreadX, 512);. The ChibiOS documentation tells me the following, but I'm not sure how to measure stack size. Any tips here @johannes?

This macro reserves x bytes of stack for the thread and space for all the required thread related structures. The total size and the alignment problems are handled inside the macro, you only need to specify the pure and simple desired stack size.

Two: I want to have a low latency pitch tracker. The buffer size is one limiting factor but another one is the thread sleep command chThdSleepMilliseconds(10); Which adds another 10ms in the worst case. So with a buffer of 1024 samples you get: 21.33ms (1024 samples) + 10ms (worst case thread sleep) + 1ms (estimation of the pitch detection calculation) = +- 33ms. If I go below 10ms of thread sleep, it seems that the whole Axoloti environment is getting less reliable (connection timeouts). Is there a hard lower bound there?

I also wanted to comment that the Axoloti environment is a very nice environment to create objects for!

thetechnobear · 2015-08-17 12:04:29 UTC

yeah I like it too...
a couple of suggestions

I tend to keep all my C code in a separate file, (.c/h) which I then include via
(this opens up some testing possibilities)
one thing I notice yesterday, is we are using -O3, and including your C code as you do, we tend to make this code 'inline', sometimes this is what you want, but it can lead to unexpected bloat... and you'll start (unexpectedly) running out of SRAM.

im not sure either is relevant for this project, but thought id mention it... as when the code increases in size, I found the above useful.

as for threading, this is an interesting 'development' topic... I use threading a lot on 'normal computers', but Im hesitating on the STMF4, its only single core, so there is no parallelisation... so whilst conceptually tidy, I wonder if it comes at a (performance) cost.
e.g. what the difference between doing this, and using your audio callback, and 'chunking' the work appropriately.

Id be interested in thoughts from those more familiar with micro controller programming... to thread or not to thread that is the question

Blindsmyth · 2016-03-18 18:38:48 UTC

Hey there! I want to use the Yin object for creating some kind of guitar bass synthesizer. Unfortunately I behaves a bit buggy. When I first load it into a project it works fine. But when I change something in the patch it won't work anymore. I try to go in live mode and takes some seconds and then disconnects axoloti. When I reconnect then, sometimes I am able to go live.
With bigger patches nothing works like it should anymore...
This is the axoloti log when I try to use Yin:

instance added, type yin
Generate code complete
Start creating directory on sdcard : /untitled
creating dir: /untitled
Done creating directory
Changing working directory on sdcard : /untitled
Change working directory: /untitled
Done changing working directory
Start compiling patch
Compiling patch... with /Applications/Axoloti.app/Contents/Java/firmware
BDIR = /Users/Simon/Documents/axoloti/build
FIRMWARE = .
RM
rm -f /Users/Simon/Documents/axoloti/build/xpatch.o /Users/Simon/Documents/axoloti/build/xpatch.elf /Users/Simon/Documents/axoloti/build/xpatch.bin /Users/Simon/Documents/axoloti/build/xpatch.d /Users/Simon/Documents/axoloti/build/xpatch.map /Users/Simon/Documents/axoloti/build/xpatch.lst
APP
arm-none-eabi-g++ -nostdlib -fno-exceptions -fno-rtti -mcpu=cortex-m4 -O3 -fomit-frame-pointer -falign-functions=16 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -Wunused-parameter -DCORTEX_USE_FPU=TRUE -DTHUMB_PRESENT -mno-thumb-interwork -DTHUMB_NO_INTERWORKING -mthumb -DTHUMB -std=c++11 -DARM_MATH_CM4 -D__FPU_PRESENT -H -I/Applications/Axoloti.app/Contents/Java/CMSIS/Include -I/Applications/Axoloti.app/Contents/Java/chibios/os/ports/common/ARMCMx/CMSIS/include -I/Applications/Axoloti.app/Contents/Java/chibios/os/ports/common/ARMCMx -I/Applications/Axoloti.app/Contents/Java/chibios/os/ports/GCC/ARMCMx -I/Applications/Axoloti.app/Contents/Java/chibios/os/ports/GCC/ARMCMx/STM32F4xx -I/Applications/Axoloti.app/Contents/Java/chibios/os/kernel/include -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/include -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32F4xx -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/GPIOv2 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/I2Cv1 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/OTGv1 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/RTCv2 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/SPIv1 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/TIMv1 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/USARTv1 -I/Applications/Axoloti.app/Contents/Java/chibios/boards/ST_STM32F4_DISCOVERY -I/Applications/Axoloti.app/Contents/Java/chibios/ext/fatfs/src -I. -I/Applications/Axoloti.app/Contents/Java/chibios -Winvalid-pch -MD -MP --include /Users/Simon/Documents/axoloti/build/xpatch.h -c /Users/Simon/Documents/axoloti/build/xpatch.cpp -o /Users/Simon/Documents/axoloti/build/xpatch.o
! /Users/Simon/Documents/axoloti/build/xpatch.h.gch
LINK
arm-none-eabi-gcc -nostartfiles -Tramlink.ld -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -mthumb -mno-thumb-interwork /Users/Simon/Documents/axoloti/build/xpatch.o -Wl,-Map=/Users/Simon/Documents/axoloti/build/xpatch.map,--cref,--just-symbols=./build/axoloti.elf -o /Users/Simon/Documents/axoloti/build/xpatch.elf
BIN
arm-none-eabi-objcopy -O binary /Users/Simon/Documents/axoloti/build/xpatch.elf /Users/Simon/Documents/axoloti/build/xpatch.bin
Done compiling patch
Start uploading patch
bin path: /Users/Simon/Documents/axoloti/build/xpatch.bin
block uploaded @ 0x20011000 length 4116
Done uploading patch
Start starting patch
Done starting patch
Start locking
Done locking
Generate code complete
Start creating directory on sdcard : /untitled
creating dir: /untitled
Done creating directory
Changing working directory on sdcard : /untitled
Change working directory: /untitled
Done changing working directory
Start compiling patch
Compiling patch... with /Applications/Axoloti.app/Contents/Java/firmware
BDIR = /Users/Simon/Documents/axoloti/build
FIRMWARE = .
RM
rm -f /Users/Simon/Documents/axoloti/build/xpatch.o /Users/Simon/Documents/axoloti/build/xpatch.elf /Users/Simon/Documents/axoloti/build/xpatch.bin /Users/Simon/Documents/axoloti/build/xpatch.d /Users/Simon/Documents/axoloti/build/xpatch.map /Users/Simon/Documents/axoloti/build/xpatch.lst
APP
arm-none-eabi-g++ -nostdlib -fno-exceptions -fno-rtti -mcpu=cortex-m4 -O3 -fomit-frame-pointer -falign-functions=16 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -Wunused-parameter -DCORTEX_USE_FPU=TRUE -DTHUMB_PRESENT -mno-thumb-interwork -DTHUMB_NO_INTERWORKING -mthumb -DTHUMB -std=c++11 -DARM_MATH_CM4 -D__FPU_PRESENT -H -I/Applications/Axoloti.app/Contents/Java/CMSIS/Include -I/Applications/Axoloti.app/Contents/Java/chibios/os/ports/common/ARMCMx/CMSIS/include -I/Applications/Axoloti.app/Contents/Java/chibios/os/ports/common/ARMCMx -I/Applications/Axoloti.app/Contents/Java/chibios/os/ports/GCC/ARMCMx -I/Applications/Axoloti.app/Contents/Java/chibios/os/ports/GCC/ARMCMx/STM32F4xx -I/Applications/Axoloti.app/Contents/Java/chibios/os/kernel/include -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/include -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32F4xx -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/GPIOv2 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/I2Cv1 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/OTGv1 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/RTCv2 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/SPIv1 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/TIMv1 -I/Applications/Axoloti.app/Contents/Java/chibios/os/hal/platforms/STM32/USARTv1 -I/Applications/Axoloti.app/Contents/Java/chibios/boards/ST_STM32F4_DISCOVERY -I/Applications/Axoloti.app/Contents/Java/chibios/ext/fatfs/src -I. -I/Applications/Axoloti.app/Contents/Java/chibios -Winvalid-pch -MD -MP --include /Users/Simon/Documents/axoloti/build/xpatch.h -c /Users/Simon/Documents/axoloti/build/xpatch.cpp -o /Users/Simon/Documents/axoloti/build/xpatch.o
! /Users/Simon/Documents/axoloti/build/xpatch.h.gch
LINK
arm-none-eabi-gcc -nostartfiles -Tramlink.ld -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -mthumb -mno-thumb-interwork /Users/Simon/Documents/axoloti/build/xpatch.o -Wl,-Map=/Users/Simon/Documents/axoloti/build/xpatch.map,--cref,--just-symbols=./build/axoloti.elf -o /Users/Simon/Documents/axoloti/build/xpatch.elf
BIN
arm-none-eabi-objcopy -O binary /Users/Simon/Documents/axoloti/build/xpatch.elf /Users/Simon/Documents/axoloti/build/xpatch.bin
Done compiling patch
Start uploading patch
bin path: /Users/Simon/Documents/axoloti/build/xpatch.bin
block uploaded @ 0x20011000 length 4116
Done uploading patch
Start starting patch
patch start taking too long, disconnecting
Disconnect request
Done starting patch
USB device found
connected
invalid CPU serial number, invalid protocol?, update firmware
Cannot obtain signature, upgrade firmware?
Ping: WaitSync Timeout, disconnecting now
Disconnect request
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
Ping: WaitSync Timeout, disconnecting now
USB device found
connected
java.lang.Exception: cpuserial has wrong length
java.lang.Exception: cpuserial has wrong length
at axoloti.HWSignature.Verify(HWSignature.java:96)
at axoloti.USBBulkConnection.connect(USBBulkConnection.java:316)
at axoloti.MainFrame.jCheckBoxConnectActionPerformed(MainFrame.java:651)
at axoloti.MainFrame.access$400(MainFrame.java:79)
at axoloti.MainFrame$6.actionPerformed(MainFrame.java:427)
at javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:2022)
at javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2348)
at javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:402)
at javax.swing.JToggleButton$ToggleButtonModel.setPressed(JToggleButton.java:308)
at javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:252)
at java.awt.Component.processMouseEvent(Component.java:6535)
at javax.swing.JComponent.processMouseEvent(JComponent.java:3324)
at java.awt.Component.processEvent(Component.java:6300)
at java.awt.Container.processEvent(Container.java:2236)
at java.awt.Component.dispatchEventImpl(Component.java:4891)
at java.awt.Container.dispatchEventImpl(Container.java:2294)
at java.awt.Component.dispatchEvent(Component.java:4713)
at java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4888)
at java.awt.LightweightDispatcher.processMouseEvent(Container.java:4525)
at java.awt.LightweightDispatcher.dispatchEvent(Container.java:4466)
at java.awt.Container.dispatchEventImpl(Container.java:2280)
at java.awt.Window.dispatchEventImpl(Window.java:2750)
at java.awt.Component.dispatchEvent(Component.java:4713)
at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:758)
at java.awt.EventQueue.access$500(EventQueue.java:97)
at java.awt.EventQueue$3.run(EventQueue.java:709)
at java.awt.EventQueue$3.run(EventQueue.java:703)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.awt.EventQueue$4.run(EventQueue.java:731)
at java.awt.EventQueue$4.run(EventQueue.java:729)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:76)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:728)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:201)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:116)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:105)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:101)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:93)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:82)

Ping: WaitSync Timeout, disconnecting now
Disconnect request
Ping: WaitSync Timeout, disconnecting now

Joren · 2016-03-21 08:48:32 UTC

Thanks for the report.I know that the object still needs some work. Especially when multiple are combined in a patch. The code needs to be updated so that it is more efficient and memory friendly. Certainly doable but it takes time.

The following is a link to a YIN implementation on the teensy, a less powerfull device. It could be give some inspiration about optimizations... https://github.com/PaulStoffregen/Audio/blob/master/analyze_notefreq.cpp

Blindsmyth · 2016-03-21 16:10:24 UTC

Hi Joren,

Thanks for the reply. Unfortunately I am not capabable of programming something like this but thanks for sharing this here anyway!
The problem with the object is that even with just one Yin axolot might not be able to load the patch. But I guess I have to wait until you or somebody else has the time to work on this.

Jan · 2016-03-21 20:43:38 UTC

This is interesting. How do these tests look like? E.g. unit tests running on an STM-Evaluation board?