ARM != ARM
There are many different versions of ARM processors. ARM Ltd. design and license designs for processor cores to manufacturers. The manufacturers then add their own peripherals (ADC, memory controllers, serial I/O, I2S, I2C, graphics processors, etc. etc.) and create the actual chips.
There are chips for (kind of) low-level embedded systems (e.g. STM32F4 used in the Axoloti) which have rich I/O but lower clock speeds, no caches, simpler pipelines, no/simple branch prediction, etc. The benefit is that they generally have lower latency and higher predictability. Also they are somewhat less volatile than their high-performance counterparts but also the processing power is lower.
And then there are advanced chips that you find in the raspberry pi and mobile phones - ranging from single core to octocore designs with multiple levels of cache and everything you would want to run a proper operating system like linux or android. With this dramatic increase in performance comes higher latencies and poor predictability but also increased software complexity.
Between those variants there are HUGE differences, even though they are all called "ARM". Porting Axoloti to a "Raspberry Pi"- class ARM processor is basically pointless as they are just so much more tailored to run a "real" operating system. While it is technically possible to run code "bare metal" on a raspberry pi (that means: without any "real" operating system like linux) it needs a GIGANTIC amount of work to make that happen. And why hassle with device drivers when there are proper linux kernels with all that onboard?
If you want to run audio stuff on a raspberry pi, it is much better to use something like PureData and use a generic USB soundcard and USB MIDI interface on it. That way, whenever a new version of the pi comes out, you can just swap the board out and run the very same application on the new device without having to worry about all the device drivers and low level stuff (as that is all done by the linux operating system already).
Basically that is what operating systems are all about: Abstract the underlying hardware so much, that swapping the hardware makes absolutely no difference for the software running on it.
The benefit of Axoloti is its low latency, fast boot times and the simplicity of the hardware that just "works". As soon as you start increasing the processing power significantly, there will be wishes for new features. Because, you know with all that new processing power, why not add video output to it? See where this is going? Ultimately it all leads to a full operating system. And then you could just use linux and PureData (or any other software).