Soundmagic Spectral


#21

since clouds is open source you might be able to figure that out pretty easy.

Another method that might be worth considering is using a combination of FM and additive synthesis, since FM produces interesting spectra which can be modelled with 1st order bessel functions. I'm envisioning a sort of "smart" additive resynthesis which uses non-sinusoidal waveforms combined in the following way: start with the spectrum you want to resynthesize, with harmonically related bands. Create an oscillator with fundamental frequency at the center of the lowest nonzero band, then subtract the pre-calculated spectrum for said oscillator from the original spectrum and repeat. said oscillators could be saws, triangles, and squares (I assume square would be the most efficient), and/or some simple FM signals. this would only really work if you're very careful about phase or do all your spectrum calculations calculations using complex numbers since complex spectra are linear and real spectra are not.

yet another tool you could use is to introduce noise equal to the minimum amplitude of any band and then reduce all bands by that amount, that way you get a guarantee of one band being 0 and a decent chance of a good number of bands being negligibly distant from 0, creating a sparse spectrum. Since sparse fft tells us that sparse spectra can be easier to compute, its fairly reasonable imo to assume the same can be said about the resynthesis of sparse spectra.

since spectra are linear barring phase issues, your design space for spectral resynthesis basically consists of using something to create enough of the partials you need and subtractive to cut out ones you don't. to get the partials you need, you can use any mix of fm, additive, wavetable, or noise synthesis.

spectral analysis isnt my main math squeeze so I'm unsure if such resynthesis has been attempted previously, whether it works, and whether some of the more specific mathematical statements above are correct

edit: added some stuff. check this paper on evolutionary FM/subtractive resynthesis of dynamic spectra out.


#22

They are all open source, so it should be possible to find out :slight_smile:


#23

That’s intriguing. I’m afraid, though I know some of the terminology, my grasp of the mathematical principles involved is tenuous at best.

In terms of which method would be most interesting, for me, it would depend on how much potential it offers for creative disruption/mangling, rather than necessarily how well it performs in terms of fidelity of reproduction.

a|x


#24

I'm reading mangling to mean unpredictable but interesting results, like blindly patching a modular drone. I think the most sonic possibilities will come from using FM, and you wont really need to know the special math for FM stuff since you can naively compute the spectra ahead of time (naive here means you can ignore the cool math and just make some preset fm waveforms and compute the spectra). However, if you use additive resynthesis from cheap waveforms you'll leave yourself as much dsp room as possible to do cool subtractive stuff on it, or things like wavefolding and pwm, which can still be set up in an unpredictable and creative way but without the high risk of stuff sounding bad that you get with FM. This is because for all the possible sounds from FM mangling, only some of them will produce harmonically simple enough waveforms to sound good.

I'll explain the math a bit for you. Usually we view audio signals as amplitude with respect to time. However, a combination of cool things said by old crazy nerds like euler, laplace, and fourier let us write those same functions as amplitude with respect to frequency. The coolest thing is that we have functions to convert between these 2 domains (domain here means the same thing it usually does, the x axis on your graph). spectral resynthesis is about finding easy and fast ways to convert from the frequency domain to the time domain. Instead of trying to mathematically convert from the frequency domain back to the time domain (inverse fourier transform) we synthesis (the resynthesis part) a wave that looks almost the same in the frequency domain (the spectral part). Since sine waves have only 1 frequency, they're the simplest way for humans to think about spectral resynthesis because you just have to make a sine wave for each point in your spectrum (graph where x is frequency and y is amplitude) and make the frequency of the sine wave equal the x coordinate of the point, and adjust its volume to the y coordinate of the point.

Unfortunately, sine waves are really hard for computers to make. Luckily, square waves arent. We can basically do the same thing as before, because square waves are just a bunch of odd harmonics with amplitude inversely proportional to frequency. In other words, a square wave at frequency f is sorta like a sin(f), and its a little more like a sin(f)+sin(3f)/3, and even more like a sin(f)+sin(3f)/3+sin(5f)/5, and so on. Because of this a square wave is very cheap for a computer to work with not only in the time domain, but in the frequency domain as well, since we can ignore all the harmonics higher than 20khz, so we have a pretty limited number of them to deal with and that number decreases as the frequency of the square wave increases. Note that I omitted some scaling factors for readability, for those factors just look up the square wave fourier series.

now I'll change the wording for how we think about spectral resynth. suppose our target spectrum is T(f), aka a function of frequency that we want to get, and that our resynthesized waveform has a spectrum of S(f). earlier we said we want T(f) = S(f), but if we subtract S(f) from both sides we get T(f)-S(f)=0. Even though we're in the frequency domain, spectra are still functions and pretty much act like it, so we can do a lot of the same stuff with spectra as we can regular functions.

The other thing we can do is say that if we add 2 simple waveforms, the spectrum of the sum is the same as the sum of the spectrum (i.e. the fourier transform is a linear transform).

If we think about our sine wave resynthesis again, what we're really doing is subtracting the spectra of a bunch of sine waves from our original spectrum in the hopes of getting close to 0. We can do the same thing with square waves, the only difference is that if we add a square wave at frequency f with amplitude A we need to subtract A/pi from the original spectrum at frequency f, A/3pi at frequency 3f, etc.

So to resynthesize a discrete spectrum (i.e. a spectrum with a finite number of points, where the x coordinate is the middle of a band and the y coordinate is the amplitude of said band) with square waves, all you have to do is look at your lowest band and make a square wave at the center frequency of that band, and adjust its volume to match the amplitude of that band. The only extra step is that you now have to subtract the higher partials before moving onto the next band, but the subtraction is easy to calculate and your next step is still to just add a square wave at the current lowest frequency band thats got an amplitude that isnt 0.

There are 2 things that complicate this, one of which matters and one of which doesnt, so I'll discuss the important one first.

the fourier series of a square wave isnt really what its spectrum is gonna look like, because it assumes the square wave goes from negative infinity to positive infinity on the time axis. What happens in real situations is that these peaks in the spectrum (called kronecker deltas, or places where the function jumps from 0 to something else and back to 0 instantly) get smoothed out and end up looking like normal distributions. that doesnt really matter since you can just run your square wave through the spectrum analyzer you're using ahead of time, and as long as you're comparing spectra that were produced by the same analyzer you should be ok.

the thing that doesnt matter for square waves but would matter if you wanted to try other waveforms is that when your x axis is frequency, your y axis is actually complex. That sounds scary, but think of it like this: your x axis (frequency) is now the middle of a cylinder. your amplitudes are vectors that point towards the wall of the cylinder, and the length of the vector is still the amplitude of that partial frequency, but the amount the arrow is rotated in the cylinder is the phase shift of that partial. if the arrow is pointing straight at you, it's got no phase shift. if its pointing away from you, then its got half a period of phase shift (pi radians or 180 degrees). if you phase shift a periodic signal by 360 degrees, you're basically spinning the cylinder around a full cycle, and all the arrows end up pointing in the same direction as they did before. The only thing this affects in our earlier model is how spectra are added and subtracted. say you have an arrow in a spectrum at frequency 220hz with amplitude 1, and its pointed towards you, so its not phase shifted. now say you add another spectrum with an identical arrow, but with 180 degree phase shift so it points the other way. if you add the spectra together (aka add the signals together) then the result is 0 at 220hz because the vectors cancel out, but since our earlier model didnt include phase shift it just reads 1+1 and spits out 2. luckily, this doesnt matter for square waves because all the partials of a square wave have no phase shift. there are other waveforms (in fact infinitely many) that also have this property, and for each one of those there are infinitely many other waveforms that don't have this property.

So we luckily dont have to worry about phase cancellation with perfectly tuned square waves, but the tradeoff is that if one of the square waves gets detuned we'll get an interference pattern. The solution to this is to sync all of the square waves to the lowest one, but that only works if all the square waves we use have a frequency thats an integer multiple of the frequency of the lowest square wave.

If you want a function written for you let me know, but understanding all this should give you enough to write the function and is honestly necessary to do anything interesting with it


#25

Wow, that's pretty comprehensive explanation, thank you very much, @watamacha..!

I have to say, it's going to take some digesting, but I'm working on it :wink:

I think I've got the bit about the importance of phase information resynthesising additively, though. I'd heard about this before, but now I have more of a grasp of why it's so important, in terms of phase cancellation effects.

I'm still vague on the bit about using square, instead of sine waves, I'm afraid. I can understand why they're easier to calculate than sines, and that if using them instead of sine waves, you'd need to remove the extra partials. What I'm still unclear on is how the partials are removed/filtered out, presumably in the time domain.

a|x


#26

Is this a possible approach, also?
http://mtg.upf.edu/node/111

a|x


#27

I think that the transform with squarish waveforms @watamacha is looking for is the Walsh Hadamar transform. Note that Walsh functions where used in early RMI Harmonic synth instead of sines, hence a bunch of characteristic overtones.

Calculating sines is an issue, but having sine functions in a table is the way to go.
In fact, the sine2t table contains a 4096 sample sine that is used in many objects.


#28

Interesting...

Isn't the issues with sine tables that the largest the phase-accumulator value, the 'steppier' the sine becomes, given a fixed sample-rate?

Maybe in practice, that's not an issue.

a|x


#29

I don't understand what you mean, can you draw something?
Tables are generally interpolated.


#30

Wow what discussion! I'm hyped but at the same time realizing that all of this is way beyond my capabilites. but it's still interesting though!

At the moment I'm getting used to the grainverb by @johannes again wich sounds pretty good actually. But what I'm missing compared to the spectral freeze or paul stretch is this smashing of transients and consonants into long noisy trails. Any Ideas how I could implement this in a cheap way that would complement the grainverb?
Some kind of vocoding/envelope followers with white noise as carrier maybe?


#31

I'm probably misunderstanding, but my thought was that as the phase accumulator increment gets higher, you'll end up skipping entries in the table, and the more entries are skipped the more approximate the waveform becomes, until eventually the resulting waveform becomes essentially random.

I must admit I haven't really got a grasp of the numbers involved, though. It could be that, with that table length, and the sample-rate the Axoloti runs at, the table will never skip values.

a|x


#32

Would these help?

http://www.keil.com/pack/doc/CMSIS/DSP/html/group__ComplexFFT.html
http://www.keil.com/pack/doc/CMSIS/DSP/html/group__RealFFT.html

And

a|x


#33

haha i know davey taylor from a LONG mail exchange, he is a brilliant programmer!


#34

Granular synthesis is probably the way to go. Have you tried the Clds object? It’s a port of the Mutable Instruments Clouds granular texture Eurorack module.

a|x


#35

As I said I'm using grainverb at the moment, but I disslike how it reacts on transients. The freeze and paul stretch really smash transients with long noisy tails, with the grainverb you rather here the tranient echoed a couple of times.
I will the try the clds, thx for the tip. Can it be used in this granular reverb fashion?


#36

You can use it in all kinds of ways. You'll have to fiddle with the settings to see what works.

There are lots of tutorials and demos of MI Clouds on YouTube. They will give you an idea of what it can do.

a|x


#37

Yes, that's what is called "aliasing", it is not random, but rather mirrored harmonics which cause high frequency inharmonics. It affects waveforms with lots of harmonics.

A pure sinewave only has one harmonic: itself.
Pure sinewave are not affected by this phenomenon (until it's frequency is above half the sampling rate).


#38

I'm saying leave the partials in, but recalculate the target spectrum by subtracting the spectrum of the last square wave you added.

check your first band, put in a square wave loud enough that subtracting the square's spectrum from the target spectrum yields 0. then move to the next nonzero band and do the same. if you end up with a band in the revised target spectrum thats less than 0, then when you get to it just apply a not gate to that square wave to phase it by 180 degrees so it sums to 0.

the approach you linked after does the same broad process (add up some signals so the sum of their spectra is equal to the target by subtracting each new signal you introduce from the target spectrum to get a residual) but instead of pure square waves they use a mix of sine waves (no overtones) and bandwidth limited white noise (equal amplitude at every frequency in its bandwidth).

you could put in a transient detector that gates a subtractive signal built on white noise, which works because transients are spectrally identical to noise. Another option would be to find a way to exaggerate the phasing effects of an all pass filter (traditionally used to produce imperceptible transient smoothing that preserves RMS power and decreases peak power, i.e. sounds just as loud but gives you more headroom on your transients)


#39

I'm aware of aliasing, Nyquist limit, etc.

Not quite what I meant, though. Something similar might happen if you had a waveform table that was being scanned through so fast that entries in the table were skipped. If too many samples are skipped, even with interpolation, the original waveform shape will be lost.

I guess that doesn't happen in practice, though.

a|x


#40

I think I found a pict that corresponds to what you mean: