Let's Design and Build a (mostly) Digital Theremin!

Anti-Aliasing ~= 1/x

It's fascinating reading recent papers on anti-aliasing oscillators.  Some of the best ones are written (or co-written) by Vesa Valimakai, and Jussi Pekonen shows up in many of the best as well.  Both sharp persons who write very clear papers and slides on the subject.

Unfortunately, it seems division, or reciprocal, shows up just about everywhere, regardless of method.  BLIT & BLEP rely on fractional delay filtering, and this requires the magnification of the error to fit the sampling period (1/F).  FIR & IIR comb filtering the aliasing away also require fractional delay.  DPW requires 1/F scaling to compensate for the differential gain.

Test

This has been a test of the TW twit filter.  All of my posts for the past month or so have been blocked and flagged for moderation.  Jason fixed it.

Free at last!

TANH Soft-Clipping

Found this quite fantastic paper: "New Perspectives on Distortion Synthesis for Virtual Analog Oscillators" by Victor Lazzarini and Joseph Timoney, published in Computer Music Journal, 34:1, pp. 28–40, Spring 2010.  One approach detailed in it soft-clips a sine wave via the tanh (hyperbolic tangent) function, which gives odd harmonics with two different roll-offs, the usual 1/f up to a point, then 1/(f^2) above that.  If you control the amplitude of the sine wave going into the clipper you can place this transition point around 10kHz, the effect of which is inaudible, and which nicely limits aliasing due to the 1/(f^2) roll-off up to Nyquist.  This gives odd harmonics - for all harmonics they mix the result with a cosine, which accentuates the second harmonic a bit.  By controlling the cosine mix they can conveniently control the odd / all harmonics ratio.

Difficulties with this approach are:
1. You need a high quality tanh function, which is difficult to do.
2. The sine gain function is 12000/(x * log(x)), which is difficult to do.
3. Reducing the sine gain below 1 to lower the harmonic content will significantly lower the output amplitude.
4. Reducing the sine gain below 1 to lower the harmonic content gives poor results when the output is mixed with cosine.

tanh(x) = (e^2x - 1) / (e^2x + 1) = (1 - e^-2x) / (1 + e^-2x)

This would take one exponentiation and one division, probably float, both of which are expensive.

Looked for papers on calculating tanh, none provided simpler solutions.  Tried polynomial fitting, which doesn't directly work well at all.  For 16 bit precision the input can be clipped to +/-6, for 32 bits +/-12, which reduces the solution space somewhat.  One approach I discovered via spreadsheet is to divide the input by a small constant, subtract it from 1, raise it to an integer power, then subtract this from 1.  Unfortunately this isn't accurate enough to avoid alias products.

Anyway, this has given me a new appreciation for "knee" type harmonic roll-off.  And I wonder if noise mixing pre-clipping would approximate the onset of vocalization better?  Something to try.  It wouldn't have occurred to me to attempt vocal dynamics via clipping, as the physics seem quite different.  And I'm not sure I want harmonic content so closely tied to amplitude, though it often is in nature.

Aliasing Effectively Vanquished! (way back in 1978 : US4249447)

I don't know why I keep thinking I can add anything fundamentally new to the research that's already been done regarding waveform generation.  People who had almost nothing in the way of processing horsepower wracked their brains and long ago mined out all of the simplest and most efficient and direct discoveries.  Spent a couple of weeks looking into soft clipping, tried many things, sadly with nothing of interest to report.  On Monday I was casting about listlessly on the old music DSP website (link) and encountered some code describing an old patented oscillator.  Found the patent and simulated three different forms detailed there (link).  Coded it up and have it running on the prototype.  WOW!  Pretty much exactly what I was looking for, a quasi band-limited version of my power-based phase modulation glottal generator.  Here is a diagram of what I've got running:

The phase increment value ("frequency") comes in at the upper left and gets routed two ways.  The lower way is fed to a second order polynomial, which gives us smaller numbers for larger phase increments.  The upper way is scaled to give C8 maximum and accumulated in the normal NCO fashion.  The accumulated phase forms a sawtooth, which is optionally modulo multiplied by 2 to double the frequency, and fed to two summing nodes, which in turn feed sine wave functions.  The lower sine wave is averaged for stability reasons (and to give a zero at Nyquist, or 24kHz), combined with a fraction (beta) of the polynomial result, and fed back to the phase inputs of the sine functions.

Beta controls the harmonic content, with 0 giving a pure sine wave, and 1 giving a 1/f harmonic roll-off (~sawtooth equivalent of 6dB / octave) and numbers in-between giving steeper slopes (we want ~12dB / octave nominal for vocal stuff).  If we do the frequency multiplication for the lower sine generator we get odd harmonics at the output (~square wave) with similarly variable harmonic content.  If you don't want just odd harmonics you can use a single sine function.  The full all / odd consumes 62 cycles max. on my Hive processor.

The beauty part though is the spectra have the secondary steeper "knee" heading into Nyquist, which really tames aliasing.  Here are some results from simulation which make that evident:

When beta = 1 there is a bit of ringing and rounding, but otherwise the waveform looks like a sawtooth.  The spectra shows a secondary steeper roll-off above 10kHz which massively controls aliasing fold-back.

Using the output sinewave itself as a feedback phase modifier is super simple and quite ingenious!  I've been avoiding all of these FM type solutions figuring they produced inferior spectra, were difficult to control, etc. but it doesn't really get much better or easier than this, particularly if you're looking for continuously variable harmonic content.

Is the knee audible?  When beta is dialed back I would say yes to some small degree as muffledness, though the knee ends up being many tens of dB below the fundamental, which masks it a lot.  At very low frequencies there is a bit of digital "burbling" going on, but it's pretty low level, and it seems all methods do this somewhat.  It's more noticeable when pitch correction / quantization is applied.

[EDIT] Here's a sample of the new oscillator in the prototype: (link).  Other than the marked lack of aliasing, and a more linear dynamic curve to the harmonic modulation input, I honestly can't tell any difference between it and the old oscillator.  My ears are kinda old, let me know if you hear any aliasing, the polynomial can be easily adjusted to fix various frequency bands.

The last main brick in the DSP wall!  (Leave those kids alone...)

[EDIT2] Tweaked the polynomial coefficients to minimize aliasing while maximizing harmonic content.  Using 0.44, 0.76, 0.54 now (L to R in the drawing above).  The "hump" out past 10kHz is now gone.

sounds good to me!
Btw, nice theremin-ey voice-ish sounds there near the end!
Now even the ugly bird dude should find it harder to complain.

"sounds good to me!"  - tinkeringdude

Thanks for listening!

"Btw, nice theremin-ey voice-ish sounds there near the end!"

Yah, though pretty much the same formant settings as before, plus a heapin' helpin' of Audition "studio reverb" (man don't never hear enough reverb...).  I mainly wanted to show that this oscillator can sub ~100% for the other one in the vocal department, which I believe it does.  

I think I'm hearing a faint bit of aliasing with beta jacked all the way up.  I've already jimmied with the polynomial a bit since that recording and will undoubtedly adjust it some more.  I've come to loathe aliasing, it's one of those "can't unhear it once you've heard it" kinds of things, where even a whiff of it triggers.

Speaking of triggering, at the end of that demo the high Q formants are "triggered" via the edge of a sub-audio sawtooth, something that happened accidentally one day, kinda neat sounding.

If you're not sure about ear reliability at the upper end - couldn't you see aliasing as "too high peaks in wrong places" in an FFT view, like if there are some at above -80 dB call it bad, or is there some gotcha I'm missing?

Btw, have you heard of the Novation Peak synth? It's said to use an FPGA to massively oversample (like 24 MHz sampling rate internally), and actually produce classic synth waveforms the naive way, and the Nyquist wall is so far away that hardly a thing makes its way back into audible range.
Brute force "because we can", well, and because mean effects like hard-sync are suddenly easy without alias, without a team of math PhDs to figure it out.
I don't know whether that was just rumor or it got confirmed somewhere.
The Commodore 64's SID was running things at 1 MHz, same principle. It was a little noisy, but kinda nice.

"If you're not sure about ear reliability at the upper end - couldn't you see aliasing as "too high peaks in wrong places" in an FFT view, like if there are some at above -80 dB call it bad, or is there some gotcha I'm missing?"  - tinkeringdude

This is an issue that is addressed best in Välimäki's papers.  He uses a masking function, I imagine rather like the ones used by lossy compression algorithms to throw away data that we can't hear, to evaluate aliasing numerically and objectively.  I can see the aliasing in a clunky one-frequency-at-a-time kind of way via simulation (FFT in Excel is low resolution and really slow), and in real-time via the FFT view in Audition (8k points max for real-time display, at least on my old PC).  What I see is the steeper reduction knee around 15kHz and the single reflections off of Nyquist before they take a dirt nap around 10kHz or so.  As one would expect it's obviously worst at maximum frequency (C8) and maximum harmonic level setting.  At Nyquist it's fairly low level, maybe 60dB below the fundamental.  I can't hear much above 12kHz so I don't know how this might be perceived by younger people.  (When I was a kid they had ultrasonic motion detectors mounted on store ceilings, I couldn't "hear" them in the usual sense but the beams would give me an instant headache, and I could always tell when one was near and pointed at me without looking.)

Also, I don't think it's my imagination, but having any "splatter" due to aliasing seems to interfere with the pitch field.  Maybe it's harmonics getting into the headphone cord?  I haven't done any controlled experiments, but when I adjusted the polynomial to get rid of the hump it seemed to get better.  I'm actually kind of shocked that I'm not having to deal with this sort of thing on a more regular basis.

"Btw, have you heard of the Novation Peak synth? It's said to use an FPGA to massively oversample (like 24 MHz sampling rate internally), and actually produce classic synth waveforms the naive way, and the Nyquist wall is so far away that hardly a thing makes its way back into audible range.
Brute force "because we can", well, and because mean effects like hard-sync are suddenly easy without alias, without a team of math PhDs to figure it out."

No, I hadn't heard of that, interesting and thanks for the pointer!  I see they're using a 4 line LCD as well, they're quite handy.

It's actually not all that difficult to make "classic" analog synth waveforms that are sufficiently band-limited, but constructs that give continuously variable harmonic content are harder (I only know of three, two of which have alias reduction).  I haven't played around with it, but reportedly one can do sync sounds as well without significant aliasing.  Once aliasing has been sufficiently tamed, I find the "burbling" low end to be more of a give-away.

I'm not sure why they went with analog filters, to me that's the really easy thing to do digitally, particularly if there is a bit of oversampling going on (the digital effects are reportedly running at >90kHz).  Though it seems the Moog 4th order (4x 1st orders w/ feedback) is a digital challenge.  But chaining two modified Chamberlins for 4th order works fine.  AFAIK, digital filters don't alias (though they can accentuate residual alias products, as can analog filters).

Hmmm. Ok, so the DSP people with their current oscillator designs build upon some psycho-acoustic "tricks" to get a way with some things which we can't hear directly - but what if further signal processing then makes things behave differently than from what the whole system would behave like if it actually was alias free? Is it really equivalent to an "alias free oscillator" then? Like, if you sell an oscillator component, and don't know what exactly people are going to do with the signal (funky non linear stuff)... not sure how else to put it... Am I making sense?

Ultrasonic, there you remind me of this weird thing, seems kinda Star Trek-ey, and the dev kit costs about as much as a "dilithium crystal", I imagine:
https://www.ultrahaptics.com/products-programs/touch-development-kit/

I'm not sure why they went with analog filters, to me that's the really easy thing to do digitally,

Easy, but only because there is an equivocation at play here
In analog synth world, when people say "VCF", they have certain ideas of what it does, and clinical filtering is only one aspect, kinda like literally amplifying is only a part of what an electric guitar's amp + speaker are about.

Have you used an analog, say MiniMoog ladder filter? And tortured it a bit?
(as for what it generally does, I guess you have seen the paper by Tim Stinchcombe? I can't tell how comprehensively it captures what's going on in what signal scenarios, too much greek, probably literally, heh)

Or a Steiner Synthacon VCF (SVF)? You can hear a modernized (stable even in self-oscillation) version in the Arturia MiniBrute - the first analog synth from Arturia, designed by known synth DIYer Yves Usson aka "YUSYNTH" - and apparenty since he wanted to give back / be oldschool, the entire schematic of that nicely nasty little SOB synth is public.
As the legend goes, the formerly only DSP based Arturia people, when they first invited Usson to demo some of his modules and see whether they'd want to hire him for the project, they quickly had a really big grin on their faces. Needless to say, there have been more Arturia analog synths out there since then.
https://www.youtube.com/watch?v=TzDjIBJ4tU8&t=1m10s

That's only some thing a found quickly, probably not the most comprehensive demonstration of what I'm trying to say.
And i't also more subtle / varied than this outright noisemaking here.

Or a Korg MS20 VCF.
When used "right", you can hear they are their own kind of beast, so an EE purist would probably be shocked how bad those circuits are, if "by the book" is the measure
Probably not a mystery, when you see diodes in an opamp feedback path it seems rather deliberate, whereas the Moog was more of an accident of miscalculation to produce gnarlyness.

So, one can say, ok, it filters, and has distortion. But that's not really all, to my humble ears anyway. Now speaking from playing with Steiner VCF and Moog VCF, there seems to be some complex interactions going on between those 2 aspects, and it gets wilder when you have funny fluctuations in the signal, e.g. complex patterns due to multiple modulation of partially very different (order of magnitude) rates, and the level of distortion and bending of filter characteristics would have non-obvious patterns, giving the whole thing some weird, alive seeming characteristic.

Probably least audible are such things when listening to a more tame, clean mid 80's poly synth from Roland, which have their own charm (and I might be wrong about because I bet there are people who can make it sound mean, but that's not what I associate with it, anyway)

---
This editor is... not fully cooperative...

"Hmmm. Ok, so the DSP people with their current oscillator designs build upon some psycho-acoustic "tricks" to get a way with some things which we can't hear directly - but what if further signal processing then makes things behave differently than from what the whole system would behave like if it actually was alias free? Is it really equivalent to an "alias free oscillator" then? Like, if you sell an oscillator component, and don't know what exactly people are going to do with the signal (funky non linear stuff)... not sure how else to put it... Am I making sense?"  - tinkeringdude

The more psycho-acoustic "trick" type approaches are low pass filtering (to reduce aliasing below the fundamental), comb filtering (to reduce aliasing between the harmonics) - both of these rely on masking (the ear being deaf to close frequencies that are lower in amplitude).  Others mimic higher rate generation and downsampling, rather like multi-rate filters that don't calculate intermediate results which will ultimately be discarded.  A couple of others, including the one I've finally picked, rely on secondary roll-off between the upper threshold of hearing and Nyquist to attenuate fold-back.

It's possible to do things like PWM and sync with base-band (i.e. non-oversampled) digital oscillators, I've read about this but haven't experimented along these lines.

I imagine the noise floor of an analog oscillator tends to white.  While the digital noise floor could be anything, and I suspect "perfect repetition" over many cycles that digital processes often do can lead to audible "tells".  And I imagine amplifying the noise floor could unmask aliasing in certain scenarios (though if they conform to something like the MP3 masking function like they are probably pretty insignificant).

"Ultrasonic, there you remind me of this weird thing, seems kinda Star Trek-ey, and the dev kit costs about as much as a "dilithium crystal", I imagine:
https://www.ultrahaptics.com/products-programs/touch-development-kit/"

~$5k USD, eek!  It must require a lot of power for one's fingers to feel the ultrasonics?

Reminds me of the "perfect speaker" from a couple of decades ago, where they heterodyned ultrasonics in the air with enough power to make the medium go non-linear.  It supposedly gives you private and localized audio sources, but I wonder what it does to the rest of the body long-term.

Easy, but only because there is an equivocation at play here 
In analog synth world, when people say "VCF", they have certain ideas of what it does, and clinical filtering is only one aspect, kinda like literally amplifying is only a part of what an electric guitar's amp + speaker are about.

Have you used an analog, say MiniMoog ladder filter? And tortured it a bit?
(as for what it generally does, I guess you have seen the paper by Tim Stinchcombe? I can't tell how comprehensively it captures what's going on in what signal scenarios, too much greek, probably literally, heh)

I see your points and very much appreciate them!  And I'll look at your links a bit later (the SO is hosting two piano recitals today and I'm the unhired hand). I haven't played around at all with modern analog synth filters, though long ago I built a second order state-variable as a guitar effect, and the digital filter in our AN-1x was really fun to control with the ribbon.  

Forcing a digital filter to do non-linear things would almost certainly cause aliasing, and perhaps this is the main reason they went with analog filters.

"This editor is... not fully cooperative..."

Yes, I've fallen back to copying and pasting from an external editor to get around all the unwanted formatting insertion.  There is a "reveal codes" button all the way to the right that allows one to clean up the remaining formatting dregs.  And drag & drop seems to work better than copy & paste.

I have to do this in Word all the time too, "Paste special..." "unformatted text" should be the default (but alas it isn't).  Word holds onto extraneous formatting like a Pit Bull on a steak...

[EDIT] OK, had some time this morning to look into this a bit and think about it.  I must say, the Moog ladder filter is one of the stranger analog circuits I've encountered, though I first saw a simpler version of the diode ladder in the PAIA low pass VCF schematic.  Not sure who invented it first, but it seems every synth designer more or less copied everyone else with just enough slight variation to avoid legal infringement / hype the difference.  I think I first saw it in this paper by Stinchcombe (link) which led me to this paper by Stilson & Smith (link) - digitizing the linear response is non-trivial, and digitizing the non-linear response probably even more problematic.  It's interesting that tanh shows up in both the analog analysis and in the paper I pointed to recently using tanh digitally to generate controlled harmonics.

It's kind of weird to me that simple analog synthesis is a "thing" in terms of sound.  I mean, everyone likes to play with knobs, and interactive filtering can be loads of fun, but the basic waveforms are really dull or harsh sounding to me, without a lot of in-between.  I suppose I've come to appreciate variable harmonic content not gotten by way of filtering, but by way of generation.  Don't get me wrong, I love electronica, ambient, dance music, etc. but I don't think what remains of my life would be any less rich if I never spent any more time playing with an analog synth or the DSP equivalent or some combination.  The thrill for me is mostly gone (and quite frankly was never all that big to begin with).

There are some things that are easy to do in analog that require a lot of study and effort to do in DSP, though with processors becoming almost free that trade-off is becoming more and more lopsided.  And there are some thing that are easy to do in DSP that are virtually impossible to do in analog (delay, stability, low noise floor, tons of virtual knobs, ease of manufacture, ease of complexity, etc.).  It's too bad that going from one domain to the other is rather difficult.