fiduciary

Author Topic: smooth swing interpolation with pitch-shifted hums  (Read 14804 times)

0 Members and 1 Guest are viewing this topic.

Offline Thexter

  • Force User
  • ***
  • Posts: 148
  • Um...Hello?
smooth swing interpolation with pitch-shifted hums
« on: February 18, 2017, 12:25:44 PM »
Background and inspirations:
Remember the hiltworks project from years ago?
HiltWorks Project Demo

Well, ever since I found that old thread, I've been trying to do something similar.

This guy is also doing something similar, but from what I can tell, he's synthesizing his hum.
Lightsaber Tuning - YouTube

And just yesterday in Profezzorn's thread Lolwel21 came up with an interesting idea for smoothly interpolating between the hum and a looping swing sound.
Announcing the Teensy Saber open source sound board

This all ties into what I'm about to describe. I had the basics of this working before Christmas, but have been holding onto it, because like any artist, I'm always wanting to make things perfect before showing them to anyone.

Theory:

The actual lightsaber swing sounds in the movies are just Ben Burtt waving a microphone in front of a speaker playing the hum. There are two main effects you get form this:
  • You get a small doppler shift of the sound (2 to 3 semitones up or down)
  • You get a volume change as the microphone nears or retreats from the speaker

When I talk about pitch shifting, I'm talking about not just changing the sample rate, but the time-invariant pitch shifting where the length of the sound stays the same. While real-time pitch-shifting is elusive on the little MCUs we use, we can fake this by playing 3 sounds at once.

Sound 1: The original looping hum from your font (main)
Sound 2: The original looping hum, but pitch shifted a few semitones higher (high)
Sound 3: The original looping hum, but pitch shifted a few semitones lower (low)

I play all three sounds at once, but the relative gains on each are controlled by the motion of the saber. When the saber is still, you have the main hum at normal gain, and the other two have gain of 0. We want the relative volumes of the two pitch-shifted swing sounds to increase according to how hard we're swinging the saber. But how do we do this?

Observations:
First, here are a few observations I noticed when watching the movies:
  • Often a high-pitched swing will be followed by a lower-pitched swing. This makes sense. When swinging the microphone towards or away from the speaker, Ben Burtt can't keep swinging in the same direction. He has to swing towards the microphone and then away from it.
  • There is often an effect where a single swing goes low-high-low, or high-low-high (in terms of pitch). This is the case when swinging the microphone across a speaker.
  • Sometimes the swing of the saber doesn't actually change pitch in the movies, only volume. As far as I can discern, this is likely the case where the microphone was nearing the speaker without moving quickly enough to warrant a noticeable doppler shift.
  • Multiple swings in the same direction don't always produce the same pitched swing sound. This means you can't just say swinging left always lerps towards the high sound, or swinging right always lerps towards the low sound.

Implementation:
So, how do we mimic these observations?

The first approach I tried was to have a single secondary track and increase the gain on one of the sounds (and decrease the gain on the main hum accordingly) in relation to the gyro readings. However, this means you are stuck with one swing sound, and no real variance in pitch. I believe this is similar to the approach Lolwel21 described.

I also tried having a single secondary track that blended various high-pitched and low-pitched hums over time, and then increased the gain on that track with the gyro readings, but that just sounded rythmic and throbbing when you swung during swings due to the premixed swing track.

That's where the 3 track method came into play. Part of the method I came up with is to treat the axes of the gyro independently. Each axis still contributes to both the high and low pitched hum. This will give you a series of weights for each of the pitch-shifted hums. The  behavior of this is that for most cases you get the high-low alternation found in observations #1 and #2. The volume is controlled by the magnitude of the total angular velocity and is independent of the weights of the pitch shifted sounds. Because of this, there will be times where you have a swing, but the weights for each of the pitch-shifted hums ends up being zero. This solves the case where a swing may not affect the pitch (observation #3), but only the volume. Remember that we're modifying volume based on total angular velocity.

Great, but what about observation #4 above? With the method I just described, the saber would behave exactly the same for each identical swing. We don't want that. We want variance.

One solution is to allow the basis that you're using to measure the angular velocity to drift or rotate by itself. Right now, when you take gyro readings, you're taking them in the frame of reference of the gyroscope. X is x, y is y, z is z. If you change the frame of reference over some period (I found 2 or 3 seconds to work well for me), you get a nice variation in the swing pitch. This is especially cool when you are spinning and the effect varies and drifts over time.

Other considerations:
  • Obviously, only polyphonic fonts work with this method.
  • You can use the hum from another font as the pitch-shifted hums for more interesting effects. Be careful with this as it can also sound absolutely awful.
  • Clipping is a huge issue here. For dark-side fonts, this is OK, as they're supposed to be grittier, but for light-side fonts, I use a pass to increase the power of the pitch-shifted hum without modifying the absolute magnitude. It's the same thing they do for pop music. I suggest having good control over your secondary stream volume per-font.

I will post a video later once it uploads to youtube.

Offline Thexter

  • Force User
  • ***
  • Posts: 148
  • Um...Hello?
Re: smooth swing interpolation with pitch-shifted hums
« Reply #1 on: February 18, 2017, 01:52:27 PM »
As promised, here's the video:


Offline profezzorn

  • Mining Colony Members
  • Master Force User
  • *
  • Posts: 900
  • May the source be with you.
    • Hubbe's Corner
Re: smooth swing interpolation with pitch-shifted hums
« Reply #2 on: February 18, 2017, 03:32:46 PM »
Very cool stuff.
Do you already have an implementation of this that we can just put in TeensySaber?  What kind of sound board does the saber in the video use?

If not, I can probably put something together from the description here, but seeing whatever code you're using would be cool. :)

Offline JakeSoft

  • Experienced Force User
  • ****
  • Posts: 392
  • The Arduino Jedi
    • Universal Saber Library
Re: smooth swing interpolation with pitch-shifted hums
« Reply #3 on: February 18, 2017, 07:46:43 PM »
That's pretty stellar, dude. Wow.

Offline Thexter

  • Force User
  • ***
  • Posts: 148
  • Um...Hello?
Re: smooth swing interpolation with pitch-shifted hums
« Reply #4 on: February 18, 2017, 08:41:59 PM »
Do you already have an implementation of this that we can just put in TeensySaber?  What kind of sound board does the saber in the video use?

The one in the video is using a Teensy 3.2. Your codebase and mine share no common ancestor, so the code won't come over directly. Here are the relevant bits. I was originally calling it multi-hum, so there are still remnants of that nomenclature. Another thing to note is that this is the simplified version. It doesn't use any fixed-point math. As soon as I got my Teensy 3.5 and 3.6, I wrote this version up. It's much easier to understand, but probably slower on the 3.2 without an FPU.

Here is the relevant code when reading the gyro. The key is that both m_flSwingStrength and m_flMultiHumAmount are stored off for later sampling by main loop:
Code: [Select]
  // gx, gy, gz are all instantaneous gyro values
  // flTotalRotation is the total angular velocity of gx, gy, gz
  // m_flMultiHumCyclePeriod is actually poorly named and is really the inverse of the period.

  // Periodicity / Rotation of the basis
  float flCycleTime = ( nCurrentTime / 1000.0f ) * m_flMultiHumCyclePeriod;
  float flYAmount = sinf( flCycleTime );
  float flZAmount = cosf( flCycleTime );

  // Normalize to the swing sensitivity and clamp
  float flNormalizedY = max( -1.0f, min( 1.0f, gy / m_flMultiHumSensitivity ) );
  float flNormalizedZ = max( -1.0f, min( 1.0f, gz / m_flMultiHumSensitivity ) );
     
  // Apply the rotation
  float flMultiHumAmount0 = flYAmount * flNormalizedY;
  float flMultiHumAmount1 = flZAmount * flNormalizedZ;

  // Store off total strength
  m_flSwingStrength = min( 1.0f, flTotalRotation / g_flMultiHumSensitivity );

  // Multi-hum amount is the sum of both axes (both go from -1..1 range) divided by sqrt(2). Think of the case where both are 1.
  m_flMultiHumAmount = ( flMultiHumAmount0 + flMultiHumAmount1 ) / 1.414214f;

In the main loop, you can use these values to basically control the sounds. m_flMultiHumAmount is a -1 to 1 value where -1 is full strength for one of the pitch shifted sounds, and 1 is full strength for the other. 0 for m_flMultiHumAmount means 0 volume for the pitch-shifted swings. m_flSwingStrength is the overall swing strength.

Offline Sethski

  • Master Force User
  • *****
  • Posts: 1202
Re: smooth swing interpolation with pitch-shifted hums
« Reply #5 on: February 19, 2017, 04:32:47 AM »
This pretty epic! Ever since the Hiltworks stuff was shared here, it's stuck in my head as the ideal approach/solution to saber sound. With the recent exciting, evolutionary strides and contributions from talented folks with open source boards, I guessed something along these lines was back on the table. Really stoked to see this - great work and cheers for sharing  :azn:

Offline BEN KENOBI

  • Crazy Old Wizard
  • FX-SABERS VIP II
  • Master Force User
  • *
  • Posts: 8473
  • The saber scientist
Re: smooth swing interpolation with pitch-shifted hums
« Reply #6 on: February 19, 2017, 05:38:46 AM »
That's really cool!! Love the hums.

Offline jbkuma

  • Mining Colony Members
  • Master Force User
  • *
  • Posts: 937
  • Pixels, everywhere.
    • Mad Science Workshoppe
Re: smooth swing interpolation with pitch-shifted hums
« Reply #7 on: February 19, 2017, 08:23:51 AM »
Wow, that's just incredible Thexter! By far the best swing sounds I've seen. This would work great with fonts like Darkblade which are very "noisy" in the canon material.

Offline Obi_1

  • Mining Colony Members
  • Experienced Force User
  • *
  • Posts: 476
  • Creator of DIYino - first open source FX-board
Re: smooth swing interpolation with pitch-shifted hums
« Reply #8 on: February 20, 2017, 12:11:14 AM »
Thexter, for this terrific motion altered hum mixing you earned an eminent place in the DIY Saber builder's Hall of Fame!!! Such a hum pitch shift is more realistic than the swing sounds, and as a further advantage, all the swing sounds could be eliminated, replacing them with 1-2 additional hum sounds (or even have only one and alter the pith of that, it already gives accurate results), making sound fonts smaller in size (going for flash memory  :wink: ).

I personally think that what Thexter describes here is the last piece in a puzzle to finally reach break-even with CF using a DIY saber board. Good times!

Offline Don

  • Force User
  • ***
  • Posts: 117
    • Ludosport Lightsaber Combat Academy
Re: smooth swing interpolation with pitch-shifted hums
« Reply #9 on: February 20, 2017, 02:10:54 AM »
Wow, this is a really impressive.
The way the sound shifts according to the movement of the saber sounds so "natural".
I believe this is one of the biggest groundbreaking developments we have seen since long time.

I definitely need to find the time to complete my saber and start playing with my DIYino.
Master Sabersmith at the Ludosport Lightsaber Combat Academy Italy

Offline WookieCookie

  • No Force
  • *
  • Posts: 28
  • Um...Hello?
Re: smooth swing interpolation with pitch-shifted hums
« Reply #10 on: February 20, 2017, 09:19:42 AM »
It's times like these that I wish I was more code savvy.  :embarrassed:

Offline Lolwel21

  • No Force
  • *
  • Posts: 9
  • Bridging mechanical and organic
Re: smooth swing interpolation with pitch-shifted hums
« Reply #11 on: February 22, 2017, 06:31:28 PM »
What audio library do you use? (May I use it?)

Offline Thexter

  • Force User
  • ***
  • Posts: 148
  • Um...Hello?
Re: smooth swing interpolation with pitch-shifted hums
« Reply #12 on: February 22, 2017, 07:09:05 PM »
I'm just using the Teensy Audio library that comes with the teensyduino software, so I have to upconvert all of the sounds to 44khz. There's an option to compile the audio library for lower sample rates (eg 22khz), but I've not done that yet.

Also, and I apologize if this is obvious, make sure to enable the Teensy sd card optimizations if you're using the 3.2, or you won't be able to read more than 1 file at a time on many sd cards.

Offline profezzorn

  • Mining Colony Members
  • Master Force User
  • *
  • Posts: 900
  • May the source be with you.
    • Hubbe's Corner
Re: smooth swing interpolation with pitch-shifted hums
« Reply #13 on: February 22, 2017, 07:56:32 PM »
I've never managed to get the teensy optimizations to work.
However, TeensySaber has better buffering code than the teensy audio library, which is why I can still play a several wav files at a time.
(I have some updates to the code that I need to post though...)

Offline profezzorn

  • Mining Colony Members
  • Master Force User
  • *
  • Posts: 900
  • May the source be with you.
    • Hubbe's Corner
Re: smooth swing interpolation with pitch-shifted hums
« Reply #14 on: February 26, 2017, 10:52:22 PM »
Thexter, have you tried this simplified code?
Does it work the same as the fixed-math code?
I'm curious, because it doesn't seem to match your description very well.

Some comments on the code:

  // Apply the rotation
  float flMultiHumAmount0 = flYAmount * flNormalizedY;
  float flMultiHumAmount1 = flZAmount * flNormalizedZ;

This is not actually a rotation. A rotation would be something like:
  rotate_x = x * cos - y * sin
  rotate_y = x * sin + y * cos


  m_flMultiHumAmount = ( flMultiHumAmount0 + flMultiHumAmount1 ) / 1.414214f;

Shouldn't this be / 2.0 ? As is, m_flMultiHumAmount can be greater than 1 or less than -1.

I thought two of the hums were supposed to be controlled independently by the rotated x/z axis?

Anyways, I'll try implement this the way the code says and see what that gets me. :)