" Example extracted from the <strong>Signal Processing repository (COM303) by Paolo Prandoni</strong>, more on: <a href=\"https://github.com/prandoni/COM303\">https://github.com/prandoni/COM303</a>.\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Hearing the phase of a sound\n",
"\n",
"In this notebook we will investigate the effect of phase on the perceptual quality of a sound. It is often said that the human ear is largely insensitive to phase and that's why most of the equalization in commercial-grade audio equipment takes place in the magnitude domain only.\n",
"\n",
"But is it really so? Let's find out."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"%matplotlib inline\n",
"import matplotlib\n",
"import matplotlib.pyplot as plt\n",
"import numpy as np\n",
"import IPython\n",
"from scipy.io import wavfile"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"plt.rcParams[\"figure.figsize\"] = (14,4)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will be synthesizing audio clips so let's set the sampling rate for the rest of the notebook:"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"Fs = 16000 # sampling freqency\n",
"TWOPI = 2 * np.pi"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We will be synthesizing and playing audio clips so let's define a convenience function to \"beautify\" the resulting sound: basically, we want a gentle fade-in and fade-out to avoid abrupt \"clicks\" when the waveform begins and ends. \n",
"\n",
"Also, there is a \"bug\" in the current version of IPython whereby audio data is forcibly normalized prior to playing (see [here](https://github.com/ipython/ipython/issues/8608) for details; this may have been solved in the meantime). On the other hand, we want to avoid normalization in order to keep control over the volume of the sound. A way to do so is to make sure that all audio clips have at least one sample at a pre-defined maximum value, and this value is the same for all clips. To do so we add a slow \"tail\" to the data which will not result in an audible sound but will set a common maximum value in all clips."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"def prepare(x, max_value = 3):\n",
" N = len(x)\n",
" # fade-in and fade-out times max 0.2 seconds\n",
" tf = min(int(0.2 * Fs), int(0.1 * N))\n",
" for n in range(0, int(tf)):\n",
" s = float(n) / float(tf)\n",
" x[n] *= s\n",
" x[N-n-1] *= s\n",
" # let's append an anti-normalization tail; drawback is one second of silence in the end\n",
" x = np.concatenate((x, np.linspace(0, max_value, int(Fs/2)), np.linspace(max_value, 0, int(Fs/2))))\n",
" return x"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1) Sustained sounds\n",
"\n",
"The first experiment will use sustained sounds, i.e. waveform whose global envelope does not change in time. A periodic sustained waveform is simply the sum of harmonically-related sinusoidal components, i.e. a set of sines or cosines with different amplitudes and phases at frequencies multiple of a fundamental. The fundamental frequency is also known as the <i>pitch</i> of the sound.\n",
"\n",
"A class of instruments that produce good sustained sounds is the woodwinds (clarinet, saxophone, etc). The mechanics of sound generation in woodwinds are beyond the scope of this notebook and plenty of references can be found on the internet. For our purposes we will choose to simulate a clarinet according to the analysis described [here](http://www.phy.mtu.edu/~suits/clarinet.html).\n",
"\n",
"\n",
"\n",
"A clarinet-like sustained sound will contain frequencies at odd multiples of the fundamental. We will just keep the fundamental and five harmonics and we be able to specify the initial phase offset for each component:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"def clarinet(f, phase = []):\n",
" # length in seconds of audio clips\n",
" T = 3\n",
" \n",
" # we will keep 5 harmonics and the fundamental\n",
" Your browser does not support the audio element.\n",
" </audio>\n",
" "
],
"text/plain": [
"<IPython.lib.display.Audio object>"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# fundamental frequency: D4\n",
"D4 = 293.665\n",
"x = clarinet(D4)\n",
"\n",
"# let's look at the waveform, nice odd-harmonics shape: \n",
"plt.plot(x[0:300])\n",
"plt.show()\n",
"\n",
"# and of course we can play it (using our preparing function):\n",
"IPython.display.Audio(prepare(x), rate=Fs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Ok, so it's not the best clarinet sound in the universe but it's not bad for just a few lines of code. Now let's see how changing the phase affects the sound. Let's just use random phase offsets for the components: we can see that the waveform doesn't look too nice anymore:"
" Your browser does not support the audio element.\n",
" </audio>\n",
" "
],
"text/plain": [
"<IPython.lib.display.Audio object>"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# but if we play it, it sounds the same! \n",
"IPython.display.Audio(prepare(xrp), rate=Fs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OK, so it seems that phase is not important after all. To check once again, run the following notebook cell as many times as you want and see if you can tell the difference between the original zero-phase and a random-phase sustained note (the phases will be different every time you run the cell):"
"In the second experiment we will use real-world dynamic sounds, i.e. sounds that display time-varying characteristics. Typically, a physical musical instrument will produce sounds whose envelope displays four subsequent portions:\n",
"\n",
"* the **attack** time is the time taken for the sound to go from silence to max amplitude\n",
"* the **decay** time is the time taken for the sound to decrease to sustain level\n",
"* the **sustain** time is the time during the sound is kept at the same amplitude\n",
"* the **release** time is the time taken for sound to go to zero after the stimulation is stopped.\n",
"\n",
"\n",
"\n",
"Consider for instance a piano note: the attack time is very quick (the hammer hits the string); the decay is quite rapid as the string settles into harmonic equilibrium but there is no sustain since once the hammer hits, the stimulation ends. So a piano note has a distinct volume envelope that rises very fast and then releases slowly:"
" Your browser does not support the audio element.\n",
" </audio>\n",
" "
],
"text/plain": [
"<IPython.lib.display.Audio object>"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"from scipy.io import wavfile\n",
"Fs, x = wavfile.read(\"resources/piano.wav\")\n",
"plt.plot(x)\n",
"plt.show() \n",
"IPython.display.Audio(x, rate=Fs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"By now we know that the \"shape\" of a waveform is largely encoded in the phase. It is no surprise, therefore, that if we mess up with the phase of the piano sample above, we will get something that looks very different.\n",
"\n",
"To see this, let's take the DFT of the audio data, set the phase to zero and take the IDFT:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"# first some prep work; let's make sure that\n",
"# the length of the signal is even \n",
"# (it will be useful later)\n",
"if len(x) % 2 != 0:\n",
" x = x[:-1]\n",
"\n",
"# let's also store the maximum value for our \n",
"# the imaginary part is negligible, as expected, \n",
"# so let's just get rid of it\n",
"xzp = np.real(xzp)\n",
"\n",
"# and now we can plot:\n",
"plt.plot(xzp)\n",
"plt.show() "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Gee, what happened?!? Well, by removing the phase, we have destroyed the timing information that, for instance, made the sharp attack possible (mathematically, note that by creating a zero-phase spectrum we did obtain a symmetric signal in the time domain!).\n",
"\n",
"If we play the waveform, we can hear that the pitch and some of the timbral quality have been preserved (after all, the magnitude spectrum is the same), but the typical piano-like envelope has been lost."
"We can amuse ourselves with even more brutal phase mangling: let's for instance set a random phase for each DFT component. The only tricky thing here is that we need to preserve the Hermitian symmetry of the DFT in order to have a real-valued time-domain signal:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"1.6535271950038364e-06\n"
]
}
],
"source": [
"# we know the signal is even-length so we need to build\n",
"# a phase vector of the form [0 p1 p2 ... pM -pM ... -p2 -p1]\n",
"Pretty bad, eh? So, in conclusion, phase is very important to the temporal aspects of the sound, but not so important for sustained sounds. In fact, the brain processes the temporal and spectral cues of sound very differently: when we concentrate on attacks and sound envelope, the brain uses time-domain processing, whereas for pitch and timbre, it uses primarily the magnitude of the spectrum!"