Homec4science
Diffusion epfl (master) Browse

R2653/teaching/COM418/booklet/time_frequency_analysis.tex
9f36f3e17ae0master

Last Change
Raw File

time_frequency_analysis.tex
View Options

\chapter{Time-frequency analysis}
Time-frequency analysis (TF) is the domain of signal processing that analyses a signal over time and frequency simultaneously. It contrasts with techniques that deal with only time or frequency at once, like for instance the frequency response of a signal which gives no information about its evolution over time. TF methods hence deal with 2-dimensional signals. The main motivation behind TF methods is that standard Fourier transform assumes that the signal is periodic over time. However, most audio signals are not. TF methods hence generally decompose the signal over time in windows using the STFT, and apply a similar treatment to each window before merging them back together.
\section{Modifying pitch and time}
One important application of TF methods in music are modifying the pitch and duration of an audio signal by a chosen ratio $r$. The challenge here is that the pitch and the duration of a music are connected. Extending the duration of an audio signal genuinely will result in a lower pitch, like playing a vinyl record at a lower speed than it is supposed to. Oppositely, speeding up an audio signal with raise its pitch. Sometimes although, it is needed to modify the signal duration without altering the pitch, and oppositely. Different techniques hence exist to tackle each of these needs.\\
\begin{figure}[p]
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=.7\textwidth]{figures/guitar_STFT.png}
\end{minipage}
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=.7\textwidth]{figures/pitch_control.png}
\end{minipage}
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=.7\textwidth]{figures/time_stretch.png}
\end{minipage}
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=.7\textwidth]{figures/time_scale.png}
\end{minipage}
\caption[Caption for LOF]%
{An example signal's STFT, first without transformation, and then with pitch control with ratio $r=0.6$. time stretch with ratio $r=0.6$, and pitch scaling with ratio $r=1.5$.}
\label{pitch_time_STFT}
\end{figure}
\textbf{Pitch control} is the simplest TF transformation, consisting in genuinely altering the pitch and the duration together, as it naturally occurs. This simply done by digitally resampling the signal at a different sample rate, and then playing it back as if it was the original sample rate. The ratio of change $r$ is the same for the duration and the pitch. For instance, by taking a signal sampled at 44.1KHz, resampling it at 88.2KHz, and then playing it back at 44.1KHz will result in a new signal two times longer and two times (one octave) lower in pitch.\\
On figure \ref{pitch_time_STFT}, on can observe the plotted STFT of a signal, before and after going through a pitch control transformation. After pitch control, one can notice that both the speed and the pitch of signal are altered, the signal gets hence vertically and horizontally stretched on the plot. Note that the stretch however occurs in a inverse fashion on each axis.\\
The \textbf{pitch scaling} transformation transposes the frequencies of an audio signal while leaving the duration unchanged. Oppositely, \textbf{time stretching} changes the duration of the audio signal without altering its pitch. Often, only time stretching is implemented from scratch, and pitch scaling is obtained by a combination of pitch control and time stretching. To this end, one simply pitch control the signal by ratio $r$ to obtain the desired pitch (yet with modified duration), and then time stretches the signal with the inverse ratio $1/r$ in order to meet back the original duration. Note that if the pitch must be increased, applying pitch control first and time stretch second results in a faster computation. Oppositely, when the pitch must be lowered, one should use time stretch first and pitch control second.\\
For instance, to increase the pitch of a signal lasting 1 second by a ratio of $3/2$, one first pitch control the signal by a ratio of $3/2$, resulting in a signal of length $2/3$ seconds, and with pitch $3/3$ times the original. one then stretches the signal with ratio $2/3$ and obtain a signal of length 1 second again. On figure \ref{pitch_time_STFT}, one can see the effect of time stretching and pitch scaling on the signal. This time, the signal is only altered vertically, or horizontally, while keeping the other axis preserved. \\
All these transformations can be implemented in time-domain and frequency-domain. Below, we overview two of the most popular TF methods for changing pitch and duration: OLA, and the phase vocoder.
\subsection{OLA}
OLA (OverLap Add) is a time-domain method for changing pitch and duration. The main idea of OLA and its derivatives are to cut the signal into overlapping chunks in time domain, and to merge these chunks back together with a different overlapping size. If the new overlapping size is greater (if the chunks overlap more than at the cutting phase), the signal will have a shorter duration, and inversely. Naive OLA uses a fixed window size, and a fixed time between windows. However this introduces echoing, humming artefacts and discontinuities in the sound.\\
One improved version of OLA, called \textbf{SOLA} (Synchronous OverLap Add) computes the cross-correlation between the current window and a fixed-size set of next window candidates, that differ in duration and start times. The exact set of candidates is to be determined by the algorithm itself, and many variants exist. The candidate with the highest cross-correlation with the current window has the highest similarity, hence reducing artifacts when merging the windows back together. One can see in figure \ref{sola} the window selection phase of the algorithm, before adding the windows back together in the output signal.\footnote{See \url{https://www.surina.net/article/time-and-pitch-scaling.html} for a short introduction.}
\begin{figure}[h]
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=.6\textwidth]{figures/sola.png}
\caption[Caption for LOF]%
{SOLA's windows selection\protect\footnotemark}.
\label{sola}
\end{minipage}
\end{figure}
\footnotetext{Diagram by Olli Parviainen, from \url{https://www.surina.net/article/time-and-pitch-scaling.html}.}
\subsection{Phase vocoder}
The phase vocoder is a vocoder (see section \ref{vocoder}) that works in the frequency-domain for changing pitch and duration. The main idea of the phase vocoder is that it first converts the audio signal into its STFT representation, using a chosen window size $w_{in}$ and a chosen hop size $h_{in}$. It then applies a transformation to all the different STFT chunks in the frequency domain, and finally converts them back to a time domain audio signal using a window size $w_{out}$ and a hop size $h_{out}$\cite{phase_vocoder}. \\
\begin{figure}[h]
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=\textwidth]{figures/phase_vocoder.png}
\caption[Caption for LOF]%
{A simple phase vocoder pipeline\protect\footnotemark.}
\label{phase_voc}
\end{minipage}
\end{figure}
\footnotetext{From the MATLAB documentation: \url{https://www.mathworks.com/help/audio/ug/pitch-shifting-and-time-dilation-using-a-phase-vocoder-in-matlab.html;jsessionid=aa5af2fbc428e67e28bd7759265c}.}
A simple yet efficient use of a phase vocoder, shown in figure \ref{phase_voc}, achieves time stretching by setting a different input and output overlap and hop size, similarly to the OLA method. The difference however is that the phase vocoder can change the phase of each chuck in the frequency domain, and make it match the phase of the previous chunk. This results in the sound being smoother and free of discontinuities, achieving better sound quality than OLA and its derivatives. Pitch scaling is then obtain by combining time stretching and pitch control, as usual.\\
An easy way to make the phases of 2 consecutive chucks match, the phase vocoder keeps a record of the global phase since the beginning of the audio signal. It then computes the difference between a chunk phase and the previous chunk phase, and each time adds it to the global phase. Finally, it replaces the chunk's phase with the global phase. Note that this is done in parallel and independently for each frequency bin of the STFT.\\
If a STFT of window size $K$ samples is run on a signal $x[n]$, the $m$\textsuperscript{th} has length $K$ samples and is converted into a Fourier Transform with $K$ frequency bins spanning from $0$ to $2\pi$. The $k$\textsuperscript{th} bin can be written as
\begin{align*}
X[m, k] = r[m,k] \cdot e^{j \phi[m,k] }
\end{align*}
where $r$ is the modulus of the bin, and $\phi$ its phase. The phase vocoder starts by setting the global phase to zero, $\phi_g[m,k]= 0$. Then, it analyzes the first chunk and computes its phase difference with the previous (non-existing) chunk's, which is said to be 0. The first chunk hence keeps its phase $\phi[1,k]$ for all $k$, which also becomes the global phase. When analyzing the second chunk, the phase vocoder computes the difference $\phi[2, k] - \phi[1,k]$ for each $k$ in parallel, and adds this to the global phase. We hence obtain the equation of the global phase for chunk $m$
\begin{align*}
\phi_g[1, k] &= \phi[1, k] , \forall k \\
\phi_g[m,k] &= \phi_g[m-1, k] + \phi[m,k] - \phi[m-1,k], \forall k, m>1
\end{align*}
When then convert back to time-domain and obtain chunks
\begin{align*}
X_g[m, k] = r[m,k] \cdot e^{j \phi_g[m,k] }
\end{align*}
before overlap-adding them together with the new hop size.
\section{Other phase vocoder effects}
Using different modifications of the phase and modulus of the frequency bins in the phase vocoder, one can achieve different aesthetic effects.
\subsection{Robotization}
By setting all the phase of all bins to $0$, the signal gets rid of all information about the pitch,
\begin{align*}
\phi[m,k] &= 0, \forall k, m .
\end{align*}
Only the envelope is kept through the modulus of the signal, which sounds monotonic, like a robot.
\subsection{Whisperization}
Similarly to the robotization, the whisperization gets totally rid of the phase information. It replaces the phase of each bin of each chunk with a random value uniformly drawn in the range $[0, 2 \pi)$, as
\begin{align*}
\phi[m,k] &= \mathcal{U}(0, 2\pi) , \forall k, m .
\end{align*}
Hence, the phase becomes pure noise and the voice sounds like it is whispered.
\subsection{Autotune}
Autotune is an audio processor first designed by Dr. Andy Hildebrand and then trademarked by Antares Audio Technologies from 1997 as a VST effect (see the chapter dedicated to VSTs), which allows to correct the pitch of a voice track, note after note. By default, Autotune automatically pitch scales each sung note to the closest semi-tone, making sure that all notes are in key. Although several implementation of Autotune exist, the phase vocoder is an efficient one.
\begin{figure}[h]
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=.7\textwidth]{figures/autotune.png}
\caption[Caption for LOF]%
{Autotune's VST interface.}
\label{}
\end{minipage}
\end{figure}
Autotune features several parameters, such as the attack time of the pitch correction. This parameter is meant to make the pitch sound more natural, as the human voice typically needs around 100 to 500ms to find the right pitch after starting to sing a note. Setting a shorter attack time (around 10ms) makes the voice sound unnaturally precise and sharp when switching note. This sound however became popular in the late 1990s as the artist \textit{Cher} released the song "Believe", featuring a desired obvious use of short attack Autotune \footnote{See the Youtube clip \url{https://youtu.be/nZXRV4MezEw?t=43}.}. Later on, short attack would become a standard voice effect in genres such as Hip-Hop, or Latino Pop, with key artists such as \textit{T-Pain}\footnote{\url{https://youtu.be/rxRvDpF2FDA?t=50}}, \textit{Akon}\footnote{\url{https://youtu.be/bKDdT_nyP54?t=53}}, or \textit{Kanye West}\footnote{\url{https://youtu.be/KEA0btSNkpw?t=285}}. A more discrete, non-perceptible Autotune with a longer attack is used on virtually every music produced today.
\section{Channel vocoder}\label{vocoder}
A \textbf{vocoder} in the broad sense is a tool that allows analyse the human voice (analysis phase), compress it based on its features, and then to recreate it (synthesis phase). Optionally, some transformation can be done during the process to obtain a different voice at synthesis. It is mostly used in telecommunications for compression purposes, where the synthesized voice is meant to be as similar as possible to the original voice. During the analysis phase, it decomposes the input voice in a given number of frequency bands. It then extracts information from each band (the needed information depends on the type of vocoder). Finally, it combines this information with possibly new signals for each band at the synthesis phase, and mix up all the bands back together. The phase vocoder that was presented before indeed follows this scheme.\\
The \textbf{channel vocoder} is a different type of vocoder that also has applications in music. It is a musical effect that allows to combine a speech signal's envelope with any audio signal as a carrier. The resulting sound keeps the intelligibility of the speech while having the carrier's timbre and pitch. The sound of the channel vocoder is quite recognizable and relatively popular in electronic music, see such examples as \url{https://www.youtube.com/watch?v=jPxxW9b76_s}.
\subsection{Brief history}
The vocoder was initially exclusively used in radio transmission for compression purposes, and is still used in mobile communication today. However, from the 1940s, researchers started having interest in applying this technology to music instruments. While attempts were made in the 1950s, the first successful vocoder design was developed in 1968 by Robert Moog (him again) and was implemented using solid-state circuitry. In 1970, Moog included his vocoder to a Moog synthesizer, using the synthesizer sound as the carrier, and a microphone for the modulator. It was hence possible to speak while playing the keyboard, producing a voice pitched like what was played. Later implementations of the vocoder would often follow the idea of Moog, as shown in figure \ref{channel_vocoder}. While never being central in music, the channel vocoder gained some popularity from the late 1970s as a futuristic effect, mainly in electronic, dance, and hip-hop music. Popular artists include \textit{Kraftwerk}\footnote{\url{https://youtu.be/wHEoMpMvz7A?t=106}}, \textit{Kavinsky}\footnote{\url{https://youtu.be/MV_3Dpw-BRY?t=31}}, or \textit{Daft Punk}\footnote{\url{https://youtu.be/yydNF8tuVmU?t=122}}.
\begin{figure}[h]
\begin{minipage}{\textwidth}
\centering
\includegraphics[width=.5\textwidth]{figures/vocoder.jpg}
\caption[Caption for LOF]%
{Waldorf STVC synthesizer and vocoder.}
\label{channel_vocoder}
\end{minipage}
\end{figure}
\subsection{Implementation}
The channel vocoder is a time-domain vocoder, by opposition to the phase vocoder. It first splits the modulator and the carrier signals in bands using a bank of bandpass filters. The bands have generally logarithmic increasing width, to fit the human perception of pitch. The envelope of the modulator is then computed using a series of a rectifier, keeping only the positive values of the signal, followed by a lowpass filter (a moving average), to smooth out the global shape of the envelope. The reason for the smoothing is that the envelope has a lower frequency than the signal itself. Then, each envelope is applied to its respective band from the carrier signal, by simply multiplicating both signals together. Sometimes, Gaussian white noise can be added to the carrier for each channel before filtering by the bandpass. This is supposed to make the sound crispier, as white noise is a good model for consonants sounds such as /s/. Note that a different white noise must be used for each channel to avoid correlation. The bands are finally summed up back together. A synthesizer generally allows to select among several oscillators for the carrier. The implementation is summarized in figure \ref{vocoder_diagram}.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\begin{figure}[h]
\center
\begin{dspBlocks}{.4}{.3}
modulator~~ & \BDsplit & \BDbandpass & \BDabs & \BDlowpass[.45em] & \BDmul && \BDadd & ~~out\\
&& \BDbandpass & \BDabs & \BDlowpass[.45em] && \BDmul && \\
&&&&& \BDbandpass & \BDbandpass &\\
carrier~~ &&&&& \BDsplit &&&
\ncline{1,1}{1,2} \ncline{->}{1,2}{1,3} \ncline{->}{1,3}{1,4} \ncline{->}{1,4}{1,5} \ncline{->}{1,5}{1,6} \ncline{->}{1,6}{1,8} \ncline{->}{1,8}{1,9}
\ncline{1,2}{2,2} \ncline{->}{2,2}{2,3} \ncline{->}{2,3}{2,4} \ncline{->}{2,4}{2,5} \ncline{->}{2,5}{2,7} \ncline{2,7}{2,8} \ncline{->}{2,8}{1,8}
\ncline{4,1}{4,6} \ncline{4,6}{4,7} \ncline{->}{4,6}{3,6} \ncline{->}{4,7}{3,7} \ncline{->}{3,6}{1,6} \ncline{->}{3,7}{2,7}
\end{dspBlocks}
\caption{Block diagram of a simple 2-band channel vocoder.}
\label{vocoder_diagram}
\end{figure}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{Exercises}
c4science · Help