# Logistic Regression

## From Regression to Classification

In the previous chapter we discussed linear regression and its application. It is a very valuable tool because

* it is easy to fit
* has simple interpretation and
* tests of statistical significance can be performed without difficulty.

But it has its drawbacks too. For example: it makes quite strong assumptions about the functional form of $f(X)$ and the error terms. Another disadvantage is that linear regression is not suited for classification problems where we deal with qualitative responses.

Below figure, where we regress the probability of default onto credit card balance, visualizes the issue.

* Linear regression produces negative probabilities for some balances
* Due to the large number of non-defaults, the linear regression's probability of default barely rises above 25%. This begs the question where we should set the threshold to qualify a default case.

Logistic regression is able to overcome these obstacles. Rather than modeling the default class $y$ (e.g. 0 for non-default, 1 for default) it models the (conditional) probability that $y$ belongs to a certain category, $\Pr(Y=y|X)$.

## Logistic Model

In above figure we display our attempt to estimated the probability of default given one feature, i.e. a client's credit card balance. In the same way we could have used a multiple linear regression (with multiple features as input) to estimate the probability.

$$\begin{equation}
\Pr(Y=y|X) = \beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p
\end{equation}$$

But as discussed above for the simple case, this too would yield unbounded probabilities with $y > 1$ and $y<0$ for some $X$ (for convenience let us assume that we are using the generic 0/1 coding for the dependent variable). The question thus is: how could we improve on that model such that we have bounded results with probabilities of default? A well known function which satisfies the inequality $0 \leq f(x) \leq 1$ (and is continuous and hence differentiable) is the so called "Sigmoid" or "Logistic" function which is defined as

$$\begin{equation}
S(x) = \frac{1}{1 + e^{-x}} = \frac{e^x}{1 + e^x}
\end{equation}$$

Above figure displays the function's shape. Its range is bounded by $[0, 1]$ and its notable "S" shape depends on the input parameter $x$. For logistic regression $x$ takes on the known functional form of a linear regression:

$$\begin{equation}
\Pr(Y=y|X) = \frac{e^{\beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p}}{1 + e^{\beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p}}
\end{equation}$$

This function is related to the linear model in that by rearranging the equation we arrive at

$$\begin{equation*}
\frac{p(X)}{1 - p(X)} = e^{\beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p}
\end{equation*}$$

Notice that for ease of use we define $p(X)$ as equivalent of $\Pr(Y=y|X)$. Taking the natural logarithm of above expression yields

$$\begin{equation}
\underbrace{\log\left(\frac{p(X)}{1 - p(X)}\right)}_{\textit{Logit}} = \beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p,
\end{equation}$$

a function where the output on the left-hand side, the so called **Logit** or **log-odds**, is linear in $X$.

## Estimating the Coefficients

To run a logistic regression we need to estimate the $\beta$ coefficients in the following equation:

$$\begin{equation}
\Pr(Y=y|X) = \frac{e^{\beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p}}{1 + e^{\beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p}}
\end{equation}$$

In the linear regression setting we used a least squares approach. Here, however, we switch to a method called **maximum likelihood**. This approach is used in many areas of statistics/machine learning to fit models and as such often studied in graduate courses. Here we will restrict ourselves to a very superficial discussion of the intuition. For the interested reader, [Elkan (2014)](http://cseweb.ucsd.edu/~elkan/250B/logreg.pdf) provides an excellent, concise introduction, whereas Bishop (2006) has a more advanced and thorough discussion of the topic.

Back to the intuition: maximum likelihood optimizes the above equation to get estimates of $\beta$ "such that the predicted probability $\hat{p}(x_i)$ of default for each individual [$\ldots$] corresponds as closely as possible to the individual's observed default status. In other words, we try to find $\hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_p$ such that plugging these estimates into the model for $p(X)$ [$\ldots$] yields a number close to one for all individuals who defaulted, and a number close to zero for all individuals who did not." (James et al. (2013, p. 133)).

The maximum likelihood function that is used to estimate the $\beta$ coefficients in the logistic regression model is derived from the probability distribution of the dependent variable $y$. Since $y$ takes on one of two values (in our example $y \in \{0, 1\}$) and assuming the response values are independent of each other, the probability mass function of $Y$ follows a Bernoulli distribution, $Y \sim Bern(p)$, which is a special case of the Binomial distribution $Bin(n, p)$ with $n=1$. Its probability mass function is

\begin{equation*}
f(y;p) = p(Y=y) = p^y (1-p)^{1-y} =
\begin{cases}
p & \text{ if } y=1 \\
(1-p) & \text{ if } y=0
\end{cases}
\end{equation*}

The joint probability mass function of $Y$ is

\begin{equation*}
f(y | \beta) = \prod_{i=1}^N p(x_i)^{y_i} \, (1-p(x_i))^{1-y_i}
\end{equation*}

and describes the values of $y$ as a function of known, fixed values for $\beta$, where $\beta$ is related to $y$ through the logistic function. Since we don't know the coefficients $\beta$ but have measured outcomes for $y$ the likelihood function reverses above joint probability mass function such that it expresses the values of $\beta$ in terms of known, fixed values for $y$. This is the likelihood function ([Czepiel (2002)](https://czep.net/stat/mlelr.pdf)).

\begin{equation}
L(\beta | y) = \prod_{i=1}^N p(x_i)^{y_i} \, (1-p(x_i))^{1-y_i}
\end{equation}

The maximum likelihood estimates are now those values for $\beta$ which maximize the likelihood function $L(\beta | y)$. Typically, to find the maximum likelihood estimates we would differentiate the (log-) likelihood with respect to the coefficients, set the derivatives equal to zero, and solve. However, there is no closed-form solution for this and thus numerical methods (such as Newton-Raphson, Newton-conjugent gradient etc.) are required to derive a maxima ([Shalizi (2017)](http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/ADAfaEPoV.pdf)).

Below clip I found to be an excellent introduction into the details of how the log-likelihood is derived with maximum likelihood.

```python
from IPython.display import YouTubeVideo

YouTubeVideo('TM1lijyQnaI')
```

Thankfully, statistical packages such as Python's `statsmodels` have the necessary tools and functions integrated into the relevant functions such that we do not need to concern ourselves with the details.

## Logistic Regression in Python

### Package Selection

To show how logistic regression is run in Python we will again rely on functions from the `statsmodels` package. The Scikit-learn package `sklearn` contains similar functions. However, when it comes to calling results, the `statsmodels` functions for logistic regression follow those for linear regression which we got to know in the previous chapter. Therefore we will work with this package.

### Data Load

We will use the generic 'Default' data set from James et al. (2013), which we discussed in this chapter's introduction. It is taken from the book's corresponding `R` package and made available in this course's data folder on GitHub. We start our journey as usual with the initial load of the necessary packages and setting a few options.

```python
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
import statsmodels.formula.api as smf
plt.style.use('seaborn-whitegrid')
```

```python
# Default data set is not available online. Data was extracted from R package "ISLR"
df = pd.read_csv('Data/Default.csv', sep=',')

# Factorize 'No' and 'Yes' in columns 'default' and 'student'
df['defaultFac'] = df.default.factorize()[0]
df['studentFac'] = df.student.factorize()[0]
df.head(5)
```

      default
      student
      balance
      income
      defaultFac
      studentFac

      0
      No
      No
      729.526495
      44361.625074
      0
      0

      1
      No
      Yes
      817.180407
      12106.134700
      0
      1

      2
      No
      No
      1073.549164
      31767.138947
      0
      0

      3
      No
      No
      529.250605
      35704.493935
      0
      0

      4
      No
      No
      785.655883
      38463.495879
      0
      0

### Simple Logistic Regression

Similar to linear regression with `statsmodels` there exist different ways to compute a logistic regression. For reference, we will show three approaches here:

1. R-Style: The `statsmodels.formula.api` library has a `glm` function mimicking `R`'s way of running glm regressions
2. Classic GLM: Using the `GLM` function from the `statsmodels.api` library
3. Logit regression: `statsmodels.api`'s `Logit` function follows the approach we used for OLS case

You might ask what GLM/glm means. It stands for *Generlized Linear Models* and as such is a generalization of linear regression approaches (linear, logistic, Poisson regression). There are many good resources available on the web for the interested. A good textbook introducing GLM is Alan Agresti's *Foundations of Linear and Generalized Linear Models* (Agresti (2015)).

#### R-Style

* This method allows for a verbose definition of the regression.
* It follows the pattern `y ~ X1 + X2 + ... + Xn` - in words we regress $y$ on $X_1, X_2, \ldots, X_n$ where $X_i$ is a vector of a particular feature.
* By default a constant is added to the regression
* `family` is a necessary argument. Remember that above, where we discussed the maximum likelihood approach of estimating the coefficients, we mentioned that the binary response $y$ follows a Bernoulli distribution, which is a special case of a Binomial distribution. Thus we chose `family=sm.families.Binomial()`.
* The `.fit()` method runs the maximum likelihood estimation of the coefficients. Default method is 'Newton-Raphson' (method=`newton`) and this fits our needs. Nonetheless, others are available. Use the help function `sm.Logit.fit?` to see what options you have (it's the same for all three approaches).

More details can be found [here](http://www.statsmodels.org/dev/example_formulas.html)

```python
# 1. R-Style

# R-Style formula
formula = 'defaultFac ~ balance'

# Regress model
logReg = smf.glm(formula=formula, data=df, family=sm.families.Binomial()).fit()
print(logReg.summary())
```

                     Generalized Linear Model Regression Results
    ==============================================================================
    Dep. Variable:             defaultFac   No. Observations:                10000
    Model:                            GLM   Df Residuals:                     9998
    Model Family:                Binomial   Df Model:                            1
    Link Function:                  logit   Scale:                          1.0000
    Method:                          IRLS   Log-Likelihood:                -798.23
    Date:                Wed, 13 Jan 2021   Deviance:                       1596.5
    Time:                        14:21:35   Pearson chi2:                 7.15e+03
    No. Iterations:                     9
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    Intercept    -10.6513      0.361    -29.491      0.000     -11.359      -9.943
    balance        0.0055      0.000     24.952      0.000       0.005       0.006
    ==============================================================================

#### Classic GLM

* `endog` = endogenous variable = response = $y$
* `exog` = exogenous variable(s) = features = $X$
* Notice that a constant needs to be added manually with `sm.add_constant(X)`

Function details are described [here](http://www.statsmodels.org/dev/generated/statsmodels.genmod.generalized_linear_model.GLM.html#statsmodels.genmod.generalized_linear_model.GLM).

```python
# 2. GLM

# Regress model
logReg = sm.GLM(endog=df.defaultFac, exog=sm.add_constant(df.balance),
                family=sm.families.Binomial()).fit()
print(logReg.summary())
```

                     Generalized Linear Model Regression Results
    ==============================================================================
    Dep. Variable:             defaultFac   No. Observations:                10000
    Model:                            GLM   Df Residuals:                     9998
    Model Family:                Binomial   Df Model:                            1
    Link Function:                  logit   Scale:                          1.0000
    Method:                          IRLS   Log-Likelihood:                -798.23
    Date:                Wed, 13 Jan 2021   Deviance:                       1596.5
    Time:                        14:21:35   Pearson chi2:                 7.15e+03
    No. Iterations:                     9
    Covariance Type:            nonrobust
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const        -10.6513      0.361    -29.491      0.000     -11.359      -9.943
    balance        0.0055      0.000     24.952      0.000       0.005       0.006
    ==============================================================================

#### Logit Regression

* `endog` = endogenous variable = response = $y$
* `exog` = exogenous variable(s) = features = $X$
* Notice that a constant needs to be added manually with `sm.add_constant(X)`
* No need to specify `family` (Logit is per default binomial)

The full function details are to be found [here](http://www.statsmodels.org/dev/generated/statsmodels.discrete.discrete_model.Logit.html)

```python
# 3. Logit

# Regress model
logReg = sm.Logit(endog=df.defaultFac, exog=sm.add_constant(df.balance)).fit()
print(logReg.summary())
```

    Optimization terminated successfully.
             Current function value: 0.079823
             Iterations 10
                               Logit Regression Results
    ==============================================================================
    Dep. Variable:             defaultFac   No. Observations:                10000
    Model:                          Logit   Df Residuals:                     9998
    Method:                           MLE   Df Model:                            1
    Date:                Wed, 13 Jan 2021   Pseudo R-squ.:                  0.4534
    Time:                        14:21:35   Log-Likelihood:                -798.23
    converged:                       True   LL-Null:                       -1460.3
    Covariance Type:            nonrobust   LLR p-value:                6.233e-290
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const        -10.6513      0.361    -29.491      0.000     -11.359      -9.943
    balance        0.0055      0.000     24.952      0.000       0.005       0.006
    ==============================================================================

    Possibly complete quasi-separation: A fraction 0.13 of observations can be
    perfectly predicted. This might indicate that there is complete
    quasi-separation. In this case some parameters will not be identified.

### Accessing Output

Though the scope of information in the summaries differ slightly between the different approaches the function yield the same results. As in the case of linear regression, the model's output can be accessed through its different methods and attributes. For example the models parameters and p-values are easily accessed as shown below.

```python
logReg.params
```

    const     -10.651331
    balance     0.005499
    dtype: float64

```python
logReg.pvalues
```

    const      3.723665e-191
    balance    2.010855e-137
    dtype: float64

### Hypothesis Testing

The $z$-statistic in above summary and the corresponding p-values play the same role as the $t$-statistic in the linear regression output. For example, the $z$-statistic associated with $\hat{\beta}_1$ is equal to $\hat{\beta}_1 / SE(\hat{\beta}_1)$. You can check this by running `logReg.params[1] / logReg.bse[1]`. A large (absolute) $z$-value, and correspondingly a small p-value, provides evidence against the null hypothesis $H_0 : \beta_1 = 0$. Notice that this null hypothesis implies that $p(X) = \dfrac{e^{\beta_0}}{(1 + e^{\beta_0})}$.

The main purpose of the intercept is to adjust the average fitted probabilities to the proportion of ones in the data. Beyond that it is not of interest.

If we are interested in the coefficient's 99% CI (instead of 95% like in the summary) we can use the same code as in the linear regression case.

```python
logReg.conf_int(alpha=0.01)
```

      0
      1

      const
      -11.581640
      -9.721022

      balance
      0.004931
      0.006067

### Making Predictions

If we wish to make predictions of the default probability we can use the `.predict()` method. Given we leave the brackets empty, this method will calculate $p(X)$ on the basis of the previously used feature training sample. If we wish to get the probability of default for a balance of, let us say, USD 2'000 we can run the following line of code:

```python
# X must include 1 in first column for intercept
logReg.predict([1, 2000])
```

    array([0.58576937])

The vector $[1, 2000]$ in the above `.predict()` function corresponds to a row vector containing the $x$-values for which we wish to predict the probability of default. If we had $p$ features, this vector would obviously have lenght $p+1$ ($p$ features + 1 for intercept).

For future reference, below code generates a confusion matrix. We will discuss confusion matrices in more detail in the chapter on $k$-Nearest Neighbor. The `threshold` parameter defines above what probability a new observation is labeled/predicted as 1 (default); if no value is given, the function reverts to its default settings which is `threshold=0.5`.

```python
# Confusion Matrix for LogRegression
logReg.pred_table(threshold=0.5)
```

    array([[9625.,   42.],
           [ 233.,  100.]])

### Plotting

Above we regressed default on balance. Thus this can still be displayed in a 2d-figure. In order to do so we need to sort the data first. Once this is done, plotting follows the usual routine.

```python
# Create sorted resutls-df
res = pd.DataFrame()
res['balance'] = df.balance
res['prob'] = logReg.predict()
res = res.sort_values('balance')

# Plot scatter & logReg
plt.figure(figsize=(12, 8))
plt.scatter(df.balance, df.defaultFac, marker='.')
plt.plot(res.balance, res.prob, c='k')
plt.axhline(y=0, color='gray', linestyle='dashed')
plt.axhline(y=1, color='gray', linestyle='dashed')
plt.ylabel('Probability of default', fontsize=12)
plt.xlabel('Balance', fontsize=12);
```

Here is an alternative way of plotting it. Notice that for the logistic function I do not use the data I trained it with but 'any possible' outcome.

```python
# Create sorted resutls-df
res = pd.DataFrame()
res['balance'] = np.linspace(start=0, stop=5000, num=10000)
res['prob'] = logReg.predict(sm.add_constant(res['balance']))

# Plot scatter & logReg
plt.figure(figsize=(12, 8))
plt.scatter(df.balance, df.defaultFac, marker='.')
plt.plot(res.balance, res.prob, c='k')
plt.axhline(y=0, color='gray', linestyle='dashed')
plt.axhline(y=1, color='gray', linestyle='dashed')
plt.ylabel('Probability of default', fontsize=12)
plt.xlabel('Balance', fontsize=12);
```

### Multiple Logistic Regression

Regressing 'default' on multiple features follows the same procedure as for simple logistic regression. As a case of use we regress 'default' on 'balance', a credit card holder's 'income' and the dummy variable 'studentFac' (1 if applicant is a student, 0 otherwise).

```python
# Assign features to matrix X and response to y
X = sm.add_constant(df[['balance', 'income', 'studentFac']])
X.income = X.income / 1000  # Income in 1'000 dollars
y = df.defaultFac

# Run Logistic Regression
logReg = sm.Logit(endog=y, exog=X).fit()
print(logReg.summary().tables[1])
```

    Optimization terminated successfully.
             Current function value: 0.078577
             Iterations 10
    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    const        -10.8690      0.492    -22.079      0.000     -11.834      -9.904
    balance        0.0057      0.000     24.737      0.000       0.005       0.006
    income         0.0030      0.008      0.370      0.712      -0.013       0.019
    studentFac    -0.6468      0.236     -2.738      0.006      -1.110      -0.184
    ==============================================================================

The p-value for `income` is surprisingly large, indicatign that there is no clear evidence of a real association between the probability of default and income. In a case study this certainly would need to be further examined.

### Multiple Logistic Regression with Scikit-learn

For reference we show below how logistic regression is run with Scikit-learn. Unfortunately, there is no summary output available in Scikit-learn like there is within the Statsmodels package. The main reason is that sklearn is not used for statistical inference but for predictive modelling/ML and the evaluation criteria are based on performance on previously unseen data.

Notice that in order to have similar coefficients/results, the hyperparameter `C` has to be set to `C=1e9`. This parameter is a regularization parameter that refers to the L1/L2 penalty scheme implemented. For details refer to Scikit-learn's [user guide](http://scikit-learn.org/stable/modules/linear_model.html#logistic-regression), [this discussion on Stackoverflow](https://stackoverflow.com/questions/43431236/how-to-perform-an-unregularized-logistic-regression-using-scikit-learn) or chapter 6.2 in James et al. (2013).

```python
from sklearn.linear_model import LogisticRegression
logReg = LogisticRegression(C=1e9, fit_intercept=False).fit(X, y)
```

```python
logReg.coef_
```

    array([[-1.08690136e+01,  5.73652683e-03,  3.03186192e-03,
            -6.46798865e-01]])

### Decision Boundary

We've seen above how one can plot a logistic regression in the simple case with just one feature. If you regress a response on two features, it is still possible to create very meaningful plots. A good example is [this answer on stackoverflow](https://stackoverflow.com/questions/28256058/plotting-decision-boundary-of-logistic-regression) where the decision boundary is drawn and the color scheme follows the probability of the response.

## Closing Comments
In closing this chapter it should be mentioned that the preceding code sections are by no means a thorough analysis but a superficial digging into the topics. Yet the tools and measures necessary to understand the output are yet to be introduced. For this we refer to the chapter on linear discriminant analysis (LDA) and quadratic discriminant analysis (QDA) (and subsequent chapters) where many tools are introduced that are of significant help in analyzing an algorithm's output.

# Further Ressources

In writing this notebook, many ressources were consulted. For internet ressources links are provided within the textflow above. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

* Agresti, Alan, 2015, *Foundations of Linear and Generalized Linear Models* (John Wiley & Sons, Hoboken, NJ).
* Bishop, Christopher M., 2006, *Pattern Recognition and Machine Learning* (Springer, New York, NY).
* Czepiel, Scott A., 2002, Maximum Likelihood Estimation of Logistic Regression Models: Theory and Implementation from website, https://czep.net/stat/mlelr.pdf, 08/24/17.
* Elkan, Charles, 2014, Maximum Likelihood, Logistic Regression, and Stochastic Gradient Training from website, http://cseweb.ucsd.edu/~elkan/250B/logreg.pdf, 08/24/17.
* James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, *An Introduction to Statistical Learning: With Applications in R* (Springer Science & Business Media, New York, NY).
* Shalizi, Cosma Rohilla, 2017, Advanced Data Analysis from an Elementary Point of View from website, http://www.stat.cmu.edu/~cshalizi/ADAfaEPoV/ADAfaEPoV.pdf, 08/24/17.


# The Discrete-Time Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Definition

The [discrete-time Fourier transform](https://en.wikipedia.org/wiki/Discrete-time_Fourier_transform) (DTFT) of a discrete signal $x[k]$ is defined as

\begin{equation}
X(e^{j \Omega}) = \sum_{k = -\infty}^{\infty} x[k] \, e^{-j \Omega k}
\end{equation}

where $\Omega \in \mathbb{R}$ denotes the normalized angular frequency. The DTFT maps a discrete signal $x[k]$ with $k \in \mathbb{Z}$ onto its continuous transform $X(e^{j \Omega}) = \mathcal{F}_* \{ x[k] \}$ with $\Omega \in \mathbb{R}$. It is frequently termed as *spectrum* of the discrete signal $x[k]$. The argument $e^{j \Omega}$ copes for the fact that the kernel $e^{-j \Omega k}$ of the DTFT is a [complex exponential signal](../discrete_signals/standard_signals.ipynb#Complex-Exponential-Signal) $z^{-k}$ with $z = e^{j \Omega}$. Other common definitions of the DTFT may be found in the literature. They differ with respect to the sign of the exponential function and normalization factors. The properties, theorems and transforms may differ from the ones given here, as a consequence.

A sufficient but not necessary condition for the existence of the DTFT is

\begin{equation}
\left|X(e^{j \Omega})\right| = \left| \sum_{k = -\infty}^{\infty} x[k] \, e^{-j \Omega k} \right|
\leq \sum_{k = -\infty}^{\infty} \left| x[k] \right| < \infty
\end{equation}

where the upper bound results from the [triangle inequality](https://en.wikipedia.org/wiki/Triangle_inequality). As sufficient condition for the existence of the DTFT, it follows that a signal $x[k]$ needs to be absolutely summable.

The DTFT is periodic with a period of $T_\text{p} = 2 \pi$,

\begin{equation}
X(e^{j \Omega}) = X(e^{j (\Omega + n \cdot 2 \pi)})
\end{equation}

for $n \in \mathbb{Z}$. This follows from the periodicity of its exponential kernel $e^{j \Omega k} = e^{j (\Omega + 2 \pi) k}$ for discrete $k \in \mathbb{Z}$. It is therefore sufficient to regard the DTFT in one period only. Typically the period $-\pi < \Omega < \pi$ is chosen. The information on the discrete signal $x[k]$ is contained in one period. As a consequence, the inverse discrete-time Fourier transform $x[k] = \mathcal{F}_*^{-1} \{ X(e^{j \Omega}) \}$ is defined as

\begin{equation}
x[k] = \frac{1}{2 \pi} \int_{-\pi}^{\pi} X(e^{j \Omega}) \, e^{j \Omega k} \; d \Omega
\end{equation}

### Transformation of the Dirac Impulse

The transform $\mathcal{F}_* \{ \delta[k] \}$ of the [Dirac impulse](../discrete_signals/standard_signals.ipynb#Dirac-Impulse) is derived by introducing $\delta[k]$ into the definition of the DTFT and exploiting the sifting property of the Dirac impulse

\begin{equation}
\mathcal{F}_* \{ \delta[k] \} = \sum_{k = -\infty}^{\infty} \delta[k] \, e^{-j \Omega k} = 1
\end{equation}

The transform of the Dirac impulse is equal to one. Hence, all normalized frequencies $\Omega$ are present with equal weight. This is an important property in the theory of discrete signals and systems, since the Dirac impulse is used to characterize linear time-invariant (LTI) systems by their [impulse response](../discrete_systems/impulse_response.ipynb) $h[k] = \mathcal{H} \{ \delta[k] \}$.

**Exercise**

* Derive the DTFT of a shifted Dirac impulse $\delta[k - \kappa]$.

### Transformation of the Rectangular Signal

The DTFT $X(e^{j \Omega}) = \mathcal{F}_* \{ \text{rect}_N[k] \}$ of the [rectangular signal](../discrete_signals/standard_signals.ipynb#Rectangular-Signal) is derived by introducing its definition into the definition of the DTFT. This results in

\begin{equation}
\mathcal{F}_* \{ \text{rect}[k] \} = \sum_{k = -\infty}^{\infty} \text{rect}[k] \, e^{-j \Omega k} =
\sum_{k = 0}^{N-1} e^{-j \Omega k} = e^{-j \Omega \frac{N-1}{2}} \cdot \frac{\sin \left(\frac{N \Omega}{2} \right)}{\sin \left( \frac{\Omega}{2} \right)}
\end{equation}

The latter equality has been derived by noting that the sum constitutes a [finite geometrical series](https://en.wikipedia.org/wiki/Geometric_series) with the common ratio $e^{-j \Omega}$. Note, that

\begin{equation}
\frac{\sin \left(\frac{N \Omega}{2} \right)}{\sin \left( \frac{\Omega}{2} \right)} \bigg\rvert_{\Omega = n \cdot 2 \pi} = N
\end{equation}

for $n \in \mathbb{Z}$ due to [L'Hôpital's rule](https://en.wikipedia.org/wiki/L'H%C3%B4pital's_rule).

**Example**

The DTFT $X(e^{j \Omega}) = \mathcal{F}_* \{ \text{rect}_N[k] \}$ of the rectangular signal is computed for a specific length $N$ by evaluating the finite series above using `SymPy`.

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

W = sym.symbols('Omega', real=True)
k = sym.symbols('k', integer=True)

N = 5
X = sym.summation(sym.exp(-sym.I*W*k), (k, 0, N-1))
X
```

The magnitude $|X(e^{j \Omega})|$ and phase $\varphi(e^{j \Omega}) = \arg \{ X(e^{j \Omega}) \}$ of the spectrum is plotted for illustration

```python
sym.plot(sym.Abs(X), (W, -3*sym.pi, 3*sym.pi), xlabel=r'$\Omega$', ylabel=r'$| X(e^{j \Omega}) |$')
sym.plot(sym.arg(X), (W, -3*sym.pi, 3*sym.pi), xlabel=r'$\Omega$', ylabel=r'$\varphi(e^{j \Omega})$');
```

**Exercise**

* What happens to the magnitude/phase if you increase/decrease the length $N$ of the rectangular signal?

### Transformation of the Exponential Signal

The DTFT $X(e^{j \Omega}) = \mathcal{F}_* \{ e^{j \Omega_0 k} \}$ of the [harmonic exponential signal](../discrete_signals/standard_signals.ipynb#Complex-Exponential-Signal) with normalized frequency $\Omega_0$ is derived by introducing it into the definition of the DTFT. This results in

\begin{equation}
\mathcal{F}_* \{ e^{j \Omega_0 k} \} = \sum_{k = -\infty}^{\infty} e^{j \Omega_0 k} \, e^{-j \Omega k} =
\sum_{k = -\infty}^{\infty} e^{-j (\Omega - \Omega_0) k} = {\bot \!\! \bot \!\! \bot}\left( \frac{\Omega - \Omega_0}{2 \pi} \right)
\end{equation}

where for the last equality the [Fourier transform of the Dirac comb](../periodic_signals/spectrum.ipynb#The-Dirac-Comb) has been used. The DTFT of the exponential signal is a periodic series of shifted Dirac impulses, as

\begin{equation}
{\bot \!\! \bot \!\! \bot}\left( \frac{\Omega - \Omega_0}{2 \pi} \right) = 2 \pi \sum_{\mu = -\infty}^{\infty} \delta(\Omega - \Omega_0 - 2 \pi \mu)
\end{equation}

### Relation to the z-Transform

The DTFT $X(e^{j \Omega}) = \mathcal{F}_* \{ x[k] \}$ of a signal $x[k]$ can be related to its [two-sided $z$-transform](../z_transform/definition.ipynb#Two-Sided-$z$-Transform) $X(z) = \mathcal{Z} \{ x[k] \}$ by inspecting the kernels of both transforms. The $z$-transform has the complex exponential function $z^{-k}$ with $z \in \mathbb{C}$ as kernel. The DTFT has the harmonic exponential function $e^{- j \Omega k}$ with $\Omega \in \mathbb{R}$ as kernel. Both can be related to each other by considering that $z = e^{\Sigma + j \Omega}$. Hence, if the ROC includes the unit circle $|z| = 1$ of the $z$-plane, the DTFT of a signal $x[k]$ can be derived from its $z$-transform by

\begin{equation}
\mathcal{F}_* \{ x[k] \} = \mathcal{Z} \{ x[k] \} \bigr\rvert_{z = e^{j \Omega}}
\end{equation}

If the ROC does not include the unit circle, the DTFT of a given signal does not exist. A benefit of the $z$-transform over the DTFT is that it can be applied to a wider class of signals.

**Example**

The DTFT $X(e^{j \Omega}) = \mathcal{F}_* \{ x[k] \}$ of the causal exponential signal

\begin{equation}
x[k] = \epsilon[k] \cdot e^{- \Sigma_0 k}
\end{equation}

for $\Sigma_0 \in \mathbb{R}^+$ is derived from the $z$-transform of the [causal complex exponential signal](#Transformation-of-the-Causal-Complex-Exponential-Signal). Using the substituting $z_0 = e^{-\Sigma_0}$ yields

\begin{equation}
X(z) = \frac{z}{z - e^{-\Sigma_0}}
\end{equation}

with the ROC $|z| > e^{-\Sigma_0}$. The ROC includes the unit circle for $0 < \Sigma_0 < \infty$. The DTFT can be derived from the $z$-transform by substituting $z$ with $e^{j \Omega}$ as

\begin{equation}
X(e^{j \Omega}) = \frac{e^{j \Omega}}{e^{j \Omega} - e^{-\Sigma_0}}
\end{equation}

### Link to Fourier Transform of a Sampled Signal

The link between the Fourier transform of a sampled signal $x_\text{s}(t)$ and the DTFT of its discrete counterpart $x[k] = x(k T)$ is established in the following. Under the assumption of [ideal sampling](../sampling/ideal.ipynb#Model-of-Ideal-Sampling), the sampled signal reads

\begin{equation}
x_\text{s}(t) = \sum_{k = -\infty}^{\infty} x(k T) \cdot \delta(t - k T) = \sum_{k = -\infty}^{\infty} x[k] \cdot \delta(t - k T)
\end{equation}

where $x(t)$ denotes the continuous signal and $T$ the sampling interval. Introducing the sampled signal into the [definition of the Fourier transform](../fourier_transform/definition.ipynb) yields the spectrum of the sampled signal

\begin{equation}
X_\text{s}(j \omega) = \int_{-\infty}^{\infty} \sum_{k = -\infty}^{\infty} x[k] \cdot \delta(t - k T) \, e^{-j \omega t} \; dt = \sum_{k = -\infty}^{\infty} x[k] \, e^{-j \omega k T}
\end{equation}

where the last equality has been derived by changing the order of summation/integration and exploiting the [sifting property of the Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse). Comparison with the definition of the DTFT yields

\begin{equation}
X_\text{s}(j \omega) = X(e^{j \Omega}) \big\rvert_{\Omega = \omega T}
\end{equation}

The spectrum of the sampled signal $X_\text{s}(j \omega)$ is equal to the DTFT of the discrete Signal $X(e^{j \Omega})$ for $\Omega = \omega T$. This result can be used to interpret the frequency axis of the DTFT.

### Interpretation of the Spectrum

It can be concluded from the definition of the inverse DTFT that a discrete signal $x[k]$ can be represented as a superposition of weighted harmonic exponential signals $X(e^{j \Omega}) \cdot e^{j \Omega k}$ for $-\pi < \Omega < \pi$. In general, the spectrum $X(e^{j \Omega})$ will also have contributions for negative normalized angular frequencies $-\pi < \Omega < 0$. The concept of [negative frequencies](https://en.wikipedia.org/wiki/Negative_frequency) has no physical meaning. However, in the context of the DTFT with its complex kernel $e^{j \Omega k}$ negative frequencies are required to express complex and real-valued signals.

The DTFT of a discrete signal $x[k] \in \mathbb{C}$ is in general complex valued, $X(e^{j \Omega}) \in \mathbb{C}$. It is commonly represented by its real and imaginary part

\begin{equation}
X(e^{j \Omega}) = \Re \{ X(e^{j \Omega}) \} + j \cdot \Im \{ X(e^{j \Omega}) \}
\end{equation}

or by its magnitude and phase

\begin{equation}
X(e^{j \Omega}) = |X(e^{j \Omega})| \cdot e^{j \varphi(e^{j \Omega})}
\end{equation}

The magnitude spectrum $|X(e^{j \Omega})|$ provides insights into the composition of a signal in terms of its harmonic contributions. For a discrete signal which has been derived by sampling from a continuous signal, the normalized angular frequency $\Omega$ can be related to the

* angular frequency $\omega$ by $\Omega = \omega T$
* frequency $f$ by $\Omega = 2 \pi \frac{f}{f_\text{s}}$

where $T$ and $f_\text{s} = \frac{1}{T}$ denote the sampling interval and frequency, respectively. This result can be used to interpret the frequency of the DTFT as follows

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


```python
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    try:
        tmp = DX(str(float(obj)))
        obj = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(obj)):
            tmp = DX(str(float(obj[i])))
            obj[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return obj

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""
# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm, deg) = ( m/1000, m/100, pi/180)
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton

half = S(1)/2

# ---

def k(phi):
    """ element stiffness matrix """
    # phi is angle between:
    # 1. vector along global x axis
    # 2. vector along 1-2-axis of truss
    # phi is counted positively about z.
    # pprint("phi / deg:")
    # pprint(N(deg(phi),3))
    (c, s) = ( cos(phi), sin(phi) )
    (cc, ss, sc) = ( c*c, s*s, s*c)
    return Matrix(
        [
        [ cc,  sc, -cc, -sc],
        [ sc,  ss, -sc, -ss],
        [-cc, -sc,  cc,  sc],
        [-sc, -ss,  sc,  ss],
        ])

F, EA, l = var("F, EA, l")

sub_list = [
    (F,               5 *kN ),
    (EA, 200 *GPa*25 *mm**2 ),
    (l,            1707 *mm ),
    ]

(l1, l2) = (l*sqrt(2), l)
(p1, p2) = (45 *pi/180, 90*pi/180)
(k1, k2) = (EA/l1*k(p1), half*EA/l2*k(p2))

pprint("\nk1 / (EA / l): ")
pprint(k1 / (EA/l) )

pprint("\nk2 / (EA / l): ")
pprint(k2 / (EA/l) )

psi = sqrt(2)/4
c = EA/l
u3y = -(F/2) / (half+psi) / c

pprint("\nu3y:")
pprint(u3y)
pprint("\nu3y / mm:")
tmp = u3y
tmp = tmp.subs(sub_list)
tmp /= mm
tmp = iso_round(tmp,0.001)
pprint(tmp)

pprint("\nF1x / N:")
F1x = - c * psi * u3y
tmp = F1x
tmp = tmp.subs(sub_list)
tmp /= Newton
tmp = iso_round(tmp,0.1)
pprint(tmp)

# k1 / (EA / l):
# ⎡ √2    √2   -√2   -√2 ⎤
# ⎢ ──    ──   ────  ────⎥
# ⎢ 4     4     4     4  ⎥
# ⎢                      ⎥
# ⎢ √2    √2   -√2   -√2 ⎥
# ⎢ ──    ──   ────  ────⎥
# ⎢ 4     4     4     4  ⎥
# ⎢                      ⎥
# ⎢-√2   -√2    √2    √2 ⎥
# ⎢────  ────   ──    ── ⎥
# ⎢ 4     4     4     4  ⎥
# ⎢                      ⎥
# ⎢-√2   -√2    √2    √2 ⎥
# ⎢────  ────   ──    ── ⎥
# ⎣ 4     4     4     4  ⎦
#
# k2 / (EA / l):
# ⎡0   0    0   0  ⎤
# ⎢                ⎥
# ⎢0  1/2   0  -1/2⎥
# ⎢                ⎥
# ⎢0   0    0   0  ⎥
# ⎢                ⎥
# ⎣0  -1/2  0  1/2 ⎦
#
# u3y:
#     -F⋅l
# ─────────────
#      ⎛√2   1⎞
# 2⋅EA⋅⎜── + ─⎟
#      ⎝4    2⎠
#
# u3y / mm:
# -1.000
#
# F1x / N:
# 1035.5

```


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Example: Linear

y = $2x$ + 3

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Example: Cubic, Logarithmic
$$ \ y = a x^3 + b x^2 + c x + d \ $$
$$ y = \log(x)$$
$$ y = \log(a x^3 + b x^2 + c x + d)$$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Example: Quadratic
$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Example: Exponential
$$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Example: Logarithmic
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Example: Sigmoidal, Logistic
$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear

Predicting China's GDP

Creating non-linear model that predicts China's GDP in USD based on historical data from 1960-2014.

```python
import numpy as np
import pandas as pd

#downloading dataset
#!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

```python
#plot dataset
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'bo')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

```python
#choose model
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Logistic function formula:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

```python
#Building Model
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y

#simple sigmoid line
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

```python
#Normalize data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#popt: optimized params
#pcov: covariance of params
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.451711, beta_2 = 0.997207

```python
#plotting regression model
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'bo', label='data')
plt.plot(x,y, 'r', linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

```python
#model accuracy
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

popt, pcov = curve_fit(sigmoid, train_x, train_y)
y_hat = sigmoid(test_x, *popt)

print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )
```

    Mean absolute error: 0.27
    Residual sum of squares (MSE): 0.23
    R2-score: -17662373831821099865453554510594048.00

    c:\users\kaycee\documents\2ca_experiments\mlwithpythoncoursera\virtualenv_courseramlp3\lib\site-packages\scipy\optimize\minpack.py:808: OptimizeWarning: Covariance of the parameters could not be estimated
      category=OptimizeWarning)

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.

```python

```


# Network Models

Epidemiological models on defined contact network.

All models are implemented in a Markov Chain Fashion (Next state $t+1$ only depend on current state $t$).

## SIR Model

States: Susceptible ($S$), Infected ($I$), Recovered ($R$)

$P_e:$ Exposure Probability

$T_r:$ Recovery time

\begin{equation}
\begin{split}
S \rightarrow^{P_e} I\\
I \rightarrow^{1/T_r} R\\
\end{split}
\end{equation}

## SEIR Model
States: Susceptible ($S$), Exposed ($I$), Infected ($I$), Recovered ($R$)

$P_e:$ Exposure Probability

$T_i:$ Incubation Period

$T_r:$ Recovery time

\begin{equation}
\begin{split}
S \rightarrow^{P_e} E\\
E \rightarrow^{1/T_i} I\\
I \rightarrow^{1/T_r} R\\
\end{split}
\end{equation}

```python
from network_models import sir_network, seir_network

import networkx as nx
import matplotlib.pyplot as plt
from tqdm import tqdm
import numpy as np
import pandas as pd
import random

# TODO: must change models to jax + numpyro

def run_simulation(graph, model, num_sims=10, t_sim=300):
    df_sims = []
    for iter_i in tqdm(range(num_sims)):
        (df_graph, df_states) = model(graph, len(graph.nodes), t_sim, num_infected_init=None)
        df_states['sim_id'] = iter_i
        df_sims.append(df_states)
    df_all = pd.concat(df_sims)

    return (df_all, df_graph)

graph = nx.scale_free_graph(500).to_undirected()
pos = nx.spring_layout(graph, iterations=100) #,prog='twopi',args='')
```

# SIR Dynamics

```python

(df_dynamics_sir, df_graph_sir) = run_simulation(graph, sir_network, num_sims=10, t_sim=100)
fig, ax = plt.subplots(1,2, figsize=(15.5, 8))
nx.draw(ax = ax[0], G=graph, pos=pos,
        node_size=5,
        node_color= 'black',#'#e35340',
        edge_color='gray',
        width=.05,
        edge_cmap=plt.cm.Blues, with_labels=False)

df_sim1 = df_dynamics_sir.copy().reset_index()
df_sim1 = df_sim1.groupby(['index']).median()

df_sim1.plot(ax=ax[1], y='S')
df_sim1.plot(ax=ax[1], y='I')
df_sim1.plot(ax=ax[1], y='R')
```

```python
(df_dynamics_seir, df_graph_seir) = run_simulation(graph, seir_network, num_sims=10, t_sim=100)

fig, ax = plt.subplots(1,2, figsize=(15.5, 8))
nx.draw(ax = ax[0], G=graph, pos=pos,
        node_size=5,
        node_color= 'black',#'#e35340',
        edge_color='gray',
        width=.05,
        edge_cmap=plt.cm.Blues, with_labels=False)

df_sim1 = df_dynamics_seir.copy().reset_index()
df_sim1 = df_sim1.groupby(['index']).median()

df_sim1.plot(ax=ax[1], y='S')
df_sim1.plot(ax=ax[1], y='E')
df_sim1.plot(ax=ax[1], y='I')
df_sim1.plot(ax=ax[1], y='R')

```

```python

```


# Music machine learning - Introduction

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. A [first definition](#definition) on the concept of machine learning
2. An introduction to a simple problem of [linear regression](#regression)
3. An explanation on [model capacity and overfitting](#capacity)
4. An introduction to the [audio datasets](#audio) that we will use

## Defining machine learning

In all natural process, there exists complex relations between sets $\mathcal{X} \mapsto \mathcal{Y}$. This can relate some objects with their names, or a cause to a consequence. In most cases, _we do not know the precise relations_ between these sets, all we have is _observations_ such as pairs $(x,y)$, composed of input data $x \in \mathcal{X}$, which have a corresponding expected output $y \in \mathcal{Y}$. The overarching goal of machine learning is to approximate such _unknown processes_ as a function $\mathcal{F}_{\theta}$, which _transforms_ input data $x$ into output data $y$.

Hence, machine learning aims to understand and model the relationship between some (usually complex and high-dimensional) inputs $\mathbf{x}\in\mathcal{X}\subset\mathbb{R}^{\mathcal{X}}$ and outputs $\mathbf{y}\in\mathcal{Y}\subset\mathbb{R}^{\mathcal{Y}}$, given by a set of data examples $\mathcal{D}=\left\{(x_1,y_1),\cdots,(x_N,y_N)\right\}$. This is achieved by defining a parametric model $f_{\mathbf{\theta}}\in\mathcal{F}$ inside a family of functions $\mathcal{F}$, which depends on parameters $\mathbf{\theta} \in \mathbf{\Theta}$ and that could approximate the underlying relationship. The _learning_ aspect refers to the adjustment of the parameters $\mathbf{\theta}$ in order to obtain the best approximation of the given task
$$
\begin{equation}
f_{\mathbf{\theta}}(\mathbf{x}) = \hat{\mathbf{y}}\approx \mathbf{y}.
\end{equation}
$$

To observe this idea in simple setups, we are going to use the `numpy` library and `matplotlib` for plotting. We also set some fixed random generator, to ensure reproducibility

```python
# Base imports
import numpy as np
import matplotlib.pyplot as plt
from helper_plot import prep_plots, finalize_plots, hdr_plot_style
rng = np.random.RandomState(1)
```

## Simple learning problem

Imagine that a certain process somewhere follows the form of a quadratic relationship

$$
 y = a x^{2} + bx + c
$$

In this case, all the **unknown parameters** are that of a polynomial model, therefore we have $\theta = \{a, b, c\}$. However, this is clearly an ideal (clean) case, whereas in natural observations, there might be some noise in our observations
$$
 y = a x^{2} + bx + c +\epsilon \quad \mbox{with} \quad \epsilon \in [-0.1, 0.1]
$$

An example of such noisy observations for different parameters is given below

```python
noise_level = 0.1
a, b, c = 3, 0, 1
# Generating
x = np.linspace(0, 1, 100)
poly = np.poly1d([a, b, c])
epsilon = np.random.uniform(-noise_level,noise_level,x.shape)
y = poly(x) + epsilon
ax1, = prep_plots(["Simple learning problem"], fig_size=(10,8), fig_num=1)
ax1.scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots([ax1], fig_title="")
#plt.xlim([-1, 2])
```

Now our main problem is that this function can follow different types of parameters

```python
params = [[3, 0, 1], [-2, 1, 0], [0.1, 1, 1]]
ax = prep_plots(["Problem 1", "Problem 2", "Problem 3"], fig_size=(20,6), fig_num=1)
# Generating
x = np.linspace(0, 1, 100)
for p in range(len(params)):
    poly = np.poly1d(params[p])
    epsilon = np.random.uniform(-noise_level,noise_level,x.shape)
    y = poly(x) + epsilon
    ax[p].scatter(x, y, color='green', s=50, marker='o', edgecolor='k', label="training samples")
finalize_plots(ax, fig_title="Different observations")
```

In real-life settings, this function can also have different levels of noise, as exemplified in the following code.

```python
params = [3, 0, 1]
noise_levels = [0.1, 1.0, 8.0]
ax = prep_plots(["Problem 1", "Problem 2", "Problem 3"], fig_size=(20,6), fig_num=1)
# Generating
x = np.linspace(0, 1, 100)
for p in range(len(noise_levels)):
    poly = np.poly1d(params)
    epsilon = np.random.uniform(-noise_levels[p],noise_levels[p],x.shape)
    y = poly(x) + epsilon
    ax[p].scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots(ax, fig_title="Different amounts of noise")
```

To summarize, we will have some observations of a function, and we would like to optimize a function that gets as close as possible to the real function that generated this data. Here, we plot the real function and also _subsample_ our number of observations (having only a few points to find the corresponding function)

```python
# Generating the data and subsampling
x_all = np.linspace(0, 1, 100); x_plot = np.linspace(0, 1, 100)
rng.shuffle(x_all); x = np.sort(x_all[:10])
poly = np.poly1d([3,0,1])
# Adding some external
epsilon = np.random.uniform(-0.1,0.1,x.shape)
y = poly(x)+ epsilon
# We keep the standard deviation for later
nnstd = np.std(epsilon); lw = 2
ax1, = prep_plots(["Simple learning problem"], fig_size=(10,8), fig_num=1)
ax1.plot(x_plot, poly(x_plot), color='green', linewidth=4, label='true function')
ax1.scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots([ax1], fig_title="")
```

To get a first grip on what machine learning does, we will rely on the `scikit-learn` library. This contains already coded models and learning procedure, that will allow us to _learn_ the parameters of this unknown function.

Here we already know that we want to use a `PolynomialFeatures` model to perfom `LinearRegression` and that this polynomial should be of degree 2.

```python
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
# Our data to fit
X = x[:, np.newaxis]
# Degree of our polynomial
degree = 2;
# Create our polynomial model for regression
model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
# Fit the parameters of this model
model.fit(X, y)
```

    Pipeline(steps=[('polynomialfeatures', PolynomialFeatures()),
                    ('linearregression', LinearRegression())])

Now that we have trained the model, we can perform _predictions_ from it, meaning that we can infer the output of the function at values that we did not observe originally.

```python
# Inference points (not observed)
X_plot = x_plot[:, np.newaxis]
# Predict the values
y_plot = model.predict(X_plot)
# Compute the error of our model at observed points
Y_model_err = np.sqrt(np.mean(np.square(y-model.predict(X))))
# Plot the result
ax1, = prep_plots(["Trained model"], fig_size=(10,8), fig_num=1)
ax1.plot(x_plot, y_plot, color='red', linewidth=4, label='trained model')
ax1.scatter(x, y, color='green', s=50, marker='o', edgecolor='w', label="training samples")
finalize_plots([ax1], fig_title="")
print(f'Model error : {Y_model_err}')
```

## Understanding model capacity and selection

In real-life problem, we are aiming to find the parameters of a model, but we do not really know what is the _real_ function underlying this process. So what we can decide to select _any_ function of _any_ **capacity** (complexity of the function). One of the problem with that, is that if we have a too simple function, it will _underfit_ (it is not complex enough for our observations). On the opposite end, if we have a function which is too complex, it might be able to _fit through all training points exactly_ ... even though there is noise in our observations ! This is examplified in the following

We can observe this idea and play with it directly by trying to find a function approximating our previous observations with a polynomial function chosen to have a degree inside \([1,2,8]\).

```python
X_plot = x_plot[:, np.newaxis]
ax1, ax2 = prep_plots(["training error", "model generalization"], fig_size=(15,4), fig_num=1)
ax1.plot([1,8], [nnstd, nnstd], label="noise level")
ax2.scatter(x, y, color='orange', edgecolor='w', s=50, marker='o', label="training samples")
colors = ['blue', 'green', 'red']; labels = ['underfitting', 'good fit', 'overfitting']
axes_sub = prep_plots(labels, fig_size=(15,4), fig_num=2)
# Optimize different degree polynomials
for count, degree in enumerate([1,2,8]):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X, y)
    Y_model_err = np.sqrt(np.mean(np.square(y-model.predict(X))))
    y_plot = model.predict(X_plot)
    ax1.scatter(degree, Y_model_err, s=50, marker="o", edgecolor='w', label=labels[count])
    ax2.plot(x_plot, y_plot, color=colors[count], linewidth=2,label=labels[count])
    ax2.set_ylim((0,4))
    axes_sub[count].scatter(x, y, color='orange', edgecolor='w', s=50, marker='o', label="Training samples")
    axes_sub[count].plot(x_plot, y_plot, color=colors[count], linewidth=2,label=labels[count])
    axes_sub[count].set_ylim((0,4))
finalize_plots(axes_sub, fig_title="Linear regression")
finalize_plots([ax1,ax2], fig_title="Linear regression")
```

Depending on the _capacity_ of the model, what we can observe is that

- `capacity too low   -> underfitting   : prediction variance >  noise variance`
- `adequate capacity  -> good fit       : prediction variance == noise variance`
- `capacity too high  -> overfitting    : prediction variance <  noise variance`

A similar example can be given for a classification problem in two dimensions as follows

## Audio applications

In order to test our algorithms on audio and music data, we will work with several datasets that should be downloaded on your local computer first from this [link ](https://nuage.ircam.fr/index.php/s/FTsaaAMFV1jEwsk)

  |**Type**|*Origin*|
  |-------:|:---------|
  |**Classification**|[*MuscleFish*](http://knight.cis.temple.edu/~vasilis/Courses/CIS750/Papers/muscle_fish.pdf) dataset|
  |**Music-speech**|[*MIREX Recognition*](http://www.music-ir.org/mirex/wiki/2015:Music/Speech_Classification_and_Detection) set|
  |**Source separation**|[*SMC Mirum*](http://smc.inesctec.pt/research/data-2/) dataset|
  |**Speech recognition**|[*CMU Arctic*](http://festvox.org/cmu_arctic/) dataset|

**Unzip the file and place the `data` folder along with the other code folders**
For the first parts of the tutorial, we will mostly rely solely on the classification dataset. In order to facilitate the interactions, we provide the function `import_dataset` that will allow to import all audio datasets along the tutorials.

```Python
def importDataset(class_path, type):
    """
    Helper function to import datasets
    % class_path  : Path to the dataset (string)
    % type       : Type of dataset (string: 'classify', 'plain', 'metadata')
    """
    # Returns the data_struct structure with
    data_struct["filenames"]  # Cell containing the list of audio files
    data_struct["classes"]    # Vector of indexes assigning each file to a class
    data_struct["class_names"] # Cell of class names
    return data_struct
```

***

**_Exercise_**

  1. Launch the import procedure  and check the corresponding structure
  2. Code a count function that prints the name and number of examples for each classes

***

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from helper_data import import_dataset
# 0.1 - Import the classification dataset
class_path = 'datasets/classification';
data_struct = import_dataset(class_path, 'classification');
```

        - Importing dataset datasets/classification.

### Preprocessing

We will rely on a set of spectral transforms that allow to obtain a more descriptive view over the audio information. As most of these are out of the scope of the machine learning course, we redirect you to a [signal processing course](https://ccrma.stanford.edu/~jos/sasp/) proposed by [Julius O. Smith](https://ccrma.stanford.edu/~jos/).

The following functions to compute various types of transforms are given as part of the basic package, in the `helper_data.py` file

  |**File**|*Transform*|
  |-------:|:----------|
  |`stft.m`       |[Short-term Fourier transform](https://en.wikipedia.org/wiki/Short-time_Fourier_transform)|
  |`fft2barkmx.m` |[Bark scale](https://en.wikipedia.org/wiki/Bark_scale) transform|
  |`fft2melmx.m`  |[Mel scale](https://en.wikipedia.org/wiki/Mel_scale) transform|
  |`fft2chromamx` |[Chromas vector](https://en.wikipedia.org/wiki/Harmonic_pitch_class_profiles)|
  |`spec2cep.m`   |[Cepstrum](https://en.wikipedia.org/wiki/Cepstrum) transform|
  |`cqt.m`        |[Constant-Q](https://en.wikipedia.org/wiki/Constant_Q_transform) transform|

In order to perform the various computations, we provide the following function, which performs the different transforms on a complete dataset.

``` Python
def computeTransforms(data_struct)
    """ data_struct   : Dataset structure with filenames """
    return data_struct

# Returns the data_struct structure with
data_struct["spectrum_power"]     # Power spectrum (STFT)
data_struct["spectrum_mel"]       # Spectrum in Mel scale
data_struct["spectrum_chroma"]    # Chroma vectors
data_struct["spectrum_CQT"]       # Constant-Q transform
```

***

**Exercise**

  1. Launch the transform computation procedure and check the corresponding structure
  2. For each class, select a random element and plot its various transforms on a single plot. You should obtain plots similar to those shown afterwards.
  3. For each transform, try to spot major pros and cons of their representation.

***

```python
# 0.2 - Pre-process the audio to obtain spectral transforms
# (may take around a minute)
from helper_data import compute_transforms
data_struct = compute_transforms(data_struct);
```

        - Performing transforms.
    21665
    1024
    11425
    1024
    12033
    1024
    11553
    1024
    21986
    1024
    11137
    1024
    26658
    1024
    5217
    1024
    7105
    1024
    12609
    1024
    3425
    1024
    6881
    1024
    10977
    1024
    7073
    1024
    10626
    1024
    8865
    1024
    7298
    1024
    7170
    1024
    3362
    1024
    3329
    1024
    5505
    1024
    4577
    1024
    6337
    1024
    7458
    1024
    175196
    1024
    41665
    1024
    134012
    1024
    55170
    1024
    22081
    1024
    127037
    1024
    118497
    1024
    34029
    1024
    27235
    1024
    31831
    1024
    24224
    1024
    29589
    1024
    30343
    1024
    28612
    1024
    24469
    1024
    32502
    1024
    28987
    1024
    29597
    1024
    27589
    1024
    30723
    1024
    35863
    1024
    26375
    1024
    31149
    1024
    31667
    1024
    24719
    1024
    36978
    1024
    27165
    1024
    30077
    1024
    33838
    1024
    35835
    1024
    30605
    1024
    29884
    1024
    29625
    1024
    28010
    1024
    30241
    1024
    27983
    1024
    33241
    1024
    31101
    1024
    28642
    1024
    31579
    1024
    28309
    1024
    18766
    1024
    13006
    1024
    12424
    1024
    18666
    1024
    13390
    1024
    18642
    1024
    27934
    1024
    22242
    1024
    23506
    1024
    22319
    1024
    23870
    1024
    6850
    1024
    7393
    1024
    9281
    1024
    5538
    1024
    3969
    1024
    5089
    1024
    2977
    1024
    4321
    1024
    3265
    1024
    7969
    1024
    6401
    1024
    6689
    1024
    8001
    1024
    6209
    1024
    7553
    1024
    3169
    1024
    10817
    1024
    8545
    1024
    3841
    1024
    6434
    1024
    3681
    1024
    4226
    1024
    4001
    1024
    4161
    1024
    5921
    1024
    4033
    1024
    3777
    1024
    6465
    1024
    8961
    1024
    4226
    1024
    11074
    1024
    6753
    1024
    3905
    1024
    6465
    1024
    5665
    1024
    3479
    1024
    67580
    1024
    5119
    1024
    5054
    1024
    4907
    1024
    4579
    1024
    4450
    1024
    4249
    1024
    62283
    1024
    5360
    1024
    4474
    1024
    59555
    1024
    4169
    1024
    4501
    1024
    4189
    1024
    4397
    1024
    4125
    1024
    5769
    1024
    9795
    1024
    6808
    1024
    44406
    1024
    19651
    1024
    6876
    1024
    7379
    1024
    6836
    1024
    4227
    1024
    38038
    1024
    23102
    1024
    7072
    1024
    20036
    1024
    20645
    1024
    4704
    1024
    8235
    1024
    24275
    1024
    9986
    1024
    7160
    1024
    4479
    1024
    34346
    1024
    17861
    1024
    9952
    1024
    7316
    1024
    5454
    1024
    10422
    1024
    50587
    1024
    7769
    1024
    4778
    1024
    9773
    1024
    8695
    1024
    5722
    1024
    4896
    1024
    12216
    1024
    31226
    1024
    9322
    1024
    10124
    1024
    23648
    1024
    20142
    1024
    11130
    1024
    19786
    1024
    5062
    1024
    33889
    1024
    15168
    1024
    5578
    1024
    14117
    1024
    19848
    1024
    8947
    1024
    10478
    1024
    10711
    1024
    12150
    1024
    13382
    1024
    3509
    1024
    20841
    1024
    8493
    1024
    8924
    1024
    17470
    1024
    17033
    1024
    17601
    1024
    7163
    1024
    15192
    1024
    20074
    1024
    9245
    1024
    7153
    1024
    9206
    1024
    6990
    1024
    5024
    1024
    13619
    1024
    21891
    1024
    22826
    1024
    33104
    1024
    25496
    1024
    7391
    1024
    6571
    1024
    87418
    1024
    11243
    1024
    7591
    1024
    16082
    1024
    6050
    1024
    4087
    1024
    8665
    1024
    45127
    1024
    23458
    1024
    19282
    1024
    10696
    1024
    9463
    1024
    18728
    1024
    10243
    1024
    5914
    1024
    10139
    1024
    12934
    1024
    7048
    1024
    33807
    1024
    22290
    1024
    8778
    1024
    8429
    1024
    10327
    1024
    9529
    1024
    8931
    1024
    5594
    1024
    3597
    1024
    3702
    1024
    4552
    1024
    3341
    1024
    5801
    1024
    5247
    1024
    3415
    1024
    6869
    1024
    5670
    1024
    5055
    1024
    5816
    1024
    6966
    1024
    3650
    1024
    3917
    1024
    13541
    1024
    4033
    1024
    3654
    1024
    3478
    1024
    5338
    1024
    6544
    1024
    6330
    1024
    4584
    1024
    6957
    1024
    5417
    1024
    5233
    1024
    5321
    1024
    5062
    1024
    5895
    1024
    8129
    1024
    6060
    1024
    3381
    1024
    7070
    1024
    6180
    1024
    3234
    1024
    5724
    1024
    5935
    1024
    6921
    1024
    6417
    1024
    6128
    1024
    23700
    1024
    28185
    1024
    32436
    1024
    32549
    1024
    22622
    1024
    36970
    1024
    32428
    1024
    126657
    1024
    218556
    1024
    74130
    1024
    89843
    1024
    64158
    1024
    129976
    1024
    125571
    1024
    70470
    1024
    82587
    1024
    37463
    1024
    32789
    1024
    23010
    1024
    23008
    1024
    20255
    1024
    28535
    1024
    23963
    1024
    18594
    1024
    24431
    1024
    21513
    1024
    24152
    1024
    22150
    1024
    23394
    1024
    23571
    1024
    21621
    1024
    24807
    1024
    21838
    1024
    22351
    1024
    25169
    1024
    24600
    1024
    23478
    1024
    21761
    1024
    60161
    1024
    79072
    1024
    82116
    1024
    76637
    1024
    79644
    1024
    35771
    1024
    51250
    1024
    41803
    1024
    15644
    1024
    39455
    1024
    37735
    1024
    40964
    1024
    40167
    1024
    40840
    1024
    37541
    1024
    39565
    1024
    36555
    1024
    34724
    1024
    40081
    1024
    43556
    1024
    34372
    1024
    40825
    1024
    42990
    1024
    38275
    1024
    39575
    1024
    40690
    1024
    37576
    1024
    34033
    1024
    39518
    1024
    40916
    1024
    31803
    1024
    31653
    1024
    36925
    1024
    33386
    1024
    36955
    1024
    37353
    1024
    31729
    1024
    41851
    1024
    39663
    1024
    36419
    1024
    36103
    1024
    35868
    1024
    40541
    1024
    33843
    1024
    32180
    1024
    38224
    1024
    37906
    1024
    35778
    1024
    36332
    1024
    31677
    1024
    35837
    1024
    39156
    1024
    35764
    1024
    34725
    1024
    33335
    1024
    39813
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=312
      warnings.warn(

    51668
    1024
    46042
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=404
      warnings.warn(

    47670
    1024
    42585
    1024
    38457
    1024
    35143
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=301
      warnings.warn(

    40147
    1024
    30855
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=483
      warnings.warn(
    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=242
      warnings.warn(

    45607
    1024
    33548
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=263
      warnings.warn(

    65551
    1024
    40027
    1024
    30128
    1024
    34709
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=236
      warnings.warn(

    41443
    1024
    21213
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=324
      warnings.warn(
    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=332
      warnings.warn(
    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=166
      warnings.warn(

    21852
    1024
    38471
    1024
    22114
    1024
    34252
    1024
    41734
    1024
    29960
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=469
      warnings.warn(

    16916
    1024
    32976
    1024
    36424
    1024
    19998
    1024
    41317
    1024
    26958
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=422
      warnings.warn(

    39794
    1024
    23099
    1024
    22850
    1024
    34472
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=270
      warnings.warn(

    54436
    1024
    36230
    1024
    50840
    1024
    42036
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=398
      warnings.warn(
    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=329
      warnings.warn(

    32290
    1024
    33000
    1024
    37032
    1024
    48575
    1024

    /Applications/anaconda3/lib/python3.8/site-packages/librosa/core/spectrum.py:222: UserWarning: n_fft=512 is too small for input signal of length=290
      warnings.warn(

    50742
    1024
    47677
    1024
    43119
    1024
    39489
    1024
    40787
    1024
    37713
    1024
    39487
    1024
    33569
    1024
    43640
    1024

### Features

As you might have noted from the previous exercice, most spectral transforms have a very high dimensionality, and might not be suited to exhibit the relevant structure of different classes. To that end, we provide a set of functions for computing several spectral features in the `helper_data` folder, we redirect interested readers to this [exhaustive article](http://recherche.ircam.fr/anasyn/peeters/ARTICLES/Peeters_2003_cuidadoaudiofeatures.pdf) on spectral features computation.

  |**File**|*Transform*|
  |-------:|:----------|
  |`spectral_centroid`|Spectral centroid|
  |`spectral_bandwidth`|Spectral bandwidth|
  |`spectral_contrast`|Spectral contrast|
  |`spectral_flatness`|Spectral flatness|
  |`spectral_rolloff`|Spectral rolloff|

Once again, we provide a function to perform the computation of different features on a complete set. Note that for each feature, we compute the temporal evolution in a vector along with the mean and standard deviation of each feature. We only detail the resulting data structure for a single feature (`SpectralCentroid`).

``` Python
def data_struct = computeFeatures(data_struct)
     """ data_struct   : Dataset structure with filenames """
    return data_struct

% Returns the data_struct structure with
data_struct["spectral_centroid"]      # Temporal value of a feature
data_struct["spectral_centroid_mean"] # Mean value of that feature
data_struct["spectral_centroid_std"]  # Standard deviation
```

***

**Exercise**

  1. Launch the feature computation procedure and check the corresponding structure
  2. This time for each class, superimpose the plots of various features on a single plot, along with a boxplot of mean and standard deviations. You should obtain plots similar to those shown afterwards.
  3. What conclusions can you make on the discriminative power of each feature ?
  4. Perform scatter plots of the mean features for all the dataset, while coloring different classes.
  5. What conclusions can you make on the discriminative power of mean features ?

***

```python
#%%
# 0.3 - Compute a set of temporal and spectral features
# (may take around 1-2 minutes)
from helper_data import compute_features
data_struct = compute_features(data_struct);
```

```python
#%% Q-0.3.2 - Plot the various features

# Just a little helper to make your figures pretty
from helper_plot import hdr_plot_style
hdr_plot_style()
# Use these styles for boxplot
boxprops=dict(linewidth=3, color='white')
whiskerprops=dict(linewidth=3, color='white')
medianprops=dict(linewidth=2.5, color='firebrick')
flierprops = dict(markeredgecolor='white', markerfacecolor='firebrick')

######################
# YOUR CODE GOES HERE
######################

```

```python
#%% Q-0.3.4 - Observe the distribution of classes for different features

# This allows to use 3D rendering in matplotlib
from mpl_toolkits.mplot3d import Axes3D
plt.figure(figsize=(12,8))
# Create a vector of random colors for each class
colorVect = np.zeros((3, len(data_struct["class_names"])));
for c in range(len(data_struct["class_names"])):
    colorVect[:,c] = np.random.rand(3);

######################
# YOUR CODE GOES HERE
######################

```

That's it for this tutorial, now remember that we can use any form of description (features) as a basis for learning algorithms. We will see in the next tutorial what we an do with these features.