```python
%matplotlib nbagg
from __future__ import division
from IPython.display import HTML
from IPython.display import display
from scipy.optimize import *
import scipy
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

## Question 1: Poiseuille's method for determining viscosity

The  volume flow rate, ${\displaystyle\frac{{\rm d}V}{{\rm d}t}}$, of fluid flowing smoothly through a horizontal tube of length $L$ and radius $r$ is given by Poiseuille's equation:
\begin{equation}
\frac{{\rm d}V}{{\rm d}t}=\frac{\pi\rho g h r^4}{8\eta L},
\end{equation}
where $\eta$ and $\rho$ are the viscosity and density, respectively, of the fluid,  $h$ is the head of pressure across the tube, and $g$ the acceleration due to gravity.
<bf>
In an experiment the graph of the flow rate versus height has a slope measured to 7%, the length is known to 0.5%, and the radius to 8%.
<bf>
Required:
<bf>

(i) What is the fractional precision to which the viscosity is known?
<bf>
<bf>
(ii) If more experimental time is available, should this be devoted to
>(a) collecting more flow-rate data,
<bf>
>(b) measuring the length,
<bf>
>(c) the radius of the tube?

### (i) What is the fractional precision to which the viscosity is known? Express your answer as a DECIMAL.

```python
def one_i():
    h = 0.07
    r = 0.08
    L = 0.005
    Z2 = h ** 2 + L ** 2 + r ** 2 * 16
    return np.sqrt(Z2)

print(one_i())
```

    0.32760494501762333

```python
'''TEST CELL- DO NOT DELETE'''
assert isinstance(one_i(), float) , 'Please make sure that one_i() returns a float.'
assert one_i()>0., 'Please make sure that one_i() returns a positive value.'

```

### (ii) If more experimental time is available, should this be devoted to
>(a) collecting more flow-rate data,
<bf>
>(b) measuring the length,
<bf>
>(c) the radius of the tube?

```python
def one_ii():
    '''Your function should return a string of A,B,C'''
    # YOUR CODE HERE
    comment = "C"
    return(comment)

print(one_ii())
```

    C

```python
'''TEST CELL- DO NOT DELETE'''
assert one_ii() == 'A' or one_ii() == 'B' or one_ii() == 'C', 'Please return A or B or C in one_ii().'
```

## Question 2: Functional error approach for Van der Waals calculation

The Van der Waals equation of state is a correction to the ideal gas law, given by the equation,

\begin{equation}
(P+\frac{a}{V_m^2})(V_m-b) = RT,
\end{equation}

where $P$ is the pressure, $V_m$ is the molar volume, $T$ is the absolute temperature, $R$ is the universal gas constant with $a$ and $b$ being species-specific Van der Waals coefficents.

A sample of Nitrogen was measured in an experiment as,
<bf>
>Molar Volume $V_m$ = $(2.000 \pm 0.003)$x$10^{-4}m^3mol^{-1}$
<bf>
>Absolute Temperature $T$ = $298.0\pm0.2K$

and the constants are,

<bf>
$a$ = $(1.408$x$10^{-1}) m^6mol^{-2}Pa$
<bf>
$b$ = $(3.913$x$10^{-5}) m^3mol^{-1}$
<bf>
$R$ = $(8.3145) JK^{-1}mol^{-1}$
<bf>

Required:
<bf>
>(i) From the given data, calculate the pressure giving your answer in MPa.
<bf>
>(ii) Calculate the uncertainty in the pressure by using the functional approach for error propagation.
<bf>
>(iii) Repeat the calculations above for
>>$V_m = (2.000\pm0.003)\times10^{-3}\,{\rm m}^{3}\,{\rm mol}^{-1}$ and  $T=400.0 \pm 0.2K$.

### (i) From the given data, calculate the pressure giving your answer in MPa.

```python
def two_i():
    '''Your function should return the pressure in MPa'''
    R = 8.3145
    b = 3.913 * 10**(-5)
    a = 1.408 * 10**(-1)
    Vm = 2.000 * 10**(-4)
    T = 298.0
    P = R * T / (Vm - b) - (a / Vm ** 2)
    P = P / 10**6
    return round(P,2)
print(two_i())
```

    11.88

```python
'''TEST CELL- DO NOT DELETE'''
assert isinstance(two_i(), float) , 'Please make sure that two_i() returns a float.'
```

### (ii) Calculate the uncertainty in the pressure by using the functional approach for error propagation.

```python
def two_ii():
    '''Your function should return the uncertainty'''
    # YOUR CODE HERE
    R = 8.3145
    b = 3.913 * 10**(-5)
    a = 1.408 * 10**(-1)
    Vshift = 0.003 * 10**(-4)
    Vm = 2.000 * 10**(-4)
    T = 298.0
    Tshift = 0.2
    P = (R * T / (Vm - b) - (a / Vm ** 2)) * 10 ** -6
    P_Vshift = (R * T / ((Vm + Vshift) - b) - (a / (Vm + Vshift) ** 2)) * 10 ** -6
    P_Tshift = (R * (T + Tshift) / (Vm - b) - (a / Vm ** 2)) * 10 ** -6
    sigma_T = P_Tshift - P
    sigma_V = P_Vshift - P
    ans = np.sqrt(sigma_T ** 2 + sigma_V ** 2)
    return round(ans,2)
print(two_ii())
```

    0.02

```python
'''TEST CELL- DO NOT DELETE'''
assert isinstance(two_ii(), float) , 'Please make sure that two_ii() returns a float.'
```

### (iii) Repeat the calculations above for
>$V_m = (2.000\pm0.003)\times10^{-3}\,{\rm m}^{3}\,{\rm mol}^{-1}$ and  $T=400.0 \pm 0.2K$.

```python
def two_iii():
    '''Your function should return both the pressure and the uncertainty'''
    pressure_2 = 0
    uncertainty_2 = 0
    # YOUR CODE HERE
    R = 8.3145
    b = 3.913 * 10**(-5)
    a = 1.408 * 10**(-1)
    Vshift = 0.003 * 10**(-3)
    Vm = 2.000 * 10**(-3)
    T = 400.0
    Tshift = 0.2
    P = (R * T / (Vm - b) - (a / Vm ** 2)) * 10 ** -6
    P_Vshift = (R * T / ((Vm + Vshift) - b) - (a / (Vm + Vshift) ** 2)) * 10 ** -6
    P_Tshift = (R * (T + Tshift) / (Vm - b) - (a / Vm ** 2)) * 10 ** -6
    sigma_T = P_Tshift - P
    sigma_V = P_Vshift - P
    pressure_2 = round(P, 3)
    uncertainty_2 = round(np.sqrt(sigma_T ** 2 + sigma_V ** 2), 3)
    return(pressure_2,uncertainty_2)

print(two_iii())
```

    (1.661, 0.003)

```python
'''TEST CELL- DO NOT DELETE'''
test_case = two_iii()
assert isinstance(test_case[0], float) , 'Please make sure that two_iii() returns two floats.'
assert isinstance(test_case[1], float) , 'Please make sure that two_iii() returns two floats.'
```

```python
'''TEST CELL- DO NOT DELETE'''
```

    'TEST CELL- DO NOT DELETE'

## Question 3: Reverse Engineering the Incredible Goal
The separation of the posts is 7.32m, and the ball is struck from a point 22m from the near post and 29m from the far post.

Required:

> (i) Plot a graph of $\alpha_\theta$ on the y-axis vs $\alpha_L$ on the x-axis for the range of values presented in the alpha_ls array.
<bf>
>(ii) To what (common) precision must these three lengths be known to justify quoting the angle to 11 significant figures?

*[Hint: use the functional approach using the values for the errors in the length measurements as the provided alpha__ls array]*

### (i) Plot a graph of $\alpha_\theta$ on the y-axis vs $\alpha_L$ on the x-axis for the range of values presented in the alpha_ls array.

```python
def three_i():
    a = 22
    b = 29
    c = 7.32
    import math
    alpha_ls = [0.1, 0.01, 0.001, 0.0001, 1e-05, 1e-06, 1e-07, 1e-08, 1e-09, 1e-10, 1e-11, 1e-12, 7.65157214e-13]
    alpha_thetas = []
    angel = degree(a, b, c)
    for i in alpha_ls:
        degree_errorA = degree(a + i, b, c) - angel
        degree_errorB = degree(a, b + i, c) - angel
        degree_errorC = degree(a, b, c + i) - angel
        alpha_theta = np.sqrt(degree_errorA ** 2 + degree_errorB ** 2 + degree_errorC ** 2)
        alpha_thetas.append(alpha_theta)
    plt.figure()
    plt.plot(-np.log10(alpha_ls) - 1, alpha_thetas)
    plt.show()
    '''Your function should plot the graph and return the alpha_thetas array.'''
    # YOUR CODE HERE
    return(alpha_thetas)

def degree(a, b, c):
    angel = 180 / np.pi * np.arccos((a**2 + b**2 - c**2) / (2 * a * b))
    return angel

three_i()

# def func_solve(i):
#     a = 22
#     b = 29
#     c = 7.32
#     angel = degree(a, b, c)
#     print(i)
#     degree_errorA = degree(a + i, b, c) - angel
#     degree_errorB = degree(a, b + i, c) - angel
#     degree_errorC = degree(a, b, c + i) - angel
#     alpha_theta = np.sqrt(degree_errorA ** 2 + degree_errorB ** 2 + degree_errorC ** 2)
#     return(alpha_theta - 1e-11)

# ans = fsolve(func_solve, 1e-10)
# print(ans)

1e-11
```

    <IPython.core.display.Javascript object>

    [1.2897314602156371,
     0.13011222242803275,
     0.013040258626603154,
     0.001304333860986745,
     0.0001304364837778715,
     1.3043679352599412e-05,
     1.304368267610034e-06,
     1.3043677404434444e-07,
     1.3043593635272353e-08,
     1.3043345643680718e-09,
     1.3050678994651162e-10,
     1.3015082449976125e-11,
     9.936340851320653e-12]

```python
'''TEST CELL- DO NOT DELETE'''
```

    'TEST CELL- DO NOT DELETE'

### (ii) To what (common) precision must these three lengths be known to justify quoting the angle to 11 significant figures?

Using fsolve in scipy, the outcome that quoting the angle to 11 significat figure need precision 7.65157214e-13, so at least, the common precision should be 1e-13.

## Question 4: Linear Regression/Weighted Fit

The data  plotted in Fig 6.1(d) relating to the degradation of the signal to noise ratio from a frequency to voltage converter near harmonics of the mains frequency are listed below.
\begin{equation}
\begin{array}{lcccccc}
\hline
{\rm frequency~(Hz)} &10&20&30&40&50&60\\
{\rm voltage~(mV)} &16&45&64&75&70&115\\
{\rm error~(mV)}   &5&5&5&5&30&5\\
\hline
{\rm frequency~(Hz)} &70&80&90&100&110&\\
{\rm voltage~(mV)} &142&167&183&160&221&\\
{\rm error~(mV)}   &5&5&5&30&5&\\
\hline
\end{array}
\end{equation}

This data is also contained in the file 'linear_regression.csv'.

Required:
<bf>
> (i) Calculate the best-fit gradient and intercept and associated errors using a weighted fit. You may use the curve_fit function.

```python
data = pd.read_csv('linear_regression.csv')
frequencies = data.iloc[:,0]
voltages = data.iloc[:,1]
voltage_errors = data.iloc[:,2]

def f(frequencies, a,b):
    return a*frequencies+b

def best_fit_params():
    '''Your function should return the gradient,gradient_error,intercept,intercept_error'''
    # YOUR CODE HERE
    gradient = 0
    gradient_error = 0
    intercept = 0
    intercept_error = 0
    plt.figure()
    popt, pcov = curve_fit(f, frequencies, voltages, sigma = voltage_errors)
    plt.plot(frequencies, voltages, 'b-', label='origin_data')
    plt.plot(frequencies, f(frequencies, *popt), 'r-', label='fit: a=%5.3f, b=%5.3f' % tuple(popt))
    plt.xlabel('x')
    plt.ylabel('y')
    plt.legend()
    plt.show()
    total_sub = 0
    delta = 0
    total_x = 0
    total_x2 = 0
    total_y = 0
    total_xy = 0
    gradient = popt[0]
    intercept = popt[1]
    for i in range(len(frequencies)):
        total_sub += (voltages[i] - gradient * frequencies[i] - intercept) ** 2
        total_x += frequencies[i]
        total_x2 += frequencies[i] ** 2
    delta = len(frequencies) * total_x2 - total_x ** 2
    common_uncertainty = np.sqrt((1 / (len(frequencies) - 2)) * total_sub)
#     gradient_error = common_uncertainty * np.sqrt(len(frequencies) / delta)
#     intercept_error = common_uncertainty * np.sqrt(total_x2 / delta)
    gradient_error = pcov[0,0]**0.5
    intercept_error = pcov[1,1]**0.5
    return(gradient,gradient_error,intercept,intercept_error)

best_fit_params()
```

    <IPython.core.display.Javascript object>

    (2.0284648194758716,
     0.05231193382948591,
     -0.9474963040606827,
     3.4085972274383725)

```python
'''TEST CELL- DO NOT DELETE'''
assert isinstance(best_fit_params(), (list, tuple, np.ndarray)), \
    'Please make sure that best_fit_params() returns a list/array of floats.'

```

    <IPython.core.display.Javascript object>

```python
'''TEST CELL- DO NOT DELETE'''
```

    'TEST CELL- DO NOT DELETE'

```python
'''TEST CELL- DO NOT DELETE'''
```

    'TEST CELL- DO NOT DELETE'

```python
'''TEST CELL- DO NOT DELETE'''
```

    'TEST CELL- DO NOT DELETE'

## Question 5: Error bars from a $\chi^2$ minimisation
|--|Unweighted | Weighted |
| --- | --- | --- |
| Gradient | (1.9$\pm$0.2)mV/Hz | (2.03$\pm$0.05)mV/Hz |
| Intercept | (0$\pm$1)x10mV | (-1$\pm$3)mV |

Required:
<bf>
>(i) For the data set of the previous question, write your own code to perform a $\chi^2$ minimisation. You may use the mininmize function, but not the curve_fit function.
<bf>
>(ii) Verify that $\chi^{2}_{\rm{min}}$ is obtained for the same values of the parameters as are listed in the table above.
<bf>
>(iii) By following the procedure of $\chi^2\rightarrow\chi^2_{\rm min}+1$ outlined in Section 6.5 in Hughes and Hase and the figure above, check that the error bars for $m$ and $c$ are in agreement with the table above.  Include explicitly the first 5 steps of the procedure shown in the figure above for the calculation of the slope.

### (i) For the data set of the previous question, write code to perform a $\chi^2$ minimisation.

```python
def func_solve(t):
    kai2 = 0
    for i in range(len(frequencies)):
        kai2 += (voltages[i] - frequencies[i] * t[0] - t[1]) ** 2 / (voltage_errors[i] ** 2)
    return kai2
m_init = 1
c_init = 1
index = [m_init, c_init]
res = minimize(func_solve, index)
print(res.fun)
```

    9.115903588053289

### (ii) Verify that $\chi^{2}_{\rm{min}}$ is obtained for the same values of the parameters as are listed in the table above.

```python
data = pd.read_csv('linear_regression.csv')

def five_ii():
    '''Your function must return the gradient and intercept'''
    gradient = 2.03
    intercept = -1
    kai2 = 0
    for i in range(len(frequencies)):
        kai2 += (voltages[i] - frequencies[i] * gradient - intercept) ** 2 / (voltage_errors[i] ** 2)
    print(kai2)
    # YOUR CODE HERE
    return(gradient,intercept)

five_ii()
```

    9.11721111111111

    (2.03, -1)

```python
'''TEST CELL- DO NOT DELETE'''
assert isinstance(five_ii(), (list, tuple, np.ndarray)), \
    'Please make sure that five_ii() returns a list/array of two floats.'
```

    9.11721111111111

### (iii) By following the procedure of $\chi^2\rightarrow\chi^2_{\rm min}+1$ outlined in Section 6.5 in Hughes and Hase and the figure above, check that the error bars for $m$ and $c$ are in agreement with the table above.  Include explicitly the first 5 steps of the procedure shown in the figure above for the calculation of the slope.

```python
def func_solvem(m, arg):
    kai2 = 0
    for i in range(len(frequencies)):
        kai2 += (voltages[i] - frequencies[i] * m - arg[0]) ** 2 / (voltage_errors[i] ** 2)
    return kai2 - (arg[1] + 1)

def func_solvec(c, arg):
    kai2 = 0
    for i in range(len(frequencies)):
        kai2 += (voltages[i] - frequencies[i] * arg[0] - c) ** 2 / (voltage_errors[i] ** 2)
    return kai2 - (arg[1] + 1)

def func_solvem2(c, arg):
    kai2 = 0
    for i in range(len(frequencies)):
        kai2 += (voltages[i] - frequencies[i] * arg[0] - c) ** 2 / (voltage_errors[i] ** 2)
    return kai2

def func_solvec2(m, arg):
    kai2 = 0
    for i in range(len(frequencies)):
        kai2 += (voltages[i] - frequencies[i] * m - arg[0]) ** 2 / (voltage_errors[i] ** 2)
    return kai2

def five_iii():
    error_m = 0
    error_c = 0
    chi_now = res.fun
    m = res.x[0]
    c = res.x[1]
    print(m)
    for i in range(10):
        m_fixed = fsolve(func_solvem, m, [c, chi_now])
        res_now = minimize(func_solvem2, c, [m_fixed])
        m = m_fixed
        c = res_now.x[0]
    error_m = abs(res.x[0] - m)

    m = res.x[0]
    c = res.x[1]
    chi_now = res.fun
    for i in range(10):
        c_fixed = fsolve(func_solvec, c, [m, chi_now])
        res_now = minimize(func_solvec2, m, [c_fixed])
        c = c_fixed
        m = res_now.x[0]

    error_c = abs(res.x[1] - c)
    # YOUR CODE HERE
    return(error_m,error_c)

five_iii()
```

    2.0284647949240395
    [2.02851608]
    [2.05400358]
    [2.0709015]
    [2.07860821]
    [2.08035088]
    [2.08044287]
    [2.08044313]
    [2.08044313]
    [2.08044313]
    [2.08044313]
    [-0.97665438]
    [-2.63113595]
    [-3.72349197]
    [-4.21832112]
    [-4.32867791]
    [-4.3343402]
    [-4.33435514]
    [-4.33435514]
    [-4.33435514]
    [-4.33435514]

    /opt/conda/lib/python3.6/site-packages/scipy/optimize/minpack.py:163: RuntimeWarning: The iteration is not making good progress, as measured by the
      improvement from the last ten iterations.
      warnings.warn(msg, RuntimeWarning)

    (array([0.05197833]), array([3.38686011]))

```python
'''TEST CELL- DO NOT DELETE'''
assert isinstance(five_iii(), (list, tuple, np.ndarray)), \
    'Please make sure that five_ii() returns a list/array of two floats.'
```

## Question 6- Strategies for error bars

\begin{equation}
\begin{array}{lccccc}
\hline
x &1&2&3&4&5\\
y &51&103&150&199&251\\
\alpha_{y}   &1&1&2&2&3\\
\hline
x &6&7&8&9&10\\
y &303&347&398&452&512\\
\alpha_{y}   &3&4&5&6&7\\
\hline
\end{array}
\end{equation}

Required:
<bf>
>(i) Calculate the weighted best-fit values of the slope, intercept, and their uncertainties.
<bf>
>(ii) If the data set had been homoscedastic, with all the errors equal, $\alpha_{y}=4$, calculate the weighted best-fit values of the slope, intercept, and their uncertainties.
<bf>
>(iii) If the experimenter took greater time to collect the first and last data points, for which $\alpha_{y}=1$, at the expense of all of the other data points, for which $\alpha_{y}=8$, calculate the weighted best-fit values of the slope, intercept, and their uncertainties.
<bf>
>(iv) Comment on your results.
<bf>
>(v) Plot the original data from the table including error bars. On the same plot, show the fitted function calculated in (i).

### (i) Calculate the weighted best-fit values of the slope, intercept, and their uncertainties.

```python
xs = [1,2,3,4,5,6,7,8,9,10]
ys = [51,103,150,199,251,303,347,398,452,512]
errors = [1,1,2,2,3,3,4,5,6,7]

def f(x,a,b):
    return(a*x+b)

def six_i():
    errors = [1,1,2,2,3,3,4,5,6,7]
    slope = 0
    intercept = 0
    slope_uncertainty = 0
    intercept_uncertainty = 0
    popt, pcov = curve_fit(f, xs, ys, sigma = errors)
    slope = popt[0]
    intercept = popt[1]
    slope_uncertainty = pcov[0,0]**0.5
    intercept_uncertainty = pcov[1,1]**0.5
    # YOUR CODE HERE
    return(slope,intercept,slope_uncertainty,intercept_uncertainty)

six_i()
```

    (49.89790138742879, 1.726770709105078, 0.332201663924493, 1.0360068862703642)

```python
'''TEST CELL- DO NOT DELETE'''
assert isinstance(six_i(), (list, tuple, np.ndarray)), \
    'Please make sure that six_i() returns a list/array of four floats.'
```

### (ii) If the data set had been homoscedastic, with all the errors equal, $\alpha_{y}=4$, calculate the weighted best-fit values of the slope, intercept, and their uncertainties.

```python
def six_ii():
    errors = [4,4,4,4,4,4,4,4,4,4]
    slope = 0
    intercept = 0
    slope_uncertainty = 0
    intercept_uncertainty = 0
    # YOUR CODE HERE
    popt, pcov = curve_fit(f, xs, ys, sigma = errors)
    slope = popt[0]
    intercept = popt[1]
    slope_uncertainty = pcov[0,0]**0.5
    intercept_uncertainty = pcov[1,1]**0.5
    return(slope,intercept,slope_uncertainty,intercept_uncertainty)

six_ii()
```

    (50.47272726877133,
     -0.9999999782423065,
     0.45720434535997834,
     2.836878357799031)

```python
'''TEST CELL- DO NOT DELETE'''
assert isinstance(six_ii(), (list, tuple, np.ndarray)), \
    'Please make sure that six_ii() returns a list/array of four floats.'
```

### (iii) If the experimenter took greater time to collect the first and last data points, for which $\alpha_{y}=1$, at the expense of all of the other data points, for which $\alpha_{y}=8$, calculate the weighted best-fit values of the slope, intercept, and their uncertainties.

```python
def six_iii():
    errors = [1,8,8,8,8,8,8,8,8,1]
    slope = 0
    intercept = 0
    slope_uncertainty = 0
    intercept_uncertainty = 0
    # YOUR CODE HERE
    popt, pcov = curve_fit(f, xs, ys, sigma = errors)
    slope = popt[0]
    intercept = popt[1]
    slope_uncertainty = pcov[0,0]**0.5
    intercept_uncertainty = pcov[1,1]**0.5
    return(slope,intercept,slope_uncertainty,intercept_uncertainty)

six_iii()
```

    (51.198747152729084,
     -0.45340345705795704,
     0.13868714281431524,
     0.9769093572145309)

```python
'''TEST CELL- DO NOT DELETE'''
assert isinstance(six_iii(), (list, tuple, np.ndarray)), \
    'Please make sure that six_iii() returns a list/array of four floats.'
```

### (iv) Comment on your results.

If the first point and the last point have very small error, the straight line is roughly fixed, so the intercept uncertainty and slope uncertainty are very small. Though the straight line may not fit with the true data, maybe with a lot of errors, but the uncertainty are very small. In my opinion, if two points, no matter whether they are the first two points or last two points, have no error, the line will be fixed without any uncertainty.

### (v) Plot the original data from the table including error bars. On the same plot, show the fitted function calculated in (i).

```python
plt.figure()
slope, intercept,_,_ = six_i()
ys_pred = []
for x in xs:
    ys_pred.append(slope * x + intercept)
plt.plot(xs, ys_pred, 'r-', label='fit: a=%5.3f, b=%5.3f' % tuple(popt))
plt.errorbar(xs, ys, yerr=errors, label='point with error bar', ls = '')# YOUR CODE HERE
plt.legend()
plt.show()
```

    <IPython.core.display.Javascript object>

```python

```

Core 1/data analysis/ipython notebook/Homework 2/Assignment 2.ipynb


# Sinais periódicos

Neste notebook avaliaremos a série de Fourier de alguns sinais periódicos.

Um sinal periódico, contínuo, é aquele para o qual a seguinte propriedade é válida:

\begin{equation}
x(t) = x(t \pm mT_p),
\end{equation}
ou seja, o valor do sinal no instante $t$ [s] é o mesmo para o instante $t \pm mT_p$ [s]. Dessa forma, o sinal se repete a cada
período $T_p$.

$T_p$ é o chamado período fundamental do sinal periódico. Neste caso, $x(t) = x(t \pm T_p) = x(t \pm 2T_p) = ... = x(t \pm kT_p)$.

Os sinais periódicos tem uma frequência fundamental $f_1 = \frac{1}{T_p}$ (ou frequência angular fundamental $\omega_1 = \frac{2\pi}{T_p}$.

A base para a estimativa do espectro de tais sinais é a ***Série de Fourier***, que nos afirma que qualquer sinal periódico pode ser representado por uma soma infinita de exponenciais complexas.

\begin{equation}
x(t) = \sum\limits_{k=-\infty}^{\infty}\tilde{c}_k \mathrm{e}^{\mathrm{j}k \omega_1 t}
\end{equation}
com os coeficientes dados por:

\begin{equation}
\tilde{c}_k = \frac{1}{T_p}\int_{T_p} x(t)\mathrm{e}^{-\mathrm{j}k\omega_1 t}\mathrm{d}t
\end{equation}

Para investigar o espectro vimos que podemos usar a informação contida em ***exatamente um período do sinal*** $x(t)$.

## Mas e se não conhecermos o período $T_p$?

Vamos investigar então o que acontece em dois casos:

### 1. Quando medirmos um número $r$ inteiro de períodos fundamentais.

Neste caso o período fundamental pode ser visto como $T^{'}_{p} = rT_p$ [s], e a frequência fundamental $f_{1}^{'}=\frac{1}{rT_p}$, $\omega_{1}^{'} = 2\pi f_{1}^{'}$ e os coeficientes de Fourier são dados por:

\begin{equation}
\tilde{c}_k = \frac{1}{rT_p}\int_{rT_p} x(t)\mathrm{e}^{-\mathrm{j}k\omega_{1}^{'} t}\mathrm{d}t
\end{equation}

Ora, se $r$ é um número inteiro, o sinal, $x(t)$ em si não mudou nada e o resultado dos coeficientes vai ser o mesmo. No entanto, $T^{'}_{p} = rT_p$, então a frequência fundamental $f_{1}^{'}$ é menor que a frequência fundamental original.

### 2. Quando medirmos um número $r$ não inteiro de períodos fundamentais.

Agora, se $r$ não for um número inteiro e, por exemplo:

Claramente, a integral que calcula os coeficientes $\tilde{c}_k$ será diferente. Podemos ver isso pela seguinte perspectiva: Ao assumirmos que $x(t)$ é periódico, então se medirmos por uma quantidade não interia de períodos, é como se o novo sinal periódico fosse:

### Assim, o que isso significa para o espectro $\tilde{c}_k$?

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
plt.rcParams.update({'font.size': 14})
from scipy import signal # some signals

fs = 100
Tp = 1
f1 = 1/Tp # Frequencia fundamental
w1 = 2*np.pi*f1
r = 17.6
t = np.arange(0, r*Tp, 1/fs) # vetor temporal

# onda quadrada
xt = signal.square(2 * np.pi * f1 * t)

# espectro original
ck = np.fft.fft(xt)
freq_vec = np.linspace(0, (len(xt)-1)*fs/len(xt), len(xt))

plt.figure()
plt.stem(freq_vec, np.abs(ck)/len(xt), '-k', basefmt=" ", use_line_collection=  True)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [-]')
plt.ylabel(r'$|\tilde{c}_k|$ [-]')
plt.xticks(np.arange(0, 11, 1))
plt.xlim((0, 10))
plt.tight_layout()
plt.show()

print('Frequency vector is: {}'.format(freq_vec))
```

```python

```

Aula 22 - SF e aspectos de resolucao/SF e resolucao do spk.ipynb


# Determinant formula from Cavalieri's principle

```python
# setup SymPy
from sympy import *
init_printing()
Vector = Matrix

# setup plotting
%matplotlib inline
import matplotlib.pyplot as mpl
from util.plot_helpers import plot_vec, plot_vecs, plot_line, plot_plane, autoscale_arrows
```

## Two dimentions

```python
a, b, c, d = symbols('a b  c d')

# Consider the volume of the parallelegram with sides:
u1 = Vector([a,b])
u2 = Vector([c,d])

# We can compute the volume of the parallelpiped by computing the deteminant of
A = Matrix([[a,b],
            [c,d]])
A.det()
```

###  Cavalieri's principle

Mathematically, we have

$$
  D(\vec{u}_1, \  \vec{u}_2)
  =
  D(\vec{u}_1 - \alpha \vec{u}_2, \ \vec{u}_2).
$$

```python
# choose alpha so A's top right entry will be zero
alpha = symbols('alpha')

alpha = b/d
A[0,:] = A[0,:] - alpha*A[1,:]
A
```

Copmputing the area is the same as computing the product of the entries on the diagonal:

```python
A[0,0]*A[1,1]
```

```python
simplify( A[0,0]*A[1,1] )
```

**Intuition:** the coefficient $\alpha$ encodes something very important about the "alternating multi-linear" structure that deteminants and cross products embody. In words, the choice of $\alpha$ and the resulting expression $a-\frac{bc}{d}$ correponds to what happens to the first component of $\vec{u}_1$ when we make it's second component zero.

(Yeah, I know, handwavy like crazy, but better than nothing.)

## Three dimentions

```python
a,b,c, d,e,f, g,h,i = symbols('a b c  d e f  g h i')

# Consider the volume of the parellelpiped with sides:
u1 = Vector([a,b,c])
u2 = Vector([d,e,f])
u3 = Vector([g,h,i])

# We can compute the volume of the parallelpiped by computing the deteminant of
A = Matrix([[a,b,c],
            [d,e,f],
            [g,h,i]])
A.det()
```

###  Cavalieri's principle

This principle leads to the following property of the determinant

$$
  D(\vec{u}_1, \ \vec{u}_2, \ \vec{u}_3)
  =
  D(\vec{u}_1,  \ \vec{u}_2- k \vec{u}_3, \ \vec{u}_3)
  =
  D(\vec{u}_1 -\ell\vec{u}_2 - m\vec{u}_3  , \  \vec{u}_2-k \vec{u}_3, \ \vec{u}_3).
$$

In particuler we make the following particular choice for the cofficients

$$
  D(\vec{u}_1, \vec{u}_2, \vec{u}_3)
  =
  D(\vec{u}_1 - \beta \vec{u}_3 - \gamma(\vec{u}_2 - \alpha\vec{u}_3), \  \vec{u}_2 - \alpha\vec{u}_3, \ \vec{u}_3).
$$

Choosing the right coefficients $\alpha$, $\beta$, and $\gamma$ can transform the matrix $A$ into a lower triangular form, which will make computing the determinant easier.

```python
alpha, beta, gamma = symbols('alpha beta gamma')

A
```

```python
# first get rid of f by subtracting third row from second row
alpha = f/i
A[1,:]= A[1,:] - alpha*A[2,:]
A
```

```python
# second get rid of c by subtracting third row from first row
beta = c/i
A[0,:]= A[0,:] - beta*A[2,:]
A
```

```python
# third get rid of b-ch/i by subtracting second row from first row
gamma = A[0,1]/A[1,1]
A[0,:]= A[0,:] - gamma*A[1,:]
A
```

Stargting from the first row, the volume of the determinant is proporitonal to the coeffieicnt `A[0,0]`. The are of the parallelepiped formd by the first two rows is `A[0,0]*A[1,1]` (we can ignore `A[1,0]`) and the overall volume is

```python
A[0,0]*A[1,1]*A[2,2]
```

```python
simplify( A[0,0]*A[1,1]*A[2,2] )
```

```python
# I still don't know how to motivate the recusive formula except to say it turns out that way...

# I tried to think about decomposing the problem into subparts,
#  but i cannot motivate det(Ai + Aj + Ak) != det(Ai) + det(Aj) + det(Ak)
#  where Ai is the same as A but with A[0,1] and A[0,2] set to zero.

```

```python
u1 = Vector([3,0,0])
u2 = Vector([2,2,0])
u3 = Vector([3,3,3])

plot_vecs(u1,u2,u3)
plot_vec(u1, at=u2, color='k')
plot_vec(u2, at=u1, color='b')
plot_vec(u1, at=u2+u3, color='k')
plot_vec(u2, at=u1+u3, color='b')
plot_vec(u1, at=u3, color='k')
plot_vec(u2, at=u3, color='b')
plot_vec(u3, at=u1, color='g')
plot_vec(u3, at=u2, color='g')
plot_vec(u3, at=u1+u2, color='g')

autoscale_arrows()
```

```python

```


# Getting started with TensorFlow

**Learning Objectives**
  1. Practice defining and performing basic operations on constant Tensors
  1. Use Tensorflow's automatic differentiation capability
  1. Learn how to train a linear regression from scratch with TensorFLow

In this notebook, we will start by reviewing the main operations on Tensors in TensorFlow and understand how to manipulate TensorFlow Variables. We explain how these are compatible with python built-in list and numpy arrays.

Then we will jump to the problem of training a linear regression from scratch with gradient descent. The first order of business will be to understand how to compute the gradients of a function (the loss here) with respect to some of its arguments (the model weights here). The TensorFlow construct allowing us to do that is `tf.GradientTape`, which we will describe.

At last we will create a simple training loop to learn the weights of a 1-dim linear regression using synthetic data generated from a linear model.

As a bonus exercise, we will do the same for data generated from a non linear model, forcing us to manual engineer non-linear features to improve our linear model performance.

```python
# Ensure the right version of Tensorflow is installed.
!pip freeze | grep tensorflow==2.0 || pip install tensorflow==2.0
```

```python
import numpy as np
from matplotlib import pyplot as plt
import tensorflow as tf
```

```python
print(tf.__version__)
```

## Operations on Tensors

### Variables and Constants

Tensors in TensorFlow are either contant (`tf.constant`) or variables (`tf.Variable`).
Constant values can not be changed, while variables values can be.

The main difference is that instances of `tf.Variable` have methods allowing us to change
their values while tensors constructed with `tf.constant` don't have these methods, and
therefore their values can not be changed. When you want to change the value of a `tf.Variable`
`x` use one of the following method:

* `x.assign(new_value)`
* `x.assign_add(value_to_be_added)`
* `x.assign_sub(value_to_be_subtracted`

```python
x = tf.constant([2, 3, 4])
x
```

```python
x = tf.Variable(2.0, dtype=tf.float32, name='my_variable')
```

```python
x.assign(45.8) # TODO 1
x
```

```python
x.assign_add(4) # TODO 1
x
```

```python
x.assign_sub(3) # TODO 1
x
```

### Point-wise operations

Tensorflow offers similar point-wise tensor operations as numpy does:

* `tf.add` allows to add the components of a tensor
* `tf.multiply` allows us to multiply the components of a tensor
* `tf.subtract` allow us to substract the components of a tensor
* `tf.math.*` contains the usual math operations to be applied on the components of a tensor
* and many more...

Most of the standard aritmetic operations (`tf.add`, `tf.substrac`, etc.) are overloaded by the usual corresponding arithmetic symbols (`+`, `-`, etc.)

```python
a = tf.constant([5, 3, 8]) # TODO 1
b = tf.constant([3, -1, 2])
c = tf.add(a, b)
d = a + b

print("c:", c)
print("d:", d)
```

```python
a = tf.constant([5, 3, 8]) # TODO 1
b = tf.constant([3, -1, 2])
c = tf.multiply(a, b)
d = a * b

print("c:", c)
print("d:", d)
```

```python
# tf.math.exp expects floats so we need to explicitly give the type
a = tf.constant([5, 3, 8], dtype=tf.float32)
b = tf.math.exp(a)

print("b:", b)
```

### NumPy Interoperability

In addition to native TF tensors, tensorflow operations can take native python types and NumPy arrays as operands.

```python
# native python list
a_py = [1, 2]
b_py = [3, 4]
```

```python
tf.add(a_py, b_py) # TODO 1
```

```python
# numpy arrays
a_np = np.array([1, 2])
b_np = np.array([3, 4])
```

```python
tf.add(a_np, b_np) # TODO 1
```

```python
# native TF tensor
a_tf = tf.constant([1, 2])
b_tf = tf.constant([3, 4])
```

```python
tf.add(a_tf, b_tf) # TODO 1
```

You can convert a native TF tensor to a NumPy array using .numpy()

```python
a_tf.numpy()
```

## Linear Regression

Now let's use low level tensorflow operations to implement linear regression.

Later in the course you'll see abstracted ways to do this using high level TensorFlow.

### Toy Dataset

We'll model the following function:

\begin{equation}
y= 2x + 10
\end{equation}

```python
X = tf.constant(range(10), dtype=tf.float32)
Y = 2 * X + 10

print("X:{}".format(X))
print("Y:{}".format(Y))
```

Let's also create a test dataset to evaluate our models:

```python
X_test = tf.constant(range(10, 20), dtype=tf.float32)
Y_test = 2 * X_test + 10

print("X_test:{}".format(X_test))
print("Y_test:{}".format(Y_test))
```

#### Loss Function

The simplest model we can build is a model that for each value of x returns the sample mean of the training set:

```python
y_mean = Y.numpy().mean()

def predict_mean(X):
    y_hat = [y_mean] * len(X)
    return y_hat

Y_hat = predict_mean(X_test)
```

Using mean squared error, our loss is:
\begin{equation}
MSE = \frac{1}{m}\sum_{i=1}^{m}(\hat{Y}_i-Y_i)^2
\end{equation}

For this simple model the loss is then:

```python
errors = (Y_hat - Y)**2
loss = tf.reduce_mean(errors)
loss.numpy()
```

This values for the MSE loss above will give us a baseline to compare how a more complex model is doing.

Now, if $\hat{Y}$ represents the vector containing our model's predictions when we use a linear regression model
\begin{equation}
\hat{Y} = w_0X + w_1
\end{equation}

we can write a loss function taking as arguments the coefficients of the model:

```python
def loss_mse(X, Y, w0, w1):
    Y_hat = w0 * X + w1
    errors = (Y_hat - Y)**2
    return tf.reduce_mean(errors)
```

### Gradient Function

To use gradient descent we need to take the partial derivatives of the loss function with respect to each of the weights. We could manually compute the derivatives, but with Tensorflow's automatic differentiation capabilities we don't have to!

During gradient descent we think of the loss as a function of the parameters $w_0$ and $w_1$. Thus, we want to compute the partial derivative with respect to these variables.

For that we need to wrap our loss computation within the context of `tf.GradientTape` instance which will reccord gradient information:

```python
with tf.GradientTape() as tape:
    loss = # computation
```

This will allow us to later compute the gradients of any tensor computed within the `tf.GradientTape` context with respect to instances of `tf.Variable`:

```python
gradients = tape.gradient(loss, [w0, w1])
```

We illustrate this procedure with by computing the loss gradients with respect to the model weights:

```python
# TODO 2
def compute_gradients(X, Y, w0, w1):
    with tf.GradientTape() as tape:
        loss = loss_mse(X, Y, w0, w1)
    return tape.gradient(loss, [w0, w1])
```

```python
w0 = tf.Variable(0.0)
w1 = tf.Variable(0.0)

dw0, dw1 = compute_gradients(X, Y, w0, w1)
```

```python
print("dw0:", dw0.numpy())
```

```python
print("dw1", dw1.numpy())
```

### Training Loop

Here we have a very simple training loop that converges. Note we are ignoring best practices like batching, creating a separate test set, and random weight initialization for the sake of simplicity.

```python
STEPS = 1000
LEARNING_RATE = .02
MSG = "STEP {step} - loss: {loss}, w0: {w0}, w1: {w1}\n"

w0 = tf.Variable(0.0)
w1 = tf.Variable(0.0)

for step in range(0, STEPS + 1):

    dw0, dw1 = compute_gradients(X, Y, w0, w1)
    w0.assign_sub(dw0 * LEARNING_RATE)
    w1.assign_sub(dw1 * LEARNING_RATE)

    if step % 100 == 0:
        loss = loss_mse(X, Y, w0, w1)
        print(MSG.format(step=step, loss=loss, w0=w0.numpy(), w1=w1.numpy()))

```

Now let's compare the test loss for this linear regression to the test loss from the baseline model that outputs always the mean of the training set:

```python
loss = loss_mse(X_test, Y_test, w0, w1)
loss.numpy()
```

This is indeed much better!

## Bonus

Try modelling a non-linear function such as: $y=xe^{-x^2}$

```python
X = tf.constant(np.linspace(0, 2, 1000), dtype=tf.float32)
Y = X * tf.exp(-X**2)
```

```python
%matplotlib inline

plt.plot(X, Y)
```

```python
def make_features(X):
    f1 = tf.ones_like(X)  # Bias.
    f2 = X
    f3 = tf.square(X)
    f4 = tf.sqrt(X)
    f5 = tf.exp(X)
    return tf.stack([f1, f2, f3, f4, f5], axis=1)
```

```python
def predict(X, W):
    return tf.squeeze(X @ W, -1)
```

```python
def loss_mse(X, Y, W):
    Y_hat = predict(X, W)
    errors = (Y_hat - Y)**2
    return tf.reduce_mean(errors)
```

```python
def compute_gradients(X, Y, W):
    with tf.GradientTape() as tape:
        loss = loss_mse(Xf, Y, W)
    return tape.gradient(loss, W)
```

```python
# TODO 3
STEPS = 2000
LEARNING_RATE = .02

Xf = make_features(X)
n_weights = Xf.shape[1]

W = tf.Variable(np.zeros((n_weights, 1)), dtype=tf.float32)

# For plotting
steps, losses = [], []
plt.figure()

for step in range(1, STEPS + 1):

    dW = compute_gradients(X, Y, W)
    W.assign_sub(dW * LEARNING_RATE)

    if step % 100 == 0:
        loss = loss_mse(Xf, Y, W)
        steps.append(step)
        losses.append(loss)
        plt.clf()
        plt.plot(steps, losses)

print("STEP: {} MSE: {}".format(STEPS, loss_mse(Xf, Y, W)))

plt.figure()
plt.plot(X, Y, label='actual')
plt.plot(X, predict(Xf, W), label='predicted')
plt.legend()
```

Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License


```python
import sympy as sm
```

```python
sm.init_printing()
```

```python
mb, mc, Ib, Ic, r, l, d = sm.symbols('m_b, m_c, I_b, I_c, r, l, d')
```

```python
mb
```

```python
Ib
```

```python
r+Ib**2
```

```python
sm.sin(r) + sm.sqrt(mb)
```

```python
t = sm.symbols('t')
```

```python
theta = sm.Function('theta')(t)
theta
```

```python
omega = sm.Function('omega')(t)
omega
```

```python
sm.diff(theta, t)
```

```python
sm.diff(theta**2+sm.sin(theta**2/2), t)
```

```python
h = l - sm.sqrt(l**2 - d**2)
h
```

```python

```


Logistic Regression with Gradient Ascent

Handwritten Digits Classification

## Gradient Ascent algorithm
From our [video](https://www.youtube.com/watch?v=TM1lijyQnaI&t=810s) on logistic regression, we derived the equation to update the logistic regression model parameters as:

\begin{equation}
\theta^{+} = \theta^{-} + \alpha (y_{i} - h(x_{i}) )\bar{x}
\end{equation}

This maximizes the following log likelihood function

\begin{equation}
J(x, \theta, y) = \sum_{i=1}^{m}y_i\log(h(x_{i})) + (1 - y_i)\log(1 - h(x_{i}))
\end{equation}

where our hypothesis is a sigmoid function
\begin{equation}
h(x_i) = \frac{1}{1 + e^{\theta^T \bar{x}}}
\end{equation}

### Batch gradient Ascent
```FOR j FROM 0 -> max_iteration:
    FOR i FROM 0 -> m:
        theta += (alpha) * (y[i] - h(x[i])) * x_bar
    ENDLOOP
ENDLOOP
```

## Multi-class Classificaton with one-vs-all (one-vs-rest)
If you have n-classes, we train n-classifiers and given a new data point we predict using all the classifiers and choose the one with the highest probability

```python
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
import numpy as np
import matplotlib.pyplot as plt
```

```python
digits = load_digits()

f, axarr = plt.subplots(3,3)
axarr[0,0].imshow(digits.images[0])
axarr[0,1].imshow(digits.images[1])
axarr[0,2].imshow(digits.images[2])
axarr[1,0].imshow(digits.images[3])
axarr[1,1].imshow(digits.images[4])
axarr[1,2].imshow(digits.images[5])
axarr[2,0].imshow(digits.images[6])
axarr[2,1].imshow(digits.images[7])
axarr[2,2].imshow(digits.images[8])
plt.show()
```

```python
class LogisticRegression():
    """Class for training and using a model for logistic regression"""

    def set_values(self, initial_params, alpha=0.01, max_iter=5000, class_of_interest=0):
        """Set the values for initial params, step size, maximum iteration, and class of interest"""
        self.params = initial_params
        self.alpha = alpha
        self.max_iter = max_iter
        self.class_of_interest = class_of_interest

    @staticmethod
    def _sigmoid(x):
        """Sigmoide function"""

        return 1.0 / (1.0 + np.exp(-x))

    def predict(self, x_bar, params):
        """predict the probability of a class"""

        return self._sigmoid(np.dot(params, x_bar))

    def _compute_cost(self, input_var, output_var, params):
        """Compute the log likelihood cost"""

        cost = 0
        for x, y in zip(input_var, output_var):
            x_bar = np.array(np.insert(x, 0, 1))
            y_hat = self.predict(x_bar, params)

            y_binary = 1.0 if y == self.class_of_interest else 0.0
            cost += y_binary * np.log(y_hat) + (1.0 - y_binary) * np.log(1 - y_hat)

        return cost

    def train(self, input_var, label, print_iter = 5000):
        """Train the model using batch gradient ascent"""

        iteration = 1
        while iteration < self.max_iter:
            if iteration % print_iter == 0:
                print(f'iteration: {iteration}')
                print(f'cost: {self._compute_cost(input_var, label, self.params)}')
                print('--------------------------------------------')

            for i, xy in enumerate(zip(input_var, label)):
                x_bar = np.array(np.insert(xy[0], 0, 1))
                y_hat = self.predict(x_bar, self.params)

                y_binary = 1.0 if xy[1] == self.class_of_interest else 0.0
                gradient = (y_binary - y_hat) * x_bar
                self.params += self.alpha * gradient

            iteration +=1

        return self.params

    def test(self, input_test, label_test):
        """Test the accuracy of the model using test data"""
        self.total_classifications = 0
        self.correct_classifications = 0

        for x,y in zip(input_test, label_test):
            self.total_classifications += 1
            x_bar = np.array(np.insert(x, 0, 1))
            y_hat = self.predict(x_bar, self.params)
            y_binary = 1.0 if y == self.class_of_interest else 0.0

            if y_hat >= 0.5 and  y_binary == 1:
                # correct classification of class_of_interest
                self.correct_classifications += 1

            if y_hat < 0.5 and  y_binary != 1:
                # correct classification of an other class
                self.correct_classifications += 1

        self.accuracy = self.correct_classifications / self.total_classifications

        return self.accuracy
```

```python
# split the data to training and test sets
digits_train, digits_test, digits_label_train, digits_label_test =\
train_test_split(digits.data, digits.target, test_size=0.20)
```

```python
# train a classifier for the ZERO digit
alpha = 1e-2
params_0 = np.zeros(len(digits.data[0]) + 1)

max_iter = 10000
digits_regression_model_0 = LogisticRegression()
digits_regression_model_0.set_values(params_0, alpha, max_iter, 0)

params =\
digits_regression_model_0.train(digits_train / 16.0, digits_label_train, 1000)

```

```python
# accuracy
digits_accuracy = digits_regression_model_0.test(digits_test / 16.0, digits_label_test)
print(f'Accuracy of prediciting a ZERO digit in test set: {digits_accuracy}')
```

```python
# train a classifier for the ONE digit
alpha = 1e-2
params_0 = np.zeros(len(digits.data[0]) + 1)

max_iter = 10000
digits_regression_model_1 = LogisticRegression()
digits_regression_model_1.set_values(params_0, alpha, max_iter, 1)

params =\
digits_regression_model_1.train(digits_train / 16.0, digits_label_train, 1000)
```

```python
#accuracy
digits_accuracy = digits_regression_model_1.test(digits_test / 16.0, digits_label_test)
print(f'Accuracy of prediciting a ONE digit in test set: {digits_accuracy}')
```

```python
# train a classifier for the TWO digit
alpha = 1e-2
params_0 = np.zeros(len(digits.data[0]) + 1)

max_iter = 10000
digits_regression_model_2 = LogisticRegression()
digits_regression_model_2.set_values(params_0, alpha, max_iter, 2)

params =\
digits_regression_model_2.train(digits_train / 16.0, digits_label_train, 1000)
```

```python
digits_accuracy = digits_regression_model_2.test(digits_test / 16.0, digits_label_test)
print(f'Accuracy of prediciting a TWO digit in test set: {digits_accuracy}')
```

```python
# train a classifier for the EIGHT digit
alpha = 1e-2
params_0 = np.zeros(len(digits.data[0]) + 1)

max_iter = 10000
digits_regression_model_8 = LogisticRegression()
digits_regression_model_8.set_values(params_0, alpha, max_iter, 8)

params =\
digits_regression_model_8.train(digits_train / 16.0, digits_label_train, 1000)
```

```python
digits_accuracy = digits_regression_model_8.test(digits_test / 16.0, digits_label_test)
print(f'Accuracy of prediciting a EIGHT digit in test set: {digits_accuracy}')
```

```python

```


# PC lab: classification

## 1 Introduction

In a binary classification setting, we are interested in assigning an observation 𝐱 to one of two possible classes, denoted by 𝑦. For example, maybe we would like to tell if a patient has a particular disease (y = 1) or not (y = 0), given certain symptoms 𝐱, in other words, we want to model $P(y|\mathbf{x})$, also called the *class posterior* or the *class-membership probability*. Two main approaches exist to model this probability: either directly using *discriminative models* or using *generative models*, which first model the joint probability $P(\mathbf{x},y)$ and obtain the class posterior using Bayes' rule.

### 1.1 Logistic regression

**Logistic regression** (LR) is the most fundamental example of a **discriminative** model. Just like linear regression, LR is a linear model. However, LR does not model the mean of a continuous outcome, but the logarithm of the [odds](https://en.wikipedia.org/wiki/Odds) of the probability $P(X)$:

\begin{equation}
log \frac{P(y|\mathbf{x})}{1-P(y|\mathbf{x})} = w_{0}x_{0} + w_{1}x_{1} + ... + w_{p}x_{p} = \mathbf{w^Tx}
\end{equation}

In practice,  we are really interested in the probability $P(y|\mathbf{x})$ and not in the odds of $P(y|\mathbf{x})$. To obtain this from the previous equation, we apply the *logistic function* (otherwise called a *sigmoid* function):

\begin{equation}
\phi(z) = \frac{1}{1 + e^{-z}} = \frac{e^{z}}{1+e^{z}}
\end{equation}

Applying this function on both sides of the LR equation, we get a model for $P(y|\mathbf{x})$ as follows:

\begin{equation}
P(y|\mathbf{x}) = \phi(\mathbf{w^Tx}) = \frac{1}{1 + e^{-\mathbf{w^Tx}}}
\end{equation}

If we want to classify a data point $\mathbf{x}$, we can calculate the probability of it being class $y$ with the above equation. A final classification is then made by assigning it to the class if the probability $P(y|\mathbf{x})$ exceeds a certain threshold. A typical threshold is 0.5.

 THOUGHT EXERCISE: What would happen to our predictions when we would choose a lower threshold, let's say 0.2? How would this affect the accuracy of our predictions? Can you think of a situation where we would want to do this?

#### 1.1.1 OPTIONAL EXTRA: Training a LR model: cross-entropy loss

Now that we know the logistic regression model to predict the probability of belonging to a certain class, all that remains is the question of how to find the weights of the model on a given set of training data. As always, this is the problem of minimizing a loss function to find an optimal set of weights. Where we used the mean squared error (MSE) for linear regression, we will use the **cross-entropy** loss function for LR. Minimizing the binary cross-entropy loss is equivalent to minimizing the negative log-likelihood of the data under a binomial distribution:

\begin{equation}
l_{log}^{i} = \frac{1}{n}\sum\limits_{i=1}^{n}-y_{i}log(P(y_i|\mathbf{x}_i))-(1-y_i)log(1-P(y_i|\mathbf{x}_i))
\end{equation}

Where $y_i$ is the class of data point $i$ and $p(\mathbf{x}_i)$ is the class-membership probability predicted by logistic regression for the observation $\mathbf{x}_i$. If we look at the cross-entropy loss **for a single data point** $l_{log}^{i}$, we can break it down in two parts:

\begin{equation}
l_{log}^{i} =
\begin{cases}
   -log(P(y_i|\mathbf{x}_i)) & \text{if} \ y_i = 1\\
   -log(1-P(y_i|\mathbf{x}_i)) & \text{if} \ y_i = 0
\end{cases}
\end{equation}

Or even more generally, the cross-entropy can be written as the log probability of predicting the correct class:

\begin{equation}
l_{log}^{i} = -log(P(\mathrm{correct} y_i|\mathbf{x}_i))
\end{equation}

From this, it should be clear that the cross-entropy loss will be larger for smaller values of $P(y_i|\mathbf{x}_i))$ if $y_i = 1$, and vice versa. If we visualize the cross-entropy loss for these two cases:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('seaborn-white')

p = np.arange(0.01,0.99,0.01) # Generate a range of predicted probabilities between zero and 1
l_0 = -np.log(p)  # cross-entropy loss if y = 1
l_1 = -np.log(1-p)

# Plot them
fig, ax = plt.subplots(figsize=(10,6))
ax.scatter(p,l_0, marker='.')
ax.scatter(p, l_1, marker='.')
ax.set_xlabel('Predicted class-membership probability $P$')
ax.set_ylabel('Cross-entropy loss')
ax.legend(['Cross-entropy loss of a sample when y is 1',
           'Cross-entropy loss of a sample when y is 0'])
plt.show()
```

#### 1.1.2 OPTIONAL EXTRA: Training a LR model: gradient descent

For linear regression, the solutions to the normal equations provide a convenient analytical solution to obtain the optimal set of model weights $\mathbf{w}$ on a set of training data. There is no such solution to find the optimal weights for a logistic regression model, so instead an optimization algorithm such as **gradient descent** is used to train a LR model.

Gradient descent is an iterative optimization algorithm that searches for the optimum of an objective function by making small changes to a set of optimization variables. Gradient descent (and its more complex variants) are widely used in machine learning to find the optimal set of model weights that minimize a certain loss function. Especially when there is no analytical solution for the weights available like for linear regression.

Generally, gradient descent uses the **gradient of the loss function with respect to the model weights** to perform updates to those weights in each iteration. At iteration $k+1$, the algoritm computes the gradient of a loss function $J(\mathbf{w})$ evaluated in the training data. Then, it performs an update to the current parameter values that is relative to the gradient multiplied with the learning rate $\gamma$, which is a constant:

\begin{equation}
\mathbf{w}_{k+1} = \mathbf{w}_{k} - \gamma\nabla{J(\mathbf{w}_{k})}
\end{equation}

Initially, the weights are often initialized with random draws from some distribution. The algorithm continues to do updates, until it converges or until some stopping criterion is reached.

In order to perform gradient descent to find the weights of a logistic regression model, we need to compute the gradient of the loss function with respect to the model parameters. Recall that, for a single data point, the cross-entropy loss function was as follows:

\begin{equation}
l_{log}^{i}(\mathbf{w}) = -y_{i}log(P(y_i|\mathbf{x}_i))-(1-y_i)log(1-P(y_i|\mathbf{x}_i)))
\end{equation}

Where $P(y_i|\mathbf{x}_i))$ is nothing else than the weighted sum of the inputs squashed through the sigmoid function:

\begin{equation}
P(y_i|\mathbf{x}_i)) = \phi(w_{0}x_{0i} + w_{1}x_{1i} + ... + w_{p}x_{pi})
\end{equation}

Before going on, let's first calculate the partial derivative of the sigmoid function:

\begin{equation}
\frac{\partial}{\partial z} \phi(z) = \frac{\partial}{\partial z} \frac{1}{1+e^{-z}} = \frac{e^{-z}}{(1+e^{-z})^2}
\end{equation}

We can rewrite this as follows:

\begin{equation}
\frac{e^{-z}}{(1+e^{-z})^2} = \frac{1 +e^{-z} -1}{(1+e^{-z})^2} = \frac{1}{1+e^{-z}} \Big( 1 - \frac{1}{1+e^{-z}}\Big) = \phi(z)(1 - \phi(z))
\end{equation}

With this result and by applying the chain rule, we can compute the partial derivative of the loss function with respect to the weight $w_j$. We will use the symbol $z$ to denote the weighted sum of the features (i.e., the input for the logistic function) and drop the subscript $i$ for clarity:

\begin{equation}
\frac{\partial l_{log}(\mathbf{w})}{\partial w_j} = \frac{\partial}{\partial w_j} \Big(-ylog(\phi(z))-(1-y)log(1-\phi(z)) \Big) \\ = \Big(  \frac{-y}{\phi(z)} + \frac{1-y}{1-\phi(z)}  \Big)\frac{\partial}{\partial w_j}\phi(z) \\ = \Big(  \frac{-y}{\phi(z)} + \frac{1-y}{1-\phi(z)}  \Big) \phi(z)(1-\phi(z))\frac{\partial}{\partial w_j}z
\end{equation}

Since $z = w_{0}x_{0} + w_{1}x_{1} + ... + w_{p}x_{p}$, $\frac{\partial}{\partial w_j}z$ is nothing more than $x_j$, so we can rewrite the above as:

\begin{equation}
\frac{\partial l_{log}(\mathbf{w})}{\partial w_j} = \Big( -y(1-\phi(z) + (1-y)\phi(z))\Big)x_j \\ = \big(  -y + \phi(z) \big)x_j = \big( \phi(z) - y \big)x_j
\end{equation}

With this partial derivative of the loss w.r.t $w_j$, we can write the update rule of the gradient descent algorithm for the $j^{th}$ weight:

\begin{equation}
w_{j,k+1} = w_{j,k} - \gamma(\phi(z_k)-y)x_{j}
\end{equation}

In other words, the algorithm will each time perform an update to the weight $w_{j}$ that is in proportion to the difference between the predicted probability of class membership in the previous iteration and the actual class. Makes sense! The entire gradient is simply the vector that contains the partial derivatives with respect to the entire weight vector $\mathbf{w}$, and in reality gradient descent acts on $\mathbf{w}$ and not on an individual weight $w_j$. Also, the gradient is typically not calculated for one data point, but evaluated over the entire training data set (unless the training set is prohibitively large, in that case mini-batch gradient descent is used).

In practice, software packages such as [scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegressionscikit-learn) do this optimization under the hood, so there is no need to implement it manually each time we want to use logistic regression.

### 1.2 Linear discriminant analysis

In contrast to the (discriminative) logistic regression model, linear discriminative analysis (LDA) is a **generative** model. Generative models are called as they are because they specify the class-conditional densities $P(\mathbf{x}|y)$, which allows to generate new observations $\mathbf{x}$ for a class $y$: for a given class $y$, we can sample from the distribution $P(\mathbf{x}|y)$.

The main assumption in LDA is that the data in each class is assumed to follow a multivariate Gaussian distribution. Furthermore, the assumption is made that the (co)variance within and between predictor variables is the same for all the classes. As such, only a limited number of parameters have to be estimated from the data to fully specify the class-conditional distributions: in the multivariate case, we need to estimate the class-specific mean vectors $\mathbf{\mu_{k}}$ and the common covariance matrix $\Sigma$. In the case of just one predictor (feature), this simplifies to estimating the mean of $x$ within each class and the variance $\sigma^2$. The priors $p(y=k)$ are simply estimated from the data by calculating the proportion of training instances in each class.

Once the class-conditional densities $P(\mathbf{x}|y)$ are modeled using LDA, predictions à la logistic regression $P(y|\mathbf{x})$ can be obtained by using Bayes' rule:

\begin{equation}
P(y=k|\mathbf{x}) = \frac{P(\mathbf{x}|y=k)P(y)}{P(\mathbf{x})} = \frac{P(\mathbf{x}|y=k)P(y=k)}{\sum_{l=1}^{K}P(\mathbf{x}|y=l)P(y=l)}
\end{equation}

where all probabilities $P(\mathbf{x}|y=l)$ is obtained by the LDA model and all probabilities $P(y=l)$ are simply observed from the class frequency in the dataset (e.g.: 70% of the training dataset samples belongs to class $l$ so: $P(y=l) = 0.70$)

## 1.3 Classification metrics

As you may imagine, the classification accuracy (the percentage of samples predicted correctly), may not always be the most appropriate metric to evaluate the performance of a classifier. In particular, the following two scenarios seem to require a different approach:

* In some applications, we want to avoid false negatives at the cost of more false positives. For instance, we want to make sure that we do not falsely predict that a patient is healthy when in reality he has a disease. When we allow for more false positives, the classification accuracy will go down.
* When the data is very imbalanced, a *null classifier* that always predicts the same class will achieve a high classification accuracy.

A first step to get some more insight in the performance of a classifier is to create a confusion matrix, which gives an overview of the kind of mistakes we have made during classification. A confusion matrix might look like this:

The concepts of *specificity* and *sensitivity* are better suited in the previous example scenarios. These are *class-specific* metrics: the sensitivity is the fraction of (true) positives that is correctly identified (as positive). The specificity is the fraction of (true negatives) that is correctly classified (as negative). A classifier might predict nearly every sample as positive, hence obtaining a high sensitivity but a very low specificity. One way to change this is to change the threshold for classification: by selecting a higher threshold, we will predict more samples as negatives, hence the sensitivity will go down because we will now predict some positives as being negative. However, the specificity will improve, because we now only predict samples as positive if the model outputs a higher probability for it being positive.

The apparent trade-off between sensitivity and specificity is summarized by the receiver operating characteristic (ROC) curve. A ROC curve is a plot of the true positive rate (sensitivity) versus the false positive rate (one minus the specificity) for various thresholds. It typically looks like this:

It illustrates the trade-off between sensitivity and specificity as we vary the threshold. A good measure to evaluate a classifier is the **area under the ROC curve (AUROC or AUC)**. A classifier that does random guesses has an AUC of 0.5 (dotted line in the plot), so if we design a classifier for a certain problem we want it to achieve an AUC higher than 0.5.

Check this [blog](https://datascienceplus.com/interpretation-of-the-auc/) to read a bit more considering the interpretation of the AUROC.

## 2 Exercises

### 2.1 Breast cancer dataset

In this set of exercises, we will use the [Breast Cancer Wisconsin (Diagnostic) Data Set](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29). The dataset contains information on the disease status of 569 breast cancer patients: they were either diagnosed with a malign (status M) or with a benign (status B) tumor.

For each patient, the dataset also contains 30 features that represent statistics of the cell nuclei present in images taken after [fine needle aspirate tissue samples](https://en.wikipedia.org/wiki/Fine-needle_aspiration). These 30 features are the mean, standard deviation and the maximum of 10 measurements on the cell nuclei:
- radius
- texture
- perimeter
- area
- smoothness
- compactness
- concavity
- concave points
- symmetry
- fractal dimension

**Based on these feature of the cell nuclei, we would like to predict whether a patient has a malign or a benign breast cancer tumor.** Let's download and read in the data:

```python
!wget https://raw.githubusercontent.com/gdewael/teaching/main/UGAIN2021/wdbc.data
```

```python
import pandas as pd
import numpy as np
data = pd.read_csv('./wdbc.data', header=None, index_col=0, names=['Patient ID', 'status'] + list(np.arange(1,31,1)))
status = data['status']
data.head()
```

First, let's look at the distribution of the disease status:

```python
pd.value_counts(data['status']).plot(kind='bar');
```

There are about 350 benign cases and roughly 200 malign cases. This is a fairly balanced dataset.

THOUGHT EXERCISE: Suppose that the dataset was unbalanced, with 525 B cases and only 25 M cases. Would you still use accuracy to evaluate the model?

In these exercises, we will use the [scikit-learn](https://scikit-learn.org/stable/) implementations of [LR](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) and [LDA](https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html). In order to fit models, we will first need to encode the labels 'Benign' (B) and 'Malign' (M) to a binary encoder:

```python
from sklearn.preprocessing import LabelEncoder
encoder = LabelEncoder().fit(status)
encoder.classes_  # 'B' will become class 0, 'M' will become class 1
```

```python
y = encoder.transform(status)
x = data.drop('status', axis=1).values # Drop the disease status from the dataframe, convert to numpy array
y[:100]
```

EXERCISE: Using scikit-learn, split the data in a 80% training and a 20% test set. Fit a logistic regression model and evaluate training and testing accuracy.

Use [this method](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) for train-test splitting and [this implementation](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) to perform logistic regression. Accuracy can be computed by using the score() method of each model.

```python

```

EXERCISE: Now do the same with LDA. Do you split your dataset again in a new random training and testing split? Think about fair evaluation/comparison between the two models.

Use [this implementation](https://scikit-learn.org/stable/modules/generated/sklearn.discriminant_analysis.LinearDiscriminantAnalysis.html) for LDA.

```python

```

Depending on your data split, LDA or LR will perform better. The model that we prefer is hence determined by a random data splitting, a behavior that we do not want. One way to avoid this is to perform [cross-validation](https://machinelearningmastery.com/k-fold-cross-validation/), we will return on this topic later in the course.

EXERCISE: Use your preferred (LDA or LR) model to predict the class probabilities $P(y|\mathbf{x})$ and the classes for the training data. Use the predict_proba() method to generate the predicted probabilities and .predict() to genereate the predicted classes. Take note of the output shape of predict_proba(), what do the columns mean? Use the code below to plot the two against each other. Which data points are most likely to be misclassified?

```python
# ** solution **
predicted_class_probabilities = ...
predicted_classes = ...

#** solution **
```

```python
import matplotlib.pyplot as plt

misclassified = predicted_classes !=  y_train

colors = ['#b2182b' if wrong else '#2166ac' for wrong in misclassified ]
fig, ax = plt.subplots(figsize=(8,6))
test_jitter = np.random.normal(scale=0.1, size=len(predicted_class_probabilities))
ax.scatter(test_jitter, predicted_class_probabilities,
           marker='.', s=25, color=colors, alpha=0.75)
ax.get_xaxis().set_ticks([]);
ax.set_ylabel('Predicted class probabilies').set_fontsize(20)
ax.set_title('Correctly classified samples in blue', size=24)
plt.show()
```

Clearly, the misclassified points are those points where the predicted probability of class membership is rather close to 0.5.

EXERCISE: Use the predicted class probabilities from the previous exercise to compute the ROC AUC score on the test dataset. Then, use the code below to plot the ROC curves of both models.

```python
# plot the roc curves
from sklearn.metrics import roc_curve, roc_auc_score

# Use test data!
#########
predicted_class_probabilities_LDA = ...
predicted_class_probabilities_LR = ...

AUC_LDA = roc_auc_score(..., ...)
AUC_LR = roc_auc_score(..., ...)

##############
```

```python
roc_LDA = roc_curve(y_test, predicted_class_probabilities_LDA)
roc_LR = roc_curve(y_test, predicted_class_probabilities_LR)

fig, ax = plt.subplots(figsize=(15,10))
ax.plot([0, 1], [0, 1], '--')
ax.plot(roc_LR[0], roc_LR[1])
ax.plot(roc_LDA[0], roc_LDA[1])
ax.set_xlabel('1-specificity').set_fontsize(18)
ax.set_ylabel('sensitivity').set_fontsize(18)

ax.legend(['Random classifier - AUC 0.5',
           'LR - test data AUC {}'.format(AUC_LR),
           'LDA - test data AUC {}'.format(AUC_LDA)],
         fontsize='large')
plt.show()
```

 EXERCISE:
We have succesfully implemented a pipeline going from data exploration to model construction and evaluation (including comparison) for the wisconsin breast cancer dataset. For the end of the PC lab, we introduce a new dataset: [Job change analytics](https://www.kaggle.com/arashnic/hr-analytics-job-change-of-data-scientists). Implement the data exploration and model construction pipeline for this dataset by reusing code from previous exercises. The goal of this exercise is to show that every dataset has its own unique set of characteristics and challenges.

Some information about the dataset: A company which is active in Big Data and Data Science wants to hire data scientists among people who successfully pass some courses which conduct by the company. Many people signup for their training. Company wants to know which of these candidates really want to work for the company after training or looking for a new employment because it helps to reduce the cost and time as well as the quality of training or planning the courses and categorization of candidates. Information related to demographics, education, experience are in hands from candidates signup and enrollment. **Hence, the input features are details about persons who took data science courses, and the output of interest (target) is whether those persons are actually looking for a new job.**

More information [here](https://www.kaggle.com/arashnic/hr-analytics-job-change-of-data-scientists).

- What do you do with missing values in the dataset?: You can throw samples with at least one missing feature value out, or you can [impute](https://scikit-learn.org/stable/modules/impute.html) those missing values.
- What about class imbalance?
- Many features are categorical, how will you encode these?

Since there is no single right way to answer the above questions, we will not provide a solution to this exercise. Experiment at free will! If you have any questions, you can always mail the teaching assistant.

Below is some preliminary data exploration. Use this to further explore the data and make design choices.

```python
!wget https://raw.githubusercontent.com/gdewael/teaching/main/UGAIN2021/HR_analytics.csv
```

```python
data = pd.read_csv('HR_analytics.csv')
data.shape
```

```python
data.head()
```

```python
# number of NaN (missing values) per feature:
pd.isnull(data).sum(0)
```

```python
# number of unique values per feature:
for k in data.columns:
    print(k, len(np.unique(data[k][~pd.isnull(data[k])])))
```

```python
# Look at what the unique values are for a specific feature:
feature = 'enrolled_university'
np.unique(data[feature][~pd.isnull(data[feature])])
```

```python
#################### YOUR OWN CODE HERE #######################
```

### 2.2 OPTIONAL EXTRA: Multiclass classification

Some classifiers inherently support multiclass classification as part of their design, for example, in LDA, the class-conditional density $P(\mathbf{x}|y)$ is modeled for every class $y$ and can be converted to predictions using Bayes' rule. For logistic regression, a natural extension to the multi-class case is [multinomial logistic regression](https://en.wikipedia.org/wiki/Multinomial_logistic_regression). Check out the [scikit-learn page on multiclass prediction](https://scikit-learn.org/stable/modules/multiclass.html) for more information. In principle, any classifier can be made multiclass using following methods:

#### One-versus-one classification

One-versus-one classification is another approach to a multiclass classification problem. For a K-class problem, the strategy consists of training $\frac{K(K-1)}{2}$ classifiers. Each of these classifiers much learn to distinguish to classes. One the classifiers are trained, a voting scheme is applied to make a prediction for an unseen data point: each classifier has to decide between two possible classes. The final predicted class is that class that gets the largest number of votes.

#### One-versus-all classification

In one-versus-all (OvA) classification (also called one-versus-rest), a single classifier is trained per class, with the samples of that class as positive samples and all other samples as negatives. The strategy proceeds as follows for a K-class classification problem:

**Inputs:**
* a classification algorithm L (learner)
* feature matrix $\mathbf{X}$
* label vector y where $y_i \in {1,...,K}$

**Procedure:**
for each k in {1,...,K}:
* construct a new label vector z where $z_i$ is 1 if $y_i$ = k and 0 otherwise
* train L on $\mathbf{X}$ to obtain a classifier $f_k$. The classifier should return class probabilities and not hard labels.

**Returns**
A list of trained classifiers $f_k$ for each  k in {1,...,K}

To make predictions for a new sample $\mathbf{x}$, the $k$ classifiers are applied to $\mathbf{x}$ and the final predicted label is the label that is predicted with the highest confidence (probability):

$\hat{y} = \underset{k \in {1,...,K}}{\mathrm{argmax}} \, f_k(\mathbf{x})$

Let's simulate a toy dataset with three classes and two features, and split it in training and test data:

```python
from sklearn.datasets import make_blobs
from sklearn.model_selection import train_test_split

X, y = make_blobs(n_samples=1000, centers= [[-2.5, 0], [0, 1], [3.5, -1]], random_state=42)

#train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
```

```python
# Make the plot
fig, ax = plt.subplots(figsize=(15,10))
colors=['#66c2a5', '#fc8d62', '#8da0cb']
for i, color in enumerate(colors):
    idx_train = np.where(y_train==i)
    idx_test = np.where(y_test==i)
    plt.scatter(X_train[idx_train,0], X_train[idx_train,1], c=color, edgecolor='black', s=30)
    plt.scatter(X_test[idx_test,0], X_test[idx_test, 1],c='white', edgecolor=color, s=70)

ax.legend(['Class 1 - train',
           'Class 1 - test',
           'Class 2 - train',
           'Class 2 - test',
           'Class 3 - train',
           'Class 3 - test']);

ax.set_xlabel('Feature 1');
ax.set_ylabel('Feature 2');
ax.set_title('Toy dataset for multiclass classification').set_fontsize(20);
```

OPTIONAL EXTRA EXERCISE: Look again at the documentation of LDA and Logistic Regression. What options do you need to specify to train these models on a multi-class problem? Train a model on this toy dataset

```python
######### exercise start ############

# instantiate model ...

# fit ...

predicted_classes = ...

classification_accuracy = np.round(np.mean(y_test == predicted_classes)*100,2)

######### exercise stop ############

# Visualize the predictions

fig, ax = plt.subplots(figsize=(15,10))
colors=['#66c2a5', '#fc8d62', '#8da0cb']

for i, color in enumerate(colors):
    idx_train = np.where(y_train==i)
    idx_test = np.where(y_test==i)

    plt.scatter(X_train[idx_train,0], X_train[idx_train,1], c=color, edgecolor='black', s=30)
    plt.scatter(X_test[idx_test,0], X_test[idx_test, 1],c='white', edgecolor=color, s=70)

ax.legend(['Class 1 - train',
           'Class 1 - test',
           'Class 2 - train',
           'Class 2 - test',
           'Class 3 - train',
           'Class 3 - test']);

# add predictions
for i, color in enumerate(colors):
    idx_predicted = np.where(predicted_classes==i)
    plt.scatter(X_test[idx_predicted,0], X_test[idx_predicted,1], c=color, marker='s', s=2)

ax.set_xlabel('Feature 1');
ax.set_ylabel('Feature 2');
ax.set_title('Toy dataset for multiclass classification - classification accuracy: {}%'.format(classification_accuracy)).set_fontsize(20);
```

### 2.3 OPTIONAL EXTRA: Data preprocessing for text inputs (example with DNA)

Since machine learning models expect numbers as inputs, not words, we need to do some preprocessing first to make them compatible with ML models. A very common feature representation for text sequences is called the **Bag of Words**. It consists of listing all the possible tokens that might occur in your text. A token might be one word, or a combination of two or multiple words. The feature representation is then a matrix with the count of the tokens for each text instance. The simplest transformation is to just look at the counts of individual tokens: the 1-grams. However, it might be interesting to count combinations of two or more words as well. Consider the following example, where we have 2 example input sample sentences:

```python
from sklearn.feature_extraction.text import CountVectorizer

data = ['this is real life',
       'is this real life']
print('data:\n', data[0], '\n', data[1])
vectorizer = CountVectorizer(analyzer='word', ngram_range=(1,2))

analyze = vectorizer.build_analyzer()
print('vocabulary:\n', analyze(data[0]))

features = vectorizer.fit_transform(data)
print('final feature representation:\n', features.toarray())
```

Note that by default, the vectorizer returns the features as a sparse array: in most applications, each row of the feature matrix will contain a lot of zeros. For this exercise, you can convert it to a regular numpy array with the ```toarray()``` method.

If we restrict ourselves to look at 1-grams, both sentences above are transformed into the exact same feature representation. However, suppose we want to do sentiment analysis and classify whether a sentence contains a question or not. Then it might be useful to look at bigrams as well, as above

```python
vectorizer = CountVectorizer(analyzer='word', ngram_range=(1,1))
analyze = vectorizer.build_analyzer()
analyze(data[0])
features = vectorizer.fit_transform(data)
print('Only 1-grams:')
print('final feature representation:\n', features.toarray())
```

When we consider bigrams, the feature representation for both sentences is no longer exactly the same. Maybe the features that represent 'this is' and 'is this' could be helpful here.

Still the bag-of-words features completely ignores where every n-gram was in the sentence. Another way to take this into account is to assign $v$ dummy features for every location in the sentence, with $v$ being the vocabulary size (number of possible different words). Every first, second, third and fourth word get assigned 1 feature for every possible word in that location (out of 4 possible words in the vocabulary). This is called [one-hot encoding](https://en.wikipedia.org/wiki/One-hot#Natural_language_processing)
This requires the sentences to be all of the same length, for example, 4 words as in the 2 sentences above.

```python
data_splitwords = [sentence.split(' ') for sentence in data]
print('data splitted by words:')
print(data_splitwords)
vocab = list(set([word for sentence in data_splitwords for word in sentence]))
print('vocabulary:')
print(vocab)
from sklearn.preprocessing import OneHotEncoder

enc = OneHotEncoder(handle_unknown='ignore',
                    categories=[vocab]*len(data_splitwords[0]))

features = enc.fit_transform(data_splitwords)
print('resulting feature representation:')
print(features.toarray())
```

If it is not immediately clear what the features mean in this example: the first four features represent which word is the first word in each sentence. The first and sentence has as first word 'this', this is the last word in the vocabulary, so of the first four features, the last is given by $1$. The second sentence has 'is' as the first word, which is the second word in the vocabulary, for this reason, the second column of the first four features is given by $1$ for this sentence. Both sentences have the same last two words 'real life', it can be seen that the last 8 features for both sentences are identical.

Armed with these encoding schemes, we can tackle a real problem. **DNA sequences** as inputs are peculiar because they are essentially text: following an alphabet containing only A, T, C or G. We will use data from a published [study](https://bmcbiol.biomedcentral.com/articles/10.1186/1741-7007-12-4) about sigma factors in E. coli. Some biological background information for those interested:

> The binding region of the transcription unit,  commonly called the promoter region, is known to play a key role  in the transcription rate of downstream genes. In Eubacteria, the sigma factor ($\sigma$) binds with the RNA  polymerase subunit ($\alpha\beta\beta'\omega$) to create RNA polymerase. Being part of the RNA polymerase holoenzyme, the sigma element acts as the connection between RNA polymerase and DNA. Depending on the sigma factor bound to the transcription unit, different binding regions of the enzyme with the DNA are observed. The data represents specific DNA regions of the *E. coli* genome to which one or more sigma factors bind.

In less-biological-jargon words: certain molecules called sigma factors control activity of genes, they do so by binding to the DNA close to the gene. If we can predict for an input DNA sequence if such a molecule will bind, we gain an understanding in the biological function of those genes and molecules.

The dataset contains 4724 DNA sequences. For each sequence, we have information on five target variables, each representing 5 sigma factors. Each time, the outcome is binary, indicating whether the sigma factor binds to the sequence or not. This is a **multi-label** problem, where an input can belong to any (multiple at once) of the classes, as opposed to only one possible class for each sample in multi-class. For simplicity in this exercise, we will focus only on one class: RPOD.

```python
!wget https://raw.githubusercontent.com/gdewael/teaching/main/UGAIN2021/promotor_list_exp_growth.csv
promotor_data = pd.read_csv('./promotor_list_exp_growth.csv', index_col=0)
promotor_data.head(20)

X = np.array(promotor_data.PROBE_SEQUENCE)
y = promotor_data['RPOD'].values

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=23)
```

 OPTIONAL EXTRA EXERCISE:
Use one of the encoding schemes to preprocess the DNA data. Construct a model and evaluate performance on the test set. Play around with different preprocessing values, and keep testing performances. What types of preprocessing lead to good performance for this dataset? In what cases are we overfitting? Think about fair evaluation: if we keep testing out configurations and evaluating on the test set, is our final test performance estimate fair? We can solve this by keeping an additional data split separate: training, validation and testing. Then we test different hyperparameters (different ways of preprocessing) on the validation set. We don't touch the test set until we have our final model. This is the only fair way of evaluating. For more information, see this [link](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets).

```python
######### exercise start ############

# instantiate encoder

# fit and transform data with encoder

# instantiate modle

# fit model

# evaluate

######### exercise stop ############
```

```python

```


# Binet's Formula

## Formula

Explicit formula to find the nth term of the Fibonacci sequence.

$\displaystyle F_n = \frac{1}{\sqrt{5}} \Bigg(\Bigg( \frac{1 + \sqrt{5}}{2} \Bigg)^n - \Bigg( \frac{1 - \sqrt{5}}{2} \Bigg)^n \Bigg)$

*Derived by Jacques Philippe Marie Binet, alreday known by Abraham de Moivre*

----

## Fibonacci Sequence

The Fibonacci sequence iterates with the next value being the sum of the previous two:

$F_{n+1} = F_n + F_{n-1}$

```python
def fib(n):
    a = b = 1
    for _ in range(n):
        yield a
        a, b = b, a + b

", ".join([str(x) for x in fib(20)])
```

    '1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765'

----
## Proof

### Fibonacci Ratios
The ratios of the Fibanacci sequence

$\displaystyle \lim_{n \rightarrow \infty} \frac{F_n}{F_{n-1}} = \varphi$ <br/>

$\displaystyle \frac{F_n}{F_{n-1}}$ converges to the limit $\phi$

```python
def fib_ratio(n):
    a = b = 1
    for _ in range(n):
        yield a/b
        a, b = b, a + b

", ".join(["{0:.6f}".format(x) for x in fib_ratio(20)])
```

    '1.000000, 0.500000, 0.666667, 0.600000, 0.625000, 0.615385, 0.619048, 0.617647, 0.618182, 0.617978, 0.618056, 0.618026, 0.618037, 0.618033, 0.618034, 0.618034, 0.618034, 0.618034, 0.618034, 0.618034'

### Compose as a Geometric Sequence

This sequence resembles a geometric sequence. Geometric sequences have terms in the form of $G_n = a \cdot r^n$ .

Therefore $F_{n+1} = F_n + F_{n-1} \implies a \cdot r^{n+1} = a \cdot r^n + a \cdot r^{n-1} \implies r^2 = r + 1$.

### Resolve Quadratic
Using the Quadratic formula we find r as $1+\varphi$, or $\displaystyle\frac{1 \pm \sqrt{5}}{2}$

Let's declare $G_n = \displaystyle\frac{1 + \sqrt{5}}{2}$, and $H_n = \displaystyle\frac{1 - \sqrt{5}}{2}$

```python
# x^2−x−1=0
from sympy import *
from sympy.plotting import plot
from sympy.solvers import solve

init_printing()
x = symbols('x')
exp = x**2 - x -1

plot(exp, (x, -2, 2))

answers = solve(x**2 -x -1, x)
[ratsimp(a) for a in answers]
```

### Conclusion
Although neither $G_n$ and $H_n$ conform to the Fibonacci sequence, through induction, $G_n - H_n$ does.

To find $a$, we can see that $F_0 = G_0 - H_0 = 0, and F_1 = G_1 - H_1 = 1 \implies a = \frac{1}{\sqrt{5}}$

----
## References
- [Art of Problem Solving - Binet's Formula][3]
- [Art of Problem Solving - Geometric Sequence][2]
- [Art of Problem Solving - Fibonacci Sequence][1]

 [1]: https://artofproblemsolving.com/wiki/index.php?title=Fibonacci_sequence
 [2]: https://artofproblemsolving.com/wiki/index.php?title=Geometric_sequence
 [3]: https://artofproblemsolving.com/wiki/index.php?title=Binet%27s_Formula