# Лабораторная работа 1
# Прикладная алгебра и численные методы
## Псевдообратная матрица, ортогонализация Грама-Шмидта, LU, QR, МНК, полиномы Лагранжа, Чебышева, сплайны, кривые Безье, нормы.

```
import numpy as np
import pandas as pd
import sympy
from sympy import S, latex, Matrix, I, simplify, expand
import scipy.linalg
import numpy.linalg
from google.colab import files
from copy import deepcopy
```

## Задание 1
Найти псевдообратную матрицу к матрице из файла ХХХ.xlsx

с помощью sympy, numpy и scipy

Записать в файл YYY.csv полученные псевдообратные матрицы. Каждый элемент матрицы должен быть в отдельной ячейке. Можно пользоваться средствами pandas, можно написать всою функцию для преобразования матрицы в строку, подходящую для записи в файл .csv.

```

```

### Задание 2
Построить LU разложение для матрицы Задания 1 с помощью sympy и scipy.

Выполнить проверку сравнением с исходной матрицей.

```

```

## Задание 3
 На основе LU разложения Задания 2 построить псевдообратную матрицу. Выполнить проверку по определению псевдообратной матрицы (должны выполняться два равенства)

```

```

## Задание 4
 Считать данные из файла ZZZ.xlsx.

Найти коэффициенты $a$, $b$, $c$, $d$ линейной регрессии $Y = at + bu + cv + d$, если
$Y$ - столбец 1, $t$  столбец 2, $u$ - столбец 3, $v$ - столбец 4.

Искать так: составить матрицу по данным задачи из файла, с помощью процесса ортогонализации Грама-Шмидта построить $QR$-разложение, по нему псевдообратную матрицу и по псевдообратной матрице найти коэффициенты.

```

```

## Задание 5
Найти как можно больше разных норм разности вектора данных $Y_{data}$ Задания 4 (столбец 1 из файла ZZZ.xlsx) и вектора значений, полученных по формуле линейной регрессии $Y = at + bu + cv + d$ с найденными в Задании 4 параметрами.

```

```

## Задание 6
Построить разложение SVD с помощью numpy для матрицы XXX. На основе полученного разложения составить псевдообратную матрицу.

```

```

## Задание 7
Для данных из файла PPP.xlsx построить полином Лагранжа

```

```

## Задание 8
Построить по данным Задания 7 кубический сплайн.

```

```

## Задание 9
Построить полином Чебышева степени не выше 4, приближающий $f(x) = 3x^6 - 4x^3 +x^2 - x + 2$.

```

```

## Задание 10
Соединить точки A и B кривой Безье так, чтобы касательные на концах совпадали с касательными к функции в точках  A и B.

 A и B точки  на графике функции  $f(x) = 3tg(x/6)$ с горизонтальными координатами  1 и 4.

```

```


# Assignment 02 Companion Notebook

This notebook contains some exercises to walk you through implementing the linear regression algorithm.  We'll pay special attention to debugging and visualization as we go along.

## A Toy Linear Regression Problem Revisited

As we discovered in the last assignment, the idea of a toy problem is very useful for validating that a machine learning algorithm is working as it is intended to.  Recall the following basic setup and role for a toy problem:

> Suppose you are given a learning algorithm designed to estimate some model parameters $\textbf{w}$ from some training data.
>
> 1.  Generate values for the model parameters $\mathbf{w}$ (e.g., set them to some known values or generate them randomly).  If you were applying your algorithm to real data, you would of course not know these parameters, but instead estimate them from data.  For our toy problem we'll proceed with values that we generate so we can test our algorithm more easily.
>
> 2.  Generate some training input data, $\mathbf{X}$, (random numbers work well for this).  Generate the training output data, $\mathbf{y}$, by applying the model with parameters $\mathbf{w}$.  For example, in a linear regression problem since $\mathbf{w}$ represents the regression coefficients, then we can generate each training label, $y_i$ as $y_i = \mathbf{x_i}^\top \mathbf{w}$.
>
> 3.  Run your learning algorithm on the synthesized training data $\mathbf{X}, \mathbf{y}$ to arrive at estimated values of the model parameters, $\hat{\mathbf{w}}$.
>
> 4.  Compare $\mathbf{w}$ and $\hat{\mathbf{w}}$ as a way of understanding whether your learning algorithm is working.

In the next code block, you'll see an example of a toy regression problem where we set $\mathbf{w} = \begin{bmatrix} 1 \\ 2 \end{bmatrix}$ and generate some training data.  To make the data a little more interesting, we'll add some noise to the training outputs (you saw this in the last assignment).  We'll also visualize the training data.

```python
%matplotlib inline
from mpl_toolkits.mplot3d import Axes3D
import numpy as np
import matplotlib.pyplot as plt

n_points = 50
X = np.random.randn(n_points,2)
w_true = np.array([1, 2])
# we'll apply a Gaussian noise with a standard deviation of 0.5 to the outputs to make it more interesting
y = X.dot(w_true) + np.random.randn(n_points,)*0.5

fig = plt.figure(figsize=(8,8))
ax = fig.add_subplot(111, projection='3d')

ax.scatter(X[:,0], X[:,1], y)
ax.set_xlabel('$x_1$')
ax.set_ylabel('$x_2$')
ax.set_zlabel('y')

plt.show()
```

### *Notebook Exercise 1 (40 minutes)*

Before implementing the algorithm you derived for computing $\mathbf{w}^\star$, let's create a visualization of the sum of squared errors as a function of the entries of $\mathbf{w}$. You should recall from exercises in the assignment document that the sum of squared errors for a particular value of $\mathbf{w}$ can be written as $\left(\mathbf{X}\mathbf{w} - \mathbf{y} \right)^\top \left (\mathbf{X}\mathbf{w} - \mathbf{y} \right)$

(a) Write a function called `sum_of_squared_errors` that takes the parameters `X`, `y`, and `w` and returns the sum of squared errors that this particular value of `w` incurs on the training data `X`, `y`.  We have included a skeletal outline of the function along with a unit test (SoftDes flashback!!).

(b) Run the visualization code in the cell below and interpret the resulting output.  What do the contour lines represent in the generated plot?  Based on the visualization, where is the optimal value of `w` (the one that minizes the squared error)?  Does this agree with the setup of the toy problem?  If not, why doesn't it match?

```python
def sum_of_squared_errors(X, y, w):
    """
    Return the sum of squared errors for the given training data (X, y) and
    model parameters w.

    >>> sum_of_squared_errors(np.array([[1, 4, 3],\
                                        [2, -1, 4]]),\
                              np.array([3, 4]),\
                              np.array([1, 2, 3]))
    289
    """
    # your code here
    pass

import doctest
doctest.testmod()
```

    **********************************************************************
    File "__main__", line 6, in __main__.sum_of_squared_errors
    Failed example:
        sum_of_squared_errors(np.array([[1, 4, 3],                                        [2, -1, 4]]),                              np.array([3, 4]),                              np.array([1, 2, 3]))
    Expected:
        289
    Got nothing
    **********************************************************************
    1 items had failures:
       1 of   1 in __main__.sum_of_squared_errors
    ***Test Failed*** 1 failures.

    TestResults(failed=1, attempted=1)

```python
w1 = np.linspace(-2, 4, 50)
w2 = np.linspace(-2, 4, 50)

W1, W2 = np.meshgrid(w1, w2)
E = np.array([[sum_of_squared_errors(X, y, np.array([W1[i, j], W2[i, j]])) \
               for j in range(W1.shape[1])] \
              for i in range(W1.shape[0])])
fig, ax = plt.subplots(figsize=(8,8))
CS = ax.contour(W1, W2, E, colors='black', levels=20)
ax.clabel(CS, inline=1, fontsize=10)
plt.xlabel('$w_1$')
plt.ylabel('$w_2$')
plt.title('Sum of Squared Errors')
plt.show()
```

#### *Expand for Solution*

```python
# ***Solution***
def sum_of_squared_errors(X, y, w):
    """
    Return the sum of squared errors for the given training data (X, y) and
    model parameters w.

    >>> sum_of_squared_errors(np.array([[1, 4, 3],\
                                        [2, -1, 4]]),\
                              np.array([3, 4]),\
                              np.array([1, 2, 3]))
    289
    """
    e = X.dot(w) - y
    return e.dot(e)

import doctest
doctest.testmod()
```

```python
# ***Solution***
w1 = np.linspace(-2, 4, 50)
w2 = np.linspace(-2, 4, 50)

W1, W2 = np.meshgrid(w1, w2)
E = np.array([[sum_of_squared_errors(X, y, np.array([W1[i, j], W2[i, j]])) \
               for j in range(W1.shape[1])] \
              for i in range(W1.shape[0])])
fig, ax = plt.subplots(figsize=(8,8))
CS = ax.contour(W1, W2, E, colors='black', levels=20)
ax.clabel(CS, inline=1, fontsize=10)
plt.xlabel('$w_1$')
plt.ylabel('$w_2$')
plt.title('Sum of Squared Errors')
plt.show()
```

***Solution***

(b) The contour lines represent values of $\mathbf{w}$ that incur equal squared error on the training set.  The optimal value of $\mathbf{w}$ (the one that minimizes the error) occurs near $w_1 = 1, w_2 = 2$, which is what we'd expect given the setup of the toy problem.

## Computing the Optimal Weights

Now you're ready to implement the formula that you derived in the assignment document.  In that document you should have arrived at the following formula for the optimal weights:

$$\mathbf{w^\star} = \left ( \mathbf{X}^\top \mathbf{X} \right)^{-1} \mathbf{X}^\top \mathbf{y}$$

### Notebook Exercise 2 (20 minutes)

Fill in the body of the function `optimal_weights` below.  You've done the hard work to derive this beautiful expression, translating it to code is the last step to glory!  Hint: `np.linalg.inv` computes the inverse of a specified matrix.  We've included code that will run your code on the training data.  Does your code compute sensible values of $\mathbf{w}$ given the setup of the toy problem?

```python
def optimal_weights(X, y):
     """ Returns the optimal weights in the least squares sense for the specified
         training inputs (X) and training outputs (y) """
     # your code here
     pass

optimal_weights(X, y)
```

#### Expand for Solution

```python
# ***Solution***

def optimal_weights(X, y):
     """ Returns the optimal weights in the least squares sense for the specified
         training inputs (X) and training outputs (y) """
     return np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)

optimal_weights(X, y)
```

***Solution***

The output makes sense since it is close to the values used to generate the data, but not identical.  We don't expect it to be identical since we added noise to the training outputs.

## Sanity Checking your Implementation

On the first day we talked a lot about evaluating ML models.  For instance, we talked about running experiments to see how well they work for some problem.  When we are implementing the algorithm ourselves a different and more basic thing we'd like to evaluate is whether we've implemented the algorithm correctly.

While you are probably feeling pretty confident right now that your implementation of linear regression is correct, for more complicated algorithms there have been cases when implementations of algorithms (even in published works) turned out to be incorrect (i.e., they didn't accurately reflect the algorithm that had been derived in the paper).  The story round the campfire (by which I mean I heard this from one of my professors in grad school, but I can't seem to find a link online verifying it) is that the initial implementation of the backpropagation algorithm (a foundational algorithm for machine learning in neural networks that we'll be learning about in the coming weeks) was wrong.  The experimental results presented in the paper were based on a flawed implementation (although clearly it wasn't so flawed that the results were garbage).

### *Notebook Exercise 3 (30 minutes)*

Let's check out a few strategies that we can use to verify that an implementation of an algorithm is correct.

a. ***Strategy 1 check for local optimality.***  If the machine learning algorithm involves optimizing some function (for example in linear regression you are optimizing squared error), you can verify that the solution you compute is locally optimal.  What does it mean for the solution to be locally optimal?  One very basic thing we can check is to see whether the value of the error gets strictly higher as we perturb the solution (e.g., add a small delta to the weights computed by your implementation of linear regression).  The following not very elegant, but illustrative code provides an implementation of this optimality check.

As a quick diagnostic of your understanding, what should be true of the output below in order for an implementation to pass the optimality check?  Why is it important to test each of the four perturbations below?

```python
w_star = optimal_weights(X, y)
w_star_err = sum_of_squared_errors(X, y, w_star)
perturbation = 10**-5
print(sum_of_squared_errors(X, y, w_star + np.array([perturbation, 0])) - w_star_err)
print(sum_of_squared_errors(X, y, w_star - np.array([perturbation, 0])) - w_star_err)
print(sum_of_squared_errors(X, y, w_star + np.array([0, perturbation])) - w_star_err)
print(sum_of_squared_errors(X, y, w_star - np.array([0, perturbation])) - w_star_err)
```

    5.583189910396413e-09
    5.583190798574833e-09
    6.337681490720115e-09
    6.337677938006436e-09

b. ***Strategy 2: check the gradient.***  For many machine learning algorithms that involve optimizing some function (linear regression is a great example) a second sanity check is to verify that the gradient is 0 at a potential solution.  Since it is not necessarily straightforward to calculate the gradient of the function we are optimizing, we can instead check that a numerical approximation of the gradient is close to 0.  We will use the finite differences method to approximate the gradient.

To help you understand what we mean by finite differences, here is the definition of the derivative of a single variable function.

$$f'(x) = \lim_{h \rightarrow 0} \frac{f(x+h) - f(x)}{h}$$

This definition suggests that we can approximate the derivative using the finite difference method as $f'(x) \approx \frac{f(x+h) - f(x)}{h}$ for some small value of $h$.  What typically works even better is to use the method of central differences where we estimate the drivative as $f'(x) \approx \frac{f(x+h) - f(x - h)}{2h}$.

In the code below, we'll apply this idea to estimating the gradient (which consists of two partial derivatives) at the optimal solution returned by your implementation of linear regression.

As a quick check of your understanding, what should be true of the output below in order for an implementation to pass the gradient check?

```python
estimate_partial_w_1 = (sum_of_squared_errors(X, y, w_star + np.array([perturbation, 0])) - sum_of_squared_errors(X, y, w_star - np.array([perturbation, 0])))/(2*perturbation)
estimate_partial_w_2 = (sum_of_squared_errors(X, y, w_star + np.array([0, perturbation])) - sum_of_squared_errors(X, y, w_star - np.array([0, perturbation])))/(2*perturbation)
print(estimate_partial_w_1, estimate_partial_w_2)
```

    -4.4408920985006255e-11 1.7763568394002502e-10

c. ***Strategy 3: compare to a known working implementation.***  Perhaps the most direct approach to validating your implementation would be to compare it to a known working implementation (assuming you have access to one).  In the cell below, we call `numpy`'s implementation of linear regression and compare it with your solution.

As a quick check of your understanding, what should be true of the output below in order for an implementation to pass the *compare to a known working implementation* check?

```python
w_known_working, _, _, _ = np.linalg.lstsq(X, y, rcond=-1)
print(w_known_working - w_star)
```

    [ 4.44089210e-16 -2.22044605e-15]

#### Expand for Solution

***Solution***

In order to pass the test all of the differences printed out should be positive since this corresponds to the error increasing as we move away from the optimal solution we computed.  It's important to test all four direction since you want to look along each dimension and in both the positive and negative directions.

The gradient should be close to 0.

The difference between the two vectors should be small.

## Training Test Splits: Bikeshare Revisited

In this next section of the notebook we're going to revisit the dataset that we met in the first assignment.  Our goals in this activity are twofold.

1.  We will introduce the notion of a train / test split for validating machine learning algorithms.
2.  We will motivate, derive, and implement an extension to linear regression called [ridge regression](https://en.wikipedia.org/wiki/Tikhonov_regularization).

For your convenience, here is the text from the previous notebook that we used to introduce the dataset.
> The [Bikeshare](https://archive.ics.uci.edu/ml/datasets/bike%20sharing%20dataset) dataset contains daily usage data over a roughly two year period. Along with each record of user counts, there are independent variables that measure various characteristics of the day in question (e.g., whether it was a weekday or a weekend, the air temperature, the wind speed).

The code below loads the dataset and produces hexplots that show various characteristics of the day versus ridership.

```python
import pandas as pd
bikeshare = pd.read_csv('https://raw.githubusercontent.com/kylecho/nd101_p1_neural_network/master/Bike-Sharing-Dataset/day.csv')

X_bikeshare = bikeshare.drop(columns=['instant', 'dteday', 'cnt', 'registered', 'casual'])
y_bikeshare = bikeshare['cnt']
X_bikeshare['bias'] = 1

plt.figure(figsize=(30, 18))
for idx, col in enumerate(X_bikeshare):
    plt.subplot(3, 4, idx+1)
    plt.hexbin(X_bikeshare[col], y_bikeshare, gridsize=25, cmap='jet')
    plt.colorbar()
    plt.xlabel(col)
    plt.ylabel('rider count')

plt.subplots_adjust(wspace=.2)
plt.show()
```

### Training and Testing Sets

One of the most fundamental ideas in evaluating machine learning algorithm involves [partitioning data into a training set (used for fitting a model) and a testing set (used for estimating the performance of the model).  There is a pretty comprehensive article on [training, validation, and testing sets](https://en.wikipedia.org/wiki/Training,_validation,_and_test_sets) on Wikipedia, but for now we are not going to be talking about the validation set.  Feel free to follow along with our presentation here and keep the link handy for future reference (no need to read the linked article now).

Remember the basic supervised learning problem setup where we are given a training data consisting of inputs $\mathbf{x_1}, \mathbf{x_2}, \ldots, \mathbf{x_n}$ and outputs $y_1, y_2, \ldots, y_n$.  So far we have been applying our learning algorithms to *all* $n$ of the training data instances.  We might be then tempted to estimate how well the resultant model would work on new data bytes computing the average squared error on these $n$ training instances.  It turns out that this approach can wildly overestimate how well the model will work on new data.  The reason is that the model parameters (e.g., the weights in linear regression) have been tuned to the training data.  Some of these model parameters will reflect genuine relationships between the inputs and outputs, and other model parameters may largely reflect particular quirks of the training data (e.g., noise) that are not applicable to new data.

In order to get an unbiased estimate of the performance of the model on new data we reserve a portion of the training data as a ***testing set***.  This testing set is not used to fit the model parameters and is only used to estimate model performance *after the model has been created*.

To clarify what we mean, in the next cell is some code that partitions the Bikeshare data into training and test sets.  We will fit the parameters of a linear regression model (using your code!) on the training set and calculate mean squared error on the testing set.  To make the code less cluttered, we will be using a helper function from the [`scikit-learn`](https://scikit-learn.org/stable/) library that creates a [training set / testing set split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html#sklearn.model_selection.train_test_split).  This function will ***randomly*** partition the given data into two disjoint sets: the training set and the testing set.  The parameter `test_size` controls the fraction of data assigned to the testing set versus the training set.  You'll also notice that we divide the `sum_of_squared_errors` by `y_train.shape[0]`, which is the number of training data instances.  Divided the sum of squared error by the number of training data instances gives us the *mean squared error.*  The mean squared error is more interpretable than the sum of squared errors since it controls for the number of data instances.

#### Notebook Exercise 4 (30 minutes)

Run the code below several times in order to answer the following questions.

What causes the results to change from run to run?

As you run the code multiple times, does there seem to be a trend that the performance on the training set is better (i.e. has lower mean squared error) than the performance on the testing set?

Since the training set was used to fit the model parameters, we might expect the training set to always have better performance than the testing set.  It appears that this is not always the case.  How is this possible?

```python
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

X_train, X_test, y_train, y_test = train_test_split(X_bikeshare, y_bikeshare, test_size=0.5)
w = optimal_weights(X_train, y_train)

print("training set mean squared error=%f" % (sum_of_squared_errors(X_train, y_train, w)/y_train.shape[0]))
print("testing set mean squared error=%f" % (sum_of_squared_errors(X_test, y_test, w)/y_test.shape[0]))
```

    training set mean squared error=831323.600458
    testing set mean squared error=686501.059788

##### Expand for Solution

***Solution***

The changes in output are driven by randomness in the `train_test_split` function.  Depending on which instances are assigned to the training versus the testing set, the output will differ.

It does appear that, on average, the performance on the testing set is worse.

On any given run it does not follow that performance on the testing set will always be worse.  It could be the case that the testing set happened to contain a lot of easy to predict instances and the training set contained a high number of outliers.

### Ridge Regression

So far we've been working with this Bikeshare dataset in cases where we have a relatively high number of training instances compared with the dimensionality of the data.  To make this more precise, the shape of `X_train` is 365 by 12, which means we have ratio of roughly 30:1 training instances to inputs features.  While there are no hard and fast rules about this, a 30:1 ratio is considered pretty good for coming up with good esitmates of model parameters.

Suppose instead that we faced a situation where we had very little training data.  To simulate this case, below we rerun our experiment with the BikeShare dataset but set the `test_size` to 0.95.  You should notice two things when running this code.

1.  The performance on the training set is markedly better than the testing set.
2.  Occasionally you will get an error message about a singular matrix.

```python
X_train, X_test, y_train, y_test = train_test_split(X_bikeshare, y_bikeshare, test_size=0.95)
print("number of training points %d, number of testing points %d" % (y_train.shape[0], y_test.shape[0]))

w = optimal_weights(X_train, y_train)
print("training set mean squared error=%f" % (sum_of_squared_errors(X_train, y_train, w)/y_train.shape[0]))
print("testing set mean squared error=%f" % (sum_of_squared_errors(X_test, y_test, w)/y_test.shape[0]))
```

    number of training points 36, number of testing points 695
    training set mean squared error=439139.914045
    testing set mean squared error=1518156.344032

The first observation (that the performance on the training set is markedly better than the testing set) is perhaps not very surprising since we now have much less training data to use to reliably estimate the model parameters.  To understand the second observation, we need remind ourselves of the formula for the optimal weights in linear regression.

$$\mathbf{w^\star} = \left ( \mathbf{X}^\top \mathbf{X} \right)^{-1} \mathbf{X}^\top \mathbf{y}$$

The error regarding a singular matrix is coming from the fact that we are computing the inverse of the matrix $\mathbf{X}^\top \mathbf{X}$.  One property of a [singular matrix](http://mathworld.wolfram.com/SingularMatrix.html) is that it is not invertible, hence the error message.  The reason it is not invertible is that the matrix $\mathbf{X}^\top \mathbf{X}$ is not full rank.  This happens when the training data does not properly span the space of the features.  This usually happens for a combination of the following reasons:

1.  There is too little training data
2.  There are features that are defined as linear combinations of each other.

In order to solve this problem, a common approach is to modify the linear regression problem to prefer solutions that have small weights.  We do this by penalizing the sum of the squares of the weights themselves.  This is called ridge regression (or Tikhonov regularization).  Below, we show the original version of ordinary least squares along with ridge regression.

Ordinary least squares:
$$\begin{align}
\mathbf{w^\star} &= \arg\min_\mathbf{w} \sum_{i=1}^n \left ( \mathbf{w}^\top \mathbf{x_i} - y_i \right)^2 \\
&= \arg\min_\mathbf{w} \left ( \mathbf{X}\mathbf{w} - \mathbf{y} \right)^\top \left ( \mathbf{X}\mathbf{w} - \mathbf{y} \right)
\end{align}$$

Ridge regression (note that $\lambda$ is a non-negative parameter that controls how much the algorithm cares about fitting the data and how much it cares about having small weights):
$$\begin{align}
\mathbf{w^\star} &= \arg\min_\mathbf{w} \sum_{i=1}^n \left ( \mathbf{w}^\top \mathbf{x_i} - y_i \right)^2 + \lambda\sum_{i=1}^d w_i^2 \\
&= \arg\min_\mathbf{w} \left ( \mathbf{X}\mathbf{w} - \mathbf{y} \right)^\top \left ( \mathbf{X}\mathbf{w} -  \mathbf{y} \right) + \lambda \mathbf{w}^\top \mathbf{w}
\end{align}$$

The penalty term may seem a little arbitrary, but it can be motivated on a conceptual level pretty easily.  The basic idea is that in the absence of sufficient training data to suggest otherwise, we should try to make the weights small.  Small weights have the property that changes to the input result in minor changes to our predictions, which is a good default behavior.

### Notebook Exercise 5 (60 minutes)
*Note: this one is really a math problem, but we didn't want to send you back to the other document and then back here again.  Let us know if you like this or not via NB.*

Derive an expression to compute the optimal weights, $\mathbf{w^\star}$, to the ridge regression problem.

* Hint 1: This is very, very similar to exercise 5 in the assignment document.
* Hint 2: If you follow the same steps as you did in exercise 5, you'll arrive at an expression that looks like this (note: $\mathbf{I}_{d \times d}$ is the $d$ by $d$ identity matrix).

$$\mathbf{w^\star} = \arg\min_\mathbf{w} \mathbf{w}^\top \mathbf{X}^\top \mathbf{X} \mathbf{w} - 2\mathbf{w}^\top \mathbf{X}^\top \mathbf{y} + \mathbf{y}^\top \mathbf{y} + \lambda \mathbf{w}^\top  \mathbf{I}_{d \times d} \mathbf{w}$$

* Hint 3: to get $\mathbf{w^\star}$, take the gradient, set it to 0 and solve for $\mathbf{w}$.

#### Expand for Solution

***Solution***

$$\begin{align}\mathbf{w^\star} &= \arg\min_\mathbf{w} \left ( \mathbf{X}\mathbf{w} - \mathbf{y} \right)^\top \left ( \mathbf{X}\mathbf{w} -  \mathbf{y} \right) + \lambda \mathbf{w}^\top \mathbf{w} \\
& = \arg\min_\mathbf{w} \mathbf{w}^\top \mathbf{X}^\top \mathbf{X} \mathbf{w} - 2\mathbf{w}^\top \mathbf{X}^\top \mathbf{y} + \mathbf{y}^\top \mathbf{y} + \lambda \mathbf{w}^\top  \mathbf{I}_{d \times d} \mathbf{w}\\
&= \arg\min_\mathbf{w} \mathbf{w}^\top \left ( \mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I_{d \times d}} \right )\mathbf{w} - 2\mathbf{w}^\top \mathbf{X}^\top \mathbf{y} + \mathbf{y}^\top \mathbf{y} \\
2 \left (  \mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I_{d \times d}} \right ) \mathbf{w^\star} - 2 \mathbf{X}^\top \mathbf{y} &=0 ~~\mbox{take the gradient and set to 0} \\
\mathbf{w}^\star &= \left ( \mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I_{d \times d}} \right)^{-1} \mathbf{X}^\top \mathbf{y}
\end{align}$$

### Notebook Exercise 6 (20 minutes)

Now we'll be revisiting the Bikeshare dataset and see if ridge regression can help.  If you'd like to implement the algorithm yourself, feel free.  Since it is a relatively small change from the implementation that you created earlier, we have gone ahead and provided you with implementation below.  Here are some questions to test your understanding of the effects of applying ridge regression to the bike share dataset.

Run the code below with the default setting of the input `lam`.  You should notice that singular matrix error does not arise anymore.  Make the value of `lam` really large (search over different orders of magnitude to find a value that is really large).  What happens to the training and test set errors?

Does there seem to be a value of `lam` that is best (we advise you to search over different orders of magnitude)?  How do you define best?  What would be a good process for determining a good value of `lam` (we'll be learning about this in much more detail coming up, but we wanted to get you thinking about some possibilities)?

```python
def optimal_weights_ridge(X, y, lam):
     """ Returns the optimal weights in the least squares sense for the specified
         training inputs (X) and training outputs (y) with ridge term `lam` """
     return np.linalg.inv(X.T.dot(X) + lam*np.eye(X.shape[1])).dot(X.T).dot(y)

X_train, X_test, y_train, y_test = train_test_split(X_bikeshare, y_bikeshare, test_size=0.95)
print("number of training points %d, number of testing points %d" % (y_train.shape[0], y_test.shape[0]))

w = optimal_weights_ridge(X_train, y_train, 1)
print("training set mean squared error=%f" % (sum_of_squared_errors(X_train, y_train, w)/y_train.shape[0]))
print("testing set mean squared error=%f" % (sum_of_squared_errors(X_test, y_test, w)/y_test.shape[0]))
```

    number of training points 36, number of testing points 695
    training set mean squared error=608917.228853
    testing set mean squared error=942739.384525

#### Expand for Solution

***Solution***

(a) When you make lambda really big, the errors become very large.  This is because the model is underfitting to the data.  In other words, it cares more about making the weights small than it does about fitting the data.

(b) It's pretty hard to tell, but it seems like a value around $0.0001$ seems to work pretty well.  We defined best by mentally averaging over the testing mean squared error measured across a few runs.  In order to do this more rigorously you'd want to have a defined space of values you'd search over, repeat the experiment a number of times, and then choose the best average performance.  We'll dig into this more systematically soon, but this is a good amount answer to get to with the current tools we have discussed.


# Linear Algebra
> ## Linearity
> ### superpostion principle (중첩의 원리)
>> a function $F(x)$ that satisfies the superposition principle is called a linear function
>> ### additivity
>>> ### $F(x_1 + x_2) = F(x_1) + F(x_2)$
>>
>> ### homogeneity
>>> ### $F(ax) = aF(x), \text{ for scalar } a.$
>> $ homogenous solution + particular solution = 0$
>
> ## Algebra
> ### synbols that stand for numbers
>> ## Geometry of Linear Equations
# Methord of Solution
> ## [Row pciture](https://twlab.tistory.com/6?category=668741)
>> 공간상에서 the line,plane of dot products
>> $
\begin{bmatrix}
2 & 5 \\ 1 & 3
\end{bmatrix}
\begin{bmatrix}
1 \\ 2
\end{bmatrix} =
\begin{bmatrix}
2 & 5 \\ 0 & 0
\end{bmatrix}
\begin{bmatrix}
1 \\ 2
\end{bmatrix} +
\begin{bmatrix}
0 & 0 \\ 1 & 3
\end{bmatrix}
\begin{bmatrix}
1 \\ 2
\end{bmatrix} =
\begin{bmatrix}
12 \\ 7
\end{bmatrix}
$
> ## [Column pciture](https://twlab.tistory.com/6?category=668741)
>> 공간상에서 linear combination of vectors
>> $
\begin{bmatrix}
2 & 5 \\ 1 & 3
\end{bmatrix}
\begin{bmatrix}
1 \\ 2
\end{bmatrix} =
1\:
\begin{bmatrix}
2 \\ 1
\end{bmatrix} +
2 \:
\begin{bmatrix}
5 \\ 3
\end{bmatrix} =
\begin{bmatrix}
12 \\ 7
\end{bmatrix}
$

```python
import sympy as sm
import numpy as np
import matplotlib.pyplot as plt
%matplotlib widget
```

### find solution.
> ### $
\begin{cases}
2x & - & y  &  & & = 0 \\
-x & + & 2y & - &z & = -1 \\
& - &3y & + & 4z &  = 4
\end{cases}$

> #### $
\begin{bmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -3 & 4 \end{bmatrix} \:
\begin{bmatrix}  x \\ y \\ z \end{bmatrix} \: =  \:
\begin{bmatrix}  0 \\ -1 \\ 4 \end{bmatrix}
$

```python
fig = plt.figure()
ax = fig.add_subplot(projection = '3d')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_xlim(-10,10)
ax.set_ylim(-10,10)
ax.set_zlim(-10,10)
xi = np.linspace(-5,5,10)
yi = np.linspace(-5,5,10)

xi,yi = np.meshgrid(xi,yi)
ax.plot_surface(xi, 2*xi, yi,alpha=0.5)
ax.plot_surface(xi, yi, -xi+2*yi+1,alpha=0.5)
ax.plot_surface(xi, yi, 3/4*yi+1,alpha=0.5)
```

    <mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x7fc9b82c7040>

        Figure

```python
x,y,z = sm.symbols('x y z')
sm.solve([2*x-y,-x+2*y-z+1,-3*y+4*z-4],[x,y,z])
```

    {x: 0, y: 0, z: 1}

```python
ax.scatter(0,0,1,marker='o',color='r',s = 100)
```

    <mpl_toolkits.mplot3d.art3d.Path3DCollection at 0x7fc97dac4e50>

# Column picture
> ## $
x\:
\begin{bmatrix}
2 \\ -1 \\ 0
\end{bmatrix} +
y\:
\begin{bmatrix}
-1 \\ 2 \\ -3
\end{bmatrix} +
z\:
\begin{bmatrix}
0 \\ -1 \\ 4
\end{bmatrix} \:= \:
\begin{bmatrix}
0 \\ -1 \\ 4
\end{bmatrix}
$

```python
fig1 = plt.figure()
ax = fig1.add_subplot(projection='3d')
ax.set_xlim(-4,4)
ax.set_ylim(-4,4)
ax.set_zlim(-4,4)
ax.quiver(0,0,0,2,-1,0)
ax.quiver(0,0,0,-1,2,-3)
ax.quiver(0,0,0,0,-1,4)
ax.scatter(0,0,0,c='r')
ax.scatter(0,-1,4,c='r')
```

    <mpl_toolkits.mplot3d.art3d.Path3DCollection at 0x7fc97c1049d0>

        Figure

```python
M = sm.Matrix([[2,-1,0,0],[-1,2,-1,-1],[0,-3,4,4]])
sm.Matrix([(2,-1,0,0),(-1,2,-1,-1),(0,-3,4,4)])
sm.Matrix(((2,-1,0,0),(-1,2,-1,-1),(0,-3,4,4)))
A[:,:-1]
A[:,-1]
```

```python
sm.linsolve(sm.Matrix([[2,-1,0,0],[-1,2,-1,-1],[0,-3,4,4]]),(x,y,z))
```

```python
sm.linsolve((M[:,:-1],M[:,-1]),x,y,z)
```


# Problem Sheet 2 Question 3 - 2nd Order Taylor

The general form of the population growth differential equation
\begin{equation}  y^{'}=ty-ty^2, \ \ (0 \leq t \leq 2)\end{equation}
with the initial condition
\begin{equation} y(0)=\frac{1}{2}.\end{equation}

Apply __2nd Order Taylor__ to approximate the solution of the given initial value problems using the indicated number of time steps. Compare the approximate solution with the given exact solution, and compare the actual error with the theoretical error.

```python
## Library
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
import pandas as pd

warnings.filterwarnings("ignore")
```

## General Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
\begin{equation}  h=\frac{b-a}{N}.\end{equation}
## Specific Discrete Interval
Here the interval is $0\leq t \leq 2$ with $N=4$
\begin{equation}  h=\frac{2-0}{4}=0.5.\end{equation}
This gives the 5 discrete points with stepsize h=0.5:
\begin{equation}  t_0=0, \ t_1=0.5, \ ... t_{4}=2. \end{equation}
This is generalised to
\begin{equation}  t_i=0+i0.5, \ \ \ i=0,1,...,4.\end{equation}
The plot below illustrates the discrete time steps from 0 to 2.

```python
### Setting up time
t_end=2
t_start=0
N=4
h=(t_end-t_start)/(N)
t=np.arange(t_start,t_end+0.01,h)
fig = plt.figure(figsize=(10,4))
plt.plot(t,0*t,'o:',color='red')
plt.plot(t[0],0*t[0],'o',color='green')

plt.title('Illustration of discrete time points for h=%s'%(h))
plt.plot();
```

## 2nd Order Taylor Solution
The 2nd Order Taylor difference equation is given by
\begin{equation}  w_{i+1}=w_i+h\left[f(t_i,w_i)+\frac{h}{2}f'(t_i,w_i)\right],\end{equation}
where
\begin{equation}  f(t_i,w_i)=t_iw_i-t_iw_i^2,\end{equation}
and
\begin{equation}  f'(t_i,w_i)=w_i+t_i(t_iw_i-t_iw_i^2)-w_i^2-t_i(2w_i(t_iw_i-t_iw_i^2)),\end{equation}
which gives
\begin{equation} w_{i+1}=w_i+h(t_iw_i-t_iw_i^2)+\frac{h^2}{2}(w_i+t_i(t_iw_i-t_iw_i^2)-w_i^2-t_i(2w_i(t_iw_i-t_iw_i^2))),\end{equation}

for $i=0,1,2,3$
with
$w_0=0.5$

```python
IC=0.5
w=np.zeros(N+1)
e=np.zeros(N+1)
w[0]=IC

#e[0]=IC
for i in range (0,N):
  #  e[i+1]=e[i]+h*(t[i]*e[i]-t[i]*e[i]*e[i])
    w[i+1]=w[i]+h*(t[i]*w[i]-t[i]*w[i]*w[i])+h*h/2*(w[i]+t[i]*(t[i]*w[i]-t[i]*w[i]*w[i])-w[i]*w[i]-t[i]*(2*w[i]*(t[i]*w[i]-t[i]*w[i]*w[i])))
```

```python
fig = plt.figure(figsize=(4,5))
# --- left hand plot
#ax = fig.add_subplot(1,4,1)
plt.plot(t,w,'o:',color='k')
plt.plot(t[0],w[0],'o',color='green')
plt.plot(t,e,'o:',color='r')

#ax.legend(loc='best')
plt.title('Numerical Solution (w) h=%s'%(h))
plt.show()

```

```python

d = {'time t_i': t,    'Euler (w_i) ':w,    'Taylor (w_i) ':y,    'Exact Error( |y_i-w_i|) ':np.abs(y-w)}
df = pd.DataFrame(data=d)
df
```

      time t_i
      Euler (w_i)
      Exact (y_i)
      Exact Error( |y_i-w_i|)
      Upper Bound

      0
      0.0
      2.000000
      2.000000
      0.000000
      0.000000

      1
      0.5
      3.145833
      3.148721
      0.002888
      0.027030

      2
      1.0
      4.708767
      4.718282
      0.009514
      0.071595

      3
      1.5
      6.958180
      6.981689
      0.023509
      0.145070

      4
      2.0
      10.337421
      10.389056
      0.051635
      0.266211

```python

```

Chapter 02 - Higher Order Methods/2_Problem Sheet/203_Problem_Sheet.ipynb


## K-Means

Given a set of datapoints ${x_1, ..., x_n}$ and a positive number $k$, find the clusters $C_1, ..., C_k$ that minimize

\begin{equation}
J = \sum_{i=1}^n \, \sum_{j=1}^k \, z_{ij} \, || x_i - \mu_j ||_2
\end{equation}

where:
- $z_{ij} \in \{0,1\}$ defines whether of not datapoint $x_i$ belongs to cluster $C_j$
- $\mu_j$ denotes the cluster center of cluster $C_j$
- $|| \, ||_2$ denotes the Euclidean distance

```python
import pandas as pd
import numpy as np
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.metrics import pairwise_distances

from utils import plot_decision_boundaries

```

```python
data = pd.read_csv('data.csv',header=None).values
data.shape
```

    (150, 2)

```python
def visualize_data(X):
    plt.scatter(X[:,0],X[:,1],c='b',s=5)
    plt.xlabel('$x_1$')
    plt.ylabel('$x_2$')
```

```python
plt.figure(figsize=(4,4))
visualize_data(data)
plt.show()
```

```python
def row_norm(X, squared=True):
    #Row-wise (squared) Euclidean norm of X.
    #Equivalent to np.sqrt((X * X).sum(axis=1))
    norms = np.einsum('ij,ij->i', X, X)
    if not squared:
        np.sqrt(norms, norms)
    return norms

def _kplus_init(X, n_clusters, random_state, x_squared_norms):
    n_samples, n_features = X.shape

    centers = np.empty((n_clusters, n_features), dtype=X.dtype)

    n_local_trials = 2 + int(np.log(n_clusters))

    center_id = random_state.randint(n_samples)

    closest_dist_sq = euclidean_distances(centers[0, np.newaxis],
                                          X,
                                          Y_norm_squared=x_squared_norms,
                                          squared=True)
    current_pot = closest_dist_sq.sum()
    for c in range(1, n_clusters):
        rand_vals = random_state.random_sample(n_local_trials) * current_pot
        candidate_ids = np.searchsorted(np.cumsum(closest_dist_sq),
                                        rand_vals)

        distance_to_candidates = euclidean_distances(
            X[candidate_ids], X, Y_norm_squared=x_squared_norms, squared=True)

        best_candidate = None
        best_pot = None
        best_dist_sq = None
        for trial in range(n_local_trials):
            new_dist_sq = np.minimum(closest_dist_sq,
                                     distance_to_candidates[trial])
            new_pot = new_dist_sq.sum()

            if (best_candidate is None) or (new_pot < best_pot):
                best_candidate = candidate_ids[trial]
                best_pot = new_pot
                best_dist_sq = new_dist_sq

        centers[c] = X[best_candidate]
        current_pot = best_pot
        closest_dist_sq = best_dist_sq

    return centers

def _random_init(X, n_clusters, random_state):
    seeds = random_state.permutation(X.shape[0])[:n_clusters]
    centers = X[seeds]
    return centers

def _init_centroids(X, n_clusters, random_state,x_squared_norms=None, init='kplus'):
    if init == 'random':
        centers = _random_init(X, n_clusters, random_state)
    else:
        centers = _kplus_init(X, n_clusters, random_state, x_squared_norms)
    return centers

def _assign_labels(X, centers):
    dist = pairwise_distances(X, centers,metric='euclidean')
    assignments = np.argmin(dist,axis=1)
    return assignments

def _get_centers(X, assignments):
    new_centers = []
    for i in np.unique(assignments):
        new_centers.append(X[assignments == i].mean(axis=0))
    return np.array(new_centers)

def _kmeans_lloyd(X, n_clusters, random_state, init, max_iter = 240):
    x_squared_norms = row_norm(X)
    centers = _init_centroids(X, n_clusters, random_state, x_squared_norms,init=init)
    distances = np.zeros(shape=(X.shape[0],), dtype=X.dtype)

    for i in range(max_iter):
        centers_old = centers.copy()
        # E-step of EM
        assignments = _assign_labels(X, centers)
        # M-step of EM
        centers = _get_centers(X, assignments)
    return centers, assignments
```

```python
class KMeans():
    def __init__(self,n_clusters:int, random_seed:int = 65):
        self.n_clusters = n_clusters
        self.random_state = np.random.RandomState(random_seed)

    def fit(self,X):
        #X_mean = X.mean(axis=0)
        #X -= X_mean
        self.cluster_centers_, self.labels_ = _kmeans_lloyd(X=X,
                                                            n_clusters = self.n_clusters,
                                                            random_state = self.random_state,
                                                            init ='random')

        return self

    def fit_predict(self,X):
        return self.fit(X).labels_

    def predict(self, X):
        #X_mean = X.mean(axis=0)
        #X -= X_mean
        if self.labels_ is not None:
            return _assign_labels(X, self.cluster_centers_)

```

```python
kmeans = KMeans(n_clusters=3, random_seed=14)
y_pred = kmeans.fit_predict(data)
y_pred is kmeans.labels_
```

    True

```python
kmeans.cluster_centers_
```

    array([[6.98823529, 4.74705882],
           [5.02307692, 3.42115385],
           [6.5787234 , 2.94893617]])

```python
a = np.arange(10)
np.sort(a[:3])
```

    array([0, 1, 2])

```python
plt.figure(figsize=(4, 4))
plot_decision_boundaries(kmeans, data)

plt.show()
```

```python
'''
def _labels_inertia(X, x_squared_norms, centers,
                    distances=None):

    n_samples = X.shape[0]
    # set the default value of centers to -1 to be able to detect any anomaly
    # easily
    labels = np.full(n_samples, -1, np.int32)
    if distances is None:
        distances = np.zeros(shape=(0,), dtype=X.dtype)

    inertia = _assign_labels_array(
        X, x_squared_norms, centers, labels,
        distances=distances)
    return labels, inertia
'''
```

    '\ndef _labels_inertia(X, x_squared_norms, centers,\n                    distances=None):\n    \n    n_samples = X.shape[0]\n    # set the default value of centers to -1 to be able to detect any anomaly\n    # easily\n    labels = np.full(n_samples, -1, np.int32)\n    if distances is None:\n        distances = np.zeros(shape=(0,), dtype=X.dtype)\n\n    inertia = _assign_labels_array(\n        X, x_squared_norms, centers, labels,\n        distances=distances)\n    return labels, inertia\n'

```python
def _compute_distances(points):
    n = points.shape[0]
    distances = np.zeros((n, n))
    for i in range(n):
        for j in range(n):
            distances[i,j] = np.linalg.norm(points[i,:] - points[j,:])
    return distances
test1 = _compute_distances(data)
X_copy = data.copy()
test2 = pairwise_distances(data, data)
# test1 == test2 with tolerance
```


# Sinais periódicos

Neste notebook avaliaremos a série de Fourier de alguns sinais periódicos.

Um sinal periódico, contínuo, é aquele para o qual a seguinte propriedade é válida:

\begin{equation}
x(t) = x(t \pm mT_p),
\end{equation}
ou seja, o valor do sinal no instante $t$ [s] é o mesmo para o instante $t \pm mT_p$ [s]. Dessa forma, o sinal se repete a cada
período $T_p$.

$T_p$ é o chamado período fundamental do sinal periódico. Neste caso, $x(t) = x(t \pm T_p) = x(t \pm 2T_p) = ... = x(t \pm kT_p)$.

Os sinais periódicos tem uma frequência fundamental $f_1 = \frac{1}{T_p}$ (ou frequência angular fundamental $\omega_1 = \frac{2\pi}{T_p}$.

A base para a estimativa do espectro de tais sinais é a ***Série de Fourier***, que nos afirma que qualquer sinal periódico pode ser representado por uma soma infinita de senos e cossenos.

\begin{equation}
x(t) = \frac{a_0}{2} + \sum\limits_{k=1}^{\infty}a_k \mathrm{cos}(k \omega_1 t)+ \sum\limits_{k=1}^{\infty}b_k \mathrm{sin}(k \omega_1 t)
\end{equation}
com os coeficientes dados por:

\begin{equation}
\frac{a_0}{2} = \frac{1}{T_p}\int_{T_p} x(t)\mathrm{d}t
\end{equation}

\begin{equation}
a_k = \frac{2}{T_p}\int_{T_p} x(t)\mathrm{cos}(k\omega_1t)\mathrm{d}t
\end{equation}

\begin{equation}
b_k = \frac{2}{T_p}\int_{T_p} x(t)\mathrm{sin}(k\omega_1t)\mathrm{d}t
\end{equation}

Quando usamos a representação complexa da série de Fourier, temos:

\begin{equation}
x(t) = \sum\limits_{k=-\infty}^{\infty}\tilde{c}_k \mathrm{e}^{\mathrm{j}k \omega_1 t}
\end{equation}
com os coeficientes dados por:

\begin{equation}
\tilde{c}_k = \frac{1}{T_p}\int_{T_p} x(t)\mathrm{e}^{-\mathrm{j}k\omega_1 t}\mathrm{d}t
\end{equation}

Vamos investigar o espectro de uma onda quadrada e a recuperação do sinal $x(t)$ por meio do somatório de Fourier.

```python
# importar as bibliotecas necessárias
import numpy as np # arrays
import matplotlib.pyplot as plt # plots
plt.rcParams.update({'font.size': 14})
from scipy import signal # some signals
```

```python
fs = 1000
t = np.arange(0, 3, 1/fs) # vetor temporal
f1 = 1 # Frequencia fundamental
w1 = 2*np.pi*f1

# onda quadrada
xt = signal.square(2 * np.pi * f1 * t)

# Figura
plt.figure()
plt.title('Onda quadrada')
plt.plot(t, xt, '--k', linewidth = 4, label = 'sinal original')
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, t[-1]))
plt.tight_layout()
plt.show()
```

# Os coeficientes de Fourier

```python
max_k = 51 # valor máximo de harmônicas
k = np.arange(-max_k, max_k+1)
freq_vec = k*f1 # vetor de frequências (proporcional a f1)
ck = (-1j/(np.pi*k))*(1-np.cos(k*np.pi)) # coeficientes de Fourier
ck[(k%2)==0] = 0 # Para calcular com os arredondamentos corretos
```

    C:\Users\ericb\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: RuntimeWarning: divide by zero encountered in true_divide
      after removing the cwd from sys.path.
    C:\Users\ericb\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: RuntimeWarning: invalid value encountered in true_divide
      after removing the cwd from sys.path.
    C:\Users\ericb\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: RuntimeWarning: invalid value encountered in multiply
      after removing the cwd from sys.path.

```python
# O espectro
plt.figure()
plt.subplot(2,1,1)
plt.stem(freq_vec, np.abs(ck), '-k', basefmt=" ", use_line_collection=  True)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [-]')
plt.ylabel(r'$|\tilde{c}_k|$ [-]')
#plt.ylim((-2.2, 2.2))
plt.subplot(2,1,2)
plt.stem(freq_vec, np.rad2deg(np.angle(ck)), '-k', basefmt=" ", use_line_collection=  True)
plt.grid(linestyle = '--', which='both')
plt.xlabel('Frequência [-]')
plt.ylabel(r'$\angle\tilde{c}_k$ [-]')
#plt.ylim((-2.2, 2.2))
plt.tight_layout()
plt.show()
```

# Reconstrução do sinal

```python
# Construir uma matriz de sinais exponenciais complexos - cada linha é um "k"
xt_mtx = np.zeros((len(k), len(t)), dtype = complex)

for jk in np.arange(0,len(k)):
    xt_mtx[jk,:] = ck[jk]*np.exp(1j*k[jk]*w1*t)

xt_recon = np.sum(xt_mtx, axis = 0)
```

```python
# Vamos plotar o sinal original e cada componente
plt.figure()
plt.title('Onda quadrada')
plt.plot(t, xt, '--k', linewidth = 4, label = 'sinal original')
for jk in np.arange(0,len(k)):
    plt.plot(t, np.real(xt_mtx[jk,:]), linewidth = 2)
plt.legend(loc = 'upper right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, t[-1]))
plt.tight_layout()
plt.show()
```

```python
# Vamos plotar o sinal original e o reconstruído
plt.figure()
plt.title('Onda quadrada')
plt.plot(t, xt, '--k', linewidth = 4, label = 'sinal original')
plt.plot(t, np.real(xt_recon), '-b', linewidth = 2, label = 'sinal reconstruído')
plt.legend(loc = 'center right')
plt.grid(linestyle = '--', which='both')
plt.xlabel('Tempo [s]')
plt.ylabel('Amplitude [-]')
plt.xlim((0, t[-1]))
plt.tight_layout()
plt.show()
```

Aula 21 - SF - espectro de alguns sinais e reconstrucao/SF de sinal periodico.ipynb


# Analytic integration of 4-nodes Finite Element

```python
from __future__ import division
from sympy import *
```

```python
init_session()
```

    IPython console for SymPy 1.2 (Python 3.6.8-64-bit) (ground types: gmpy)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.2/

```python
E, rho = symbols('E rho', positive=True)
nu, r, s = symbols('nu r s')
```

Constitutive equations in Voigt's notation

```python
factor = E/((1+nu)*(1-2*nu))
factor = 1
C = factor*Matrix([
     [1 - nu,nu,0],
     [nu,1,0],
     [0,0,(1-2*nu)/2]])
display(C)
```

Interpolation functions

    2-----1
    |     |
    |     |
    3-----4

```python
N = S(1)/4*Matrix([(1 + r) * (1 + s),
                  (1 - r) * (1 + s),
                  (1 - r) * (1 - s),
                  (1 + r) * (1 - s)])
```

Interpolation matrix

```python
H = zeros(2, 8)
for i in range(4):
    H[0, 2*i] = N[i]
    H[1, 2*i + 1] = N[i]

display(H)
```

Mass matrix

```python
M_inte = H.T * H
M = integrate(integrate(rho*M_inte, [r, -1, 1]), [s, -1, 1])
display(M)
```

Derivatives interpolation matrix

```python
B = zeros(3, 8)
for i in range(4):
    B[0, 2*i] = diff(N[i], r)
    B[1, 2*i + 1] = diff(N[i], s)
    B[2, 2*i] = diff(N[i], s)
    B[2, 2*i + 1] = diff(N[i], r)

display(B)
```

Stiffness matrix

```python
K_inte = B.T * C * B
K = simplify(integrate(integrate(K_inte, [r, -1, 1]), [s, -1, 1]))
```

```python
display(K)
```

$$\left[\begin{matrix}- \frac{2 \nu}{3} + \frac{1}{2} & \frac{1}{8} & \frac{\nu}{6} - \frac{1}{4} & \frac{\nu}{2} - \frac{1}{8} & \frac{\nu}{3} - \frac{1}{4} & - \frac{1}{8} & \frac{\nu}{6} & - \frac{\nu}{2} + \frac{1}{8}\\\frac{1}{8} & - \frac{\nu}{3} + \frac{1}{2} & - \frac{\nu}{2} + \frac{1}{8} & \frac{\nu}{3} & - \frac{1}{8} & \frac{\nu}{6} - \frac{1}{4} & \frac{\nu}{2} - \frac{1}{8} & - \frac{\nu}{6} - \frac{1}{4}\\\frac{\nu}{6} - \frac{1}{4} & - \frac{\nu}{2} + \frac{1}{8} & - \frac{2 \nu}{3} + \frac{1}{2} & - \frac{1}{8} & \frac{\nu}{6} & \frac{\nu}{2} - \frac{1}{8} & \frac{\nu}{3} - \frac{1}{4} & \frac{1}{8}\\\frac{\nu}{2} - \frac{1}{8} & \frac{\nu}{3} & - \frac{1}{8} & - \frac{\nu}{3} + \frac{1}{2} & - \frac{\nu}{2} + \frac{1}{8} & - \frac{\nu}{6} - \frac{1}{4} & \frac{1}{8} & \frac{\nu}{6} - \frac{1}{4}\\\frac{\nu}{3} - \frac{1}{4} & - \frac{1}{8} & \frac{\nu}{6} & - \frac{\nu}{2} + \frac{1}{8} & - \frac{2 \nu}{3} + \frac{1}{2} & \frac{1}{8} & \frac{\nu}{6} - \frac{1}{4} & \frac{\nu}{2} - \frac{1}{8}\\- \frac{1}{8} & \frac{\nu}{6} - \frac{1}{4} & \frac{\nu}{2} - \frac{1}{8} & - \frac{\nu}{6} - \frac{1}{4} & \frac{1}{8} & - \frac{\nu}{3} + \frac{1}{2} & - \frac{\nu}{2} + \frac{1}{8} & \frac{\nu}{3}\\\frac{\nu}{6} & \frac{\nu}{2} - \frac{1}{8} & \frac{\nu}{3} - \frac{1}{4} & \frac{1}{8} & \frac{\nu}{6} - \frac{1}{4} & - \frac{\nu}{2} + \frac{1}{8} & - \frac{2 \nu}{3} + \frac{1}{2} & - \frac{1}{8}\\- \frac{\nu}{2} + \frac{1}{8} & - \frac{\nu}{6} - \frac{1}{4} & \frac{1}{8} & \frac{\nu}{6} - \frac{1}{4} & \frac{\nu}{2} - \frac{1}{8} & \frac{\nu}{3} & - \frac{1}{8} & - \frac{\nu}{3} + \frac{1}{2}\end{matrix}\right]$$

## Gauss integration

```python
wts = [2, [1,1], [S(5)/9, S(8)/9, S(5)/9],
        [(18+sqrt(30))/36,(18+sqrt(30))/36, (18-sqrt(30))/36, (18-sqrt(30))/36]];
pts = [0, [-sqrt(S(1)/3), sqrt(S(1)/3)], [-sqrt(S(3)/5), 0, sqrt(S(3)/5)],
[-sqrt(S(3)/7 - S(2)/7*sqrt((6)/5)), sqrt(S(3)/7 - S(2)/7*sqrt(S(6)/5)),
-sqrt(S(3)/7 + S(2)/7*sqrt(S(6)/5)), sqrt(S(3)/7 + S(2)/7*sqrt(S(6)/5))]]
```

```python
K_1pts = simplify(K_inte.subs({r:0, s:0})*wts[0])
display(K_1pts)
```

$$\left[\begin{matrix}- \frac{\nu}{4} + \frac{3}{16} & \frac{1}{16} & - \frac{1}{16} & \frac{\nu}{4} - \frac{1}{16} & \frac{\nu}{4} - \frac{3}{16} & - \frac{1}{16} & \frac{1}{16} & - \frac{\nu}{4} + \frac{1}{16}\\\frac{1}{16} & - \frac{\nu}{8} + \frac{3}{16} & - \frac{\nu}{4} + \frac{1}{16} & \frac{\nu}{8} + \frac{1}{16} & - \frac{1}{16} & \frac{\nu}{8} - \frac{3}{16} & \frac{\nu}{4} - \frac{1}{16} & - \frac{\nu}{8} - \frac{1}{16}\\- \frac{1}{16} & - \frac{\nu}{4} + \frac{1}{16} & - \frac{\nu}{4} + \frac{3}{16} & - \frac{1}{16} & \frac{1}{16} & \frac{\nu}{4} - \frac{1}{16} & \frac{\nu}{4} - \frac{3}{16} & \frac{1}{16}\\\frac{\nu}{4} - \frac{1}{16} & \frac{\nu}{8} + \frac{1}{16} & - \frac{1}{16} & - \frac{\nu}{8} + \frac{3}{16} & - \frac{\nu}{4} + \frac{1}{16} & - \frac{\nu}{8} - \frac{1}{16} & \frac{1}{16} & \frac{\nu}{8} - \frac{3}{16}\\\frac{\nu}{4} - \frac{3}{16} & - \frac{1}{16} & \frac{1}{16} & - \frac{\nu}{4} + \frac{1}{16} & - \frac{\nu}{4} + \frac{3}{16} & \frac{1}{16} & - \frac{1}{16} & \frac{\nu}{4} - \frac{1}{16}\\- \frac{1}{16} & \frac{\nu}{8} - \frac{3}{16} & \frac{\nu}{4} - \frac{1}{16} & - \frac{\nu}{8} - \frac{1}{16} & \frac{1}{16} & - \frac{\nu}{8} + \frac{3}{16} & - \frac{\nu}{4} + \frac{1}{16} & \frac{\nu}{8} + \frac{1}{16}\\\frac{1}{16} & \frac{\nu}{4} - \frac{1}{16} & \frac{\nu}{4} - \frac{3}{16} & \frac{1}{16} & - \frac{1}{16} & - \frac{\nu}{4} + \frac{1}{16} & - \frac{\nu}{4} + \frac{3}{16} & - \frac{1}{16}\\- \frac{\nu}{4} + \frac{1}{16} & - \frac{\nu}{8} - \frac{1}{16} & \frac{1}{16} & \frac{\nu}{8} - \frac{3}{16} & \frac{\nu}{4} - \frac{1}{16} & \frac{\nu}{8} + \frac{1}{16} & - \frac{1}{16} & - \frac{\nu}{8} + \frac{3}{16}\end{matrix}\right]$$

```python
K_2pts = zeros(8, 8)
for row in range(8):
    for col in range(8):
        K_2pts[row, col] = sum(wts[1][kx]*wts[1][ky]*
                               K_inte[row, col].subs({r: pts[1][kx], s:pts[1][ky]})
                               for kx in range(2) for ky in range(2))

```

```python
simplify(K_2pts - K)
```

$$\left[\begin{matrix}0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right]$$

```python
K_3pts = zeros(8, 8)
for row in range(8):
    for col in range(8):
        K_3pts[row, col] = sum(wts[2][kx]*wts[2][ky]*
                        K_inte[row, col].subs({r: pts[2][kx], s:pts[2][ky]})
                        for kx in range(3) for ky in range(3))

```

```python
simplify(K_3pts - K)
```

$$\left[\begin{matrix}0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right]$$

```python

```

```python

```

# References

1. Bathe, Klaus-Jürgen. Finite element procedures. Klaus-Jurgen Bathe, 2006.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
/* Based on Lorena Barba template available at: https://github.com/barbagroup/AeroPython/blob/master/styles/custom.css*/
@font-face {
font-family: "Computer Modern";
src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}
div.cell{
width:800px;
margin-left:16% !important;
margin-right:auto;
}
h1 {
font-family: 'Alegreya Sans', sans-serif;
}
h2 {
font-family: 'Fenix', serif;
}
h3{
font-family: 'Fenix', serif;
margin-top:12px;
margin-bottom: 3px;
}
h4{
font-family: 'Fenix', serif;
}
h5 {
font-family: 'Alegreya Sans', sans-serif;
}
div.text_cell_render{
font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
line-height: 135%;
font-size: 120%;
width:600px;
margin-left:auto;
margin-right:auto;
}
.CodeMirror{
font-family: "Source Code Pro";
font-size: 90%;
}
/* .prompt{
display: None;
}*/
.text_cell_render h1 {
font-weight: 200;
font-size: 50pt;
line-height: 100%;
color:#CD2305;
margin-bottom: 0.5em;
margin-top: 0.5em;
display: block;
}
.text_cell_render h5 {
font-weight: 300;
font-size: 16pt;
color: #CD2305;
font-style: italic;
margin-bottom: .5em;
margin-top: 0.5em;
display: block;
}
.warning{
color: rgb( 240, 20, 20 )
}
</style>

```python

```


### Introduction to Random Variables and Statistical Distributions

```python
from scipy.stats import bernoulli, poisson, binom, norm, mvn
import numpy as np
from matplotlib import pyplot as plt
import matplotlib
%matplotlib inline
```

```python
headimg = plt.imread('../data/quarterheads.jpg')
tailimg = plt.imread('../data/quartertails.jpg')
```

### Discrete random variables
A discrete random variable can take a finite number of possible values. A toss of a two-sided coin can be thought of as a random variable, or a roll of a 6 sided dice.

#### Probability distributions for discrete random variables
A random variable is generated from a probability distribution. There are different types of distributions defined for discrete random variables. These include:
- Bernoulli distribution
- Binomial distribution
- Multinoulli distribution
- Poisson distribution

#### Bernoulli distribution
Bernoulli distribution represents a binary random variable which takes value 1 with success probability $p$ and value 0 with failure probability $q=1-p$. A bernoulli distribution has only one parameter: $p$.

```python
theta = 0.5
# let us draw a sample from a bernoulli distribution
b = bernoulli.rvs(theta,size=1)
print(b)
if b[0] == 0:
    plt.imshow(tailimg)
    plt.axis('off')
else:
    plt.imshow(headimg)
    plt.axis('off')

```

```python
# you can also draw samples simultaneously
samples = bernoulli.rvs(theta,size=1000)
print(samples)
# count the number of successes (sample = 1). What happens when you change p?
print(np.count_nonzero(samples==1))
```

    [0 0 0 1 0 1 0 0 1 0 1 1 0 1 1 1 0 0 1 0 0 0 0 0 1 1 1 0 1 1 1 0 0 0 0 0 0
     1 0 1 0 1 0 1 0 0 1 1 1 0 0 0 1 1 0 0 0 1 0 0 1 0 0 1 1 1 1 1 0 0 1 0 0 1
     0 0 0 1 1 0 1 0 0 1 0 1 1 0 1 1 1 1 1 1 1 1 1 1 0 0 1 1 0 0 0 0 0 0 0 1 0
     1 1 1 0 0 1 1 1 0 0 1 1 0 0 1 1 1 1 0 0 1 1 0 1 0 0 0 0 0 1 1 1 1 1 1 1 1
     1 1 1 1 1 0 0 0 1 0 1 1 1 1 1 0 0 1 1 0 1 0 1 0 0 0 0 1 0 0 0 1 0 1 0 0 1
     0 0 1 0 1 0 1 1 1 0 0 0 0 1 1 0 0 1 1 0 0 1 0 1 0 0 1 0 1 0 0 0 1 1 1 1 1
     1 0 1 0 1 0 1 0 1 0 1 1 0 0 0 0 1 0 1 1 1 0 1 1 0 0 0 1 0 0 0 1 1 0 1 0 0
     0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 1
     1 0 0 1 1 0 1 0 0 1 1 1 0 0 1 0 1 1 1 0 0 1 0 0 0 1 1 0 0 0 0 1 0 0 1 1 1
     0 0 0 0 0 1 1 1 0 1 0 1 1 1 1 0 1 0 0 1 0 1 1 1 1 1 0 0 1 1 1 0 0 1 1 1 1
     0 1 0 0 0 1 0 0 1 0 1 0 0 0 0 1 0 1 0 1 1 0 0 1 1 0 1 0 0 1 1 1 1 0 1 1 0
     0 1 1 1 1 1 1 0 1 0 1 1 0 1 1 1 1 1 0 0 1 0 0 0 0 1 0 0 0 1 0 0 1 1 0 1 0
     0 0 1 0 0 0 1 0 1 0 1 0 1 1 1 1 0 0 0 1 1 0 0 1 0 1 0 1 0 0 1 1 1 1 0 0 0
     1 1 0 0 0 1 0 0 1 1 0 0 0 0 0 1 0 0 1 0 0 1 1 0 1 1 1 0 1 1 1 1 1 1 0 1 1
     0 0 0 0 1 1 1 1 1 1 1 0 1 0 1 0 0 0 1 0 0 0 1 0 0 1 0 1 0 0 1 1 0 0 1 1 0
     1 1 1 0 0 0 1 0 0 1 0 1 0 0 0 1 1 1 1 1 1 1 0 1 1 0 0 0 0 0 1 1 0 0 1 1 1
     1 0 0 0 0 0 1 1 0 1 1 1 0 0 0 1 1 1 1 0 0 0 1 0 1 0 1 1 1 0 1 1 0 1 1 0 0
     0 1 0 0 0 1 0 1 1 1 0 0 1 1 0 0 1 0 1 1 0 1 0 0 1 1 1 1 1 0 1 1 1 0 1 1 0
     0 0 1 1 0 1 0 0 0 1 0 1 0 0 1 1 0 1 0 1 1 0 1 1 1 0 0 1 1 1 1 1 1 0 0 1 1
     0 0 1 1 0 0 0 1 0 1 1 0 0 0 0 0 1 1 1 0 1 1 1 1 0 0 0 0 1 1 1 0 0 1 0 0 0
     1 0 1 1 0 0 1 0 0 0 1 0 0 0 0 0 1 1 0 1 0 1 1 0 0 0 1 1 1 1 1 1 0 1 1 0 0
     0 0 0 0 1 0 0 0 1 0 0 0 1 1 1 1 0 1 0 1 1 0 1 1 1 1 1 1 0 0 0 0 0 0 1 1 1
     0 1 0 1 0 0 0 1 0 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 0 1 0 0 1 1 0 0 1 1 1 0
     0 0 1 0 1 1 0 0 0 0 0 0 0 1 1 0 1 0 0 1 0 1 0 0 1 0 1 1 0 1 1 1 1 1 0 0 1
     1 0 1 1 1 0 0 1 0 1 1 1 1 1 1 1 1 1 1 1 0 0 1 0 1 0 0 0 0 0 0 0 1 0 0 0 0
     0 0 0 0 1 0 1 0 1 1 1 1 0 0 0 0 1 0 0 1 0 1 0 1 0 1 0 0 1 0 1 1 1 1 0 1 0
     0 1 1 0 0 1 1 1 1 0 0 1 0 1 0 0 0 1 0 0 1 0 1 1 0 1 1 0 1 1 0 1 0 1 0 1 0
     0]
    487

```python
# plotting the probability mass function for the Bernoulli distribution
a = np.arange(2)

colors = matplotlib.rcParams['axes.color_cycle']
plt.figure(figsize=(12,8))
for i, theta in enumerate([0.1, 0.2, 0.6, 0.7]):
    ax = plt.subplot(1, 4, i+1)
    plt.bar(a, bernoulli.pmf(a, theta), label=theta, color=colors[i], alpha=0.2)
    ax.xaxis.set_ticks(a)

    plt.legend(loc=0)
    if i == 0:
        plt.ylabel("PDF at $k$")

plt.suptitle("Bernoulli probability")
```

#### Binomial distribution
Another popular distribution for a discrete random variable is the binomial distribution. A binomial distribution has two parameters $n$ and $\theta$, where $0 \le \theta \le 1$. The sample generated by a binomial distribution denotes the number of successes observed in a sequence of $n$ binary trials (e.g., toss of a coin) when the probability of each success is $\theta$.

The samples that are drawn from a binomial distribution range between 0 and $n$.

The probability distribution is defined as:
\begin{equation}
p(k;n,\theta) = P(X = k) = \binom{n}{k}\theta^k (1 - \theta)^{n-k}
\end{equation}

```python
#sampling from a binomial distribution
sample = binom.rvs(20,0.4,1)
print sample
```

    10

```python
#plotting the pmf for a bernoulli distribution
plt.figure(figsize=(12,6))
k = np.arange(0, 22)
for p, color in zip([0.1, 0.3, 0.6, 0.8], colors):
    rv = binom(20, p)
    plt.plot(k, rv.pmf(k), lw=2, color=color, label=p)
    plt.fill_between(k, rv.pmf(k), color=color, alpha=0.3)
plt.legend()
plt.title("Binomial distribution")
plt.tight_layout()
plt.ylabel("PDF at $k$")
plt.xlabel("$k$")
```

#### Multinoulli distribution
A multinoulli distribution is a generalization of Bernoulli distribution for trials which can take more than two possibilities ($k > 2$). The parameter for multinoulli distribution is a vector ${\bf \theta}$ which has $k$ entries. Each entry $\theta_i$ indicates the probability of observing the category $i$ in a single trial.

```python
# generate samples from a multinoulli distribution. Essentially simulated a single roll of dice. Note that the output is a vector of length $k = 6$
np.random.multinomial(1, [1/6.]*6, size=1)
```

#### Multinomial distribution
A mulinomial distribution is a generalization of Binomial distribution for trials which can take more than two possibilities. The parameters for the multinomial distribution is a vector ${\bf \theta}$ and $n$.

```python
# generate samples from a multinomial distribution. Note that the output is a vector of length $k = 6$
np.random.multinomial(20, [1/6.]*6, size=1)
```

### Continuous random variables
A continuous random variable can take an infinite number of possible values. Several interesting distributions exist:
- alpha An alpha continuous random variable.
- beta A beta continuous random variable.
- gamma A gamma continuous random variable.
- expon An exponential continuous random variable.
- gauss Gaussian random variable

#### Gaussian distribution
One of the most popular distribution is the Gaussian distribution. This distribution is defined for any number of variables. For a single variable case, the distribution is defined using two parameters: $\mu$ and $\sigma$. $\mu$ or the mean can take any value and $\sigma$ or the standard deviation is $\ge 0$.

For a continuous distribution, you cannot compute the probability mass at any value of the random variable. Instead, you can compute the density using the probability density function:
$$p(x) = \frac{1}{\sigma\sqrt{2\pi}}\exp[-\frac{1}{2}(\frac{x - \mu}{\sigma})^2]$$
The random variable represented using a Gaussian distribution can take any value from $-\infty$ to $\infty$.

```python
# set the parameters
mu = 0
sigma = 1
# draw 1000 samples from this distribution
samples = norm(mu, sigma).rvs(1000)
# plot an empirical distribution, i.e., a histogram
#plt.hist(samples, 30, normed=True, alpha=.3)

# Compute the density at several instances of the random variable
x = np.linspace(-10, 10, 10001)
# plot the density
plt.plot(x, norm(mu, sigma).pdf(x), linewidth=2)

```

#### Multi-dimensional or multivariate Gaussian distribution
A distribution can be defined for multivariate random variables. One example is the multivariate Gaussian. In general, the random variable is a $D$ length vector ${\bf X}$. The two parameters of this distribution are a mean vector ${\bf \mu}$ and a covariance matrix $\Sigma$. The pdf at any value of ${\bf X}$ is given by:
$$
      \mathcal{N}({\bf X}| {\bf \mu,\Sigma}) \triangleq \frac{1}{(2\pi)^{D/2}|{\bf \Sigma}|^{D/2}}exp\left[-\frac{1}{2}{\bf (x-\mu)^\top\Sigma^{-1}(x-\mu)}\right]
$$
Note that if $D = 1$, it reduces to a univariate Gaussian distribution.

```python
#define the parameters for D = 2
mu = np.array([10,10])
Sigma = np.array([[1,0.],[0.,1]])
rv = np.random.multivariate_normal(mu,Sigma)
#sample some points
s = np.random.multivariate_normal(mu,Sigma,10000)

fig = plt.figure()
plt.subplot(111)
plt.scatter(s[:,0],s[:,1])

# add a contour plot
smin = np.min(s,axis=0)
smax = np.max(s,axis=0)
t1=np.linspace(smin[0],smax[0],1000)
t2=np.linspace(smin[1],smax[1],1000)

# evaluate pdf at each of these mesh points

```

```python
np.cov(s.transpose())
```

    array([[  9.89485661e-01,  -1.43077154e-02],
           [ -1.43077154e-02,   9.97270669e+01]])


This notebook is structured as follows:

*    Evaluate the disadvantages of implementing the logistic regression algorithm.

* Learn the basics concepts of Naive Bayes model.

* Learn about the how the algorithm works.

* Implementation of NB classifier.

* Contrast the performance between the algorithms: logistic regression and NB, for classification problems, using AUC/ROC metrics.

```
import pandas as pd
import pydotplus
import numpy  as np
import matplotlib.pyplot as plt
import seaborn as sns
import statsmodels.api as sm

plt.style.use('ggplot')

from io import StringIO
from IPython.display import Image, SVG
from graphviz import Source
from matplotlib import cm
from matplotlib.colors import ListedColormap, LinearSegmentedColormap
from matplotlib.patches import Patch
from scipy.stats import chi2_contingency, norm
from sklearn.tree import export_graphviz
from sklearn.metrics import roc_curve, auc, accuracy_score
from sklearn.model_selection import StratifiedKFold, train_test_split
from statsmodels.formula.api import ols
from sympy import var, plot_implicit, Eq

# ignore log(0) and divide by 0 warning
np.seterr(divide='ignore');
```

```
df = pd.read_csv('/content/drive/MyDrive/data_davivienda/Lending_club.csv',
                 dtype = {'loan_status':'category', 'annual_inc':'float',
                          'verification_status':'category', 'emp_length':'category',
                          'home_ownership':'category', 'int_rate':'object',
                          'loan_amnt':'float', 'purpose':'category',
                          'term':'category', 'grade':'category'})
df['int_rate'] = df['int_rate'].str.rstrip('%').astype('float')
```

```
df.head()
```

      loan_status
      annual_inc
      verification_status
      emp_length
      home_ownership
      int_rate
      loan_amnt
      purpose
      term
      grade

      0
      Fully Paid
      24000.0
      Verified
      10+ years
      RENT
      10.65
      5000.0
      credit_card
      36 months
      B

      1
      Charged Off
      30000.0
      Source Verified
      < 1 year
      RENT
      15.27
      2500.0
      car
      60 months
      C

      2
      Fully Paid
      12252.0
      Not Verified
      10+ years
      RENT
      15.96
      2400.0
      small_business
      36 months
      C

      3
      Fully Paid
      49200.0
      Source Verified
      10+ years
      RENT
      13.49
      10000.0
      other
      36 months
      C

      4
      Fully Paid
      80000.0
      Source Verified
      1 year
      RENT
      12.69
      3000.0
      other
      60 months
      B

##Disadvantages of logistic regression##
Although logistic regression is one of the most commonly used classification algorithms, it is not the only one. If the underlying relationship between the logit of odds and the covariates is not linear, we need to consider higher order terms of the covariates to make the logistic model valid. Doing so requires a lot of tuning and experimenting. In such cases, it is often better to look at a classification algorithm that does not rely on a specific assumption about the relationship between the probability of the outcome and the covariates. Therefore we are going to introduce some of the most popular models

**Bayes Theorem**

To solve a problem using Bayesian methods, we have to specify two functions: the likelihood function p(X|θ),which describes the probability of observing a dataset X for a given value of the unknown parameters θ, and the prior distribution p(θ), which describes any knowledge we have about the parameters before we collect the data. Note that the likelihood should be considered as a function of the parameters θ with the data X held fixed. The prior distribution and the likelihood functionare used to compute the posterior distribution p(θ|X) via Bayes’ rule:
>  p($\theta$|X)=$\frac{p(X|\theta) \ p(\theta)}{\int d\theta'\ p(X|\theta')\ p(\theta')}$

In many cases, it will not be possible to analytically compute the normalizing constant in the denominator ofthe posterior distribution, p(X) =∫dθp(X|θ)p(θ),and Markov Chain Monte Carlo (MCMC) methods are needed to draw random samples from p(θ|X).[1]

**Deduction of Bayes Theorem**

Let B be an event whose chance of occurrence must be measured under the assumption that an event A has been observed. If the experiment is repeated, under the same conditions, n times then the relative frequency of B under condition A is defined as follows:

> $fr(B|A)=\frac{n(A\cap B)}{n(A)}; if\quad n(A)>0$

where n(A∩B) is the number of success cases to A∩B. When the experiment is performed a large enough number of times, we have:

> $fr(B|A)=\frac{\frac{n(A\cap B)}{n}}{\frac{n(A)}{n}}; if\quad n(A)>0$

If n$\rightarrow \ \infty$, the last equation becomes:

>  $P(B|A)= \frac{P(A\cap B)}{P(A)}$

From the last equation we can define the independency (other important concept)of two events as follows:

> $P(A \cap B)= P(A)P(B)$

Recalling the following definitions we can get two important results:

> **sample space**: $\Omega=\{ \omega_n \}\quad n \in (0, \infty)$ $\quad$where $\omega_i\cap\omega_j=\phi$:

**Total probability rule**

$P(B)= P(B\cap \Omega)$ = $P(\cup_n(B \cap \omega_n )$ = $\sum_n P(B\cap \omega_n)$ = $\sum_n P(B|\omega_n)P(\omega_n)$

from the last rule we get the next corollary:

Corollary:  Be it $\omega_1, \omega_2...$ a finite sample of $\Omega$ with P(A_i)> 0 for all $i$ we have:

> $P(\omega_i| B)=\frac{P(\omega_i) P(B| \omega_i)}{\sum_j P(B| \omega_j) P(\omega_j)} $

> Proof:

> $P(\omega_i|B)=\frac{P(\omega_i \cap B)}{P(B)}= \frac{P(\omega_i)P(B|\omega_i)}{P(B)}= \frac{P(\omega_i)P(B|\omega_i)}{\sum_j P(B|\omega_j)P(\omega_j)} $

if $j\ \rightarrow \ \infty$ then:

> p($\omega$|X)=$\frac{p(X| \omega) \ p(\omega)}{\int d\omega'\ p(X|\omega')\ p(\omega')}$

The last equation is the continuous version of Bayes rule[2]

##Naive Bayes Clasiffier##

This algorithm is based on Bayes' theorem, which is a conditional probability theorem (see previous section). The principal idea for this project is:

> Given a client with certain characteristics, we will use Bayes' theorem and the observed data to estimate the probability that defaulters and non-defaulters have the same characteristics as the client. The larger of these two probabilities would then determine how we classify the client.

Let's go to determine whether $\text{P(defaulter| income verified)}$ is larger than $\text{P(non-defaulter| income verified)}$ using Bayes' theorem:

>  $\text{P(defaulter| income verified)}=\frac{\text{P(income verified|  defaulter)}\ \text{P(defaulter)}}{\text{P(income verified)}}$ ; $\text{P(non-defaulter|  income verified)}=\frac{\text{P(income verified|  non-defaulter)} \ \text{P(non-defaulter)}}{\text{P(income verified)}}$

In order to get the answer we need to do an assumption and verify if this is true or not:

>  $\text{P(defaulter|  income verified)} \ \text{P(defaulter)}$< $\text{P(non defaulter|  income verified)}\ \text{P(non-defaulter)}\quad (1)$

First we are going to calculate the probability $\text{P(defaulter)}$ and $\text{P(non-defaulter)}$:

```
p_def=df[df['loan_status']=='Charged Off']['loan_status'].count()/df['loan_status'].count()
print('P(defaulter)={:.3f} \nP(non-defaulter)={:.3f}'.format(p_def, 1-p_def))
```

    P(defaulter)=0.141
    P(non-defaulter)=0.859

Now we are goingt to calculate the probability $\text{P(income verified| defaulter)}$ and $\text{P(income verified| non-defaulter)}$:

```
p_id=df[(df['loan_status']=='Charged Off') & (df['verification_status']=='Verified')]['loan_status'].count()/df[df['loan_status']=='Charged Off']['loan_status'].count()
p_ind=df[(df['loan_status']=='Fully Paid') & (df['verification_status']=='Verified')]['loan_status'].count()/df[df['loan_status']=='Fully Paid']['loan_status'].count()
print('P(income verified given defaulter)={:.3f} \nP(income verified given non-defaulter)={:.3f}'.format(p_id, p_ind))
```

    P(income verified given defaulter)=0.363
    P(income verified given non-defaulter)=0.313

We can evaluate the equation (1)

```
p_def_iv=p_id*p_def
p_ndef_iv=p_ind*(1-p_def)
print('P(defaulter|income verified)  is proportional to {:.3f}  \
\nP(non-defaulter|income verified) is proportional to {:.3f}'.format(p_def_iv, p_ndef_iv))
```

    P(defaulter|income verified)  is proportional to 0.051
    P(non-defaulter|income verified) is proportional to 0.269

 If we only have information about the income verification status of our clients, we would probably give loans to those clients whose income has been verified. What happens if the client is either `Source Verified` or `Not Verified`? We can repeat the same procedure:

```
p_id2=df[(df['loan_status']=='Charged Off') & (df['verification_status']=='Source Verified')]['loan_status'].count()/df[df['loan_status']=='Charged Off']['loan_status'].count()
p_ind2=df[(df['loan_status']=='Fully Paid') & (df['verification_status']=='Source Verified')]['loan_status'].count()/df[df['loan_status']=='Fully Paid']['loan_status'].count()

p_def_iv2=p_id2*p_def
p_ndef_iv2=p_ind2*(1-p_def)
print('P(defaulter|income source verified)  is proportional to {:.3f}  \
\nP(non-defaulter|income source verified) is proportional to {:.3f}'.format(p_def_iv2, p_ndef_iv2))
```

    P(defaulter|income source verified)  is proportional to 0.037
    P(non-defaulter|income source verified) is proportional to 0.217

```
p_id3=df[(df['loan_status']=='Charged Off') & (df['verification_status']=='Not Verified')]['loan_status'].count()/df[df['loan_status']=='Charged Off']['loan_status'].count()
p_ind3=df[(df['loan_status']=='Fully Paid') & (df['verification_status']=='Not Verified')]['loan_status'].count()/df[df['loan_status']=='Fully Paid']['loan_status'].count()

p_def_iv3=p_id3*p_def
p_ndef_iv3=p_ind3*(1-p_def)
print('P(defaulter|income not verified)  is proportional to {:.3f}  \
\nP(non-defaulter| income not verified) is proportional to {:.3f}'.format(p_def_iv3, p_ndef_iv3))
```

    P(defaulter|income not verified)  is proportional to 0.053
    P(non-defaulter| income not verified) is proportional to 0.373

The conditional probabilities of being a non-defaulter are always bigger than those for being a defaulter, remenber that in the notebook of logistic regression we find a Simsonp paradox for the `verified_status` and `loan_status`

##The algorithm and assumptions##

There are two possible routes that we can follow to create a classification algorithm from Bayes' theorem.

#### Probabilistic Model

Recall that in logistic regression, the main quantity of interest is the probability of no default (remember that our encoding was 1 for non-defaulters and 0 for defaulters) given specific values for the covariates. We explicitly modeled this probability as a linear function of covariates after transformation by the logit function.

The Naive Bayes classifier aims to estimate this probability from a different perspective. From Bayes' theorem, we can express this conditional probability in the following way:

$$\text{P(non-defaulter|covariates=values)} = \frac{\text{P(covariates=values| non-defaulter)P(defaulter)}}{\text{P(covariates=values)}}.$$

**Then, we create our classifier as in the logistic model by selecting a threshold $t$ and classify a client with given covariates values as a defaulter if:**

$$
\text{P(non-defaulter|covariates=values)}>t.
$$

#### Maximum A Posteriori Model (MAP)

For this classifier, a client with given covariates values is classified as a non-defaulter if:

$$
\text{P(non-defaulter|covariates=values)}>\text{P(defaulter|covariates=values)}
$$

which is equivalent to:

$\text{P(covariates=values|non-defaulter)P(non-defaulter)}>\text{P(covariates=values|defaulter)P(defaulter)}$
**Note: This assumption is the same of the equation (1)**

Let's take a look at each of these quantities:

1. $\text{P(defaulter)}$ and $\text{P(non-defaulter)}$. These are the probabilities of having a defaulter and a non-defaulter. These can be easily estimated using the proportion of defaulters in the dataset.
2. $\text{P(covariates=values)}$. This is the probability of having a customer whose covariates are equal to $\text{values}$. This can also be estimated by proportions, but we will actually find out that it is not necessary to estimate this quantity at all (more on this later).
3. $\text{P(covariates=values| non-defaulter)}$ and $\text{P(covariates=values| defaulter)}$. We can estimate these by looking at the proportion of defaulters and non-defaulters with covariates equal to $\text{values}$. But if we have a lot of covariates, the number of such defaulters might be extremely small or even equal to zero. The main challenge in building a Naive Bayes classifier is estimating these particular probabilities.

Indeed, when the number of covariates is large, it is hard to estimate $\text{P(covariates=values given non-defaulter)}$ by proportions directly. For instance, suppose that we have 10 covariates and that all of them are binary variables. Then, the total number of possible values these covariates can take is equal to $2^{10}=1024$. Thus, in order to estimate $\text{P(covariates=values given non-defaulter)}$ we would need at least $1024$ samples! If we want to go even further, suppose we want to classify texts having most 82 words, where each word in the text has 10 possible alternatives. The number of samples we would need will be larger than the number of atoms in the universe!

##Assumption of Naive##

All classifiers start from a master algorithm and are differentiated from a different assumption for each classifier. For example, in this case,the Naive Bayes classifier makes the assumption that *all covariates are independent when conditioned on the outcome*.

$\text{P(covariates=values given non-defaulter)} = \text{P(covariate}_1\text{=value}_1\text{ given non-defaulter)}\times\cdots\times\text{P(covariate}_p\text{=value}_p\text{ given non-defaulter)}$

IMPORTANT: **Here we assume there are $p$ covariates in total. This significantly reduces the amount of samples required for the estimation! For instance, consider the example with 10 covariates, all of which are binary variables. Here, we need to estimate 10 simpler conditional probabilities and each one of them only concerns one binary variable, which has 2 possible outcomes. So we only need at least $2×10=20$ samples now, which is much smaller than $1024$.**

##What about continuous covariates?##
Computing proporitions works great for calculating the probabilities for discrete covariates. But what about variables like `annual_inc` that take on a continuous range of values? For these, we need to fit a *distribution* to the data. In the natural world, many things are governed by a normal distribution. To fit a normal distribution, all we need to specify is its mean and variance. The normal distribution with mean equal to the sample mean and variance equal to the sample variance of the covariate turns out to be a very good approximation. Once we get this normal distribution, the probabilities can be calculated directly for any covariate value.

Find the normal distribution that most aligned with the observed distribution of `annual_inc` in defaulters. Based on this normal distribution, calculate $\text{P(annual_inc=10,000|defaulter)}$. Function `pdf()` in `scipy.stats.norm`
calculates the density of a normal distribution at a given value.

Let's fit the continuous variables with some distribution in this case (normal):

```
df_default = df[df['loan_status']=='Charged Off']
df_ndefault=df[df['loan_status']=='Fully Paid']

mean_est = df_default['annual_inc'].mean()
var_est = df_default['annual_inc'].var()

# pdf takes standard deviation as scale parameter
# note we have log10 transformed annual_inc so when call norm.pdf, we should evaluate at 4
print(f"P(annual_inc=10,000|defaulter) = {norm.pdf(10000, loc = mean_est, scale = np.sqrt(var_est))}")
```

    P(annual_inc=10,000|defaulter) = 4.473123137400813e-06

The last exercise was very easy to obtain from the data but, what about the inverse case? $\text{P(defaulter|annual_inc=10,000)}$ It is not trivial, right?

Now, we claim that we can use the following procedure to get $\text{P(non-defaulter|covariates=values)}$:

1. Estimate $\text{P(defaulter)}$ and $\text{P(non-defaulter)}$.
2. Estimate $\text{P(covariates=values|defaulter)}$ and $\text{P(covariates=values given non-defaulter)}$
3. Estimate $\text{P(non-defaulter|covariates=values)}$ with the following equation:

$$
    \frac{\text{P(covariates=values| non-defaulter)}\text{P(non-defaulter)}}{\text{P(covariates=values|defaulter)}\text{P(defaulter)}+\text{P(covariates=values|non-defaulter)}\text{P(non-defaulter)}}
$$

##Feature engenier##

```
# plot the distribution of the continuous variable
sns.violinplot(x='loan_status', y='loan_amnt', data=df);
```

```
#We have to standardize  this variable to view better results
ax = sns.violinplot(x=df['loan_status'], y=np.log10(df['annual_inc']));
ax.set(ylabel='log annual_inc');
```

```
sns.violinplot(x='loan_status', y='int_rate', data=df);
 #Here the distribution are very different in the two types of users
```

However, the distribution for `int_rate` is close enough to normal, but that of `loan_amnt` seems a bit skewed (although nowhere near the degree of `annual_inc`. Let's apply a square root transformation:

```
ax = sns.violinplot(x = df['loan_status'], y = np.sqrt(df['loan_amnt']));
ax.set(ylabel='sqrt loan_amnt');
```

From the above exercise, we find that we need to transform `annual_inc` and `loan_amnt` so that the normality assumption can be approximately satisfied. We will use the transformed variables to construct the Naive Bayes estimator:

```
df['annual_inc_log'] = np.log10(df['annual_inc'])
df['loan_amnt_sqrt'] = np.sqrt(df['loan_amnt'])
```

## **Implementing the classifier**------------------------------------**Algorithm**----------------------------------

```
#function to fit the continous variables a normal distribution

"""
Learn parameters for continuous covariate
y is a binary outcome
X is a pandas series of a continuous covariate
"""
def nb_fit_con(y, X):
    y_code = y.cat.codes
    X_0 = X[y_code==0]
    X_1 = X[y_code==1]
    return [[np.mean(X_0), np.std(X_0)], [np.mean(X_1), np.std(X_1)]]
```

Next, we write a function `nb_fit_dis` to estimate the probabilities of each level of a discrete covariate in defaulters and non-defaulters:

```
"""
Learn parameters for discrete covariate
y is a binary outcome
X is a pandas series of a discrete covariate
"""
def nb_fit_dis(y, X):
    y_code = y.cat.codes
    X_code = X.cat.codes
    X_nlvl = len(X.cat.categories)

    X_0 = X_code[y_code==0]
    X_1 = X_code[y_code==1]

    X_0_freq = [np.sum(X_0==x)/len(X_0) for x in range(X_nlvl)]
    X_1_freq = [np.sum(X_1==x)/len(X_1) for x in range(X_nlvl)]
    return [X_0_freq, X_1_freq]
```

The following function uses the two estimation functions we wrote above to train a Naive Bayes classifier:

```
"""
y is a binary outcome
X is a pandas dataframe of covariates
"""
def train_nb_classifier(y, X):
    trained_model = {}
    for ind_col in X.columns:
        # check data type
        if X[ind_col].dtype=='float':
            params_fit = nb_fit_con(y, X[ind_col])
        else:
            params_fit = nb_fit_dis(y, X[ind_col])
        trained_model[ind_col] = params_fit
    margin_p_0 = np.sum(y.cat.codes==0)/len(y)
    trained_model['margin_p'] = [margin_p_0, 1-margin_p_0]
    return trained_model
```

Once the model is trained, the following function `predict_nb_classifier` predicts the probability of not defaulting for new samples:

```
"""
log of normal density
"""
def log_dnorm(x, loc, scale):
    return -(x-loc)**2/2/(scale**2) - np.log(scale) - 0.5*np.log(2*np.pi)

"""
prediction
X_test: a pandas dataframe of covariates for prediction
        discrete vars have the same coding scheme as in the train dataset
"""

def predict_nb_classifier( nb_fit, X_test ):
    pred_0_p = np.ones(X_test.shape[0] )*np.log(nb_fit['margin_p'][0])
    pred_1_p = np.ones(X_test.shape[0] )*np.log(nb_fit['margin_p'][1])

    for ind_col in X_test.columns:
        params = nb_fit[ind_col]
        if X_test[ind_col].dtype=='float':
            #increment prob for outcome 0 for all samples
            update_0 = log_dnorm(X_test[ind_col], loc = params[0][0], scale = params[0][1])
            update_1 = log_dnorm(X_test[ind_col], loc = params[1][0], scale = params[1][1])
        else:
            update_0 = [np.log(params[0][x]) for x in X_test[ind_col].cat.codes]
            update_1 = [np.log(params[1][x]) for x in X_test[ind_col].cat.codes]
        pred_0_p = pred_0_p + update_0
        pred_1_p = pred_1_p + update_1

    #calculate the estimated probability of outcome=1
    #specifically handle Inf
    p_ratio = np.exp(pred_1_p-pred_0_p)
    p_1 = p_ratio/(1+p_ratio)
    p_1[p_ratio==np.Inf] = 1

    return p_1
```

```
def cv_roc_nb(X, y, n_split):

    cv = StratifiedKFold(n_split)

    tprs = []
    aucs = []
    mean_fpr = np.linspace(0, 1, 100)
    for train, test in cv.split(X, y):
        nb_trained = train_nb_classifier(y.iloc[train], X.iloc[train])
        probas_ = predict_nb_classifier(nb_trained, X.iloc[test])

        # Compute ROC curve and area the curve

        fpr, tpr, thresholds = roc_curve(y.iloc[test].cat.codes, probas_)
        tprs.append(np.interp(mean_fpr, fpr, tpr))
        tprs[-1][0] = 0.0
        roc_auc = auc(fpr, tpr)
        aucs.append(roc_auc)

    return tprs, aucs, mean_fpr
```

```
# fit NB
tprs_nb, aucs_nb, mean_fpr_nb = cv_roc_nb(df[['term', 'verification_status', 'annual_inc_log']], df['loan_status'], 5)
```

```
#@title LogisticR DATA
# normalize and set-up the data
df_log = pd.concat([(df['loan_amnt'] - df['loan_amnt'].mean())/df['loan_amnt'].std(),
                     (df['int_rate'] - df['int_rate'].mean())/df['int_rate'].std(),
                     (df['annual_inc'] - df['annual_inc'].mean())/df['annual_inc'].std(),
                     pd.get_dummies(df['home_ownership'], prefix='home', drop_first=True),
                     pd.get_dummies(df['purpose'], prefix='purpose', drop_first=True),
                     pd.get_dummies(df['grade'], prefix='grade', drop_first=True)], axis=1)
df_log['verification_status'] = df['verification_status'].cat.codes
df_log['emp_length'] = df['emp_length'].cat.codes
df_log['term'] = df['term'].cat.codes
df_log['Intercept'] = 1

# fit the logistic regression model
logit_full1 = sm.Logit(df['loan_status'].cat.codes, df_log)
logit_full1_res = logit_full1.fit()
predict_withvs = logit_full1_res.predict(df_log)

# generate cross-validated ROC and AUC for logistic regression model

def cv_roc(X, y, n_split):

    cv = StratifiedKFold(n_split)

    tprs = []
    aucs = []
    mean_fpr = np.linspace(0, 1, 100)
    for train, test in cv.split(X, y):
        logit = sm.Logit(y.iloc[train], X.iloc[train])
        logit_res = logit.fit(disp=0)
        probas_ = logit_res.predict(X.iloc[test])
        # Compute ROC curve and area under the curve
        fpr, tpr, thresholds = roc_curve(y.iloc[test], probas_)
        tprs.append(np.interp(mean_fpr, fpr, tpr))
        tprs[-1][0] = 0.0
        roc_auc = auc(fpr, tpr)
        aucs.append(roc_auc)

    return tprs, aucs, mean_fpr
```

    Optimization terminated successfully.
             Current function value: 0.378195
             Iterations 7

```
# fit logistic
tprs_lgs, aucs_lgs, mean_fpr_lgs = cv_roc(df_log[['Intercept', 'term', 'verification_status', 'annual_inc']], df['loan_status'].cat.codes, 5)

```

```
#Comparation
plt.plot([0, 1], [0, 1], linestyle='--', lw=2, color='blue',
         label='Random guess', alpha=.8)

mean_tpr_nb = np.mean(tprs_nb, axis=0)
mean_tpr_nb[-1] = 1.0
mean_tpr_lgs = np.mean(tprs_lgs, axis=0)
mean_tpr_lgs[-1] = 1.0

mean_auc_nb = auc(mean_fpr_nb, mean_tpr_nb)
std_auc_nb = np.std(aucs_nb)
mean_auc_lgs = auc(mean_fpr_lgs, mean_tpr_lgs)
std_auc_lgs = np.std(aucs_lgs)

plt.plot(mean_fpr_nb, mean_tpr_nb, color='darkorange', lw=2, alpha=.8,
         label=f"Naive Bayes (AUC = {mean_auc_nb:0.2f} $\pm$ {std_auc_nb:0.2f})")
plt.plot(mean_fpr_lgs, mean_tpr_lgs, color='red', lw=2, alpha=.8,
         label=f"Logistic regression(AUC = {mean_auc_lgs:0.2f} $\pm$ {std_auc_lgs:0.2f})")

plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend(loc='lower right')
plt.title('Comparison of NB and Logistic')
plt.show()
```

## Conclusions

In this case, we learned about a few new classification models and the advantages and disadvantages they have versus the standard logistic model. We learned that the Naive Bayes classifier, although "naive", generally does not suffer much and in fact is the fastest to train. Furthermore, the implementation of Naive Bayes is quite math-heavy and relies on a simplifying assumption of independence. This makes Naive Bayes bad when covariates are obviously dependent.
Note: For the Naive Bayes classifier can assume distributions other than the normal distribution a get better results.

#References

[1] Pankaj Mehta, Ching-Hao Wang, Alexandre G. R. Day, and Clint Richardson, A high-bias, low-variance introduction to Machine Learning for physicists. 27 May 2019. https://arxiv.org/pdf/1803.08823.pdf

[2] Liliana Blanco, Probabilidad. Universidad Nacional de Colombia. Facultad de Ciencias 2004.

[3] DS4A Correlation 3.0, Course materials, 2020.

```

```

```

```


# Molar Volume and Compressibility Factor from Van Der Waals Equation
This is the first problem of the famous set of [Ten Problems in Chemical Engineering](https://www.polymath-software.com/ASEE/Tenprobs.pdf).  Here, the goal is to solve a single nonlinear algabraic equation.

Jacob Albrecht, 2019

# Problem Setup
Use the van der Waals equation of state, given by:

$$\left(P+\frac{a}{V^2}\right)\left(V-b\right)=RT$$

where

$$a=\frac{27}{64}\left(\frac{R^2T^2_c}{P_c}\right)$$

$$b=\frac{RT_c}{8P_c}$$

- $P$ is the pressure in atm
- $V$ is the molar volume in liter/g-mol
- $T$ is temperature in Kelvin
- $R$ is the ideal gas constant (R = 0.08206 atm.liter/g-mol.K)
- $T_c$ is the critical temperature (405.5 K for ammonia)
- $P_c$ is the critical pressure (111.3 atm for ammonia)

Reduced pressure is:

$$P_r=\frac{P}{P_c}$$

And compressibility is:

$$Z=\frac{PV}{RT}$$

# Problem Tasks
a) Calculate the molar volume and compressibility factor for gaseous ammonia at 56 atm and 450 K by using the van der Waals equation of state.

b) Repeat the calculations for the following reduced pressures: $P_r$ = 1, 2, 4, 10, and 20.

c. How does the compressibility factor vary as a function of $P_r$?

# Solutions

Because we need to rearrange the equation, we can use the `sympy` package to do the algebra for us:

```python
 import sympy
```

```python
P,V,N,R,T,a,b,Pc,Pr,Tc,Z = sympy.symbols('P V N R T a b Pc Pr Tc Z')
```

```python
eq1 = sympy.Eq((P+a/V**2)*(V-b),R*T)
a_def = sympy.Eq(27/64*(R**2*Tc**2/Pc),a)
b_def = sympy.Eq(R*Tc/(8*Pc),b)
```

Lets confirm we've entered in the equations

```python
eq1
```

$\displaystyle \left(P + \frac{a}{V^{2}}\right) \left(V - b\right) = R T$

```python

```

```python
a_def
```

$\displaystyle \frac{0.421875 R^{2} Tc^{2}}{Pc} = a$

```python
b_def
```

$\displaystyle \frac{R Tc}{8 Pc} = b$

We can refresh the values for $a$ and $b$ with subsitution:

```python
eq1_subs = eq1.subs({a:27/64*(R**2*Tc**2/Pc),b:R*Tc/(8*Pc)})
eq1_subs
```

$\displaystyle \left(P + \frac{0.421875 R^{2} Tc^{2}}{Pc V^{2}}\right) \left(V - \frac{R Tc}{8 Pc}\right) = R T$

## Solution to part a)
Subsitute the Pressure and Temperature given for part a:

```python
eq1_subs_part_a = eq1_subs.subs({P:56,R:0.08206,T:450,Pc:111.3,Tc:405.5})
```

Next, we can substitute the other values, and solve for $V$

```python
root = sympy.solveset(eq1_subs_part_a,symbol = V, domain=sympy.S.Reals)
root.args[0]
```

$\displaystyle 0.574891913557151$

Solve for $Z$

```python
(P*V/(R*T)).subs({P:56,R:0.08206,T:450,V:root.args[0]})
```

$\displaystyle 0.871826770633966$

## Solution to part b)

Loop through the different reduced pressures, and make a table of the results:

```python
Vs = []
Zs = []
Prs = [1,2,4,10,20]
for Pr in Prs:
    eq1_subs_part_b = eq1_subs.subs({P:Pr*111.3,R:0.08206,T:450,Pc:111.3,Tc:405.5})
    root = sympy.solveset(eq1_subs_part_b,symbol = V, domain=sympy.S.Reals)
    Vs.append(root.args[0])
    Zs.append((P*V/(R*T)).subs({P:Pr*111.3,R:0.08206,T:450,V:root.args[0]}))
```

```python
import pandas as pd
%matplotlib inline
part_b = pd.DataFrame({'Reduced Pressure':Prs,'Molar Volume':Vs,'Compressibility factor':Zs},dtype='float')
part_b
```

      Reduced Pressure
      Molar Volume
      Compressibility factor

      0
      1.0
      0.233509
      0.703808

      1
      2.0
      0.077268
      0.465777

      2
      4.0
      0.060654
      0.731261

      3
      10.0
      0.050875
      1.533409

      4
      20.0
      0.046175
      2.783482

## Solution to part c)
Plot the Compressibility factor vs. Reduced pressure

```python
part_b.plot('Reduced Pressure','Compressibility factor');
```

# Reference
“The Use of Mathematical Software packages in Chemical Engineering”, Michael B. Cutlip, John J. Hwalek, Eric H.
Nuttal, Mordechai Shacham, Workshop Material from Session 12, Chemical Engineering Summer School, Snowbird,
Utah, Aug., 1997.

```python
%load_ext watermark
%watermark -v -p sympy,pandas,matplotlib
```

    CPython 3.7.3
    IPython 7.6.1

    sympy 1.4
    pandas 0.25.1
    matplotlib 3.1.0

```python

```