# Machine Learning: Basic Principles 2018
# Round 2 - Regression

## Learning goals

In this exercise you will learn how to use **linear regression**, in order to predict a quantity of interest based on data. The implementation of linear regression amounts to the minimization of a function (**the empirical risk**) which measures the error of a predictor when applied to training data. You will hear about **gradient descent (GD)** which is a simple but powerful algorithm for finding the minimum of a function. Variations of GD are the main algorithmic tools behind many state-of-the art machine learning methods (such as deep learning).
A good understanding of GD is therefore a worthy asset for an ML engineer. Expanding the basic linear regression model you will learn some additional techniques such as **feature scaling** (or **data normalization**)
and **feature mapping** to obtain **polynomial regression**.

(Hint: Completing this exercise will help you in future rounds where you might
reuse substantial parts of the code developed in this exercise.)

### Keywords

`Linear Regression`, `Gradient Descent (GD)`, `Mean Squared Error (MSE)`, `Empirical Risk Minimization (ERM)`, `Polynomial Regression`,`Data Visualization`

## 1 Introduction

In this exercise we will apply __linear regression__ to explore the dependencies between the two cryptocurrencies **Ethereum** and **Bitcoin**. In particular, we will try to __predict the price of the cryptocurrency Ethereum [2] from the price of the cryptocurrency Bitcoin [1]__.

> ### References
>(only background info, not required for completing the exercise)
>
>[1] Cointelegraph, 2013-2018. What is bitcoin? Here's everything you need to know. Blockchains, bubbles and the future of money. https://www.cnet.com/how-to/what-is-bitcoin/ [Accessed 13th May 2018].
>
>[2] Jaffe, J., 2018. What is Ethereum. Guide for Beginners. https://cointelegraph.com/ethereum-for-beginners/what-is-ethereum  [Accessed 13th May 2018].
>

## 2 The Data
In order to predict one currency from the other we start by looking at some historic data. The data is stored in two files; one file for each cryptocurrency:
* BTC-USD.csv (Bitcoin prices in US Dollar)
* ETC-USD.csv (Ethereum prices in US dollar)

These files contain time series of daily prices (in US dollar) and traded volumes of Bitcoin and
Etherum throughout the last three years. The files can be found on the website https://finance.yahoo.com

The following code snippet provides you some basic tools for reading in the dataset stored in these files and how to visualize this data.

```python
import pandas as pd # import Pandas library (and defining shorthand "pd") for reading and manipulating the data files
from matplotlib import pyplot as plt # import and define shorthand "plt" for library "pyplot" providing plotting functions
from IPython.display import display, HTML
import numpy as np   # import and define shorthand "np" for library "numpy" for advanced mathematical operations in python

#read in data from csv files
#parse_dates function is used on Date-column to change them from string to date-object

df_bc = pd.read_csv("BTC-USD.csv",parse_dates=['Date'])
df_eth = pd.read_csv("ETH-USD.csv",parse_dates=['Date'])

## Show top rows of each file.
# function "display()" is Jupyter Notebook command to show multiple function outputs from one cell.
display(HTML(df_bc.head(5).to_html(max_rows=5)))
display(HTML(df_eth.head(5).to_html(max_rows=5)))

## Plot original data to same figure
# Example of use: plt.plot(x,y)
plt.plot(df_bc.Date.values, df_bc.Close.values, label=("Bitcoin price (in USD)"))
plt.plot(df_eth.Date.values, df_eth.Close.values, label=("Ethereum price (in USD)"))

## set up figure title and labels for x- and y-axis
plt.title(r'$\bf{Figure\ 1.}$ Bitcoin vs Ethereum')
plt.xlabel('date')
plt.ylabel('price')

## rotate x-ticks by 20 degrees
plt.xticks(rotation=20)

## enable legend-box for plot labels
plt.legend()

## show the plot
plt.show()
```

## 3 Exercise

Our ML problem involves data points $\mathbf{z}$ which contain all parameters of the two cryptocurrencies Bitcoin and Ethereum at a particular day. As the feature $x$ of a data point $\mathbf{z}$ we choose the closing price of Bitcoin on a particular day, while the label $y$ is the Ethereum closing price.

The actual exercise starts from here and it has been divided into 6 tasks:

* 3.1 Getting Hands on the Data
* 3.2 Scatterplots
* 3.3 Linear Regression
* 3.4 Polynomial Regression
* 3.5 Gradient Descent

Your task is to fill in `...` under `### STUDENT TASK ###` in each step.

## 3.1 Getting Hands on the Data

In order to learn (or find) a good predictor $h(x)$ which approximates the Ethereum closing price $y$ based on the Bitcoin closing price $x$, we make use of the data stored in the two csv files, `BTC-USD.csv` and `ETC-USD.csv`. In particular, these files provide us with a labeled dataset $\mathcal{X} = \{(x^{(i)},y^{(i)} \}_{i=1}^{N}$. The dataset $\mathcal{X}$ contains $N$ data points $z^{(i)}$, for $i=1,\ldots,N$, with the features $x^{(i)}$ which we define as the closing price of Bitcoin at day $i$. The quantity of interest (the label) $y^{(i)}$ of the $i$-th data point is the closing price of Ethereum at day $i$. We would like to predict the Ethereum closing price $y^{(i)}$ solely from the Bitcoin closing price $x^{(i)}$ on the same day.

As evident from Figure 1, comparing the closing prices of Bitcoin and Ethereum is difficult visually because of the significantly different value ranges of Bitcoin and Ethereum prices. Therefore, in order to facilitate a better intuition for the relation between the two currencies, we will normalize the prices to the same value range [3]:

\begin{equation}
\label{scale_features_labels}
\tag{1}
 x^{(i)}_{\rm scaled} = \frac{x^{(i)}-\min_{i} x^{(i)}}{\max_{i} x^{(i)}-\min_{i} x^{(i)}},
   y^{(i)}_{\rm scaled} = \frac{y^{(i)}-\min_{i} y^{(i)}}{\max_{i} y^{(i)}-\min_{i} y^{(i)}}
\end{equation}

The normalization results in having the same value range [0,1] for both currencies.

> ### References
>
> [3] Feature Scaling. In Wikipedia. https://en.wikipedia.org/wiki/Feature_scaling#Methods [Accessed 13th June 2018].
>

### Tasks:
1. Read in the closing prices of Ethereum and Bitcoin (stored in column with the caption "Close") from the csv files
2. Preprocess the data
    1. Normalize \eqref{scale_features_labels} Bitcoin prices from column called __Close__ and save them into a variable `bitcoin`.
    2. Normalize \eqref{scale_features_labels} Ethereum prices from column called __Close__ and save them into a variable `ethereum`.
4. Create a **Figure 2** which depicts the time series of rescaled prices.

The following code snippet imports all Python libraries which are required for this exercise.

```python
# import all libraries to be used in this exercise

import pandas as pd
from matplotlib import pyplot as plt
import numpy as np
from IPython.display import display
```

```python
import pandas as pd
from matplotlib import pyplot as plt
import numpy as np
from plotchecker import LinePlotChecker

def normalize(x, min, max):
    return (x - min) / (max - min)

def normalize_features(features):
    return list(map(lambda i: normalize(i, min(features), max(features)), features))

def solution1():
    # We assign Global variables for testing purposes
    global df_bc, df_eth, bitcoin, ethereum, axis, best_alpha
    # read in historic Bitcoin and Ethereum statistics data from the files "BTC-USD.csv" and "ETH-USD.csv"

    df_bc = pd.read_csv("BTC-USD.csv", parse_dates=['Date'])
    df_eth = pd.read_csv("ETH-USD.csv", parse_dates=['Date'])

    # read the dates for the individual records from column "Date"
    bitcoin_date = df_bc.Date.values
    ethereum_date = df_eth.Date.values

    ### STUDENT TASK ###
    ## read in closing prices for Bitcoin and Ethereum from the column "Close" and computed the rescaled values
    ## Replace '...' with your solution.
    #bitcoin = [...]
    #ethereum = [...]

    # YOUR CODE HERE
    xs = df_bc.Close.values
    ys = df_eth.Close.values

    bitcoin = normalize_features(xs)
    ethereum = normalize_features(ys)

    # Show cryptocurrency prices over transaction date
    axis = plt.gca()
    plt.plot(bitcoin_date, bitcoin, label=("Bitcoin"))
    plt.plot(ethereum_date,ethereum, label=("Ethereum"))
    plt.title(r'$\bf{Figure\ 2.}$ Normalized cryptocurrency prices')
    plt.xlabel('Date')
    plt.ylabel('Normalized price')
    plt.xticks(rotation=20)
    plt.legend()
```

```python
solution1()
assert len(bitcoin) == 948,"Your bitcoin data is incorrect length"
assert len(ethereum) == 948, "Your ethereum data is incorrect length"
assert np.max(bitcoin) == np.max(ethereum), "Incorrect max values after normalisation"
assert np.min(bitcoin) == np.min(ethereum), "Incorrect min values after normalisation"

# Run test that check that plot renders correctly. Requires plotchecker to be installed.
from plotchecker import LinePlotChecker
pc = LinePlotChecker(axis)
pc.assert_num_lines(2)
pc.find_permutation('title',r'$\bf{Figure\ 2.}$ Normalized cryptocurrency prices')
pc.find_permutation('xlabel','Date')
pc.find_permutation('ylabel','Normalized price')
pc.assert_labels_equal(['Bitcoin','Ethereum'])
```

## 3.2 Scatterplots

Scatter plots are a helpful tool for getting an idea of the relation between features and labels.

### Task:
1. Generate a scatter plot using the Python function `plt.scatter()`. The x-axis of the scatter plot represents the values of the rescaled Bitcoin prices $x_{\rm scaled}^{(i)}$ and y-axis represents the values of the rescaled Ethereum prices $y_{\rm scaled}^{(i)}$.

```python
### STUDENT TASK ###
#x=...
#y=...
# YOUR CODE HERE

x = bitcoin
y = ethereum

plt.figure(figsize=(8, 8))
plt.scatter(x,y)
plt.title(r'$\bf{Figure\ 3.}$Normalized cryptocurrency prices ($x^{(i)},y^{(i)}$)')
plt.xlabel('normalized Bitcoin prices')
plt.ylabel('normalized Ethereum prices')
plt.annotate('$(x^{(i)},y^{(i)})$', xy=(x[913], y[913]), xytext=(0.1, 0.5),
             arrowprops=dict(arrowstyle="->",facecolor='black'),
             )
axis = plt.gca()
```

```python
# Check that plot is a scatterplot. Requires plotchecker
from plotchecker import ScatterPlotChecker
pc = ScatterPlotChecker(axis)
assert len(pc.x_data) == 948
assert len(pc.y_data) == 948
```

## 3.3 Linear Regression

Our goal is to predict the closing price $y^{(i)}$ of Ethereum at some day $i$ based on the Bitcoin closing price $x^{(i)}$ on the very same day. Within linear regression, we try to do this prediction using linear functions which belong to the hypothesis space

\begin{equation*}
 \mathcal{H} = \{h^{(\mathbf{w})}(\mathbf{x}) = \mathbf{w}^{T} \mathbf{x} \mbox{ for some } \mathbf{w} \in \mathbb{R}\}.
\label{eq1}
\tag{1}
\end{equation*}

In particular, we try to predict (or approximate) $y^{(i)}$ by the function value $h^{(\mathbf{w})}(\mathbf{x}^{(i)}) = \mathbf{w}^{T}\mathbf{x}^{(i)}$ using the augmented feature vector $\mathbf{x}^{(i)} = (x^{(i)},1)^{T}$. At first sight, it seems strange to add a constant one to each feature value $x^{(i)}$ (which is the closing price of Bitcoin at day i). However, it will turn out to allow for a larger class of functions $h^{(\mathbf{w})}$ which can be used for the prediction.

The prediction $h^{(\mathbf{w})}(\mathbf{x}^{(i)})$ will typically incur a non-zero __prediction error__ $y^{(i)} - h^{(\mathbf{w})}(\mathbf{x}^{(i)})$, which we quantify using its square $(y^{(i)} - h^{(\mathbf{w})}(\mathbf{x}^{(i)})^2$ (squared error loss). Since we have $N$ data points $(x^{(i)},y^{(i)})$ available, we can compute the __empirical risk__ (average loss)
\begin{equation*}
 \mathcal{E} (\mathbf{w}) = \frac{1}{N}\sum^{N}_{i=1}(y^{(i)} - \mathbf{w}^{T} \mathbf{x}^{(i)})^2.
\label{eq2}
\tag{2}
\end{equation*}

Our goal is to find the optimal predictor $h_{\rm opt}(\cdot)$:
\begin{equation}
h_{\rm opt}(\cdot) = \underset{{h(\mathbf{w})} \in \mathcal{H}}{\operatorname{argmin}} \mathcal{E} (\mathbf{w}).
\label{eq3}
\tag{3}
\end{equation}

Note that every function $h^{(\mathbf{w})}(\mathbf{x}) \in \mathcal{H}$ corresponds to a particular choice of the weight vector $\mathbf{w} \in \mathbb{R}^{2}$. This allows us to rewrite the optimization problem (Eq. \ref{eq3}) as an optimization problem for the weight vector:
\begin{align}
\mathbf{w}_{\rm opt} &= \underset{ \mathbf{w} \in \ \mathbb{R}^{2}}{\operatorname{argmin}} \{ \frac{1}{N}\sum^{N}_{i=1}(y^{(i)} - \mathbf{w}^{T} \mathbf{x}^{(i)})^2 \}\\
&= \underset{ \mathbf{w} \in \ \mathbb{R}^{2}}{\operatorname{argmin}} \frac{1}{N} {||\mathbf{y}-\mathbf{X}\mathbf{w}||}_2^2,
\label{eq4}
\tag{4}
\end{align}
where
\begin{align}
\mathbf{X} = \big(\mathbf{x}^{(1)},\ldots,\mathbf{x}^{(N)}\big)^{T} \text{and }\mathbf{y} = \begin{bmatrix}y^{(1)}\\\vdots\\y^{(n)}\end{bmatrix}\ \mathbf{X} \in \mathbb{R}^{N \times 2},\mathbf{y}\in\mathbb{R}^{N}.
\end{align}
An optimal weight vector $\mathbf{w}_{\rm opt}$ solving \eqref{eq4} induces an optimal predictor $h_{\rm opt}= h^{(\mathbf{w}_{\rm opt})}$ which solves \eqref{eq3}.

Whenever the matrix $\mathbf{X}^T \mathbf{X}$ is invertible, the solution of (\ref{eq4}), i.e., the optimal weight which minimizes the empirical risk, is obtained as
\begin{align}
\mathbf{w}_{\rm opt} = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}.
\label{eq5}
\tag{5}
\end{align}

### Tasks
0. Implement a Python function `featureMatrix(x)` which takes a length-$N$ vector of Bitcoin closing prices $x^{(i)}$, for $i=1,\ldots,N$ as input and outputs the feature matrix $\mathbf{X}=\big(\mathbf{x}^{(1)},\ldots,\mathbf{x}^{(N)}\big)^{T} \in \mathbb{R}^{N \times 2}$ whose rows are the augmented feature vectors $\mathbf{x}^{(i)} = \big(x^{(i)},1\big)^{T} \in \mathbb{R}^{2}$.
1. Implement a Python function `fit(X,y)` that takes feature matrix $\mathbf{X}$ and label vector $\mathbf{y}$ as inputs and returns the optimal weight vector according to (Eq. \ref{eq5}.)
2. Implement a Python function `predict(X,w)` that takes the feature matrix $\mathbf{X}$ and weight vector $\mathbf{w}$ as input and returns a vector containing the predicted labels according to (Eq. \ref{eq1}.)
3. Implement a Python function `empirical_risk(X, y, w)` taking feature matrix $\mathbf{X}$, label vector $\mathbf{y}$ and weight vector $\mathbf{w}$ and returns the empirical error (Eq. \ref{eq2}).
4. Implement a Python function `linearRegression(X,y)` which outputs the empirical error of your prediction and the optimal weight vector
    * Modify our feature matrix and label vector by adding a dummy feature (i.e. 1). This can be done by using helper functions `featureMatrix(x)` and `labelVector(y)`
    * use these modified $\mathbf{X}$ and $\mathbf{y}$ to fit the data and calculate the empirical error using the Python functions that you defined in previous steps.
5. Execute the cell. You should see a plot similar to $\bf{Figure\ 3.}$, but with a legend showing the empirical error and your prediction line in red.

```python
def featureMatrix(x):
    # Generate x_i = (x_i, 1) feature matrix
    # x_i = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    return x_i

def fit(x, y):
    ### STUDENT TASK ###
    ## Compute optimal w by replacing '...' with your solution.
    ## Hints: Check out numpy's linalg.inv(), dot() and transpose() functions.
    # w_opt = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    return w_opt

# Predict new y data
# return numpy.ndarray:
#      [ [y_pred_1]
#          ....
#        [y_pred_N]]
def predict(X, w_opt):
    ### STUDENT TASK ###
    ## Predict new y data by replacing '...' with your solution.
    ## Hint! Use X and w_opt to get necessary matrices.
    # y_pred ...
    # YOUR CODE HERE
    raise NotImplementedError()
    return y_pred

# Calculate empirical error of the prediction
# return float
def empirical_risk(X, Y, w_opt):
    ### STUDENT TASK ###
    ## Compute empirical error by replacing '...' with your solution.
    ## Hints! Use X, Y and w_opt to get necessary matrices.
    ##        Check out numpy's dot(), mean(), power() and subtract() functions.
    # empirical_error = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    return empirical_error

def linearRegression(x, y):
    ### STUDENT TASK ###
    ## Calculate X, Y, w_opt and empirical_error
    ## Hints! Use featureMatrix() and labelVector() to get necessary matrices, X and Y.
    # X = ...
    # Y = ...
    # w_opt=...
    # empirical_error=...
    # YOUR CODE HERE
    raise NotImplementedError()
    return w_opt, empirical_error

def labelVector(y):
    # Reshape y to ensure correct behavior when doing matrix operations
    return np.reshape(y,(len(y),1))

def draw_plot(x, y, title=''):
    w_opt, empirical_error = linearRegression(x, y)
    x_pred = np.linspace(0,1.1,100)
    y_pred = predict(featureMatrix(x_pred), w_opt)
    # Plot data points and linear regression fitting line
    plt.figure(figsize=(8, 8))
    plt.scatter(x, y)
    plt.plot(x_pred,y_pred,'r', label=("Empirical = %.4f" % empirical_error))
    plt.title(title)
    plt.xlabel('Bitcoin')
    plt.ylabel('Ethereum')
    plt.legend()
    axis = plt.gca()

######### Linear regression model for x and y data #########
draw_plot(x, y, r'$\bf{Figure\ 4.}$ Normalized cryptocurrency prices')
```

```python
w_opt, empirical_error = linearRegression(x, y)
assert empirical_error < 0.015
w_opt, empirical_error = linearRegression([0,1,2,3], [0,1,2,3])
# Because of computational rounding errors, empirical error is almost never exactly 0
assert empirical_error < 1e-30

```

## 3.4 Polynomial Regression

By looking at $\bf{Figure\ 3.}$ we see that relation between label $y$ (Ethereum price) and feature $x$ (Bitcoin price) is highly non-linear. Therefore it is useful to consider a hypothesis space which is constituted by polynomial functions
\begin{equation}
\label{equ_def_poly_hyposapce}
\mathcal{H}^{(d)}_{\rm poly} = \{ h^{(\mathbf{w})}(\cdot): \mathbb{R} \rightarrow \mathbb{R}: h^{(\mathbf{w})}(x) = \sum_{r=1}^{d} w_{r} x^{r-1} \mbox{, with some } \mathbf{w} =(w_{1},\ldots,w_{d})^{T} \in \mathbb{R}^{d} \}.
\end{equation}

As in task `3.3 Linear Regression`, the quality of a predictor is measured by the squared error loss and linear regression amounts to minimizing the  average squared error loss (mean squared error):
\begin{equation}
\min_{h \in \mathcal{H}_{\rm poly} } \frac{1}{N} \sum_{i=1}^{d} (y^{(i)} - h^{(\mathbf{w})}(x^{(i)}))^{2} \mbox{, where } h^{(\mathbf{w})}(x)=\sum_{r=1}^{d} w_{r} x^{r-1}.
\end{equation}

As discussed in the course book (Section 3.2 "Polynomial Regression") polynomial regression is equivalent to combining linear regression with a feature map. In particular, we transform the features $x$ (Bitcoin closing price) of the data points to a higher dimensional feature space using the feature map

\begin{equation}
\phi(x) = (x^{d-1},..,x^1, x^0)^{T} \in \mathbb{R}^{d}.
\label{eq7}
\tag{7}
\end{equation}

This feature map takes the original feature $x \in \mathbb{R}$ (the Bitcoin closing price) as input and returns a new feature vector $\mathbf{x}= \phi(x) \in \mathbb{R}^{d}$ of length $d$. The resulting feature matrix is
\begin{equation*}
    \mathbf{X}_{poly}=\begin{bmatrix}
        (x^{(1)})^{d-1} & \dots & (x^{(1)})^{1}& (x^{(1)})^{0}\\
        (x^{(2)})^{d-1} & \dots & (x^{(2)})^{1}& (x^{(2)})^{0}\\
      \vdots & \ddots& \vdots & \vdots\\
       (x^{(N-1)})^{d-1} & \dots & (x^{(N-1)})^{1}& (x^{(N-1)})^{0}\\
      (x^{(N)})^{d-1} & \dots & (x^{(N)})^{1}& (x^{(N)})^{0}
    \end{bmatrix}.
\label{eq8}
\tag{8}
\end{equation*}

Polyonimal regression is then nothing but linear regression with the tranformed feature vectors $\mathbf{x}^{(i)} = \phi(x^{(i)})$, with $x^{(i)}$ being the Bitcoin closing price at day $i$.

### Tasks

1. Implement a Python function `feature_mapping(x,d)` which reads in vector of feature values $x^{(i)}$ of dataset $\mathcal{X}$ and returns the feature matrix \eqref{eq8} with given degree $d$ (by default $d=2$).
2. Implement a Python function `polynomialRegression(x,y,degree=1)` which is based on the following steps
    * use `feature_mapping(x,d)` to obtain the feature matrix \eqref{eq8}
    * compute the optimal weight vector `w_opt` by inserting the feature matrix \eqref{eq8} into (5)
    * use weight vector to computed predictions $\hat{y}$ for the labels $y$
    * compute the empirical risk obtained when predicting the labels in the dataset $\mathcal{X}$
3. Generate a scatter plot of the dataset $\mathcal{X}$ which includes also the optimal polynomial predictor as a red curve by executing the cell

```python
# Linear regression model for feature mapping.
def polynomialRegression(x, y, degree=1):
    ### STUDENT TASK ###
    ## Calculate w_opt, empirical_error, X and Y
    # X = ...
    # Y = ...
    # w_opt=...
    # empirical_error=...
    # YOUR CODE HERE
    raise NotImplementedError()
    return w_opt, empirical_error

# Extract feature to higher dimensional by computing feature mapping
# return numpy.ndarray with following mappings:
#      [[x_1^(d), x_1^(d-1), x_1^(d-2), ... , x_1^(0)]
#        ...         ...        ...     ...     ...
#       [x_N^(d), x_N^(d-1), x_N^(d-2), ... ,x_N^(0)]]
def feature_mapping(x, degree=1):
    ### STUDENT TASK ####
    ## Compute specified feature mapping by replacing '...' with your solution.
    ## Hints! Use x to get all the feature vectors of the data set.
    ##        Check out numpy's vstack(), hstack() and column_stack() functions.
    # polynomial_features = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    return polynomial_features

def draw_plot(x, y, title='', degree=1):
    w_opt, empirical_error = polynomialRegression(x, y, degree)

    # Change feature values into continues one from 0 to 1.
    x_pred = np.linspace(0,1.1,100)

    # predict new y values using feature mapping
    y_pred = predict(feature_mapping(x_pred, degree), w_opt)

    # Plot data points and linear regression fitting line
    plt.figure(figsize=(8, 8))
    plt.scatter(x, y)
    plt.plot(x_pred,y_pred,'r', label=("Empirical = %.4f" % empirical_error))
    plt.title(title)
    plt.xlabel('Bitcoin')
    plt.ylabel('Ethereum')
    plt.xlim(0,1)
    plt.ylim(0,1)
    plt.legend()
    plt.show()

######### Linear regression model for x and y data #########
draw_plot(x,y,r'$\bf{Figure 5.}$Normalized cryptocurrency prices',degree=11)
```

```python
w_opt, empirical_error = polynomialRegression(x, y)
assert empirical_error < 0.01
w_opt, empirical_error = polynomialRegression([0,1,2,3], [0,1,2,3])
# Because of computational rounding errors, empirical error is almost never 0
assert empirical_error < 1e-30
for i in range(0,100):
    x_test = feature_mapping([0,1,2,3],i)
    assert x_test.shape == (4,1+i)
```

## 3.5 Gradient Descent

Recall `3.3 Linear Regression`, where the optimal predictor $h^{(\mathbf{w}_{\rm opt})}$ was found by minimizing empirical error $\mathcal{E}(\mathbf{w})$ using the closed-form solution

\begin{align}
\mathbf{w}_{\rm opt} = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T \mathbf{y},
\label{eq5}
\tag{5}
\end{align}

with the matrix $\mathbf{X}$ containing the feature vectors $\mathbf{x}^{(i)}$ and the vector $\mathbf{y}$ containing
the labels $y^{(i)}$ of the data points. There are two challenges with using the closed-form expression \eqref{eq5} to get the optimal weight vector $\mathbf{w}_{\rm opt}$: First, it only applies if the matrix $(\mathbf{X}^T \mathbf{X})$ is invertible (note that we have little control over the features as those are obtained from measurements). Second, inverting the matrix $\mathbf{X}^T \mathbf{X}$ might be computationally infeasible if we have a lot (billions) of features. Luckily, there is an alternative method for finding the optimal weight vector $\mathbf{w}_{\rm opt}$ which avoids these two challenges. In particular, this method is based on a very simple method for finding the minimum of a function: gradient descent **(GD)**.

GD amounts to iteratively updating a current guess (or approximation) $\mathbf{w}^{(k)}$ for the optimal weight (which minimizes ([2](#mjx-eqn-eq2))) according to the rule:

\begin{equation*}
    \mathbf{w}^{(k+1)} = \mathbf{w}^{(k)} - \alpha {\nabla}_{\mathbf{w}}\mathcal{E} (\mathbf{w}^{(k)}).
    \label{eq10}
    \tag{10}
\end{equation*}
The choice of the step-size $\alpha$ is crucial for how quickly (and if at all) the iterates $\mathbf{w}^{(k)}$ converge to an optimal weight vector $\mathbf{w}_{opt}$ which yields smallest empirical risk (see `3.3 Linear Regression`).

Note that the implementation of GD requires to compute the gradient ${\nabla}_{\mathbf{w}}\mathcal{E} (\mathbf{w})$.
For linear regression and squared error loss this gradient is obtained as

\begin{equation*}
{\nabla}_{\mathbf{w}}\mathcal{E} (\mathbf{w}) =-\frac{2}{N}\sum^{N}_{i=1}\mathbf{x}^{(i)}(y^{(i)} - {{\mathbf{w}}}^{T} \mathbf{x}^{(i)}) = -\frac{2}{N} \mathbf{X}^{T}(\mathbf{y}-\mathbf{X} \mathbf{w}).
\label{eq11}
\tag{11}
\end{equation*}

GD is started by initalizing the weight vector $\mathbf{w}$ to some value, either randomly or setting $\mathbf{w}^{(0)}=\big(0,\dots,0\big)^{T}$ and then repeating the GD update \eqref{eq10} until some termination criterion is met. Two simple options for the termination criterion are: (i) using a fixed number of iterations or (ii) checking the decrease of the objective function and stopping when the decrease is below a threshold.

It can shown that for a sufficiently small step size $\alpha$, GD always converges to the optimal weight vector $\mathbf{w}_{\rm opt}$, regardless of which inital guess $w^{(0)}$ has been used. The convergence speed depends crucially on the precise choice of the step size $\alpha$: If $\alpha$ is too large, the iterations may not converge at all while if $\alpha$ is too small, the convergence is unacceptably slow.

### Tasks

1. Implement a Python function `gradient()` which computes the gradient according to (Eq. \ref{eq11}). This Python function requires as input a feature matrix $X$, label vector $y$ and returns a vector.
2. Complete Python function `gradient_descent()` which implements the GD update (Eq. \ref{eq10}). This Python function takes feature matrix $X$, label vector $y$ learning rate $\alpha$ and max iterations as inputs and outputs an array of empirical errors.
3. The code snippet below creates a figure entitled "$\bf{Figure\ 6.}$ Converge of GD" which depicts the convergence of GD for different values of the step-size (these different values are in variable `learning_rates`
4. Determine which learning rate yields fastest convergence and store in the variable `best_alpha`
5. Create a $\bf{Figure\ 7.}$ which is identical fo $\bf{Figure\ 6.}$ except for the color coding of the curves (each curve corrsponds to one parituclar step size): the curve corresponding to `best_alpha` should be red and the remaining curves in blue.

### Tips
* `empirical_risk()` and `X`, `Y` -variables are same as in `linearRegression()`!

```python
# Calculate empirical error of the prediction
def empirical_risk(X, Y, w_opt):
    # Assign current weight vector according to iterative weight vector
    ### STUDENT TASK ###
    ## Compute empirical error by replacing '...' with your solution.
    ## Hints! Use X, Y and w_opt to get necessary matrices.
    ##        Check out numpy's dot(), mean(), power() and subtract() functions.
    # empirical_error = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    return empirical_error

# Compute gradient
# return numpy.ndarray:
#      [ [w_1]
#          ....
#        [w_d]]
def gradient(X,Y,w):
    N = X.shape[0]
    ### STUDENT TASK ###
    ## Compute gradient by replacing '...' with your solution.
    ## Hint! Use X and Y to get necessary matrices.
    # gradient = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    return gradient

# Run GD for k steps
# a = alpha/learning rate
# k = iteration steps
# returns empirical error array with lenght k: [ Empirical_1, ..., Empirical_k ]
def gradient_descent(x, y, a, k):
    ### STUDENT TASK ###
    ## Hints! Same as in linearRegression()!
    # X = ...
    # Y = ...
    # YOUR CODE HERE
    raise NotImplementedError()
    # Initial weigth vector (all values 0)
    w = np.zeros((X.shape[1], Y.shape[1]))
    empirical_errors = []

    for i in range(k):
        # Calculate gradient
        grad = gradient(X,Y,w)

        ### STUDENT TASK ###
        ## Update weight vector by replacing '...' with your solution.
        # w = ...
        # YOUR CODE HERE
        raise NotImplementedError()
        ### STUDENT TASK ###
        # Calculate Empirical Risk and append the error into empirical_errors array
        # YOUR CODE HERE
        raise NotImplementedError()
    return empirical_errors

def visualize_error(x, y, learning_rates, best_alpha = None):
    fig, axes = plt.subplots(1, 2,figsize=(12, 4))

    for learning_rate in learning_rates:
        # Plot Error against Step Size
        GD_converge=gradient_descent(x,y,learning_rate, 1000)
        axes[0].plot(GD_converge,label=(r'$\alpha=$'+str(learning_rate)))

    axes[0].set_xlabel('Number of Iterations')
    axes[0].set_ylabel('Empirical Error')
    axes[0].legend(loc=0)
    axes[0].set_title(r'$\bf{Figure\ 6.}$Converge of GD')

    for learning_rate in learning_rates:
        # Plot Error against Step Size.
        # Now mark the best converge in red. Use value from best as a correct step size.
        GD_converge=gradient_descent(x,y,learning_rate, 1000)

        if learning_rate == best_alpha:
            axes[1].plot(GD_converge,label=(r'$\alpha=$'+str(learning_rate)), color="red")
        else:
            axes[1].plot(GD_converge,label=(r'$\alpha=$'+str(learning_rate)), color="blue")

    axes[1].set_xlabel('Number of Iterations')
    axes[1].set_ylabel('Empirical Error')
    axes[1].legend(loc=0)
    axes[1].set_title(r'$\bf{Figure\ 7.}$Converge of GD')
    plt.tight_layout()
    return axes, best_alpha

learning_rates = [0.001, 0.01, 0.02, 0.1, 0.2, 0.3, 0.9, 0.98]

### STUDENT TASK ###
# Change best=None into step size from the list that provides the fastest converge. e.g best=1
###
GD_plots, best = visualize_error(x, y, best_alpha=None, learning_rates=learning_rates)
```

```python
assert best != None, "You haven't specified the best learning rate"
for i in [1,10,100,200,243]:
    res = gradient_descent([0,0.5,1], [0,0.5,1], 1e-5, i)
    assert len(res) == i, "Size of the Error array is incorrect"
    assert np.sum(res)/i < 1, "Your error is way higher than it should be."

# Check that plots are rendered.
from plotchecker import LinePlotChecker
for GD_plot in GD_plots:
    pc = LinePlotChecker(GD_plot)
    pc.assert_num_lines(8)
```


# Repaso (Módulo 1)
> Recordar que el tema principal del módulo 1 son las ecuaciones diferenciales. Entonces, al finalizar este módulo, las competencias principales que deben tener ustedes es
> - Resolver de forma numérica ecuaciones diferenciales ordinarias (EDO) de cualquier orden.
> - Graficar soluciones de dichas EDO en diferentes representaciones.
> - Interpretar o concluir acerca de las gráficas que se obtuvieron.

___
## Ejemplo 1. Recordemos la ecuación logística.
Un modelo popular de crecimiento poblacional de organismos es la llamada **ecuación lógistica**, publicada por Pierre Verhulst en 1838.

$$\frac{dx}{dt} = \mu(x) \; x = r\; (1- x)\; x.$$

En este modelo, $x$ es una variable que representa *cualitativamente* la población. El valor $x=1$ representa la capacidad máxima de población y el valor $x=0$ representa extinción.

Además, $r$ es la tasa de crecimiento máxima de la población.

### Primer punto
Dibuje $\mu(x)=r(1-x)$ para $r=1$, e interprete su significado.

```python
# Importar librerías
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
```

```python
# Definimos función mu
def mu(x, r):
    return r*(1-x)
```

```python
# Parámetro r
r = 1
# Vector de población
x = np.linspace(0, 1.1)
# Calculamos valores de mu para x
y = mu(x, r)
```

```python
# Gráfica
plt.figure(figsize=(8,6))
plt.plot(x, y)
plt.xlabel('Población $x$')
plt.ylabel('Tasa de crecimiento per cápita $\mu(x)$')
plt.grid()
plt.show()
```

**Interpretación:**
- La tasa de crecimiento depende de la población. Cuando la población $x$ es cercana a cero, la tasa de crecimiento per cápita $\mu(x)$ es máxima e igual a $r=1$.
- Adicionalmente, cuando la población es máxima $x=1$, la tasa de crecimiento per cápita $\mu(x)$ es igual a cero.
- Cuando hay sobrepoblación $x>1$, la tasa de crecimiento per cápita es negativa.

### Segundo punto
Para $r=1$ y $x(0)=x_0=0.1$ resuelva numéricamente esta ecuación y grafique $x$ vs. $t$. ¿Qué se puede decir de la población cuando $t\to\infty$?

```python
# Importamos función para integración numérica
from scipy.integrate import odeint
```

```python
# Definimos campo de la ecuación diferencial
def logistica(x, t, r):
    return mu(x, r)*x
# Definimos el parámetro r
r = 1
# Definimos condición inicial
x0 = 0.1
# Definimos el vector de tiempo
t = np.linspace(0, 10)
```

```python
# solución de la ecuación
x = odeint(logistica, x0, t, args=(r,))
```

```python
# Gráfica x vx. t
plt.figure(figsize=(8,6))
plt.plot(t, x)
plt.xlabel('Tiempo')
plt.ylabel('Población')
plt.grid()
plt.show()
```

**Conclusión:**
- Cuando $t\to\infty$, podemos observar que $x(t)\to 1$.
- Además, la población no decrece, de hecho, parece que se va a mantener constante en 1, es decir en su capacidad máxima.

### Tercer punto
Haga un barrido de $-1\leq r\leq 1$ con pasos de $0.5$, resolviendo la ecuación logística numéricamente cada vez y graficando los resultados en una misma gráfica. ¿Para qué valores de $r$ se tiene crecimiento de la población? ¿Para qué valores se tiene extinción?

```python
# Vector de parámetros
rr = np.array([-1, -0.5, 0, 0.5, 1])
```

```python
plt.figure(figsize=(8,6))
for r in rr:
    x = odeint(logistica, x0, t, args=(r,))
    plt.plot(t, x, label='r=%f'%r)

plt.xlabel('Tiempo')
plt.ylabel('Población')
#plt.legend(loc='center left', bbox_to_anchor=(1.05, 0.5))
plt.legend(loc='best')
plt.grid()
plt.show()
```

**Conclusión:**
- Para los valores negativos de $r$, la población se extingue conforme pasa el tiempo.
- Por el contrario, cuando $r$ es positiva, la población llega a su límite (tope, capacidad máxima) conforme pasa el tiempo.
- Cuando $r=0$, la población se mantiene constante en su valor inicial.

___
## Ejemplo 2. Conejos vs. Ovejas.

Imaginemos que en un mismo ecosistema se encuentran conejos y ovejas. Supongamos, además, que ambas compiten por el mismo alimento (hierba) y que la cantidad total de alimento es limitada. Se ignoran otros factores como depredadores, efectos de temporada (estaciones), y otras fuentes de comida. El modelo de competición entre dos especies **Lotka-Volterra** nos sirve para describir este fenómeno.

Dos fenómenos importantes:
- Cada especie crecería hasta su capacidad máxima en ausencia de la otra especie. Esto se puede modelar con la ecuación logística para cada especie. Los conejos tienen una habilidad soprendente para reproducirse, entonces comparativamente deberían crecer más.
- Cuando conejos y ovejas se encuentran, empieza la competencia. Algunas veces los conejos comen, pero las ovejas (al ser más grandes) ganarán el derecho a la comida la mayoría de las veces. Supondremos que dichos conflictos se dan a una tasa proporcional al tamaño de cada población (si hay dos veces más ovejas, la probabilidad de que un conejo encuentre a una oveja será el doble). Supondremos que dicha competencia disminuirá la tasa de crecimiento para cada especie, y el efecto será mayor para los conejos.

Con las consideraciones anteriores, un modelo específico es:

\begin{align}
\frac{dx_1}{dt} &= x_1(3-x_1-2x_2)\\
\frac{dx_2}{dt} &= x_2(2-x_2-x_1),
\end{align}

donde $x_1(t)\geq 0$ es la población de conejos al instante $t$ y $x_2(t)\geq 0$ es la población de ovejas al instante $t$. Definimos $x=\left[x_1\quad x_2\right]^T$.

Esta selección de coeficientes se tiene para recrear el escenario descrito. Sin embargo, este modelo se puede utilizar para estudiar competición entre especies en general y los coeficientes cambiarán en cada caso.

Simule el sistema para cada una de las siguientes condiciones iniciales. Para cada caso, obtenga gráficas de $x_1$ vs. $t$, $x_2$ vs. $t$ y $x_2$ vs. $x_1$. ¿Qué pasa con las poblaciones de conejos y de ovejas cuando $t\to\infty$? ¿Pueden coexistir?

1. $x(0)=\left[x_1(0)\quad x_2(0)\right]^T = [0.5 \quad 1]^T$.
2. $x(0)=\left[x_1(0)\quad x_2(0)\right]^T = [1 \quad 0.5]^T$.
3. $x(0)=\left[x_1(0)\quad x_2(0)\right]^T = [1.5 \quad 1]^T$.
4. $x(0)=\left[x_1(0)\quad x_2(0)\right]^T = [1 \quad 1.5]^T$.
5. $x(0)=\left[x_1(0)\quad x_2(0)\right]^T = [1 \quad 1]^T$.

```python
def lotkaVolterra(x, t):
    x1 = x[0]
    x2 = x[1]
    return np.array([x1*(3-x1-2*x2), x2*(2-x2-x1)])
```

```python
# Primer condición inicial
x0 = np.array([0.5, 1])
# Vector de tiempo
t = np.linspace(0, 20, 100)
```

```python
# Resolver numéricamente
xx = odeint(lotkaVolterra, x0, t)
# Separamos soluciones
x1 = xx[:, 0]
x2 = xx[:, 1]
```

```python
# Gráficas de x1 vs. t y x2 vs. t
plt.figure(figsize=(8,6))
plt.plot(t, x1, label='Población de conejos $x_1(t)$')
plt.plot(t, x2, label='Población de ovejas $x_2(t)$')
plt.xlabel('Tiempo $t$')
plt.axhline(y=0, color='k')
plt.axvline(x=0, color='k')
plt.legend(loc='best')
plt.grid()
plt.show()
```

```python
plt.figure(figsize=(8,6))
plt.plot(x1, x2, label='Trayectoria de las poblaciones')
plt.plot(x1[0], x2[0], '*', ms=10, label = 'Poblaciones iniciales')
plt.plot(x1[-1], x2[-1], '*', ms=10, label = 'Poblaciones finales')
plt.xlabel('Población de conejos $x_1$')
plt.ylabel('Población de ovejas $x_2$')
plt.axhline(y=0, color='k')
plt.axvline(x=0, color='k')
plt.legend(loc = 'best')
plt.grid()
plt.show()
```

**Conclusión:**
- Cuando $t\to\infty$, la población de los conejos se extingue ($x_1(t)\to0$), mientras que la población de ovejas llega a su capacidad máxima ($x_2(t)\to 2$).
- Lo anterior evidencia que bajo las poblaciones iniciales seleccionadas, estas especies no pueden coexistir.

```python

```

```python

```

```python

```

```python

```

```python

```

Referencia:
- Strogatz, Steven. *NONLINEAR DYNAMICS AND CHAOS*, ISBN: 9780813349107, (eBook disponible en biblioteca).

<footer id="attribution" style="float:right; color:#808080; background:#fff;">
Created with Jupyter by Esteban Jiménez Rodríguez.
</footer>


# LU Decomposition

+ This notebook is part of lecture 4 *Factorization into LU* in the OCW MIT course 18.06 [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
import numpy as np
from sympy import *
import matplotlib.pyplot as plt
import seaborn as sns
from IPython.display import Image
from warnings import filterwarnings

init_printing(use_latex = 'mathjax')
%matplotlib inline
filterwarnings('ignore')
```

# LU decomposition of a matrix A

* We will decompose the matrix A into and upper and lower triangular matrix, such that multiplying these will result back into A
$$ A = LU $$

## Turning the matrix of coefficients into Upper triangular form

* Consider the following matrix of coefficients
$$ \begin{bmatrix} 1 & -2 & 1 \\ 3 & 2 & -2 \\ 6 & -1 & -1 \end{bmatrix} $$
* Successive elementary row operation follow
    * Which is nothing other than matrix multiplication of the elementary matrices
    * An elementary matrix is an identity matrix on which one elementary row operation was performed

```python
A = Matrix([[1, -2, 1], [3, 2, -2], [6, -1, -1]])
A
```

$$\left[\begin{matrix}1 & -2 & 1\\3 & 2 & -2\\6 & -1 & -1\end{matrix}\right]$$

```python
eye(3)
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

* We have to get a -3 in the first **pivot** (the 1 in row 1, column 1) to get rid of the 3 in position row 2, column 1 (we call the resulting matrix E21, referring to the row 2, column 1)
* Then we add the new row 1 to row two
    Row one of the identity matrix is then (-3,0,0) (but we leave it (1,0,0) in E21) and adding this to row 2 leaves (-3,1,0)

```python
E21 = Matrix([[1, 0, 0], [-3, 1, 0], [0, 0, 1]])
E21
```

$$\left[\begin{matrix}1 & 0 & 0\\-3 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

```python
E21 * A # The resulting matrix after multiplication by E21
```

$$\left[\begin{matrix}1 & -2 & 1\\0 & 8 & -5\\6 & -1 & -1\end{matrix}\right]$$

* We do the same to get rid of the 6 in row 3, column 1
    * Multiplying row 1 (of the identity matrix) by -6 and adding this new row to row 3 (but again leaving row 1 as (1,0,0) in E31)

```python
E31 = Matrix([[1, 0, 0], [0, 1, 0], [-6, 0, 1]])
E31
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\-6 & 0 & 1\end{matrix}\right]$$

```python
E31 * E21 * A # This got rid of the leading 6 in row 3
```

$$\left[\begin{matrix}1 & -2 & 1\\0 & 8 & -5\\0 & 11 & -7\end{matrix}\right]$$

* Now the 8 in row 2, column 2 is the **pivot** and we need to get rid of the 11 in row 3, column 2
* Unfortunately we have an 8 and an 11 to deal with
* We will have to do two elementary row operations
    * -11 times row 2 of the identity matrix (0,-11,0)
    * Added to 8 times row 3 (0,0,8) ∴ (0,-11,8)

```python
E32 = Matrix([[1, 0 , 0], [0, 1, 0], [0, -11, 8]])
E32
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & -11 & 8\end{matrix}\right]$$

```python
U = E32 * E31 * E21 * A
U # We call is U for upper triangular
```

$$\left[\begin{matrix}1 & -2 & 1\\0 & 8 & -5\\0 & 0 & -1\end{matrix}\right]$$

* The matrix is now in upper triangular form
$$ \left( { E }_{ 32 } \right) \left( { E }_{ 31 } \right) \left( { E }_{ 21 } \right) A=U $$

## Calculating the Lower triangular from

* Note, to reverse this process we would have to do the following:
$$ { \left( { E }_{ 21 } \right)  }^{ -1 }{ \left( { E }_{ 31 } \right)  }^{ -1 }{ \left( { E }_{ 32 } \right)  }^{ -1 }\left( { E }_{ 32 } \right) \left( { E }_{ 31 } \right) \left( { E }_{ 21 } \right) A=A $$
* The inverse of a matrix can be calculated using the sympy method .*inv*()

* We can check this with a Boolean request

```python
E21.inv() * E31.inv() * E32.inv() * E32 * E31 * E21 * A == A # The Boolean double equal signs asks: Is the
# left-hand side equal to the right-hand side?
```

    True

* Indeed, we will be back with the identity matrix just multiplying the inverse elementary matrices and the elementary matrices

```python
E21.inv() * E31.inv() * E32.inv() * E32 * E31 * E21
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

* Multiplying the inverse elementary matrices on the left, must also have it happen on the right
$$ { \left( { E }_{ 21 } \right)  }^{ -1 }{ \left( { E }_{ 31 } \right)  }^{ -1 }{ \left( { E }_{ 32 } \right)  }^{ -1 }\left( { E }_{ 32 } \right) \left( { E }_{ 31 } \right) \left( { E }_{ 21 } \right) A={ \left( { E }_{ 21 } \right)  }^{ -1 }{ \left( { E }_{ 31 } \right)  }^{ -1 }{ \left( { E }_{ 32 } \right)  }^{ -1 }U $$

* The multiplication of these inverse elementary matrices is lower triangular
* We can call in L
$$ A=LU $$

```python
L = E21.inv() * E31.inv() * E32.inv()
L
```

$$\left[\begin{matrix}1 & 0 & 0\\3 & 1 & 0\\6 & \frac{11}{8} & \frac{1}{8}\end{matrix}\right]$$

```python
A == L * U # Checking this with a Boolean question
```

    True

```python
A, L * U # They are identical
```

$$\left ( \left[\begin{matrix}1 & -2 & 1\\3 & 2 & -2\\6 & -1 & -1\end{matrix}\right], \quad \left[\begin{matrix}1 & -2 & 1\\3 & 2 & -2\\6 & -1 & -1\end{matrix}\right]\right )$$

## Doing this in one go using sympy

```python
L, U, _ = A.LUdecomposition()
```

```python
L
```

$$\left[\begin{matrix}1 & 0 & 0\\3 & 1 & 0\\6 & \frac{11}{8} & 1\end{matrix}\right]$$

```python
U # Note the difference from the U above
```

$$\left[\begin{matrix}1 & -2 & 1\\0 & 8 & -5\\0 & 0 & - \frac{1}{8}\end{matrix}\right]$$

```python
L * U # Back to A
```

$$\left[\begin{matrix}1 & -2 & 1\\3 & 2 & -2\\6 & -1 & -1\end{matrix}\right]$$

## What's special about L?

* This only works when no row interchange happens
* It also actually only works when doing the conventional subtracting the scalar multiplication of a row from another row, leaving the positive scalar as opposed to the negatives I use, allowing me to add the two rows (as opposed to subtraction)
* Note the 3 (in row 2, column 1) and the 6 (in row 3, column 1)
* They are the row multiplications we have to do for E21 and E31
* The ¹¹ / ₈ is what we did for E32 (we just did it in two steps so as not to use fractions)

## Row exchanges

* We have to allow row exchanges if the pivot contains a zero

* For an example, from a 3×3 identity matrix we could have:

```python
eye(3)
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

* Exchanging rows one and two would be:

```python
Matrix([[0, 1, 0], [1, 0, 0], [0, 0, 1]])
```

$$\left[\begin{matrix}0 & 1 & 0\\1 & 0 & 0\\0 & 0 & 1\end{matrix}\right]$$

```python
A, Matrix([[0, 1, 0], [1, 0, 0], [0, 0, 1]]) * A # Showing row exchange
```

$$\left ( \left[\begin{matrix}1 & -2 & 1\\3 & 2 & -2\\6 & -1 & -1\end{matrix}\right], \quad \left[\begin{matrix}3 & 2 & -2\\1 & -2 & 1\\6 & -1 & -1\end{matrix}\right]\right )$$

* How many permutations of row exchanges are there?
$$ {n!} $$

* In a 3×3 matrix there are 3! = 6 permutations
    * Multiplying any of them will result in one of the 6
    * They are inverses of each other
    * The inverse are the transposes
* For 4×4 there are 4! = 24

## Example problems

### Example problem 01

* Perform LU decomposition of:
$$  \begin{bmatrix} 1 & 0 & 1 \\ a & a & a \\ b & b & a \end{bmatrix} $$
* For which values of *a* and *b* does L and U exist?

#### Solution

```python
a, b = symbols('a b')
```

```python
A = Matrix([[1, 0, 1], [a, a, a], [b, b, a]])
A
```

$$\left[\begin{matrix}1 & 0 & 1\\a & a & a\\b & b & a\end{matrix}\right]$$

```python
L,U, _ = A.LUdecomposition()
```

```python
L, U
```

$$\left ( \left[\begin{matrix}1 & 0 & 0\\a & 1 & 0\\b & \frac{b}{a} & 1\end{matrix}\right], \quad \left[\begin{matrix}1 & 0 & 1\\0 & a & 0\\0 & 0 & a - b\end{matrix}\right]\right )$$

* Checking

```python
L * U == A
```

    True

* For existence:
    * *a* ≠ 0
* It's easy to see why, since if a equals zero, we will have a zero in a pivot position and we will have to do row exchange, which is not allowed for LU-decomposition

## Hints and tips

```python
E21, E21.inv() # To take the inverse of an elementary matrix, simply change the sign of the off-diagonal elements and
# multiply each element by 1 over the determinant
# The determinant is easy to do for these *n* = 3 square matrices, since the top row is (1,0,0)
```

$$\left ( \left[\begin{matrix}1 & 0 & 0\\-3 & 1 & 0\\0 & 0 & 1\end{matrix}\right], \quad \left[\begin{matrix}1 & 0 & 0\\3 & 1 & 0\\0 & 0 & 1\end{matrix}\right]\right )$$

```python
E31, E31.inv()
```

$$\left ( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\-6 & 0 & 1\end{matrix}\right], \quad \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\6 & 0 & 1\end{matrix}\right]\right )$$

```python
E32, E32.inv()
```

$$\left ( \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & -11 & 8\end{matrix}\right], \quad \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & \frac{11}{8} & \frac{1}{8}\end{matrix}\right]\right )$$

* By keeping track of the elementary matrices it is easy to get L and U
* It's easy to get the inverses of L and U
* This means it is easy to calculate **x**

$$ Ax=LUx=b\\ Ux={ L }^{ -1 }b\\ x={ U }^{ -1 }{ L }^{ -1 }b $$

```python

```


```python
import numpy as np
```

## numpy金融函数

```python
from matplotlib.pyplot import plot, show
# 买一个房子，首付1000w，剩余贷款5年，每年还4次，每期还款10w，年利率0.03
# 计算到期末时一共花了多少钱？即相当于如果5年后全款买这套房子花了多少钱
print("Future value", np.fv(0.03/4, 5 * 4, -10, -1000))
```

    Future value 1376.0963320407982

```python
r=0.03/4
a1=1000*(1+r)**20
a2=10*((1+r)**20-1)/r
```

```python
a1+a2
```

    1376.0963320407982

```python
#现值
print("Present value", np.pv(0.03/4, 5 * 4, -10, 1376.09633204))
```

    Present value -999.9999999993126

```python
#净现值
cashflows = np.random.randint(100, size=5)
cashflows = np.insert(cashflows, 0, -100)
print("Cashflows", cashflows)
```

    Cashflows [-100   24    6   76   79   67]

```python
print("Net present value", np.npv(0.03, cashflows))
```

    Net present value 126.49257777850747

```python
-100+24/(1+0.03)+6/(1+0.03)**2+76/(1+0.03)**3+79/(1+0.03)**4+67/(1+0.03)**5
```

    126.49257777850747

```python
#内部收益率irr
print("Internal rate of return", np.irr([-100, 24 , 6, 76, 79, 67]))
```

    Internal rate of return 0.3092467363712803

```python
print("Internal rate of return", np.irr([-100, 110]))
```

    Internal rate of return 0.10000000000000009

```python
print("Modified internal rate of return", np.mirr([-100, 38, 48, 90, 17, 36], 0.03, 0.03))

```

    Modified internal rate of return 0.1955097613307959

```python
#分期付款  等额还款
print("Payment", np.pmt(0.01/12, 12 * 30, 10000000))
```

    Payment -32163.95204465004

```python
r=0.01/12
q=1/(1+r)
# pmt*q*(q**360-1)/(q-1)=10000000
10000000*(q-1)/((q**360-1)*q)
```

    32163.952044646507

```python
#付款期数
print("Number of payments", np.nper(0.10/12, -100, 9000))
```

    Number of payments 167.04751180075235

```python
#利率

print("Interest rate", 12 * np.rate(167, -100, 9000, 0))
```

    Interest rate 0.09997564206640422

## 内存优化方面的技巧

```python
x = np.ones((3, 4))
y = np.ones((3, 4))
before = id(y)
before
```

    4496921504

```python
y = y + x
id(y) == before
```

    False

可以看到在y赋值操作后是新开了内存的。如果不想让新开内存

```python
y = np.ones((3, 4))
before = id(y)
y[:] = x + y
id(y) == before
```

    True

## 用numpy生成向量

### 注意如下方式生成的只是数组，而不是向量！！！

```python
a = np.random.randn(5)
a
```

    array([-1.07168447, -1.37614176, -0.54540806,  0.54994061,  0.3317809 ])

```python
a.T
```

    array([-1.07168447, -1.37614176, -0.54540806,  0.54994061,  0.3317809 ])

```python
a.shape
```

    (5,)

```python
a * a.T
```

    array([1.14850759, 1.89376615, 0.29746995, 0.30243468, 0.11007857])

### 采用以下方式生成向量，

```python
a = np.random.randn(5,1)
a
```

    array([[ 1.01622595],
           [ 0.33205095],
           [ 0.47420917],
           [-1.65770468],
           [-1.90864558]])

```python
a.T
```

    array([[ 1.01622595,  0.33205095,  0.47420917, -1.65770468, -1.90864558]])

```python
a.shape
```

    (5, 1)

```python
a * a.T
```

    array([[ 1.03271519,  0.33743879,  0.48190367, -1.68460252, -1.93961518],
           [ 0.33743879,  0.11025783,  0.15746161, -0.55044241, -0.63376758],
           [ 0.48190367,  0.15746161,  0.22487434, -0.78609876, -0.90509724],
           [-1.68460252, -0.55044241, -0.78609876,  2.74798482,  3.16397072],
           [-1.93961518, -0.63376758, -0.90509724,  3.16397072,  3.64292795]])

## 一些属性

### 无穷大

```python
np.inf
```

    inf

```python
np.inf > 10000000
```

    True

```python
arr = np.arange(15).reshape(5, 3)
```

### 获取数组的shape

```python
arr.shape
```

    (5, 3)

```python
np.shape(arr)
```

    (5, 3)

### 生成matrix对象(一般不建议使用吧。。。

```python
arr = np.arange(9).reshape((3, 3))
arr
```

    array([[0, 1, 2],
           [3, 4, 5],
           [6, 7, 8]])

```python
np.asmatrix(arr)
```

    matrix([[0, 1, 2],
            [3, 4, 5],
            [6, 7, 8]])

```python
np.matrix(arr)
```

    matrix([[0, 1, 2],
            [3, 4, 5],
            [6, 7, 8]])

## 维度转换

### reshape

```python
arr = np.arange(8)
arr
```

    array([0, 1, 2, 3, 4, 5, 6, 7])

```python
arr.reshape((2, 4))
```

    array([[0, 1, 2, 3],
           [4, 5, 6, 7]])

```python
'''
Return a copy of the array collapsed(坍塌) into one dimension.
@order: {'C', 'F', 'A', 'K'}, optional
'C' means to flatten in row-major (C-style) order.
'F' means to flatten in column-major (Fortran-
style) order. 'A' means to flatten in column-major
order if `a` is Fortran *contiguous* in memory,
row-major order otherwise. 'K' means to flatten
`a` in the order the elements occur in memory.
The default is 'C'.
'''
arr.reshape((2, 4), order='F')
```

    array([[0, 2, 4, 6],
           [1, 3, 5, 7]])

### 打平成1维

```python
arr.ravel()
```

    array([0, 0, 1, 3, 4, 0, 4, 3])

```python
arr.reshape((2, 4)).ravel()
```

    array([0, 0, 1, 3, 4, 0, 4, 3])

```python
np.ravel(arr.reshape((2, 4)))
```

    array([0, 0, 1, 3, 4, 0, 4, 3])

```python
np.arange(8).reshape((2, 4)).flat
```

    <numpy.flatiter at 0x7fc4c48a2a00>

```python
for a in np.arange(8).reshape((2, 4)).flat:
    print(a)
```

    0
    1
    2
    3
    4
    5
    6
    7

```python
np.arange(8).reshape((2, 4)).flatten()
```

    array([0, 1, 2, 3, 4, 5, 6, 7])

### 增加一个维度

```python
arr
```

    array([0, 1, 2, 3, 4, 5, 6, 7])

```python
arr.reshape((-1, 1))
```

    array([[0],
           [1],
           [2],
           [3],
           [4],
           [5],
           [6],
           [7]])

```python
np.expand_dims(arr, axis=0)
```

    array([[0, 1, 2, 3, 4, 5, 6, 7]])

```python
np.expand_dims(arr, axis=1)
```

    array([[0],
           [1],
           [2],
           [3],
           [4],
           [5],
           [6],
           [7]])

```python
arr[:, np.newaxis]
```

    array([[0],
           [1],
           [2],
           [3],
           [4],
           [5],
           [6],
           [7]])

### squeeze 函数：从数组的形状中删除单维度条目，即把shape中为1的维度去掉

```python
a = np.arange(10).reshape((1,1,2,1,5, 1))
a
```

    array([[[[[[0],
               [1],
               [2],
               [3],
               [4]]],

             [[[5],
               [6],
               [7],
               [8],
               [9]]]]]])

```python
a.shape
```

    (1, 1, 2, 1, 5, 1)

```python
a.squeeze()
```

    array([[0, 1, 2, 3, 4],
           [5, 6, 7, 8, 9]])

## ndarray切片

```python

```

## ndarray合并

```python
arr1 = np.arange(8).reshape((2, 4))
arr2 = np.random.randn(2, 4)
```

### np.concatenate

```python
np.concatenate((arr1, arr2))
```

    array([[ 0.        ,  1.        ,  2.        ,  3.        ],
           [ 4.        ,  5.        ,  6.        ,  7.        ],
           [-0.20558031, -1.87406768,  0.66159744, -2.27991831],
           [ 1.80279172, -0.57716586, -0.58042856, -0.65255574]])

```python
np.concatenate((arr1, arr2), axis=1)
```

    array([[ 0.        ,  1.        ,  2.        ,  3.        , -0.20558031,
            -1.87406768,  0.66159744, -2.27991831],
           [ 4.        ,  5.        ,  6.        ,  7.        ,  1.80279172,
            -0.57716586, -0.58042856, -0.65255574]])

### np.vstack | np.hstack

```python
np.vstack((arr1, arr2))
```

    array([[ 0.        ,  1.        ,  2.        ,  3.        ],
           [ 4.        ,  5.        ,  6.        ,  7.        ],
           [-0.20558031, -1.87406768,  0.66159744, -2.27991831],
           [ 1.80279172, -0.57716586, -0.58042856, -0.65255574]])

```python
np.hstack((arr1, arr2))
```

    array([[ 0.        ,  1.        ,  2.        ,  3.        , -0.20558031,
            -1.87406768,  0.66159744, -2.27991831],
           [ 4.        ,  5.        ,  6.        ,  7.        ,  1.80279172,
            -0.57716586, -0.58042856, -0.65255574]])

### np.c_ | np.r_
np.r_是按列连接两个矩阵，就是把两矩阵上下相加，要求列数相等。

np.c_是按行连接两个矩阵，就是把两矩阵左右相加，要求行数相等。

```python
np.r_[arr1, arr2]
```

    array([[ 0.        ,  1.        ,  2.        ,  3.        ],
           [ 4.        ,  5.        ,  6.        ,  7.        ],
           [-0.35656944, -0.76316801,  2.60948819,  0.79686435],
           [ 0.64158651, -0.26508957, -0.60328599, -0.22472195]])

```python
np.c_[arr1, arr2]
```

    array([[ 0.        ,  1.        ,  2.        ,  3.        , -0.35656944,
            -0.76316801,  2.60948819,  0.79686435],
           [ 4.        ,  5.        ,  6.        ,  7.        ,  0.64158651,
            -0.26508957, -0.60328599, -0.22472195]])

```python
np.c_[np.ones(arr1.shape[0]), arr1]
```

    array([[1., 0., 1., 2., 3.],
           [1., 4., 5., 6., 7.]])

## 赋值

```python
arr = np.arange(8).reshape((2, 4))
arr
```

    array([[0, 1, 2, 3],
           [4, 5, 6, 7]])

```python
arr[:, 2] = [8, 9]
```

```python
arr
```

    array([[0, 1, 8, 3],
           [4, 5, 9, 7]])

## 比较

```python
import numpy as np
```

```python
a = np.arange(8).reshape((2, 4))
b = np.arange(8).reshape((2, 4))
```

```python
a == b
```

    array([[ True,  True,  True,  True],
           [ True,  True,  True,  True]])

```python
np.array_equal(a, b)
```

    True

```python
np.allclose(a, b)
```

    True

## 描述性统计

```python
max(), min(), sum(), sqrt(), square()
```

```python
arr = np.arange(8).reshape((2, 4))
arr
```

    array([[0, 1, 2, 3],
           [4, 5, 6, 7]])

### mean

```python
np.mean(arr)
```

    3.5

```python
np.mean(arr, axis=0)
```

    array([2., 3., 4., 5.])

```python
arr.mean(axis=0)
```

    array([2., 3., 4., 5.])

### 中位数median

```python
np.median(arr)
```

    5.0

```python
arr1
```

    array([[0, 1, 2],
           [3, 4, 5]])

### prod乘积。
* axis=None: 返回所有元素乘积
* axis=0: 返回每列的乘积

```python
np.prod(arr1)
```

    0

```python
np.prod(arr1, axis=0)
```

    array([ 0,  4, 10])

### 众数mode

```python
arr = np.array([1, 9, 6, 4, 8, 5, 7, 3, 2, 8, 3, 8, 3, 9, 7, 9, 5, 1, 5, 3])
```

```python
np.mode(arr)
```

#### 方法一

##### 计算非负整数的出现次数
Count number of occurrences of each value in array of non-negative ints.

```python
np.bincount(arr)
```

    array([0, 2, 1, 4, 1, 3, 1, 2, 3, 3])

```python
#np中没有直接求众数的方法
#bincount统计了每个索引位置出现的次数，注意arr的元素只能是正整数
#如结果第一个元素0表示0在arr中出现了0次
# #间接求众数
np.argmax(np.bincount(arr))
```

    3

#### 方法二：scipy.stats

```python
from scipy import stats
```

```python
r = stats.mode(arr)  # 表示3出现了4次
r
```

    ModeResult(mode=array([3]), count=array([4]))

```python
r.mode
```

    array([3])

```python
r.count
```

    array([4])

#### 方法3 转换为pd.DataFrame或者Series

```python
import pandas as pd
```

```python
s = pd.Series(arr)
```

```python
s.mode()
```

    0    3
    dtype: int64

### cumsum累积和

```python
arr
```

    array([1, 9, 6, 4, 8, 5, 7, 3, 2, 8, 3, 8, 3, 9, 7, 9, 5, 1, 5, 3])

```python
np.cumsum(arr)
```

    array([  1,  10,  16,  20,  28,  33,  40,  43,  45,  53,  56,  64,  67,
            76,  83,  92,  97,  98, 103, 106])

### ndarray中非0元素的个数

```python
np.count_nonzero(arr)
```

    5

```python
np.count_nonzero(arr.reshape((2, 4)))
```

    5

## elementwise运算

```python
arr1 = np.arange(6).reshape((2, 3))
arr1
```

    array([[0, 1, 2],
           [3, 4, 5]])

```python
np.log(arr1+1)
```

    array([[0.        , 0.69314718, 1.09861229],
           [1.38629436, 1.60943791, 1.79175947]])

### np.clip

```python
np.clip(np.random.rand(5), 0.4, 0.7)
```

    array([0.4611924 , 0.7       , 0.4       , 0.63426303, 0.7       ])

```python
np.max(arr1)
```

    5

```python
np.maximum(arr1, 3)
```

    array([[3, 3, 3],
           [3, 4, 5]])

```python
arr1 - 1
```

    array([[-1,  0,  1],
           [ 2,  3,  4]])

```python
arr1 - np.array([[1],
                [2]])
```

    array([[-1,  0,  1],
           [ 1,  2,  3]])

```python
arr1 - np.array([1, 2, 3])
```

    array([[-1, -1, -1],
           [ 2,  2,  2]])

```python
arr1 - np.array([[1, 2, 3]])
```

    array([[-1, -1, -1],
           [ 2,  2,  2]])

```python
arr1 - np.array([[1, 2]])
```

### np.power 幂

```python
np.power(2, 3)  # 2**3
```

    8

```python
np.power([2, 3], 3)  # 2**3, 3**3
```

    array([ 8, 27])

```python
np.power([2, 3], [2, 3])  # 2**2, 3**3
```

    array([ 4, 27])

### 对应位置的元素相乘/除

```python
arr1 * arr1
```

    array([[ 0,  1,  4],
           [ 9, 16, 25]])

```python
arr1 / (arr1 + 1)
```

    array([[0.        , 0.5       , 0.66666667],
           [0.75      , 0.8       , 0.83333333]])

## 沿着某一维度运算

```python
import pandas as pd
```

```python
df2 = pd.DataFrame({'A': 1.,
                    'B': pd.Timestamp('20130102'),
                    'C': pd.Series(1, index=list(range(4)), dtype='float32'),
                    'D': np.array([3] * 4, dtype='int32'),
                    'E': pd.Categorical(["test", "train", "test", "train"]),
                    'F': 'foo'})
df2
```

      A
      B
      C
      D
      E
      F

      0
      1.0
      2013-01-02
      1.0
      3
      test
      foo

      1
      1.0
      2013-01-02
      1.0
      3
      train
      foo

      2
      1.0
      2013-01-02
      1.0
      3
      test
      foo

      3
      1.0
      2013-01-02
      1.0
      3
      train
      foo

```python
np.apply_along_axis(lambda v: dict(zip(df2.columns, v)), axis=1, arr=df2.values)
```

    array([{'A': 1.0, 'B': Timestamp('2013-01-02 00:00:00'), 'C': 1.0, 'D': 3, 'E': 'test', 'F': 'foo'},
           {'A': 1.0, 'B': Timestamp('2013-01-02 00:00:00'), 'C': 1.0, 'D': 3, 'E': 'train', 'F': 'foo'},
           {'A': 1.0, 'B': Timestamp('2013-01-02 00:00:00'), 'C': 1.0, 'D': 3, 'E': 'test', 'F': 'foo'},
           {'A': 1.0, 'B': Timestamp('2013-01-02 00:00:00'), 'C': 1.0, 'D': 3, 'E': 'train', 'F': 'foo'}],
          dtype=object)

## 多项式运算

### 多项式表达方式

```python
p = np.array([1,0,-3,5])
x = 5
print(np.polyval(p,x))
```

    115

```python
x**3 - 3*x**1 + 5
```

    115

### 多项式求解

```python
p = np.array([1,0,-3,5])
b = np.roots(p)
b
```

    array([-2.27901879+0.j        ,  1.13950939+0.94627954j,
            1.13950939-0.94627954j])

```python
np.real(b)
```

    array([-2.27901879,  1.13950939,  1.13950939])

### 多项式乘法(卷积)

```python
a =[1,2,3,4]; b = [1,4,9,16]
print(np.convolve(a,b))
```

    [ 1  6 20 50 75 84 64]

### 多项式的曲线拟合

#### 函数拟合

```python
x = [1,2,3,4,5]
y = [5.6,40,150,250,498]
p = np.polyfit(x,y,3)
print (p)
```

    [  6.03333333 -24.5         83.06666667 -62.48      ]

```python
import matplotlib.pyplot as plt
```

```python
#拟合结果
x2 = np.arange(0,5.1,0.1)
y2 = np.polyval(p,x2)
plt.plot(x,y,'*',x2,y2)
plt.show()
```

#### 多项式插值

```python
import math as math
x = list(np.arange(0,2*math.pi,0.2))

def g(x):
    return 4.8*math.cos(math.pi*x/20.0)

#y = math.cos(math.pi*x/20.0)
y = []
for t in x:
    g1 = g(t)
    y.append(g1)

plt.plot(x,y,'*')
plt.show()
```

```python
#第二种方法创建x和y
g = lambda x: 4.8*math.cos(math.pi*x/20.0)
x = list(np.arange(0,2*math.pi,0.2))
y = [g(x) for x in list(np.arange(0,2*math.pi,0.2))]
plt.plot(x,y,'*')
plt.show()
```

```python
#多项式插值
import scipy.interpolate as itp
x1 = np.arange(0,2*math.pi,0.1)
g2 = itp.interp1d(x,y,kind='cubic')
y1 =g2(x1)
#作图
plt.plot(x,y,'*',x1,y1)
plt.show()
```

## 数值积分计算
在一个实数区 间上的定积分可以理解为在坐标平面上，由曲线、直线以及轴围成的曲边梯形的面积值

```python
import scipy.integrate
import math
F = lambda x: x**3 - 2 *x -5
Q =scipy.integrate.quad(F,0,2)

print(Q[0])
```

    -10.0

```python
t = np.arange(0,2.1,0.01)
g = F(t)

plt.plot(t,g,'*')
plt.show()
```

```python
# 计算积分求解
def h(x):
    return 0.25*x**4 - x**2 -5*x

h(2) - h(0)
```

    -10.0

```python
# 4.3.2符号积分计算
#SymPy 的核心功能是将数学符号表示为 Python 对象。
#在 SymPy 库中，类 sympy.Symbol 可用于此目的
#==============================================================================
from sympy import *
x =  Symbol("x")
y =  Symbol('y',real=True)
s = integrate(x * y, (x,int("0"), int("1")), (y,int("1"),int("2")))
print(s)
```

    3/4

```python
#或者
s =  integrate(x * y, (x,int("0"), int("1")))
ss = integrate(s, (y,int("1"), int("2")))
print(s)
print(ss)
```

    y/2
    3/4

## 导数计算

```python
#求导
x = Symbol('x',real=True)
y = Symbol('y',real=True)
z = Symbol("z")
expr = x**4 + x**3 + x**2 + x + 1
expr.diff(x)
```

    4*x**3 + 3*x**2 + 2*x + 1

```python
#多元函数
from sympy import *
expr = (x + 1) ** 3 * y ** 2 *(z - 1)
expr.diff(x,y)
```

    6*y*(x + 1)**2*(z - 1)

## 线性代数&矩阵运算

### 线性方程组求解

```python
A = np.array([[4,-2,1],[-2,4,-2],[1,-2,4]])
B = np.array([11,-16,17])
print(np.linalg.solve(A,B))
```

    [ 1. -2.  3.]

### 矩阵特征值和特征向量

```python
A = np.array([[4,-2,1],[-2,4,-2],[1,-2,4]])
d,v = np.linalg.eig(A)
print(d,v)
```

    [7.37228132 3.         1.62771868] [[-5.41774320e-01  7.07106781e-01  4.54401349e-01]
     [ 6.42620551e-01  2.73458602e-16  7.66184591e-01]
     [-5.41774320e-01 -7.07106781e-01  4.54401349e-01]]

```python
#验证
print(np.dot( np.matrix(A), np.matrix(v[:,1].reshape(-1,1))))
print(np.dot(np.matrix(v[:,1].reshape(-1,1)),d[1]))
```

    [[ 2.12132034e+00]
     [ 1.33226763e-15]
     [-2.12132034e+00]]
    [[ 2.12132034e+00]
     [ 8.20375805e-16]
     [-2.12132034e+00]]

### 求逆

```python
arr = np.random.RandomState(14).randn(3, 3)
arr
```

    array([[ 1.55133911,  0.07918602,  0.17397653],
           [-0.07233657, -2.0043294 ,  0.14467781],
           [-1.50116862,  0.21110945, -0.55820506]])

```python
np.linalg.inv(arr)
```

    array([[ 0.95314825,  0.07088078,  0.31544017],
           [-0.22558183, -0.52969762, -0.20759654],
           [-2.64859426, -0.39094625, -2.71827435]])

### 行列式

```python
np.linalg.det(arr)
```

    1.1417782564454833

### norm 范数

```python
arr
```

    array([[ 1.55133911,  0.07918602,  0.17397653],
           [-0.07233657, -2.0043294 ,  0.14467781],
           [-1.50116862,  0.21110945, -0.55820506]])

#### 向量的范数

```python
np.linalg.norm(arr.ravel())
```

    3.0160171097951203

##### 可以看出, 默认参数下 是向量的2范数就是每个元素的平方和再开平方根 = 向量模长！

```python
np.sqrt(arr.ravel().dot(arr.ravel()))
```

    3.0160171097951203

#### 矩阵的范数

```python
np.linalg.norm(arr)
```

    3.0160171097951203

```python
np.abs(arr)
```

    array([[1.55133911, 0.07918602, 0.17397653],
           [0.07233657, 2.0043294 , 0.14467781],
           [1.50116862, 0.21110945, 0.55820506]])

```python
np.sum(np.abs(arr), axis=0)
```

    array([3.1248443 , 2.29462488, 0.8768594 ])

### SVD分解

```python
a = np.arange(48).reshape((12, 4))
#svd分解
U, s, Vh = np.linalg.svd(a, full_matrices=True)
```

```python
U.shape, s, Vh.shape
```

    ((12, 12),
     array([1.88956663e+02, 3.92167338e+00, 1.34838487e-14, 9.73158362e-16]),
     (4, 4))

## ndarray数据保存和加载

### 默认情况下，数组以未压缩的原始二进制格式保存在扩展名为npy的文件中

```python
np.save('name.npy', arr)
```

### 加载save保存的npy文件

```python
arr = np.load('name.npy')
```

### 将1D/2D数组写入以某种分隔符隔开的文本文件中, 注意保存三维以上的数据会报错

```python
np.savetxt('name.txt', arr)
```

### 加载savetxt保存的txt文件

```python
arr = np.loadtxt('name.txt')
```

### Save several arrays into a single file in uncompressed .npz format.

```python
np.savez('tmp/a.npz', arr1=arr1, arr2=arr2)
```

#### 或者压缩格式

```python
np.savez_compressed('tmp/a_compressed.npz', arr1=arr1, arr2=arr2)
```

### 加载npz文件

```python
a = np.load('tmp/a.npz')
print(a['arr1'], a['arr2'])
```


    Run in Google Colab

    View source on GitHub

# Implicit Feedback

There are two types of feedback on which most recommender systems would be based. Explicit feedback deals with clear and quantified expression of preferences by a user, in terms of both likes and dislikes. An excellent example would be ratings. However, much more common is implicit feedback, where users provide some positive signals, but the contrast to negative signals may be absent or at least much less evident. An example would be clicks on links or views when browsing.  In this tutorial, we will focus on implicit feedback.

## 1. Setup

```
!pip install --quiet cornac==1.6.0 adjustText
```

```
import os
import sys
from collections import defaultdict

import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from adjustText import adjust_text
%matplotlib inline

import cornac
from cornac.utils import cache
from cornac.datasets import movielens
from cornac.eval_methods import RatioSplit
from cornac.models import MF, BPR, WMF

%tensorflow_version 1.x
import tensorflow as tf

print(f"System version: {sys.version}")
print(f"Cornac version: {cornac.__version__}")
print(f"Tensorflow version: {tf.__version__}")

SEED = 42
VERBOSE = False
```

    System version: 3.6.9 (default, Apr 18 2020, 01:56:04)
    [GCC 8.4.0]
    Cornac version: 1.6.0
    Tensorflow version: 1.15.2

## 2. Weighted Matrix Factorization (WMF)

We use the term **adoption matrix** to refer to interactions between users and items.  In the case of explicit feedback, it is the rating matrix, while in the case of implicit feedback, it can be a matrix of click counts or binary.

Weighted matrix factorization predicts an adoption of user $i$ to item $j$ using inner product of user and item factors as $\hat{p}_{i,j} = \mathbf{u}_i^T \mathbf{v}_j$.  The difference between Weighted MF and Original MF is that WMF tries to predict *unobserved* adoptions while MF ignores them during training. The convention by Cornac library is to derive observed and unobserved adoptions from the rating matrix as follows.

\begin{equation}
p_{i,j} =
\begin{cases}
r_{i, j} &\mbox{if } r_{i,j} > 0 \\
0 & \mbox{otherwise}
\end{cases}
\end{equation}

The important principle is that WMF usually assigns a higher confidence weight when fitting *observed* adoptions than when fitting *unobserved* adoptions. There are several ways to introduce the confidence scores.  Here we use the convention by Cornac library, which implements the following rules:

\begin{equation}
c_{i,j} =
\begin{cases}
a &\mbox{if } r_{i,j} > 0 \\
b & \mbox{otherwise }
\end{cases}
\end{equation}

Concretely, WMF minimizes the following loss function:

$$ \mathcal{L}(\mathbf{U,V}|\lambda) = \frac{1}{2} \sum_{i,j} c_{i,j} (p_{i,j} - \mathbf{u}_i^T \mathbf{v}_j)^2 + \frac{\lambda}{2} \sum_{i=1}^{N} ||\mathbf{u}_i||^2 + \frac{\lambda}{2} \sum_{j=1}^{M} ||\mathbf{v}_j||^2 $$

Let's now compare how WMF and MF are different.

```
K = 50
wmf = WMF(k=K, max_iter=100, a=1.0, b=0.01, learning_rate=0.001, lambda_u=0.01, lambda_v=0.01,
          verbose=VERBOSE, seed=SEED, name=f"WMF(K={K})")
mf = MF(k=K, max_iter=20, learning_rate=0.01, lambda_reg=0.01,
        verbose=VERBOSE, seed=SEED, name=f"MF(K={K})")

eval_metrics = [
  cornac.metrics.RMSE(),
  cornac.metrics.AUC(),
  cornac.metrics.Precision(k=10),
  cornac.metrics.Recall(k=10),
  cornac.metrics.FMeasure(k=10),
  cornac.metrics.NDCG(k=[10, 20, 30]),
  cornac.metrics.MRR(),
  cornac.metrics.MAP()
]

data = movielens.load_feedback(variant="100K")
rs = RatioSplit(data, test_size=0.2, seed=SEED, verbose=VERBOSE)

cornac.Experiment(eval_method=rs, models=[wmf, mf], metrics=eval_metrics).run()
```

    TEST:
    ...
              |   RMSE |    AUC |  F1@10 |    MAP |    MRR | NDCG@10 | NDCG@20 | NDCG@30 | Precision@10 | Recall@10 | Train (s) | Test (s)
    --------- + ------ + ------ + ------ + ------ + ------ + ------- + ------- + ------- + ------------ + --------- + --------- + --------
    WMF(K=50) | 1.1012 | 0.8980 | 0.0734 | 0.0956 | 0.2387 |  0.1068 |  0.1293 |  0.1506 |       0.0968 |    0.0800 |    4.8214 |   2.0496
    MF(K=50)  | 0.8909 | 0.6573 | 0.0290 | 0.0402 | 0.1453 |  0.0539 |  0.0565 |  0.0626 |       0.0524 |    0.0241 |    0.2575 |   1.7880

As we can observe, the strength of the MF model is the ability to predict ratings well (lower RMSE). However, WMF model is designed to rank items, by fitting binary adoptions, thus it outperforms MF across all the listed ranking metrics.

## 3. Bayesian Personalized Ranking (BPR)

Implicit feedback can also be turned into another form of expression, namely ordinal preferences.  Given an *observed* feedback $(i, j)$ between user $i$ and item $j$, another item $l$ (with *no observed interaction* with user $i$) is coupled with the pair to form a triple $(i, j, l)$.  Thus, there are now notions of *positive* and *negative* items.  The training data consists of ordinal triples $D_S = \{(i, j, l) | r_{ij} \in \mathbf{R}^+ \land r_{il} \in \mathbf{R}^-\}$.

BPR maximizes the joint probability over all triples:

$$ \prod_{(i,j,l) \in D_S} p(j >_{i} l) = \prod_{(i,j,l) \in D_S} \sigma(\mathbf{u}_i^T \mathbf{v}_j - \mathbf{u}_i^T \mathbf{v}_l) $$

where $\sigma(*)$ is the logistic *sigmoid* function and the preference scoring function is the inner product of user and item latent factors, inspired by the matrix factorization method.

By minimizing *negative log-likelihood* with *l2-regularization*, we arrive at the following loss function:

$$ \mathcal{L}(\mathbf{U,V}|\lambda) = \sum_{i,j,l} \ln (1 + \exp\{-(\mathbf{u}_i^T \mathbf{v}_j - \mathbf{u}_i^T \mathbf{v}_l)\}) + \frac{\lambda}{2} \sum_{i=1}^{N} ||\mathbf{u}_i||^2 + \frac{\lambda}{2} \sum_{j=1}^{M} ||\mathbf{v}_j||^2 $$

Let's compare the two models BPR and WMF.

```
K = 50
bpr = BPR(k=K, max_iter=200, learning_rate=0.01, lambda_reg=0.001,
          verbose=VERBOSE, seed=SEED, name=f"BPR(K={K})")
wmf = WMF(k=K, max_iter=100, a=1.0, b=0.01, learning_rate=0.001, lambda_u=0.01, lambda_v=0.01,
          verbose=VERBOSE, seed=SEED, name=f"WMF(K={K})")

rs = RatioSplit(data, test_size=0.2, seed=SEED, verbose=VERBOSE)
cornac.Experiment(eval_method=rs, models=[bpr, wmf], metrics=eval_metrics).run()
```

    TEST:
    ...
              |   RMSE |    AUC |  F1@10 |    MAP |    MRR | NDCG@10 | NDCG@20 | NDCG@30 | Precision@10 | Recall@10 | Train (s) | Test (s)
    --------- + ------ + ------ + ------ + ------ + ------ + ------- + ------- + ------- + ------------ + --------- + --------- + --------
    BPR(K=50) | 1.5631 | 0.9356 | 0.0859 | 0.1146 | 0.2337 |  0.1096 |  0.1393 |  0.1662 |       0.0938 |    0.1000 |    5.8177 |   1.8928
    WMF(K=50) | 1.1012 | 0.8980 | 0.0734 | 0.0956 | 0.2387 |  0.1068 |  0.1293 |  0.1506 |       0.0968 |    0.0800 |    3.9621 |   2.0072

BPR only tries to preserve the ordinal constraints without learning to predict the rating values.  Thus, RMSE is not the right metric to evaluate BPR model.   Minimizing the loss function of BPR is analogous to maximizing AUC, therefore, we expect BPR to do well on that metric.

Both BPR and WMF models are designed to obtain good performances in terms of ranking metrics.  With reasonable efforts for hyper-parameter tuning, we should see comparable performance between the two models.

## 4. Interpreting Latent Factors

Both presented models (WMF and BPR) are based on the matrix factorization technique, let us see how the latent factors could give us some information about how items are associated with one another, as well as which items tend to be associated with a particular factor.

```
# Download some information of MovieLens 100K dataset
item_df = pd.read_csv(
  cache("http://files.grouplens.org/datasets/movielens/ml-100k/u.item"),
  sep="|", encoding="ISO-8859-1",
  names=["ItemID", "Title", "Release Date", "Video Release Date", "IMDb URL",
         "unknown", "Action", "Adventure", "Animation", "Children's", "Comedy",
         "Crime", "Documentary", "Drama", "Fantasy", "Film-Noir", "Horror",
         "Musical", "Mystery", "Romance", "Sci-Fi", "Thriller", "War", "Western"]
).set_index("ItemID").drop(columns=["Video Release Date", "IMDb URL", "unknown"])
```

### WMF Latent Factors

Below is the variance of each latent dimension across different items.  If a factor or dimension has higher variance, it is better at discriminating among different items.  Conversely, if the variance of a factor is too small, it implies that most items would have a similar value for that factor.

```
var_df = pd.DataFrame({"Factor": np.arange(K), "Variance": np.var(wmf.V, axis=0)})
fig, ax = plt.subplots(figsize=(13, 5))
sns.barplot(x="Factor", y="Variance", data=var_df, palette="ch:.25", ax=ax);
```

```
TOP2F = (0, 18)
SAMPLE_SIZE = 20

rng = np.random.RandomState(SEED)
sample_inds = rng.choice(np.arange(wmf.V.shape[0]), size=SAMPLE_SIZE, replace=False)
sample_df = pd.DataFrame(data=wmf.V[sample_inds][:, TOP2F], columns=["x", "y"])

sns.lmplot(x="x", y="y", data=sample_df, height=11.0, fit_reg=False)
item_idx2id = list(wmf.train_set.item_ids)
titles = item_df.loc[[int(item_idx2id[i]) for i in sample_inds]]["Title"].values
adjust_text(
  [plt.text(*sample_df.loc[i].values, titles[i], size=10) for i in range(len(titles))]
);
```

### Interpreting Recommendations of WMF

Let's pick a user and see what the top-K recommendations by WMF model are.

```
UIDX = 0
TOPK = 5

item_id2idx = wmf.train_set.iid_map
item_idx2id = list(wmf.train_set.item_ids)

recommendations, scores = wmf.rank(UIDX)
print(f"\nTOP {TOPK} RECOMMENDATIONS:")
print("Scores:", scores[recommendations[:TOPK]])
item_df.loc[[int(item_idx2id[i]) for i in recommendations[:TOPK]]]
```

    TOP 5 RECOMMENDATIONS:
    Scores: [4.5135827 4.466052  4.3171253 4.25356   4.202481 ]

      Title
      Release Date
      Action
      Adventure
      Animation
      Children's
      Comedy
      Crime
      Documentary
      Drama
      Fantasy
      Film-Noir
      Horror
      Musical
      Mystery
      Romance
      Sci-Fi
      Thriller
      War
      Western

      ItemID

      318
      Schindler's List (1993)
      01-Jan-1993
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      1
      0

      50
      Star Wars (1977)
      01-Jan-1977
      1
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      1
      0
      1
      0

      496
      It's a Wonderful Life (1946)
      01-Jan-1946
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      64
      Shawshank Redemption, The (1994)
      01-Jan-1994
      0
      0
      0
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      169
      Wrong Trousers, The (1993)
      01-Jan-1993
      0
      0
      1
      0
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

The solution of alternating least squares model can be used to explain the recommendations.  Please refer to Section 5 in the [paper](http://yifanhu.net/PUB/cf.pdf) for more details.  In summary, the predicted preference of user $i$ for item $j$ is rewritten as:

$$ \hat{p}_{i,j} = \sum_{l=1}^M sim_i (j, l) c_{i, l} p_{i,l} $$

where $sim_i (j, l)$ is the similarity between item $j$ and item $l$ from user $i$’s viewpoint.

Let's try to explain why the WMF model recommends the movie **Star Wars (1977)**, *ItemID* = 50 to the user.

```
REC_ID = 50
print("Recommended movie:", item_df.loc[REC_ID].Title)
rec_idx = item_id2idx[str(REC_ID)]

U, V, lbd = wmf.U, wmf.V, wmf.lambda_v
pi = wmf.train_set.matrix[UIDX].A.ravel()
ci = (pi > 0) * (wmf.a - wmf.b) + wmf.b

Wu = np.linalg.inv(V.T.dot(np.diag(ci)).dot(V) + lbd * np.eye(wmf.k))
sim = V[rec_idx].dot(Wu).dot(V.T)
contribution = sim * ci * pi

print("\nTop contributors:")
df = defaultdict(list)
for idx in np.argsort(contribution)[-TOPK:][::-1]:
  df["ItemID"].append(item_df.loc[[int(item_idx2id[idx])]].index.values[0])
  df["Title"].append(item_df.loc[[int(item_idx2id[idx])]]["Title"].values[0])
  df["Similarity"].append(sim[idx])
  df["Confidence"].append(ci[idx])
  df["Adoption"].append(pi[idx])
  df["Contribution"].append(contribution[idx])
pd.DataFrame.from_dict(df)
```

    Recommended movie: Star Wars (1977)

    Top contributors:

      ItemID
      Title
      Similarity
      Confidence
      Adoption
      Contribution

      0
      258
      Contact (1997)
      0.241282
      1.0
      4.0
      0.965128

      1
      237
      Jerry Maguire (1996)
      0.231635
      1.0
      4.0
      0.926541

      2
      971
      Mediterraneo (1991)
      0.205750
      1.0
      4.0
      0.823000

      3
      56
      Pulp Fiction (1994)
      0.140434
      1.0
      5.0
      0.702171

      4
      690
      Seven Years in Tibet (1997)
      0.115898
      1.0
      4.0
      0.463592

### BPR Latent Factors

Below is the variance of each latent dimension.

```
var_df = pd.DataFrame({"Factor": np.arange(K), "Variance": np.var(bpr.i_factors, axis=0)})
fig, ax = plt.subplots(figsize=(13, 5))
sns.barplot(x="Factor", y="Variance", data=var_df, palette="ch:.25", ax=ax);
```

```
TOP2F = (24, 49)
SAMPLE_SIZE = 20

rng = np.random.RandomState(SEED)
sample_inds = rng.choice(np.arange(bpr.i_factors.shape[0]), size=SAMPLE_SIZE, replace=False)
sample_df = pd.DataFrame(data=bpr.i_factors[sample_inds][:, TOP2F], columns=["x", "y"])

sns.lmplot(x="x", y="y", data=sample_df, height=11.0, fit_reg=False)
item_idx2id = list(bpr.train_set.item_ids)
titles = item_df.loc[[int(item_idx2id[i]) for i in sample_inds]]["Title"].values
adjust_text(
  [plt.text(*sample_df.loc[i].values, titles[i], size=10) for i in range(len(titles))]
);
```

## References

1.   Aggarwal, C. C. (2016). Recommender systems (Vol. 1). Cham: Springer International Publishing.
2.   Hu, Y., Koren, Y., & Volinsky, C. (2008, December). Collaborative filtering for implicit feedback datasets. In 2008 Eighth IEEE International Conference on Data Mining (pp. 263-272). Ieee.
3.   Rendle, S., Freudenthaler, C., Gantner, Z., & Schmidt-Thieme, L. (2012). BPR: Bayesian personalized ranking from implicit feedback. arXiv preprint arXiv:1205.2618.
4.   Cornac - A Comparative Framework for Multimodal Recommender Systems (https://cornac.preferred.ai/)
5.   https://github.com/microsoft/recommenders/blob/master/notebooks/02_model/cornac_bpr_deep_dive.ipynb