# Numerical Linear Algebra In Machine Learning
----------------

## Contents

1. Introduction

2. Introduction To Function Approximation

3. Introduction To Regression

4. Linear Solvers For Least Squares Regression
   - Cholesky Factorization For Normal Equations

   - Singular Values Decomposition

   - Controlling For Overfitting With Regularization

   - Implementation in Scikit-learn

5. A Touch Of Recommendation Systems

6. Where To Go From Here

## Introduction
------------------

In this blogpost we'll go over applications of numerical linear algebra in machine learning starting out with regression and ending with modern recommender systems! Numerical linear algebra (and numerical analysis more generally) was one of thoses courses that I learned, thought was boring and never wanted to study again. Only with maturity that comes with age (and a PhD) was I able to understand and appreciate the true power of numerical linear alebra.  Infact *understanding (distribued) linear algebra is probably one of the most important and useful tools I have ever learned.*  It has allowed me to contribute to open source libraries for scientific computing and understand how big data and machine learning systems work.  The reason why numerical linear algebra is so important is because it allows us to approximate functions.  In scientific computing and machine learning one is interested in **how to approximate a function** $f(x)$.  Numerical analysis and statistics concerns itself with **how good is our approximation to** $f(x)$?

Traditionally algorithms in Numerical Linear Algebra are taught/learned in [Matlab](https://www.mathworks.com/products/matlab.html) for easy learning, but written for production Fortran and C/C++ for high performance. The fields of Data Science, Machine Learning and Artifical Intelligence have a seen recent boom and given a new life and applications to these algorithms. These fields lean more to using Python (though R is often used as well) as the primaty programming language, but make heavy use of wrappers to [BLAS](http://www.netlib.org/blas/) and [LAPACK](http://www.netlib.org/lapack/) libraries written in Fortran and C/C++.

Many Data Scientists and Machine Learning Researchers also heavily make use of [Jupyter](https://jupyter.org/) notebook as a tool for rapid prototyping and writing experiments in Python (though it also allows for the use of other languages likes R, Julia and Scala). Jupyter notebook is as an interactive envrionment for computing that runs in your web browser and is very similar to [Mathematica's](https://www.wolfram.com/notebooks/) notebook. The fact it runs in a web browser is quite convenient as it will allows users to easily work on a remote computer without any complicated set up.  See this [post](http://michael-harmon.com/blog/JupyterOnGCP.html) about how to get set up with Python & Jupyter notebooks on Google Cloud with all the dependencies installed for this post.

##  Introduction To Function Approximation
------------------------

Let's now move onto function approximation.  One learns in Calculus to represent smooth functions $f : \, \mathbb{R} -> \mathbb{R}$ about some point $x_0$ using special polynomials called [power series](http://tutorial.math.lamar.edu/Classes/CalcII/PowerSeries.aspx) or [Taylor series](http://mathworld.wolfram.com/TaylorSeries.html):

\begin{equation}
f(x) \, = \sum_{n=0}^{\infty} \, a_n (x - x_0)^{n}
\end{equation}

We can find the coefficients to the series by the equation,

\begin{equation}
a_{n} \; = \; \frac{1}{n!} \, f^{n}(x_0)
\end{equation}

We observed that there is a radius of convergence $R \, \in \mathbb{R}$ such that within $(x_0 - R, x_0 + R)$ (one should test the end points too) the series converges to $f(x)$ [uniformly](http://web.math.ucsb.edu/~padraic/caltech/math1d_2010/ma1d_wk5_notes_2010.pdf) and outside that it does not necessarily converge to $f(x)$.

We then learned that we can approximate our function $f(x)$ with the finite degree polynomial,

\begin{equation}
f(x) \, \simeq \, f_N(x) \, = \, a_0  \, + \, a_1 (x - x_0) \, + \,a_2 (x - x_0)^2 \, + \ldots \, + \, a_N (x - x_0)^N
\end{equation}

We can then use that approximation to perform calculations on our function that might not have been possible otherwise.  The most memorialble for me being learning how to [integrate a Gaussian](http://www.faculty.umassd.edu/michele.mandrioli/424book/fchm4.5.html). Though in practice [numerical integration](https://en.wikipedia.org/wiki/Numerical_integration) is usually perferred for finite integrals.

Power series were originaly developed to approximate solutions to differential equations. As science and engineering progressed the differential equations become more complicated and harder to solve.  Other approximation methods were invented like Fourier Series:

\begin{equation}
f(x) \, \simeq \, f_N(x) \, = \, \frac{a_0}{2} \, + \sum_{k=1}^{N} \left(a_n \cos \left(\frac{2 \pi k x}{L} \right) + b_n \sin \left(\frac{2 \pi k x}{L} \right) \right)
\end{equation}

We remark that the functions $\cos\left(\frac{2 \pi k x}{L}\right)$ and $\sin\left(\frac{2 \pi k x}{L}\right)$ form an **orthogonal basis for the Hilbert Space** [$L^{2}([-L,L])$](http://mathworld.wolfram.com/L2-Space.html).  **Coefficients for the approximations are determined projecting the function onto the basis functions:**

\begin{equation}
a_n \; = \; \frac{2}{L} \, \int_{-L}^{L} \, f(x) \,\cos \left(\frac{2 \pi n x}{L} \right) \, dx
\end{equation}

\begin{equation}
b_n \; = \; \frac{2}{L} \, \int_{-L}^{L} \, f(x) \,\sin \left(\frac{2 \pi n x}{L} \right) \, dx
\end{equation}

The convergence of the Fourier series is much subtle issue. While the Taylor series was guaranteed to converge uniformly, the regularity conditions imposed to achieve this were quite strong.  The class of functions in $L^{2}([-L,L])$ is quite broad and depending on the regularity conditions of our specific function we will get different modes of [convergence](https://en.wikipedia.org/wiki/Convergence_of_Fourier_series).

Let's derive the second equation for $b_n$ by using a "projection" onto the basis function $\sin \left(\frac{2 \pi n x}{L} \right)$,

\begin{align}
\int_{-L}^{L} \sin \left(\frac{2 \pi n x}{L} \right) \, \left[\frac{a_0}{2} \, + \sum_{k=1}^{N} \left(a_n \cos \left(\frac{2 \pi k x}{L} \right) + b_n \sin\left(\frac{2 \pi k  x}{L} \right) \right) \right] \, dx \; &= \; \int_{-L}^{L} \sin \left(\frac{2 \pi n x}{L} \right) \, f(x) \, dx \\
b_n \, \int_{-L}^{L} \sin^2  \left(\frac{2 \pi n x}{L} \right)\, dx  \; &= \; \int_{-L}^{L} \sin \left(\frac{2 \pi n x}{L} \right) \, f(x) \, dx \\
b_n \, \frac{L}{2} \; &= \;  \int_{-L}^{L} \, f(x) \,\sin \left(\frac{2 \pi n x}{L} \right) \, dx
\end{align}

Which finally becomes,

\begin{equation}
b_n \; = \; \left(\frac{L}{2} \right)^{-1} \int_{-L}^{L} \, f(x) \,\sin \left(\frac{2 \pi n x}{L} \right) \, dx
\end{equation}

The above example illustrates the two steps in our approximation methods:

1. **Projecting the function (or its values) onto a finite dimensional space**

2. **Solving for the co-efficients corresponding to each of the basis functions**

The last step is not usually as easy as the above example was and often requires **solving a system of linear equations:**

\begin{equation}
A x \, = \, b
\end{equation}

The need to solve a system of linear equations occurs most often because the basis for our finite dimensional space is not orthogonal.  If the basis were orthogonal, then the matrix would be **diagonal** and we could be invert it by hand, leading to equations like the Fourier coefficient equations.

Each of these steps in our approximations methods introduces their own errors:

1. The error in your model approximation.

2. The error in solving the linear system of equations.

The first error best summarized in the [bias-variance trade off](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff) (see Section 2.2 in this [book](http://faculty.marshall.usc.edu/gareth-james/ISL/) as well). By this I mean that by assuming that our function has only $p$ features we introduce a bias into our model and as we increase the number we will most likely increase the variance in our model. The second error is due to rounding errors and [numerical stability](https://en.wikipedia.org/wiki/Numerical_analysis#Numerical_stability_and_well-posed_problems).  We won't touch on either of these topics too much, but mention them for completeness.

With the advent of the computer and seemingly cheap and unlimited computational resources numerical methods like finite difference and finite element methods were invented to approximate solutions to differential equations. The finite element method is one that is particularly dear to my heart and has concepts that have proved useful in understanding models in statistics and machine learning, particularly, Generalized Addative Models.

In the next section well go into the basics of Linear Regression models which is a subset of the larger class of Generalized Additive Models.

## Introduction To Regression
----------------------------

Linear regression is generally the first statistical modeling technique one learns.  It involves relating some target $y$ that is a continuous random variable to $p$ "features" represented as a vector: $\textbf{x}_{i} = [1, x_{1}, x_{2}, \ldots, x_{p}] \; \in \, \mathbb{R}^{p+1}$ using coefficients $\boldsymbol \omega = [\omega_0, \omega_1, \omega_2, \ldots, \omega_p] \; \in \, \mathbb{R}^{p+1}$.  The coefficient $\omega_0$ is called the "intercept." The features can have continuous and non-negative integer values and are generally considered to be non-random. The relationship between the target values and features values is described by the model,

\begin{equation}
y_{i} \; =\; \boldsymbol  \omega^{T} \textbf{x}_{i} + \epsilon_{i}
\end{equation}

Where $\epsilon_{i} \, \sim \, N(0,1)$ are independent and normal distributed with mean 0 and variance 1 and $\boldsymbol  \omega \, \in \mathbb{R}^{p+1}$ are the unknown co-efficients.  The fitting the linear regression model then becomes,

\begin{equation}
h_{\boldsymbol \omega}( \textbf{x}_{i}) \; =\; \boldsymbol  \omega^{T} \textbf{x}_{i}
\end{equation}

Fitting the model then becomes finding the cofficients $\boldsymbol \omega$ that best approximates $y_i \; = \; h_{\boldsymbol \omega}( \textbf{x}_{i})$ for $i=1, \ldots, n$.  This model is often called [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares) and [converges in probability](https://en.wikipedia.org/wiki/Consistent_estimator).

In general we will have many distinct measurements $\textbf{x}_{i}$ corresponding to values $y_i$ that are collected into a **training set**, $D_{\text{Train}} \, = \,  \left\{ (\textbf{x}_{i}, \, y_{i})  \right\}_{i=1}^{n}$.  This can be represend in the form $D_{\text{Train}} \, = \, (\textbf{X}_{\text{Train}}, \textbf{y}_{\text{Train}}$), where $\textbf{X}_{\text{Train}} \in \mathbb{R}^{n \times (p+1)}$ is the Design Matrix and $\textbf{y}_{\text{Train}} \in \mathbb{R}^{n}$ is the vector corresponding of target values. We also have a **test set**, $D_{\text{Test}} \, = \,  \left\{ (\textbf{x}_{i}, \, y_{i})  \right\}_{i=1}^{m}$.  This can be represend in the form $D_{\text{Test}} \, = \, (\textbf{X}, \textbf{y}$), where $\textbf{X}_{\text{Test}} \in \mathbb{R}^{m \times (p+1)}$ and $\textbf{y}_{\text{Test}} \in \mathbb{R}^{m}$ is the vector corresponding of target values.  The training set is the set of points we use to determine $\boldsymbol \omega$ and the testing evaluating the performance of our model.  For ease of notation we will simply drop the +1 from the $p+1$, but the reader should always remember there is an intectept.  Let's make things concrete using some example data.

The data we will use (`data.csv`) comes from the NYC Mayor's Office of Sustainbility and contains energy efficiency measurements of multi-family housing units in New York City.  We will use some of the measurements to predict the Green House Gas Emissions of the buildings, you can read more how this data was cleaned here.  The file has the following columns:

- `NGI` : Nautral Gas Use Intensity (kBtu/ft 2 )
- `EI` : Electricty Use Intensity (kBtu/ft 2 )
- `WI` : Water Use Intensity (kga/ft 2 )
- `Site_EUI` : Site Energy Usage Intensity (kBtu/ft 2 )
- `GHGI` : Green House Gas Emissions Intensity (Metric Tons CO2e / ft 2 )

We can import the data using Pandas which gives us a fast way to read in data from csv documents:

```python
import pandas  as pd

#columns we care about
cols = ['NGI','EI','WI','Site_EUI','GHGI']

# read the data
df = pd.read_csv("data.csv",
                 usecols = cols)

#view the first 5 lines
df.head()
```

      Site_EUI
      NGI
      EI
      WI
      GHGI

      0
      -0.037249
      -0.016434
      -0.012849
      -0.104832
      -0.037384

      1
      -0.037921
      -0.017372
      -0.007096
      -0.040264
      -0.037782

      2
      -0.033047
      -0.013440
      0.019025
      -0.047608
      -0.031731

      3
      -0.034623
      -0.012073
      -0.026548
      -0.118878
      -0.034398

      4
      -0.033804
      -0.013676
      0.008066
      -0.077564
      -0.032913

We split out dataframe into a training and test set:

```python
df_test  = df.sample(frac=0.25, random_state=42)
df_train = df.drop(df_test.index, axis=0)
```

Then our target variable then is the `GHGI` and design matrix becomes the rest of the columns:

```python
import numpy as np

# Design matrix
X = df_train[['NGI','EI','WI','Site_EUI']].values

# Target vector
y = df_train[["GHGI"]].values[:,0]

# Design matrix
X_test  = df_test[['NGI','EI','WI','Site_EUI']].values

# Target vector
y_test  = df_test[["GHGI"]].values[:,0]

print("X = \n", X)
print()
print("X.shape = ", X.shape)
print()
print("y = \n", y)
print()
print("X_test.shape = ", X_test.shape)
```

    X =
     [[-0.0164345  -0.01284852 -0.1048316  -0.0372486 ]
     [-0.01737222 -0.00709637 -0.04026423 -0.03792081]
     [-0.01344039  0.01902526 -0.04760821 -0.0330473 ]
     ...
     [-0.01034947 -0.03023676  0.00994682 -0.03592519]
     [-0.00979158 -0.03166883 -0.00486112 -0.0356311 ]
     [-0.00874449 -0.02223016 -0.05749552 -0.03100967]]

    X.shape =  (3702, 4)

    y =
     [-0.03738401 -0.03778169 -0.03173064 ... -0.03306691 -0.03265591
     -0.03082855]

    X_test.shape =  (1234, 4)

$X$ and $y$ are now a [NumPy](https://numpy.org/) matrix and array respectively and the same for $X_{\text{test}}$ and $y_{\text{test}}$. NumPy is Python library that has highly optimized data structures and algorithms for linear algebra.  Infact, [Pandas is written on top of NumPy](https://stackoverflow.com/questions/11077023/what-are-the-differences-between-pandas-and-numpyscipy-in-python).

We can then look at the relationship or correlation each of the features has on the target variables by looking at their scatter plots using a function I wrote:

```python
from plotting import plot_results
%matplotlib inline

plot_results(df_test)
```

These scatter plots show us the correlation between `GHGI` and the different features.  The far panel on the right shows us that as we increase `Site_EUI` on average we see `GHGI` increase linearly.  This makes sense as the more energy a building uses the more green house gas it produces.  The same is true of natural gas (`NGI`) as it is used for heating or cooking.  Therefore if this goes up so should our green house gas emissions.  Less intuitive may be electricity usage. We might not expect that as our electicity usage increases our green house gases increases, however, this could be due to the fact the source of the electricity comes from gas or coal.  That fact this is less intuitive could also be why the shape is less linear than `Site_EUI`.  The amount of water we use shoul not effect the green house gas emissions (maybe for heating hot water). This realiztion is displayed in the scater plot for `WI` as it does not show any distinct pattern.

Let's now return to the linear regression. Our goal is to approximate the target variable $y_i$ with a function $h_{\boldsymbol \omega}(\textbf{x}_{i})$. This function is a linear combination of our features,

\begin{equation}
h_{\boldsymbol \omega}(\textbf{x}_{i}) \, = \, \boldsymbol \omega^{T} \textbf{x}_{i}
\end{equation}

We find the solution vector or coefficients $\boldsymbol \omega$ by minimizing the cost function $J \left( \boldsymbol \omega \right)$,

\begin{align}
\hat{\boldsymbol \omega}
 \; &= \;
 \min_{\boldsymbol \omega} \, \frac{1}{2} \sum_{i=1}^{n} \left( y_{i}  -h_{\boldsymbol \omega}( \textbf{x}_{i}) \right)^{2} \\
&= \;
 \min_{\boldsymbol \omega} \, \frac{1}{2} \Vert \textbf{y} - h_{\boldsymbol \omega}(\textbf{x}) \Vert^{2} \\
 &= \;
 \min_{\boldsymbol \omega} \, J \left( \boldsymbol \omega \right)
\end{align}

Mininmizing $J \left( \boldsymbol \omega \right)$ is equivalent to setting the $\nabla J \left( \boldsymbol \omega \right) \; = \; 0$.  We can expand the cost function,

\begin{align}
J(\boldsymbol \omega )
\; &= \;
\frac{1}{2} \, \Vert  \textbf{y} - \textbf{X} \boldsymbol \omega \Vert^{2} \\
&= \;
 \min_{\boldsymbol \theta} \, \frac{1}{2} \Vert \textbf{y} - \textbf{X} \boldsymbol \omega \Vert^{2} \\
&= \; \frac{1}{2} \, \left( \textbf{y} - \textbf{X} \boldsymbol \omega \right)^{T}  \left( \textbf{y} - \textbf{X} \boldsymbol \omega  \right) \\
&= \; \frac{1}{2} \, \left( \textbf{y}^{T} \textbf{y}  - \textbf{y}^{T} \textbf{X} \boldsymbol \omega  - \boldsymbol \omega^{T} \textbf{X}^{T} \textbf{y} + \boldsymbol \omega^{T} \textbf{X}^{T} \textbf{X}  \boldsymbol \omega \right)
\end{align}

Taking the gradient of both sides we then have (for a review on matrix calculus see [here](https://atmos.washington.edu/~dennis/MatrixCalculus.pdf)),

\begin{equation}
\boldsymbol \nabla J(\boldsymbol \omega )
\; = \;
\textbf{X}^{T} \left( \textbf{y} - \textbf{X} \boldsymbol \omega \right)
\end{equation}

Setting the above equal to zero yields the linear system of equations,

\begin{equation}
\left(\textbf{X}^{T}\textbf{X}\right) \,
\hat{\boldsymbol \omega}
\; = \;
\textbf{X}^{T} \textbf{y}
\end{equation}

The above formulation is the so-called **"Normal Equations"**.  We can rewrite them as,

\begin{equation}
S \,
\hat{\boldsymbol \omega}
\; = \;
\textbf{b}
\end{equation}

Where $\textbf{b} \, \in \mathbb{R}^{p}$ and $S \, = \, \textbf{X}^{T}\textbf{X} \, \in \mathbb{R}^{p \times p}$. When the features have been scaled (as we have assumed), the  matrix $S$ is the **covariance matrix**

\begin{equation}
\left(\textbf{X}^{T}\textbf{X}\right)_{i,j} \; = \; \text{Cov}(x_{i},x_{j} )
\end{equation}

The vector $b \, = \, \textbf{X}^{T} \textbf{y} $ is the *projection of the target values onto the finite dimensional space that is spanned by the features.*  Defining the **residual r** as,

\begin{equation}
\textbf{r} \; = \; \textbf{y} - \textbf{X} \boldsymbol \omega
\end{equation}

Then we note $\nabla J \left( \hat{\boldsymbol \omega} \right) \; = \; 0$ is the same thing as saying,

\begin{equation}
X^{T} \, \textbf{r} \; = \; 0
\end{equation}

Or the solution $\hat{\boldsymbol \omega}$ is *value such that the residual is orthogonal to our feature space or the residual is statistically uncorrelated with any of the features.*

The covariance matrix is symmetric and positive definite, but if the two features are highly correlated then two rows in the matrix will be nearly identical and the matrix will be nearly singular. Statistically, having correlated features is bad since it makes it hard to determine the impact on any one feature has on the target. Numerically, having correlated features is bad because the condition number of the covariance matrix explodes.  The reason is that the condition number is definted as,

\begin{equation}
\kappa(S) \; = \; \Vert S \Vert_2 \, \Vert S^{-1} \Vert_2 \; = \; \frac{\sigma_{\text{max}}}{\sigma_{\text{min}}}
\end{equation}

Where $\sigma$ are the singular values from the singular value decomposition.  As the matrix $S$ becomes more singular $\Vert S^{-1} \Vert_2 \, \rightarrow \, \infty$, forcing the $\kappa \, \rightarrow \, \infty$.   We remark that the condition number affects not only the accuracy of your solution, but also the stability of the solvers, see [Lecture 12](https://people.maths.ox.ac.uk/trefethen/text.html).

There are two we will go over about solving the over-determined system of equations $X \boldsymbol \omega \, = \, \textbf{y}$:

1. Solving the Normal Equations using the Cholesky Decomposition

2. Solving the linear system using the Singular Value Decomposition

However, one can also solve the Normal Equations using an [LU factorization](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.linalg.lu_solve.html) or the original system with the [QR factorization](https://en.wikipedia.org/wiki/QR_decomposition).

## Linear Solvers For Least Squares Regression
------------------------
See [Lecture 11](https://people.maths.ox.ac.uk/trefethen/text.html) for more detailed discussions.

### Linear Solver 1: Cholesky Decomposition Of Normal Equations

Let's first solve the Normal Equations:

\begin{equation}
S \,
\boldsymbol \omega
\; = \;
X^{T} \textbf{y}
\end{equation}

Our covariance matrix $S$ is symmetrix positive definite we can use the [Cholesky decomposition](https://en.wikipedia.org/wiki/Cholesky_decomposition):

\begin{equation}
S = L L^{T}
\end{equation}

**where $L$ is an lower triangular matrix.** To solve the system from the nomrmal equations then becomes,

\begin{align}
S \, \boldsymbol \omega \; &= \; X^{T} \textbf{y} \\
L L^{T} \, \boldsymbol \omega \; &= \; X^{T} \textbf{y}
\end{align}

Which we can rewrite as first solving,

\begin{equation}
L \textbf{z} \; = \; X^{T} \textbf{y}
\end{equation}

Then solve,

\begin{equation}
L^T \boldsymbol \omega \; = \; \textbf{z}
\end{equation}

As $L$ is an lower triangular matrix each of these linear system of equations is simple to solve and the forward/backward substitutions made to solve them are backwards stable.

Let's try this out by first analyzing the covariance matrix

```python
# Build the covariance matrix
X_t = X.transpose()
S   = X_t.dot(X)

print("S = \n", S)
```

    S =
     [[ 0.74547278  1.35358196  2.7794211   1.72619886]
     [ 1.35358196  4.09897316  5.16118203  3.39404027]
     [ 2.7794211   5.16118203 16.53112563  6.69851448]
     [ 1.72619886  3.39404027  6.69851448  4.2625346 ]]

Let's take a look at the eigen values to understand the condititioning of our matrix

```python
eigs = np.linalg.eigvals(S)
print(eigs)
```

    [22.1809954   2.73222512  0.03890524  0.6859804 ]

```python
cond_num = np.sqrt( eigs.max() / eigs.min())
print(f"Condition number = {cond_num}")
```

    Condition number = 23.87736746652693

The eigenvalues are all positive and the condition number are not too big which is good!

We will now be using many SciPy function calls. SciPy is a sister Python library with Numpy and supports many scientific algorithms.  [SciPy's](https://docs.scipy.org/doc/scipy-0.16.1/reference/linalg.html) linear algebra routines expand on [NumPy's](https://docs.scipy.org/doc/numpy/reference/routines.linalg.html) and even allow for use sparse matrices and vectors. SciPy acts a wrappers around LAPACK, indeed the eigenvalue calculator directly calls [LAPACK's](http://www.icl.utk.edu/~mgates3/docs/lapack.html#geev) eigenvalue decomposition.

Let's now perform the Cholesky Decomposition using [SciPy](https://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.linalg.cholesky.html):

```python
from scipy import linalg

# Perform the Cholesky Factsaorization
L = linalg.cholesky(S, lower=True)
```

```python
L
```

    array([[0.86340766, 0.        , 0.        , 0.        ],
           [1.56772058, 1.28110317, 0.        , 0.        ],
           [3.21912954, 0.08936547, 2.48200412, 0.        ],
           [1.99928602, 0.2027303 , 0.0984836 , 0.46324014]])

We can check to make sure that $L L^T \, = \, S$:

```python
L.dot(L.T)
```

    array([[ 0.74547278,  1.35358196,  2.7794211 ,  1.72619886],
           [ 1.35358196,  4.09897316,  5.16118203,  3.39404027],
           [ 2.7794211 ,  5.16118203, 16.53112563,  6.69851448],
           [ 1.72619886,  3.39404027,  6.69851448,  4.2625346 ]])

Then use the matrix $L$ to solve the normal equations using SciPy's [solve_triangular](https://github.com/scipy/scipy/blob/v1.3.3/scipy/linalg/basic.py#L261) (which solves a lower/upper linear system).  This function directly calls [LAPACK's](http://www.icl.utk.edu/~mgates3/docs/lapack.html#trtrs) implementation of forward/backward substitution:

```python
# solve for the coefficents
z     = linalg.solve_triangular(L,   X_t.dot(y), lower=True)
omega = linalg.solve_triangular(L.T, z, lower=False)

print(f"omega : {omega}")
```

    omega : [-0.2376972   0.03247505  0.01313829  1.02531297]

We can now get the predicted values from $\hat{y} \, = \, X \hat{\boldsymbol \omega}$ on the test set and plot the results against the true values `y_test`:

```python
# Get the in sample predicted values
df_test["y_ols"] = X_test.dot(omega)

plot_results(df_test, 'y_ols')
```

Not bad! Let's move onto solving the system of equations with the singular value decomposition. I should note SciPy implements a [Cholesky solver](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.cho_solve.html#scipy.linalg.cho_solve) that directly calls [LAPACK's](http://www.icl.utk.edu/~mgates3/docs/lapack.html#posv) routine.

## Linear Solver 2: The Singular Value Decomposition (SVD)

Let's solve the overdetermined system of equations,

\begin{equation}
X \, \boldsymbol \omega \; = \; y
\end{equation}

using the [Singular Value Decomposition (SVD)](https://en.wikipedia.org/wiki/Singular_value_decomposition):

\begin{equation}
X \; = \; U \Sigma V^{T}
\end{equation}

Where $U \, \in \mathbb{R}^{n \times n}$ and $V \, \in  \mathbb{R}^{p \times p}$ are [unitary matrices](https://en.wikipedia.org/wiki/Unitary_matrix) and $\Sigma \, \in \mathbb{R}^{n \times p}$ is a diagonal matrix with the singular values along the diagonal. We can substitute in the SVD into our system of equations to obtain,

\begin{align}
X \, \boldsymbol \omega \; &= \; y \\
U \Sigma V^{T} \boldsymbol \omega \; &= \; y \\
U^{T} U \Sigma V^{T} \boldsymbol \omega \; &= \; U^{T} \, y \\
\Sigma V^{T} \boldsymbol \omega \; &= \; U^{T} \, y
\end{align}

We then transform this to a system of equations:

\begin{equation}
\Sigma \, \textbf{z} \; = \; U^{T} \, y
\end{equation}

and

\begin{equation}
V^{T} \boldsymbol \omega \; = \; \textbf{z}
\end{equation}

The last system of equation can be transformed using the fact $V$ is unitary ($VV^{T} \, = \, I$):

\begin{equation}
\boldsymbol \omega \; = \; V \textbf{z}
\end{equation}

We can then solve the original system of equations $X \, \boldsymbol \omega \; = \; y$ by

1. Finding the SVD of the Design Matrix $X \; = \; U \Sigma V^{T}$ then solve,

2. Inverting the diagonal matrix to solve $ \textbf{z} \; = \Sigma^{-1} \; U^{T} \, y$

3. Perform the matrix-vector multiplication: $\boldsymbol \omega \; = \; V \textbf{z}$

We can perform the SVD using [SciPy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.svd.html#scipy.linalg.svd)  (which to no surprise calls [LAPACK's](http://www.icl.utk.edu/~mgates3/docs/lapack.html#gesvd) routine):

```python
U, Sig, Vt = linalg.svd(X, full_matrices=False)
```

We can then look at the shapes of each of the resulting matrices:

```python
U.shape, Sig.shape, Vt.shape
```

    ((3702, 4), (4,), (4, 4))

And then perform the steps described above to find $\boldsymbol \omega$,

```python
z = (1 / Sig) * U.transpose().dot(y)

omega = Vt.transpose().dot(z)

print(f"omega : {omega}")
```

    omega : [-0.2376972   0.03247505  0.01313829  1.02531297]

The solution via Cholesky Factorization and the SVD have the same solutions!

One very nice thing about linear regression is that it is very easy to interpret. In linear regression, the coefficients tell us how much an increase in one unit of the feature increases the target variable by one unit.  Therefore increases in the sites energy usage intensity (`Site_EUI`) increase the `GHGI` the most and intuitively this makes sense.  However, the model is also telling us that the increases in natural gas usage (`NGI`) *decrease* the `GHGI` which doesn't make sense. This most likely happened because the features are correlated (see off diagonal values in covariance matrix $S$) or because the model was over-fitting, i.e. giving to much weight to a specific feature.

### Controling For Overfitting With Regularization
----------------------------
Often times a linear regression will give us good results on the training set, but on very bad results on another independent (test) dataset.  This is an example of overfitting our model.  We can control for overfitting our model by introducting regularization of the form,

\begin{align}
J(\boldsymbol \omega )
 \; = \;
 \frac{1}{2n} \sum_{i=1}^{n} \left( y_{i} -  h_{\boldsymbol \omega}(\textbf{x}_{i}) \right)^{2}   + \frac{\lambda}{q} \sum_{i=1}^{p} \vert \omega_{i} \vert^{q}
\end{align}

where $\lambda \, \geq \, 0$.  For $q=2$ this is called [Ridge Regression](https://en.wikipedia.org/wiki/Tikhonov_regularization) and for $q=1$ it is called [Lasso Regression](https://en.wikipedia.org/wiki/Lasso_(statistics)). Ridge Regression serves to control for over-fitting by shrinking the parameters $\omega_{i}$ as $\lambda$ increases.  Not all individual parameters must shrink as $\lambda$ increases, but in aggregate $\Vert \boldsymbol \omega \Vert^{2}_{\ell_2}$ decreases. We note that *both methods seek to reduce the variance in model, and subsequently increase the bias.*

**Remark: In neither method do we consider penalizing the value $\omega_0$.**

In the case of Ridge Regression we can view the cost function as,

\begin{equation}
\hat{\boldsymbol \omega}
 \; = \;
 \min_{\boldsymbol \omega}  \Vert \textbf{y} - \textbf{X} \boldsymbol \omega \Vert^{2}_{2}  +  \lambda  \Vert \boldsymbol \omega \Vert^{2}_{2}
\end{equation}

Which after expanding the values and setting the derivative to zero yields,

\begin{equation}
\hat{\boldsymbol \omega}
\; = \;
\left(\textbf{X}^{T} \textbf{X}  + \lambda I \right)^{-1}
 \left(\textbf{X}^{T} \textbf{y} \right)
\end{equation}

We can see that the regularization term acts to remove the singularity from the covariance matrix if there is correlation among features.  Using the SVD we arrive at:

\begin{align}
\left(\textbf{X}^{T}\textbf{X}  + \lambda I \right) \, \boldsymbol \omega
\; &= \;
 \left(\textbf{X}^{T} \textbf{y} \right) \\
\left( V \Sigma^{T} \Sigma V^{T} + \lambda I \right) \, \boldsymbol \omega
\; &= \;
 \left(V \Sigma^{T} U^{T} \right) \, \textbf{y} \\
 V \left( \Sigma^{T} \Sigma   + \lambda I \right) V^{T} \, \boldsymbol \omega
\; &= \;
\left(V \Sigma^{T} U^{T} \right) \, \textbf{y} \\
\left( \Sigma^{T} \Sigma  + \lambda I \right) V^{T} \, \boldsymbol \omega
\; &= \; \Sigma^{T} U \, \textbf{y}
\end{align}
We can transform the last equation to the system of equations:

\begin{equation}
\left( \Sigma^{T} \Sigma + \lambda I \right) \textbf{z}
\; = \; \Sigma^{T} U^{T} \, \textbf{y}
\end{equation}

with the system corresponding equation $V^{T} \, \hat{\boldsymbol \omega} \; = \; \textbf{z}$ which can be rewritten using the unitary property of $V$ as the matrix-vector multiplication:

\begin{equation}
\hat{\boldsymbol \omega}\; = \; V \textbf{z}
\end{equation}

The matrix, $\left( \Sigma^{T} \Sigma + \lambda I \right) \, \in \mathbb{R}^{p \times p}$ is diagonal and can be inverted by hand.  This gives us a way of solving the problem for Ridge Regression for a given $\lambda$:

1. Finding the SVD $X \; = \; U \Sigma V^{T}$ then solve

2. "Invert" the matrix to obtain $ \textbf{z} \; = \; \left( \Sigma^{T} \Sigma  + \lambda I \right)^{-1} \, \Sigma^{T} \, U^{T} \, \textbf{y}$

3. Perform the matrix-vector multiplication $\hat{\boldsymbol \omega} \; = \; V \textbf{z}$

We remark we can combine the 2. and 3. to obtain the formula:

\begin{equation}
\hat{\boldsymbol \omega} \; = V \, \left( \Sigma^{T} \Sigma  + \lambda I \right)^{-1} \, \Sigma^{T} \, U^{T} \, \textbf{y}
\end{equation}

And then our predictions $\hat{y} \; = \; X \, \hat{\omega}$ become:

\begin{align}
\hat{y} \; &= \; X \, \hat{\omega} \\
&= \; U \, \Sigma \, \left( \Sigma^{T} \Sigma  + \lambda I \right)^{-1} \, \Sigma^{T} \, U^{T} \, \textbf{y} \\
&= \; U \, D \, U^{T} \, \textbf{y}
\end{align}

As you may have learned the matrix $U$ is unitary so it acts a change of basis transformation and the matrix $D \, = \, \Sigma \, \left( \Sigma^{T} \Sigma + \lambda I \right)^{-1} \, \Sigma^{T} \, \in  \mathbb{R}^{n \times n}$ is a diagonal matrix that either shrinks or elongates along this new basis as shown below:

|  |
|:--:|
| *From https://upload.wikimedia.org/wikipedia/commons/b/bb/Singular-Value-Decomposition.svg* |

The values along the diagonal of $D$ take the form,

\begin{equation}
D_{i,i} \; = \; \frac{\sigma_{i}^{2}}{\sigma_{i}^{2} + \lambda}
\end{equation}

From this formulation $D$ acts to shrink the influence of low frequency singular vectors ($\sigma$ small) more than it does high frequency singular vector ($\sigma$ large)!

Let's see how our code will look for solving the Ridge Regression problem using the SVD.

```python
# using function annotations in Python 3
# (see, https://www.python.org/dev/peps/pep-3107/)
import scipy as sp
def ridge_svd(
    X         : np.ndarray,
    y         : np.ndarray,
    reg_param : float = 1.0
) -> np.ndarray:
    """
    Solves the Ridge Regression problem using the SVD.

    Params:
    -------
    X : The matrix of features
    Y : The target vector
    reg_param : The regularization parameter

    Returns:
    --------
    omega : The array of coefficent for the solution
    """

    # compute the SVD
    U, Sig, Vt   = sp.linalg.svd(X, full_matrices=False)

    # for Sigma^{T} Sigma + \lambda (this is an array)
    Sig_plus_reg = Sig * Sig + reg_param

    # find z by "inverting" the matrix and then get omega
    z     = (1.0 / Sig_plus_reg) * Sig * U.T.dot(y)
    omega = Vt.T.dot(z)

    return omega
```

Let's test this with $\lambda \, = \, 0$ which should give us the least squares regression solution:

```python
print("omega: ", ridge_svd(X, y, 0.0))
```

    omega:  [-0.2376972   0.03247505  0.01313829  1.02531297]

It does! Let's try adding a little regularization and see what happens.

```python
omega = ridge_svd(X, y, 0.1)
print(f"omega: {omega}")
```

    omega: [0.14161464 0.07955296 0.03040791 0.78857811]

We can see that adding regulaization shrank the last feature, i.e. (`Site_EUI`) and made turned the coefficient for `NGI` positive which makes sense!  Let's take a look at the results:

```python
df_test["y_ridge"] = X_test.dot(omega)

plot_results(df_test, 'y_ols', 'y_ridge')
```

It's hard to say whether least squares or ridge regression is better from these figures, but the coefficients for Ridge Regression make much more sense.  Notice that I have left out how to find the regularization parameter $\lambda$.  That topic is out of scope for this post. However, for reference, the best value for $\lambda$ is often chosen via cross-validation, see [these slides](http://statweb.stanford.edu/~owen/courses/305a/Rudyregularization.pdf) for more information.

### Implementation In Scikit-Learn

[Scikit-Learn](https://scikit-learn.org/) is the industry standard open source machine leanring library in Python. As it is open source we can easily see how the developers wrote their linear sovlers by going to the library's [GitHub](https://github.com/scikit-learn/scikit-learn).

For regression, Scikit-learn mostly acts as a wrapper around Scipy which act as a wrapper to BLAS/LAPACK. The regular linear regression [Scikit-Learn](https://github.com/scikit-learn/scikit-learn/blob/1495f6924/sklearn/linear_model/base.py#L367) implementation utilizes Scipy's [least square solver](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.lstsq.html), which if the matrix is dense calls [BLAS/LAPACK](https://github.com/numpy/numpy/blob/master/numpy/linalg/umath_linalg.c.src#L3167).  If the matrix $X$ is sparse then it calls SciPy's [sparse least squares solver](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.lstsq.html) which uses least squres with QR factorization [LSQR](https://github.com/scipy/scipy/blob/v1.3.2/scipy/sparse/linalg/isolve/lsqr.py#L98-L570).  Note that while $X \, \in \mathbb{R}^{n \times p}$ may be a large sparse matrix, the covariance matrix $S \, \in \mathbb{R}^{p \times p}$ is typically quite small and dense.

For ridge regression, unless the user specifices the linear sovler to use then if the intercept is needed Scikit-learn solve the linear system with the [Stochastic Average Gradient](https://arxiv.org/abs/1309.2388) method, otherwise it uses the [Cholesky Decomposition](https://github.com/scikit-learn/scikit-learn/blob/1495f6924/sklearn/linear_model/ridge.py#L134) for dense matrices and [Conjugate Gradient](https://github.com/scikit-learn/scikit-learn/blob/1495f6924/sklearn/linear_model/ridge.py#L37) method for sparse matrices. Although, all the algorithms are written for dense or sparse matrices. The congugate gradient method makes use of SciPy's [Sparse CG function](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.cg.html) which does not need to explicitly form the matrix, but rather only needs its action on a vector making it extremely efficient for sparse matrices.  If any of these methods fail, the least square QR solver is used.

## A Touch Of Recommendation Systems
------------------------------------

Recommendation systems have become very popular and are one of the best examples of Machine Learning that we interact with daily, i.e. Amazon and Netflix. One popular technique for making recommendations is called Alternating Least Squares.

The idea is that we have some table of ratings of items by users as shown below:

|  |
|:--:|
| *From https://recsysjd.wordpress.com/2016/09/10/matrix-factorization-techniques-for-recommender-systems* |

However, not all users will have rated all items and we want to predict how a user may feel about an item they have not rated.  This is how recomendations happen, through the prediction of how a user might rate an item they have not seen before!

Let's for example take the idea of recommending movies to users of a webservice.  The idea is we want to say a given user has some preference on a movie by the perfence of the generes that make up the movie.  For simplicity let's say movies are a combination of the genres (so-called [latent factors](https://en.wikipedia.org/wiki/Latent_variable) in machine learning):

- Horror
- Action
- Comedy

Then we can write a user's preference as well a movie as a linear combinatin of these genres:

\begin{align}
\text{Sally} \; &= \; 1 \times \text{Horror} +  2 \times \text{Action} - 0.2 \times \text{Comedy} \\
\ \text{Weekend At Bernies} \; &= \; 0 \times \text{Horror} +  1 \times  \text{Action} + 5 \times \text{Comedy}  \\
\text{Sally's Rating of Weekend At Bernies}  \; &= \; 1 \times 0 + 2 \times 1 - 0.2 \times 5 \\
&= \; 1 \, \text{Star}
\end{align}

The above shows how Sally would rate "Weekend at Bernies" given that she likes films involving Horror and Action, but dislikes action films and that "Weekend at Bernies" is a little bit of action and a lot of comedy. If we label $R_{u,i}$ as user $u$'s rating of movie $i$, then the inner product of the vector of movie coefficients $m_{i}$ and user preferences $p_{u}$ is $R_{u,i}$ :

\begin{equation}
R_{u,i} \; = \; \textbf{m}_{i}^{T} \textbf{p}_{u}
\end{equation}

We can then factorize the review matrix $R$ into a product of movies $M$ and users $P$ as depicted in the diagram above,

\begin{equation}
R \; \simeq \; M^{T} P
\end{equation}

If $R \in \mathbb{R}^{n \times k}$ and we have $f$ latent factors that make a movie then the movie genres are $G \in \mathbb{R}^{f \times n}$ and user preferences are $P \in \mathbb{R}^{f \times k}$.  How do we find the coefficient for each user and each movies, i.e. how do we find the matrices $M$ and $P$?

The idea is we can set up a cost function almost like Ridge Regression and minimize with respect to $M$ and $P$:

\begin{equation}
\min_{M,P} \, J(M,P) \; = \; \frac{1}{2} \Vert R - M^{T} P \Vert^{2}_{F}  +
\frac{\lambda}{2} \left( \Vert M \Vert^{2}_{F} + \Vert P \Vert^{2}_{F} \right) \
\end{equation}

Where $\Vert \cdot \Vert_{F}$ is the [Frobenius Norm](https://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm).  How do we minimize for both $M$ and $P$? Well one way is to:

0. Randomly initialized $M$ and $P$ then

1. Fix $M$ and solve for $P$ such that $\nabla_{P} J(M,P) \; = \; 0$

2. Fix $P$ and solve for $M$ such that $\nabla_{M} J(M,P) \; = \; 0$

3. Repeat steps 1 and 2 until convergence or max iterations reached

At each stage in 1. and 2. you are updating the choice and $M$ and $P$ by essentially solving a Least Squares Ridge Regression problem for $P$ and $M$. The alternating nature of this iterative algorithm gives rise to its name [Alternating Least Squares](https://link.springer.com/chapter/10.1007%2F978-3-540-68880-8_32)! Some links to good references for Alternating Least Squares below.

## Where To Go From Here
--------------

We've gone over a few topics in this blogpost and I want to supply the reader with some links for more information on the topics discussed.

### Numerical Linear Algebra

- http://people.maths.ox.ac.uk/~trefethen/text.html

- https://jhupbooks.press.jhu.edu/title/matrix-computations

- https://www.springer.com/gp/book/9783540346586

### Regression & Machine Learning

- http://faculty.marshall.usc.edu/gareth-james/ISL/

- https://www.crcpress.com/An-Introduction-to-Generalized-Linear-Models/Dobson-Barnett/p/book/9781138741515

- https://web.stanford.edu/~hastie/Papers/ESLII.pdf

- http://stanford.edu/~rezab/classes/cme323/S15/

### Recommendation Systems:
- https://spark.apache.org/docs/latest/ml-collaborative-filtering.html

- http://www.mattmoocar.me/recsys/

- https://web.stanford.edu/~rezab/papers/fastals.pdf

- http://yifanhu.net/PUB/cf.pdf


```python
import numpy as np
```

```python
A = np.array([[2,3],[2,2]])
```

```python
A
```

    array([[2, 3],
           [2, 2]])

```python
np.sum(A, axis=0)
```

    array([4, 5])

```python
np.sum(A, axis=1)
```

    array([5, 4])

```python
np.dot(A,A)
```

    array([[10, 12],
           [ 8, 10]])

```python
import scipy
import scipy.integrate
```

```python
def velocity(t):
    return 25 + 10.0*t
```

```python
distance = scipy.integrate.quad(velocity, 0, 3)
```

```python
print distance
```

    (120.0, 1.3322676295501878e-12)

```python
import sympy
from sympy import sin, cos
from sympy.abc import x, y
```

```python
# Differentiate below expression
sympy.diff(x**2 + x**3 + cos(x), x)
```

    3*x**2 + 2*x - sin(x)

```python

```

```python

```


# Modelproject - Cournot competition

## Introduction
In this project, we  find the optimal production quantity for each of two firms in a Cournot - competition. We compare the situation with two identical firms and two non-identical firms.

### We apply following assumptions for the model:
* There are two firms in the economy. In the first part they are identical, in the second part they are non-identical
* The firms choose the output simultanously ($Q_1$ and $Q_2$)
* The total quantity in the economy is $Q = Q_1 + Q_2$
* The marketprice is decreasing in quantity produced $P(Q) = a - b*Q$
* For identical firms: Both firms share equal marginal cost $0 \leq c<a$, $ATC = MC = c$
* For non-identical firms: The firms has different marginal cost: ($c_1$, $c_2$)

In the end of the paper, we compare the outcomes for a market with identical firms vs non-identical firms

```python
# Importing packages
from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
import numpy as np
import sympy as sm
from IPython.display import display
from sympy import simplify
```

```python
# Set symbols according to the equation
q1, q2 = sm.symbols('Q_1 Q_2') # Quantity for each firm
c = sm.symbols('c') # Marginal cost for bth firms on the identical-market
c1, c2 = sm.symbols('c_1 c_2') # Marginal cost for each firm on the non-identical market
a, b = sm.symbols('a b') # Constants
```

# Two Identical Firms
Define functions

```python
# We define p being the demand (price) on the market for two identical firms
def p(q1, q2, b, a):
    return a-b*q1-b*q2

```

```python
# Define the total cost functions for firms 1 and firm 2.
def tc1(q1,c):
    return c*q1

def tc2(q2,c):
    return c*q2
```

```python
# Define profits for firm 1 and firm 2
def profit1(q1, p, tc1):
    return q1*p(q1, q2, b, a)-tc1(q1,c)

def profit2(q2, p, tc2):
    return q2*p(q1, q2, b, a)-tc2(q2,c)
```

To maximize profits for each firm, we differentiate the profit functions wrt. Q1 and Q2 respectively and find the first-order-conditions

```python
# Diff and define FOCs:
print('First order condition for firm 1:')
FOC1 = sm.diff(profit1(q1, p, tc1), q1)
display(FOC1)

print('First order condition for firm 2:')
FOC2 = sm.diff(profit2(q2, p, tc2), q2)
display(FOC2)
```

    First order condition for firm 1:

$\displaystyle - 2 Q_{1} b - Q_{2} b + a - c$

    First order condition for firm 2:

$\displaystyle - Q_{1} b - 2 Q_{2} b + a - c$

To determine how each firm reacts (the choice of quantity) to the opposite firm's optimal quantity, we define the reactionfunctions

```python
#Define reaction functions
print('Reaction function R(q1):')
R1 = sm.solve(FOC1,q1)
q1_solve=(R1[0])
display(q1_solve)

print('Reaction function R(q2):')
R2 = sm.solve(FOC2,q2)
q2_solve=(R2[0])
display(q2_solve)
```

    Reaction function R(q1):

$\displaystyle \frac{- Q_{2} b + a - c}{2 b}$

    Reaction function R(q2):

$\displaystyle \frac{- Q_{1} b + a - c}{2 b}$

We solve for Q1, Q2 and find the total quantity of goods in the market:

```python
print('The equation with two unknowns (Q1 and Q2), we have to solve')
q_all = sm.Eq((q1_solve + q2_solve))
display(q_all)

sol_dict = sm.solve((FOC1,FOC2), (q1, q2))

print('Optimal Quantity for firm 1:')
q1_optimal = sol_dict[q1]
display(q1_optimal)

print('Optimal Quantity for firm 2:')
q2_optimal = sol_dict[q2]
display(q2_optimal)

print('Total Quantity in the economy with two identical firms:')
q_total = q1_optimal + q2_optimal
display(q_total)
```

    The equation with two unknowns (Q1 and Q2), we have to solve

$\displaystyle \frac{- Q_{1} b + a - c}{2 b} + \frac{- Q_{2} b + a - c}{2 b} = 0$

    Optimal Quantity for firm 1:

$\displaystyle \frac{a - c}{3 b}$

    Optimal Quantity for firm 2:

$\displaystyle \frac{a - c}{3 b}$

    Total Quantity in the economy with two identical firms:

$\displaystyle \frac{2 \left(a - c\right)}{3 b}$

When total quantity (Q) is found, we can apply this to the demand (price)

```python
print('Optimal demand (price) in Cournot-competition for two identical firms:')
p_opt = a-b*q_total
display(p_opt)
```

    Optimal demand (price) in Cournot-competition for two identical firms:

$\displaystyle \frac{a}{3} + \frac{2 c}{3}$

We find the profit for each firm and total profit in the economy by inserting optimal price, marginal cost and optimal quantity in the profit function

```python
print('Profit for firm 1:')
profit_1 = (p_opt-c)*q1_optimal
display(sm.simplify(profit_1))

print('Profit for firm 2:')
profit_2 = (p_opt-c)*q2_optimal
display(sm.simplify(profit_2))

print('Total profit in the economy:')
profit_total = profit_1 + profit_2
display(sm.simplify(profit_total))
```

    Profit for firm 1:

$\displaystyle \frac{\left(a - c\right)^{2}}{9 b}$

    Profit for firm 2:

$\displaystyle \frac{\left(a - c\right)^{2}}{9 b}$

    Total profit in the economy:

$\displaystyle \frac{2 \left(a - c\right)^{2}}{9 b}$

Not surprisingly, the profits are equivalent for the identical firms. Now, what happens when the firms are not identical?

# Two non-identical Firms
 Now, the two firms does not share the same level of costs (c). For this reason, we apply $c_1$ and $c_2$ for firm 1 and firm 2 respectively
* Firm 1: $ATC1 = MC1 = c_1$
* Firm 2: $ATC2 = MC2 = c_2$

The demand function (P) and the reaction functions $R(Q_i)$ are equal to the ones found before with two identical firms.
Therefore, we apply the same procedure

```python
# Define the total cost functions

def tc1_non(q1,c1):
    return c1*q1

def tc2_non(q2,c1):
    return c1*q2
```

```python
# Define the profit functions

def profit1_non(q1, p, tc1):
    return q1*p(q1, q2, b, a)-tc1(q1,c1)

def profit2_non(q2, p, tc2):
    return q2*p(q1, q2, b, a)-tc2(q2,c2)
```

```python
# To determine the optimal quantity produced for each firm, we first find the first-order-conditions (FOC)

print('First order condition for firm_non 1:')
FOC1_non = sm.diff(profit1_non(q1, p, tc1), q1)
display(FOC1_non)

print('First order condition for firm_non 2:')
FOC2_non = sm.diff(profit2_non(q2, p, tc2), q2)
display(FOC2_non)
```

    First order condition for firm_non 1:

$\displaystyle - 2 Q_{1} b - Q_{2} b + a - c_{1}$

    First order condition for firm_non 2:

$\displaystyle - Q_{1} b - 2 Q_{2} b + a - c_{2}$

```python
# To determine how each firm reacts (the choice of quantity) to the opposite firm's optimal quantity, we define the reactionfunctions
print('Reaction function R(q1) non-identical firms:')
R1_non = sm.solve(FOC1_non,q1)
q1_solve_non=(R1_non[0])
display(q1_solve_non)

print('Reaction function R(q2) non-identical firms:')
R2_non = sm.solve(FOC2_non,q2)
q2_solve_non=(R2_non[0])
display(q2_solve_non)

# As there are two firms on the market, we set these reaction functions equal to eachother

print('The equation for non-identical firms with two unknowns (Q1 and Q2), we have to solve:')
q_all_non = sm.Eq((q1_solve_non + q2_solve_non), 0)
display(sm.simplify(q_all_non))
```

    Reaction function R(q1) non-identical firms:

$\displaystyle \frac{- Q_{2} b + a - c_{1}}{2 b}$

    Reaction function R(q2) non-identical firms:

$\displaystyle \frac{- Q_{1} b + a - c_{2}}{2 b}$

    The equation for non-identical firms with two unknowns (Q1 and Q2), we have to solve:

$\displaystyle \frac{- Q_{1} b - Q_{2} b + 2 a - c_{1} - c_{2}}{2 b} = 0$

Now we solve for Q1 and Q2 and find the total quantity of goods in the market:

```python
# Define a dictionary to solve for optimal quantities
sol_dict_non = sm.solve((FOC1_non,FOC2_non), (q1, q2))

# Optimal quantity for the firms
print('Optimal Quantity for firm 1:')
q1_optimal_non = sol_dict_non[q1]
display(q1_optimal_non)

print('Optimal Quantity for firm 2:')
q2_optimal_non = sol_dict_non[q2]
display(q2_optimal_non)

print('Total Quantity in the economy:')
q_total_non = (q1_optimal_non + q2_optimal_non)
display(sm.simplify(q_total_non))
```

    Optimal Quantity for firm 1:

$\displaystyle \frac{a - 2 c_{1} + c_{2}}{3 b}$

    Optimal Quantity for firm 2:

$\displaystyle \frac{a + c_{1} - 2 c_{2}}{3 b}$

    Total Quantity in the economy:

$\displaystyle \frac{2 a - c_{1} - c_{2}}{3 b}$

When total quantity (Q) is found, we can apply this to the demand (price)

```python
print('Optimal demand (price) in Cournot-competition for two non-identical firms:')
p_opt_non = a-b*q_total_non
sm.simplify(p_opt_non)
```

    Optimal demand (price) in Cournot-competition for two non-identical firms:

$\displaystyle \frac{a}{3} + \frac{c_{1}}{3} + \frac{c_{2}}{3}$

We find the profit for each non-identical firm and total profit in the economy by inserting optimal price, marginal cost and optimal quantity in the profit function

```python
# Profit for each firm is given by

print('Profit for firm 1:')
profit_1_non = (p_opt_non*q1_optimal_non)-c1*q1_optimal_non
display(simplify(profit_1_non))

print('Profit for firm 2:')
profit_2_non = (p_opt_non*q2_optimal_non)-c2*q2_optimal_non
display(sm.simplify(profit_2_non))

# Total profit in the economy

print('Total profit in the economy for two non-identical firms:')
profit_total_non = profit_1_non + profit_2_non
display(sm.simplify(profit_total_non))
```

    Profit for firm 1:

$\displaystyle \frac{\left(a - 2 c_{1} + c_{2}\right)^{2}}{9 b}$

    Profit for firm 2:

$\displaystyle \frac{\left(a + c_{1} - 2 c_{2}\right)^{2}}{9 b}$

    Total profit in the economy for two non-identical firms:

$\displaystyle \frac{2 a^{2} - 2 a c_{1} - 2 a c_{2} + 5 c_{1}^{2} - 8 c_{1} c_{2} + 5 c_{2}^{2}}{9 b}$

## Compare the two situations with figure-plots and Sliders to change the values.
Here, you can play around with the sliders to observe the impact of the
different parameters, given in the model

```python
# Equilibrium quantities and profits for 2 identical firms

def f(a, b, c):
    display(f'Optimal quantity for identical firms is {(a-c)/(3*b)}')
    display(f'Profit for identical firms is {(a-c)**2/(9*b)}')
interact(f, a=(0.0,10.0), b=(0.0,1.0), c=(0.0,10.0));
```

    interactive(children=(FloatSlider(value=5.0, description='a', max=10.0), FloatSlider(value=0.5, description='b…

```python
def f(a, b, c1, c2):
    display(f'Optimal quantity non-identical firm 1 is {(a-2*c1+c2)/(3*b)}')
    display(f'Optimal quantity non-identical firm 2 is {(a-2*c2+c1)/(3*b)}')
    display(f'The profit given parameter values for non-identical firm 1 is {(((a-2*c1+c2))**2)/(9*b)}')
    display(f'The profit given parameter values for non-identical firm 2 is {(((a-2*c2+c1))**2)/(9*b)}')
interact(f, a=(0.0,10.0), b=(0.0,1.0), c1=(0.0,5.0), c2=(0.0,5.0));
```

    interactive(children=(FloatSlider(value=5.0, description='a', max=10.0), FloatSlider(value=0.5, description='b…

# conclusion

### We have created an algorithm that can solve Cournot-competition problems and created sliders that show  results dependent on parameter values.
### We find for all firms that the price is positvely correlated with a and negatively correlated with b.
### Identical firms are affected negatively by higher prices.
### Profits of non-identical firms are negatively affected if their own production costs increase, but profits are positively affected if competition production costs increase

____________________________________

## Comparing the market for Identical firms vs non-identical firms

### Identical firms

* Total quantity produced:
$ Q= \frac{2(a-c)}{3b}$
* price: $ P = \frac{a+2c}{3}$
* Total profit:
$ = \frac{2(a-c)^2}{9b}$
* Profit for each firm: $Profit=\frac{(a-c)^2}{9b}$

### non-identical firms
* Total quantity produced:
  $Q=\frac{2a-c_1-c_2}{3b}$
* Quantity firm 1: $Q_1=\frac{a-2c_1+c_2}{3b}$
* Quantity firm 2: $Q_2=\frac{a+c_1-2c_2}{3b}$
* price:
  $P=\frac{a+c_1+c_2}{3}$
* Total profit:
$ = \frac{(2a-c_1-c_2)^2}{9b}$
* profit firm 1: $ = \frac{(a-2c_1+c_2)^2}{9b}$
* Profit firm 2: $ = \frac{(a+c_1-2c_2)^2}{9b}$

The main difference on the two markets is given by the non-identical level of marginal cost, $c$

For $c > c_1+c_2$:
* Total produced quantity will be lower for identical firms. Vice Versa for non-identical firms.
* The price of the product will be higher in the economy for identical firms. Vice Versa for non-identical firms
* Ceterus Parabus profit will be lower for identical firms vs non-identical firms.


# A/B Testing from Scratch: Bayesian Approach

We reuse the simple problem of comparing two online ads campaigns (or teatments, user interfaces or slot machines). We details how Bayesian A/B test is conducted and highlights the differences between it and the frequentist approaches. Readers are encouraged to tinker with the widgets provided in order to explore the impacts of each parameter.

```python
import numpy as np
import pandas as pd

#widgets
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from IPython.display import display

#plots
import matplotlib.pyplot as plt
from plotnine import *
from mizani import *

#stats
import scipy as sp
import statsmodels as sm

import warnings
warnings.filterwarnings('ignore')
import collections
```

## Start with A Problem

A typical situation marketers (research physicians, UX researchers, or gamblers) find themselves in is that they have two variations of ads (treatments, user interfaces, or slot machines) and want to find out which one has the better performance in the long run.

Practitioners know this as A/B testing and statisticians as **hypothesis testing**. Consider the following problem. We are running an online ads campaign `A` for a period of time, but now we think a new ads variation might work better so we run an experiemnt by dividing our audience in half: one sees the existing campaign `A` whereas the other sees a new campaign `B`. Our performance metric is conversion (sales) per click (ignore [ads attribution problem](https://support.google.com/analytics/answer/1662518) for now). After the experiment ran for two months, we obtain daily clicks and conversions of each campaign and determine which campaign has the better performance.

We simulate the aforementioned problem with both campaigns getting randomly about a thousand clicks per day. The secrete we will pretend to not know is that hypothetical campaign `B` has slightly better conversion rate than `A` in the long run (10.5% vs 10%). With this synthetic data, we will explore some useful statistical concepts and exploit them for our frequentist A/B testing.

```python
def gen_campaigns(p1,p2,nb_days,scaler,seed):
    #generate fake data
    np.random.seed(seed)
    ns = np.random.triangular(50,100,150,size=nb_days*2).astype(int)
    np.random.seed(seed)
    es = np.random.randn(nb_days*2) / scaler

    n1 = ns[:nb_days]
    c1 = ((p1 + es[:nb_days]) * n1).astype(int)
    n2 = ns[nb_days:]
    c2 = ((p2 + es[nb_days:]) * n2).astype(int)
    conv_days = pd.DataFrame({'click_day':range(nb_days),'click_a':n1,'conv_a':c1,'click_b':n2,'conv_b':c2})

    conv_days =  conv_days[['click_day','click_a','click_b','conv_a','conv_b']]
    conv_days['cumu_click_a'] = conv_days.click_a.cumsum()
    conv_days['cumu_click_b'] = conv_days.click_b.cumsum()
    conv_days['cumu_conv_a'] = conv_days.conv_a.cumsum()
    conv_days['cumu_conv_b'] = conv_days.conv_b.cumsum()
    conv_days['cumu_rate_a'] = conv_days.cumu_conv_a / conv_days.cumu_click_a
    conv_days['cumu_rate_b'] = conv_days.cumu_conv_b / conv_days.cumu_click_b
    return conv_days

conv_days = gen_campaigns(p1 = 0.10,
                          p2 = 0.105,
                          nb_days = 24,
                          scaler=300,
                          seed = 1412) #god-mode
conv_days.head()
```

      click_day
      click_a
      click_b
      conv_a
      conv_b
      cumu_click_a
      cumu_click_b
      cumu_conv_a
      cumu_conv_b
      cumu_rate_a
      cumu_rate_b

      0
      0
      125
      87
      12
      9
      125
      87
      12
      9
      0.096000
      0.103448

      1
      1
      114
      86
      11
      9
      239
      173
      23
      18
      0.096234
      0.104046

      2
      2
      67
      91
      6
      9
      306
      264
      29
      27
      0.094771
      0.102273

      3
      3
      96
      103
      9
      10
      402
      367
      38
      37
      0.094527
      0.100817

      4
      4
      89
      125
      9
      13
      491
      492
      47
      50
      0.095723
      0.101626

```python
rates_df = conv_days[['click_day','cumu_rate_a','cumu_rate_b']].melt(id_vars='click_day')
g = (ggplot(rates_df, aes(x='click_day', y='value', color='variable')) + geom_line() + theme_minimal() +
          xlab('Hours of Experiment Run') + ylab('Cumulative Conversions / Cumulative Clicks'))
g
```

```python
#sum after 2 months
conv_df = pd.DataFrame({'campaign_id':['A','B'], 'clicks':[conv_days.click_a.sum(),conv_days.click_b.sum()],
                        'conv_cnt':[conv_days.conv_a.sum(),conv_days.conv_b.sum()]})
conv_df['conv_per'] =  conv_df['conv_cnt'] / conv_df['clicks']
conv_df
```

      campaign_id
      clicks
      conv_cnt
      conv_per

      0
      A
      2488
      234
      0.094051

      1
      B
      2209
      222
      0.100498

## Think Like A Bayesian

The core idea of Bayesian A/B testing is to formulate a posterior distribution for each variation. Recall that frequentist approach does not do this; it instead first assume a null hypothesis that there is no difference between the variations and see the data to determine the false positive rate. The advantage is that we can have hypotheses directly about the true values we are interested in such as the conversion rate for each variation. The cost is that now we cannot leverate central limit theorem to assume normal distribution for our hypotheses; instead we need to construct our own distributions, called **posterior distriubtion**, using Bayes' Rule:

\begin{align}
P(H|D) &= \frac{P(H \cap D)}{P(D)} \\
&= \frac{P(D|H)P(H)}{P(D)} \text{; chain rule of probability}\\
& = \frac{P(D|H)P(H)}{\sum_{j=1}^k P(D|H_j)P(H_j)} \text{; summing over all possible hypotheses}
\end{align}
where $H$ is the hypothesis or model and $D$ is the data or evidence

* $P(H|D)$ is the probability of the hypothesis being true given the observed data, called the **posterior**. This is the distribution we use to estimate the true values such as true conversion rates in the Bayesian approach.
* $P(D|H)$ is the probability of seeing the data given that the hypothesis is true, called the **likelihood**, this is similar to p-value of rejection of the null hypothesis under frequentist approach.
* $P(H)$ is the probability of the hypothesis being true aka our belief in the hypothesis, called the **prior**. Refer to a table of [conjugate priors](https://en.wikipedia.org/wiki/Conjugate_prior) to choose the suitable prior for your posterior and likelihood.
* $P(D)$ is the probability of the data being present, called the **evidence**.

## Derive The Posterior Distribution

In our case, the likelihood that we would see this set of clicks and conversions data given that the true conversion rate is `p` can be described as the probability mass function of a Bernouilli distribution:

\begin{align}
P(D|H) &= \prod_{i=1}^{n} p^{x_i} (1-p)^{1-x_i} \\
&= p^{\sum_{i=1}^{n} x_i} (1-p)^{\sum_{i=1}^{n} (1-x_i)}
\end{align}

where $x_i$ is the binary flag for conversion and `p` is the true conversion rate by the hypothesis $H$.

The prior is a beta distribution with the following probability density function:

\begin{align}
P(H) &= \frac{p^{\alpha-1} (1-p)^{\beta-1}}{B(\alpha,\beta)}
\end{align}

where $\alpha$ and $\beta$ are hyperparameters correlated with number of successes and failures respectively, and $B(\alpha, \beta)$ is the beta function to normalize the distribution to be between 0 and 1. Intuitively, the beta distribution is shaped by the number of successes and failures initialized by $\alpha$ and $\beta$; moreover, higher number means we are more certain about our distribution thus resulting in less variance:

```python
beta_df = pd.DataFrame({'x': [i/100 for i in range(100)],
                        '1_1': [sp.stats.beta.pdf(i/100, a=1,b=1) for i in range(100)],
                        '2_3': [sp.stats.beta.pdf(i/100, a=2,b=3) for i in range(100)],
                        '4_6': [sp.stats.beta.pdf(i/100, a=4,b=6) for i in range(100)],
                        '20_30': [sp.stats.beta.pdf(i/100, a=20,b=30) for i in range(100)],
                       }).melt(id_vars='x')
g = (ggplot(beta_df,aes(x='x',y='value',color='variable')) +
     geom_line() + theme_minimal() +
     xlab('Values') + ylab('Probability Density')
    )
g
```

Here we notice that the evidence $P(D)$ and the normalizing factor $B(\alpha, \beta)$ are constants in regards to our hypothesis so we can bypass them and consider that our posterior distribution is proportional to:

\begin{align}
P(H|D) &\propto \left( p^{\sum_{i=1}^{n} x_i} (1-p)^{\sum_{i=1}^{n} (1-x_i)} \right) \left(p^{\alpha-1} (1-p)^{\beta-1} \right) \\
&\propto p^{\alpha + \sum_{i=1}^{n} x_i - 1} (1-p)^{\beta + \sum_{i=1}^{n} (1-x_i) -1}
\end{align}

We can see that our resulting terms are equivalent to an unnormalized beta distribution. Thus, by normalizing it, we can get a beta distribution as our posterior:

\begin{align}
P(H|D) &\propto p^{\alpha + \sum_{i=1}^{n} x_i - 1} (1-p)^{\beta + \sum_{i=1}^{n} (1-x_i) -1} \\
&= \frac{p^{\alpha + \sum_{i=1}^{n} x_i - 1} (1-p)^{\beta + \sum_{i=1}^{n} (1-x_i) -1}}{B(\alpha + \sum_{i=1}^{n} x_i, \beta + \sum_{i=1}^{n} (1-x_i))} \\
&= Beta(\alpha + \text{number of conversions}, \beta + \text{number of non-conversions})
\end{align}

## Choose The Right Prior

We can see that for events that acts like a Bernoulli trial the posterior distribution takes a very convenient form. What we need to do is simply choose a reasonable set of $\alpha$ and $\beta$ for the prior. This is where our assumption factors into a Bayesian A/B test. One way is to set the prior based on past campaign performance; for instance, if we have run this type of campaign and had the average conversion rate of 10%, we can scale the prior according to how certain we are about the conversion rates. Note that different priors can represent 10%, for instance $Beta(1,9)$ and $Beta(10,90)$, but at different level of certainty. See how $Beta(10,90)$ is more pointy around 10% than $Beta(1,9)$.

```python
def plot_beta(a=1,b=9,scaler=1):
    beta_df = pd.DataFrame({'x': [i/100 for i in range(100)],
                           'value': [sp.stats.beta.pdf(i/100, a=a*scaler,b=b*scaler) for i in range(100)]})
    g = (ggplot(beta_df,aes(x='x',y='value')) +
         geom_line() + theme_minimal() +
         xlab('Values') + ylab('Probability Density') +
         ggtitle(f'alpha = {a*scaler}; beta={b*scaler}')
        )
    return g

widgets.interact(plot_beta,
                 a=widgets.IntSlider(min=1,max=100,step=1,value=1),
                 b=widgets.IntSlider(min=1,max=100,step=1,value=9),
                 scaler=widgets.FloatSlider(min=0.1,max=100,step=0.5,value=1))
```

    interactive(children=(IntSlider(value=1, description='a', min=1), IntSlider(value=9, description='b', min=1), …

    <function __main__.plot_beta(a=1, b=9, scaler=1)>

```python
conv_df
```

      campaign_id
      clicks
      conv_cnt
      conv_per

      0
      A
      2488
      234
      0.094051

      1
      B
      2209
      222
      0.100498

Depending on what we choose as our priors for `A` and `B`, we can derive their posterior distributions as follows. Note that the prior has an impact on the certainty about the values of the posterior (pointiness) and the expected values of the posterior themselves. For instance, even if your observed conversion rate is around 10%, if you gives an absurdly strong prior like $Beta(999,1)$, implying 99.9% conversion rate, your posteriors' expectations can be around 30%.

```python
def plot_posterior(clicks, conv_cnt, a = 1, b = 1, scaler=1):
    beta_df = pd.DataFrame({'x': [i/100 for i in range(100)]})
    if not isinstance(a, collections.MutableSequence):
        a = [a for i in range(len(clicks))]
        b = [b for i in range(len(clicks))]
    for v in range(len(clicks)):
        beta_df[f'value_{v}'] = [sp.stats.beta.pdf(i/100, a=a[v]*scaler + conv_cnt[v], \
                                 b=b[v]*scaler + clicks[v] - conv_cnt[v]) for i in range(100)]
    beta_df = beta_df.melt(id_vars='x')
    g = (ggplot(beta_df,aes(x='x',y='value',color='variable',group='variable')) +
         geom_line() + theme_minimal() +
         xlab('Values') + ylab('Probability Density') +
         ggtitle(f'alpha = {[i*scaler for i in a]}; beta={[i*scaler for i in b]}')
        )
    return g

widgets.interact(plot_posterior,
                 clicks=fixed(conv_df.clicks),
                 conv_cnt=fixed(conv_df.conv_cnt),
                 a=widgets.IntSlider(min=1,max=1000,step=1,value=999),
                 b=widgets.IntSlider(min=1,max=1000,step=1,value=1),
                 scaler=widgets.FloatSlider(min=1,max=100,step=1,value=1))
```

    interactive(children=(IntSlider(value=999, description='a', max=1000, min=1), IntSlider(value=1, description='…

    <function __main__.plot_posterior(clicks, conv_cnt, a=1, b=1, scaler=1)>

## Who Wins by How Much

One way to determine which variation is better is to look at the posterior distributions and their expectations; however, that does not tell us the margin of difference. Since we have a posterior distribution for each variation we can derive what people sometimes confuse with the frequentist p-value: **what is the probability that one variation is better than another?** One way to do this is by using Monte Carlo simulation to draw a large number of samples from each posterior then calculate the percentage of times each sample won.

```python
def sample_proportion(c,n,a=1,b=1,sim_size=100000): return np.random.beta(c+a,n-c+b,sim_size)
def proportion_test_b(c1,c2,n1,n2,a1=1,a2=1,b1=9,b2=9,sim_size=100000):
    p1 = sample_proportion(c1,n1,a1,b1,sim_size)
    p2 = sample_proportion(c2,n2,a2,b2,sim_size)
    return (p1 > p2).mean()
def proportion_ratio(c1,c2,n1,n2,a1=1,a2=1,b1=9,b2=9,sim_size=100000):
    p1 = sample_proportion(c1,n1,a1,b1,sim_size)
    p2 = sample_proportion(c2,n2,a2,b2,sim_size)
    return p1/p2
def proportion_ci_b(c1,c2,n1,n2,p_value=0.05,a1=1,a2=1,b1=9,b2=9,sim_size=100000):
    ratios = proportion_ratio(c1,c2,n1,n2,a1,a2,b1,b2,sim_size)
    return np.quantile(ratios,[p_value/2,1-p_value/2])

p_value= proportion_test_b(*conv_df.conv_cnt,*conv_df.clicks)
ratios = proportion_ratio(*conv_df.conv_cnt,*conv_df.clicks)
credible = proportion_ci_b(*conv_df.conv_cnt,*conv_df.clicks,p_value=0.05)
print(f'Probability that A is greater than B: {p_value}')
print(f'Average A/B ratio: {ratios.mean()}')
print(f'Credible interval of A/B ratio: {credible}')
```

    Probability that A is greater than B: 0.22743
    Average A/B ratio: 0.9394908467261973
    Credible interval of A/B ratio: [0.78622231 1.11331594]

Plotting the ratio of A over B also gives us a look into the magnitude of the difference. For instace, below we can see that for about 80% of the ratio distribution `A` is worse than `B`, and in case that we were wrong `A` is at most about 20% better than `B`. And similar to the frequentist approach, we can calculate the range where most of these ratios (say 95%) falls into, called **credible interval**.

```python
g = (ggplot(pd.DataFrame({'x':ratios}), aes(x='x')) + geom_histogram() +
    theme_minimal() + geom_vline(xintercept=1,color='red') +
    geom_vline(xintercept=credible[0],color='green') + geom_vline(xintercept=credible[1],color='green') +
    xlab('Ratio of sampled conversion rates A / B')
    )
g
```

## When to Stop: Value Remaining

The freedom from frequentist p-values allows us to not run into the issue that infinite number of samples *always* yield statistical significance even when the true values are exactly the same. As we can see from the plot below, the probility of `A` beating `B` as a result of Monte Carlo sampling from the posteriors stay about 50% when there is no true difference and gradually decreases when there is.

```python
conv_days2 = gen_campaigns(p1 = 0.10,
                          p2 = 0.10,
                          nb_days = 60,
                          scaler=300,
                          seed = 1412) #god-mode
conv_days2['prob_same'] = conv_days2.apply(lambda row: proportion_test_b(row['cumu_conv_a'],
                                                                   row['cumu_conv_b'],row['cumu_click_a'],
                                                                   row['cumu_click_b']),1)
conv_days3 = gen_campaigns(p1 = 0.10,
                          p2 = 0.11,
                          nb_days = 60,
                          scaler=300,
                          seed = 1412) #god-mode

conv_days3['prob_diff'] = conv_days3.apply(lambda row: proportion_test_b(row['cumu_conv_a'],
                                                                   row['cumu_conv_b'],row['cumu_click_a'],
                                                                   row['cumu_click_b']),1)
prob_df = pd.DataFrame({'click_day':conv_days2.click_day,'prob_same':conv_days2.prob_same,
                        'prob_diff':conv_days3.prob_diff}).melt(id_vars='click_day')
g = (ggplot(prob_df,aes(x='click_day',y='value',color='variable')) +
    geom_line() + theme_minimal() +
    geom_hline(yintercept=[0.9,0.1],color=['green','red']) +
    xlab('Number of timesteps') + ylab('Probability of A beating B') +
    scale_y_continuous(labels=formatters.percent_format()) +
    annotate("text", label = "Above this line A is better than B", x = 20, y = 1, color = 'green') +
    annotate("text", label = "Below this line B is better than A", x = 20, y = 0, color = 'red') +
    ggtitle('Comparison between probabilities of A beating B when B is actually better and when both are the same')
    )
g
```

The immediate question then comes up: when should we stop and declare a winner. Technically when there is a true difference, we can stop at any point in time and we will end up choosing the winner at varying degrees of probability (this is a clear improvement from the frequentist approach where we can only say we are not sure most of the time). We can also use a simple rule like if the probability of `A` beating `B` is lower than 10% or greater than 90%, we declare a winner like the plot above.

Stopping criteria for experiments such as ROPE and expected loss as described in [Bayesian A/B Testing: a step-by-step guide](http://www.claudiobellei.com/2017/11/02/bayesian-AB-testing/); here we are using **value remaining** as introduced by [Google](https://support.google.com/analytics/answer/2846882?hl=en). Value remaining per round of experiment is defined as:

$$V_t = \frac{rate_{max}-rate_{opt}}{rate_{opt}}$$

As experiment goes on, we plot the distribution of $V_t$ and stops when the $1-\alpha$ percentile is lower than our threshold. Intuitively, this is to say that we are $1-\alpha$% confident that our "best" arm might be beaten by the margin equals to the threshold. For practical purpose, we try 95th percentile and threshold of 1%.

```python
def value_remaining(c1,c2,n1,n2,q=95,sim_size=100000,a1=1,a2=1,b1=9,b2=9):
    p1 = sample_proportion(c1,n1,a1,b1,sim_size)[:,None]
    p2 = sample_proportion(c2,n2,a2,b2,sim_size)[:,None]
    p = np.concatenate([p1,p2],1)
    p_max = p.max(1)
    best_idx = np.argmax([p1.mean(),p2.mean()])
    p_best = p[:,best_idx]
    vs = (p_max-p_best)/p_best
    return np.percentile(vs,q)

value_remaining(*conv_df.conv_cnt,*conv_df.clicks)
```

    0.0835620586721222

You can see that in the case where the true difference is 1%, value remaining gradually decreases below our 1% threshold. On the other hand, when the true difference is 0%, value remaining always hover around 10%.

```python
conv_days2['value_remaining'] = conv_days2.apply(lambda row: value_remaining(row['cumu_conv_a'],
                                                                   row['cumu_conv_b'],row['cumu_click_a'],
                                                                   row['cumu_click_b']),1)
conv_days3['value_remaining'] = conv_days3.apply(lambda row: value_remaining(row['cumu_conv_a'],
                                                                   row['cumu_conv_b'],row['cumu_click_a'],
                                                                   row['cumu_click_b']),1)
value_df = pd.DataFrame({'click_day':conv_days2.click_day,'value_same':conv_days2.value_remaining,
                        'value_diff':conv_days3.value_remaining}).melt(id_vars='click_day')
g = (ggplot(value_df,aes(x='click_day',y='value',color='variable')) +
    geom_line() + theme_minimal() +
    geom_hline(yintercept=0.01, color='red') +
    scale_y_continuous(labels=formatters.percent_format(), breaks=[i/100 for i in range(0,101,10)]) +
    xlab('Number of timesteps') + ylab('Value Remaining')
    )
g
```

## References

* [An Introduction to Bayesian Thinking: A Companion to the Statistics with R Course](https://statswithr.github.io/book/)
* [Formulas for Bayesian A/B Testing](https://www.evanmiller.org/bayesian-ab-testing.html)
* [20 - Beta conjugate prior to Binomial and Bernoulli likelihoods](https://youtu.be/hKYvZF9wXkk)
* [Bayesian A/B Testing: A Hypothesis Test that Makes Sense](https://www.countbayesie.com/blog/2015/4/25/bayesian-ab-testing)
* [What's the difference between a confidence interval and a credible interval?](https://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval)
* [Is Bayesian A/B Testing Immune to Peeking? Not Exactly](http://varianceexplained.org/r/bayesian-ab-testing/)


# Quality metrics for classification

We will consider a range of classification quality metrics solving a classification problem with a neural network.

```python
%matplotlib inline
import pandas as pd
import numpy as np
import numpy.testing as np_testing
import matplotlib.pyplot as plt
```

# Load MAGIC Data Set

Source: https://magic.mpp.mpg.de/

```python
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/magic/magic04.data
```

Features description:
- **Length:** continuous # major axis of ellipse [mm]
- **Width:** continuous # minor axis of ellipse [mm]
- **Size:** continuous # 10-log of sum of content of all pixels [in #phot]
- **Conc:** continuous # ratio of sum of two highest pixels over fSize [ratio]
- **Conc1:** continuous # ratio of highest pixel over fSize [ratio]
- **Asym:** continuous # distance from highest pixel to center, projected onto major axis [mm]
- **M3Long:** continuous # 3rd root of third moment along major axis [mm]
- **M3Trans:** continuous # 3rd root of third moment along minor axis [mm]
- **Alpha:** continuous # angle of major axis with vector to origin [deg]
- **Dist:** continuous # distance from origin to center of ellipse [mm]
- **Label:** g,h # gamma (signal), hadron (background)

g = gamma (signal): 12332 \
h = hadron (background): 6688

```python
f_names = np.array(["Length", "Width", "Size", "Conc", "Conc1", "Asym", "M3Long", "M3Trans", "Alpha", "Dist"])

data = pd.read_csv("magic04.data", header=None, names=list(f_names)+["Label"])
data.head()
```

# Data preparation

```python
# prepare a matrix of input features
X = data[f_names].values

# prepare a vector of true labels
y = 1 * (data['Label'].values == "g")
```

```python
# print sizes of X and y
X.shape, y.shape
```

```python
X[:2]
```

```python
y[:5]
```

# Train / test split

```python
from sklearn.model_selection import train_test_split

# Split data into train and test samples
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=42, stratify=y)
```

# Preprocessing

Scale input data using StandardScaler:
$$
X_{new} = \frac{X - \mu}{\sigma}
$$

```python
# Import StandardScaler
from sklearn.preprocessing import StandardScaler

# Create object of the class and set up its parameters
ss = StandardScaler()

# Estimate mean and sigma values
ss.fit(X_train)

# Scale train and test samples
X_train = ss.transform(X_train)
X_test = ss.transform(X_test)
```

# Fit a classifier

Now let's create a neural network and fit it.

```python
#!pip install pytorch-lightning
```

```python
import torch
from torch.nn import functional as F
from torch import nn
import pytorch_lightning as pl

class Model(pl.LightningModule):

    def __init__(self):
        super().__init__()

        # define all layers of the netwrok
        self.net = nn.Sequential(
                                nn.Linear(10, 10),
                                nn.Tanh(),
                                nn.Linear(10, 1),
                                nn.Sigmoid())

    def forward(self, x):
        # make a prediction for x
        return self.net(x)

    # calculate loss function values
    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self(x)
        loss = F.binary_cross_entropy(y_hat, y)
        return loss

    # define optimizer to fit the network
    def configure_optimizers(self):
        return torch.optim.Adam(self.parameters(), lr=0.02)
```

```python
from torch.utils.data import TensorDataset, DataLoader

# combine X and y into one pytorch tensor dataset
dataset_train = TensorDataset(torch.tensor(X_train, dtype=torch.float),
                              torch.tensor(y_train.reshape(-1, 1), dtype=torch.float))

# loader divides our train data into batches
train_loader = DataLoader(dataset_train, batch_size=128)
```

```python
# define trainer to fit out network
trainer = pl.Trainer(max_epochs=10)

# init our netwrok
model = Model()

# fit the netwrok
trainer.fit(model, train_loader)
```

# Make Predictions

Make prediction of **probability** of the positive class.

```python
# make predictions
y_proba_test = model(torch.tensor(X_test, dtype=torch.float))[:, 0].detach().numpy()
```

```python
print("Truth  : ", y_test[:10])
print("Proba  : ", y_proba_test[:10])
```

Make prediction of class **label**.

```python
# transform the predicted probabilities into predicted labels {0, 1}
y_pred_test = 1 * (y_proba_test > 0.5)
```

```python
print("Truth  : ", y_test[:10])
print("Pred   : ", y_pred_test[:10])
```

# Label-based Quality Metrics

Consider a confusion matrix:

* TP (true positive) - currectly predicted positives
* FP (false positive) - incorrectly predicted negatives (1st order error)
* FN (false negative) - incorrectly predicted positives (2nd order error)
* TN (true negative) - currectly predicted negatives
* Pos (Neg) - total number of positives (negatives)

Quality metrics:

* $ \text{Accuracy} = \frac{TP + TN}{Pos+Neg}$
* $ \text{Error rate} = 1 -\text{accuracy}$
* $ \text{Precision} =\frac{TP}{TP + FP}$
* $ \text{Recall} =\frac{TP}{TP + FN} = \frac{TP}{Pos}$
* $ \text{F}_\beta \text{-score} = (1 + \beta^2) \cdot \frac{\mathrm{precision} \cdot \mathrm{recall}}{(\beta^2 \cdot \mathrm{precision}) + \mathrm{recall}}$

### Task 1
Complete a function that computes TP, FP, TN, FN, Accuracy, Error rate, Precision, Recall and F1-score metrics for a classifier.

**Hint:** use implementation of the metrics from `sklearn.metrics` as it is shown below. Example for confusin matrix: https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix

```python
from sklearn.metrics import confusion_matrix, accuracy_score, precision_score, recall_score, f1_score

def quality_metrics_report(y_true, y_pred):
    """
    Parameters
    ----------
    y_true: array-like of shape (n_samples,)
        Ground truth (correct) target values.
    y_pred: array-like of shape (n_samples,)
        Estimated targets as returned by a classifier.

    Returns
    -------
    List of metric values: [tp, fp, fn, tn, accuracy, error_rate, precision, recall, f1]
    """

    ### BEGIN SOLUTION

    ### END SOLUTION

    return [tp, fp, fn, tn, accuracy, error_rate, precision, recall, f1]
```

```python
quality_metrics_report([0, 1, 0, 1], [1, 1, 1, 0])
```

Expected output:

```python
[1, 2, 1, 0, 0.25, 0.75, 0.3333333333333333, 0.5, 0.4]

```

```python
### BEGIN HIDDEN TESTS
actual  = quality_metrics_report([0, 1, 0, 1], [1, 1, 1, 0])
desired = [1, 2, 1, 0, 0.25, 0.75, 0.3333333333333333, 0.5, 0.4]
np_testing.assert_almost_equal(actual, desired, decimal=1)
### END HIDDEN TESTS
```

Now let's compute all these quality metrics for all classifiers considered above.

```python
metrics_report = pd.DataFrame(columns=['TP', 'FP', 'FN', 'TN', 'Accuracy', 'Error rate', 'Precision', 'Recall', 'F1'])

metrics_report.loc['Model', :] = quality_metrics_report(y_test, y_pred_test)

metrics_report
```

## Probability-based Quality Metrics

### ROC curve

The receiver operating characteristic curve (ROC) measures how well a classifier separates two classes.

Let $y_{\rm i}$ is a true label and $\hat{y}_{\rm i}$ is a predicted score for the $i^{\rm th}$ observation.

The numbers of positive and negative observations: $\mathcal{I}_{\rm 1} = \{i: y_{\rm i}=1\}$ and $\mathcal{I}_{\rm 0} = \{i: y_{\rm i}=0\}$.

The sum of observation weights for each class: $W_{\rm 1} = \sum_{i \in \mathcal{I}_{\rm 1}} w_{\rm i}$ and  $W_{\rm 0} = \sum_{i \in \mathcal{I}_{\rm 0}} w_{\rm i}$.

For each predicted score threshold value $\tau$, True Positive Rate (TPR) and False Positive Rate (FPR) are calculated:

\begin{equation}
TPR(\tau) = \frac{1}{W_{\rm 1}} \sum_{i \in \mathcal{I}_{\rm 1}} I[\hat{y}_{\rm i} \ge \tau] w_{\rm i}
\end{equation}

\begin{equation}
FPR(\tau) = \frac{1}{W_{\rm 0}} \sum_{i \in \mathcal{I}_{\rm 0}} I[\hat{y}_{\rm i} \ge \tau] w_{\rm i}
\end{equation}

### Task 2
Complete the fucntion below, that computes a ROC curve and ROC AUC for a classifier.

**Hint:** use `roc_curve` and `auc` from `from sklearn.metrics`.

```python
from sklearn.metrics import roc_curve, auc

def roc_curve_report(y_true, y_proba):
    """
    Parameters
    ----------
    y_true: array-like of shape (n_samples,)
        Ground truth (correct) target values.
    y_proba: array-like of shape (n_samples,)
        Predicted probabilities of the positive class predicted by a classifier.

    Returns
    -------
    fpr : array, shape = [>2]
        Increasing false positive rates such that element i is the false
        positive rate of predictions with score >= thresholds[i].
    tpr : array, shape = [>2]
        Increasing true positive rates such that element i is the true
        positive rate of predictions with score >= thresholds[i].
    roc_auc : float
        Area under the ROC curve defined by the fpr and tpr.
    """

    ### BEGIN SOLUTION

    ### END SOLUTION

    return fpr, tpr, roc_auc
```

```python
roc_curve_report([0, 1, 0, 1], [0.6, 0.9, 0.1, 0.4])
```

Expected output:

```python
(array([0. , 0. , 0.5, 0.5, 1. ]), array([0. , 0.5, 0.5, 1. , 1. ]), 0.75)

```

```python
### BEGIN HIDDEN TESTS
actual  = roc_curve_report([0, 1, 0, 1], [0.6, 0.9, 0.1, 0.4])[0]
desired = np.array([0. , 0. , 0.5, 0.5, 1. ])
np_testing.assert_almost_equal(actual, desired, decimal=1)
### END HIDDEN TESTS
```

Now let's plot ROC curves for all classifiers considered above.

```python
from sklearn.metrics import roc_curve, auc

fpr, tpr, roc_auc = roc_curve_report(y_test, y_proba_test)
```

```python
plt.figure(figsize=(9, 6))
plt.plot(fpr, tpr, linewidth=3, label='Model')

plt.xlabel('FPR', size=18)
plt.ylabel('TPR', size=18)

plt.legend(loc='best', fontsize=14)
plt.grid(b=1)
plt.show()

print('ROC AUC:', roc_auc)
```

```python

```