# Modeling data 2

## Building a model

Recall that in notebook 3, we saw that we could use a mathematical function to classify an image as an apple or a banana, based on the average amount of green in an image:

A common function for performing this kind of **classification** is the sigmoid that we saw in the last notebook, and that we will now extend by adding two **parameters**, $w$ and $b$:

$$\sigma(x; w, b) := \frac{1}{1 + \exp(-wx + b)}$$

$$ x = \mathrm{data} $$

\begin{align}
\sigma(x;w,b) &\approx 0 \implies \mathrm{apple} \\
\sigma(x;w,b) &\approx 1 \implies \mathrm{banana}
\end{align}

In our mathematical notation above, the `;` in the function differentiates between the **data** and the **parameters**. `x` is the data and is determined from the image. The parameters, `w` and `b`, are numbers which we choose to make our function match the results it should be modeling.

Note that in the code below, we don't distinguish between data and parameters - both are just inputs to our function, σ!

```julia
using Images

apple = load("d
    ata/10_100.jpg")
banana = load("data/104_100.jpg")

apple_green_amount = mean(Float64.(green.(apple)))
banana_green_amount = mean(Float64.(green.(banana)))

"Average green for apple = $apple_green_amount; " *
"Average green for banana = $banana_green_amount; "
```

```julia
σ(x, w, b) = 1 / (1 + exp(-w * x + b))
```

What we want is that when we give σ as input the average green for the apple, roughly `x = 0.3385`, it should return as output something close to 0, meaning "apple". And when we give σ the input `x = 0.8808`, it should output something close to 1, meaning "banana".

By changing the parameters of the function, we can change the shape of the function, and hence make it represent, or **fit**, the data better!

## Data fitting by varying parameters

We can understand how our choice of `w` and `b` affects our model by seeing how our values for `w` and `b` change the plot of the $\sigma$ function.

To do so, we will use the `Interact.jl` Julia package, which provides "widgets" for controlling parameters interactively via sliders:

```julia
using Plots; gr()   # GR works better for interactive manipulations
using Interact      # package for interactive manipulation
```

Run the code in the next cell. You should see two "sliders" appear, one for `w` and one for `b`.

**Game**:
Move both of those sliders around until the blue curve, labeled "model", which is the graph of the `\sigma` function, passes through *both* of the data points at the same time.

```julia
@manipulate for w in -10:0.01:30, b in 0:0.1:20

    plot(x -> σ(x, w, b), xlim=(-0,1), ylim=(-0.1,1.1), label="model", legend=:topleft, lw=3)

    scatter!([apple_green_amount],  [0.0], label="apple", ms=5)   # marker size = 5
    scatter!([banana_green_amount], [1.0], label="banana", ms=5)

end
```

Notice that the two parameters do two very different things. The **weight**, `w`, determines *how fast* the transition between 0 and 1 occurs. It encodes how trustworthy we think our data  actually is, and in what range we should be putting points between 0 and 1 and thus calling them "unsure". The **bias**, `b`, encodes *where* on the $x$-axis the switch should take place. It can be seen as shifting the function left-right. We'll come to understand these *parameters* more in notebook 6.

Here are some parameter choices that work well:

```julia
w = 25.58; b = 15.6

plot(x -> σ(x, w, b), xlim=(0,1), ylim=(-0.1,1.1), label="model", legend=:topleft, lw=3)

scatter!([apple_green_amount], [0.0], label="apple")
scatter!([banana_green_amount],[1.0], label="banana")
```

(Note that in this problem there are many combinations of `w` and `b` that fit the data well.)

Once we have a model, we have a computational representation for how to choose between "apple" and "banana". So let's pull in some new images and see what our model says about them!

```julia
apple2 = load("data/107_100.jpg")
```

```julia
green_amount = mean(Float64.(green.(apple2)))
@show green_amount

scatter!([green_amount], [0.0], label="new apple")
```

Our model successfully says that our new image is an apple! Pat yourself on the back: you've actually just trained your first neural network!

#### Exercise 1

Load the image of a banana in `data/8_100.jpg` as `mybanana`. Edit the code below to calculate the amount of green in `mybanana` and to overlay data for this image with the existing model and data points.

# To get the desired overlay, the code we need is

```julia
mybanana = load("data/8_100.jpg")
mybanana_green_amount = mean(Float64.(green.(banana)))
scatter!([mybanana_green_amount], [1.0], label="my banana")
```

## Closing remarks: bigger models, more data, more accuracy

That last apple should start making you think: not all apples are red; some are yellow. "Redness" is one attribute of being an apple, but isn't the whole thing. What we need to do is incorporate more ideas into our model by allowing more inputs. However, more inputs would mean more parameters to play with. Also, we would like to have the computer start "learning" on its own, instead of modifying the parameters ourselves until we think it "looks right". How do we take the next step?

The first thing to think about is, if you wanted to incorporate more data into the model, how would you change the sigmoid function? Play around with some ideas. But also, start thinking about how you chose parameters. What process did you do to finally end up at good parameters? These two problems (working with models with more data and automatically choosing parameters) are the last remaining step to understanding deep learning.

introductory-tutorials/broader-topics-and-ecosystem/intro-to-ml/05. ML - Building models.ipynb


# ML 101

## 2. Linear Algebra

The following system of equations:

$\begin{split}
4 x_1 - 5 x_2 & = -13 \\
-2x_1 + 3 x_2 & = 9
\end{split}$

We are looking for a unique solution for the two variables $x_1$ and $x_2$.  The system can be described as:

$$Ax = b$$

as matrices:

$A = \begin{bmatrix}
       4  & -5 \\[0.3em]
       -2 &  3
     \end{bmatrix},\
 b = \begin{bmatrix}
       -13 \\[0.3em]
       9
     \end{bmatrix}$

```python
import numpy as np

a = np.array([[4, -5], [-2, 3]])
b = np.array([-13, 9])
x = np.linalg.solve(a, b)
print(f'{x}')
```

A **scalar** is an element in a vector, containing a real number **value**. In a vector space model or a vector mapping of (symbolic, qualitative, or quantitative) properties the scalar holds the concrete value or property of a variable.

A **vector** is an array, tuple, or ordered list of scalars (or elements) of size $n$, with $n$ a positive integer. The **length** of the vector, that is the number of scalars in the vector, is also called the **order** of the vector.

**Vectorization** is the process of creating a vector from some data using some process.

Vectors of the length $n$ could be treated like points in $n$-dimensional space. One can calculate the distance between such points using measures like [Euclidean Distance](https://en.wikipedia.org/wiki/Euclidean_distance). The similarity of vectors could also be calculated using [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity).

```python
np.show_config()
```

## Notation

A **matrix** is a list of vectors that all are of the same length. $A$ is a matrix with $m$ rows and $n$ columns, entries of $A$ are real numbers:

$$A \in \mathbb{R}^{m \times n}$$

A vector $x$ with $n$ entries of real numbers, could also be thought of as a matrix with $n$ rows and $1$ column, or as known as a **column vector**.

$x = \begin{bmatrix}
       x_1 \\[0.3em]
       x_2 \\[0.3em]
       \vdots \\[0.3em]
       x_n
     \end{bmatrix}$

Representing a **row vector**, that is a matrix with $1$ row and $n$ columns, we write $x^T$ (this denotes the transpose of $x$, see above).

$x^T = \begin{bmatrix}
       x_1 & x_2 & \cdots & x_n
     \end{bmatrix}$

We use the notation $a_{ij}$ (or $A_{ij}$, $A_{i,j}$, etc.) to denote the entry of $A$ in the $i$th row and
$j$th column:

$A = \begin{bmatrix}
       a_{11} & a_{12} & \cdots & a_{1n} \\[0.3em]
       a_{21} & a_{22} & \cdots & a_{2n} \\[0.3em]
       \vdots & \vdots & \ddots & \vdots \\[0.3em]
       a_{m1} & a_{m2} & \cdots & a_{mn}
     \end{bmatrix}$

We denote the $j^{th}$ column of $A$ by $a_j$ or $A_{:,j}$:

$A = \begin{bmatrix}
       \big| & \big| &  & \big| \\[0.3em]
       a_{1} & a_{2} & \cdots & a_{n} \\[0.3em]
       \big| & \big| &  & \big|
     \end{bmatrix}$

We denote the $i^{th}$ row of $A$ by $a_i^T$ or $A_{i,:}$:

$A = \begin{bmatrix}
      -- & a_1^T  & -- \\[0.3em]
       -- & a_2^T  & -- \\[0.3em]
          & \vdots &  \\[0.3em]
       -- & a_m^T  & --
     \end{bmatrix}$

A $n \times m$ matrix is a two-dimensional array with $n$ rows and $m$ columns.

## Matrix Multiplication

The result of the multiplication of two matrixes $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times p}$ is the matrix:

$$C = AB \in \mathbb{R}^{m \times n}$$

That is, we are multiplying the columns of $A$ with the rows of $B$:

$$C_{ij}=\sum_{k=1}^n{A_{ij}B_{kj}}$$

The number of columns in $A$ must be equal to the number of rows in $B$.

### Vector-Vector Products

#### Inner or Dot Product of Two Vectors

For two vectors $x, y \in \mathbb{R}^n$, the **inner product** or **dot product** $x^T y$ is a real number:

$x^T y \in \mathbb{R} = \begin{bmatrix}
       x_1 & x_2 & \cdots & x_n
     \end{bmatrix} \begin{bmatrix}
       y_1 \\[0.3em]
       y_2 \\[0.3em]
       \vdots \\[0.3em]
       y_n
     \end{bmatrix} = \sum_{i=1}^{n}{x_i y_i}$

The **inner products** are a special case of matrix multiplication.

It is always the case that $x^T y = y^T x$.

##### Example

To calculate the inner product of two vectors $x = [1 2 3 4]$ and $y = [5 6 7 8]$, we can loop through the vector and multiply and sum the scalars (this is simplified code):

```python
x = (1, 2, 3, 4)
y = (5, 6, 7, 8)
n = len(x)
if n == len(y):
    result = 0
    for i in range(n):
        result += x[i] * y[i]
    print(result)
```

It is clear that in the code above we could change line 7 to `result += y[i] * x[i]` without affecting the result.

We can use the *numpy* module to apply the same operation, to calculate the **inner product**. We import the *numpy* module and assign it a name *np* for the following code:

```python
import numpy as np
```

We define the vectors $x$ and $y$ using *numpy*:

```python
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
print("x:", x)
print("y:", y)
```

We can now calculate the $dot$ or $inner product$ using the *dot* function of *numpy*:

```python
np.dot(x, y)
```

The order of the arguments is irrelevant:

```python
np.dot(y, x)
```

Note that both vectors are actually **row vectors** in the above code. We can transpose them to column vectors by using the *shape* property:

```python
print("x:", x)
x.shape = (4, 1)
print("xT:", x)
print("y:", y)
y.shape = (4, 1)
print("yT:", y)
```

In fact, in our understanding of Linear Algebra, we take the arrays above to represent **row vectors**. *Numpy* treates them differently.

We see the issues when we try to transform the array objects. Usually, we can transform a row vector into a column vector in *numpy* by using the *T* method on vector or matrix objects:

```python
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
print("x:", x)
print("y:", y)
print("xT:", x.T)
print("yT:", y.T)
```

The problem here is that this does not do, what we expect it to do. It only works, if we declare the variables not to be arrays of numbers, but in fact a matrix:

```python
x = np.array([[1, 2, 3, 4]])
y = np.array([[5, 6, 7, 8]])
print("x:", x)
print("y:", y)
print("xT:", x.T)
print("yT:", y.T)

```

Note that the *numpy* functions *dot* and *outer* are not affected by this distinction. We can compute the dot product using the mathematical equation above in *numpy* using the new $x$ and $y$ row vectors:

```python
print("x:", x)
print("y:", y.T)
np.dot(x, y.T)
```

Or by reverting to:

```python
print("x:", x.T)
print("y:", y)
np.dot(y, x.T)
```

To read the result from this array of arrays, we would need to access the value this way:

```python
np.dot(y, x.T)[0][0]
```

#### Outer Product of Two Vectors

For two vectors $x \in \mathbb{R}^m$ and $y \in \mathbb{R}^n$, where $n$ and $m$ do not have to be equal, the **outer product** of $x$ and $y$ is:

$xy^T \in \mathbb{R}^{m\times n}$

The **outer product** results in a matrix with $m$ rows and $n$ columns by $(xy^T)_{ij} = x_i y_j$:

$xy^T \in \mathbb{R}^{m\times n} = \begin{bmatrix}
       x_1 \\[0.3em]
       x_2 \\[0.3em]
       \vdots \\[0.3em]
       x_n
     \end{bmatrix} \begin{bmatrix}
       y_1 & y_2 & \cdots & y_n
     \end{bmatrix} = \begin{bmatrix}
       x_1 y_1 & x_1 y_2 & \cdots & x_1 y_n \\[0.3em]
       x_2 y_1 & x_2 y_2 & \cdots & x_2 y_n \\[0.3em]
       \vdots  & \vdots  & \ddots & \vdots \\[0.3em]
       x_m y_1 & x_m y_2 & \cdots & x_m y_n \\[0.3em]
     \end{bmatrix}$

Some useful property of the outer product: assume $\mathbf{1} \in \mathbb{R}^n$ is an $n$-dimensional vector of scalars with the value $1$. Given a matrix $A \in \mathbb{R}^{m\times n}$ with all columns equal to some vector $x \in \mathbb{R}^m$, using the outer product $A$ can be represented as:

$A = \begin{bmatrix}
       \big| & \big| &  & \big| \\[0.3em]
       x & x & \cdots & x \\[0.3em]
       \big| & \big| &  & \big|
     \end{bmatrix} = \begin{bmatrix}
       x_1    & x_1    & \cdots & x_1    \\[0.3em]
       x_2    & x_2    & \cdots & x_2    \\[0.3em]
       \vdots & \vdots & \ddots & \vdots \\[0.3em]
       x_m    &x_m     & \cdots & x_m
     \end{bmatrix} = \begin{bmatrix}
       x_1 \\[0.3em]
       x_2 \\[0.3em]
       \vdots \\[0.3em]
       x_m
     \end{bmatrix} \begin{bmatrix}
       1 & 1 & \cdots & 1
     \end{bmatrix} = x \mathbf{1}^T$

##### Example

If we want to compute the outer product of two vectors $x$ and $y$, we need to transpose the row vector $x$ to a column vector $x^T$. This can be achieved by the *reshape* function in *numpy*, the *T* method, or the *transpose()* function. The *reshape* function takes a parameter that describes the number of colums and rows for the resulting transposing:

```python
x = np.array([[1, 2, 3, 4]])
print("x:", x)
print("xT:", np.reshape(x, (4, 1)))
print("xT:", x.T)
print("xT:", x.transpose())
```

We can now compute the **outer product** by multiplying the column vector $x$ with the row vector $y$:

```python
x = np.array([[1, 2, 3, 4]])
y = np.array([[5, 6, 7, 8]])
x.T * y
```

*Numpy* provides an *outer* function that does all that:

```python
np.outer(x, y)
```

Note, in this simple case using the simple arrays for the data structures of the vectors does not affect the result of the *outer* function:

```python
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
np.outer(x, y)
```

### Matrix-Vector Products

Assume a matrix $A \in \mathbb{R}^{m\times n}$ and a vector $x \in \mathbb{R}^n$ the product results in a vector $y = Ax \in \mathbb{R}^m$.

$Ax$ could be expressed as the dot product of row $i$ of matrix $A$ with the column value $j$ of vector $x$. Let us first consider matrix multiplication with a scalar:

$A = \begin{bmatrix}
       1 & 2 \\[0.3em]
       3 & 4
     \end{bmatrix}$

We can compute the product of $A$ with a scalar $n = 2$ as:

$A = \begin{bmatrix}
       1 \times n & 2 \times n \\[0.3em]
       3 \times n & 4 \times n
     \end{bmatrix} = \begin{bmatrix}
       1 \times 2 & 2 \times 2 \\[0.3em]
       3 \times 2 & 4 \times 2
     \end{bmatrix} = \begin{bmatrix}
       2 & 4 \\[0.3em]
       6 & 8
     \end{bmatrix} $

Using *numpy* this can be achieved by:

```python
import numpy as np
A = np.array([[4, 5, 6],
             [7, 8, 9]])
A * 2
```

Assume that we have a column vector $x$:

$x = \begin{bmatrix}
       1 \\[0.3em]
       2 \\[0.3em]
       3
     \end{bmatrix}$

To be able to multiply this vector with a matrix, the number of columns in the matrix must correspond to the number of rows in the column vector. The matrix $A$ must have $3$ columns, as for example:

$A = \begin{bmatrix}
       4 & 5 & 6\\[0.3em]
       7 & 8 & 9
     \end{bmatrix}$

To compute $Ax$, we multiply row $1$ of the matrix with column $1$ of $x$:

$$
\begin{bmatrix}
  4 & 5 & 6
\end{bmatrix}
\begin{bmatrix}
 1 \\[0.3em]
 2 \\[0.3em]
 3
\end{bmatrix} = 4 \times 1 + 5 \times 2 + 6 \times 3 = 32
$$

We do the compute the dot product of row $2$ of $A$ and column $1$ of $x$:

$$ \begin{bmatrix}
  7 & 8 & 9
 \end{bmatrix}
 \begin{bmatrix}
 1 \\[0.3em]
 2 \\[0.3em]
 3
\end{bmatrix} = 7 \times 1 + 8 \times 2 + 9 \times 3 = 50 $$

The resulting column vector $Ax$ is:

$Ax = \begin{bmatrix}
       32 \\[0.3em]
       50
     \end{bmatrix}$

Using *numpy* we can compute $Ax$:

```python
A = np.array([[4, 5, 6],
             [7, 8, 9]])
x = np.array([1, 2, 3])
A.dot(x)
```

We can thus describe the product writing $A$ by rows as:

$y = Ax = \begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix} x = \begin{bmatrix}
 a_1^T x \\[0.3em]
 a_2^T x \\[0.3em]
 \vdots \\[0.3em]
 a_m^T x
\end{bmatrix}$

This means that the $i$th scalar of $y$ is the inner product of the $i$th row of $A$ and $x$, that is $y_i = a_i^T x$.

If we write $A$ in column form, then:

$$ y = Ax =
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 a_1 & a_2 & \cdots & a_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix}
\begin{bmatrix}
 x_1 \\[0.3em]
 x_2 \\[0.3em]
 \vdots \\[0.3em]
 x_n
\end{bmatrix} =
\begin{bmatrix}
 a_1
\end{bmatrix} x_1 +
\begin{bmatrix}
 a_2
\end{bmatrix} x_2 + \dots +
\begin{bmatrix}
 a_n
\end{bmatrix} x_n
$$

In this case $y$ is a **[linear combination](https://en.wikipedia.org/wiki/Linear_combination)** of the *columns* of $A$, the coefficients taken from $x$.

The above examples multiply be the right with a column vector. One can multiply on the left by a row vector as well, $y^T = x^T A$ for $A \in \mathbb{R}^{m\times n}$, $x\in \mathbb{R}^m$, $y \in \mathbb{R}^n$. There are two ways to express $y^T$, with $A$ expressed by its columns, with $i$th scalar of $y^T$ corresponds to the inner product of $x$ and the $i$th column of $A$:

$$
y^T = x^T A = x^t \begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 a_1 & a_2 & \cdots & a_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} =
\begin{bmatrix}
 x^T a_1 & x^T a_2 & \dots & x^T a_n
\end{bmatrix}
$$

One can express $A$ by rows, where $y^T$ is a linear combination of the rows of $A$ with the scalars from $x$.

$$
\begin{equation}
\begin{split}
y^T & = x^T A \\
    & = \begin{bmatrix}
 x_1 & x_2 & \dots & x_n
\end{bmatrix}
\begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix} \\
   & = x_1 \begin{bmatrix}-- & a_1^T  & --\end{bmatrix} + x_2 \begin{bmatrix}-- & a_2^T  & --\end{bmatrix} + \dots + x_n \begin{bmatrix}-- & a_n^T  & --\end{bmatrix}
\end{split}
\end{equation}
$$

### Matrix-Matrix Products

One can view matrix-matrix multiplication $C = AB$ as a set of vector-vector products. The $(i,j)$th entry of $C$ is the inner product of the $i$th row of $A$ and the $j$th column of $B$:

$$
C = AB =
\begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 b_1 & b_2 & \cdots & b_p \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} =
\begin{bmatrix}
 a_1^T b_1 & a_1^T b_2 & \cdots & a_1^T b_p \\[0.3em]
 a_2^T b_1 & a_2^T b_2 & \cdots & a_2^T b_p \\[0.3em]
 \vdots    & \vdots    & \ddots & \vdots    \\[0.3em]
 a_m^T b_1 & a_m^T b_2 & \cdots & a_m^T b_p
\end{bmatrix}
$$

Here $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{n\times p}$, $a_i \in \mathbb{R}^n$ and $b_j \in \mathbb{R}^n$, and $A$ is represented by rows, $B$ by columns.

If we represent $A$ by columns and $B$ by rows, then $AB$ is the sum of the outer products:

$$
C = AB =
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 a_1 & a_2 & \cdots & a_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix}
\begin{bmatrix}
 -- & b_1^T  & -- \\[0.3em]
 -- & b_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & b_n^T  & --
\end{bmatrix}
= \sum_{i=1}^n a_i b_i^T
$$

This means that $AB$ is the sum over all $i$ of the outer product of the $i$th column of $A$ and the $i$th row of $B$.

One can interpret matrix-matrix operations also as a set of matrix-vector products. Representing $B$ by columns, the columns of $C$ are matrix-vector products between $A$ and the columns of $B$:

$$
C = AB = A
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 b_1 & b_2 & \cdots & b_p \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} =
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 A b_1 & A b_2 & \cdots & A b_p \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix}
$$

In this interpretation the $i$th column of $C$ is the matrix-vector product with the vector on the right, i.e. $c_i = A b_i$.

Representing $A$ by rows, the rows of $C$ are the matrix-vector products between the rows of $A$ and $B$:

$$
C = AB = \begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}
B =
\begin{bmatrix}
 -- & a_1^T B & -- \\[0.3em]
 -- & a_2^T B & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_n^T B & --
\end{bmatrix}
$$

The $i$th row of $C$ is the matrix-vector product with the vector on the left, i.e. $c_i^T = a_i^T B$.

#### Notes on Matrix-Matrix Products

**Matrix multiplication is associative:** $(AB)C = A(BC)$

**Matrix multiplication is distributive:** $A(B + C) = AB + AC$

**Matrix multiplication is, in general, not commutative;** It can be the case that $AB \neq BA$. (For example, if $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{n\times q}$, the matrix product $BA$ does not even exist if $m$ and $q$ are not equal!)

## Identity Matrix

The **identity matrix** $I \in \mathbb{R}^{n\times n}$ is a square matrix with the value $1$ on the diagonal and $0$ everywhere else:

$$
I_{ij} = \left\{
\begin{array}{lr}
 1 & i = j\\
 0 & i \neq j
\end{array}
\right.
$$

For all $A \in \mathbb{R}^{m\times n}$:

$AI = A = IA$

In the equation above multiplication has to be made possible, which means that in the portion $AI = A$ the dimensions of $I$ have to be $n\times n$, while in $A = IA$ they have to be $m\times m$.

We can generate an *identity matrix* in *numpy* using:

```python
import numpy as np
A = np.array([[0, 1, 2],
              [3, 4, 5],
              [6, 7, 8],
              [9, 10, 11]])
print("A:", A)
```

We can ask for the shape of $A$:

```python
A.shape
```

The *shape* property of a matrix contains the $m$ (number of rows) and $n$ (number of columns) properties in a tuple, in that particular order. We can create an identity matrix for the use in $AI$ by using the $n$ value:

```python
np.identity(A.shape[1], dtype="int")
```

Note that we specify the *dtype* parameter to *identity* as *int*, since the default would return a matrix of *float* values.

To generate an identity matrix for the use in $IA$ we would use the $m$ value:

```python
np.identity(A.shape[0], dtype="int")
```

We can compute the dot product of $A$ and its identity matrix $I$:

```python
n = A.shape[1]
I = np.array(np.identity(n, dtype="int"))
np.dot(A, I)
```

The same is true for the other direction:

```python
m = A.shape[0]
I = np.array(np.identity(m, dtype="int"))
np.dot(I, A)
```

## Diagonal Matrix

In the **diagonal matrix** non-diagonal elements are $0$, that is $D = diag(d_1, d_2, \dots{}, d_n)$, with:

$$
D_{ij} = \left\{
\begin{array}{lr}
 d_i & i = j\\
 0 & i \neq j
\end{array}
\right.
$$

The identity matrix is a special case of a diagonal matrix: $I = diag(1, 1, \dots{}, 1)$.

In *numpy* we can create a *diagonal matrix* from any given matrix using the *diag* function:

```python
import numpy as np
A = np.array([[0,   1,  2,  3],
              [4,   5,  6,  7],
              [8,   9, 10, 11],
              [12, 13, 14, 15]])
np.diag(A)
```

An optional parameter *k* to the *diag* function allows us to extract the diagonal above the main diagonal with a positive *k*, and below the main diagonal with a negative *k*:

```python
np.diag(A, k=1)
```

```python
np.diag(A, k=-1)
```

## Transpose of a Matrix

**Transposing** a matrix is achieved by *flipping* the rows and columns. For a matrix $A \in \mathbb{R}^{m\times n}$ the transpose $A^T \in \mathbb{R}^{n\times m}$ is the $n\times m$ matrix given by:

$(A^T)_{ij} = A_{ji}$

Properties of transposes:

- $(A^T)^T = A$
- $(AB)^T = B^T A^T$
- $(A+B)^T = A^T + B^T$

## Symmetric Metrices

Square metrices $A \in \mathbb{R}^{n\times n}$ are **symmetric**, if $A = A^T$.

$A$ is **anti-symmetric**, if $A = -A^T$.

For any matrix $A \in \mathbb{R}^{n\times n}$, the matrix $A + A^T$ is **symmetric**.

For any matrix $A \in \mathbb{R}^{n\times n}$, the matrix $A - A^T$ is **anti-symmetric**.

Thus, any square matrix $A \in \mathbb{R}^{n\times n}$ can be represented as a sum of a symmetric matrix and an anti-symmetric matrix:

$A = \frac{1}{2} (A + A^T) + \frac{1}{2} (A - A^T)$

The first matrix on the right, i.e. $\frac{1}{2} (A + A^T)$ is symmetric. The second matrix $\frac{1}{2} (A - A^T)$ is anti-symmetric.

$\mathbb{S}^n$ is the set of all symmetric matrices of size $n$.

$A \in \mathbb{S}^n$ means that $A$ is symmetric and of the size $n\times n$.

## The Trace

The **trace** of a square matrix $A \in \mathbb{R}^{n\times n}$ is $tr(A)$ (or $trA$) is the sum of the diagonal elements in the matrix:

$trA = \sum_{i=1}^n A_{ii}$

Properties of the **trace**:

- For $A \in \mathbb{R}^{n\times n}$, $\mathrm{tr}A = \mathrm{tr}A^T$
- For $A,B \in \mathbb{R}^{n\times n}$, $\mathrm{tr}(A + B) = \mathrm{tr}A + \mathrm{tr}B$
- For $A \in \mathbb{R}^{n\times n}$, $t \in \mathbb{R}$, $\mathrm{tr}(tA) = t \mathrm{tr}A$
- For $A,B$ such that $AB$ is square, $\mathrm{tr}AB = \mathrm{tr}BA$
- For $A,B,C$ such that $ABC$ is square, $\mathrm{tr}ABC = \mathrm{tr}BCA = \mathrm{tr}CAB$, and so on for the product of more matrices.

## Norms

The **norm** of a vector $x$ is $\| x\|$, informally the length of a vector.

Example: the Euclidean or $\mathscr{l}_2$ norm:

$\|x\|_2 = \sqrt{\sum_{i=1}^n{x_i^2}}$

Note: $\|x\|_2^2 = x^T x$

A **norm** is any function $f : \mathbb{R}^n \rightarrow \mathbb{R}$ that satisfies the following properties:

- For all $x \in \mathbb{R}^n$, $f(x) \geq 0$ (non-negativity)
- $f(x) = 0$ if and only if $x = 0$ (definiteness)
- For all $x \in \mathbb{R}^n$, $t \in \mathbb{R}$, $f(tx) = |t|\ f(x)$ (homogeneity)
- For all $x, y \in \mathbb{R}^n$, $f(x + y) \leq f(x) + f(y)$ (triangle inequality)

Norm $\mathscr{l}_1$:

$\|x\|_1 = \sum_{i=1}^n{|x_i|}$

Norm $\mathscr{l}_\infty$:

$\|x\|_\infty = \max_i|x_i|$

All these three norms are examples of the $\mathscr{l}_p$ norms, with $p$ a real number parameter $p \geq 1$:

$\|x\|_p = \left(\sum_{i=1}^n{|x_i|^p}\right)^{\frac{1}{p}}$

*Frobenius norm* for matrices:

$\|A\|_F = \sqrt{\sum_{i=1}^m\sum_{i=1}^n A_{ij}^2} = \sqrt{\mathrm{tr}(A^T A)}$

And many more.

## Linear Independence and Rank

A set of vectors $\{x_1, x_2, \dots{}, x_n\} \subset \mathbb{R}^m$ is said to be **(linearly) independent** if no vector can be represented as a linear combination of the remaining vectors.

A set of vectors $\{x_1, x_2, \dots{}, x_n\} \subset \mathbb{R}^m$ is said to be **(lineraly) dependent** if one vector from this set can be represented as a linear combination of the remaining vectors.

For some scalar values $\alpha_1, \dots{}, \alpha_{n-1} \in \mathbb{R}$ the vectors $x_1, \dots{}, x_n$ are linerly dependent, if:

$\begin{equation}
x_n = \sum_{i=1}^{n-1}{\alpha_i x_i}
\end{equation}$

Example: The following vectors are lineraly dependent, because $x_3 = -2 x_1 + x_2$

$x_1 = \begin{bmatrix}
 1 \\[0.3em]
 2 \\[0.3em]
 3
\end{bmatrix}
\quad
x_2 = \begin{bmatrix}
 4 \\[0.3em]
 1 \\[0.3em]
 5
\end{bmatrix}
\quad
x_3 = \begin{bmatrix}
 2 \\[0.3em]
 -1 \\[0.3em]
 -1
\end{bmatrix}
$

### Column Rank of a Matrix

The **column rank** of a matrix $A \in \mathbb{R}^{m\times n}$ is the size of the largest subset of columns of $A$ that constitute a linear independent set. Informaly this is the number of linearly independent columns of $A$.

### Row Rank of a Matrix

The **row rank** of a matrix $A \in \mathbb{R}^{m\times n}$ is the largest number of rows of $A$ that constitute a lineraly independent set.

### Rank of a Matrix

For any matrix $A \in \mathbb{R}^{m\times n}$, the column rank of $A$ is equal to the row rank of $A$. Both quantities are referred to collectively as the rank of $A$, denoted as $rank(A)$. Here are some basic properties of the rank:

- For $A \in \mathbb{R}^{m\times n}$, $rank(A) \leq \min(m, n)$. If $rank(A) = \min(m, n)$, then $A$ is said to be
**full rank**.
- For $A \in \mathbb{R}^{m\times n}$, $rank(A) = rank(A^T)$
- For $A \in \mathbb{R}^{m\times n}$, $B \in \mathbb{R}^{n\times p}$, $rank(AB) \leq \min(rank(A), rank(B))$
- For $A,B \in \mathbb{R}^{m\times n}$, $rank(A + B) \leq rank(A) + rank(B)$

## Subtraction and Addition of Metrices

Assume $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{m\times n}$, that is $A$ and $B$ are of the same size, to add $A$ to $B$, or to subtract $B$ from $A$, we add or subtract corresponding entries:

$$
A + B =
\begin{bmatrix}
 a_{11} & a_{12} & \cdots & a_{1n} \\[0.3em]
 a_{21} & a_{22} & \cdots & a_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} & a_{m2} & \cdots & a_{mn}
\end{bmatrix} +
\begin{bmatrix}
 b_{11} & b_{12} & \cdots & b_{1n} \\[0.3em]
 b_{21} & b_{22} & \cdots & b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 b_{m1} & b_{m2} & \cdots & b_{mn}
\end{bmatrix} =
\begin{bmatrix}
 a_{11} + b_{11} & a_{12} + b_{12} & \cdots & a_{1n} + b_{1n} \\[0.3em]
 a_{21} + b_{21} & a_{22} + b_{22} & \cdots & a_{2n} + b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} + b_{m1} & a_{m2} + b_{m2} & \cdots & a_{mn} + b_{mn}
\end{bmatrix}
$$

The same is applies to subtraction:

$$
A - B =
\begin{bmatrix}
 a_{11} & a_{12} & \cdots & a_{1n} \\[0.3em]
 a_{21} & a_{22} & \cdots & a_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} & a_{m2} & \cdots & a_{mn}
\end{bmatrix} -
\begin{bmatrix}
 b_{11} & b_{12} & \cdots & b_{1n} \\[0.3em]
 b_{21} & b_{22} & \cdots & b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 b_{m1} & b_{m2} & \cdots & b_{mn}
\end{bmatrix} =
\begin{bmatrix}
 a_{11} - b_{11} & a_{12} - b_{12} & \cdots & a_{1n} - b_{1n} \\[0.3em]
 a_{21} - b_{21} & a_{22} - b_{22} & \cdots & a_{2n} - b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} - b_{m1} & a_{m2} - b_{m2} & \cdots & a_{mn} - b_{mn}
\end{bmatrix}
$$

In Python using *numpy* this can be achieved using the following code:

```python
import numpy as np
print("np.arange(9):", np.arange(9))
print("np.arange(9, 18):", np.arange(9, 18))
A = np.arange(9, 18).reshape((3, 3))
B = np.arange(9).reshape((3, 3))
print("A:", A)
print("B:", B)
```

The *numpy* function *arange* is similar to the standard Python function *range*. It returns an array with $n$ elements, specified in the one parameter version only. If we provide to parameters to *arange*, it generates an array starting from the value of the first parameter and ending with a value one less than the second parameter. The function *reshape* returns us a matrix with the corresponding number of rows and columns.

We can now add and subtract the two matrices $A$ and $B$:

```python
A + B
```

```python
A - B
```

## Inverse

The **inverse** of a square matrix $A \in \mathbb{R}^{n\times n}$ is $A^{-1}$:

$A^{-1} A = I = A A^{-1}$

Not all matrices have inverses. Non-square matrices do not have inverses by definition. For some square matrices $A$ the inverse might not exist.

$A$ is **invertible** or **non-singular** if $A^{-1}$ exists.

$A$ is **non-invertible** or **singular** if $A^{-1}$ does not exist.

For $A$ to have an inverse $A^{-1}$, $A$ must be **full rank**.

Assuming that $A,B \in \mathbb{R}^{n\times n}$ are non-singular, then:

- $(A^{-1})^{-1} = A$
- $(AB)^{-1} = B^{-1} A^{-1}$
- $(A^{-1})^T = (A^T)^{-1}$ (often simply $A^{-T}$)

## Orthogonal Matrices

Two vectors $x, y \in \mathbb{R}^n$ are **orthogonal** if $x^T y = 0$.

A vector $x \in \mathbb{R}^n$ is **normalized** if $\|x\|^2 = 1$.

A square matrix $U \in \mathbb{R}^{n\times n}$ is **orthogonal** if all its columns are orthogonal to each other and are **normalized**. The columns are then referred to as being **orthonormal**.

It follows immediately from the definition of orthogonality and normality that:

$U^T U = I = U U^T$

This means that the inverse of an orthogonal matrix is its transpose.

If U is not square - i.e., $U \in \mathbb{R}^{m\times n}$, $n < m$ - but its columns are still orthonormal, then $U^T U = I$, but $U U^T \neq I$.

We generally only use the term orthogonal to describe the case, where $U$ is square.

Another nice property of orthogonal matrices is that operating on a vector with an orthogonal matrix will not change its Euclidean norm. For any $x \in \mathbb{R}^n$, $U \in \mathbb{R}^{n\times n}$ orthogonal.

$\|U_x\|^2 = \|x\|^2$

## Range and Nullspace of a Matrix

The **span** of a set of vectors $\{ x_1, x_2, \dots{}, x_n\}$ is the set of all vectors that can be expressed as
a linear combination of $\{ x_1, \dots{}, x_n \}$:

$\mathrm{span}(\{ x_1, \dots{}, x_n \}) = \{ v : v = \sum_{i=1}^n \alpha_i x_i, \alpha_i \in \mathbb{R} \}$

It can be shown that if $\{ x_1, \dots{}, x_n \}$ is a set of n linearly independent vectors, where each $x_i \in \mathbb{R}^n$, then $\mathrm{span}(\{ x_1, \dots{}, x_n\}) = \mathbb{R}^n$. That is, any vector $v \in \mathbb{R}^n$ can be written as a linear combination of $x_1$ through $x_n$.

The projection of a vector $y \in \mathbb{R}^m$ onto the span of $\{ x_1, \dots{}, x_n\}$ (here we assume $x_i \in \mathbb{R}^m$) is the vector $v \in \mathrm{span}(\{ x_1, \dots{}, x_n \})$, such that $v$ is as close as possible to $y$, as measured by the Euclidean norm $\|v − y\|^2$. We denote the projection as $\mathrm{Proj}(y; \{ x_1, \dots{}, x_n \})$ and can define it formally as:

$\mathrm{Proj}( y; \{ x_1, \dots{}, x_n \}) = \mathrm{argmin}_{v\in \mathrm{span}(\{x_1,\dots{},x_n\})}\|y − v\|^2$

The **range** (sometimes also called the columnspace) of a matrix $A \in \mathbb{R}^{m\times n}$, denoted $\mathcal{R}(A)$, is the the span of the columns of $A$. In other words,

$\mathcal{R}(A) = \{ v \in \mathbb{R}^m : v = A x, x \in \mathbb{R}^n\}$

Making a few technical assumptions (namely that $A$ is full rank and that $n < m$), the projection of a vector $y \in \mathbb{R}^m$ onto the range of $A$ is given by:

$\mathrm{Proj}(y; A) = \mathrm{argmin}_{v\in \mathcal{R}(A)}\|v − y\|^2 = A(A^T A)^{−1} A^T y$

The **nullspace** of a matrix $A \in \mathbb{R}^{m\times n}$, denoted $\mathcal{N}(A)$ is the set of all vectors that equal $0$ when multiplied by $A$, i.e.,

$\mathcal{N}(A) = \{ x \in \mathbb{R}^n : A x = 0 \}$

Note that vectors in $\mathcal{R}(A)$ are of size $m$, while vectors in the $\mathcal{N}(A)$ are of size $n$, so vectors in $\mathcal{R}(A^T)$ and $\mathcal{N}(A)$ are both in $\mathbb{R}^n$. In fact, we can say much more. It turns out that:

$\{ w : w = u + v, u \in \mathcal{R}(A^T), v \in \mathcal{N}(A) \} = \mathbb{R}^n$ and $\mathcal{R}(A^T) \cap \mathcal{N}(A) = \{0\}$

In other words, $\mathcal{R}(A^T)$ and $\mathcal{N}(A)$ are disjoint subsets that together span the entire space of
$\mathbb{R}^n$. Sets of this type are called **orthogonal complements**, and we denote this $\mathcal{R}(A^T) = \mathcal{N}(A)^\perp$.

## Determinant

The determinant of a square matrix $A \in \mathbb{R}^{n\times n}$, is a function $\mathrm{det} : \mathbb{R}^{n\times n} \rightarrow \mathbb{R}$, and is denoted $|A|$ or $\mathrm{det}A$ (like the trace operator, we usually omit parentheses).

### A geometric interpretation of the determinant

Given

$\begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_n^T  & --
\end{bmatrix}$

consider the set of points $S \subset \mathbb{R}^n$ formed by taking all possible linear combinations of the row vectors $a_1, \dots{}, a_n \in \mathbb{R}^n$ of $A$, where the coefficients of the linear combination are all
between $0$ and $1$; that is, the set $S$ is the restriction of $\mathrm{span}( \{ a_1, \dots{}, a_n \})$ to only those linear combinations whose coefficients $\alpha_1, \dots{}, \alpha_n$ satisfy $0 \leq \alpha_i \leq 1$, $i = 1, \dots{}, n$. Formally:

$$
S = \{v \in \mathbb{R}^n : v = \sum_{i=1}^n \alpha_i a_i  \textrm{ where }  0 \leq \alpha_i \leq 1, i = 1, \dots{}, n \}
$$

The absolute value of the determinant of $A$, it turns out, is a measure of the *volume* of the set $S$. The volume here is intuitively for example for $n = 2$ the area of $S$ in the Cartesian plane, or with $n = 3$ it is the common understanding of *volume* for 3-dimensional objects.

Example:

$A = \begin{bmatrix}
 1 & 3\\[0.3em]
 3 & 2
\end{bmatrix}$

The rows of the matrix are:

$a_1 = \begin{bmatrix}
 1 \\[0.3em]
 3
\end{bmatrix}
\quad
a_2 = \begin{bmatrix}
 3 \\[0.3em]
 2
\end{bmatrix}$

The set S corresponding to these rows is shown in:

The figure above is an illustration of the determinant for the $2\times 2$ matrix $A$ above. Here, $a_1$ and $a_2$
are vectors corresponding to the rows of $A$, and the set $S$ corresponds to the shaded region (i.e., the parallelogram). The absolute value of the determinant, $|\mathrm{det}A| = 7$, is the area of the parallelogram.

For two-dimensional matrices, $S$ generally has the shape of a parallelogram. In our example, the value of the determinant is $|A| = −7$ (as can be computed using the formulas shown later), so the area of the parallelogram is $7$.

In three dimensions, the set $S$ corresponds to an object known as a parallelepiped (a three-dimensional box with skewed sides, such that every face has the shape of a parallelogram). The absolute value of the determinant of the $3 \times 3$ matrix whose rows define $S$ give the three-dimensional volume of the parallelepiped. In even higher dimensions, the set $S$ is an object known as an $n$-dimensional parallelotope.

Algebraically, the determinant satisfies the following three properties (from which all other properties follow, including the general formula):

- The determinant of the identity is $1$, $|I| = 1$. (Geometrically, the volume of a unit hypercube is $1$).
- Given a matrix $A \in \mathbb{R}^{n\times n}$, if we multiply a single row in $A$ by a scalar $t \in \mathbb{R}$, then the determinant of the new matrix is $t|A|$,<br/>
$\left| \begin{bmatrix}
 -- & t a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}\right|  = t|A|$<br/>
(Geometrically, multiplying one of the sides of the set $S$ by a factor $t$ causes the volume
to increase by a factor $t$.)
- If we exchange any two rows $a^T_i$ and $a^T_j$ of $A$, then the determinant of the new matrix is $−|A|$, for example<br/>
$\left| \begin{bmatrix}
 -- & a_2^T  & -- \\[0.3em]
 -- & a_1^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}\right|  = -|A|$

Several properties that follow from the three properties above include:

- For $A \in \mathbb{R}^{n\times n}$, $|A| = |A^T|$
- For $A,B \in \mathbb{R}^{n\times n}$, $|AB| = |A||B|$
- For $A \in \mathbb{R}^{n\times n}$, $|A| = 0$ if and only if $A$ is singular (i.e., non-invertible). (If $A$ is singular then it does not have full rank, and hence its columns are linearly dependent. In this case, the set $S$ corresponds to a "flat sheet" within the $n$-dimensional space and hence has zero volume.)
- For $A \in \mathbb{R}^{n\times n}$ and $A$ non-singular, $|A−1| = 1/|A|$

## Tensors

A [**tensor**](https://en.wikipedia.org/wiki/Tensor) could be thought of as an organized multidimensional array of numerical values. A vector could be assumed to be a sub-class of a tensor. Rows of tensors extend alone the y-axis, columns along the x-axis. The **rank** of a scalar is 0, the rank of a **vector** is 1, the rank of a **matrix** is 2, the rank of a **tensor** is 3 or higher.

## Hyperplane

The **hyperplane** is a sub-space in the ambient space with one dimension less. In a two-dimensional space the hyperplane is a line, in a three-dimensional space it is a two-dimensional plane, etc.

Hyperplanes divide an $n$-dimensional space into sub-spaces that might represent clases in a machine learning algorithm.

# Summary

## Dot Product

This is also the *inner product*. It is a function that returns a number computed from two vectors of the same length by summing up the product of the corresponding dimensions.

For two vectors $a = [a_1, a_2, \dots{}, a_n]$ and $b = [b_1, b_2, \dots{}, b_n]$ the dot product is:

$\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_{i} b_{i} = a_{1} b_{1} + a_{2} b_{2} + \cdots + a_{n} b_{n}$

If we normalize two vectors and compute the dot product, we get the *cosine similarity*, which can be used as a metric for cimilarity of vectors. Independent of the absolute length we look at the angle between the vectors, i.e. the lenght is neutralized via normalization.

The cosine of two non-zero vectors can be derived by using the Euclidean dot product formula (see [Wikipedia: Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)):

$\mathbf{a} \cdot \mathbf{b} = \left\|\mathbf{a}\right\| \left\|\mathbf{b}\right\| \cos\theta$

Given two vectors of attributes, $A$ and $B$, the cosine similarity, $cos(\theta)$, is represented using a dot product and magnitude as:

$\text{similarity} = \cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{ \|\mathbf{A} \|\|\mathbf{B} \| } = \frac{\sum \limits_{i=1}^{n}{A_{i}B_{i}}}{{\sqrt {\sum \limits _{i=1}^{n}{A_{i}^{2}}}}{\sqrt {\sum \limits _{i=1}^{n}{B_{i}^{2}}}}}$, with $A_i$ and $B_i$ components of vector $A$ and $B$ respectively.

## Hadamard Product

This is also known as the **entrywise product**. For two matrices $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{m\times n}$ the Hadamard product $A\circ B$ is:

$(A\circ B)_{i,j} = (A)_{i,j} (B)_{i,j}$

For example:

$$\begin{bmatrix}
 a_{11} & a_{12} & a_{13} \\[0.3em]
 a_{21} & a_{22} & a_{23} \\[0.3em]
 a_{31} & a_{32} & a_{33}
\end{bmatrix} \circ
\begin{bmatrix}
 b_{11} & b_{12} & b_{13} \\[0.3em]
 b_{21} & b_{22} & b_{23} \\[0.3em]
 b_{31} & b_{32} & b_{33}
\end{bmatrix} =
\begin{bmatrix}
 a_{11}b_{11} & a_{12}b_{12} & a_{13}b_{13} \\[0.3em]
 a_{21}b_{21} & a_{22}b_{22} & a_{23}b_{23} \\[0.3em]
 a_{31}b_{31} & a_{32}b_{32} & a_{33}b_{33}
\end{bmatrix}$$

## Outer Product

This is also called the **tensor product** of two vectors. Compute the resulting matrix by multiplying each element from a column vector with all alements in a row vector.

(Based on the work of [Damir Cavar](https://github.com/dcavar/python-tutorial-notebooks))


# Simply Supported Beam subjected to Linearly Varying Load

## Objective
Design, develop and test a Python program to analyse a simply supported beam subjcted to linearly varying load applied over the entire span. The program should calculate the support reactions, SF and BM at regular intervals along the span, point of zero shear and maximum bending moment.

## Theory

The equations for reactions at supports, intensity of load ($w_x$), sfear force ($V_x$) and bending moment ($M_x$) at a distance $x$ from left support are as shown below:

$$
\begin{align}
R_a &= \left( 2 w_1 + w_2 \right) \frac{L}{6} \\
R_b &= \left( w_1 + 2 w_2 \right) \frac{L}{6} \\
w_x &= w_1 + \frac{w_2 - w_1}{L} x \\
V_x &= R_a - \frac{w_1 + w_x}{2} x \\
M_x &= R_a x - \frac{w_1 + w_x}{2} x \cdot \frac{2 w_1 + w_x}{w_1 + w_x} \frac{x}{3} = R_a x - (2 w_1 + w_x) \frac{x^2}{6}
\end{align}$$

It can be seen that when the intensity of load $w_x$ varies linearly with $x$, variation of $V_x$ and $M_x$ with respect to $x$ is parabolic and cubic, respectively.

## Input Data
The primary input data are:
1. **Span $L$** in metres
2. **Intensity of load at left support $w_1$** in kN/m
3. **Intensity of load at right support $w_2$** in kN/m
4. **Number of equal intervals along span** at which shear force and bending moment are to be calculated

## Output Data
1. Shear force at regular intervals
2. Bending moment at regular intervals

## Other Data and Sign Convention
1. Distance $x$ of a chosen cross section from left support is positive when measured to the right of left support and $0 \leq x \leq L$
2. Intensity of load $w_1, w_2, w_x$ are positive when acting downwards.
3. Reactions at supports $R_a, R_b$ are positive when acting upwards.
4. Shear force $V_x$ is positive when net force on left of section is upwards or when net force to the right of section is downwards. Negative otherwise.
5. Bending moment $M_x$ is positive when bending moment is sagging and negative if hogging.

## Program Organization
We will develop functions to calculate each of these results, namely, reactions, intensity of load, sgear force and bending moment, as follows:

Function Name	Input Data	Output Data	Purpose

**`reactions()`**	$L$, $w_1$, $w_2$	$R_a, R_b$	Calculate reactions at supports

**`calc_wx()`**	$x$, $L$, $w_1$, $w_2$	$w_x$	Calculate intensity of load at $x$

**`calc_Vx()`**	$x$, $L$, $w_1$, $w_2$	$V_x$	Calculate shear force at $x$

**`calc_Mx()`**	$x$, $L$, $w_1$, $w_2$	$M_x$	Calculate bending moment at $x$

The function to calculate the reactions uses the above equations is shown below. It is tested for different types of input, namely, UDL ($w_1 = w_2$), triangular loads ($w_1 = 0$ and $w_2 = 0$).

```python
from __future__ import division, print_function
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def reactions(L, w1, w2):
    '''Returns the reactions at left and right supports of a simply supported beam
       subjected to a linearly varying load with intensity w1 and w2 at the left and right
       ends, respectively'''
    Ra = (2 * w1 + w2) * L / 6
    Rb = (w1 + 2 * w2) * L / 6
    return (Ra, Rb)

Ra, Rb = reactions(6, 10, 10)
print(Ra, Rb)

Ra, Rb = reactions(6, 0, 30)
print(Ra, Rb)

Ra, Rb = reactions(6, 30, 0)
print(Ra, Rb)

help(reactions)
```

    30.0 30.0
    30.0 60.0
    60.0 30.0
    Help on function reactions in module __main__:

    reactions(L, w1, w2)
        Returns the reactions at left and right supports of a simply supported beam
        subjected to a linearly varying load with intensity w1 and w2 at the left and right
        ends, respectively

Function to calculate intensity of load $w_x$ at distance $x$ from left support is shown below. It is tested for the intensity of load at midspan of a $6~m$ span beam with triangular loading ($w_1 = 0$) and UDL ($w_1 = w_2$).

```python
def calc_wx(x, L, w1, w2):
    '''Returns the intensity of a linearly varying load at distance 'x' from left support
       when intensity at x=0 is w1 and at x=L is w2'''
    return w1 + (w2 - w1) * x / L

print(calc_wx(3, 6, 0, 20))
print(calc_wx(3, 6, 10, 10))

help(calc_wx)

x = np.array([0, 1, 2, 3, 4, 5, 6], dtype=float)
wx = calc_wx(x, 6.0, 0.0, 12.0)
print(type(x), len(x))
print(type(wx), len(wx))
print(wx)
```

    10.0
    10.0
    Help on function calc_wx in module __main__:

    calc_wx(x, L, w1, w2)
        Returns the intensity of a linearly varying load at distance 'x' from left support
        when intensity at x=0 is w1 and at x=L is w2

    <type 'numpy.ndarray'> 7
    <type 'numpy.ndarray'> 7
    [  0.   2.   4.   6.   8.  10.  12.]

Functions to calculate shear force $V_x$ and bending moment $M_x$ at $x$ are shown below. They are tested for 13 data points spaced equally along the span. Graphs of $w_x, V_x$ and $M_x$ are plotted and verified for the nature of their variation.

```python
def calc_Vx(x, L, w1, w2):
    '''Returns the shear force at a distance 'x' from left support in a simply supported beam
       of span L subjected to a linearly varying load of intensity w1 at x=0 and w2 at x=L'''

    Ra, Rb = reactions(L, w1, w2)
    return (Ra  - (w1 + calc_wx(x, L, w1, w2)) / 2 * x)

def calc_Mx(x, L, w1, w2):
    '''Returns the bending moment at a distance 'x' from left support in a simply supported beam
       of span L subjected to a linearly varying load of intensity w1 at x=0 and w2 at x=L'''
    Ra, Rb = reactions(L, w1, w2)
    wx = calc_wx(x, L, w1, w2)
    return (Ra * x - (2*w1+wx)*x*x/6)

L = 6.0
w1 = 0.0
w2 = 12.0
numint = 12

Ra, Rb = reactions(L, w1, w2)
print('Ra = %.4f, Rb = %.4f' % (Ra, Rb))

x = np.linspace(0, L, numint+1)
wx = calc_wx(x, 6.0, w1, w2)
Vx = calc_Vx(x, L, w1, w2)
Mx = calc_Mx(x, L, w1, w2)

for xx, yy, zz, mm in zip(x, wx, Vx, Mx):
    print("%8.2f %12.4f %12.4f %12.4f" % (xx, yy, zz, mm))

plt.subplot(311)
plt.plot(x, wx)
plt.grid()

plt.subplot(312)
plt.plot(x, Vx)
plt.grid()

plt.subplot(313)
plt.plot(x, Mx)
plt.grid()

plt.show()
```

We must now determine the value of $x$ at which $V_x = 0$. This requires the calculation of root of the shear force equation. To achieve this, we will write two functions **`bracket()`** and **`false_pos()`** which will accomplish the following tasks, which are general in that they can be used in other problems which require finding of roots.

**`bracket()`** takes as input two vectors, namely the values of the independent variable $x$ and the corresponding dependent variable $y$. It returs **`None`** if no roots are bracketed else it returns the first brackets if a root is bracketed

**`false_pos()`** takes as input the function $f(x, L, w_1, w_2)$ whose roots are to be found, the brackets $x_1$, $x_2$, tolerance and the maximum number of iterations desired.

The function **`false_pos()`** is tested and verified by calculating shear force at the distance calculated by **`false_pos()`**, which must be zero, subjected to the chosen tolerance. Once verified, we can calculate the bending moment at the same location.

```python
def bracket(x, y):
    '''Returns an List with two elements corresponding to the first root that is bracketed
       within the array x[], else returns None'''

    b = []
    for i in range(len(x)-1):
        if y[i] * y[i+1] <= 0.0:
            b.append(x[i])
            b.append(x[i+1])
            return b
    return None

print(type(x), len(x))
b = bracket(x, Vx)
print(b)
```

    <type 'numpy.ndarray'> 13
    [3.0, 3.5]

```python
def bisection(f, x1, x2, L, w1, w2, tol=1e-6, maxiter=50):
    '''Returns the root within the bracket [x1, x2] using Bisection method. Convergence is determined
       by the tolerance 'tol', which defaults to 1e-6 if not specified. A maximum of 'maxiter' iterations
       are performed, which defaults to 50 if not specified. If no root with the required tolerance can be
       found within the specified number of iterations, function returns None

       When calling this function, keyword arguments 'L', 'w1' and 'w2' must be supplied, which in turn will
       be passed on to function 'f', whose root is being determined.'''

    y1 = f(x1, L, w1, w2)
    y2 = f(x2, L, w1, w2)
    if abs(y1) <= tol:
        return x1
    elif abs(y2) <= tol:
        return x2

    if (y1 * y2) > 0:
        print('Root not bracketed')
        return None

    k = 0
    while (k <= maxiter):
        k += 1
        x = (x1 + x2) / 2
        y = f(x, L, w1, w2)
        if abs(y) <= tol:
            return x
        else:
            if y1 * y < 0:
                x2 = x
                y2 = y
            else:
                x1 = x
                y1 = y

    if abs(y1) <= tol:
        return(x1)
    elif abs(y2) <= tol:
        return x2
    else:
        return None

x = bisection(calc_Vx, b[0],  b[1], 6.0, 0.0, 12.0, 1e-12)
print(x, calc_Vx(x, L, w1, w2))

help(bisection)
```

    3.46410161514 5.24025267623e-13
    Help on function bisection in module __main__:

    bisection(f, x1, x2, L, w1, w2, tol=1e-06, maxiter=50)
        Returns the root within the bracket [x1, x2] using Bisection method. Convergence is determined
        by the tolerance 'tol', which defaults to 1e-6 if not specified. A maximum of 'maxiter' iterations
        are performed, which defaults to 50 if not specified. If no root with the required tolerance can be
        found within the specified number of iterations, function returns None

        When calling this function, keyword arguments 'L', 'w1' and 'w2' must be supplied, which in turn will
        be passed on to function 'f', whose root is being determined.

```python
print(calc_Mx(x, L, w1, w2))
```

    27.7128129211

Once each function is tested, we can now write the main function that will set up the input data, call the required functions in the correct sequence and print the results. A separate function is developed to plot graphs.

```python
def plot_graphs(x, wx, Vx, Mx):
    '''Function to plot graphs for intensity of load, shear force and bending moment. Does not return any value.'''

    plt.subplot(311)
    plt.plot(x, wx)
    plt.grid()

    plt.subplot(312)
    plt.plot(x, Vx)
    plt.grid()

    plt.subplot(313)
    plt.plot(x, Mx)
    plt.grid()

    plt.show()
    return

def main(L, w1, w2, numint):
    '''Main function that sets up input, prints output and generates graphs. Does not return any value'''

    Ra, Rb = reactions(L, w1, w2)
    print('INPUT')
    print('Span = %.4f m' % L)
    print('Intensity of load at left support  = %.4f kN/m' % w1)
    print('Intensity of load at right support = %.4f kN/m' % w2)
    print()
    print('OUTPUT')
    print('Reactions')
    print('Ra = %.4f, Rb = %.4f' % (Ra, Rb))
    x = np.linspace(0.0, L, numint+1)
    wx = calc_wx(x, L, w1, w2)
    Vx = calc_Vx(x, L, w1, w2)
    Mx = calc_Mx(x, L, w1, w2)
    for xx, ww, sf, bm in zip(x, wx, Vx, Mx):
        print("%8.2f %12.4f %12.4f %12.4f" % (xx, ww, sf, bm))
    plot_graphs(x, wx, Vx, Mx)
    b = bracket(x, Vx)
    xmax = bisection(calc_Vx, b[0], b[1], L, w1, w2, 1e-12)
    Mmax = calc_Mx(xmax, L, w1, w2)
    print('Maximum Bending Moment')
    print('x = %.4f V = %.4f Mmax = %.4f' % (xmax, calc_Vx(xmax, L, w1, w2), calc_Mx(xmax, L, w1, w2)))
    return

L = 6.0
w1 = 0.0
w2 = 12.0
main(L, w1, w2, 13)
```

```python
L = 6.0
w1 = 10.0
w2 = 10.0
main(L, w1, w2, 13)
```

```python
L = 6.0
w1 = 12.0
w2 = 0.0
main(L, w1, w2, 13)
```

## Results and Discussion
* Discuss the results for different types of loads that can be generated using this program, namely
  * Triangular load ($w_1=0, w_2 \neq 0$ or $w1 \neq 0, w_2=0$)
  * Unifomly distributed load ($w_1 = w_2$)
  * Trapezoidal load ($w_1 \neq 0$ and $w_2 \neq 0$)
* A study on the effect of changing number of intervals could be made

## Limitations of the Program
The defined scope of the problem limits the program to simple supports, load applied over entire span and varying linearly from $w_1$ at left support to $w_2$ at the right support.

## Improvements to the Program
* Replacing Bisection with False Position method may reduce the number of iterations required.

## Conclusions

```python
def bisection(f, x1, x2, L, w1, w2, tol, maxiter):
    y1 = f(x1, L, w1, w2)

    if abs(y1) <= tol:
        return x1

    y2 = f(x2, L, w1, w2)
    if abs(y2) <= tol:
        return x2
    k = 0
    while k <= maxiter:
        k += 1
        x = (x1 + x2) / 2
        y = f(x, L, w1, w2)
        # print(x, y)
        if abs(y) <= tol:
            return x
        if y1 * y <= 0.0:  # Root is bracketed by left half
            x2 = x
            y2 = y
        else: # Root is bracketed by right half
            x1 = x
            y1 = y
    return None
root = bisection(calc_Vx, 3.0, 3.5, 6, 0, 12, 1e-6, 50)
Vmax = calc_Vx(root, 6, 0, 12)
Mmax = calc_Mx(root, 6, 0, 12)
print(root, Vmax, Mmax)
print()
root = bisection(calc_Vx, 3.0, 3.5, 6, 0, 12, 1e-12, 50)
Vmax = calc_Vx(root, 6, 0, 12)
Mmax = calc_Mx(root, 6, 0, 12)
print(root, Vmax, Mmax)
print()
root = bisection(calc_Vx, 3.0, 3.5, 6, 0, 12, 1e-16, 50)
Vmax = calc_Vx(root, 6, 0, 12)
Mmax = calc_Mx(root, 6, 0, 12)
print(root, Vmax, Mmax)
```

    3.46410155296 4.30757552294e-07 27.7128129211

    3.46410161514 5.24025267623e-13 27.7128129211

    3.46410161514 0.0 27.7128129211

```python
class UDL(object):
    def __init__(self, w, d1, d2):
        self.w = w
        if d2 > d1:
            self.d1 = d1
            self.d2 = d2
        else:
            seld.d1 = d2
            self.d2 = d1
        return

    def getResultant(self):
        return self.w * (self.d2 - self.d1)

    def getCG(self):
        return (self.d1 + self.d2) / 2.0

    def getMagnitude(self):
        return self.getResultant()

    def getMoment(self):
        return self.getResultant() * self.getCG()

    def calcV(self, x):
        if x <= self.d1:
            return 0.0
        elif x <= self.d2:
            return -(self.w * (x - self.d1))
        else:
            return -self.getResultant()

    def calcM(self, x):
        if x <= self.d1:
            return 0.0
        elif x <= self.d2:
            return -(self.w * (x - self.d1)**2 / 2.0)
        else:
            return -(self.getResultant() * (x - self.getCG()))

class SSBeam(object):
    def __init__(self, L, loads, ndiv=10):
        self.L = L
        self.loads = loads[:]
        self.ndiv = ndiv
        self.sections = np.linspace(0, self.L, self.ndiv+1)
        self.V = np.zeros(self.sections.shape)
        self.M = np.zeros(self.sections.shape)
        return

    def getSections(self):
        self.sections = np.linspace(0, self.L, self.ndiv+1)
        return

    def __str__(self):
        s = "Simply Supported Beam, Span = %.2f" % (self.L,)
        return s

    def calcReactions(self):
        P = 0.0
        M = 0.0
        for load in self.loads:
            P += load.getMagnitude()
            M += load.getMoment()
        Rb = M / self.L
        Ra = P - Rb
        return (Ra, Rb)

    def calcV_M(self):
        Ra, Rb = self.calcReactions()
        for i, x in enumerate(self.sections):
            V = Ra
            M = Ra * x
            for load in self.loads:
                V += load.calcV(x)
                M += load.calcM(x)
            self.V[i] = V
            self.M[i] = M
        print(self.V)
        print(self.M)
        return

    def plot(self):
        plt.subplot(211)
        plt.plot(self.sections, self.V)
        plt.subplot(212)
        plt.plot(self.sections, self.M)
        plt.grid()
        plt.show()
        return

loads = [UDL(10.0, 0.0, 6.0)]
B = SSBeam(6, loads, 12)
print(B)
Ra, Rb = B.calcReactions()
print("Ra = %.2f, Rb = %.2f" % (Ra, Rb))
B.calcV_M()
B.plot()
```

```python

```


# Linear Regression Tutorial
by Marc Deisenroth

The purpose of this notebook is to practice implementing some linear algebra (equations provided) and to explore some properties of linear regression.

```python
import numpy as np
import scipy.linalg
import matplotlib.pyplot as plt
%matplotlib inline
```

We consider a linear regression problem of the form
$$
y = \boldsymbol x^T\boldsymbol\theta + \epsilon\,,\quad \epsilon \sim \mathcal N(0, \sigma^2)
$$
where $\boldsymbol x\in\mathbb{R}^D$ are inputs and $y\in\mathbb{R}$ are noisy observations. The parameter vector $\boldsymbol\theta\in\mathbb{R}^D$ parametrizes the function.

We assume we have a training set $(\boldsymbol x_n, y_n)$, $n=1,\ldots, N$. We summarize the sets of training inputs in $\mathcal X = \{\boldsymbol x_1, \ldots, \boldsymbol x_N\}$ and corresponding training targets $\mathcal Y = \{y_1, \ldots, y_N\}$, respectively.

In this tutorial, we are interested in finding good parameters $\boldsymbol\theta$.

```python
# Define training set
X = np.array([-3, -1, 0, 1, 3]).reshape(-1,1) # 5x1 vector, N=5, D=1
y = np.array([-1.2, -0.7, 0.14, 0.67, 1.67]).reshape(-1,1) # 5x1 vector

# Plot the training set
plt.figure()
plt.plot(X, y, '+', markersize=10)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

## 1. Maximum Likelihood
We will start with maximum likelihood estimation of the parameters $\boldsymbol\theta$. In maximum likelihood estimation, we find the parameters $\boldsymbol\theta^{\mathrm{ML}}$ that maximize the likelihood
$$
p(\mathcal Y | \mathcal X, \boldsymbol\theta) = \prod_{n=1}^N p(y_n | \boldsymbol x_n, \boldsymbol\theta)\,.
$$
From the lecture we know that the maximum likelihood estimator is given by
$$
\boldsymbol\theta^{\text{ML}} = (\boldsymbol X^T\boldsymbol X)^{-1}\boldsymbol X^T\boldsymbol y\in\mathbb{R}^D\,,
$$
where
$$
\boldsymbol X = [\boldsymbol x_1, \ldots, \boldsymbol x_N]^T\in\mathbb{R}^{N\times D}\,,\quad \boldsymbol y = [y_1, \ldots, y_N]^T \in\mathbb{R}^N\,.
$$

Let us compute the maximum likelihood estimate for a given training set

```python
## EDIT THIS FUNCTION
def max_lik_estimate(X, y):

    # X: N x D matrix of training inputs
    # y: N x 1 vector of training targets/observations
    # returns: maximum likelihood parameters (D x 1)

    N, D = X.shape
    theta_ml = np.linalg.solve(X.T @ X, X.T @ y) ## <-- SOLUTION
    return theta_ml
```

```python
# get maximum likelihood estimate
theta_ml = max_lik_estimate(X,y)
```

Now, make a prediction using the maximum likelihood estimate that we just found

```python
## EDIT THIS FUNCTION
def predict_with_estimate(Xtest, theta):

    # Xtest: K x D matrix of test inputs
    # theta: D x 1 vector of parameters
    # returns: prediction of f(Xtest); K x 1 vector

    prediction = Xtest @ theta ## <-- SOLUTION

    return prediction
```

Now, let's see whether we got something useful:

```python
# define a test set
Xtest = np.linspace(-5,5,100).reshape(-1,1) # 100 x 1 vector of test inputs

# predict the function values at the test points using the maximum likelihood estimator
ml_prediction = predict_with_estimate(Xtest, theta_ml)

# plot
plt.figure()
plt.plot(X, y, '+', markersize=10)
plt.plot(Xtest, ml_prediction)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Questions
1. Does the solution above look reasonable?
2. Play around with different values of $\theta$. How do the corresponding functions change?
3. Modify the training targets $\mathcal Y$ and re-run your computation. What changes?

Let us now look at a different training set, where we add 2.0 to every $y$-value, and compute the maximum likelihood estimate

```python
ynew = y + 2.0

plt.figure()
plt.plot(X, ynew, '+', markersize=10)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

```python
# get maximum likelihood estimate
theta_ml = max_lik_estimate(X, ynew)
print(theta_ml)

# define a test set
Xtest = np.linspace(-5,5,100).reshape(-1,1) # 100 x 1 vector of test inputs

# predict the function values at the test points using the maximum likelihood estimator
ml_prediction = predict_with_estimate(Xtest, theta_ml)

# plot
plt.figure()
plt.plot(X, ynew, '+', markersize=10)
plt.plot(Xtest, ml_prediction)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Question:
1. This maximum likelihood estimate doesn't look too good: The orange line is too far away from the observations although we just shifted them by 2. Why is this the case?
2. How can we fix this problem?

Let us now define a linear regression model that is slightly more flexible:
$$
y = \theta_0 + \boldsymbol x^T \boldsymbol\theta_1 + \epsilon\,,\quad \epsilon\sim\mathcal N(0,\sigma^2)
$$
Here, we added an offset (bias) parameter $\theta_0$ to our original model.

#### Question:
1. What is the effect of this bias parameter, i.e., what additional flexibility does it offer?

If we now define the inputs to be the augmented vector $\boldsymbol x_{\text{aug}} = \begin{bmatrix}1\\\boldsymbol x\end{bmatrix}$, we can write the new linear regression model as
$$
y = \boldsymbol x_{\text{aug}}^T\boldsymbol\theta_{\text{aug}} + \epsilon\,,\quad \boldsymbol\theta_{\text{aug}} = \begin{bmatrix}
\theta_0\\
\boldsymbol\theta_1
\end{bmatrix}\,.
$$

```python
N, D = X.shape
X_aug = np.hstack([np.ones((N,1)), X]) # augmented training inputs of size N x (D+1)
theta_aug = np.zeros((D+1, 1)) # new theta vector of size (D+1) x 1
```

Let us now compute the maximum likelihood estimator for this setting.
_Hint:_ If possible, re-use code that you have already written

```python
## EDIT THIS FUNCTION
def max_lik_estimate_aug(X_aug, y):

    theta_aug_ml = max_lik_estimate(X_aug, y) ## <-- SOLUTION

    return theta_aug_ml
```

```python
theta_aug_ml = max_lik_estimate_aug(X_aug, y)
```

Now, we can make predictions again:

```python
# define a test set (we also need to augment the test inputs with ones)
Xtest_aug = np.hstack([np.ones((Xtest.shape[0],1)), Xtest]) # 100 x (D + 1) vector of test inputs

# predict the function values at the test points using the maximum likelihood estimator
ml_prediction = predict_with_estimate(Xtest_aug, theta_aug_ml)

# plot
plt.figure()
plt.plot(X, y, '+', markersize=10)
plt.plot(Xtest, ml_prediction)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

It seems this has solved our problem!
#### Question:
1. Play around with the first parameter of $\boldsymbol\theta_{\text{aug}}$ and see how the fit of the function changes.
2. Play around with the second parameter of $\boldsymbol\theta_{\text{aug}}$ and see how the fit of the function changes.

### Nonlinear Features
So far, we have looked at linear regression with linear features. This allowed us to fit straight lines. However, linear regression also allows us to fit functions that are nonlinear in the inputs $\boldsymbol x$, as long as the parameters $\boldsymbol\theta$ appear linearly. This means, we can learn functions of the form
$$
f(\boldsymbol x, \boldsymbol\theta) = \sum_{k = 1}^K \theta_k \phi_k(\boldsymbol x)\,,
$$
where the features $\phi_k(\boldsymbol x)$ are (possibly nonlinear) transformations of the inputs $\boldsymbol x$.

Let us have a look at an example where the observations clearly do not lie on a straight line:

```python
y = np.array([10.05, 1.5, -1.234, 0.02, 8.03]).reshape(-1,1)
plt.figure()
plt.plot(X, y, '+')
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Polynomial Regression
One class of functions that is covered by linear regression is the family of polynomials because we can write a polynomial of degree $K$ as
$$
\sum_{k=0}^K \theta_k x^k = \boldsymbol \phi(x)^T\boldsymbol\theta\,,\quad
\boldsymbol\phi(x)=
\begin{bmatrix}
x^0\\
x^1\\
\vdots\\
x^K
\end{bmatrix}\in\mathbb{R}^{K+1}\,.
$$
Here, $\boldsymbol\phi(x)$ is a nonlinear feature transformation of the inputs $x\in\mathbb{R}$.

Similar to the earlier case we can define a matrix that collects all the feature transformations of the training inputs:
$$
\boldsymbol\Phi = \begin{bmatrix}
\boldsymbol\phi(x_1) & \boldsymbol\phi(x_2) & \cdots & \boldsymbol\phi(x_n)
\end{bmatrix}^T \in\mathbb{R}^{N\times K+1}
$$

Let us start by computing the feature matrix $\boldsymbol \Phi$

```python
## EDIT THIS FUNCTION
def poly_features(X, K):

    # X: inputs of size N x 1
    # K: degree of the polynomial
    # computes the feature matrix Phi (N x (K+1))

    X = X.flatten()
    N = X.shape[0]

    #initialize Phi
    Phi = np.zeros((N, K+1))

    # Compute the feature matrix in stages
    for k in range(K+1):
        Phi[:,k] = X**k ## <-- SOLUTION
    return Phi
```

With this feature matrix we get the maximum likelihood estimator as
$$
\boldsymbol \theta^\text{ML} = (\boldsymbol\Phi^T\boldsymbol\Phi)^{-1}\boldsymbol\Phi^T\boldsymbol y
$$
For reasons of numerical stability, we often add a small diagonal "jitter" $\kappa>0$ to $\boldsymbol\Phi^T\boldsymbol\Phi$ so that we can invert the matrix without significant problems so that the maximum likelihood estimate becomes
$$
\boldsymbol \theta^\text{ML} = (\boldsymbol\Phi^T\boldsymbol\Phi + \kappa\boldsymbol I)^{-1}\boldsymbol\Phi^T\boldsymbol y
$$

```python
## EDIT THIS FUNCTION
def nonlinear_features_maximum_likelihood(Phi, y):
    # Phi: features matrix for training inputs. Size of N x D
    # y: training targets. Size of N by 1
    # returns: maximum likelihood estimator theta_ml. Size of D x 1

    kappa = 1e-08 # 'jitter' term; good for numerical stability

    D = Phi.shape[1]

    # maximum likelihood estimate
    Pt = Phi.T @ y # Phi^T*y
    PP = Phi.T @ Phi + kappa*np.eye(D) # Phi^T*Phi + kappa*I

    # maximum likelihood estimate
    C = scipy.linalg.cho_factor(PP)
    theta_ml = scipy.linalg.cho_solve(C, Pt) # inv(Phi^T*Phi)*Phi^T*y

    return theta_ml
```

Now we have all the ingredients together: The computation of the feature matrix and the computation of the maximum likelihood estimator for polynomial regression. Let's see how this works.

To make predictions at test inputs $\boldsymbol X_{\text{test}}\in\mathbb{R}$, we need to compute the features (nonlinear transformations) $\boldsymbol\Phi_{\text{test}}= \boldsymbol\phi(\boldsymbol X_{\text{test}})$ of $\boldsymbol X_{\text{test}}$ to give us the predicted mean
$$
\mathbb{E}[\boldsymbol y_{\text{test}}] = \boldsymbol \Phi_{\text{test}}\boldsymbol\theta^{\text{ML}}
$$

```python
K = 5 # Define the degree of the polynomial we wish to fit
Phi = poly_features(X, K) # N x (K+1) feature matrix

theta_ml = nonlinear_features_maximum_likelihood(Phi, y) # maximum likelihood estimator

# test inputs
Xtest = np.linspace(-4,4,100).reshape(-1,1)

# feature matrix for test inputs
Phi_test = poly_features(Xtest, K)

y_pred = Phi_test @ theta_ml # predicted y-values

plt.figure()
plt.plot(X, y, '+')
plt.plot(Xtest, y_pred)
plt.xlabel("$x$")
plt.ylabel("$y$");
```

Experiment with different polynomial degrees in the code above.
#### Questions:
1. What do you observe?
2. What is a good fit?

## Evaluating the Quality of the Model

Let us have a look at a more interesting data set

```python
def f(x):
    return np.cos(x) + 0.2*np.random.normal(size=(x.shape))

X = np.linspace(-4,4,20).reshape(-1,1)
y = f(X)

plt.figure()
plt.plot(X, y, '+')
plt.xlabel("$x$")
plt.ylabel("$y$");
```

Now, let us use the work from above and fit polynomials to this dataset.

```python
## EDIT THIS CELL
K = 2 # Define the degree of the polynomial we wish to fit

Phi = poly_features(X, K) # N x (K+1) feature matrix

theta_ml = nonlinear_features_maximum_likelihood(Phi, y) # maximum likelihood estimator

# test inputs
Xtest = np.linspace(-5,5,100).reshape(-1,1)
ytest = f(Xtest) # ground-truth y-values

# feature matrix for test inputs
Phi_test = poly_features(Xtest, K)

y_pred = Phi_test @ theta_ml # predicted y-values

# plot
plt.figure()
plt.plot(X, y, '+')
plt.plot(Xtest, y_pred)
plt.plot(Xtest, ytest)
plt.legend(["data", "prediction", "ground truth observations"])
plt.xlabel("$x$")
plt.ylabel("$y$");
```

#### Questions:
1. Try out different degrees of polynomials.
2. Based on visual inspection, what looks like the best fit?

Let us now look at a more systematic way to assess the quality of the polynomial that we are trying to fit. For this, we compute the root-mean-squared-error (RMSE) between the $y$-values predicted by our polynomial and the ground-truth $y$-values. The RMSE is then defined as
$$
\text{RMSE} = \sqrt{\frac{1}{N}\sum_{n=1}^N(y_n - y_n^\text{pred})^2}
$$
Write a function that computes the RMSE.

```python
## EDIT THIS FUNCTION
def RMSE(y, ypred):
    rmse = np.sqrt(np.mean((y-ypred)**2)) ## SOLUTION
    return rmse
```

Now compute the RMSE for different degrees of the polynomial we want to fit.

```python
## EDIT THIS CELL
K_max = 20
rmse_train = np.zeros((K_max+1,))

for k in range(K_max+1):

    # feature matrix
    Phi = poly_features(X, k)

    # maximum likelihood estimate
    theta_ml = nonlinear_features_maximum_likelihood(Phi, y)

    # predict y-values of training set
    ypred_train = Phi @ theta_ml

    # RMSE on training set
    rmse_train[k] = RMSE(y, ypred_train)

plt.figure()
plt.plot(rmse_train)
plt.xlabel("degree of polynomial")
plt.ylabel("RMSE");
```

#### Question:
1. What do you observe?
2. What is the best polynomial fit according to this plot?
3. Write some code that plots the function that uses the best polynomial degree (use the test set for this plot). What do you observe now?

```python
# WRITE THE PLOTTING CODE HERE
plt.figure()
plt.plot(X, y, '+')

# feature matrix
Phi = poly_features(X, 5)

# maximum likelihood estimate
theta_ml = nonlinear_features_maximum_likelihood(Phi, y)

# feature matrix for test inputs
Phi_test = poly_features(Xtest, 5)

ypred_test = Phi_test @ theta_ml

plt.plot(Xtest, ypred_test)
plt.xlabel("$x$")
plt.ylabel("$y$")
plt.legend(["data", "maximum likelihood fit"]);
```

The RMSE on the training data is somewhat misleading, because we are interested in the generalization performance of the model. Therefore, we are going to compute the RMSE on the test set and use this to choose a good polynomial degree.

```python
## EDIT THIS CELL
K_max = 20
rmse_train = np.zeros((K_max+1,))
rmse_test = np.zeros((K_max+1,))

for k in range(K_max+1):

    # feature matrix
    Phi = poly_features(X, k)

    # maximum likelihood estimate
    theta_ml = nonlinear_features_maximum_likelihood(Phi, y)

    # predict y-values of training set
    ypred_train = Phi @ theta_ml

    # RMSE on training set
    rmse_train[k] = RMSE(y, ypred_train)

    # feature matrix for test inputs
    Phi_test = poly_features(Xtest, k)

    # prediction
    ypred_test = Phi_test @ theta_ml

    # RMSE on test set
    rmse_test[k] = RMSE(ytest, ypred_test)

plt.figure()
plt.semilogy(rmse_train) # this plots the RMSE on a logarithmic scale
plt.semilogy(rmse_test) # this plots the RMSE on a logarithmic scale
plt.xlabel("degree of polynomial")
plt.ylabel("RMSE")
plt.legend(["training set", "test set"]);
```

#### Questions:
1. What do you observe now?
2. Why does the RMSE for the test set not always go down?
3. Which polynomial degree would you choose now?
4. Plot the fit for the "best" polynomial degree.

```python
# WRITE THE PLOTTING CODE HERE
plt.figure()
plt.plot(X, y, '+')
k = 5
# feature matrix
Phi = poly_features(X, k)

# maximum likelihood estimate
theta_ml = nonlinear_features_maximum_likelihood(Phi, y)

# feature matrix for test inputs
Phi_test = poly_features(Xtest, k)

ypred_test = Phi_test @ theta_ml

plt.plot(Xtest, ypred_test)
plt.xlabel("$x$")
plt.ylabel("$y$")
plt.legend(["data", "maximum likelihood fit"]);
```

#### Question
If you did not have a designated test set, what could you do to estimate the generalization error (purely using the training set)?

## 2. Maximum A Posteriori Estimation

We are still considering the model
$$
y = \boldsymbol\phi(\boldsymbol x)^T\boldsymbol\theta + \epsilon\,,\quad \epsilon\sim\mathcal N(0,\sigma^2)\,.
$$
We assume that the noise variance $\sigma^2$ is known.

Instead of maximizing the likelihood, we can look at the maximum of the posterior distribution on the parameters $\boldsymbol\theta$, which is given as
$$
p(\boldsymbol\theta|\mathcal X, \mathcal Y) = \frac{\overbrace{p(\mathcal Y|\mathcal X, \boldsymbol\theta)}^{\text{likelihood}}\overbrace{p(\boldsymbol\theta)}^{\text{prior}}}{\underbrace{p(\mathcal Y|\mathcal X)}_{\text{evidence}}}
$$
The purpose of the parameter prior $p(\boldsymbol\theta)$ is to discourage the parameters to attain extreme values, a sign that the model overfits. The prior allows us to specify a "reasonable" range of parameter values. Typically, we choose a Gaussian prior $\mathcal N(\boldsymbol 0, \alpha^2\boldsymbol I)$, centered at $\boldsymbol 0$ with variance $\alpha^2$ along each parameter dimension.

The MAP estimate of the parameters is
$$
\boldsymbol\theta^{\text{MAP}} = (\boldsymbol\Phi^T\boldsymbol\Phi + \frac{\sigma^2}{\alpha^2}\boldsymbol I)^{-1}\boldsymbol\Phi^T\boldsymbol y
$$
where $\sigma^2$ is the variance of the noise.

```python
## EDIT THIS FUNCTION
def map_estimate_poly(Phi, y, sigma, alpha):
    # Phi: training inputs, Size of N x D
    # y: training targets, Size of D x 1
    # sigma: standard deviation of the noise
    # alpha: standard deviation of the prior on the parameters
    # returns: MAP estimate theta_map, Size of D x 1

    D = Phi.shape[1]

    # SOLUTION
    PP = Phi.T @ Phi + (sigma/alpha)**2 * np.eye(D)
    theta_map = scipy.linalg.solve(PP, Phi.T @ y)

    return theta_map
```

```python
# define the function we wish to estimate later
def g(x, sigma):
    p = np.hstack([x**0, x**1, np.sin(x)])
    w = np.array([-1.0, 0.1, 1.0]).reshape(-1,1)
    return p @ w + sigma*np.random.normal(size=x.shape)
```

```python
# Generate some data
sigma = 1.0 # noise standard deviation
alpha = 1.0 # standard deviation of the parameter prior
N = 20

np.random.seed(42)

X = (np.random.rand(N)*10.0 - 5.0).reshape(-1,1)
y = g(X, sigma) # training targets

plt.figure()
plt.plot(X, y, '+')
plt.xlabel("$x$")
plt.ylabel("$y$");
```

```python
# get the MAP estimate
K = 8 # polynomial degree

# feature matrix
Phi = poly_features(X, K)

theta_map = map_estimate_poly(Phi, y, sigma, alpha)

# maximum likelihood estimate
theta_ml = nonlinear_features_maximum_likelihood(Phi, y)

Xtest = np.linspace(-5,5,100).reshape(-1,1)
ytest = g(Xtest, sigma)

Phi_test = poly_features(Xtest, K)
y_pred_map = Phi_test @ theta_map

y_pred_mle = Phi_test @ theta_ml

plt.figure()
plt.plot(X, y, '+')
plt.plot(Xtest, y_pred_map)
plt.plot(Xtest, g(Xtest, 0))
plt.plot(Xtest, y_pred_mle)

plt.legend(["data", "map prediction", "ground truth function", "maximum likelihood"]);
```

```python
print(np.hstack([theta_ml, theta_map]))
```

    [[-1.49712990e+00 -1.08154986e+00]
     [ 8.56868912e-01  6.09177023e-01]
     [-1.28335730e-01 -3.62071208e-01]
     [-7.75319509e-02 -3.70531732e-03]
     [ 3.56425467e-02  7.43090617e-02]
     [-4.11626749e-03 -1.03278646e-02]
     [-2.48817783e-03 -4.89363010e-03]
     [ 2.70146690e-04  4.24148554e-04]
     [ 5.35996050e-05  1.03384719e-04]]

Now, let us compute the RMSE for different polynomial degrees and see whether the MAP estimate addresses the overfitting issue we encountered with the maximum likelihood estimate.

```python
## EDIT THIS CELL

K_max = 12 # this is the maximum degree of polynomial we will consider
assert(K_max < N) # this is the latest point when we'll run into numerical problems

rmse_mle = np.zeros((K_max+1,))
rmse_map = np.zeros((K_max+1,))

for k in range(K_max+1):

    # feature matrix
    Phi = poly_features(X, k)

    # maximum likelihood estimate
    theta_ml = nonlinear_features_maximum_likelihood(Phi, y)

    # predict the function values at the test input locations (maximum likelihood)
    y_pred_test = 0*Xtest ## <--- EDIT THIS LINE

    ####################### SOLUTION
    # feature matrix for test inputs
    Phi_test = poly_features(Xtest, k)

    # prediction
    ypred_test_mle = Phi_test @ theta_ml
    #######################

    # RMSE on test set (maximum likelihood)
    rmse_mle[k] = RMSE(ytest, ypred_test_mle)

    # MAP estimate
    theta_map = map_estimate_poly(Phi, y, sigma, alpha)

    # Feature matrix
    Phi_test = poly_features(Xtest, k)

    # predict the function values at the test input locations (MAP)
    ypred_test_map = Phi_test @ theta_map

    # RMSE on test set (MAP)
    rmse_map[k] = RMSE(ytest, ypred_test_map)

plt.figure()
plt.semilogy(rmse_mle) # this plots the RMSE on a logarithmic scale
plt.semilogy(rmse_map) # this plots the RMSE on a logarithmic scale
plt.xlabel("degree of polynomial")
plt.ylabel("RMSE")
plt.legend(["Maximum likelihood", "MAP"])
```

#### Questions:
1. What do you observe?
2. What is the influence of the prior variance on the parameters ($\alpha^2$)? Change the parameter and describe what happens.

## 3. Bayesian Linear Regression

```python
# Test inputs
Ntest = 200
Xtest = np.linspace(-5, 5, Ntest).reshape(-1,1) # test inputs

prior_var = 2.0 # variance of the parameter prior (alpha^2). We assume this is known.
noise_var = 1.0 # noise variance (sigma^2). We assume this is known.

pol_deg = 3 # degree of the polynomial we consider at the moment
```

Assume a parameter prior $p(\boldsymbol\theta) = \mathcal N (\boldsymbol 0, \alpha^2\boldsymbol I)$. For every test input $\boldsymbol x_*$ we obtain the
prior mean
$$
E[f(\boldsymbol x_*)] = 0
$$
and the prior (marginal) variance (ignoring the noise contribution)
$$
V[f(\boldsymbol x_*)] = \alpha^2\boldsymbol\phi(\boldsymbol x_*) \boldsymbol\phi(\boldsymbol x_*)^\top
$$
where $\boldsymbol\phi(\cdot)$ is the feature map.

```python
## EDIT THIS CELL

# compute the feature matrix for the test inputs
Phi_test = poly_features(Xtest, pol_deg) # N x (pol_deg+1) feature matrix SOLUTION

# compute the (marginal) prior at the test input locations
# prior mean
prior_mean = np.zeros((Ntest,1)) # prior mean <-- SOLUTION

# prior variance
full_covariance = Phi_test @ Phi_test.T * prior_var # N x N covariance matrix of all function values
prior_marginal_var =  np.diag(full_covariance)

# Let us visualize the prior over functions
plt.figure()
plt.plot(Xtest, prior_mean, color="k")

conf_bound1 = np.sqrt(prior_marginal_var).flatten()
conf_bound2 = 2.0*np.sqrt(prior_marginal_var).flatten()
conf_bound3 = 2.0*np.sqrt(prior_marginal_var + noise_var).flatten()
plt.fill_between(Xtest.flatten(), prior_mean.flatten() + conf_bound1,
             prior_mean.flatten() - conf_bound1, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), prior_mean.flatten() + conf_bound2,
                 prior_mean.flatten() - conf_bound2, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), prior_mean.flatten() + conf_bound3,
                 prior_mean.flatten() - conf_bound3, alpha = 0.1, color="k")

plt.xlabel('$x$')
plt.ylabel('$y$')
plt.title("Prior over functions");
```

Now, we will use this prior distribution and sample functions from it.

```python
## EDIT THIS CELL

# samples from the prior
num_samples = 10

# We first need to generate random weights theta_i, which we sample from the parameter prior
random_weights = np.random.normal(size=(pol_deg+1,num_samples), scale=np.sqrt(prior_var))

# Now, we compute the induced random functions, evaluated at the test input locations
# Every function sample is given as f_i = Phi * theta_i,
# where theta_i is a sample from the parameter prior

sample_function = Phi_test @ random_weights # <-- SOLUTION

plt.figure()
plt.plot(Xtest, sample_function, color="r")
plt.title("Plausible functions under the prior")
print("Every sampled function is a polynomial of degree "+str(pol_deg));
```

Now we are given some training inputs $\boldsymbol x_1, \dotsc, \boldsymbol x_N$, which we collect in a matrix $\boldsymbol X = [\boldsymbol x_1, \dotsc, \boldsymbol x_N]^\top\in\mathbb{R}^{N\times D}$

```python
N = 10
X = np.random.uniform(high=5, low=-5, size=(N,1)) # training inputs, size Nx1
y = g(X, np.sqrt(noise_var)) # training targets, size Nx1
```

Now, let us compute the posterior

```python
## EDIT THIS FUNCTION

def polyfit(X, y, K, prior_var, noise_var):
    # X: training inputs, size N x D
    # y: training targets, size N x 1
    # K: degree of polynomial we consider
    # prior_var: prior variance of the parameter distribution
    # sigma: noise variance

    jitter = 1e-08 # increases numerical stability

    Phi = poly_features(X, K) # N x (K+1) feature matrix

    # Compute maximum likelihood estimate
    Pt = Phi.T @ y # Phi*y, size (K+1,1)
    PP = Phi.T @ Phi + jitter*np.eye(K+1) # size (K+1, K+1)
    C = scipy.linalg.cho_factor(PP)
    # maximum likelihood estimate
    theta_ml = scipy.linalg.cho_solve(C, Pt) # inv(Phi^T*Phi)*Phi^T*y, size (K+1,1)

#     theta_ml = scipy.linalg.solve(PP, Pt) # inv(Phi^T*Phi)*Phi^T*y, size (K+1,1)

    # MAP estimate
    theta_map = scipy.linalg.solve(PP + noise_var/prior_var*np.eye(K+1), Pt)

    # parameter posterior
    iSN = (np.eye(K+1)/prior_var + PP/noise_var) # posterior precision
    SN = scipy.linalg.pinv(noise_var*np.eye(K+1)/prior_var + PP)*noise_var  # posterior covariance
    mN = scipy.linalg.solve(iSN, Pt/noise_var) # posterior mean

    return (theta_ml, theta_map, mN, SN)
```

```python
theta_ml, theta_map, theta_mean, theta_var = polyfit(X, y, pol_deg, alpha, sigma)
```

Now, let's make predictions (ignoring the measurement noise). We obtain three predictors:
\begin{align}
&\text{Maximum likelihood: }E[f(\boldsymbol X_{\text{test}})] = \boldsymbol \phi(X_{\text{test}})\boldsymbol \theta_{ml}\\
&\text{Maximum a posteriori: } E[f(\boldsymbol X_{\text{test}})] = \boldsymbol \phi(X_{\text{test}})\boldsymbol \theta_{map}\\
&\text{Bayesian: } p(f(\boldsymbol X_{\text{test}})) = \mathcal N(f(\boldsymbol X_{\text{test}}) \,|\, \boldsymbol \phi(X_{\text{test}}) \boldsymbol\theta_{\text{mean}},\, \boldsymbol\phi(X_{\text{test}}) \boldsymbol\theta_{\text{var}}  \boldsymbol\phi(X_{\text{test}})^\top)
\end{align}
We already computed all quantities. Write some code that implements all three predictors.

```python
## EDIT THIS CELL

# predictions (ignoring the measurement/observations noise)

Phi_test = poly_features(Xtest, pol_deg) # N x (K+1)

# maximum likelihood predictions (just the mean)
m_mle_test = Phi_test @ theta_ml

# MAP predictions (just the mean)
m_map_test = Phi_test @ theta_map

# predictive distribution (Bayesian linear regression)
# mean prediction
mean_blr = Phi_test @ theta_mean
# variance prediction
cov_blr =  Phi_test @ theta_var @ Phi_test.T
```

```python
# plot the posterior
plt.figure()
plt.plot(X, y, "+")
plt.plot(Xtest, m_mle_test)
plt.plot(Xtest, m_map_test)
var_blr = np.diag(cov_blr)
conf_bound1 = np.sqrt(var_blr).flatten()
conf_bound2 = 2.0*np.sqrt(var_blr).flatten()
conf_bound3 = 2.0*np.sqrt(var_blr + sigma).flatten()

plt.fill_between(Xtest.flatten(), mean_blr.flatten() + conf_bound1,
                 mean_blr.flatten() - conf_bound1, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), mean_blr.flatten() + conf_bound2,
                 mean_blr.flatten() - conf_bound2, alpha = 0.1, color="k")
plt.fill_between(Xtest.flatten(), mean_blr.flatten() + conf_bound3,
                 mean_blr.flatten() - conf_bound3, alpha = 0.1, color="k")
plt.legend(["Training data", "MLE", "MAP", "BLR"])
plt.xlabel('$x$');
plt.ylabel('$y$');
```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/MACHINE_LEARNING/MATHS_FOR_ML/LINEAR_REGRESSION_SOLUTION.ipynb


# SymPy

```python
from datascience import *
import numpy as np
from sympy import *
import sympy
init_printing()
import matplotlib.pyplot as plt
%matplotlib inline
solve = lambda x,y: sympy.solve(x-y)[0] if len(sympy.solve(x-y))==1 else "Not Single Solution"
```

Python has many tools, such as the [SymPy library](https://docs.sympy.org/latest/tutorial/index.html) that we can use for expressing and evaluating formulas and functions in economics.

Since SymPy helps with symbolic math, we start out by create a symbol using `Symbol`, which we assign to a variable name. Then, we can use the symbols to construct symbolic expressions.

```python
x = Symbol('x')
x
```

Now let's try using SymPy to create a symbolic expression for some hypothetical supply and demand curves.

To define an upward sloping supply curve with price expressed as a function of quantity, we start off defining the symbol $Q$, which represents quantity. Then, we set up a negative relationship expressing $P_S$, which denotes the price of the supplied good (how much the producer earns), in terms of $Q$.

```python
Q = Symbol('Q')
P_S = 2 * Q - 3
P_S
```

Similarly, we will also use $Q$ to express a relationship with $P_D$, the price of the good purchased (how much the consumer pays), creating a downward sloping demand curve.

Note that both functions are of the variable $Q$; this will be important in allowing us to solve for the equilibrium.

```python
P_D = 2 - Q
P_D
```

To solve for the equilibrium given the supply and demand curve, we know that the price paid by consumers must equal to the price earned by suppliers. Thus, $P_D = P_S$, allowing us to set the two equations equal to each other and solve for the equilibrium quantity and thus equilibrium price. To solve this by hand, we would set up the following equation to solve for $Q$:

$$
P_D = P_S\\
2-Q = 2Q-3
$$

Using SymPy, we call `solve`, which takes in 2 arguments that represent the 2 sides of an equation and solves for the underlying variable such that the equation holds. Here, we pass in $P_D$ and $P_S$, both represented in terms of $Q$, to solve for the value of $Q$ such that $P_D=P_S$. It's good to know that `solve` is a custom function built for this class, and will be provided in the notebooks for you.

```python
Q_star = solve(P_D, P_S)
Q_star
```

The value of $Q$ that equates $P_D$ and $P_S$ is known as the market equilibrium quantity, and we denote it as $Q^*$. Here, $Q^* = \frac{5}{3}$.

With $Q^*$ determined, we can substitute this value as $Q$ to thus calculate $P_D$ or $P_S$. We substitute values using the `subs` function, which follows the syntax `expression.subs(symbol_we_want_to_substitute, value_to_substitute_with)`.

```python
P_D.subs(Q, Q_star)
```

We can also substitute $Q^*$ into $P_S$, and should get the same results.

```python
P_S.subs(Q, Q_star)
```

Thus, the equilibrium price and quantity are \$0.33 and $\frac{5}{3}$, respectively.

Let's try another example. Suppose our demand function is $\text{Price}_{D}=-2 \cdot \text{Quantity} + 10$. Using SymPy, this would be

```python
demand = -2 * Q + 10
demand
```

In addition, let the supply function be $\text{Price}_{S}=3 \cdot \text{Quantity} + 1$. Using SymPy, this would be

```python
supply = 3 * Q + 1
supply
```

We will now try to find the market equilibrium. The market price equilibrium $P^*$ is the price at which the quantity supplied and quantity demanded of a good or service are equal to each other. Similarly, the market quantity equilibrium $Q^*$ is the quantity at which the price paid by consumers is equal to the price received by producers.

Combined, the price equilibrium and quantity equilibrium form a point on the graph with quantity and price as its axes, called the equilibrium point. This point is the point at which the demand and supply curves intersect.

First, we solve for the quantity equilibrium.

```python
Q_star = solve(demand, supply)
Q_star
```

Next, we plug the quantity equilibrium into our demand or supply expression to get the price equilibrium:

```python
demand.subs(Q, 9/5)
```

Graphically, we can plot the supply and demand curves with quantity on the $x$ axis and price on the $y$ axis. The point at which they intersect is the equilibrium point.

```python
def plot_equation(equation, price_start, price_end, label=None):
    plot_prices = [price_start, price_end]
    plot_quantities = [equation.subs(list(equation.free_symbols)[0], c) for c in plot_prices]
    plt.plot(plot_prices, plot_quantities, label=label)

def plot_intercept(eq1, eq2):
    ex = sympy.solve(eq1-eq2)[0]
    why = eq1.subs(list(eq1.free_symbols)[0], ex)
    plt.scatter([ex], [why])
    return (ex, why)

plot_equation(demand, 0, 5)
plot_equation(supply, 0, 5)
plt.ylim(0,20)
plt.xlabel("Quantity")
plt.ylabel("Price")
plot_intercept(supply, demand);
```