#
Linear Algebra for Data Scientists

One of the most common questions we get on Data science is:

How much maths do I need to learn to be a data scientist?

If you get confused and ask experts what should you learn at this stage, most of them would suggest / agree that you go ahead with Linear Algebra!
in this simple tutorials you can learn all of the thing you need for being a data scientist with Linear Algabra

last update: 11/07/2018

you can follow me on:

> ###### [ GitHub](https://github.com/mjbahmani)

-------------------------------------------------------------------------------------------------------------
 I hope you find this kernel helpful and some UPVOTES would be very much appreciated.

 -----------

## Notebook  Content
1. [Introduction](#1)
1. [Basic Concepts and Notation](#2)
1. [Notation ](#2)
1. [Matrix Multiplication](#3)
    1. [Vector-Vector Products](#4)
    1. [Outer Product of Two Vectors](#5)
    1. [Matrix-Vector Products](#6)
    1. [Matrix-Matrix Products](#7)
1. [Identity Matrix](#8)
1. [Diagonal Matrix](#9)
1. [Transpose of a Matrix](#10)
1. [Symmetric Metrices](#11)
1. [The Trace](#12)
1. [Norms](#13)
1. [Linear Independence and Rank](#14)
    1. [Column Rank of a Matrix](#15)
    1. [Row Rank of a Matrix](#16)
    1. [Rank of a Matrix](#17)
1. [Subtraction and Addition of Metrices](#18)
    1. [Inverse](#19)
1. [Orthogonal Matrices](#20)
1. [Range and Nullspace of a Matrix](#21)
1. [Determinant](#22)
    1. [geometric interpretation of the determinant](#23)
1. [Tensors](#24)
1. [Hyperplane](#25)
1. [Summary](#26)
    1. [Dot Product](#27)
    1. [Hadamard Product](#28)
    1. [Outer Product](#29)
1. [Eigenvalues and Eigenvectors](#30)
1. [Exercise](#31)
1. [Conclusion](#32)
1. [References](#33)

#  1-Introduction
Linear algebra is the branch of mathematics that deals with vector spaces. good understanding of Linear Algebra is intrinsic to analyze Machine Learning algorithms, especially for Deep Learning where so much happens behind the curtain.you have my word that I will try to keep mathematical formulas & derivations out of this completely mathematical topic and I try to cover all of subject that you need as data scientist.

*Is there anything more useless or less useful than Algebra?*

**Billy Connolly**

## 1-1 Import

```python
import matplotlib.pyplot as plt
import matplotlib.patches as patch
from scipy.stats import norm
from scipy import linalg
import pandas as pd
import numpy as np
import glob
import sys
import os
```

##  1-2 Setup

```python
%matplotlib inline
%precision 4
plt.style.use('ggplot')
np.set_printoptions(suppress=True)
```

# 2- Basic Concepts and Notation

The following system of equations:

$\begin{equation}
\begin{split}
4 x_1 - 5 x_2 & = -13 \\
 -2x_1 + 3 x_2 & = 9
\end{split}
\end{equation}$

We are looking for a unique solution for the two variables $x_1$ and $x_2$.  The system can be described as:

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

$$
Ax=b
$$

as matrices:

$$A = \begin{bmatrix}
       4  & -5 \\[0.3em]
       -2 &  3
     \end{bmatrix},\
 b = \begin{bmatrix}
       -13 \\[0.3em]
       9
     \end{bmatrix}$$

A **scalar** is an element in a vector, containing a real number **value**. In a vector space model or a vector mapping of (symbolic, qualitative, or quantitative) properties the scalar holds the concrete value or property of a variable.

A **vector** is an array, tuple, or ordered list of scalars (or elements) of size $n$, with $n$ a positive integer. The **length** of the vector, that is the number of scalars in the vector, is also called the **order** of the vector.

**Vectorization** is the process of creating a vector from some data using some process.

Vectors of the length $n$ could be treated like points in $n$-dimensional space. One can calculate the distance between such points using measures like [Euclidean Distance](https://en.wikipedia.org/wiki/Euclidean_distance). The similarity of vectors could also be calculated using [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity).

## 3- Notation

A **matrix** is a list of vectors that all are of the same length. $A$ is a matrix with $m$ rows and $n$ columns, antries of $A$ are real numbers:

$A \in \mathbb{R}^{m \times n}$

A vector $x$ with $n$ entries of real numbers, could also be thought of as a matrix with $n$ rows and $1$ column, or as known as a **column vector**.

$x = \begin{bmatrix}
       x_1 \\[0.3em]
       x_2 \\[0.3em]
       \vdots \\[0.3em]
       x_n
     \end{bmatrix}$

Representing a **row vector**, that is a matrix with $1$ row and $n$ columns, we write $x^T$ (this denotes the transpose of $x$, see above).

$x^T = \begin{bmatrix}
       x_1 & x_2 & \cdots & x_n
     \end{bmatrix}$

We use the notation $a_{ij}$ (or $A_{ij}$, $A_{i,j}$, etc.) to denote the entry of $A$ in the $i$th row and
$j$th column:

$A = \begin{bmatrix}
       a_{11} & a_{12} & \cdots & a_{1n} \\[0.3em]
       a_{21} & a_{22} & \cdots & a_{2n} \\[0.3em]
       \vdots & \vdots & \ddots & \vdots \\[0.3em]
       a_{m1} & a_{m2} & \cdots & a_{mn}
     \end{bmatrix}$

We denote the $j$th column of $A$ by $a_j$ or $A_{:,j}$:

$A = \begin{bmatrix}
       \big| & \big| &  & \big| \\[0.3em]
       a_{1} & a_{2} & \cdots & a_{n} \\[0.3em]
       \big| & \big| &  & \big|
     \end{bmatrix}$

We denote the $i$th row of $A$ by $a_i^T$ or $A_{i,:}$:

$A = \begin{bmatrix}
      -- & a_1^T  & -- \\[0.3em]
       -- & a_2^T  & -- \\[0.3em]
          & \vdots &  \\[0.3em]
       -- & a_m^T  & --
     \end{bmatrix}$

A $n \times m$ matrix is a two-dimensional array with $n$ rows and $m$ columns.

## 4-Matrix Multiplication

The result of the multiplication of two matrixes $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times p}$ is the matrix:

$C = AB \in \mathbb{R}^{m \times n}$

That is, we are multiplying the columns of $A$ with the rows of $B$:

$C_{ij}=\sum_{k=1}^n{A_{ij}B_{kj}}$

The number of columns in $A$ must be equal to the number of rows in $B$.

## 4-1 Vector-Vector Products

#### Inner or Dot Product of Two Vectors

For two vectors $x, y \in \mathbb{R}^n$, the **inner product** or **dot product** $x^T y$ is a real number:

$x^T y \in \mathbb{R} = \begin{bmatrix}
       x_1 & x_2 & \cdots & x_n
     \end{bmatrix} \begin{bmatrix}
       y_1 \\[0.3em]
       y_2 \\[0.3em]
       \vdots \\[0.3em]
       y_n
     \end{bmatrix} = \sum_{i=1}^{n}{x_i y_i}$

The **inner products** are a special case of matrix multiplication.

It is always the case that $x^T y = y^T x$.

##### Example

To calculate the inner product of two vectors $x = [1 2 3 4]$ and $y = [5 6 7 8]$, we can loop through the vector and multiply and sum the scalars (this is simplified code):

```python
x = (1, 2, 3, 4)
y = (5, 6, 7, 8)
n = len(x)
if n == len(y):
    result = 0
    for i in range(n):
        result += x[i] * y[i]
    print(result)
```

    70

It is clear that in the code above we could change line 7 to `result += y[i] * x[i]` without affecting the result.

We can use the *numpy* module to apply the same operation, to calculate the **inner product**. We import the *numpy* module and assign it a name *np* for the following code:

```python
import numpy as np
```

We define the vectors $x$ and $y$ using *numpy*:

```python
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
print("x:", x)
print("y:", y)
```

    x: [1 2 3 4]
    y: [5 6 7 8]

We can now calculate the $dot$ or $inner product$ using the *dot* function of *numpy*:

```python
np.dot(x, y)
```

    70

The order of the arguments is irrelevant:

```python
np.dot(y, x)
```

    70

Note that both vectors are actually **row vectors** in the above code. We can transpose them to column vectors by using the *shape* property:

```python
print("x:", x)
x.shape = (4, 1)
print("xT:", x)
print("y:", y)
y.shape = (4, 1)
print("yT:", y)
```

    x: [1 2 3 4]
    xT: [[1]
     [2]
     [3]
     [4]]
    y: [5 6 7 8]
    yT: [[5]
     [6]
     [7]
     [8]]

In fact, in our understanding of Linear Algebra, we take the arrays above to represent **row vectors**. *Numpy* treates them differently.

We see the issues when we try to transform the array objects. Usually, we can transform a row vector into a column vector in *numpy* by using the *T* method on vector or matrix objects:

```python
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
print("x:", x)
print("y:", y)
print("xT:", x.T)
print("yT:", y.T)
```

    x: [1 2 3 4]
    y: [5 6 7 8]
    xT: [1 2 3 4]
    yT: [5 6 7 8]

The problem here is that this does not do, what we expect it to do. It only works, if we declare the variables not to be arrays of numbers, but in fact a matrix:

```python
x = np.array([[1, 2, 3, 4]])
y = np.array([[5, 6, 7, 8]])
print("x:", x)
print("y:", y)
print("xT:", x.T)
print("yT:", y.T)

```

    x: [[1 2 3 4]]
    y: [[5 6 7 8]]
    xT: [[1]
     [2]
     [3]
     [4]]
    yT: [[5]
     [6]
     [7]
     [8]]

Note that the *numpy* functions *dot* and *outer* are not affected by this distinction. We can compute the dot product using the mathematical equation above in *numpy* using the new $x$ and $y$ row vectors:

```python
print("x:", x)
print("y:", y.T)
np.dot(x, y.T)
```

    x: [[1 2 3 4]]
    y: [[5]
     [6]
     [7]
     [8]]

    array([[70]])

Or by reverting to:

```python
print("x:", x.T)
print("y:", y)
np.dot(y, x.T)
```

    x: [[1]
     [2]
     [3]
     [4]]
    y: [[5 6 7 8]]

    array([[70]])

To read the result from this array of arrays, we would need to access the value this way:

```python
np.dot(y, x.T)[0][0]
```

    70

## 4-2 Outer Product of Two Vectors

For two vectors $x \in \mathbb{R}^m$ and $y \in \mathbb{R}^n$, where $n$ and $m$ do not have to be equal, the **outer product** of $x$ and $y$ is:

$xy^T \in \mathbb{R}^{m\times n}$

The **outer product** results in a matrix with $m$ rows and $n$ columns by $(xy^T)_{ij} = x_i y_j$:

$xy^T \in \mathbb{R}^{m\times n} = \begin{bmatrix}
       x_1 \\[0.3em]
       x_2 \\[0.3em]
       \vdots \\[0.3em]
       x_n
     \end{bmatrix} \begin{bmatrix}
       y_1 & y_2 & \cdots & y_n
     \end{bmatrix} = \begin{bmatrix}
       x_1 y_1 & x_1 y_2 & \cdots & x_1 y_n \\[0.3em]
       x_2 y_1 & x_2 y_2 & \cdots & x_2 y_n \\[0.3em]
       \vdots  & \vdots  & \ddots & \vdots \\[0.3em]
       x_m y_1 & x_m y_2 & \cdots & x_m y_n \\[0.3em]
     \end{bmatrix}$

Some useful property of the outer product: assume $\mathbf{1} \in \mathbb{R}^n$ is an $n$-dimensional vector of scalars with the value $1$. Given a matrix $A \in \mathbb{R}^{m\times n}$ with all columns equal to some vector $x \in \mathbb{R}^m$, using the outer product $A$ can be represented as:

$A = \begin{bmatrix}
       \big| & \big| &  & \big| \\[0.3em]
       x & x & \cdots & x \\[0.3em]
       \big| & \big| &  & \big|
     \end{bmatrix} = \begin{bmatrix}
       x_1    & x_1    & \cdots & x_1    \\[0.3em]
       x_2    & x_2    & \cdots & x_2    \\[0.3em]
       \vdots & \vdots & \ddots & \vdots \\[0.3em]
       x_m    &x_m     & \cdots & x_m
     \end{bmatrix} = \begin{bmatrix}
       x_1 \\[0.3em]
       x_2 \\[0.3em]
       \vdots \\[0.3em]
       x_m
     \end{bmatrix} \begin{bmatrix}
       1 & 1 & \cdots & 1
     \end{bmatrix} = x \mathbf{1}^T$

##### Example

If we want to compute the outer product of two vectors $x$ and $y$, we need to transpose the row vector $x$ to a column vector $x^T$. This can be achieved by the *reshape* function in *numpy*, the *T* method, or the *transpose()* function. The *reshape* function takes a parameter that describes the number of colums and rows for the resulting transposing:

```python
x = np.array([[1, 2, 3, 4]])
print("x:", x)
print("xT:", np.reshape(x, (4, 1)))
print("xT:", x.T)
print("xT:", x.transpose())
```

    x: [[1 2 3 4]]
    xT: [[1]
     [2]
     [3]
     [4]]
    xT: [[1]
     [2]
     [3]
     [4]]
    xT: [[1]
     [2]
     [3]
     [4]]

We can now compute the **outer product** by multiplying the column vector $x$ with the row vector $y$:

```python
x = np.array([[1, 2, 3, 4]])
y = np.array([[5, 6, 7, 8]])
x.T * y
```

    array([[ 5,  6,  7,  8],
           [10, 12, 14, 16],
           [15, 18, 21, 24],
           [20, 24, 28, 32]])

*Numpy* provides an *outer* function that does all that:

```python
np.outer(x, y)
```

    array([[ 5,  6,  7,  8],
           [10, 12, 14, 16],
           [15, 18, 21, 24],
           [20, 24, 28, 32]])

Note, in this simple case using the simple arrays for the data structures of the vectors does not affect the result of the *outer* function:

```python
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
np.outer(x, y)
```

    array([[ 5,  6,  7,  8],
           [10, 12, 14, 16],
           [15, 18, 21, 24],
           [20, 24, 28, 32]])

## 4-3 Matrix-Vector Products

Assume a matrix $A \in \mathbb{R}^{m\times n}$ and a vector $x \in \mathbb{R}^n$ the product results in a vector $y = Ax \in \mathbb{R}^m$.

$Ax$ could be expressed as the dot product of row $i$ of matrix $A$ with the column value $j$ of vector $x$. Let us first consider matrix multiplication with a scalar:

$A = \begin{bmatrix}
       1 & 2 \\[0.3em]
       3 & 4
     \end{bmatrix}$

We can compute the product of $A$ with a scalar $n = 2$ as:

$A = \begin{bmatrix}
       1 * n & 2 * n \\[0.3em]
       3 * n & 4 * n
     \end{bmatrix} = \begin{bmatrix}
       1 * 2 & 2 * 2 \\[0.3em]
       3 * 2 & 4 * 2
     \end{bmatrix} = \begin{bmatrix}
       2 & 4 \\[0.3em]
       6 & 8
     \end{bmatrix} $

Using *numpy* this can be achieved by:

```python
import numpy as np
A = np.array([[4, 5, 6],
             [7, 8, 9]])
A * 2
```

    array([[ 8, 10, 12],
           [14, 16, 18]])

Assume that we have a column vector $x$:

$x = \begin{bmatrix}
       1 \\[0.3em]
       2 \\[0.3em]
       3
     \end{bmatrix}$

To be able to multiply this vector with a matrix, the number of columns in the matrix must correspond to the number of rows in the column vector. The matrix $A$ must have $3$ columns, as for example:

$A = \begin{bmatrix}
       4 & 5 & 6\\[0.3em]
       7 & 8 & 9
     \end{bmatrix}$

To compute $Ax$, we multiply row $1$ of the matrix with column $1$ of $x$:

$\begin{bmatrix}
  4 & 5 & 6
 \end{bmatrix}
 \begin{bmatrix}
 1 \\[0.3em]
 2 \\[0.3em]
 3
\end{bmatrix} = 4 * 1 + 5 * 2 + 6 * 3 = 32 $

We do the compute the dot product of row $2$ of $A$ and column $1$ of $x$:

$\begin{bmatrix}
  7 & 8 & 9
 \end{bmatrix}
 \begin{bmatrix}
 1 \\[0.3em]
 2 \\[0.3em]
 3
\end{bmatrix} = 7 * 1 + 8 * 2 + 9 * 3 = 50 $

The resulting column vector $Ax$ is:

$Ax = \begin{bmatrix}
       32 \\[0.3em]
       50
     \end{bmatrix}$

Using *numpy* we can compute $Ax$:

```python
A = np.array([[4, 5, 6],
             [7, 8, 9]])
x = np.array([1, 2, 3])
A.dot(x)
```

    array([32, 50])

We can thus describe the product writing $A$ by rows as:

$y = Ax = \begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix} x = \begin{bmatrix}
 a_1^T x \\[0.3em]
 a_2^T x \\[0.3em]
 \vdots \\[0.3em]
 a_m^T x
\end{bmatrix}$

This means that the $i$th scalar of $y$ is the inner product of the $i$th row of $A$ and $x$, that is $y_i = a_i^T x$.

If we write $A$ in column form, then:

$y = Ax =
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 a_1 & a_2 & \cdots & a_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix}
\begin{bmatrix}
 x_1 \\[0.3em]
 x_2 \\[0.3em]
 \vdots \\[0.3em]
 x_n
\end{bmatrix} =
\begin{bmatrix}
 a_1
\end{bmatrix} x_1 +
\begin{bmatrix}
 a_2
\end{bmatrix} x_2 + \dots +
\begin{bmatrix}
 a_n
\end{bmatrix} x_n
$

In this case $y$ is a **[linear combination](https://en.wikipedia.org/wiki/Linear_combination)** of the *columns* of $A$, the coefficients taken from $x$.

The above examples multiply be the right with a column vector. One can multiply on the left by a row vector as well, $y^T = x^T A$ for $A \in \mathbb{R}^{m\times n}$, $x\in \mathbb{R}^m$, $y \in \mathbb{R}^n$. There are two ways to express $y^T$, with $A$ expressed by its columns, with $i$th scalar of $y^T$ corresponds to the inner product of $x$ and the $i$th column of $A$:

$y^T = x^T A = x^t \begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 a_1 & a_2 & \cdots & a_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} =
\begin{bmatrix}
 x^T a_1 & x^T a_2 & \dots & x^T a_n
\end{bmatrix}$

One can express $A$ by rows, where $y^T$ is a linear combination of the rows of $A$ with the scalars from $x$.

$\begin{equation}
\begin{split}
y^T & = x^T A \\
    & = \begin{bmatrix}
 x_1 & x_2 & \dots & x_n
\end{bmatrix}
\begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix} \\
   & = x_1 \begin{bmatrix}-- & a_1^T  & --\end{bmatrix} + x_2 \begin{bmatrix}-- & a_2^T  & --\end{bmatrix} + \dots + x_n \begin{bmatrix}-- & a_n^T  & --\end{bmatrix}
\end{split}
\end{equation}$

## 4-4 Matrix-Matrix Products

One can view matrix-matrix multiplication $C = AB$ as a set of vector-vector products. The $(i,j)$th entry of $C$ is the inner product of the $i$th row of $A$ and the $j$th column of $B$:

```python
matrix1 = np.matrix(
    [[0, 4],
     [2, 0]]
)
matrix2 = np.matrix(
    [[-1, 2],
     [1, -2]]
)
```

```python
matrix1 + matrix2
```

    matrix([[-1,  6],
            [ 3, -2]])

### 4-4-1  Multiplication
To multiply two matrices with numpy, you can use the np.dot method:

```python
np.dot(matrix1, matrix2)
```

    matrix([[ 4, -8],
            [-2,  4]])

```python

matrix1 * matrix2
```

    matrix([[ 4, -8],
            [-2,  4]])

$C = AB =
\begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 b_1 & b_2 & \cdots & b_p \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} =
\begin{bmatrix}
 a_1^T b_1 & a_1^T b_2 & \cdots & a_1^T b_p \\[0.3em]
 a_2^T b_1 & a_2^T b_2 & \cdots & a_2^T b_p \\[0.3em]
 \vdots    & \vdots    & \ddots & \vdots    \\[0.3em]
 a_m^T b_1 & a_m^T b_2 & \cdots & a_m^T b_p
\end{bmatrix}$

Here $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{n\times p}$, $a_i \in \mathbb{R}^n$ and $b_j \in \mathbb{R}^n$, and $A$ is represented by rows, $B$ by columns.

If we represent $A$ by columns and $B$ by rows, then $AB$ is the sum of the outer products:

$C = AB =
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 a_1 & a_2 & \cdots & a_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix}
\begin{bmatrix}
 -- & b_1^T  & -- \\[0.3em]
 -- & b_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & b_n^T  & --
\end{bmatrix}
= \sum_{i=1}^n a_i b_i^T
$

This means that $AB$ is the sum over all $i$ of the outer product of the $i$th column of $A$ and the $i$th row of $B$.

One can interpret matrix-matrix operations also as a set of matrix-vector products. Representing $B$ by columns, the columns of $C$ are matrix-vector products between $A$ and the columns of $B$:

$C = AB = A
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 b_1 & b_2 & \cdots & b_p \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} =
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 A b_1 & A b_2 & \cdots & A b_p \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix}
$

In this interpretation the $i$th column of $C$ is the matrix-vector product with the vector on the right, i.e. $c_i = A b_i$.

Representing $A$ by rows, the rows of $C$ are the matrix-vector products between the rows of $A$ and $B$:

$C = AB = \begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}
B =
\begin{bmatrix}
 -- & a_1^T B & -- \\[0.3em]
 -- & a_2^T B & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_n^T B & --
\end{bmatrix}$

The $i$th row of $C$ is the matrix-vector product with the vector on the left, i.e. $c_i^T = a_i^T B$.

#### Notes on Matrix-Matrix Products

**Matrix multiplication is associative:** $(AB)C = A(BC)$

**Matrix multiplication is distributive:** $A(B + C) = AB + AC$

**Matrix multiplication is, in general, not commutative;** It can be the case that $AB \neq BA$. (For example, if $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{n\times q}$, the matrix product $BA$ does not even exist if $m$ and $q$ are not equal!)

## 5- Identity Matrix

The **identity matrix** $I \in \mathbb{R}^{n\times n}$ is a square matrix with the value $1$ on the diagonal and $0$ everywhere else:

```python
np.eye(4)
```

    array([[1., 0., 0., 0.],
           [0., 1., 0., 0.],
           [0., 0., 1., 0.],
           [0., 0., 0., 1.]])

$I_{ij} = \left\{
\begin{array}{lr}
 1 & i = j\\
 0 & i \neq j
\end{array}
\right.
$

For all $A \in \mathbb{R}^{m\times n}$:

$AI = A = IA$

In the equation above multiplication has to be made possible, which means that in the portion $AI = A$ the dimensions of $I$ have to be $n\times n$, while in $A = IA$ they have to be $m\times m$.

We can generate an *identity matrix* in *numpy* using:

```python
import numpy as np
A = np.array([[0, 1, 2],
              [3, 4, 5],
              [6, 7, 8],
              [9, 10, 11]])
print("A:", A)
```

    A: [[ 0  1  2]
     [ 3  4  5]
     [ 6  7  8]
     [ 9 10 11]]

We can ask for the shape of $A$:

```python
A.shape
```

    (4, 3)

The *shape* property of a matrix contains the $m$ (number of rows) and $n$ (number of columns) properties in a tuple, in that particular order. We can create an identity matrix for the use in $AI$ by using the $n$ value:

```python
np.identity(A.shape[1], dtype="int")
```

    array([[1, 0, 0],
           [0, 1, 0],
           [0, 0, 1]])

Note that we specify the *dtype* parameter to *identity* as *int*, since the default would return a matrix of *float* values.

To generate an identity matrix for the use in $IA$ we would use the $m$ value:

```python
np.identity(A.shape[0], dtype="int")
```

    array([[1, 0, 0, 0],
           [0, 1, 0, 0],
           [0, 0, 1, 0],
           [0, 0, 0, 1]])

We can compute the dot product of $A$ and its identity matrix $I$:

```python
n = A.shape[1]
I = np.array(np.identity(n, dtype="int"))
np.dot(A, I)
```

    array([[ 0,  1,  2],
           [ 3,  4,  5],
           [ 6,  7,  8],
           [ 9, 10, 11]])

The same is true for the other direction:

```python
m = A.shape[0]
I = np.array(np.identity(m, dtype="int"))
np.dot(I, A)
```

    array([[ 0,  1,  2],
           [ 3,  4,  5],
           [ 6,  7,  8],
           [ 9, 10, 11]])

### 5-1  Inverse Matrices

```python
inverse = np.linalg.inv(matrix1)
print(inverse)
```

    [[0.   0.5 ]
     [0.25 0.  ]]

## 6- Diagonal Matrix

In the **diagonal matrix** non-diagonal elements are $0$, that is $D = diag(d_1, d_2, \dots{}, d_n)$, with:

$D_{ij} = \left\{
\begin{array}{lr}
 d_i & i = j\\
 0 & i \neq j
\end{array}
\right.
$

The identity matrix is a special case of a diagonal matrix: $I = diag(1, 1, \dots{}, 1)$.

In *numpy* we can create a *diagonal matrix* from any given matrix using the *diag* function:

```python
import numpy as np
A = np.array([[0,   1,  2,  3],
              [4,   5,  6,  7],
              [8,   9, 10, 11],
              [12, 13, 14, 15]])
np.diag(A)
```

    array([ 0,  5, 10, 15])

An optional parameter *k* to the *diag* function allows us to extract the diagonal above the main diagonal with a positive *k*, and below the main diagonal with a negative *k*:

```python
np.diag(A, k=1)
```

    array([ 1,  6, 11])

```python
np.diag(A, k=-1)
```

    array([ 4,  9, 14])

## 7- Transpose of a Matrix

**Transposing** a matrix is achieved by *flipping* the rows and columns. For a matrix $A \in \mathbb{R}^{m\times n}$ the transpose $A^T \in \mathbb{R}^{n\times m}$ is the $n\times m$ matrix given by:

$(A^T)_{ij} = A_{ji}$

Properties of transposes:

- $(A^T)^T = A$
- $(AB)^T = B^T A^T$
- $(A+B)^T = A^T + B^T$

## 8- Symmetric Matrices

Square metrices $A \in \mathbb{R}^{n\times n}$ are **symmetric**, if $A = A^T$.

$A$ is **anti-symmetric**, if $A = -A^T$.

For any matrix $A \in \mathbb{R}^{n\times n}$, the matrix $A + A^T$ is **symmetric**.

For any matrix $A \in \mathbb{R}^{n\times n}$, the matrix $A - A^T$ is **anti-symmetric**.

Thus, any square matrix $A \in \mathbb{R}^{n\times n}$ can be represented as a sum of a symmetric matrix and an anti-symmetric matrix:

$A = \frac{1}{2} (A + A^T) + \frac{1}{2} (A - A^T)$

The first matrix on the right, i.e. $\frac{1}{2} (A + A^T)$ is symmetric. The second matrix $\frac{1}{2} (A - A^T)$ is anti-symmetric.

$\mathbb{S}^n$ is the set of all symmetric matrices of size $n$.

$A \in \mathbb{S}^n$ means that $A$ is symmetric and of the size $n\times n$.

## 9-The Trace

The **trace** of a square matrix $A \in \mathbb{R}^{n\times n}$ is $tr(A)$ (or $trA$) is the sum of the diagonal elements in the matrix:

$trA = \sum_{i=1}^n A_{ii}$

Properties of the **trace**:

- For $A \in \mathbb{R}^{n\times n}$, $\mathrm{tr}A = \mathrm{tr}A^T$
- For $A,B \in \mathbb{R}^{n\times n}$, $\mathrm{tr}(A + B) = \mathrm{tr}A + \mathrm{tr}B$
- For $A \in \mathbb{R}^{n\times n}$, $t \in \mathbb{R}$, $\mathrm{tr}(tA) = t \mathrm{tr}A$
- For $A,B$ such that $AB$ is square, $\mathrm{tr}AB = \mathrm{tr}BA$
- For $A,B,C$ such that $ABC$ is square, $\mathrm{tr}ABC = \mathrm{tr}BCA = \mathrm{tr}CAB$, and so on for the product of more matrices.

```python
a = np.arange(8).reshape((2,2,2))
np.trace(a)
```

    array([6, 8])

```python
print(np.trace(matrix1))
```

    0

```python
det = np.linalg.det(matrix1)
print(det)
```

    -7.999999999999998

```python
a = np.array([[1, 2], [3, 4]])
a
```

    array([[1, 2],
           [3, 4]])

```python

a.transpose()
```

    array([[1, 3],
           [2, 4]])

# 10- Norms

The **norm** of a vector $x$ is $\| x\|$, informally the length of a vector.

Example: the Euclidean or $\mathscr{l}_2$ norm:

$\|x\|_2 = \sqrt{\sum_{i=1}^n{x_i^2}}$

Note: $\|x\|_2^2 = x^T x$

A **norm** is any function $f : \mathbb{R}^n \rightarrow \mathbb{R}$ that satisfies the following properties:

- For all $x \in \mathbb{R}^n$, $f(x) \geq 0$ (non-negativity)
- $f(x) = 0$ if and only if $x = 0$ (definiteness)
- For all $x \in \mathbb{R}^n$, $t \in \mathbb{R}$, $f(tx) = |t|\ f(x)$ (homogeneity)
- For all $x, y \in \mathbb{R}^n$, $f(x + y) \leq f(x) + f(y)$ (triangle inequality)

Norm $\mathscr{l}_1$:

$\|x\|_1 = \sum_{i=1}^n{|x_i|}$

Norm $\mathscr{l}_\infty$:

$\|x\|_\infty = \max_i|x_i|$

All these three norms are examples of the $\mathscr{l}_p$ norms, with $p$ a real number parameter $p \geq 1$:

$\|x\|_p = \left(\sum_{i=1}^n{|x_i|^p}\right)^{\frac{1}{p}}$

*Frobenius norm* for matrices:

$\|A\|_F = \sqrt{\sum_{i=1}^m\sum_{i=1}^n A_{ij}^2} = \sqrt{\mathrm{tr}(A^T A)}$

And many more.

```python
v = np.array([1,2,3,4])
norm.median(v)
```

    array([1., 2., 3., 4.])

# 11- Linear Independence and Rank

A set of vectors $\{x_1, x_2, \dots{}, x_n\} \subset \mathbb{R}^m$ is said to be **(linearly) independent** if no vector can be represented as a linear combination of the remaining vectors.

A set of vectors $\{x_1, x_2, \dots{}, x_n\} \subset \mathbb{R}^m$ is said to be **(lineraly) dependent** if one vector from this set can be represented as a linear combination of the remaining vectors.

For some scalar values $\alpha_1, \dots{}, \alpha_{n-1} \in \mathbb{R}$ the vectors $x_1, \dots{}, x_n$ are linerly dependent, if:

$\begin{equation}
x_n = \sum_{i=1}^{n-1}{\alpha_i x_i}
\end{equation}$

Example: The following vectors are lineraly dependent, because $x_3 = -2 x_1 + x_2$

$x_1 = \begin{bmatrix}
 1 \\[0.3em]
 2 \\[0.3em]
 3
\end{bmatrix}
\quad
x_2 = \begin{bmatrix}
 4 \\[0.3em]
 1 \\[0.3em]
 5
\end{bmatrix}
\quad
x_3 = \begin{bmatrix}
 2 \\[0.3em]
 -1 \\[0.3em]
 -1
\end{bmatrix}
$

## 11-1 Column Rank of a Matrix

The **column rank** of a matrix $A \in \mathbb{R}^{m\times n}$ is the size of the largest subset of columns of $A$ that constitute a linear independent set. Informaly this is the number of linearly independent columns of $A$.

## 11-2 Row Rank of a Matrix

The **row rank** of a matrix $A \in \mathbb{R}^{m\times n}$ is the largest number of rows of $A$ that constitute a lineraly independent set.

## 11-3 Rank of a Matrix

For any matrix $A \in \mathbb{R}^{m\times n}$, the column rank of $A$ is equal to the row rank of $A$. Both quantities are referred to collectively as the rank of $A$, denoted as $rank(A)$. Here are some basic properties of the rank:

- For $A \in \mathbb{R}^{m\times n}$, $rank(A) \leq \min(m, n)$. If $rank(A) = \min(m, n)$, then $A$ is said to be
**full rank**.
- For $A \in \mathbb{R}^{m\times n}$, $rank(A) = rank(A^T)$
- For $A \in \mathbb{R}^{m\times n}$, $B \in \mathbb{R}^{n\times p}$, $rank(AB) \leq \min(rank(A), rank(B))$
- For $A,B \in \mathbb{R}^{m\times n}$, $rank(A + B) \leq rank(A) + rank(B)$

# 12-  Subtraction and Addition of Metrices

Assume $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{m\times n}$, that is $A$ and $B$ are of the same size, to add $A$ to $B$, or to subtract $B$ from $A$, we add or subtract corresponding entries:

$A + B =
\begin{bmatrix}
 a_{11} & a_{12} & \cdots & a_{1n} \\[0.3em]
 a_{21} & a_{22} & \cdots & a_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} & a_{m2} & \cdots & a_{mn}
\end{bmatrix} +
\begin{bmatrix}
 b_{11} & b_{12} & \cdots & b_{1n} \\[0.3em]
 b_{21} & b_{22} & \cdots & b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 b_{m1} & b_{m2} & \cdots & b_{mn}
\end{bmatrix} =
\begin{bmatrix}
 a_{11} + b_{11} & a_{12} + b_{12} & \cdots & a_{1n} + b_{1n} \\[0.3em]
 a_{21} + b_{21} & a_{22} + b_{22} & \cdots & a_{2n} + b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} + b_{m1} & a_{m2} + b_{m2} & \cdots & a_{mn} + b_{mn}
\end{bmatrix}
$

The same is applies to subtraction:

$A - B =
\begin{bmatrix}
 a_{11} & a_{12} & \cdots & a_{1n} \\[0.3em]
 a_{21} & a_{22} & \cdots & a_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} & a_{m2} & \cdots & a_{mn}
\end{bmatrix} -
\begin{bmatrix}
 b_{11} & b_{12} & \cdots & b_{1n} \\[0.3em]
 b_{21} & b_{22} & \cdots & b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 b_{m1} & b_{m2} & \cdots & b_{mn}
\end{bmatrix} =
\begin{bmatrix}
 a_{11} - b_{11} & a_{12} - b_{12} & \cdots & a_{1n} - b_{1n} \\[0.3em]
 a_{21} - b_{21} & a_{22} - b_{22} & \cdots & a_{2n} - b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} - b_{m1} & a_{m2} - b_{m2} & \cdots & a_{mn} - b_{mn}
\end{bmatrix}
$

In Python using *numpy* this can be achieved using the following code:

```python
import numpy as np
print("np.arange(9):", np.arange(9))
print("np.arange(9, 18):", np.arange(9, 18))
A = np.arange(9, 18).reshape((3, 3))
B = np.arange(9).reshape((3, 3))
print("A:", A)
print("B:", B)
```

    np.arange(9): [0 1 2 3 4 5 6 7 8]
    np.arange(9, 18): [ 9 10 11 12 13 14 15 16 17]
    A: [[ 9 10 11]
     [12 13 14]
     [15 16 17]]
    B: [[0 1 2]
     [3 4 5]
     [6 7 8]]

The *numpy* function *arange* is similar to the standard Python function *range*. It returns an array with $n$ elements, specified in the one parameter version only. If we provide to parameters to *arange*, it generates an array starting from the value of the first parameter and ending with a value one less than the second parameter. The function *reshape* returns us a matrix with the corresponding number of rows and columns.

We can now add and subtract the two matrices $A$ and $B$:

```python
A + B
```

    array([[ 9, 11, 13],
           [15, 17, 19],
           [21, 23, 25]])

```python
A - B
```

    array([[9, 9, 9],
           [9, 9, 9],
           [9, 9, 9]])

## 12-1 Inverse

The **inverse** of a square matrix $A \in \mathbb{R}^{n\times n}$ is $A^{-1}$:

$A^{-1} A = I = A A^{-1}$

Not all matrices have inverses. Non-square matrices do not have inverses by definition. For some square matrices $A$ the inverse might not exist.

$A$ is **invertible** or **non-singular** if $A^{-1}$ exists.

$A$ is **non-invertible** or **singular** if $A^{-1}$ does not exist.

Note: **non-singular** means the opposite of **non-invertible**!

For $A$ to have an inverse $A^{-1}$, $A$ must be **full rank**.

Assuming that $A,B \in \mathbb{R}^{n\times n}$ are non-singular, then:

- $(A^{-1})^{-1} = A$
- $(AB)^{-1} = B^{-1} A^{-1}$
- $(A^{-1})^T = (A^T)^{-1}$ (often simply $A^{-T}$)

## 13- Orthogonal Matrices

Two vectors $x, y \in \mathbb{R}^n$ are **orthogonal** if $x^T y = 0$.

A vector $x \in \mathbb{R}^n$ is **normalized** if $\|x\|^2 = 1$.

A square matrix $U \in \mathbb{R}^{n\times n}$ is **orthogonal** if all its columns are orthogonal to each other and are **normalized**. The columns are then referred to as being **orthonormal**.

It follows immediately from the definition of orthogonality and normality that:

$U^T U = I = U U^T$

This means that the inverse of an orthogonal matrix is its transpose.

If U is not square - i.e., $U \in \mathbb{R}^{m\times n}$, $n < m$ - but its columns are still orthonormal, then $U^T U = I$, but $U U^T \neq I$.

We generally only use the term orthogonal to describe the case, where $U$ is square.

Another nice property of orthogonal matrices is that operating on a vector with an orthogonal matrix will not change its Euclidean norm. For any $x \in \mathbb{R}^n$, $U \in \mathbb{R}^{n\times n}$ orthogonal.

$\|U_x\|^2 = \|x\|^2$

```python
#How to create random orthonormal matrix in python numpy

def rvs(dim=3):
     random_state = np.random
     H = np.eye(dim)
     D = np.ones((dim,))
     for n in range(1, dim):
         x = random_state.normal(size=(dim-n+1,))
         D[n-1] = np.sign(x[0])
         x[0] -= D[n-1]*np.sqrt((x*x).sum())
         # Householder transformation
         Hx = (np.eye(dim-n+1) - 2.*np.outer(x, x)/(x*x).sum())
         mat = np.eye(dim)
         mat[n-1:, n-1:] = Hx
         H = np.dot(H, mat)
         # Fix the last sign such that the determinant is 1
     D[-1] = (-1)**(1-(dim % 2))*D.prod()
     # Equivalent to np.dot(np.diag(D), H) but faster, apparently
     H = (D*H.T).T
     return H
```

## 14- Range and Nullspace of a Matrix

The **span** of a set of vectors $\{ x_1, x_2, \dots{}, x_n\}$ is the set of all vectors that can be expressed as
a linear combination of $\{ x_1, \dots{}, x_n \}$:

$\mathrm{span}(\{ x_1, \dots{}, x_n \}) = \{ v : v = \sum_{i=1}^n \alpha_i x_i, \alpha_i \in \mathbb{R} \}$

It can be shown that if $\{ x_1, \dots{}, x_n \}$ is a set of n linearly independent vectors, where each $x_i \in \mathbb{R}^n$, then $\mathrm{span}(\{ x_1, \dots{}, x_n\}) = \mathbb{R}^n$. That is, any vector $v \in \mathbb{R}^n$ can be written as a linear combination of $x_1$ through $x_n$.

The projection of a vector $y \in \mathbb{R}^m$ onto the span of $\{ x_1, \dots{}, x_n\}$ (here we assume $x_i \in \mathbb{R}^m$) is the vector $v \in \mathrm{span}(\{ x_1, \dots{}, x_n \})$, such that $v$ is as close as possible to $y$, as measured by the Euclidean norm $\|v − y\|^2$. We denote the projection as $\mathrm{Proj}(y; \{ x_1, \dots{}, x_n \})$ and can define it formally as:

$\mathrm{Proj}( y; \{ x_1, \dots{}, x_n \}) = \mathrm{argmin}_{v\in \mathrm{span}(\{x_1,\dots{},x_n\})}\|y − v\|^2$

The **range** (sometimes also called the columnspace) of a matrix $A \in \mathbb{R}^{m\times n}$, denoted $\mathcal{R}(A)$, is the the span of the columns of $A$. In other words,

$\mathcal{R}(A) = \{ v \in \mathbb{R}^m : v = A x, x \in \mathbb{R}^n\}$

Making a few technical assumptions (namely that $A$ is full rank and that $n < m$), the projection of a vector $y \in \mathbb{R}^m$ onto the range of $A$ is given by:

$\mathrm{Proj}(y; A) = \mathrm{argmin}_{v\in \mathcal{R}(A)}\|v − y\|^2 = A(A^T A)^{−1} A^T y$

See for more details in the notes page 13.

The **nullspace** of a matrix $A \in \mathbb{R}^{m\times n}$, denoted $\mathcal{N}(A)$ is the set of all vectors that equal $0$ when multiplied by $A$, i.e.,

$\mathcal{N}(A) = \{ x \in \mathbb{R}^n : A x = 0 \}$

Note that vectors in $\mathcal{R}(A)$ are of size $m$, while vectors in the $\mathcal{N}(A)$ are of size $n$, so vectors in $\mathcal{R}(A^T)$ and $\mathcal{N}(A)$ are both in $\mathbb{R}^n$. In fact, we can say much more. It turns out that:

$\{ w : w = u + v, u \in \mathcal{R}(A^T), v \in \mathcal{N}(A) \} = \mathbb{R}^n$ and $\mathcal{R}(A^T) \cap \mathcal{N}(A) = \{0\}$

In other words, $\mathcal{R}(A^T)$ and $\mathcal{N}(A)$ are disjoint subsets that together span the entire space of
$\mathbb{R}^n$. Sets of this type are called **orthogonal complements**, and we denote this $\mathcal{R}(A^T) = \mathcal{N}(A)^\perp$.

# 15-  Determinant

The determinant of a square matrix $A \in \mathbb{R}^{n\times n}$, is a function $\mathrm{det} : \mathbb{R}^{n\times n} \rightarrow \mathbb{R}$, and is denoted $|A|$ or $\mathrm{det}A$ (like the trace operator, we usually omit parentheses).

## 15-1 A geometric interpretation of the determinant

Given

$\begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_n^T  & --
\end{bmatrix}$

consider the set of points $S \subset \mathbb{R}^n$ formed by taking all possible linear combinations of the row vectors $a_1, \dots{}, a_n \in \mathbb{R}^n$ of $A$, where the coefficients of the linear combination are all
between $0$ and $1$; that is, the set $S$ is the restriction of $\mathrm{span}( \{ a_1, \dots{}, a_n \})$ to only those linear combinations whose coefficients $\alpha_1, \dots{}, \alpha_n$ satisfy $0 \leq \alpha_i \leq 1$, $i = 1, \dots{}, n$. Formally:

$S = \{v \in \mathbb{R}^n : v = \sum_{i=1}^n \alpha_i a_i \mbox{ where } 0 \leq \alpha_i \leq 1, i = 1, \dots{}, n \}$

The absolute value of the determinant of $A$, it turns out, is a measure of the *volume* of the set $S$. The volume here is intuitively for example for $n = 2$ the area of $S$ in the Cartesian plane, or with $n = 3$ it is the common understanding of *volume* for 3-dimensional objects.

Example:

$A = \begin{bmatrix}
 1 & 3\\[0.3em]
 3 & 2
\end{bmatrix}$

The rows of the matrix are:

$a_1 = \begin{bmatrix}
 1 \\[0.3em]
 3
\end{bmatrix}
\quad
a_2 = \begin{bmatrix}
 3 \\[0.3em]
 2
\end{bmatrix}$

The set S corresponding to these rows is shown in:

The figure above is an illustration of the determinant for the $2\times 2$ matrix $A$ above. Here, $a_1$ and $a_2$
are vectors corresponding to the rows of $A$, and the set $S$ corresponds to the shaded region (i.e., the parallelogram). The absolute value of the determinant, $|\mathrm{det}A| = 7$, is the area of the parallelogram.

For two-dimensional matrices, $S$ generally has the shape of a parallelogram. In our example, the value of the determinant is $|A| = −7$ (as can be computed using the formulas shown later), so the area of the parallelogram is $7$.

In three dimensions, the set $S$ corresponds to an object known as a parallelepiped (a three-dimensional box with skewed sides, such that every face has the shape of a parallelogram). The absolute value of the determinant of the $3 \times 3$ matrix whose rows define $S$ give the three-dimensional volume of the parallelepiped. In even higher dimensions, the set $S$ is an object known as an $n$-dimensional parallelotope.

Algebraically, the determinant satisfies the following three properties (from which all other properties follow, including the general formula):

- The determinant of the identity is $1$, $|I| = 1$. (Geometrically, the volume of a unit hypercube is $1$).
- Given a matrix $A \in \mathbb{R}^{n\times n}$, if we multiply a single row in $A$ by a scalar $t \in \mathbb{R}$, then the determinant of the new matrix is $t|A|$,

$\left| \begin{bmatrix}
 -- & t a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}\right|  = t|A|$

(Geometrically, multiplying one of the sides of the set $S$ by a factor $t$ causes the volume
to increase by a factor $t$.)
- If we exchange any two rows $a^T_i$ and $a^T_j$ of $A$, then the determinant of the new matrix is $−|A|$, for example

$\left| \begin{bmatrix}
 -- & a_2^T  & -- \\[0.3em]
 -- & a_1^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}\right|  = -|A|$

Several properties that follow from the three properties above include:

- For $A \in \mathbb{R}^{n\times n}$, $|A| = |A^T|$
- For $A,B \in \mathbb{R}^{n\times n}$, $|AB| = |A||B|$
- For $A \in \mathbb{R}^{n\times n}$, $|A| = 0$ if and only if $A$ is singular (i.e., non-invertible). (If $A$ is singular then it does not have full rank, and hence its columns are linearly dependent. In this case, the set $S$ corresponds to a "flat sheet" within the $n$-dimensional space and hence has zero volume.)
- For $A \in \mathbb{R}^{n\times n}$ and $A$ non-singular, $|A−1| = 1/|A|$

# 16- Tensors

A [**tensor**](https://en.wikipedia.org/wiki/Tensor) could be thought of as an organized multidimensional array of numerical values. A vector could be assumed to be a sub-class of a tensor. Rows of tensors extend alone the y-axis, columns along the x-axis. The **rank** of a scalar is 0, the rank of a **vector** is 1, the rank of a **matrix** is 2, the rank of a **tensor** is 3 or higher.

# 17- Hyperplane

The **hyperplane** is a sub-space in the ambient space with one dimension less. In a two-dimensional space the hyperplane is a line, in a three-dimensional space it is a two-dimensional plane, etc.

Hyperplanes divide an $n$-dimensional space into sub-spaces that might represent clases in a machine learning algorithm.

# 18- Summary
let me summary what we read in this kernel

## 18-1 Dot Product

This is also the *inner product*. It is a function that returns a number computed from two vectors of the same length by summing up the product of the corresponding dimensions.

For two vectors $a = [a_1, a_2, \dots{}, a_n]$ and $b = [b_1, b_2, \dots{}, b_n]$ the dot product is:

$\mathbf{a} \cdot \mathbf{b} = \sum_{i=1}^{n} a_{i} b_{i} = a_{1} b_{1} + a_{2} b_{2} + \cdots + a_{n} b_{n}$

If we normalize two vectors and compute the dot product, we get the *cosine similarity*, which can be used as a metric for cimilarity of vectors. Independent of the absolute length we look at the angle between the vectors, i.e. the lenght is neutralized via normalization.

The cosine of two non-zero vectors can be derived by using the Euclidean dot product formula (see [Wikipedia: Cosine similarity](https://en.wikipedia.org/wiki/Cosine_similarity)):

$\mathbf{a} \cdot \mathbf{b} = \left\|\mathbf{a}\right\| \left\|\mathbf{b}\right\| \cos\theta$

Given two vectors of attributes, $A$ and $B$, the cosine similarity, $cos(\theta)$, is represented using a dot product and magnitude as:

$\text{similarity} = \cos(\theta) = \frac{\mathbf{A} \cdot \mathbf{B}}{ \|\mathbf{A} \|\|\mathbf{B} \| } = \frac{\sum \limits_{i=1}^{n}{A_{i}B_{i}}}{{\sqrt {\sum \limits _{i=1}^{n}{A_{i}^{2}}}}{\sqrt {\sum \limits _{i=1}^{n}{B_{i}^{2}}}}}$, with $A_i$ and $B_i$ components of vector $A$ and $B$ respectively.

## 18-2 Hadamard Product

This is also known as the **entrywise product**. For two matrices $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{m\times n}$ the Hadamard product $A\circ B$ is:

$(A\circ B)_{i,j} = (A)_{i,j} (B)_{i,j}$

For example:

$\begin{bmatrix}
 a_{11} & a_{12} & a_{13} \\[0.3em]
 a_{21} & a_{22} & a_{23} \\[0.3em]
 a_{31} & a_{32} & a_{33}
\end{bmatrix} \circ
\begin{bmatrix}
 b_{11} & b_{12} & b_{13} \\[0.3em]
 b_{21} & b_{22} & b_{23} \\[0.3em]
 b_{31} & b_{32} & b_{33}
\end{bmatrix} =
\begin{bmatrix}
 a_{11}b_{11} & a_{12}b_{12} & a_{13}b_{13} \\[0.3em]
 a_{21}b_{21} & a_{22}b_{22} & a_{23}b_{23} \\[0.3em]
 a_{31}b_{31} & a_{32}b_{32} & a_{33}b_{33}
\end{bmatrix}$

## 18-3 Outer Product

This is also called the **tensor product** of two vectors. Compute the resulting matrix by multiplying each element from a column vector with all alements in a row vector.

# 19- Eigenvalues and Eigenvectors
Assume, we have two interest bearing accounts. The first gives an interest rate of 5%, the second a 3% interest, with annual compound.

Assume that after $t$ years the amounts in the two accounts are represented by a 2-vector:

$x^{(t)} = \begin{bmatrix}
 amount in Account 1 \\[0.3em]
 amount in Account 2
\end{bmatrix}$

The growth of the amounts in one year can be described in a matrix:

$x^{(t+1)} = \begin{bmatrix}
 a_{11} & a_{12} \\[0.3em]
 a_{21} & a_{22}
\end{bmatrix} x^{(t)}$

Given the specification of the interest rate above, this simple case gives us:

$x^{(t+1)} = \begin{bmatrix}
 1.05 & 0    \\[0.3em]
 0    & 1.03
\end{bmatrix} x^{(t)}$

Let $A$ denote the matrix: $\begin{bmatrix}
 1.05 & 0    \\[0.3em]
 0    & 1.03
\end{bmatrix}$

$A$ is a diagonal.

```python
import numpy as np
x = np.array([[100],
              [100]])
A = np.array([[1.05, 0],
              [0,    1.03]])
A.dot(x)
```

    array([[105.],
           [103.]])

After two years the accounts would be:

```python
A.dot(A.dot(x))
```

    array([[110.25],
           [106.09]])

If we might want to know how $x^{(100)}$ compares to $x^{(0)}$, we could iterate over:

$\begin{align}
x^{(100)} & = A x^{(99)} \\
          & = A(Ax^{(98)}) \\
          & = A(A(Ax^{(97)})) \\
          & \vdots \\
          & = \underbrace{A \cdot A \dots A}_\text{100 times} \ x^{(0)}
\end{align}$

We can also write the product as $A^{100}$.

Note that $A$ is a diagonal, thus the entries of $A^{100}$ are $1.05^{100}$ and $1.03^{100}$:

$A^{100} = \begin{bmatrix}
 131.50125784630401 & 0    \\[0.3em]
 0    & 19.218631980856298
\end{bmatrix}$

What we can see is that account 1 dominates account 2, account 2 becoming less and less relevant over time.

# 20- Exercise

```python
# Students may (probably should) ignore this code. It is just here to make pretty arrows.

def plot_vectors(vs):
    """Plot vectors in vs assuming origin at (0,0)."""
    n = len(vs)
    X, Y = np.zeros((n, 2))
    U, V = np.vstack(vs).T
    plt.quiver(X, Y, U, V, range(n), angles='xy', scale_units='xy', scale=1)
    xmin, xmax = np.min([U, X]), np.max([U, X])
    ymin, ymax = np.min([V, Y]), np.max([V, Y])
    xrng = xmax - xmin
    yrng = ymax - ymin
    xmin -= 0.05*xrng
    xmax += 0.05*xrng
    ymin -= 0.05*yrng
    ymax += 0.05*yrng
    plt.axis([xmin, xmax, ymin, ymax])
```

```python
# Again, this code is not intended as a coding example.

a1 = np.array([3,0])         # axis
a2 = np.array([0,3])

plt.figure(figsize=(8,4))
plt.subplot(1,2,1)
plot_vectors([a1, a2])
v1 = np.array([2,3])
plot_vectors([a1,v1])
plt.text(2,3,"(2,3)",fontsize=16)
plt.tight_layout()

```

```python
#Matrices, Transformations and Geometric Interpretation
a1 = np.array([7,0])         # axis
a2 = np.array([0,5])

A = np.array([[2,1],[1,1]])  # transformation f in standard basis
v2 =np.dot(A,v1)
plt.figure(figsize=(8,8))
plot_vectors([a1, a2])
v1 = np.array([2,3])
plot_vectors([v1,v2])
plt.text(2,3,"v1 =(2,3)",fontsize=16)
plt.text(6,5,"Av1 = ", fontsize=16)
plt.text(v2[0],v2[1],"(7,5)",fontsize=16)
print(v2[1])
```

```python
#Change to a Different Basis
e1 = np.array([1,0])
e2 = np.array([0,1])
B = np.array([[1,4],[3,1]])
plt.figure(figsize=(8,4))
plt.subplot(1,2,1)
plot_vectors([e1, e2])
plt.subplot(1,2,2)
plot_vectors([B.dot(e1), B.dot(e2)])
plt.Circle((0,0),2)
#plt.show()
#plt.tight_layout()
```

```python
#Inner Products
e1 = np.array([1,0])
e2 = np.array([0,1])
A = np.array([[2,3],[3,1]])
v1=A.dot(e1)
v2=A.dot(e2)
plt.figure(figsize=(8,4))
plt.subplot(1,2,1)
plot_vectors([e1, e2])
plt.subplot(1,2,2)
plot_vectors([v1,v2])
plt.tight_layout()
#help(plt.Circle)
plt.Circle(np.array([0,0]),radius=1)
plt.Circle.draw
```

# 21-Conclusion
If you have made this far – give yourself a pat at the back. We have covered different aspects of **Linear algebra** in this Kernel. I have tried to give sufficient amount of information as well as keep the flow such that everybody can understand the concepts and be able to do necessary calculations. Still, if you get stuck somewhere, feel free to comment below.

you can follow me on:
> ###### [ GitHub](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist)
> ###### [Kaggle](https://www.kaggle.com/mjbahmani/)

 I hope you find this kernel helpful and some UPVOTES would be very much appreciated.

# 22-References

* [1] [Linear Algbra1](https://github.com/dcavar/python-tutorial-for-ipython)
* [2] [Linear Algbra2](https://www.oreilly.com/library/view/data-science-from/9781491901410/ch04.html)

3- Mathematics and Linear Algebra/Linear Algebra for Data Scientists.ipynb


# Neuromatch Academy: Week 1, Day 3, Tutorial 1
# Model Fitting: Linear regression with MSE

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil with help from Byron Galbraith

**Content reviewers**: Lina Teichmann, Saeed Salehi, Patrick Mineault,  Ella Batty, Michael Waskom

___
#Tutorial Objectives

This is Tutorial 1 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will learn how to fit simple linear models to data.
- Learn how to calculate the mean-squared error (MSE)
- Explore how model parameters (slope) influence the MSE
- Learn how to find the optimal model parameter using least-squares optimization

---

**acknowledgements:**
- we thank Eero Simoncelli, much of today's tutorials are inspired by exercises asigned in his mathtools class.

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
#@title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Helper functions

def plot_observed_vs_predicted(x, y, y_hat, theta_hat):
  """ Plot observed vs predicted data

  Args:
      x (ndarray): observed x values
  y (ndarray): observed y values
  y_hat (ndarray): predicted y values

  """
  fig, ax = plt.subplots()
  ax.scatter(x, y, label='Observed')  # our data scatter plot
  ax.plot(x, y_hat, color='r', label='Fit')  # our estimated model
  # plot residuals
  ymin = np.minimum(y, y_hat)
  ymax = np.maximum(y, y_hat)
  ax.vlines(x, ymin, ymax, 'g', alpha=0.5, label='Residuals')
  ax.set(
      title=fr"$\hat{{\theta}}$ = {theta_hat:0.2f}, MSE = {mse(x, y, theta_hat):.2f}",
      xlabel='x',
      ylabel='y'
  )
  ax.legend()

```

---
# Section 1: Mean Squared Error (MSE)

```python
#@title Video 1: Linear Regression & Mean Squared Error
from IPython.display import IFrame
class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
        self.id=id
        src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
        super(BiliVideo, self).__init__(src, width, height, **kwargs)

video = BiliVideo(id='BV1tA411e7NW', width=854, height=480, fs=1)
print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
video

```

    Video available at https://www.bilibili.com/video/BV1tA411e7NW

**Linear least squares regression** is an old but gold  optimization procedure that we are going to use for data fitting. Least squares (LS) optimization problems are those in which the objective function is a quadratic function of the
parameter(s) being optimized.

Suppose you have a set of measurements, $y_{n}$ (the "dependent" variable) obtained for different input values, $x_{n}$ (the "independent" or "explanatory" variable). Suppose we believe the measurements are proportional to the input values, but are corrupted by some (random) measurement errors, $\epsilon_{n}$, that is:

$$y_{n}= \theta x_{n}+\epsilon_{n}$$

for some unknown slope parameter $\theta.$ The least squares regression problem uses **mean squared error (MSE)** as its objective function, it aims to find the value of the parameter $\theta$ by minimizing the average of squared errors:

\begin{align}
\min _{\theta} \frac{1}{N}\sum_{n=1}^{N}\left(y_{n}-\theta x_{n}\right)^{2}
\end{align}

We will now explore how MSE is used in fitting a linear regression model to data. For illustrative purposes, we will create a simple synthetic dataset where we know the true underlying model. This will allow us to see how our estimation efforts compare in uncovering the real model (though in practice we rarely have this luxury).

First we will generate some noisy samples $x$ from [0, 10) along the line $y = 1.2x$ as our dataset we wish to fit a model to.

```python
# @title

# @markdown Execute this cell to generate some simulated data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence
np.random.seed(121)

# Let's set some parameters
theta = 1.2
n_samples = 30

# Draw x and then calculate y
x = 10 * np.random.rand(n_samples)  # sample from a uniform distribution over [0,10)
noise = np.random.randn(n_samples)  # sample from a standard normal distribution
y = theta * x + noise

# Plot the results
fig, ax = plt.subplots()
ax.scatter(x, y)  # produces a scatter plot
ax.set(xlabel='x', ylabel='y');
```

Now that we have our suitably noisy dataset, we can start trying to estimate the underlying model that produced it. We use MSE to evaluate how successful a particular slope estimate $\hat{\theta}$ is for explaining the data, with the closer to 0 the MSE is, the better our estimate fits the data.

## Exercise 1: Compute MSE

In this exercise you will implement a method to compute the mean squared error for a set of inputs $x$, measurements $y$, and slope estimate $\hat{\theta}$. We will then compute and print the mean squared error for 3 different choices of theta

```python
def mse(x, y, theta_hat):
  """Compute the mean squared error

  Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta_hat (float): An estimate of the slope parameter

  Returns:
    float: The mean squared error of the data with the estimated parameter.
  """
  ####################################################
  ## TODO for students: compute the mean squared error
  # Fill out function and remove
  raise NotImplementedError("Student exercise: compute the mean squared error")
  ####################################################

  # Compute the estimated y
  y_hat = ...

  # Compute mean squared error
  mse = ...

  return mse

# Uncomment below to test your function
theta_hats = [0.75, 1.0, 1.5]
# for theta_hat in theta_hats:
#   print(f"theta_hat of {theta_hat} has an MSE of {mse(x, y, theta_hat):.2f}")
```

```python
# to_remove solution
def mse(x, y, theta_hat):
  """Compute the mean squared error

  Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta_hat (float): An estimate of the slope parameter

  Returns:
    float: The mean squared error of the data with the estimated parameter.
  """

  # Compute the estimated y
  y_hat = theta_hat * x

  # Compute mean squared error
  mse = np.mean((y - y_hat)**2)

  return mse

theta_hats = [0.75, 1.0, 1.5]
for theta_hat in theta_hats:
  print(f"theta_hat of {theta_hat} has an MSE of {mse(x, y, theta_hat):.2f}")
```

    theta_hat of 0.75 has an MSE of 9.08
    theta_hat of 1.0 has an MSE of 3.01
    theta_hat of 1.5 has an MSE of 4.52

The result should be:

theta_hat of 0.75 has an MSE of 9.08\
theta_hat of 1.0 has an MSE of 3.0\
theta_hat of 1.5 has an MSE of 4.52

We see that $\hat{\theta} = 1.0$ is our best estimate from the three we tried. Looking just at the raw numbers, however, isn't always satisfying, so let's visualize what our estimated model looks like over the data.

```python
#@title

#@markdown Execute this cell to visualize estimated models

fig, axes = plt.subplots(ncols=3, figsize=(18, 4))
for theta_hat, ax in zip(theta_hats, axes):

  # True data
  ax.scatter(x, y, label='Observed')  # our data scatter plot

  # Compute and plot predictions
  y_hat = theta_hat * x
  ax.plot(x, y_hat, color='r', label='Fit')  # our estimated model

  ax.set(
      title= fr'$\hat{{\theta}}$= {theta_hat}, MSE = {mse(x, y, theta_hat):.2f}',
      xlabel='x',
      ylabel='y'
  );

axes[0].legend()
```

## Interactive Demo: MSE Explorer

Using an interactive widget, we can easily see how changing our slope estimate changes our model fit. We display the **residuals**, the differences between observed and predicted data, as line segments between the data point (observed response) and the corresponding predicted response on the model fit line.

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(theta_hat=widgets.FloatSlider(1.0, min=0.0, max=2.0))
def plot_data_estimate(theta_hat):
  y_hat = theta_hat * x
  plot_observed_vs_predicted(x, y, y_hat, theta_hat)

```

While visually exploring several estimates can be instructive, it's not the most efficient for finding the best estimate to fit our data. Another technique we can use is choose a reasonable range of parameter values and compute the MSE at several values in that interval. This allows us to plot the error against the parameter value (this is also called an **error landscape**, especially when we deal with more than one parameter). We can select the final $\hat{\theta}$ ($\hat{\theta}_{MSE}$) as the one which results in the lowest error.

```python
# @title

# @markdown Execute this cell to loop over theta_hats, compute MSE, and plot results

# Loop over different thetas, compute MSE for each
theta_hat_grid = np.linspace(-2.0, 4.0)
errors = np.zeros(len(theta_hat_grid))
for i, theta_hat in enumerate(theta_hat_grid):
  errors[i] = mse(x, y, theta_hat)

# Find theta that results in lowest error
best_error = np.min(errors)
theta_hat = theta_hat_grid[np.argmin(errors)]

# Plot results
fig, ax = plt.subplots()
ax.plot(theta_hat_grid, errors, '-o', label='MSE', c='C1')
ax.axvline(theta, color='g', ls='--', label=r"$\theta_{True}$")
ax.axvline(theta_hat, color='r', ls='-', label=r"$\hat{{\theta}}_{MSE}$")
ax.set(
  title=fr"Best fit: $\hat{{\theta}}$ = {theta_hat:.2f}, MSE = {best_error:.2f}",
  xlabel=r"$\hat{{\theta}}$",
  ylabel='MSE')
ax.legend();
```

We can see that our best fit is $\hat{\theta}=1.18$ with an MSE of 1.45. This is quite close to the original true value $\theta=1.2$!

---
# Section 2: Least-squares optimization

While the approach detailed above (computing MSE at various values of $\hat\theta$) quickly got us to a good estimate, it still relied on evaluating the MSE value across a grid of hand-specified values. If we didn't pick a good range to begin with, or with enough granularity, we might miss the best possible estimator. Let's go one step further, and instead of finding the minimum MSE from a set of candidate estimates, let's solve for it analytically.

We can do this by minimizing the cost function. Mean squared error is a convex objective function, therefore we can compute its minimum using calculus. Please see video or appendix for this derivation! After computing the minimum, we find that:

\begin{align}
\hat\theta = \frac{\vec{x}^\top \vec{y}}{\vec{x}^\top \vec{x}}
\end{align}

This is known as solving the normal equations. For different ways of obtaining the solution, see the notes on [Least Squares Optimization](https://www.cns.nyu.edu/~eero/NOTES/leastSquares.pdf) by Eero Simoncelli.

### Exercise 2: Solve for the Optimal Estimator

In this exercise, you will write a function that finds the optimal $\hat{\theta}$ value using the least squares optimization approach (the equation above) to solve MSE minimization. It shoud take arguments $x$ and $y$ and return the solution $\hat{\theta}$.

We will then use your function to compute $\hat{\theta}$ and plot the resulting prediction on top of the data.

```python
def solve_normal_eqn(x, y):
  """Solve the normal equations to produce the value of theta_hat that minimizes
    MSE.

    Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    float: the value for theta_hat arrived from minimizing MSE
  """

  ################################################################################
  ## TODO for students: solve for the best parameter using least squares
  # Fill out function and remove
  raise NotImplementedError("Student exercise: solve for theta_hat using least squares")
  ################################################################################

  # Compute theta_hat analytically
  theta_hat = ...

  return theta_hat

# Uncomment below to test your function
# theta_hat = solve_normal_eqn(x, y)
# y_hat = theta_hat * x
# plot_observed_vs_predicted(x, y, y_hat, theta_hat)

```

```python
# to_remove solution
def solve_normal_eqn(x, y):
  """Solve the normal equations to produce the value of theta_hat that minimizes
    MSE.

    Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    float: the value for theta_hat arrived from minimizing MSE
  """

  theta_hat = (x.T @ y) / (x.T @ x)

  return theta_hat

theta_hat = solve_normal_eqn(x, y)
y_hat = theta_hat * x

with plt.xkcd():
  plot_observed_vs_predicted(x, y, y_hat, theta_hat)

```

We see that the analytic solution produces an even better result than our grid search from before, producing $\hat{\theta} = 1.21$ with MSE = 1.43!

---
# Summary

- Linear least squares regression is an optimization procedure that can be used for data fitting:
    - Task: predict a value for $y$ given $x$
    - Performance measure: $\textrm{MSE}$
    - Procedure: minimize $\textrm{MSE}$ by solving the normal equations
- **Key point**: We fit the model by defining an *objective function* and minimizing it.
- **Note**: In this case, there is an *analytical* solution to the minimization problem and in practice, this solution can be computed using *linear algebra*. This is *extremely* powerful and forms the basis for much of numerical computation throughout the sciences.

---
# Appendix

## Least Squares Optimization Derivation

We will outline here the derivation of the least squares solution.

We first set the derivative of the error expression with respect to $\theta$ equal to zero,

\begin{align}
\frac{d}{d\theta}\frac{1}{N}\sum_{i=1}^N(y_i - \theta x_i)^2 = 0 \\
\frac{1}{N}\sum_{i=1}^N-2x_i(y_i - \theta x_i) = 0
\end{align}

where we used the chain rule. Now solving for $\theta$, we obtain an optimal value of:

\begin{align}
\hat\theta = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}
\end{align}

Which we can write in vector notation as:

\begin{align}
\hat\theta = \frac{\vec{x}^\top \vec{y}}{\vec{x}^\top \vec{x}}
\end{align}

This is known as solving the *normal equations*. For different ways of obtaining the solution, see the notes on [Least Squares Optimization](https://www.cns.nyu.edu/~eero/NOTES/leastSquares.pdf) by Eero Simoncelli.


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 17: Exogenous Shocks in the New-Keynesian Model

In this notebook, we will compute impulse responses to exogenous shocks in the new-Keynesian model and interpret the results.

### Households

\begin{align}
C_t^{-\sigma} & = \beta (1+r_t)E_t C_{t+1}^{-\sigma}\\
W_t & = \varphi (1-L_t)^{-\eta} C^{\sigma}\\
C_t^{-\sigma} & = \beta E_t C_{t+1}^{-\sigma}\left( R_{t+1} + 1-\delta \right)\\
\end{align}

### Firms, Production, and Aggregate Supply

\begin{align}
W_t & = (1-\alpha) A_t K_{t}^{\alpha}L_t^{-\alpha}/X_t\\
R_t & = \alpha A_t K_{t}^{\alpha-1}L_t^{1-\alpha}/X_t\\
Y_t & = A_t K_t^{\alpha}L_t^{1-\alpha}\\
\pi_t & = \beta E_t\pi_{t+1} - \kappa \log(X_t/\bar{X}) + u_t\\
Y_t & = C_t + I_t + G_t
\end{align}

### Monetary Policy

\begin{align}
i_{t} & = \phi_{\pi}\pi_t + \phi_{y} \log (Y_t/\bar{Y}) + v_t\\
\end{align}

### Evolution of State Variables

\begin{align}
K_{t+1} & = I_{t} + (1-\delta) K_t\\
\log A_{t+1} & = \rho_a \log A_{t} + \epsilon^a_{t+1}\\
\log (G_{t+1}/\bar{G}) & = \rho_g \log (G_{t}/\bar{G}) + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}\\
v_{t+1} & = \rho_v v_{t} + \epsilon^v_{t+1}.
\end{align}

### Fisher Equation

\begin{align}
i_{t} & = r_{t} + E_t \pi_{t+1}\\
\end{align}

The goal is to compute impulse responses in the model to all three exogenous shocks. We will use the following parameterization:

| $\bar{y}$ | $\beta$ | $\bar{r}$    | $\kappa$ | $\pi^T$ | $\phi_{\pi}$ | $\phi_y$ | $\rho_g$ | $\rho_u$ | $\rho_v$ |
|-----------|---------|--------------|----------|---------|--------------|----------|----------|----------|----------|
| 0         | 0.995   | $-\log\beta$ | 0.1      | 0.02/4  | 1.5          | 0.5/4    | 0.5      | 0.5      | 0.5      |

```python

# values = pd.Series(guess,index=varNames)
# print(values)

```

```python
# Define a function that evaluates the equilibrium conditions of the model solved for zero.

    # Parameters

    # Current variables

    # Forward variables

    # Euler equation

    # Labor supply

    # Capital supply

    # Labor demand

    # Capital demand

    # Production

    # Market clearing

    # Monetary policy

    # Fisher equation

    # Phillips curve
#     phillips_curve = (1-p.beta) + p.beta*(fwd.pi) - p.kappa*np.log(cur.x/p.x_bar)  + np.log(cur.u) - cur.pi

    # Capital evolution

    # Government purchases process

    # TFP process

    # Inflation shock process

    # Monetary policy shock process

    # Stack equilibrium conditions into a numpy array

# Initialize the model into a variable named 'nk_model'

# Compute the steady state numerically using .compute_ss() method of nk_model

```

```python

```

```python

# model.impulse(t0=5,shocks=[0,0.01,0.01,0.01/4,0.01/4,0.01/4],percent=True,T=26)
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

## Solution

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series. CELL PROVIDED
parameters = pd.Series()
parameters['y_bar'] = 0
parameters['beta'] = 0.995
parameters['r_bar'] = -np.log(parameters.beta)
parameters['kappa'] = 0.1
parameters['pi_T'] = 0.02/4
parameters['phi_pi'] = 1.5
parameters['phi_y'] = 0.5/4
parameters['rho_g'] = 0.5
parameters['rho_u'] = 0.5
parameters['rho_v'] = 0.5

# Create variable called 'varNames' that stores the variable names in a list with state variables ordered first
varNames = ['g','u','v','y','pi','i','r']

# Create variable called 'shockNames' that stores an exogenous shock name for each state variable.
shockNames = ['e_g','e_u','e_v']

# Define a function that evaluates the equilibrium conditions of the model solved for zero.
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters
    p = parameters

    # Current variables
    cur = variables_current

    # Forward variables
    fwd = variables_forward

    # IS equation
    is_equation = fwd.y - (cur.r -p.r_bar) + cur.g - cur.y

    # Fisher_equation
    fisher_equation = cur.r + fwd.pi - cur.i

    # Monetary policy
    monetary_policy = p.r_bar + p.pi_T + p.phi_pi*(cur.pi - p.pi_T) + p.phi_y*cur.y + cur.v - cur.i

    # Phillips curve
    phillips_curve = p.beta*(fwd.pi- p.pi_T) + p.kappa*cur.y + cur.u - (cur.pi-p.pi_T)

    # Demand process
    demand_process = p.rho_g*cur.g - fwd.g

    # Monetary policy process
    monetary_policy_process = p.rho_v*cur.v - fwd.v

    # Inflation process
    inflation_process = p.rho_u*cur.u - fwd.u

    # Stack equilibrium conditions into a numpy array
    return np.array([
            is_equation,
            fisher_equation,
            monetary_policy,
            phillips_curve,
            demand_process,
            monetary_policy_process,
            inflation_process
        ])

# Initialize the model into a variable named 'nk_model'
nk_model = ls.model(equations = equilibrium_equations,
                 nstates=3,
                 varNames=varNames,
                 shockNames=shockNames,
                 parameters = parameters)

# Compute the steady state numerically using .compute_ss() method of nk_model
guess = [k,a,g,y,c,i,l,x,pi,real,nominal_rate,w,r]
nk_model.compute_ss(guess)
```

```python
# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'loglinear' to False

# Print the solution to the model

```

**Questions**

1. Will a positive shock to $g_t$ increase or decrease $y_t$?
2. Will a positive shock to $g_t$ increase or decrease $\pi_t$?
3. Will a positive shock to $u_t$ increase or decrease $y_t$?
4. Will a positive shock to $u_t$ increase or decrease $\pi_t$?

**Answers**

1.

2.

1.

4.

## Impulse Responses

Compute a 21 period impulse response of the model's variables to a:

* 0.01 unit shock to $g_t$
* 0.01/4 unit shock to $u_t$
* 0.01/4 unit shock to $v_t$

```python
# Compute impulse responses

```

### Demand Shock

Plot the computed impulses responses of the nominal_rate interest rate, the real interest rate, output, and inflation to the 0.01 unit demand shock. Express inflation and interest rates in *annualized* (e.g., multiplied by 4) terms.

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,8))

# Create upper-left axis. PROVIDED
ax1 = fig.add_subplot(2,2,1)

# Create upper-right axis. PROVIDED
ax2 = fig.add_subplot(2,2,2)

# Create lower-left axis. PROVIDED
ax3 = fig.add_subplot(2,2,3)

# Create lower-right axis. PROVIDED
ax4 = fig.add_subplot(2,2,4)

```

### Inflation Shock

Plot the computed impulses responses of the nominal_rate interest rate, the real interest rate, output, and inflation to the 0.01/4 unit inflation shock. Express inflation and interest rates in *annualized* (e.g., multiplied by 4) terms.

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,8))

# Create upper-left axis. PROVIDED
ax1 = fig.add_subplot(2,2,1)

# Create upper-right axis. PROVIDED
ax2 = fig.add_subplot(2,2,2)

# Create lower-left axis. PROVIDED
ax3 = fig.add_subplot(2,2,3)

# Create lower-right axis. PROVIDED
ax4 = fig.add_subplot(2,2,4)

```

### Monetary Policy Shock

Plot the computed impulses responses of the nominal_rate interest rate, the real interest rate, output, and inflation to the 0.01/4 unit monetary policy shock. Express inflation and interest rates in *annualized* (e.g., multiplied by 4) terms.

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,8))

# Create upper-left axis. PROVIDED
ax1 = fig.add_subplot(2,2,1)

# Create upper-right axis. PROVIDED
ax2 = fig.add_subplot(2,2,2)

# Create lower-left axis. PROVIDED
ax3 = fig.add_subplot(2,2,3)

# Create lower-right axis. PROVIDED
ax4 = fig.add_subplot(2,2,4)

```


# What is Machine Learning?

It is a study of computer algorithm that improve automatically through experience. These algorithms build a Mathematical model (Neural Network) that is based on a sample called the "training data".

# What are Neural Networks?

It is a series of algorithms built such that they find the patterns and relations in a set of data, that mimics the working of the human brain.

Each of the nodes in the neural network is called a perceptron.

## Perceptrons

Every unit of a Neural Network is called a perceptron. It is essentially an artificial neuron.

Each perceptron has some number of weighted inputs and a bias, which are summed up (linear combination) and then passed through an activation function to get the output.

Mathematically this looks like:

$$
\begin{align}
y &= f(w_1 x_1 + w_2 x_2 + b) \\
y &= f\left(\sum_i w_i x_i +b \right)
\end{align}
$$

With vectors this is the dot/inner product of two vectors:

$$
h = \begin{bmatrix}
x_1 \, x_2 \cdots  x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_1 \\
           w_2 \\
           \vdots \\
           w_n
\end{bmatrix}
$$

## Activation Functions

They are mathematical equations used to determine the output of a neural network.

## Tensors

Neural network computation is essentially just a bunch of linear algebra operations on *tensors*. Tensors are the data structures used for Neural Network computation and PyTorch and other such Deep Learning Libararies are built around them.

## Single Perceptron

Now let's create our first perceptron/neuron.

```python
#Import pytorch
import torch
```

```python
#Define activation function (sigmoid)
def activation(x):
    #x is a Tensor.

    return 1/(1+torch.exp(-x))
```

```python
#Generate Data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 5 random normal variables
features = torch.randn((1, 5))
# True weights for our data, random normal variables again
weights = torch.randn_like(features)
# and a true bias term
bias = torch.randn((1, 1))

```

```python
#Obtaining the output
y= activation(torch.sum(features*weights)+bias)
print(y)

```

    tensor([[0.1595]])
    tensor([[0.1595]])

You can do multiplication and sum in the same operation using matrix multiplication. Matrix multiplication is much more efficient with modern libraries and GPUs.

Here we'll use [`torch.mm()`](https://pytorch.org/docs/stable/torch.html#torch.mm) to perform matrix multiplication on features and weights.

**Note** : For matrix multiplication we need to have the two matrix in the form [M x N] and [N x O] .

Here in our case we have both the Tensors in a vector form which is [1 x 5].

To counter this issue we need to reshape this Tensor. We have a few options [`weights.reshape()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.reshape), [`weights.resize_()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.resize_), and [`weights.view()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.view).

* `weights.reshape(a, b)` will return a new tensor with the same data as `weights` with size `(a, b)` sometimes, and sometimes a clone, as in it copies the data to another part of memory.
* `weights.resize_(a, b)` returns the same tensor with a different shape. However, if the new shape results in fewer elements than the original tensor, some elements will be removed from the tensor (but not from memory). If the new shape results in more elements than the original tensor, new elements will be uninitialized in memory. Here I should note that the underscore at the end of the method denotes that this method is performed **in-place**. Here is a great forum thread to [read more about in-place operations](https://discuss.pytorch.org/t/what-is-in-place-operation/16244) in PyTorch.
* `weights.view(a, b)` will return a new tensor with the same data as `weights` with size `(a, b)`.

> In today's session we'll use `.view()` .

```python
#Using torch.mm()
y = activation(torch.mm(features, weights.view( 5, 1 ) ) + bias)
print(y)
# 1 x 5 ... 1 x 5
# 1 x 5 ... 5 x 1

```

    tensor([[0.1595]])

## Creating our first neural network

Well that was a single neuron, which alone doesn't do much but when stacked together they showcase their real potential.

Here the bottom layer provides the inputs and is called the **Input layer**. The layer that follows is what is called as the **Hidden layer**. The topmost layer, provides us with the output and is fittingly called the **Output layer**.

This network can be represented mathematically with matrices as shown below.

$$
\vec{h} = [h_1 \, h_2] =
\begin{bmatrix}
x_1 \, x_2 \cdots \, x_n
\end{bmatrix}
\cdot
\begin{bmatrix}
           w_{11} & w_{12} \\
           w_{21} &w_{22} \\
           \vdots &\vdots \\
           w_{n1} &w_{n2}
\end{bmatrix}
$$

The output for this small network is found by treating the hidden layer as inputs for the output unit. The network output is expressed simply

$$
y =  f_2 \! \left(\, f_1 \! \left(\vec{x} \, \mathbf{W_1}\right) \mathbf{W_2} \right)
$$

```python
### Generate some data
torch.manual_seed(7)

# Features
features = torch.randn((1,3))

# Define the size of each layer in our network
n_input = features.shape[1]
n_hidden = 2
n_output = 1

# Weights for inputs to hidden layer
W1 = torch.randn(n_input, n_hidden)
                # 3    x    2

# Weights for hidden layer to output layer
W2 = torch.randn(n_hidden, n_output)
                # 2    x    1

# and bias terms for hidden and output layers
B1 = torch.randn(1, n_hidden)  # [bias1, bias2]

B2 = torch.randn(1, n_output)  # [bias1_1]

```

```python
# Output
y1 = activation(torch.mm(features, W1) + B1)
output = activation(torch.mm(y1, W2) + B2)
print(output)
```

    tensor([[0.3171]])

Once executed correctly you should see the output `tensor([[ 0.3171]])`.