#
Linear Algebra for Data Scientists

One of the most common questions we get on Data science is:

How much maths do I need to learn to be a data scientist?

If you get confused and ask experts what should you learn at this stage, most of them would suggest / agree that you go ahead with Linear Algebra!
This is the third step of the [10 Steps to Become a Data Scientist](https://www.kaggle.com/mjbahmani/10-steps-to-become-a-data-scientist). and you can learn all of the thing you need for being a data scientist with Linear Algabra.

last update: 12/12/2018

You can Fork code  and  Follow me on:

> ###### [ GitHub](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist)
> ###### [Kaggle](https://www.kaggle.com/mjbahmani/)
-------------------------------------------------------------------------------------------------------------
 I hope you find this kernel helpful and some UPVOTES would be very much appreciated.

 -----------

## Notebook  Content
1. [Introduction](#1)
1. [Basic Concepts](#2)
1. [Notation ](#2)
1. [Matrix Multiplication](#3)
    1. [Vector-Vector Products](#4)
    1. [Outer Product of Two Vectors](#5)
    1. [Matrix-Vector Products](#6)
    1. [Matrix-Matrix Products](#7)
1. [Identity Matrix](#8)
1. [Diagonal Matrix](#9)
1. [Transpose of a Matrix](#10)
1. [Symmetric Metrices](#11)
1. [The Trace](#12)
1. [Norms](#13)
1. [Linear Independence and Rank](#14)
    1. [Column Rank of a Matrix](#15)
    1. [Row Rank of a Matrix](#16)
    1. [Rank of a Matrix](#17)
1. [Subtraction and Addition of Metrices](#18)
    1. [Inverse](#19)
1. [Orthogonal Matrices](#20)
1. [Range and Nullspace of a Matrix](#21)
1. [Determinant](#22)
    1. [geometric interpretation of the determinant](#23)
1. [Tensors](#24)
1. [Hyperplane](#25)
1. [Eigenvalues and Eigenvectors](#30)
1. [Exercise](#31)
1. [Conclusion](#32)
1. [References](#33)

#  1-Introduction
This is the third step of the [10 Steps to Become a Data Scientist](https://www.kaggle.com/mjbahmani/10-steps-to-become-a-data-scientist).
**Linear algebra** is the branch of mathematics that deals with **vector spaces**. good understanding of Linear Algebra is intrinsic to analyze Machine Learning algorithms, especially for **Deep Learning** where so much happens behind the curtain.you have my word that I will try to keep mathematical formulas & derivations out of this completely mathematical topic and I try to cover all of subject that you need as data scientist.

*Is there anything more useless or less useful than Algebra?*

**Billy Connolly**

## 1-1 Import

```python
import matplotlib.patches as patch
import matplotlib.pyplot as plt
from scipy.stats import norm
from scipy import linalg
from sklearn import svm
import tensorflow as tf
import pandas as pd
import numpy as np
import glob
import sys
import os
```

##  1-2 Setup

```python
%matplotlib inline
%precision 4
plt.style.use('ggplot')
np.set_printoptions(suppress=True)
```

# 2- Basic Concepts

The following system of equations:

$\begin{equation}
\begin{split}
4 x_1 - 5 x_2 & = -13 \\
 -2x_1 + 3 x_2 & = 9
\end{split}
\end{equation}$

We are looking for a unique solution for the two variables $x_1$ and $x_2$.  The system can be described as:

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

$$
Ax=b
$$

as matrices:

$$A = \begin{bmatrix}
       4  & -5 \\[0.3em]
       -2 &  3
     \end{bmatrix},\
 b = \begin{bmatrix}
       -13 \\[0.3em]
       9
     \end{bmatrix}$$

A **scalar** is an element in a vector, containing a real number **value**. In a vector space model or a vector mapping of (symbolic, qualitative, or quantitative) properties the scalar holds the concrete value or property of a variable.

A **vector** is an array, tuple, or ordered list of scalars (or elements) of size $n$, with $n$ a positive integer. The **length** of the vector, that is the number of scalars in the vector, is also called the **order** of the vector.

```python
#3-dimensional vector in numpy
a = np.zeros((2, 3, 4))
#l = [[[ 0.,  0.,  0.,  0.],
    #      [ 0.,  0.,  0.,  0.],
     #     [ 0.,  0.,  0.,  0.]],
     #     [[ 0.,  0.,  0.,  0.],
    #      [ 0.,  0.,  0.,  0.],
     #     [ 0.,  0.,  0.,  0.]]]
a
```

```python

# Declaring Vectors

x = [1, 2, 3]
y = [4, 5, 6]

print(type(x))

# This does'nt give the vector addition.
print(x + y)

# Vector addition using Numpy

z = np.add(x, y)
print(z)
print(type(z))

# Vector Cross Product
mul = np.cross(x, y)
print(mul)
```

**Vectorization** is the process of creating a vector from some data using some process.

Vectors of the length $n$ could be treated like points in $n$-dimensional space. One can calculate the distance between such points using measures like [Euclidean Distance](https://en.wikipedia.org/wiki/Euclidean_distance). The similarity of vectors could also be calculated using [Cosine Similarity](https://en.wikipedia.org/wiki/Cosine_similarity).
###### [Go to top](#top)

## 3- Notation

A **matrix** is a list of vectors that all are of the same length. $A$ is a matrix with $m$ rows and $n$ columns, antries of $A$ are real numbers:

$A \in \mathbb{R}^{m \times n}$

A vector $x$ with $n$ entries of real numbers, could also be thought of as a matrix with $n$ rows and $1$ column, or as known as a **column vector**.

$x = \begin{bmatrix}
       x_1 \\[0.3em]
       x_2 \\[0.3em]
       \vdots \\[0.3em]
       x_n
     \end{bmatrix}$

Representing a **row vector**, that is a matrix with $1$ row and $n$ columns, we write $x^T$ (this denotes the transpose of $x$, see above).

$x^T = \begin{bmatrix}
       x_1 & x_2 & \cdots & x_n
     \end{bmatrix}$

We use the notation $a_{ij}$ (or $A_{ij}$, $A_{i,j}$, etc.) to denote the entry of $A$ in the $i$th row and
$j$th column:

$A = \begin{bmatrix}
       a_{11} & a_{12} & \cdots & a_{1n} \\[0.3em]
       a_{21} & a_{22} & \cdots & a_{2n} \\[0.3em]
       \vdots & \vdots & \ddots & \vdots \\[0.3em]
       a_{m1} & a_{m2} & \cdots & a_{mn}
     \end{bmatrix}$

We denote the $j$th column of $A$ by $a_j$ or $A_{:,j}$:

$A = \begin{bmatrix}
       \big| & \big| &  & \big| \\[0.3em]
       a_{1} & a_{2} & \cdots & a_{n} \\[0.3em]
       \big| & \big| &  & \big|
     \end{bmatrix}$

We denote the $i$th row of $A$ by $a_i^T$ or $A_{i,:}$:

$A = \begin{bmatrix}
      -- & a_1^T  & -- \\[0.3em]
       -- & a_2^T  & -- \\[0.3em]
          & \vdots &  \\[0.3em]
       -- & a_m^T  & --
     \end{bmatrix}$

A $n \times m$ matrix is a **two-dimensional** array with $n$ rows and $m$ columns.
###### [Go to top](#top)

## 4-Matrix Multiplication

The result of the multiplication of two matrixes $A \in \mathbb{R}^{m \times n}$ and $B \in \mathbb{R}^{n \times p}$ is the matrix:

```python
# initializing matrices
x = np.array([[1, 2], [4, 5]])
y = np.array([[7, 8], [9, 10]])
```

$C = AB \in \mathbb{R}^{m \times n}$

That is, we are multiplying the columns of $A$ with the rows of $B$:

$C_{ij}=\sum_{k=1}^n{A_{ij}B_{kj}}$

The number of columns in $A$ must be equal to the number of rows in $B$.

###### [Go to top](#top)

```python
# using add() to add matrices
print ("The element wise addition of matrix is : ")
print (np.add(x,y))
```

```python
# using subtract() to subtract matrices
print ("The element wise subtraction of matrix is : ")
print (np.subtract(x,y))
```

```python
# using divide() to divide matrices
print ("The element wise division of matrix is : ")
print (np.divide(x,y))
```

```python
# using multiply() to multiply matrices element wise
print ("The element wise multiplication of matrix is : ")
print (np.multiply(x,y))
```

## 4-1 Vector-Vector Products

Inner or Dot **Product** of Two Vectors.

For two vectors $x, y \in \mathbb{R}^n$, the **inner product** or **dot product** $x^T y$ is a real number:

$x^T y \in \mathbb{R} = \begin{bmatrix}
       x_1 & x_2 & \cdots & x_n
     \end{bmatrix} \begin{bmatrix}
       y_1 \\[0.3em]
       y_2 \\[0.3em]
       \vdots \\[0.3em]
       y_n
     \end{bmatrix} = \sum_{i=1}^{n}{x_i y_i}$

The **inner products** are a special case of matrix multiplication.

It is always the case that $x^T y = y^T x$.

##### Example

To calculate the inner product of two vectors $x = [1 2 3 4]$ and $y = [5 6 7 8]$, we can loop through the vector and multiply and sum the scalars (this is simplified code):

```python
x = (1, 2, 3, 4)
y = (5, 6, 7, 8)
n = len(x)
if n == len(y):
    result = 0
    for i in range(n):
        result += x[i] * y[i]
    print(result)
```

It is clear that in the code above we could change line 7 to `result += y[i] * x[i]` without affecting the result.
###### [Go to top](#top)

We can use the *numpy* module to apply the same operation, to calculate the **inner product**. We import the *numpy* module and assign it a name *np* for the following code:

We define the vectors $x$ and $y$ using *numpy*:

```python
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
print("x:", x)
print("y:", y)
```

We can now calculate the $dot$ or $inner product$ using the *dot* function of *numpy*:

```python
np.dot(x, y)
```

The order of the arguments is irrelevant:

```python
np.dot(y, x)
```

Note that both vectors are actually **row vectors** in the above code. We can transpose them to column vectors by using the *shape* property:

```python
print("x:", x)
x.shape = (4, 1)
print("xT:", x)
print("y:", y)
y.shape = (4, 1)
print("yT:", y)
```

In fact, in our understanding of Linear Algebra, we take the arrays above to represent **row vectors**. *Numpy* treates them differently.

We see the issues when we try to transform the array objects. Usually, we can transform a row vector into a column vector in *numpy* by using the *T* method on vector or matrix objects:
###### [Go to top](#top)

```python
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
print("x:", x)
print("y:", y)
print("xT:", x.T)
print("yT:", y.T)
```

The problem here is that this does not do, what we expect it to do. It only works, if we declare the variables not to be arrays of numbers, but in fact a matrix:

```python
x = np.array([[1, 2, 3, 4]])
y = np.array([[5, 6, 7, 8]])
print("x:", x)
print("y:", y)
print("xT:", x.T)
print("yT:", y.T)

```

Note that the *numpy* functions *dot* and *outer* are not affected by this distinction. We can compute the dot product using the mathematical equation above in *numpy* using the new $x$ and $y$ row vectors:
###### [Go to top](#top)

```python
print("x:", x)
print("y:", y.T)
np.dot(x, y.T)
```

Or by reverting to:

```python
print("x:", x.T)
print("y:", y)
np.dot(y, x.T)
```

To read the result from this array of arrays, we would need to access the value this way:

```python
np.dot(y, x.T)[0][0]
```

## 4-2 Outer Product of Two Vectors

For two vectors $x \in \mathbb{R}^m$ and $y \in \mathbb{R}^n$, where $n$ and $m$ do not have to be equal, the **outer product** of $x$ and $y$ is:

$xy^T \in \mathbb{R}^{m\times n}$

The **outer product** results in a matrix with $m$ rows and $n$ columns by $(xy^T)_{ij} = x_i y_j$:

$xy^T \in \mathbb{R}^{m\times n} = \begin{bmatrix}
       x_1 \\[0.3em]
       x_2 \\[0.3em]
       \vdots \\[0.3em]
       x_n
     \end{bmatrix} \begin{bmatrix}
       y_1 & y_2 & \cdots & y_n
     \end{bmatrix} = \begin{bmatrix}
       x_1 y_1 & x_1 y_2 & \cdots & x_1 y_n \\[0.3em]
       x_2 y_1 & x_2 y_2 & \cdots & x_2 y_n \\[0.3em]
       \vdots  & \vdots  & \ddots & \vdots \\[0.3em]
       x_m y_1 & x_m y_2 & \cdots & x_m y_n \\[0.3em]
     \end{bmatrix}$

Some useful property of the outer product: assume $\mathbf{1} \in \mathbb{R}^n$ is an $n$-dimensional vector of scalars with the value $1$. Given a matrix $A \in \mathbb{R}^{m\times n}$ with all columns equal to some vector $x \in \mathbb{R}^m$, using the outer product $A$ can be represented as:

$A = \begin{bmatrix}
       \big| & \big| &  & \big| \\[0.3em]
       x & x & \cdots & x \\[0.3em]
       \big| & \big| &  & \big|
     \end{bmatrix} = \begin{bmatrix}
       x_1    & x_1    & \cdots & x_1    \\[0.3em]
       x_2    & x_2    & \cdots & x_2    \\[0.3em]
       \vdots & \vdots & \ddots & \vdots \\[0.3em]
       x_m    &x_m     & \cdots & x_m
     \end{bmatrix} = \begin{bmatrix}
       x_1 \\[0.3em]
       x_2 \\[0.3em]
       \vdots \\[0.3em]
       x_m
     \end{bmatrix} \begin{bmatrix}
       1 & 1 & \cdots & 1
     \end{bmatrix} = x \mathbf{1}^T$

```python
x = np.array([[1, 2, 3, 4]])
print("x:", x)
print("xT:", np.reshape(x, (4, 1)))
print("xT:", x.T)
print("xT:", x.transpose())
```

Example
###### [Go to top](#top)

We can now compute the **outer product** by multiplying the column vector $x$ with the row vector $y$:

```python
x = np.array([[1, 2, 3, 4]])
y = np.array([[5, 6, 7, 8]])
x.T * y
```

*Numpy* provides an *outer* function that does all that:

```python
np.outer(x, y)
```

Note, in this simple case using the simple arrays for the data structures of the vectors does not affect the result of the *outer* function:

```python
x = np.array([1, 2, 3, 4])
y = np.array([5, 6, 7, 8])
np.outer(x, y)
```

## 4-3 Matrix-Vector Products

Assume a matrix $A \in \mathbb{R}^{m\times n}$ and a vector $x \in \mathbb{R}^n$ the product results in a vector $y = Ax \in \mathbb{R}^m$.

$Ax$ could be expressed as the dot product of row $i$ of matrix $A$ with the column value $j$ of vector $x$. Let us first consider matrix multiplication with a scalar:
###### [Go to top](#top)

$A = \begin{bmatrix}
       1 & 2 \\[0.3em]
       3 & 4
     \end{bmatrix}$

We can compute the product of $A$ with a scalar $n = 2$ as:

$A = \begin{bmatrix}
       1 * n & 2 * n \\[0.3em]
       3 * n & 4 * n
     \end{bmatrix} = \begin{bmatrix}
       1 * 2 & 2 * 2 \\[0.3em]
       3 * 2 & 4 * 2
     \end{bmatrix} = \begin{bmatrix}
       2 & 4 \\[0.3em]
       6 & 8
     \end{bmatrix} $

Using *numpy* this can be achieved by:

```python
import numpy as np
A = np.array([[4, 5, 6],
             [7, 8, 9]])
A * 2
```

Assume that we have a column vector $x$:

$x = \begin{bmatrix}
       1 \\[0.3em]
       2 \\[0.3em]
       3
     \end{bmatrix}$

To be able to multiply this vector with a matrix, the number of columns in the matrix must correspond to the number of rows in the column vector. The matrix $A$ must have $3$ columns, as for example:

$A = \begin{bmatrix}
       4 & 5 & 6\\[0.3em]
       7 & 8 & 9
     \end{bmatrix}$

To compute $Ax$, we multiply row $1$ of the matrix with column $1$ of $x$:

$\begin{bmatrix}
  4 & 5 & 6
 \end{bmatrix}
 \begin{bmatrix}
 1 \\[0.3em]
 2 \\[0.3em]
 3
\end{bmatrix} = 4 * 1 + 5 * 2 + 6 * 3 = 32 $

We do the compute the dot product of row $2$ of $A$ and column $1$ of $x$:

$\begin{bmatrix}
  7 & 8 & 9
 \end{bmatrix}
 \begin{bmatrix}
 1 \\[0.3em]
 2 \\[0.3em]
 3
\end{bmatrix} = 7 * 1 + 8 * 2 + 9 * 3 = 50 $

The resulting column vector $Ax$ is:

$Ax = \begin{bmatrix}
       32 \\[0.3em]
       50
     \end{bmatrix}$

Using *numpy* we can compute $Ax$:

```python
A = np.array([[4, 5, 6],
             [7, 8, 9]])
x = np.array([1, 2, 3])
A.dot(x)
```

We can thus describe the product writing $A$ by rows as:

$y = Ax = \begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix} x = \begin{bmatrix}
 a_1^T x \\[0.3em]
 a_2^T x \\[0.3em]
 \vdots \\[0.3em]
 a_m^T x
\end{bmatrix}$

This means that the $i$th scalar of $y$ is the inner product of the $i$th row of $A$ and $x$, that is $y_i = a_i^T x$.

If we write $A$ in column form, then:

$y = Ax =
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 a_1 & a_2 & \cdots & a_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix}
\begin{bmatrix}
 x_1 \\[0.3em]
 x_2 \\[0.3em]
 \vdots \\[0.3em]
 x_n
\end{bmatrix} =
\begin{bmatrix}
 a_1
\end{bmatrix} x_1 +
\begin{bmatrix}
 a_2
\end{bmatrix} x_2 + \dots +
\begin{bmatrix}
 a_n
\end{bmatrix} x_n
$

In this case $y$ is a **[linear combination](https://en.wikipedia.org/wiki/Linear_combination)** of the *columns* of $A$, the coefficients taken from $x$.

The above examples multiply be the right with a column vector. One can multiply on the left by a row vector as well, $y^T = x^T A$ for $A \in \mathbb{R}^{m\times n}$, $x\in \mathbb{R}^m$, $y \in \mathbb{R}^n$. There are two ways to express $y^T$, with $A$ expressed by its columns, with $i$th scalar of $y^T$ corresponds to the inner product of $x$ and the $i$th column of $A$:

$y^T = x^T A = x^t \begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 a_1 & a_2 & \cdots & a_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} =
\begin{bmatrix}
 x^T a_1 & x^T a_2 & \dots & x^T a_n
\end{bmatrix}$

One can express $A$ by rows, where $y^T$ is a linear combination of the rows of $A$ with the scalars from $x$.

$\begin{equation}
\begin{split}
y^T & = x^T A \\
    & = \begin{bmatrix}
 x_1 & x_2 & \dots & x_n
\end{bmatrix}
\begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix} \\
   & = x_1 \begin{bmatrix}-- & a_1^T  & --\end{bmatrix} + x_2 \begin{bmatrix}-- & a_2^T  & --\end{bmatrix} + \dots + x_n \begin{bmatrix}-- & a_n^T  & --\end{bmatrix}
\end{split}
\end{equation}$

###### [Go to top](#top)

## 4-4 Matrix-Matrix Products

One can view matrix-matrix multiplication $C = AB$ as a set of vector-vector products. The $(i,j)$th entry of $C$ is the inner product of the $i$th row of $A$ and the $j$th column of $B$:

```python
matrix1 = np.matrix(
    [[0, 4],
     [2, 0]]
)
matrix2 = np.matrix(
    [[-1, 2],
     [1, -2]]
)
```

```python
matrix1 + matrix2
```

```python
matrix1 - matrix2
```

### 4-4-1  Multiplication
To multiply two matrices with numpy, you can use the np.dot method:

```python
np.dot(matrix1, matrix2)
```

```python

matrix1 * matrix2
```

$C = AB =
\begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 b_1 & b_2 & \cdots & b_p \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} =
\begin{bmatrix}
 a_1^T b_1 & a_1^T b_2 & \cdots & a_1^T b_p \\[0.3em]
 a_2^T b_1 & a_2^T b_2 & \cdots & a_2^T b_p \\[0.3em]
 \vdots    & \vdots    & \ddots & \vdots    \\[0.3em]
 a_m^T b_1 & a_m^T b_2 & \cdots & a_m^T b_p
\end{bmatrix}$

Here $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{n\times p}$, $a_i \in \mathbb{R}^n$ and $b_j \in \mathbb{R}^n$, and $A$ is represented by rows, $B$ by columns.

If we represent $A$ by columns and $B$ by rows, then $AB$ is the sum of the outer products:

$C = AB =
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 a_1 & a_2 & \cdots & a_n \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix}
\begin{bmatrix}
 -- & b_1^T  & -- \\[0.3em]
 -- & b_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & b_n^T  & --
\end{bmatrix}
= \sum_{i=1}^n a_i b_i^T
$

This means that $AB$ is the sum over all $i$ of the outer product of the $i$th column of $A$ and the $i$th row of $B$.

One can interpret matrix-matrix operations also as a set of matrix-vector products. Representing $B$ by columns, the columns of $C$ are matrix-vector products between $A$ and the columns of $B$:

$C = AB = A
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 b_1 & b_2 & \cdots & b_p \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix} =
\begin{bmatrix}
 \big| & \big| &  & \big| \\[0.3em]
 A b_1 & A b_2 & \cdots & A b_p \\[0.3em]
 \big| & \big| &  & \big|
\end{bmatrix}
$

In this interpretation the $i$th column of $C$ is the matrix-vector product with the vector on the right, i.e. $c_i = A b_i$.

Representing $A$ by rows, the rows of $C$ are the matrix-vector products between the rows of $A$ and $B$:

$C = AB = \begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}
B =
\begin{bmatrix}
 -- & a_1^T B & -- \\[0.3em]
 -- & a_2^T B & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_n^T B & --
\end{bmatrix}$

The $i$th row of $C$ is the matrix-vector product with the vector on the left, i.e. $c_i^T = a_i^T B$.

#### Notes on Matrix-Matrix Products

**Matrix multiplication is associative:** $(AB)C = A(BC)$

**Matrix multiplication is distributive:** $A(B + C) = AB + AC$

**Matrix multiplication is, in general, not commutative;** It can be the case that $AB \neq BA$. (For example, if $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{n\times q}$, the matrix product $BA$ does not even exist if $m$ and $q$ are not equal!)
###### [Go to top](#top)

## 5- Identity Matrix

The **identity matrix** $I \in \mathbb{R}^{n\times n}$ is a square matrix with the value $1$ on the diagonal and $0$ everywhere else:

```python
np.eye(4)
```

$I_{ij} = \left\{
\begin{array}{lr}
 1 & i = j\\
 0 & i \neq j
\end{array}
\right.
$

For all $A \in \mathbb{R}^{m\times n}$:

$AI = A = IA$

In the equation above multiplication has to be made possible, which means that in the portion $AI = A$ the dimensions of $I$ have to be $n\times n$, while in $A = IA$ they have to be $m\times m$.

We can generate an *identity matrix* in *numpy* using:

```python
A = np.array([[0, 1, 2],
              [3, 4, 5],
              [6, 7, 8],
              [9, 10, 11]])
print("A:", A)
```

We can ask for the shape of $A$:

```python
A.shape
```

The *shape* property of a matrix contains the $m$ (number of rows) and $n$ (number of columns) properties in a tuple, in that particular order. We can create an identity matrix for the use in $AI$ by using the $n$ value:

```python
np.identity(A.shape[1], dtype="int")
```

Note that we specify the *dtype* parameter to *identity* as *int*, since the default would return a matrix of *float* values.

To generate an identity matrix for the use in $IA$ we would use the $m$ value:

```python
np.identity(A.shape[0], dtype="int")
```

We can compute the dot product of $A$ and its identity matrix $I$:

```python
n = A.shape[1]
I = np.array(np.identity(n, dtype="int"))
np.dot(A, I)
```

The same is true for the other direction:

```python
m = A.shape[0]
I = np.array(np.identity(m, dtype="int"))
np.dot(I, A)
```

### 5-1  Inverse Matrices

```python
inverse = np.linalg.inv(matrix1)
print(inverse)
```

## 6- Diagonal Matrix

In the **diagonal matrix** non-diagonal elements are $0$, that is $D = diag(d_1, d_2, \dots{}, d_n)$, with:

$D_{ij} = \left\{
\begin{array}{lr}
 d_i & i = j\\
 0 & i \neq j
\end{array}
\right.
$

The identity matrix is a special case of a diagonal matrix: $I = diag(1, 1, \dots{}, 1)$.

In *numpy* we can create a *diagonal matrix* from any given matrix using the *diag* function:

```python
import numpy as np
A = np.array([[0,   1,  2,  3],
              [4,   5,  6,  7],
              [8,   9, 10, 11],
              [12, 13, 14, 15]])
np.diag(A)
```

An optional parameter *k* to the *diag* function allows us to extract the diagonal above the main diagonal with a positive *k*, and below the main diagonal with a negative *k*:
###### [Go to top](#top)

```python
np.diag(A, k=1)
```

```python
np.diag(A, k=-1)
```

## 7- Transpose of a Matrix

**Transposing** a matrix is achieved by *flipping* the rows and columns. For a matrix $A \in \mathbb{R}^{m\times n}$ the transpose $A^T \in \mathbb{R}^{n\times m}$ is the $n\times m$ matrix given by:

$(A^T)_{ij} = A_{ji}$

Properties of transposes:

- $(A^T)^T = A$
- $(AB)^T = B^T A^T$
- $(A+B)^T = A^T + B^T$

```python
a = np.array([[1, 2], [3, 4]])
a
```

```python
a.transpose()
```

## 8- Symmetric Matrices

Square metrices $A \in \mathbb{R}^{n\times n}$ are **symmetric**, if $A = A^T$.

$A$ is **anti-symmetric**, if $A = -A^T$.

For any matrix $A \in \mathbb{R}^{n\times n}$, the matrix $A + A^T$ is **symmetric**.

For any matrix $A \in \mathbb{R}^{n\times n}$, the matrix $A - A^T$ is **anti-symmetric**.

Thus, any square matrix $A \in \mathbb{R}^{n\times n}$ can be represented as a sum of a symmetric matrix and an anti-symmetric matrix:

$A = \frac{1}{2} (A + A^T) + \frac{1}{2} (A - A^T)$

The first matrix on the right, i.e. $\frac{1}{2} (A + A^T)$ is symmetric. The second matrix $\frac{1}{2} (A - A^T)$ is anti-symmetric.

$\mathbb{S}^n$ is the set of all symmetric matrices of size $n$.

$A \in \mathbb{S}^n$ means that $A$ is symmetric and of the size $n\times n$.

```python
def symmetrize(a):
    return a + a.T - np.diag(a.diagonal())
```

```python
a = np.array([[1, 2], [3, 4]])
print(symmetrize(a))
```

## 9-The Trace

The **trace** of a square matrix $A \in \mathbb{R}^{n\times n}$ is $tr(A)$ (or $trA$) is the sum of the diagonal elements in the matrix:

$trA = \sum_{i=1}^n A_{ii}$

Properties of the **trace**:

- For $A \in \mathbb{R}^{n\times n}$, $\mathrm{tr}A = \mathrm{tr}A^T$
- For $A,B \in \mathbb{R}^{n\times n}$, $\mathrm{tr}(A + B) = \mathrm{tr}A + \mathrm{tr}B$
- For $A \in \mathbb{R}^{n\times n}$, $t \in \mathbb{R}$, $\mathrm{tr}(tA) = t \mathrm{tr}A$
- For $A,B$ such that $AB$ is square, $\mathrm{tr}AB = \mathrm{tr}BA$
- For $A,B,C$ such that $ABC$ is square, $\mathrm{tr}ABC = \mathrm{tr}BCA = \mathrm{tr}CAB$, and so on for the product of more matrices.

###### [Go to top](#top)

```python
a = np.arange(8).reshape((2,2,2))
np.trace(a)
```

```python
print(np.trace(matrix1))
```

```python
det = np.linalg.det(matrix1)
print(det)
```

```python
a = np.array([[1, 2], [3, 4]])
a
```

```python

a.transpose()
```

# 10- Norms
a norm is a function that assigns a strictly positive length or size to each vector in a vector space—except for the zero vector, which is assigned a length of zero. A **seminorm**, on the other hand, is allowed to assign zero length to some non-zero vectors (in addition to the zero vector).

The **norm** of a vector $x$ is $\| x\|$, informally the length of a vector.

Example: the Euclidean or $\mathscr{l}_2$ norm:

$\|x\|_2 = \sqrt{\sum_{i=1}^n{x_i^2}}$

Note: $\|x\|_2^2 = x^T x$

A **norm** is any function $f : \mathbb{R}^n \rightarrow \mathbb{R}$ that satisfies the following properties:

- For all $x \in \mathbb{R}^n$, $f(x) \geq 0$ (non-negativity)
- $f(x) = 0$ if and only if $x = 0$ (definiteness)
- For all $x \in \mathbb{R}^n$, $t \in \mathbb{R}$, $f(tx) = |t|\ f(x)$ (homogeneity)
- For all $x, y \in \mathbb{R}^n$, $f(x + y) \leq f(x) + f(y)$ (triangle inequality)

Norm $\mathscr{l}_1$:

$\|x\|_1 = \sum_{i=1}^n{|x_i|}$

How to calculate norm in python? **it is so easy**
###### [Go to top](#top)

```python
v = np.array([1,2,3,4])
norm.median(v)
```

# 11- Linear Independence and Rank

A set of vectors $\{x_1, x_2, \dots{}, x_n\} \subset \mathbb{R}^m$ is said to be **(linearly) independent** if no vector can be represented as a linear combination of the remaining vectors.

A set of vectors $\{x_1, x_2, \dots{}, x_n\} \subset \mathbb{R}^m$ is said to be **(lineraly) dependent** if one vector from this set can be represented as a linear combination of the remaining vectors.

For some scalar values $\alpha_1, \dots{}, \alpha_{n-1} \in \mathbb{R}$ the vectors $x_1, \dots{}, x_n$ are linerly dependent, if:

$\begin{equation}
x_n = \sum_{i=1}^{n-1}{\alpha_i x_i}
\end{equation}$

Example: The following vectors are lineraly dependent, because $x_3 = -2 x_1 + x_2$

$x_1 = \begin{bmatrix}
 1 \\[0.3em]
 2 \\[0.3em]
 3
\end{bmatrix}
\quad
x_2 = \begin{bmatrix}
 4 \\[0.3em]
 1 \\[0.3em]
 5
\end{bmatrix}
\quad
x_3 = \begin{bmatrix}
 2 \\[0.3em]
 -1 \\[0.3em]
 -1
\end{bmatrix}
$

```python
#How to find linearly independent rows from a matrix
matrix = np.array(
    [
        [0, 1 ,0 ,0],
        [0, 0, 1, 0],
        [0, 1, 1, 0],
        [1, 0, 0, 1]
    ])

lambdas, V =  np.linalg.eig(matrix.T)
# The linearly dependent row vectors
print (matrix[lambdas == 0,:])
```

## 11-1 Column Rank of a Matrix

The **column rank** of a matrix $A \in \mathbb{R}^{m\times n}$ is the size of the largest subset of columns of $A$ that constitute a linear independent set. Informaly this is the number of linearly independent columns of $A$.
###### [Go to top](#top)

```python
A = np.matrix([[1,3,7],[2,8,3],[7,8,1]])
np.linalg.matrix_rank(A)

```

```python
from numpy.linalg import matrix_rank
matrix_rank(np.eye(4)) # Full rank matrix
```

```python
I=np.eye(4); I[-1,-1] = 0. # rank deficient matrix
matrix_rank(I)
```

```python
matrix_rank(np.ones((4,))) # 1 dimension - rank 1 unless all 0
```

```python
matrix_rank(np.zeros((4,)))
```

## 11-2 Row Rank of a Matrix

The **row rank** of a matrix $A \in \mathbb{R}^{m\times n}$ is the largest number of rows of $A$ that constitute a lineraly independent set.

## 11-3 Rank of a Matrix

For any matrix $A \in \mathbb{R}^{m\times n}$, the column rank of $A$ is equal to the row rank of $A$. Both quantities are referred to collectively as the rank of $A$, denoted as $rank(A)$. Here are some basic properties of the rank:
###### [Go to top](#top)

- For $A \in \mathbb{R}^{m\times n}$, $rank(A) \leq \min(m, n)$. If $rank(A) = \min(m, n)$, then $A$ is said to be
**full rank**.
- For $A \in \mathbb{R}^{m\times n}$, $rank(A) = rank(A^T)$
- For $A \in \mathbb{R}^{m\times n}$, $B \in \mathbb{R}^{n\times p}$, $rank(AB) \leq \min(rank(A), rank(B))$
- For $A,B \in \mathbb{R}^{m\times n}$, $rank(A + B) \leq rank(A) + rank(B)$

```python
from numpy.linalg import matrix_rank
print(matrix_rank(np.eye(4))) # Full rank matrix
I=np.eye(4); I[-1,-1] = 0. # rank deficient matrix
print(matrix_rank(I))
print(matrix_rank(np.ones((4,)))) # 1 dimension - rank 1 unless all 0
print (matrix_rank(np.zeros((4,))))

```

# 12-  Subtraction and Addition of Metrices

Assume $A \in \mathbb{R}^{m\times n}$ and $B \in \mathbb{R}^{m\times n}$, that is $A$ and $B$ are of the same size, to add $A$ to $B$, or to subtract $B$ from $A$, we add or subtract corresponding entries:

$A + B =
\begin{bmatrix}
 a_{11} & a_{12} & \cdots & a_{1n} \\[0.3em]
 a_{21} & a_{22} & \cdots & a_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} & a_{m2} & \cdots & a_{mn}
\end{bmatrix} +
\begin{bmatrix}
 b_{11} & b_{12} & \cdots & b_{1n} \\[0.3em]
 b_{21} & b_{22} & \cdots & b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 b_{m1} & b_{m2} & \cdots & b_{mn}
\end{bmatrix} =
\begin{bmatrix}
 a_{11} + b_{11} & a_{12} + b_{12} & \cdots & a_{1n} + b_{1n} \\[0.3em]
 a_{21} + b_{21} & a_{22} + b_{22} & \cdots & a_{2n} + b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} + b_{m1} & a_{m2} + b_{m2} & \cdots & a_{mn} + b_{mn}
\end{bmatrix}
$

The same is applies to subtraction:

$A - B =
\begin{bmatrix}
 a_{11} & a_{12} & \cdots & a_{1n} \\[0.3em]
 a_{21} & a_{22} & \cdots & a_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} & a_{m2} & \cdots & a_{mn}
\end{bmatrix} -
\begin{bmatrix}
 b_{11} & b_{12} & \cdots & b_{1n} \\[0.3em]
 b_{21} & b_{22} & \cdots & b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 b_{m1} & b_{m2} & \cdots & b_{mn}
\end{bmatrix} =
\begin{bmatrix}
 a_{11} - b_{11} & a_{12} - b_{12} & \cdots & a_{1n} - b_{1n} \\[0.3em]
 a_{21} - b_{21} & a_{22} - b_{22} & \cdots & a_{2n} - b_{2n} \\[0.3em]
 \vdots & \vdots & \ddots & \vdots \\[0.3em]
 a_{m1} - b_{m1} & a_{m2} - b_{m2} & \cdots & a_{mn} - b_{mn}
\end{bmatrix}
$

In Python using *numpy* this can be achieved using the following code:

```python
import numpy as np
print("np.arange(9):", np.arange(9))
print("np.arange(9, 18):", np.arange(9, 18))
A = np.arange(9, 18).reshape((3, 3))
B = np.arange(9).reshape((3, 3))
print("A:", A)
print("B:", B)
```

The *numpy* function *arange* is similar to the standard Python function *range*. It returns an array with $n$ elements, specified in the one parameter version only. If we provide to parameters to *arange*, it generates an array starting from the value of the first parameter and ending with a value one less than the second parameter. The function *reshape* returns us a matrix with the corresponding number of rows and columns.

We can now add and subtract the two matrices $A$ and $B$:

```python
A + B
```

```python
A - B
```

## 12-1 Inverse

The **inverse** of a square matrix $A \in \mathbb{R}^{n\times n}$ is $A^{-1}$:

$A^{-1} A = I = A A^{-1}$

Not all matrices have inverses. Non-square matrices do not have inverses by definition. For some square matrices $A$ the inverse might not exist.

$A$ is **invertible** or **non-singular** if $A^{-1}$ exists.

$A$ is **non-invertible** or **singular** if $A^{-1}$ does not exist.

Note: **non-singular** means the opposite of **non-invertible**!

For $A$ to have an inverse $A^{-1}$, $A$ must be **full rank**.

Assuming that $A,B \in \mathbb{R}^{n\times n}$ are non-singular, then:

- $(A^{-1})^{-1} = A$
- $(AB)^{-1} = B^{-1} A^{-1}$
- $(A^{-1})^T = (A^T)^{-1}$ (often simply $A^{-T}$)
###### [Go to top](#top)

## 13- Orthogonal Matrices

Two vectors $x, y \in \mathbb{R}^n$ are **orthogonal** if $x^T y = 0$.

A vector $x \in \mathbb{R}^n$ is **normalized** if $\|x\|^2 = 1$.

A square matrix $U \in \mathbb{R}^{n\times n}$ is **orthogonal** if all its columns are orthogonal to each other and are **normalized**. The columns are then referred to as being **orthonormal**.

It follows immediately from the definition of orthogonality and normality that:

$U^T U = I = U U^T$

This means that the inverse of an orthogonal matrix is its transpose.

If U is not square - i.e., $U \in \mathbb{R}^{m\times n}$, $n < m$ - but its columns are still orthonormal, then $U^T U = I$, but $U U^T \neq I$.

We generally only use the term orthogonal to describe the case, where $U$ is square.

Another nice property of orthogonal matrices is that operating on a vector with an orthogonal matrix will not change its Euclidean norm. For any $x \in \mathbb{R}^n$, $U \in \mathbb{R}^{n\times n}$ orthogonal.

$\|U_x\|^2 = \|x\|^2$

```python
#How to create random orthonormal matrix in python numpy
def rvs(dim=3):
     random_state = np.random
     H = np.eye(dim)
     D = np.ones((dim,))
     for n in range(1, dim):
         x = random_state.normal(size=(dim-n+1,))
         D[n-1] = np.sign(x[0])
         x[0] -= D[n-1]*np.sqrt((x*x).sum())
         # Householder transformation
         Hx = (np.eye(dim-n+1) - 2.*np.outer(x, x)/(x*x).sum())
         mat = np.eye(dim)
         mat[n-1:, n-1:] = Hx
         H = np.dot(H, mat)
         # Fix the last sign such that the determinant is 1
     D[-1] = (-1)**(1-(dim % 2))*D.prod()
     # Equivalent to np.dot(np.diag(D), H) but faster, apparently
     H = (D*H.T).T
     return H
```

## 14- Range and Nullspace of a Matrix

The **span** of a set of vectors $\{ x_1, x_2, \dots{}, x_n\}$ is the set of all vectors that can be expressed as
a linear combination of $\{ x_1, \dots{}, x_n \}$:

$\mathrm{span}(\{ x_1, \dots{}, x_n \}) = \{ v : v = \sum_{i=1}^n \alpha_i x_i, \alpha_i \in \mathbb{R} \}$

It can be shown that if $\{ x_1, \dots{}, x_n \}$ is a set of n linearly independent vectors, where each $x_i \in \mathbb{R}^n$, then $\mathrm{span}(\{ x_1, \dots{}, x_n\}) = \mathbb{R}^n$. That is, any vector $v \in \mathbb{R}^n$ can be written as a linear combination of $x_1$ through $x_n$.

The projection of a vector $y \in \mathbb{R}^m$ onto the span of $\{ x_1, \dots{}, x_n\}$ (here we assume $x_i \in \mathbb{R}^m$) is the vector $v \in \mathrm{span}(\{ x_1, \dots{}, x_n \})$, such that $v$ is as close as possible to $y$, as measured by the Euclidean norm $\|v − y\|^2$. We denote the projection as $\mathrm{Proj}(y; \{ x_1, \dots{}, x_n \})$ and can define it formally as:

$\mathrm{Proj}( y; \{ x_1, \dots{}, x_n \}) = \mathrm{argmin}_{v\in \mathrm{span}(\{x_1,\dots{},x_n\})}\|y − v\|^2$

The **range** (sometimes also called the columnspace) of a matrix $A \in \mathbb{R}^{m\times n}$, denoted $\mathcal{R}(A)$, is the the span of the columns of $A$. In other words,

$\mathcal{R}(A) = \{ v \in \mathbb{R}^m : v = A x, x \in \mathbb{R}^n\}$

Making a few technical assumptions (namely that $A$ is full rank and that $n < m$), the projection of a vector $y \in \mathbb{R}^m$ onto the range of $A$ is given by:

$\mathrm{Proj}(y; A) = \mathrm{argmin}_{v\in \mathcal{R}(A)}\|v − y\|^2 = A(A^T A)^{−1} A^T y$

See for more details in the notes page 13.

The **nullspace** of a matrix $A \in \mathbb{R}^{m\times n}$, denoted $\mathcal{N}(A)$ is the set of all vectors that equal $0$ when multiplied by $A$, i.e.,

$\mathcal{N}(A) = \{ x \in \mathbb{R}^n : A x = 0 \}$

Note that vectors in $\mathcal{R}(A)$ are of size $m$, while vectors in the $\mathcal{N}(A)$ are of size $n$, so vectors in $\mathcal{R}(A^T)$ and $\mathcal{N}(A)$ are both in $\mathbb{R}^n$. In fact, we can say much more. It turns out that:

$\{ w : w = u + v, u \in \mathcal{R}(A^T), v \in \mathcal{N}(A) \} = \mathbb{R}^n$ and $\mathcal{R}(A^T) \cap \mathcal{N}(A) = \{0\}$

In other words, $\mathcal{R}(A^T)$ and $\mathcal{N}(A)$ are disjoint subsets that together span the entire space of
$\mathbb{R}^n$. Sets of this type are called **orthogonal complements**, and we denote this $\mathcal{R}(A^T) = \mathcal{N}(A)^\perp$.

###### [Go to top](#top)

# 15-  Determinant

The determinant of a square matrix $A \in \mathbb{R}^{n\times n}$, is a function $\mathrm{det} : \mathbb{R}^{n\times n} \rightarrow \mathbb{R}$, and is denoted $|A|$ or $\mathrm{det}A$ (like the trace operator, we usually omit parentheses).

## 15-1 A geometric interpretation of the determinant

Given

$\begin{bmatrix}
 -- & a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_n^T  & --
\end{bmatrix}$

consider the set of points $S \subset \mathbb{R}^n$ formed by taking all possible linear combinations of the row vectors $a_1, \dots{}, a_n \in \mathbb{R}^n$ of $A$, where the coefficients of the linear combination are all
between $0$ and $1$; that is, the set $S$ is the restriction of $\mathrm{span}( \{ a_1, \dots{}, a_n \})$ to only those linear combinations whose coefficients $\alpha_1, \dots{}, \alpha_n$ satisfy $0 \leq \alpha_i \leq 1$, $i = 1, \dots{}, n$. Formally:

$S = \{v \in \mathbb{R}^n : v = \sum_{i=1}^n \alpha_i a_i \mbox{ where } 0 \leq \alpha_i \leq 1, i = 1, \dots{}, n \}$

The absolute value of the determinant of $A$, it turns out, is a measure of the *volume* of the set $S$. The volume here is intuitively for example for $n = 2$ the area of $S$ in the Cartesian plane, or with $n = 3$ it is the common understanding of *volume* for 3-dimensional objects.

Example:

$A = \begin{bmatrix}
 1 & 3 & 4\\[0.3em]
 3 & 2 & 1\\[0.3em]
  3 & 2 & 1
\end{bmatrix}$

The rows of the matrix are:

$a_1 = \begin{bmatrix}
 1 \\[0.3em]
 3 \\[0.3em]
 3
\end{bmatrix}
\quad
a_2 = \begin{bmatrix}
 3 \\[0.3em]
 2 \\[0.3em]
 2
\end{bmatrix}
\quad
a_3 = \begin{bmatrix}
 4 \\[0.3em]
 1 \\[0.3em]
 1
\end{bmatrix}$

The set S corresponding to these rows is shown in:

The figure above is an illustration of the determinant for the $2\times 2$ matrix $A$ above. Here, $a_1$ and $a_2$
are vectors corresponding to the rows of $A$, and the set $S$ corresponds to the shaded region (i.e., the parallelogram). The absolute value of the determinant, $|\mathrm{det}A| = 7$, is the area of the parallelogram.

For two-dimensional matrices, $S$ generally has the shape of a parallelogram. In our example, the value of the determinant is $|A| = −7$ (as can be computed using the formulas shown later), so the area of the parallelogram is $7$.

In three dimensions, the set $S$ corresponds to an object known as a parallelepiped (a three-dimensional box with skewed sides, such that every face has the shape of a parallelogram). The absolute value of the determinant of the $3 \times 3$ matrix whose rows define $S$ give the three-dimensional volume of the parallelepiped. In even higher dimensions, the set $S$ is an object known as an $n$-dimensional parallelotope.

Algebraically, the determinant satisfies the following three properties (from which all other properties follow, including the general formula):

- The determinant of the identity is $1$, $|I| = 1$. (Geometrically, the volume of a unit hypercube is $1$).
- Given a matrix $A \in \mathbb{R}^{n\times n}$, if we multiply a single row in $A$ by a scalar $t \in \mathbb{R}$, then the determinant of the new matrix is $t|A|$,

$\left| \begin{bmatrix}
 -- & t a_1^T  & -- \\[0.3em]
 -- & a_2^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}\right|  = t|A|$

(Geometrically, multiplying one of the sides of the set $S$ by a factor $t$ causes the volume
to increase by a factor $t$.)
- If we exchange any two rows $a^T_i$ and $a^T_j$ of $A$, then the determinant of the new matrix is $−|A|$, for example

$\left| \begin{bmatrix}
 -- & a_2^T  & -- \\[0.3em]
 -- & a_1^T  & -- \\[0.3em]
    & \vdots &  \\[0.3em]
 -- & a_m^T  & --
\end{bmatrix}\right|  = -|A|$

Several properties that follow from the three properties above include:

- For $A \in \mathbb{R}^{n\times n}$, $|A| = |A^T|$
- For $A,B \in \mathbb{R}^{n\times n}$, $|AB| = |A||B|$
- For $A \in \mathbb{R}^{n\times n}$, $|A| = 0$ if and only if $A$ is singular (i.e., non-invertible). (If $A$ is singular then it does not have full rank, and hence its columns are linearly dependent. In this case, the set $S$ corresponds to a "flat sheet" within the $n$-dimensional space and hence has zero volume.)
- For $A \in \mathbb{R}^{n\times n}$ and $A$ non-singular, $|A−1| = 1/|A|$
###### [Go to top](#top)

# 16- Tensors

A [**tensor**](https://en.wikipedia.org/wiki/Tensor) could be thought of as an organized multidimensional array of numerical values. A vector could be assumed to be a sub-class of a tensor. Rows of tensors extend alone the y-axis, columns along the x-axis. The **rank** of a scalar is 0, the rank of a **vector** is 1, the rank of a **matrix** is 2, the rank of a **tensor** is 3 or higher.

###### [Go to top](#top)

```python
A = tf.Variable(np.zeros((5, 5), dtype=np.float32), trainable=False)
new_part = tf.ones((2,3))
update_A = A[2:4,2:5].assign(new_part)
sess = tf.InteractiveSession()
tf.global_variables_initializer().run()
print(update_A.eval())
```

# 17- Hyperplane

The **hyperplane** is a sub-space in the ambient space with one dimension less. In a two-dimensional space the hyperplane is a line, in a three-dimensional space it is a two-dimensional plane, etc.

Hyperplanes divide an $n$-dimensional space into sub-spaces that might represent clases in a machine learning algorithm.

```python
np.random.seed(0)
X = np.r_[np.random.randn(20, 2) - [2, 2], np.random.randn(20, 2) + [2, 2]]
Y = [0] * 20 + [1] * 20

fig, ax = plt.subplots()
clf2 = svm.LinearSVC(C=1).fit(X, Y)

# get the separating hyperplane
w = clf2.coef_[0]
a = -w[0] / w[1]
xx = np.linspace(-5, 5)
yy = a * xx - (clf2.intercept_[0]) / w[1]

# create a mesh to plot in
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx2, yy2 = np.meshgrid(np.arange(x_min, x_max, .2),
                     np.arange(y_min, y_max, .2))
Z = clf2.predict(np.c_[xx2.ravel(), yy2.ravel()])

Z = Z.reshape(xx2.shape)
ax.contourf(xx2, yy2, Z, cmap=plt.cm.coolwarm, alpha=0.3)
ax.scatter(X[:, 0], X[:, 1], c=Y, cmap=plt.cm.coolwarm, s=25)
ax.plot(xx,yy)

ax.axis([x_min, x_max,y_min, y_max])
plt.show()
```

## 20- Exercises
let's do some exercise.

```python
# Students may (probably should) ignore this code. It is just here to make pretty arrows.

def plot_vectors(vs):
    """Plot vectors in vs assuming origin at (0,0)."""
    n = len(vs)
    X, Y = np.zeros((n, 2))
    U, V = np.vstack(vs).T
    plt.quiver(X, Y, U, V, range(n), angles='xy', scale_units='xy', scale=1)
    xmin, xmax = np.min([U, X]), np.max([U, X])
    ymin, ymax = np.min([V, Y]), np.max([V, Y])
    xrng = xmax - xmin
    yrng = ymax - ymin
    xmin -= 0.05*xrng
    xmax += 0.05*xrng
    ymin -= 0.05*yrng
    ymax += 0.05*yrng
    plt.axis([xmin, xmax, ymin, ymax])
```

```python
# Again, this code is not intended as a coding example.

a1 = np.array([3,0])         # axis
a2 = np.array([0,3])

plt.figure(figsize=(8,4))
plt.subplot(1,2,1)
plot_vectors([a1, a2])
v1 = np.array([2,3])
plot_vectors([a1,v1])
plt.text(2,3,"(2,3)",fontsize=16)
plt.tight_layout()

```

```python
#Matrices, Transformations and Geometric Interpretation
a1 = np.array([7,0])         # axis
a2 = np.array([0,5])

A = np.array([[2,1],[1,1]])  # transformation f in standard basis
v2 =np.dot(A,v1)
plt.figure(figsize=(8,8))
plot_vectors([a1, a2])
v1 = np.array([2,3])
plot_vectors([v1,v2])
plt.text(2,3,"v1 =(2,3)",fontsize=16)
plt.text(6,5,"Av1 = ", fontsize=16)
plt.text(v2[0],v2[1],"(7,5)",fontsize=16)
print(v2[1])
```

```python
#Change to a Different Basis
e1 = np.array([1,0])
e2 = np.array([0,1])
B = np.array([[1,4],[3,1]])
plt.figure(figsize=(8,4))
plt.subplot(1,2,1)
plot_vectors([e1, e2])
plt.subplot(1,2,2)
plot_vectors([B.dot(e1), B.dot(e2)])
plt.Circle((0,0),2)
#plt.show()
#plt.tight_layout()
```

```python
#Inner Products
e1 = np.array([1,0])
e2 = np.array([0,1])
A = np.array([[2,3],[3,1]])
v1=A.dot(e1)
v2=A.dot(e2)
plt.figure(figsize=(8,4))
plt.subplot(1,2,1)
plot_vectors([e1, e2])
plt.subplot(1,2,2)
plot_vectors([v1,v2])
plt.tight_layout()
#help(plt.Circle)
plt.Circle(np.array([0,0]),radius=1)
plt.Circle.draw
```

```python
# using sqrt() to print the square root of matrix
print ("The element wise square root is : ")
print (np.sqrt(x))
```

# 21-Conclusion
If you have made this far – give yourself a pat at the back. We have covered different aspects of **Linear algebra** in this Kernel. You are now finishing the **third step** of the course to continue, return to the [**main page**](https://www.kaggle.com/mjbahmani/10-steps-to-become-a-data-scientist/) of the course.

###### [Go to top](#top)

you can follow me on:
> ###### [ GitHub](https://github.com/mjbahmani/)
> ###### [Kaggle](https://www.kaggle.com/mjbahmani/)

 I hope you find this kernel helpful and some UPVOTES would be very much appreciated.

# 22-References

1. [Linear Algbra1](https://github.com/dcavar/python-tutorial-for-ipython)
1. [Linear Algbra2](https://www.oreilly.com/library/view/data-science-from/9781491901410/ch04.html)
1. [GitHub](https://github.com/mjbahmani/10-steps-to-become-a-data-scientist)

>###### If you have read the notebook, you can follow next steps: [**Course Home Page**](https://www.kaggle.com/mjbahmani/10-steps-to-become-a-data-scientist)

3- Mathematics and Linear Algebra/Linear Algebra for Data Scientists.ipynb


```python
from IPython.core.display import HTML
HTML("<style>.container { width:95% !important; }</style>")
```

<style>.container { width:95% !important; }</style>

## Optimization problem to be studied

We will start studying functions of multiple variables by studying unconstrained optimization problems
$$
\begin{align}
\min \quad &f(x)\\
\text{s.t.}\quad &x\in \mathbb R^n
\end{align}
$$

As an example, we study the optimization problem
$$
\begin{align}
\min \quad & (x_1-10)^2+(x_2+5)^2+x_1^2\\
\text{s.t.}\quad &x_1,x_2\in\mathbb R
\end{align}
$$
This problem is unconstrained, because there are no constraints.

Now we need to define a function in Python that evaluates that function. That is, we define a two-variable function
$$f:(x_1,x_2)\to (x_1-10)^2+(x_2+5)^2+x_1^2$$

```python
def f_simple(x):
    return (x[0] - 10.0)**2 + (x[1] + 5.0)**2+x[0]**2
```

Now we can call that function

```python
print "At point (3,-8) the value of the function is ",f_simple([3,-8])
```

    At point (3,-8) the value of the function is  67.0

We can also plot that function

```python
import numpy as np
from pylab import meshgrid
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

def plot_2d_function(lb1,lb2,ub1,ub2,f):
    x = np.arange(lb1,ub1,0.1)
    y = np.arange(lb2,ub2,0.1)
    X,Y = meshgrid(x, y) # grid of point
    Z = [f([x,y]) for (x,y) in zip (X,Y)] # evaluation of the function on the grid
    fig = plt.figure()
    ax = fig.gca(projection='3d')
    surf = ax.plot_surface(X, Y, Z)
    return plt
```

```python
plot_2d_function(3,-7,7,-3,f_simple).show()
```

    /usr/lib64/python2.7/site-packages/matplotlib/collections.py:590: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
      if self._edgecolors == str('face'):

# Direct search methods

Direct search methods (also called pattern search methods) just rely on the function values to find a (local) optimum. Direct search methods consist of a set of
1. exploratory moves that acquire information about the function $f$ in th eneighbourhood of the current solution, and
2. pattern moves that attempt to speed up the search using the information acquired in the exploratory moves.

## Hooke&Jeeves

**input:** a minimum step length $L>0$, initial step length $\epsilon_0$, constant $0<\delta<1$ for reducing the step length, exploratory step multiplier $\gamma>1$, and starting solution $x_0$
**output:** an approximation of a local optimum (no guarantees of quality in general cases)
```
set epsilon as the initial step length epsilon0
set x as the starting solution
while epsilon is greater than L:
    for each coordinate direction i:
        find the smallest of function values by incrementing and reducing the variable value in that coordinate by epsilon, let this value be xi*
    if x is the same as (x1*,...,xn*):
        reduce epsilon to delta*epsilon
    else:
        if the value of f at (x1*,...,xn*) is smaller than at x+gamma*((x1*,...,xn*)-x):
            set x as (x1*,...xn*)
        else:
            set x as x+gamma*((x1*,...,xn*)-x)
return x

```
Thus, the exploratory step of Hooke&Jeeves is performed by incrementing and reducing the variable to each coordinate direction and the pattern move is just a multiplication of the exploratory move.

```python
import copy #Copying vectors
import numpy as np #Import vector calculus and much more!
def hookejeeves(L,epsilon0,delta,gamma,x0,f):
    #Set up the initial values
    epsilon = epsilon0
    x = np.array(x0)
    #Loop while step length greater than L:
    while epsilon>L:
        #our exploratory move is initially [0,..,0]
        xtest = np.zeros(len(x))
        for coordinate in range(len(x)):
            #First points to be explored are the all x, to be changed
            exp_points = [copy.copy(x) for _ in range(3)] #points to be explored
            #Change exp_points[0] and exp_points[1] to reflect
            #moving along the coordinate
            exp_points[0][coordinate]-=epsilon
            exp_points[1][coordinate]+=epsilon
            #Assign the function values given by exp_points to a list
            f_exp_points = [f(exp_point) for exp_point in exp_points]
            #pick the smallest one of them
            min_value = min(f_exp_points)
            #The exploratory move to the coordinate direction is given by the
            #move giving the smallest value of f
            xtest[coordinate] = exp_points[f_exp_points.index(min_value)][coordinate] #The coordinate value is the one where the minimum is attained
        #If no move at all, then reduce the exploratory move step size
        if all(xtest==x):
            epsilon = delta*epsilon
        else:
            #if exploratory move is better than pattern move
            if f(xtest)<f(x+gamma*(xtest-x)):
                #...set x as the exploratory move
                x = xtest
            else:
                #Otherwise we take the pattern move
                x = x+gamma*(xtest-x)
    return x
```

```python
L = 0.001
epsilon0 = 1
delta = 0.1
gamma = 2.0
start = [0.0,0.0]
#%timeit hookejeeves(L,epsilon0,delta,gamma,x0,f_simple)
x = hookejeeves(L,epsilon0,delta,gamma,start,f_simple); print "Optimal solution is" + str(x)
```

    Optimal solution is[ 5. -5.]

### How does it actually work?

Let us define a function that can plot a set of steps:

```python
%pylab inline
import matplotlib.pyplot as plt

def plot_2d_steps(steps,start):
    myvec = np.array([start]+steps).transpose()
    plt.plot(myvec[0,],myvec[1,],'ro')
    for label,x,y in zip([str(i) for i in range(len(steps)+1)],myvec[0,],myvec[1,]):
        plt.annotate(label,xy = (x, y))
    return plt
```

    Populating the interactive namespace from numpy and matplotlib

    WARNING: pylab import has clobbered these variables: ['copy', 'gamma']
    `%matplotlib` prevents importing * from pylab and numpy

Now we need to save the steps of the algorithm. For that, let us modify the algorithm slightly:

```python
import copy #Copying vectors
import numpy as np #Import vector calculus and much more!
def hookejeeves_savesteps(L,epsilon0,delta,gamma,x0,f):
    epsilon = epsilon0
    x = np.array(x0)
    steps = []
    while epsilon>L:
        xtest = np.zeros(len(x))
        for coordinate in range(len(x)):
            exp_points = [copy.copy(x) for _ in range(3)] #points to be explored
            exp_points[0][coordinate]-=epsilon
            exp_points[1][coordinate]+=epsilon
            f_exp_points = [f(exp_point) for exp_point in exp_points]
            min_value = min(f_exp_points)
            xtest[coordinate] = exp_points[f_exp_points.index(min_value)][coordinate] #The coordinate value is the one where the minimum is attained
        if all(xtest==x):
            epsilon = delta*epsilon
        else:
            if f(xtest)<f(x+gamma*(xtest-x)):
                x = xtest
            else:
                x = x+gamma*(xtest-x)
            steps.append(x)
    return x,steps
```

```python
L = 0.001
epsilon0 = 1.0
delta = 0.1
gamma = 2
start = [0.,0.]
(x,steps) = hookejeeves_savesteps(L,epsilon0,delta,gamma,start,f_simple)
```

```python
plot_2d_steps(steps,start).show()
```

## Powell's method

Powell's method is similar to Hooke&Jeeves, but the first step in exploratory moves is taken to the direction of the last pattern move. This speeds up the convergence in most cases.

```python
import copy
import numpy as np
def powell(L,epsilon0,delta,gamma,x0,f):
    epsilon = epsilon0
    exp_direction = np.array([0,1])
    x = np.array(x0)
    while epsilon>L:
        #Comparing among exploratory points to firtst exploratory direction:
        if f(x+exp_direction)<f(x):
            exp_step1=exp_direction
        elif f(x-exp_direction)<f(x):
            exp_step1=-exp_direction
        else:
            exp_step1 = np.zeros(2)
        #The following only works in 2d!!
        exp_direction2 = np.array([exp_direction[1],-exp_direction[0]])
        if f(x+exp_direction2)<f(x):
            exp_step2=exp_direction2
        elif f(x-exp_direction2)<f(x):
            exp_step2=-exp_direction2
        else:
            exp_step2 = np.zeros(2)
        if all(exp_step1+exp_step2==0):
            epsilon = delta*epsilon
        else:
            if f(x+(exp_step1+exp_step2))<f(x+gamma*(exp_step1+exp_step2)):
                x = x+(exp_step1+exp_step2)
            else:
                x = x+gamma*(exp_step1+exp_step2)
            exp_direction = epsilon*(exp_step1+exp_step2)/np.linalg.norm(exp_step1+exp_step2)
    return x
```

```python
L = 0.001
epsilon0 = 1.
delta = 0.1
gamma = 2.0
start = [0.,0.]
#%timeit powell(L,epsilon0,delta,gamma,x0,f_simple)
x = powell(L,epsilon0,delta,gamma,start,f_simple)
print "Optimal solution is" + str(x)
```

    Optimal solution is[ 4.82842712 -5.        ]

### How does it actually work?

Let us again modify the algorithm slightly:

```python
import copy
import numpy as np
def powell_savesteps(L,epsilon0,delta,gamma,x0,f):
    epsilon = epsilon0
    exp_direction = np.array([0,1])
    x = np.array(x0)
    steps = []
    while epsilon>L:
        exp_direction=epsilon*exp_direction
        #Comparing among exploratory points to firtst exploratory direction:
        if f(x+exp_direction)<f(x):
            exp_step1=exp_direction
        elif f(x-exp_direction)<f(x):
            exp_step1=-exp_direction
        else:
            exp_step1 = np.zeros(2)
        #The following only works in 2d!!
        exp_direction2 = np.array([exp_direction[1],-exp_direction[0]])
        if f(x+exp_direction2)<f(x):
            exp_step2=exp_direction2
        elif f(x-exp_direction2)<f(x):
            exp_step2=-exp_direction2
        else:
            exp_step2 = np.zeros(2)
        if all(exp_step1+exp_step2==0):
            epsilon = delta*epsilon
        else:
            if f(x+(exp_step1+exp_step2))<f(x+gamma*(exp_step1+exp_step2)):
                x = x+(exp_step1+exp_step2)
            else:
                x = x+gamma*(exp_step1+exp_step2)
            steps.append(x)
            exp_direction = (exp_step1+exp_step2)/np.linalg.norm(exp_step1+exp_step2)
    return x,steps
```

```python
L = 0.001
epsilon0 = 10
delta = 0.01
gamma = 2.0
start = [-2.,1.]
(x,steps) = powell_savesteps(L,epsilon0,delta,gamma,start,f_simple)
```

```python
plot_2d_steps(steps,start).show()
```

Lecture 3, Direct search methods; Hooke&Jeeves and Powell's method.ipynb


 ILI286 - Computación Científica II

 EDP Parabólicas: Diferencias Finitas

  [S]cientific [C]omputing [T]eam

 Version: 1.2

# Tabla de Contenidos
* [Introducción](#intro)
* [EDP Parabólica: Ecuación de Onda](#para)
* [Diferencias Finitas (Forward Difference)](#fd)
* [Diferencias Finitas (Backward Difference)](#bd)
* [Ecuación de calor 1D: Condición de borde de Dirichlet](#heat1d:dirichlet)
* [Ecuación de calor 1D: Condición de borde Periódicas](#heat1d:periodic)
* [Ecuación de calor 2D](#heat2d)
* [Acknowledgements](#acknowledgements)

### Librerias necesarias

```python
%matplotlib notebook

import numpy as np

from matplotlib import pyplot as plt
from matplotlib import animation
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
from mpl_toolkits.mplot3d import Axes3D

from IPython.display import HTML
```

```python
def show(x, t, u):
    Nt = len(t)
    fig, ax = plt.subplots()
    l, = plt.plot(x, u[:,0],'ko:')
    dx = 0.05*(x[-1]-x[0])
    ymin = u.min().min()
    ymax = u.max().max()
    dy = 0.05*(ymax-ymin)
    ax.set_xlim([x[0]-dx, x[-1]+dx])
    ax.set_ylim([ymin-dy, ymax+dy])
    plt.xlabel("$x$", fontsize=20)
    plt.ylabel("$u(x,t)$", fontsize=20)

    def animate(i):
        l.set_ydata(u[:,i])
        return l,

    #Init only required for blitting to give a clean slate.
    def init():
        l.set_ydata(np.ma.array(u[:,0], mask=True))
        return l,

    dt = t[1]-t[0]
    #interval = 4 * 100. * 200/Nt   # So simulations run in the same time regardless of Nt
    interval = 100
    anim = animation.FuncAnimation(
        fig, animate, np.arange(1, Nt),
        init_func=init, interval=interval, blit=True
    )
    return anim

def animate(all_sims, interval=100):
    NumSims = all_sims.shape[0]
    fig, ax = plt.subplots()
    img = plt.imshow(all_sims[0,:,:], cmap=cm.coolwarm, interpolation='nearest', origin='lower')
    plt.xticks(range(0,101,10), np.linspace(0.,1.0,11))
    plt.yticks(range(10,101,10), np.linspace(0.1,1.0,10))
    plt.colorbar(img)

    def animate(i):
        img.set_data(all_sims[i,:,:])
        return img,

    #Init only required for blitting to give a clean slate.
    def init():
        img.set_data(all_sims[0,:,:])
        return img,

    anim = animation.FuncAnimation(
        fig, animate, np.arange(1, NumSims),
        init_func=init, interval=interval, blit=True
    )
    return anim
```

## EDP Parabólica: Ecuación de Onda

En esta sección estudiaremos una clase distinta (y muy común) de EDPs: Parabólicas, en partícular, la **ecuación de calor**:

$$
u_t(x,t) = D\, u_{xx}(x,t).
$$

Este corresponde a un modelo de difusión, que explica como una variable $u(x,t)$ a lo largo de una dimensión espacial $x$, se propaga desde regiones de mayores concentraciones hacia menores concentraciones, a lo largo del tiempo $t$, hasta alcanzar un estado estable.  En este sentido el coeficiente $D$ representa la capacidad de dufisividad, esto es, la fácilidad con la que puede propagarse a lo largo del medio.

## Diferencias Finitas (Forward Difference)

Similar a como se hizo para solucionar numéricamente ODEs, se puede discretizar cada derivada parcial de la PDE con *finite differences*.  Sea la siguiente PDE (ecuación de calor) con sus condiciones de borde:

\begin{align*}
u_t(x,t) &= D\, u_{xx}(x,t) \\
u(x,0) &= f(x) \\
u(a,t) &= l(t) \\
u(b,t) &= r(t) \\
\end{align*}

Un primer enfoque es utilizar *forward difference* para aproximar la derivada temporal, y *centered difference formula* para la segunda derivada espacial, del siguiente modo:

\begin{align}
u_t(x,t) &\approx \frac{1}{k} \left( u(x, t+k) - u(x,t) \right) \\
u_{xx}(x,t) &\approx \frac{1}{h^2} \left( u(x+h,t) - 2 u(x,t) + u(x-h, t)  \right),
\end{align}

siendo $k,h$ el *time step* y *spatial step* respectivamente. Si por simplicidad se denota $w_{i,j} = u(x_i, t_j)$, entonces la discretización de la ecuación queda del siguiente modo:

$$
\frac{1}{k} (w_{i,j+1} - w_{i,j}) \approx \frac{D}{h^2} ( w_{i+1,j}-2 w_{i,j} + w_{i-1,j} ),
$$

con error de aproximación del orden $O(k) + O(h^2)$.

(**Idea**) La idea del método es la siguiente; Dado que por la condiciones de borde y las condiciones iniciales, sabemos los valores de $w_{0,j}, w_{M,j}, w_{i,0} \ \ \forall i,j$, determinar sucesivamente los valores $w_{i,j}$ en los tiempos siguientes,

\begin{align}
t_1:  \ (w_{1,1}, w_{2,1}, &\ldots, w_{M-1,1}) \\
t_2:  \ (w_{1,2}, w_{2,2}, &\ldots, w_{M-1,2}) \\
 & \cdots \\
t_N:  \ (w_{1,N}, w_{2,N}, &\ldots, w_{M-1,N}).
\end{align}

Por medio de la ecuación discretizada anteriormente, es posible obtener un método iterativo que permite conocer los valores aproximados $w_{i,j}$, secuencialmente sobre cada instante siguiente:

$$
\begin{pmatrix}
w_{1,j+1} \\
\\
\vdots \\
\\
w_{m,j+1}
\end{pmatrix}  =
\begin{pmatrix}
1-2\sigma & \sigma & 0 & \cdots & 0 \\
\sigma & 1-2\sigma & \sigma & \ddots & \vdots \\
0 & \sigma & 1-2\sigma & \ddots & 0 \\
\vdots & \ddots & \ddots & \ddots & \sigma \\
0 & \cdots & 0 & \sigma & 1-2\sigma
\end{pmatrix}
\begin{pmatrix}
w{1,j} \\
\\
\vdots \\
\\
w_{m,j}
\end{pmatrix}
+
\sigma \begin{pmatrix}
w_{0,j} \\
0 \\
\vdots \\
0 \\
w_{m+1,j}
\end{pmatrix} \ \ \  \text{con} \ \sigma = \frac{Dk}{h^2}
$$

Para ver más detalles, consultar texto guía (*Numerical Analysis, Timothy Sauer, 2nd edition*). Tal método se encuentra implementado en `heat_equation_forward_differences()`.

## Diferencias Finitas (Backward Difference)

Una de las desventajas del método anterior, es que es **condicionalmente estable**, esto es, que converge únicamente bajo ciertas condiciones sobre el tamaño de discretización de la malla: $\sigma = \frac{Dk}{h^2} < \frac{1}{2}$.

Un enfoque distinto y que intenta sobrellevar este problema, es utilizar **backward differences** para la aproximación de derivadas (tal como se hizo en ODEs). Consideremos ahora, que la derivada temporal en la ecuación de calor, es aproximada con el tiempo anterior en vez del tiempo siguiente:

$$
u_t(x,t) \approx \frac{1}{k} \left( u(x,t) - u(x,t-k) \right),
$$
dejando la segunda derivada espacial intacta. Luego se obtiene una ecuación discretizada como se muestra a continuación:

$$
\frac{1}{k}\left( w_{i,j} - w_{i,j-1} \right) = \frac{D}{h^2} \left( w_{i+1,j} - 2 w_{i,j} + w_{i-1,j} \right).
$$

Evaluando esta ecuación y organizando correctamente, se obtiene el método iterativo siguiente:

$$
\begin{pmatrix}
1+2\sigma & -\sigma & 0 & \cdots & 0 \\
-\sigma & 1+2\sigma & -\sigma & \ddots & \vdots \\
0 & -\sigma & 1+2\sigma & \ddots & 0 \\
\vdots & \ddots & \ddots & \ddots & \sigma \\
0 & \cdots & 0 & -\sigma & 1+2\sigma
\end{pmatrix}
\begin{pmatrix}
w_{1,j} \\
\\
\vdots \\
\\
w_{m,j}
\end{pmatrix}
=
\begin{pmatrix}
w_{1,j-1} \\
\\
\vdots \\
\\
w_{m,j-1}
\end{pmatrix}
+
\sigma
\begin{pmatrix}
w_{0,j} \\
0 \\
\vdots \\
0 \\
w_{m+1,j}
\end{pmatrix}
$$

Para más detalles consultar texto guía (*Numerical Analysis, Timothy Sauer, 2nd edition*). Puede ver la implementación de este método en `heat_equation_backward_differences()`.

A diferencia del anterior, este no impone ninguna restricción para su convergencia!, esto es, es ** incondicionalmente estable **.

## Ecuación de calor 1D: Condición de borde de Dirichlet

Consideremos la siguiente EDP parabólica, para $ x \in [a,b]$ y $t \in [0,T_{max}]$:
\begin{align*}
u_t(x,t) = D\, u_{xx}(x,t)
\end{align*}
\begin{align*}
u(x,0) = f(x)
\end{align*}
\begin{align*}
u(a,t) = l(t)
\end{align*}
\begin{align*}
u(b,t) = r(t)
\end{align*}

```python
def heat_equation_forward_differences(P, Nx, Nt, bc='boundary'):
    """
    Solves the heat equation using forward differences
    """
    if bc=='dirichlet':
        x = np.linspace(P["xmin"], P["xmax"], Nx)
        t = np.linspace(P["tmin"], P["tmax"], Nt)
        dx = x[1]-x[0]
        dt = t[1]-t[0]
        S = P["D"]*dt/dx**2
        print("CFL condition: D*dt/dx^2 = %.1f <= 0.5 ?",S)
        # Storage
        u = np.zeros((Nx, Nt))
        # Time Loop
        for i, ti in enumerate(t):
            if i==0:
                u[:,0] = P["f"](x)
            else:
                u[ 0,i] = P["l"](ti)
                u[-1,i] = P["r"](ti)
                u[1:-1, i] = S*u[:-2, i-1] + (1-2*S)*u[1:-1,i-1]+S*u[2:,i-1]
        return x, t, u

    elif bc=='periodic':
        x = np.linspace(P["xmin"], P["xmax"], Nx)
        t = np.linspace(P["tmin"], P["tmax"], Nt)
        dx = x[1]-x[0]
        dt = t[1]-t[0]
        S = P["D"]*dt/dx**2
        print("CFL condition: D*dt/dx^2 = ",S, " <= 0.5 ?")
        # Storage
        u = np.zeros((Nx, Nt))
        # Time Loop
        for i, ti in enumerate(t):
            if i==0:
                u[:,0] = P["f"](x)
            else:
                u[0, i] = S*u[-2, i-1] + (1-2*S)*u[0,i-1]+S*u[1,i-1]
                u[1:-1, i] = S*u[:-2, i-1] + (1-2*S)*u[1:-1,i-1]+S*u[2:,i-1]
                u[-1, i] = S*u[-2, i-1] + (1-2*S)*u[-1,i-1]+S*u[1,i-1]
        return x, t, u
```

```python
def heat_equation_backward_differences(P, Nx, Nt, bc='dirichlet'):
    """
    Solves the heat equation using forward differences
    """
    if bc=='dirichlet':
        x = np.linspace(P["xmin"], P["xmax"], Nx)
        t = np.linspace(P["tmin"], P["tmax"], Nt)
        dx = x[1]-x[0]
        dt = t[1]-t[0]
        S = P["D"]*dt/dx**2
        print("CFL condition not required: D*dt/dx^2 = ",S)
        # Constructing the matrix
        A = np.diag(-S*np.ones(Nx-3),-1) + np.diag((1+2*S)*np.ones(Nx-2),0) + np.diag(-S*np.ones(Nx-3),+1)
        # Storage
        u = np.zeros((Nx, Nt))
        # Time Loop
        for i, ti in enumerate(t):
            if i==0:
                u[:,0] = P["f"](x)
            else:
                u[ 0,i] = P["l"](ti)
                u[-1,i] = P["r"](ti)
                b = u[1:-1,i-1].copy()
                b[ 0] += S*u[ 0,i]
                b[-1] += S*u[-1,i]
                u[1:-1, i] = np.linalg.solve(A, b)
        return x, t, u
```

### Experimentación

```python
f1 = lambda x: np.sin(2*np.pi*x)**2
l1 = lambda t: t
r1 = lambda t: t
P1 = {"xmin":0, "xmax":1, "tmin":0, "tmax":1.0, "D":1, "f":f1, "l":l1, "r":r1}

f2 = lambda x: 10*np.exp(-(x-.5)**2/0.01)
l2 = lambda t: 0
r2 = lambda t: 0
P2 = {"xmin":0, "xmax":1, "tmin":0, "tmax":1.0, "D":1, "f":f2, "l":l2, "r":r2}

f3 = lambda x: x
l3 = lambda t: 0
r3 = lambda t: 0
P3 = {"xmin":0, "xmax":1, "tmin":0, "tmax":1.0, "D":1, "f":f3, "l":l3, "r":r3}
```

```python
P = P1
#x, t, u = heat_equation_forward_differences(P, 10, 10, bc='dirichlet') # Unstable
#x, t, u = heat_equation_forward_differences(P, 10, 200, bc='dirichlet') # Stable
#x, t, u = heat_equation_forward_differences(P, 100, 20000,bc='dirichlet') # Stable
#x, t, u = heat_equation_backward_differences(P, 10, 10, bc='dirichlet') # Stable
#x, t, u = heat_equation_backward_differences(P, 10, 100, bc='dirichlet') # Stable
x, t, u = heat_equation_backward_differences(P, 100, 100, bc='dirichlet') # Stable

show(x,t,u)
```

```python
fig = plt.figure()
ax = fig.gca(projection='3d')
X,T = np.meshgrid(x,t)
surf = ax.plot_surface(X, T, u, rstride=1, cstride=1, cmap=cm.coolwarm, linewidth=0, antialiased=False)
#ax.set_zlim(-1.01, 1.01)
ax.zaxis.set_major_locator(LinearLocator(10))
ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
```

```python
fig = plt.figure()
ax = fig.gca(projection='3d')
X = np.arange(-5, 5, 0.25)
Y = np.arange(-5, 5, 0.25)
X, Y = np.meshgrid(X, Y)
R = np.sqrt(X**2 + Y**2)
Z = np.sin(R)
surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.coolwarm, linewidth=0, antialiased=False)
ax.set_zlim(-1.01, 1.01)
ax.zaxis.set_major_locator(LinearLocator(10))
ax.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
```

## Ecuación de calor 1D: Condición de borde Periódica

Consideremos la siguiente EDP parabólica, para $ x \in [a,b]\ \ $ y $t \in [0,T_{max}] \ \ $:
\begin{align*}
u_t(x,t) = D\, u_{xx}(x,t)
\end{align*}
\begin{align*}
u(x,0) = f(x)
\end{align*}
\begin{align*}
u(-x,t) = u(1-x,t)
\end{align*}

La condición de frontera es periódica, esto es, estudiamos la evolución de la temperatura en un anillo,
por lo que consideramos que para $u(-x,t) = u(1-x,t)$

```python
f1 = lambda x: np.sin(2*np.pi*x)**2
P1 = {"xmin":0, "xmax":1, "tmin":0, "tmax":1.0, "D":1, "f":f1}

f2 = lambda x: 10*np.exp(-(x-.5)**2/0.01)
P2 = {"xmin":0, "xmax":1, "tmin":0, "tmax":1.0, "D":1, "f":f2}

f3 = lambda x: x
P3 = {"xmin":0, "xmax":1, "tmin":0, "tmax":1.0, "D":1, "f":f3}
```

```python
P = P3
#x, t, u = heat_equation_forward_differences(P, 10, 10, bc='periodic') # Unstable
#x, t, u = heat_equation_forward_differences(P, 10, 200, bc='periodic') # Stable
x, t, u = heat_equation_forward_differences(P, 100, 20000, bc='periodic') # Stable
show(x,t,u)
```

## Ecuación de calor 2D

De forma equivalente, la ecuación de calor puede ser planteada para situaciones n-dimensionales. En partícular para en caso 2D puede ser escrita como se muestra a continuación

$$
\frac{\partial u}{\partial t} = D \ \Delta u = D \ \left(\frac{\partial^2 u}{\partial x^2} + \frac{\partial^2 u}{\partial y^2} \right)
$$

con sus respectivas condiciones de borde e iniciales. Las funciones que se muestran a continuación (`evolve()` y `solver()`) se encargan de la resolución *explícita* e *iterativa* del la ecuación 2D.

```python
# This is the 2D version of Explicit Finite Differences
def evolve(u_new, u_old, dt, D, dx2, dy2):
    u_new[1:-1, 1:-1]  = u_old[1:-1, 1:-1]
    u_new[1:-1, 1:-1] += D * dt * (u_old[2:, 1:-1] - 2*u_old[1:-1, 1:-1] + u_old[:-2, 1:-1]) / dx2
    u_new[1:-1, 1:-1] += D * dt * (u_old[1:-1, 2:] - 2*u_old[1:-1, 1:-1] + u_old[1:-1, :-2]) / dy2
    return u_new

####################################################
# MAIN 2D-PARABOLIC FUNCTION SOLVER
####################################################
def solver(u0, D=1., Nx=100, Ny=100, num_steps=10000):
    dx = 1./Nx
    dy = 1./Ny
    dx2 = dx**2
    dy2 = dy**2

    # For stability, this is the largest interval possible
    # for the size of the time-step:
    dt = 1.0*dx2*dy2 / ( 2*D*(dx2+dy2) )

    u_aux = u0
    u = np.zeros([Nx,Ny])
    all_sims = np.zeros([num_steps, Nx, Ny])

    # Iterative step
    for n in range(num_steps):
        evolve(u, u_aux, D, dt, dx2, dy2)
        u_aux = u
        all_sims[n,:,:] = u

    return all_sims
```

A continuación se genera como condición inicial, un distribución de *temperatura* con una distribución en forma de anillo en un dominio $[0,1]\times [0,1]$. **Pruebe variando los parámetros!**.

```python
####################################################
# Initial conditions generation: Ring distribution
####################################################
Ri = 0.05 # Internal Radii
Re = 0.10 # External Radii
Nx = 100
Ny = 100
dx = 1./Nx # Interval size in x-direction.
dy = 1./Ny # Interval size in y-direction.
D = 1.   # Diffusion constant.

u0 = np.zeros([Nx,Ny])

# Now, set the initial conditions (ui).
for i in range(Nx):
    for j in range(Ny):
        v=np.array([i*dx-0.5,j*dy-0.5])
        C1 = np.linalg.norm(v,1) <= np.sqrt(Re)
        C2 = np.linalg.norm(v,1) >= np.sqrt(Ri)
        if C1 and C2 : u0[i,j] = 1
```

```python
####################################################
# ANIMATION
####################################################
all_sims = solver(u0)
animate(all_sims, interval=50)
```

```python
########################################################
# Initial conditions generation: Write your own IC here
########################################################
x,y = np.meshgrid(np.linspace(0.,1.,Nx), np.linspace(0.,1.,Ny), sparse=True)

# initial condition
f0 = lambda x,y : 10*np.sin(8*np.pi*x)*np.sin(8*np.pi*y)
u0 = f0(x,y)
```

```python
####################################################
# ANIMATION
####################################################
all_sims = solver(u0)
animate(all_sims, interval=50)
```

# Acknowledgements
* _Material creado por profesor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _y ayudantes: Alvaro Salinas y Martín Villanueva. DI UTFSM. Noviembre 2016._

***
### DISCLAIMER ###

El presente notebook ha sido creado para el curso **ILI286 - Computación Científica 2**, del [Departamento de Informática](http://www.inf.utfsm.cl/), [Universidad Técnica Federico Santa María](http://www.utfsm.cl/).

El material ha sido creado por Claudio Torres <ctorres@inf.utfsm.cl> y Sebastian Flores <sebastian.flores@usm.cl>, y es distribuido sin restricciones. En caso de encontrar un error, por favor no dude en contactarnos.

[Update 2016] (Martín) Notebook creado como una fusión de las tres partes anteriores. Agregado contexto / Marco teórico. Agregadas/Arregladas animaciones de los experimentos 1D. Se agregó la simulación y animación en 2D de distribución de temperatura en forma de anillo.
***