#  Review of Basics of Linear Algebra
---
**Agenda**
>1. Matrix Vector Operations using NumPy
>1. Vector Spaces and Matrices: Four fundamental fubspaces
>1. Motivating Examples: Image and text manipulations
>1. Eigen-decomposition, determinant and trace
>1. Special Matrices: Orthogonal Matrices
>1. Norms
>

```python
# Following lines are for Python 2.x to 3.xx compatibility
from __future__ import print_function
from __future__ import division
```

```python
#IMPORT
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
%matplotlib inline

## Set a seed for the random number generator
np.random.seed(100)
```

## NumPy Arrays: Vectors, Matrices and Tensors

### Creating some special vectors and matrices
***
>-  Fixed ones: having a set of given elements
>-  Random ones
>-  Reshaping vectors to get matrices
>-  Zero matrix
>-  One matrix
>-  Identity matrix
>-  Permutation matrix

```python
## Create a vectors of  length 4
v1 =  np.array([1, 2, 3, 4])
v2 = np.array([3, 2,1,-1])

print("v1:", v1)
print ("v2:", v2)
```

    v1: [1 2 3 4]
    v2: [ 3  2  1 -1]

```python
# Create a random vector of Integers
v3 = np.random.randint(0, high=15, size=(4,))
print ("v3: ",v3)
```

    v3:  [8 8 3 7]

```python
# Create a random COLUMN vector of Integers
v31 = np.random.randint(0, high=8, size=(4,1))
print ("v31: \n",v31)
```

    v31:
     [[7]
     [7]
     [0]
     [2]]

```python
v4 = np.arange(5,9)
print ("v4: ",v4)
```

    v4:  [5 6 7 8]

```python
## Create a matrix of order 4x3
A = np.array([[1, 2, 3],
              [2, 1, 4],
              [2, 4, 7],
              [1, 2, 3]], dtype=float)

## Following is no longer recommeded
matA = np.matrix([[1, 2, 3],
              [2, 1, 4],
              [2, 4, 7],
              [1, 2, 3]])

matA2 = np.matrix("1, 2; 3, 4")

## Create a random matrix of order 4x3 whose elements are chosen uniformly randomly

## CHANGES NEEDED WITH Python 3.x for compatibilty

B = np.random.rand(4,3)

## Create a matrix of order 4x3 made of all zeros
zero_43 = np.zeros((4,3), dtype=float)

## Create a matrix of all ones of order 3x5
ones_35 = np.ones((3,5), dtype=float)

## Create an identity matrix of order 4x4
# eye_4 = np.identity(4)
eye_4 = np.eye(4) # This is more general. See the documentation.
```

```python
print("The random matrix B is:\n", B)

print ("\n The identity matrix of order 4: ",eye_4)
```

    The random matrix B is:
     [[ 0.81168315  0.17194101  0.81622475]
     [ 0.27407375  0.43170418  0.94002982]
     [ 0.81764938  0.33611195  0.17541045]
     [ 0.37283205  0.00568851  0.25242635]]

     The identity matrix of order 4:  [[ 1.  0.  0.  0.]
     [ 0.  1.  0.  0.]
     [ 0.  0.  1.  0.]
     [ 0.  0.  0.  1.]]

```python
## Create a vector of order 12 (such as [3,5,7,...])and
## Rearrange its elements to create a matrix of order 4-by-3.
v12 = np.arange(3,26,2)

A43 = v12.reshape(4,3) # or v12.reshape(4,-1)
print( "An array v12=",v12, "\n Reshaped into a matrix:\n", A43)
```

    An array v12= [ 3  5  7  9 11 13 15 17 19 21 23 25]
     Reshaped into a matrix:
     [[ 3  5  7]
     [ 9 11 13]
     [15 17 19]
     [21 23 25]]

```python
## Create a vector of order 12 (such as [3,5,7,...])and
## Rearrange its elements to create a matrix of order 4-by-3.
v12 = np.arange(3,42,2)

A43 = v12.reshape(4,-1) # or v12.reshape(4,-1)
print( "An array v12=",v12, "\n Reshaped into a matrix:\n", A43)
```

    An array v12= [ 3  5  7  9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41]
     Reshaped into a matrix:
     [[ 3  5  7  9 11]
     [13 15 17 19 21]
     [23 25 27 29 31]
     [33 35 37 39 41]]

```python
v12.shape
```

    (20,)

```python

```

```python

```

```python

```

### Basic array operations
---
Revise the following operation on matrices. Study the properties.
>- Transpose of a matrix
>- Addition of two matrices
>- Elementwise product of two matrices and other elementwise operations
>- Multiplication of two matrices (dot product)
>- Finding Submatrices
>- Broadcasting in NumPy

```python
## Add the two vectors above
v3 = v1 + v2
## np.add(v1,v2)

## Multiply the two vectors (element-wise)
v4=v1*v2

#Dot product
dotp = np.dot(v1,v2)

print("The sum of the vectors", v1,"+", v2, "=",v3)

print("The elementwise product of the vectors is:", v4)

print("The dot product of the vectors is A NUMBER: ", dotp)

```

    The sum of the vectors [1 2 3 4] + [ 3  2  1 -1] = [4 4 4 3]
    The elementwise product of the vectors is: [ 3  4  3 -4]
    The dot product of the vectors is A NUMBER:  6

```python
## Adding two matrices
A_plus_B = A + B  #np.add(A,B)
print ("A:\n", A)
print ("\n B:\n", B)
print ("\n The sum is: \n", A_plus_B)

## Can you multiply the two matrices, A and B? How?
multAB = A*B
print ("\n The element wise product is: \n", multAB)

```

    A:
     [[ 1.  2.  3.]
     [ 2.  1.  4.]
     [ 2.  4.  7.]
     [ 1.  2.  3.]]

     B:
     [[ 0.81168315  0.17194101  0.81622475]
     [ 0.27407375  0.43170418  0.94002982]
     [ 0.81764938  0.33611195  0.17541045]
     [ 0.37283205  0.00568851  0.25242635]]

     The sum is:
     [[ 1.81168315  2.17194101  3.81622475]
     [ 2.27407375  1.43170418  4.94002982]
     [ 2.81764938  4.33611195  7.17541045]
     [ 1.37283205  2.00568851  3.25242635]]

     The element wise product is:
     [[ 0.81168315  0.34388203  2.44867425]
     [ 0.54814749  0.43170418  3.76011928]
     [ 1.63529876  1.3444478   1.22787318]
     [ 0.37283205  0.01137701  0.75727906]]

```python
print ('A: \n',A)

print ('Transpose of A, A.T \n ', A.T)

# Let us create a matrix
C = np.random.randint(0, high=2, size=(3,5))
print ("\n Random matrix C:\n",C)

## MULTIPLICATION (4,3) and (3,5): Find the transpose of B and Multiply to A in appropriate order
AtimesC = np.dot(A, C)

print ('\n Product of A and C in the sense of dot product : \n', AtimesC)

```

    A:
     [[ 1.  2.  3.]
     [ 2.  1.  4.]
     [ 2.  4.  7.]
     [ 1.  2.  3.]]
    Transpose of A, A.T
      [[ 1.  2.  2.  1.]
     [ 2.  1.  4.  2.]
     [ 3.  4.  7.  3.]]

     Random matrix C:
     [[0 1 1 1 0]
     [0 0 0 0 0]
     [1 0 1 0 1]]

     Product of A and C in the sense of dot product :
     [[ 3.  1.  4.  1.  3.]
     [ 4.  2.  6.  2.  4.]
     [ 7.  2.  9.  2.  7.]
     [ 3.  1.  4.  1.  3.]]

```python
# This has been depricated. No longer advised to be used.
mat1 = np.matrix('1 0; 0 2')
mat2 = np.matrix('1 2 3; -1 0 1')

print ("Product of two matrices of data type matrix:\n",mat1*mat2) # no need of np.dot(mat1, mat2)

```

    Product of two matrices of data type matrix:
     [[ 1  2  3]
     [-2  0  2]]

### Submatrices by slicing

```python
# Random Example Matrix
S = np.random.rand(4,4)

print("S:\n",S)

print ("S[:,[1,3]]")
print (S[:,[1,3]])

print ("S[[1,3],:]")
print (S[[1,3],:])

E = S[:,[1,3]][0:2] # Submatrix by slicing

print ('A submatrix is E: ----------------------------')
print (E)

print ("S[:,[1,2]][0:2]")
print (S[:,[1,2]][0:2])
```

    S:
     [[0.00607005 0.82080849 0.26320965 0.50106643]
     [0.71096644 0.02219788 0.53582866 0.24812327]
     [0.55302567 0.81867102 0.04103166 0.56728301]
     [0.2774666  0.75213614 0.62362505 0.81281704]]
    S[:,[1,3]]
    [[0.82080849 0.50106643]
     [0.02219788 0.24812327]
     [0.81867102 0.56728301]
     [0.75213614 0.81281704]]
    S[[1,3],:]
    [[0.71096644 0.02219788 0.53582866 0.24812327]
     [0.2774666  0.75213614 0.62362505 0.81281704]]
    A submatrix is E: ----------------------------
    [[0.82080849 0.50106643]
     [0.02219788 0.24812327]]
    S[:,[1,2]][0:2]
    [[0.82080849 0.26320965]
     [0.02219788 0.53582866]]

### Broadcasting

Can you add the following?
$$
A = \left( \begin{array}{ccc} 1 & 1 & 2 \\
1 & -1 & 3
\end{array}
\right) +
\left( \begin{array}{cc} 5 \\
-5
\end{array}
\right)
$$

OR

$$
B = \left( \begin{array}{ccc} 1 & 1 & 2 \\
1 & -1 & 3
\end{array}
\right) +
\left( \begin{array}{ccc} 1 & 1 &1\\
\end{array}
\right)
$$

```python
A = np.array([[1, 1, 2],[1,-1,3]]) + np.array([[5],[-5]])
B = np.array([[1, 1, 2],[1,-1,3]]) + np.array([1,1,1])
print("A is\n",A)
print("B is\n",B)
```

    A is
     [[ 6  6  7]
     [-4 -6 -2]]
    B is
     [[2 2 3]
     [2 0 4]]

## Vector Spaces

### Independence, Orthogonality and Subspaces
---

 What does it mean for a set of vectors $\{ v_1, v_2, \cdots, v_r\}$ to be linearly independent?

$$ \alpha_1 v_1 + \alpha_2 v_2 + \cdots + \alpha_r v_r \implies \alpha_i = 0,\  i=1,\cdots,r.$$

In other words these vectors can not be combined to form a zero vector, or, one of these vectors can not be given as a linear combination of the remaining.

 What conditions two vectors that are perpendicular (orthogonal) to each other satisfy?

$$ u \perp v \iff \langle u, v\rangle = v^Tu = 0$$

 Provide the conditions so that two subspaces of a vector space are mutually orthogonal.

$$ U \perp V \iff \langle u, v\rangle = v^Tu = 0,\ \forall (u \in U, v \in V).$$

---

## The Four Fundamental Subspaces
---

The matrix $A \in \mathbb{R}^{m \times n}$ represents a linear  transformation,  $T_A(x): \mathbb{R}^n \to \mathbb{R}^m$ given by $T_A(x) = A x$.

 We can define the following four space with respect to matrix $A$.

  >1. Row space of $A$ is a subspace of $\mathbb{R}^n$:
    $$ \mathcal{R}(A) =\mathcal{C}(A^T) = \left\{ x \in \mathbb{R}^n : x = A^T y \textrm{ for some }y \in \mathbb{R}^n \right\}.$$
  >1. Null space of $A$ is a subspace of $\mathbb{R}^n$:
    $$ \mathcal{N}(A) = \left\{ x \in \mathbb{R}^n : Ax = 0 \right\}.$$
 >1. Column space of $A$ is a subspace of $\mathbb{R}^m$:
    $$ \mathcal{C}(A) = \left\{ y \in \mathbb{R}^m : y = Ax \textrm{ for some }x \in \mathbb{R}^n \right\}.$$
 >1. Left null space of $A$, or null space of $A^T$, is a subspace of $\mathbb{R}^m$:
    $$ \mathcal{N}(A^T) = \left\{ y \in \mathbb{R}^m : A^T y = 0 \right\}.$$

The above four subspaces satisfy the following properties:

 >1. $ \mathcal{R}(A) \perp \mathcal{N}(A) $, or $ \mathcal{C}(A^T) \perp \mathcal{N}(A) $.
>1. $ \mathcal{C}(A) \perp \mathcal{N}(A^T) $.
>1. $ \mathcal{R}(A) \oplus \mathcal{N}(A) =\mathbb{R}^n $.
>1. $ \mathcal{C}(A) \oplus \mathcal{N}(A^T) =\mathbb{R}^m $.

Here $\perp $ stands for perpendicular, and  $\oplus$ means that every element $x \in \mathbb{R}^n$ could be written  as $x = x_n+x_r$, where $x_n \in \mathcal{N}(A)$ and $x_r \in \mathcal{R}(A)$.

 If rank$(A)$ = r, then

 >- dim $\mathcal{R}(A) = $ dim $\mathcal{C}(A) = r$,

 >- dim $\mathcal{N}(A)=n-r$,

 >- and dim$\mathcal{N}(A^T)=m-r$.

**Group-work**: Consider matrix A and its echelon form to answer the following
$$
A = \begin{bmatrix}
    1 & 2 & 0 & 1\\
    0 & 1 & 1 & 0\\
    1 & 2 & 0 & 1
\end{bmatrix}
\Rightarrow U = \begin{bmatrix}
    1 & 2 & 0 & 1\\
    0 & 1 & 1 & 0\\
    0 & 0 & 0 & 0
\end{bmatrix}
$$
>1. Find all its four subspaces as spans of vectors.

### Motivating Example: An image as a tensor, a matrix, and a vector
***
Do the following
 1. Read an image from your computer (I downloaded a 'Baby Yoda')
 1. Verify the datatype and check the dimensions of the data.
 1. Convert the image to a monochrome image (a matrix) by using weighted addition of the RGB values.
 1. Put a frame around the image.
 1. Add some Gaussian-noise to the image.

```python
pix = mpimg.imread("./images/BabyYodaDoll.jpg") # OR BabyYodaDoll.jpg, BabyYodaXmas

plt.axis('off')
plt.imshow(pix)
plt.show()
```

```python
type(pix)
```

    numpy.ndarray

```python
dim = pix.shape
print ("Original order of the image tensor:", dim)

X = pix.reshape(-1,3)/255.0
print("After vectorizing the image, the dimensions are:", X.shape )

```

    Original order of the image tensor: (368, 700, 3)
    After vectorizing the image, the dimensions are: (257600, 3)

#### Conversion to a black & white image

[Check this out](https://www.prasannakumarr.in/journal/color-to-grayscale-python-image-processing
)

```python
## Converting a color image to black-n-white
### Read this blog: https://www.prasannakumarr.in/journal/color-to-grayscale-python-image-processing
color_weight = [0.2125, 0.7154, 0.0721]; # LUMA-REC.709
#color_weight = [1, 0, 0]
pix_gray = np.dot(pix[..., 0:3], color_weight) # Note conversion to gray-scale is not unique
print ("Order of the gray-scale image matrix:", pix_gray.shape)
plt.axis('off')
plt.imshow(pix_gray, cmap='gray')
plt.show()
```

#### Submatrices by Slicing
***

```python
# Create a frame around the image
pix_framed = np.zeros(np.add(pix_gray.shape, tuple([20,20])))
print ("Order of the framed gray-scale image matrix:", pix_framed.shape)
pix_framed[10:pix_gray.shape[0]+10, 10:pix_gray.shape[1]+10] = pix_gray
plt.axis('off')
plt.imshow(pix_framed, cmap='gray')
plt.show()
```

#### Adding Gaussian noise to a black-&-white image.

```python
#classwork: Make the frame lighter and wider.
# Add gaussian noise to the image
#size = pix_gray.shape
gauss_noise =  np.random.normal(0.0,10.0, (368,700))
noisy_pix= pix_gray + gauss_noise
plt.axis('off')
plt.imshow(pix)
#plt.imshow(gauss_noise.reshape(368,700))
plt.imshow(noisy_pix, cmap='gray')
plt.show()
```

#### Normal (Gaussian) Distribution
$$\large
p(x) = \frac{1}{\sqrt{ 2 \pi \sigma^2 }}
                 e^{ - \frac{ (x - \mu)^2 } {2 \sigma^2} },
                 $$
                 where $\mu$ is the mean and $\sigma$ is the standard deviation of the  distribution.

### Motivating Example: Term-document Matrix
---
AIM: Load some text data in a term-document  matrix.
You can try removing the stop-words.

```python
from sklearn.feature_extraction.text import CountVectorizer,ENGLISH_STOP_WORDS
```

```python
vectorizer = CountVectorizer()
```

```python
document1 = " I bought this game as a gift for my 8 year old daughter who loves games. I was expecting lots of gross foods--but I was surprised at the inappropriate cards--eyeball, human burger, blood salsa, and fresh brains. Those are not foods that typical people find in their refrigerators. We do not practice cannibalism. She was very upset when I suggested that we just take out those cards. I seriously wonder who thinks that those cards are appropriate for kids. The rest of the game is funny, but I wish I would have looked through the cards before I gave it to her."
document2 = "Absolutely love Taco vs burrito. I bought it as a kickstarter. I originally bought this game because my husband and I love to play games with friends but most of them are not targeted to children so I got this to add to our collection so we had options when our friends with kids came. I’m not gonna lie I did No have high expectations for this to be a game for adult but I was Sooooo wrong!!!!!! We have now played with several different groups of friends and it’s a hit!!!!! With adults it becomes a major strategy game. I have Now bought it as a Christmas present bc it was so well received!!!!"
document3 = " Unlike several of the reviewers here, I didn't purchase this originally for when kids are around. I bought it because of the reviews that said the adults all loved it too! I'm always on the lookout for games playable by 2 people and this was a great one. It's incredibly simple, but brings a lot of laughs with the competition and sabotage. I'm really glad I gave this game a chance."
doc_list = [document1, document2, document3]
```

```python
# Fit a bag of words
bow = vectorizer.fit_transform(doc_list)
print(type(bow))
print ("Feature (terms) Names: \n",vectorizer.get_feature_names())
```

    <class 'scipy.sparse.csr.csr_matrix'>
    Feature (terms) Names:
     ['absolutely', 'add', 'adult', 'adults', 'all', 'always', 'and', 'appropriate', 'are', 'around', 'as', 'at', 'bc', 'be', 'because', 'becomes', 'before', 'blood', 'bought', 'brains', 'brings', 'burger', 'burrito', 'but', 'by', 'came', 'cannibalism', 'cards', 'chance', 'children', 'christmas', 'collection', 'competition', 'daughter', 'did', 'didn', 'different', 'do', 'expectations', 'expecting', 'eyeball', 'find', 'foods', 'for', 'fresh', 'friends', 'funny', 'game', 'games', 'gave', 'gift', 'glad', 'gonna', 'got', 'great', 'gross', 'groups', 'had', 'have', 'her', 'here', 'high', 'hit', 'human', 'husband', 'in', 'inappropriate', 'incredibly', 'is', 'it', 'just', 'kickstarter', 'kids', 'laughs', 'lie', 'looked', 'lookout', 'lot', 'lots', 'love', 'loved', 'loves', 'major', 'most', 'my', 'no', 'not', 'now', 'of', 'old', 'on', 'one', 'options', 'originally', 'our', 'out', 'people', 'play', 'playable', 'played', 'practice', 'present', 'purchase', 'really', 'received', 'refrigerators', 'rest', 'reviewers', 'reviews', 'sabotage', 'said', 'salsa', 'seriously', 'several', 'she', 'simple', 'so', 'sooooo', 'strategy', 'suggested', 'surprised', 'taco', 'take', 'targeted', 'that', 'the', 'their', 'them', 'thinks', 'this', 'those', 'through', 'to', 'too', 'typical', 'unlike', 'upset', 'very', 'vs', 'was', 'we', 'well', 'when', 'who', 'wish', 'with', 'wonder', 'would', 'wrong', 'year']

```python
# Check the matrix
print("Bag of words sparse matrix (data structure CSR-compressed sparse row):\n",bow, "\n To an array: \n", bow.toarray())
```

    Bag of words sparse matrix (data structure CSR-compressed sparse row):
       (0, 18)	1
      (0, 129)	1
      (0, 47)	2
      (0, 10)	1
      (0, 50)	1
      (0, 43)	2
      (0, 84)	1
      (0, 149)	1
      (0, 89)	1
      (0, 33)	1
      (0, 143)	2
      (0, 81)	1
      (0, 48)	1
      (0, 139)	3
      (0, 39)	1
      (0, 78)	1
      (0, 88)	2
      (0, 55)	1
      (0, 42)	2
      (0, 23)	2
      (0, 120)	1
      (0, 11)	1
      (0, 125)	4
      (0, 66)	1
      (0, 27)	4
      :	:
      (2, 35)	1
      (2, 102)	1
      (2, 9)	1
      (2, 108)	1
      (2, 110)	1
      (2, 4)	1
      (2, 80)	1
      (2, 133)	1
      (2, 5)	1
      (2, 90)	1
      (2, 76)	1
      (2, 98)	1
      (2, 24)	1
      (2, 54)	1
      (2, 91)	1
      (2, 67)	1
      (2, 115)	1
      (2, 20)	1
      (2, 77)	1
      (2, 73)	1
      (2, 32)	1
      (2, 109)	1
      (2, 103)	1
      (2, 51)	1
      (2, 28)	1
     To an array:
     [[0 0 0 0 0 0 1 1 2 0 1 1 0 0 0 0 1 1 1 1 0 1 0 2 0 0 1 4 0 0 0 0 0 1 0 0
      0 1 0 1 1 1 2 2 1 0 1 2 1 1 1 0 0 0 0 1 0 0 1 1 0 0 0 1 0 1 1 0 1 1 1 0
      1 0 0 1 0 0 1 0 0 1 0 0 1 0 2 0 2 1 0 0 0 0 0 1 1 0 0 0 1 0 0 0 0 1 1 0
      0 0 0 1 1 0 1 0 0 0 0 1 1 0 1 0 3 4 1 0 1 1 3 1 1 0 1 0 1 1 0 3 2 0 1 2
      1 0 1 1 0 1]
     [1 1 1 1 0 0 2 0 1 0 2 0 1 1 1 1 0 0 3 0 0 0 1 2 0 1 0 0 0 1 1 1 0 0 1 0
      1 0 1 0 0 0 0 2 0 3 0 3 1 0 0 0 1 1 0 0 1 1 3 0 0 1 1 0 1 0 0 0 0 5 0 1
      1 0 1 0 0 0 0 2 0 0 1 1 1 1 2 2 2 0 0 0 1 1 2 0 0 1 0 1 0 1 0 0 1 0 0 0
      0 0 0 0 0 1 0 0 3 1 1 0 0 1 0 1 0 0 0 1 0 3 0 0 5 0 0 0 0 0 1 2 2 1 1 0
      0 4 0 0 1 0]
     [0 0 0 1 1 1 2 0 1 1 0 0 0 0 1 0 0 0 1 0 1 0 0 1 1 0 0 0 1 0 0 0 1 0 0 1
      0 0 0 0 0 0 0 2 0 0 0 1 1 1 0 1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 3 0 0
      1 1 0 0 1 1 0 0 1 0 0 0 0 0 0 0 3 0 1 1 0 1 0 0 1 0 1 0 0 0 1 1 0 0 0 1
      1 1 1 0 0 1 0 1 0 0 0 0 0 0 0 0 1 5 0 0 0 3 0 0 0 1 0 1 0 0 0 1 0 0 1 0
      0 1 0 0 0 0]]

```python
print ("Data Type of variable bow:",type(bow))
# Find the index of specific words
print (vectorizer.vocabulary_.get("children"))
print (vectorizer.vocabulary_.get("salsa"))
print (bow.getcol(vectorizer.vocabulary_.get("great")))
```

    Data Type of variable bow: <class 'scipy.sparse.csr.csr_matrix'>
    29
    111
      (2, 0)	1

### Inverse and genralized inverse
***
A square matrix is said to have an inverse if all its rows or all its columns are linearly independent. Inverse of square matrix $A$ is denoted by $A^{-1}$. It has the following property:
$$ A^{-1} A = A A^{-1} = I$$
**Review Gauss-Jordan Elimination** for finding inverse of a matrix.
Ref: __[WikiPedia](https://en.wikipedia.org/wiki/Gaussian_elimination)__

```python
# Using linear algebra module for finding inverse
# Random Example Matrix
S = np.random.rand(4,4)
inv_S = np.linalg.inv(S)
print ("Inverse of S: \n",inv_S)
print ("Verify inverse:\n",np.dot(S,inv_S))

```

    Inverse of S:
     [[ 6.29223122e-02  4.65377320e-01  1.11496530e+00 -1.38531775e+00]
     [ 5.39687908e-01 -9.77003966e-01  1.14702973e+00  2.37923981e-01]
     [-6.96882874e-01  4.52175993e-01 -9.25863481e-01  1.51010216e+00]
     [ 9.12693835e-01  3.45607950e-01 -5.85756832e-01 -3.29197827e-04]]
    Verify inverse:
     [[ 1.00000000e+00 -4.16333634e-17  4.16333634e-17 -1.38777878e-17]
     [ 1.11022302e-16  1.00000000e+00 -1.11022302e-16  2.77555756e-17]
     [ 5.55111512e-17  0.00000000e+00  1.00000000e+00  0.00000000e+00]
     [ 0.00000000e+00  5.55111512e-17  0.00000000e+00  1.00000000e+00]]

### Eigenvalues and eigenvectors

An eigenvector of a square matrix $A$ is a special non-zero vector such that
$$ A v  = \lambda v$$
where $\lambda$ is called the associated eigenvalue.

**Example**: Find the eigenvalues and eigenvectors of
$
A = \left( \begin{array}{cc} 2 & 4  \\
1 & -1
\end{array}
\right)
$
### Eigen-decomposition

If the square matrix $A\in \mathbb{C}^{n\times n}$ has $n$ linearly independent eigenvectors  then $A$ can be given in the following factorized form as
$$
A = Q \Lambda Q^{-1},
$$
where $\Lambda = $ diag$(\lambda_1, \cdots, \lambda_n)$ and columns of matrix $Q$ are made of the eigenvector $q_i$ of $A$ $(i=1,\cdots,n)$, arranged in the same order as the eigenvalues in $\Lambda$.
>- When $A$ is a real and symmetric matrix: $A = Q \Lambda Q^T$, where $Q$ is orthogonal $(Q^TQ = I = QQ^T)$ and $\Lambda$ is made of real diagonal entries.
>- If a function $f(x)$ has power series expansion in $x$, then $f(A) = Q f(\Lambda) Q^{-1}$.

### Daterminant
---

Laplace's Formula for determinant of a square matrix
$$\det(A) = \sum_{j=1}^n (-1)^{i+j} a_{ij} M_{ij}\ \textrm{ for a fixed row } i,$$
$$\det(A) = \sum_{i=1}^n (-1)^{i+j} a_{ij} M_{ij} \textrm{ for a fixed column } j .$$
where minor $M_{ij}$ si the determinant of the submatrix without the $i$-th row and $j$-th column.

>- ##### Determinant of a matrix is the product of all its eigenvalues.
>- ##### $\det\left(I_n\right) = 1$, where $I_n$ is the $n\times n$  identity matrix.
>- #####  $\det\left(A^\textsf{T}\right) = \det(A)$, where $A^\textsf{T}$ denotes the transpose of $A$.
>- #####  $\det\left(A^{-1}\right) = \frac{1}{\det(A)} = [\det(A)]^{-1}.$

>- #####  For square matrices $A$ and $B$ of equal size,
 $$\det(AB) = \det(A) \det(B).$$
>- #####  $\det(cA) = c^n\det(A)$, for an $n\times n$ matrix $A$.

### Trace
---
Trace of a square matrix is the sum of its diagonal elements
$$
\operatorname{tr}(\mathbf{A}) = \sum_{i=1}^n a_{ii} = a_{11} + a_{22} + \dots + a_{nn}
$$

>- Trace is the sum of the eigenvalues of a matrix.
>- For any nonsingular matrix $\mathbf{S}$: $\operatorname{tr}(\mathbf{S} \mathbf{A} \mathbf{S}^{-1})  = \operatorname{tr}(\mathbf{A}) $ (i.e., invariance under change of base)

>- Following holds
$$
\begin{align}
  \operatorname{tr}(\mathbf{A}^T) &= \operatorname{tr}(\mathbf{A}),\\\\
  \operatorname{tr}(\mathbf{A} + \mathbf{B}) &= \operatorname{tr}(\mathbf{A}) + \operatorname{tr}(\mathbf{B}), \\\\
     \operatorname{tr}(c\mathbf{A}) &= c \operatorname{tr}(\mathbf{A}), \\\\
     \operatorname{tr}(\mathbf{A}\mathbf{B}) &= \operatorname{tr}(\mathbf{B}\mathbf{A}),\\\\
     \operatorname{tr}(\mathbf{A}\mathbf{B}) &\ne \operatorname{tr}(\mathbf{A})\operatorname{tr}(\mathbf{B})
\end{align}
$$

>- $$  \operatorname{tr}\left(\mathbf{A}^\mathsf{T}\mathbf{B}\right) =
  \operatorname{tr}\left(\mathbf{A}\mathbf{B}^\mathsf{T}\right) =
  \operatorname{tr}\left(\mathbf{B}^\mathsf{T}\mathbf{A}\right) =
  \operatorname{tr}\left(\mathbf{B}\mathbf{A}^\mathsf{T}\right) =
  \sum_{i,j}A_{ij}B_{ij}.
  $$

---
# DISREGARD THE REMAINING
---
## The Review continues in Notebook 2.

### Some Special Matrices
***
- Symmetric ans Skew-symmetric Matrics
- Upper and Lower Triangular Matrices
- Banded Matrices
- Orthogonal and Unitary Matrices
- Positive definite, positive semidefinite matrices
- Negative definite, negative semidefinite matrices
- Indefinite Matrices
- Permutation Matrix
- Diagonally Dominant Matrices
- Nonnegative Matrices

### Derivatives of Matrices of functions of some variables
***
#### Review the basics of derivatives, partial derivatives and gradients.

 Definition of derivative of a matrix of functions.
$$
\frac{d}{d \alpha} C(\alpha) = \dot{C}(\alpha) = [\dot{c}_{i,j}(\alpha)].
$$
Product Rule

$$
\frac{d}{d \alpha} [ A(\alpha)\, B(\alpha)] = \dot{A}(\alpha)\,B(\alpha) + A(\alpha)\,\dot{B}(\alpha).
$$

[See Matrix Calculus on WikiPedia](https://en.wikipedia.org/wiki/Matrix_calculus)

**Exercise: ** If $\phi(x) = \frac{1}{2} x^T A x - b^T x$, show that the gradient is given by
$$
\nabla \phi(x) = \frac{1}{2} (A^T + A)x - b.
$$
This result will be quite useful as we move along.

$$\large
\nabla_x \left( \frac{1}{2} x^T A x - b^T x \right) = \frac{1}{2} (A^T + A)x - b.
$$

### Norms
---
Please see Text-5 for a comprehensive review.

>- What is the norms of a vector?
>- $\|\cdot\|_p$ norms: Euclidean norm, 1-norms, Manhattan distance.
>- Matrix Norms, subordinate matrix norms
>- Unit circles in different norms.

[Image Source: WikiMedia](https://upload.wikimedia.org/wikipedia/commons/f/f8/L1_and_L2_balls.svg)

```python
# NORM : Euclidean, Frobenius

D1 = np.array([[1,1],[1,-1]])
print D1
print np.linalg.norm(D1)

D2 = np.array([[1,2, -1],[3,4, -6]])
print D2
print np.linalg.norm(D2)

#print np.linalg.norm(D2, ord=np.inf)
```

```python

```

```python

```

```python

```

### Derivatives of Matrices of functions of some variables
***
#### Review the basics of derivatives, partial derivatives and gradients.

 Definition of derivative of a matrix of functions.
$$
\frac{d}{d \alpha} C(\alpha) = \dot{C}(\alpha) = [\dot{c}_{i,j}(\alpha)].
$$
Product Rule

$$
\frac{d}{d \alpha} [ A(\alpha)\, B(\alpha)] = \dot{A}(\alpha)\,B(\alpha) + A(\alpha)\,\dot{B}(\alpha).
$$

[See Matrix Calculus on WikiPedia](https://en.wikipedia.org/wiki/Matrix_calculus)

**Exercise: ** If $\phi(x) = \frac{1}{2} x^T A x - b^T x$, show that the gradient is given by
$$
\nabla \phi(x) = \frac{1}{2} (A^T + A)x - b.
$$
This result will be quite useful as we move along.

$$\large
\nabla_x \left( \frac{1}{2} x^T A x - b^T x \right) = \frac{1}{2} (A^T + A)x - b.
$$

### Linear Systems of Equations
---
Example: Solve the following system by using inversion of the coefficient matrix.
> $2x+3y-z = 4$

> $-x+y+2z = 2$

> $3+2x-4z = 1$

```python
# FIND the solution using coding
A1 =  np.array([[2, 3, -1],
               [-1, 1, 2],
               [3, 2, -4]], dtype=float)
b = np.array([4, 2, 1]).reshape(3, -1)

# X = inverse(A) b
X = np.dot(np.linalg.inv(A1), b)
print X
```

```python
%%html
<style>
.eqnbox{
    margin:auto;width:500px;padding:20px;
    border: 3px solid green; border-radius:15px;margin-top:20px;margin-bottom:20px;
}

.eqnbox2{
    margin:auto;width:500px;padding:20px;
    border: 1px solid green; border-radius:15px;margin-top:20px;margin-bottom:20px;
}
</style>
```

```python

```

course_notes/MA544 Share/NB1 MA544 updated.ipynb


# Imports

```python
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle, Arrow
from matplotlib.animation import FuncAnimation
from mpl_toolkits.axes_grid.inset_locator import inset_axes
%matplotlib notebook

import re
from scipy.integrate import odeint, solve_ivp
```

# Decalaring Paths

```python
results_path = '../Notebook Results/Chapter 1/'
```

# Initializations

```python
%%html
<style>
.output_wrapper button.btn.btn-default,
.output_wrapper .ui-dialog-titlebar {
  display: none;
}
</style>
```

<style>
.output_wrapper button.btn.btn-default,
.output_wrapper .ui-dialog-titlebar {
  display: none;
}
</style>

```python
plt.rcParams['figure.dpi'] = 80
plt.rcParams['figure.figsize'] = (10,5)
```

# Pendulum Simulation

**Pendulum**

\begin{equation}
\begin{aligned}
F = ma &\Leftrightarrow m\frac{d^2 \theta}{d x^2} \\
\end{aligned}
\end{equation}

\begin{equation}
\begin{aligned}
x_{displacement} &= L\theta\\ \\ \\
\end{aligned}
\end{equation}

\begin{aligned}
m\frac{d^2 x}{d x^2} &= mgsin(\theta) - \mu\omega \\
m\frac{d^2 (L\theta)}{d x^2} &= mgsin(\theta) - \mu\omega \\
mL\frac{d^2 (\theta)}{d x^2} &= mgsin(\theta) - \mu\omega \\
\frac{d^2 (\theta)}{d x^2} &= \frac{g}{L}sin(\theta) - \frac{\mu}{mL}\omega \\
\frac{d^2 (\theta)}{d x^2} &= \frac{g}{L}sin(\theta) - \frac{\mu}{mL}\frac{d (\theta)}{d x} \\
\end{aligned}

```python
class PendulumSimulator:

    def __init__(self, theta_0 = 60.0, omega_0 = 3.9, air_resistance_factor = 0.01,
                 gravitational_force = 9.8, length = 2, timesteps = 1000, vector_skips=5, interval=50):
        self.DEGREES = np.pi/180
        self.theta_0 = theta_0*self.DEGREES
        self.omega_0 = omega_0
        self.air_resistance_factor = air_resistance_factor
        self.gravitational_force = gravitational_force
        self.length = length
        self.timesteps = timesteps
        self.simulation_time = np.linspace(0, self.timesteps, 10*(self.timesteps)+1)
        self.labels_changed = False
        self.vector_skips = vector_skips
        self.interval = interval

    @staticmethod
    def get_pendulum_deriv(t, params, g, l, a):
        _theta = params[0]
        _omega = params[1]
        d_theta = _omega
        d_omega = -(g/l)*np.sin(_theta) - (a*d_theta)
        return d_theta, d_omega

    @staticmethod
    def update_rad2deg(labels):
        _labels = [float(re.sub(r'[^\x00-\x7F]+','-', s)) for s in labels]
        _labels = [round((int(k)*(180/np.pi))/180, 1) for k in _labels]
        _labels = [r'${0}\pi$'.format(k) for k in _labels]
        return _labels

    def calculate_deriv(self):

        y0 = [self.theta_0, self.omega_0]
        res = odeint(self.get_pendulum_deriv, y0, self.simulation_time,
                     args=(self.gravitational_force, self.length,self.air_resistance_factor),
                     tfirst=True)
        self._theta, self._omega = res.T
        self._alpha = -(self.gravitational_force/self.length)*np.sin(self._theta) - (self.air_resistance_factor*self._omega)
        self.pendulum_coord = self.length*np.c_[np.cos((1.5*np.pi)+self._theta),
                                                np.sin((1.5*np.pi)+self._theta)]

    def init_plot(self):

        self.axes.set_title(r'$Dynamic Of A Pendulum$')
        self.ot_axes = inset_axes(self.axes, width="40%", height="30%", loc=4,
                                  bbox_to_anchor=(0,0.1,1,1), bbox_transform=self.axes.transAxes)
        self.ot_axes.patch.set_alpha(0.5)
        self.pen_axes = inset_axes(self.axes, width="15%", height="30%", loc=3,
                                   bbox_to_anchor=(0.055,0.1,1,1), bbox_transform=self.axes.transAxes)
        self.pen_axes.patch.set_alpha(0.5)

        # Setup Time v/s Theta Axes
        self._p1, = self.axes.plot(self.simulation_time[0], self._theta[0], marker = 'o', markersize=1.2, lw=0.1, color='w')
        self._p2, = self.ot_axes.plot(self._theta[0], self._omega[0], color='w', lw=0.8)
        self._p3  = self.ot_axes.scatter(self._theta[0], self._omega[0], facecolor='r', edgecolor='r', s=5)

        self.axes.set_facecolor('k')
        self.axes.grid()
        self.axes.set_xlabel(r'$Time(sec) \longrightarrow$')
        self.axes.set_ylabel(r'$\theta^\circ_{rad} \longrightarrow$')
        self.axes.set_ylim(self._theta.min()*1.1, self._theta.max()*1.1)

        # Setup Omega Theta Axes
        self.ot_axes.set_facecolor('#ffefd6')
        for spine in self.ot_axes.spines.values():
            spine.set_edgecolor('white')

        self.ot_axes.grid(color='w', linestyle='-.', linewidth=0.3)
        _xrange = np.arange(self._theta.min()*1.2,self._theta.max()*1.2)
        _yrange = np.arange(self._omega.min()*1.2,self._omega.max()*1.2)
        self.ot_axes.plot([0]*len(_yrange),_yrange, lw=1, c='w')
        self.ot_axes.plot(_xrange,[0]*len(_xrange), lw=1, c='w')
        self.ot_axes.tick_params(axis='both', colors='w')
        self.ot_axes.set_xlabel(r'$\theta^\circ_{rad} \longrightarrow$', color='w')
        self.ot_axes.set_ylabel(r'$\omega \longrightarrow$', color='w')

        skip = slice(None, None, self.vector_skips)
        self.ot_axes.quiver(self._theta[skip], self._omega[skip], self._omega[skip], self._alpha[skip],
                              color='#cffaac', pivot='tip', width=0.005, scale=1 / 0.015, alpha=0.5)

        # Setup Pendulum Axes
        self.pen_axes.text(0, 1.5*self.length, r'$0^\circ$', fontsize=12, color='w')
        self.pen_axes.tick_params(labelbottom = False, labeltop = False,
                                  labelleft = False, labelright = False)
        self.pen_axes.set_xlim(-1.5*self.length,1.5*self.length)
        self.pen_axes.set_ylim(-1.5*self.length,1.5*self.length)
        for spine in self.pen_axes.spines.values():
            spine.set_edgecolor('white')
        self.pen_axes.plot([0]*50, np.arange(-25,25), color='w', lw=0.8, linestyle='-.')
        self.pen_axes.plot(np.arange(-25,25), [0]*50, color='w', lw=0.4, linestyle='-.')
        self.pen_axes.scatter(0, 0, facecolor='k', edgecolor='w', s=3)
        self._p4, = self.pen_axes.plot([0,self.pendulum_coord[0][0]],
                                       [0,self.pendulum_coord[0][1]],
                                       color='#ffd77a')
        self._p5 = self.pen_axes.scatter(self.pendulum_coord[0][0],
                                         self.pendulum_coord[0][1],
                                         facecolor='brown', edgecolor='w', s=25)

        return [self._p1, self._p2, self._p3, self._p4, self._p5]

    def update_plot(self, i):
        self.axes.set_xlim(0, self.simulation_time[i])

        if self.labels_changed == False:
            actual_labels1 = [k.get_text() for k in self.axes.get_yticklabels()]
            updated_labels1 = self.update_rad2deg(actual_labels1)
            self.axes.set_yticklabels(updated_labels1)

            actual_labels2 = [k.get_text() for k in self.ot_axes.get_xticklabels()]
            updated_labels2 = self.update_rad2deg(actual_labels2)
            self.ot_axes.set_xticklabels(updated_labels2)
            self.labels_changed = True

        self._p1.set_data(self.simulation_time[:i], self._theta[:i])
        self._p2.set_data(self._theta[i-10:i], self._omega[i-10:i])
        self._p3.set_offsets(np.c_[self._theta[i-1], self._omega[i-1]])
        self._p4.set_data([0,self.pendulum_coord[i][0]],
                          [0,self.pendulum_coord[i][1]])
        self._p5.set_offsets(self.pendulum_coord[i])

        return [self._p1, self._p2, self._p3, self._p4, self._p5]

    def start_simulation(self):

        self.fig, self.axes = plt.subplots()
        self.calculate_deriv()
        self.ani=FuncAnimation(fig=self.fig, func=self.update_plot, init_func=self.init_plot,
                               interval=self.interval, blit=True, save_count = 1000)
        plt.show()

```

```python
psHandler = PendulumSimulator(theta_0 = 60.0, omega_0 = 4,
                              air_resistance_factor = 0.01, gravitational_force = 9.8,
                              length = 2, timesteps = 1000, vector_skips = 20, interval=50)
plt.rcParams['figure.figsize'] = (13,6)
psHandler.start_simulation()
```

    <IPython.core.display.Javascript object>

```python

```


# Recursion examples

adopted from: https://github.com/rasbt/algorithms_in_ipython_notebooks

https://github.com/AbhishekSinhaCoder/Data-Structures-Concepts-through-Notebook

# Examples using Recursion

**Important Note**

For most cases, using function recursion should be avoided in Python are better be implemented using for/while loops most of the time (although, I must admit that recursive solutions do look elegant). One of the reasons is that stacking recursive calls can easily blow up memory or at least result in the popular yet nasty "RuntimeError: maximum recursion depth exceeded". Also, keep in mind that Python does not optimize tail recursion in favor of having the full tracebacks for debugging; related to that, please see Guido van Rossums blog posts "[Tail Recursion Elimination](http://neopythonic.blogspot.com.au/2009/04/tail-recursion-elimination.html)" and "[Final Words on Tail Calls](http://neopythonic.blogspot.com.au/2009/04/final-words-on-tail-calls.html)." If you do like to play around with recursion more efficiently, I highly recommend taking a look at [Haskell](https://www.haskell.org) or other functional programming languages. That being said, below are some examples of recursive function implementations in Python for illustrative purposes.

## Review: Running time and big-$\mathcal{O}$: Recursive Algorithms for Computing Powers

As example of the use of linear recursion, we consider the problem of raising a number $x$ to an arbitrary nonnegative integer, $n$. That is, we wish to compute the power function, defined as $power(x,n) = xn$. (We use the name “power” for this discussion, to differentiate from the built-in Python function _pow_ that provides such functionality.) We will consider two different recursive formulations for the problem that lead to algorithms with very different performance.
A trivial recursive definition follows from the fact that $x^n = x \cdot x^{n−1}$ for $n > 0$.

\begin{equation}
power(x,n) ==\left\{
                \begin{array}{ll}
                        1 \ if \ n = 0\\
                        x \ power(x, n − 1) \ otherwise.
                \end{array}
              \right.
\end{equation}

This definition leads to a recursive algorithm shown below:

```python
# in case we did not know about Pythons x**n statement :)
```

```python
2**8
```

```python
# so the idea is to reduce the problem to a simpler instance of the same problem
## to start with we need a base case
## so base case for power would be x to 0 which is 1
```

```python
2**0
```

```python
def basicpower(x, n):
    '''Compute the value x**n for integer n.'''
    if n == 0:
        return 1 # base case without base case our recursive function will run forever
    else:
        return x  * basicpower(x, n-1) # our recursive call so we are calling the function on itself on a smaller problem space
```

# How to make a recursive function in two easy steps
* figure out the base case
* figure out how to reduce problem to a subset of the same problem

```python
basicpower(2,8)
```

A recursive call to this version of power(x,n) runs in $\mathcal{O}(n)$ time.

```python
%%timeit
basicpower(2,1000)
```

```python
%%timeit
basicpower(2,2000)
```

```python
%%timeit
basicpower(2,2500)
```

```python
# not satisfied with performance of our basic power we have an improvement in mind
```

```python

def power(x, n):
    '''Compute the value x**n for integer n.'''
    if n == 0: # this is our base case
        return 1
    else:
        partial = power(x, n // 2) # remember that x^4*x^4 == x^8
        result = partial * partial
    if n % 2 == 1:
        result *= x
    return result
```

```python
power(2,8)
```

```python
power(2,9)
```

```python
import math

```

```python
math.log2(1024)
```

```python
math.log2(1024*1024)
```

```python
math.log2(10**100) # log n of googool
```

```python
# so if we were playing the guess the number with yes and no questions we could figure out any number up to googool in 333 questions

# not bad when we consider that number of atoms in Universe is something 10**80
```

```python
%%timeit
power(2,1000)
```

```python
%%timeit
power(2,2000)
```

```python
%%timeit
power(2,2_000_000)
```

To analyze the running time of the revised algorithm, we observe that the exponent in each recursive call of function power(x,n) is at most half of the preceding exponent. As we saw with the analysis of binary search, the number of times that we can divide $n$ in half before getting to one or less is $\mathcal{O}(log n)$. Therefore, our new formulation of the power function results in $\mathcal{O}(logn)$ recursive calls. Each individual activation of the function uses $\mathcal{O}(1)$ operations (excluding the recursive calls), and so the total number of operations for computing power(x,n) is O(logn). This is a significant improvement over the original $\mathcal{O}(n)$-time algorithm.

**Exercise:** Test the two functions with a timer and see the difference. Identify the base case and stopping conditions.

```python
# so we got the famous stack overflow
# technically we could adjust the recursion depth in Python but it would not really help in the long run
```

```python
%%timeit
power(2,100_000)
```

```python
%%timeit
2**100_000
```

```python
%%timeit
power(2,1_000_000)
```

```python
%%timeit
2**1_000_000
```

# We humans operate better on visual data - 1 picture and 1000 words

```python
# in python we can pass functions as arguments
import matplotlib.pyplot as plt
import time
def automatic_time(input_array, func_name):
    # input_array contains all elements of input,
    #   preferably sorted from small to big,
    #   each element could also be a tuple, which is requried
    #   for functions that need more than 1 input
    # func_name is the name of the function (as object, not str)
    # this function also suppresses printing output from the {func_name}
    ret = []
    for val in input_array:
#         print(time.process_time_ns())
#         start = time.process_time()
        start = time.time()
        from IPython.utils import io as iuio
        with iuio.capture_output() as suppressed:
            if isinstance(val, tuple):
                _ = func_name(*val) #unrolling the tuple to pass multiple arguments to our function
            else:
                _ = func_name(val) # just a single argument function

        ret.append((time.time() - start)) # 10e9 converts unit from second to nanosecond
    return ret

# my_input_arr = [(2,1_000),(2,2_000)]
# print("testing function_1 with the automatic measure")
# print(F"run-time for function 1 result in nanoseconds: \n{automatic_time(my_input_arr, basicpower)}")
```

```python

```

```python
input_arr = [(2,i) for i in range(100, 2_500, 100)]

# input_arr_t = [(i,i) for i in range(1, 1000)]

# fun1 = automatic_time(input_arr, function_1)
results = automatic_time(input_arr, basicpower)
# fun3 = automatic_time(input_arr, function_3)
# fun4 = automatic_time(input_arr_t, function_4)
# fun5 = automatic_time(input_arr_t, function_5)

plt.plot([el[1] for el in input_arr], results) # we only want to see the i as x
plt.xlabel("input size")
plt.ylabel("running time in seconds")
plt.show()
```

```python
input_arr = [(2,i) for i in range(100_000, 20_000_000, 100_000)]

results = automatic_time(input_arr, power)
# mathpow_results = automatic_time(input_arr, math.pow) # math.pow does n ot like large powers

plt.plot([el[1] for el in input_arr], results)
# plt.plot([el[1] for el in input_arr], mathpow_results)
plt.xlabel("input size")
plt.ylabel("running time in seconds")
plt.show()
```

```python
math.pow(2,800_000)
```

# so it certainly looks linear but the jumps in our speed indicates how OS and computer hardware might affect an algorithm in real life

## Factorial

```python
# so again base case for stopping will be 1
```

```python
def factorial(x):
    if x <= 1:
        return 1 # not x! as factorial of 0 is considered 1!
    else:
        return x * factorial(x-1) # so we want to be sure that our problem space keeps getting smaller
```

```python
math.factorial(0)
```

```python
# regular loop based algorithm should need less memory
def factorial_loop(x):
    res = 1
    while x > 1:
        res *= x
        x -=1
    return res
```

```python
factorial_loop(5)
```

```python
factorial(5)
```

```python
factorial(1)
```

5! = 5 x 4 x 3 x 2 x 1 = 120

```python
factorial(5)
```

```python
%%timeit
factorial(10)
```

```python
%%timeit
factorial(100)
```

```python
%%timeit
factorial_loop(100)
```

```python
%%timeit
factorial(1000)
```

```python
%%timeit
factorial(2000)
```

```python
%%timeit
factorial_loop(1000)
```

```python
%%timeit
factorial_loop(2000)
```

```python
factorial(5000)
```

```python
%%timeit
factorial(5000) #
```

```python
input_arr = [i for i in range(1_000, 30_000, 1_000)]

fun2 = automatic_time(input_arr, factorial_loop)
plt.plot(input_arr, fun2)
plt.xlabel("input size")
plt.ylabel("running time in seconds")
plt.show()
```

```python
input_arr = [i for i in range(100, 2_500, 100)]

fun2 = automatic_time(input_arr, factorial_loop)
plt.plot(input_arr, fun2)
fun1 = automatic_time(input_arr, factorial)
plt.plot(input_arr, fun1)
plt.xlabel("input size")
plt.ylabel("running time in seconds")
plt.show()
```

```python

```

# Exercise: Fibonacci

The sequence, in which each number is the sum of the two preceding numbers is known as the Fibonacci sequence: $1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, \cdots$ (each number is the sum of the previous two, e.g., $144=55 + 89$).

As you covered Fibonacci in Semester 1, this should be familiar. The twist is we are going to look at a variety of ways of implementing it and the complexity of execution for each.

A direct implementation of the description above would lead to pseudo code like this:

```python
def bad_fibonacci(n):
  """Return the nth Fibonacci number."""
  if n <= 1:
    return n
  else:
    return bad_fibonacci(n-2) + bad_fibonacci(n-1)
```

```python
for n in range(10):
    print(bad_fibonacci(n))
```

```python
%%timeit
bad_fibonacci(10)
```

```python
%%timeit
bad_fibonacci(12)
```

```python
%%timeit
bad_fibonacci(16)
```

```python
%%timeit
bad_fibonacci(20)
```

```python
%%timeit
bad_fibonacci(24)
```

```python
%%timeit
bad_fibonacci(32)
```

```python
input_arr = [i for i in range(1, 35, 1)]

fun2 = automatic_time(input_arr, bad_fibonacci)
plt.plot(input_arr, fun2)
plt.xlabel("input size")
plt.ylabel("running time in seconds")
plt.show()
```

```python
# so bad_fibonacci looks like an exponational complexity
```

```python
arr = [2**n for n in range(30)]
plt.plot(list(range(30)), arr)
plt.show()
```

```python
# sure smells like exponential
```

```python
# what is so bad about bad fibonacci ?
# so for each recursive call you get new 2 recursive calls
# it is like that story of inventor of chess in India
# he asked for 1 grain of rice on first square
# 2 on 2nd , 4 on 3rd, 8 on 4th,
# the story modification is that king realized the problem
# to get out of it king said ok, you have to count the rice

```

```python
2**32 # he could count half a board
```

```python
2**64 # not likely for a human...
```

```python

def good_fibonacci(n):
  """Return pair of Fibonacci numbers, F(n) and F(n-1)."""
  if n <= 1:
    return (n,0)
  else:
    (a, b) = good_fibonacci(n-1) # looks linear here
    return (a+b, a)
```

```python
%%timeit
good_fibonacci(10)
```

```python
%%timeit
good_fibonacci(20)
```

```python
%%timeit
good_fibonacci(30)
```

```python
input_arr = [i for i in range(100, 3_000, 100)]

fun2 = automatic_time(input_arr, good_fibonacci)
plt.plot(input_arr, fun2)
plt.xlabel("input size")
plt.ylabel("running time in seconds")
plt.show()
```

```python
import sys
print(sys.getrecursionlimit())
```

```python
#https://stackoverflow.com/questions/3323001/what-is-the-maximum-recursion-depth-in-python-and-how-to-increase-it
sys.setrecursionlimit(100_000)
print(sys.getrecursionlimit())
```

```python
good_fibonacci(5_000)
```

```python
## Fibonacci Solutions

1. timing
```

```python
def fibonacci_loop(n):
    a, b = 1, 1
    cnt = 3
    while cnt <= n:
        a, b = a+b, a
        cnt += 1
    return a
```

```python
fibonacci_loop(2)
```

```python
fibonacci_loop(10)
```

```python
input_arr = [i for i in range(1000, 200_000, 2000)]

fun2 = automatic_time(input_arr, fibonacci_loop)
plt.plot(input_arr, fun2)
plt.xlabel("input size")
plt.ylabel("running time in seconds")
plt.show()
```

```python
# 2. It is not tail recursive becuase there is something to do after returning the values (adding them together). We can use 2 accumulators to avoid this.
```

# Example: Binary Search

Binary search is a classic recursive algorithm to find a target value within a sorted sequence. This is among the most important of computer algorithms, and it is the reason that we so often store data in sorted order.

e.g. For the sorted sequence below stored in a Python list with indexes above:

0	1	 2	3	4	 5	6	7	 8	9	10	 11

2	4	 5	6	8	 9	15	16	 17	22	30	 31

If the sequence was unsorted a simple solution is a _sequential search algorithm_: use a loop to examine every element. You either reach the end of the list or find the target.

It is linear complexity, running in $\mathcal{O}(n)$ time as worst case it inspects every element in the sequence.

A sorted sequence allows a much faster approach. Think about how you would accomplish this task by hand: divide in two, and choose the middle digit as a candidate to compare to the target. Everything to the left of the candidate is lower than it and everything to the right is higher. Compare the target to the candidate and discard the left if the candidate is lower and the right if it is higher. Then repeat your _binary search_ algorithm. This is much more efficient, running in $\mathcal{O}($log$n$) time.

Review the iterative and recursive binary search algorithm implementations below.

```python
def binary_search_iterative(data, target):
    """Return True if target is found in the given Python list."""
    low = 0
    high = len(data)-1
    while low <= high: # so while array/list pointers do not meet we keep going
        mid = (low + high) // 2 # // meaning even half
        if target == data[mid]:         # found a match
            return True
        elif target < data[mid]:
            high = mid - 1                # only consider values left of mid
        else:
            low = mid + 1                 # only consider values right of mid
    return False                      # loop ended without success
```

```python
def binary_search(data, target, low, high):
    """Return True if target is found in indicated portion of a Python list.

      The search only considers the portion from data[low] to data[high] inclusive.
      """
    if low > high:
        return False                    # interval is empty; no match
    else:
        mid = (low + high) // 2
    if target == data[mid]:         # found a match
        return True
    elif target < data[mid]:
        # recur on the portion left of the middle
        return binary_search(data, target, low, mid - 1)
    else: # target > data[mid]
        # recur on the portion right of the middle
        return binary_search(data, target, mid + 1, high)
```

When a function makes two recursive calls, we say that it uses binary recursion. Clearly the binary_search above is a binary recursion. Drawing the English ruler and the bad fibonacci function are also examples of binary recursion.

```python
data=[2,4,5,6,8,9,15,16,17,22,30,31]
iter_ans = binary_search_iterative(data,100)
iter_ans
```

```python
binary_search_iterative(data,15)
```

```python
binary_search_iterative(data,7)
```

```python
%%timeit
binary_search_iterative(list(range(1_000_000)), 9000)
```

```python

```

```python
input_arr = [(list(range(i)), 9000) for i in range(100_000, 1_000_000, 50_000)]

fun1 = automatic_time(input_arr, binary_search_iterative)
plt.plot([len(el[0]) for el in input_arr], fun1)
# fun2 = automatic_time(input_arr, binary_search) # the recursive one
# plt.plot([len(el[0]) for el in input_arr], fun2)
plt.xlabel("input size")
plt.ylabel("running time in seconds")
plt.show()
```

```python
# so binary search is in log n time we would need a trulyl large data sets to see difference
# most of our time is spent linearly setting up our data sets
```

```python
arr = list(range(1_000_000))
big_arr = list(range(100_000_000))
```

```python
%%timeit
binary_search(arr, 9000, 0, len(arr)-1)
```

    9.56 µs ± 246 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%%timeit
binary_search_iterative(arr, 9000)
```

    7.23 µs ± 120 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%%timeit
binary_search(big_arr, 9000, 0, len(big_arr)-1)
```

    17.3 µs ± 483 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%%timeit
binary_search(big_arr, 50_000_000, 0, len(big_arr)-1) # so this should hit with 1 comparison
```

    16 µs ± 150 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%%timeit
binary_search_iterative(big_arr, 9000)
```

    13.2 µs ± 513 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%%timeit
binary_search_iterative(big_arr, 50_000_000) # so this should hit with 1 comparison
```

    12.6 µs ± 468 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
# so creating a recursive algorithm yourself
# check if problem is already solved
# otherwise reduce problem to simpler instance of the same problem
# in other words Simplify and Delegate the problem
```

```python
# all recursive algorithms can be transformed to regular loop based algorithms
# worst case you build your stack for holding the function calls
```

## Length of an array

```python
# in case we forgout about len() function ...
```

```python
def array_len(x):
    if x == []:
        return 0
    else:
        return 1 + array_len(x[1:])
```

```python
array_len([])
```

    0

```python
array_len([1, 2, 3])
```

    3

```python
array_len(list(range(2961))) # so it looks like 3000 limit is actually 2961 (presumably stack needs space for other functions)
```

    2961

## Sum of the elements in an array

```python
# in case we forgot about sum() function ...
```

```python
def array_sum(x):
    if x == []:
        return 0
    else:
        return x[0] + array_sum(x[1:])
```

```python
array_sum([])
```

    0

```python
array_sum([5])
```

    5

```python
array_sum([1, 2, 3, 4, 5])
```

    15

```python
array_sum(list(range(101))) # remember the formula for summing arithmethic series n(n-1)/2
```

    5050

```python
array_sum(list(range(2961)))
```

    4382280

# Tower of Hanoi

* must not move more than one disc at a time
* must place this disc on a needle so that there is no smaller disc below it

## Quicksort - more about sorting later on

```python
def quicksort(array):
    if len(array) < 2:
        return array
    else:
        pivot = array[0]
        smaller, bigger = [], []
        for ele in array[1:]:
            if ele <= pivot:
                smaller.append(ele)
            else:
                bigger.append(ele)
        return quicksort(smaller) + [pivot] + quicksort(bigger)
```

```python
quicksort([])
```

```python
quicksort([5])
```

```python
quicksort([5, 4])
```

```python
quicksort([1, 2, 7, 5, 4])
```

```python
quicksort([5, 4, 3, 2])
```


```python
import sympy as sy

sy.init_printing()
```

## 2 equations, 2 unknowns

```python
x, y = sy.symbols('x y')
a, b, c, d = sy.symbols('a b c d')

eq1 = sy.Eq(a*x + b*y + 2, 0)
eq2 = sy.Eq(c*x + d*y - 2, 0)

sol = sy.solve((eq1, eq2), (x, y))
sol
```

## Try subscripts

```python
x, y = sy.symbols('x_mz y_ny')
a, b, c, d = sy.symbols('a_1 b_2 c_3 d_4')

eq1 = sy.Eq(a*x + b*y + 2, 0)
eq2 = sy.Eq(c*x + d*y - 2, 0)

sol = sy.solve((eq1, eq2), (x, y))
sol
```

## 3 equations, 3 unknowns

```python
x, y, z = sy.symbols('x y z')
a, b, c, d, e, f, g, h, i, j = sy.symbols('a b c d e, f, g, h, i, j')

eq1 = sy.Eq(a*x + b*y + c*z + 2, 0)
eq2 = sy.Eq(d*x + e*y + f*z - 2, 0)
eq3 = sy.Eq(g*x + h*y + i*z, 0)

sol = sy.solve((eq1, eq2, eq3), (x, y, z))
sol
```

```python
sol[x]
```

```python

```


## Код Хэмминга (n,k)

- $n$ - число бит в кодовой последовательности
- $k$ - число информационных бит
- $r = n - k$ - число контрольных бит

```python
import numpy as np
from sympy import *
from functools import reduce
from IPython.display import Math
```

### Пример для (7,4)

```python
n,k = (7,4)
r = n - k
```

Контрольные биты четности $p_{1..r}$ вычисляются из битов данных $d_{1..k}$ следующим образом:

> $p_1 = d_1 \oplus d_2 \oplus d_4$
> $p_2 = d_1 \oplus d_3 \oplus d_4$
> $p_3 = d_2 \oplus d_3 \oplus d_4$

Вектор кодовой последовательности $\langle d_1, d_2, d_3, d_4, p_1, p_2, p_3 \rangle$ можно получить умножением вектора исходных данных $\langle d_1, d_2, d_3, d_4 \rangle$ на матрицу-генератор $G$, где $P$ - матрица перестановки, соответствующая вышеупомянутым комбинациям информационных битов для вычисления битов четности:

> $G = \begin{bmatrix}I_{k} ~\vdots~ P\end{bmatrix}$

Для проверки наличия ошибок в кодовой последовательности вычисляется вектор синдромов $\vec{S} = \langle S_1, S_2, S_3 \rangle$, такой что:

> $S_1 = p_1 \oplus d_1 \oplus d_2 \oplus d_4$
> $S_2 = p_2 \oplus d_1 \oplus d_3 \oplus d_4$
> $S_3 = p_3 \oplus d_2 \oplus d_3 \oplus d_4$

Если вектор синдромов равен нулевому вектору $\langle 0, 0, 0 \rangle$, то кодовая последовательность не содержит ошибок. Вектор синдромов можно получить умножением вектора кодовой последовательности на матрицу контроля четности:

> $H = \begin{bmatrix}P \\ \cdots \\ I_{r}\end{bmatrix}$

---
#### Матрица-генератор:

```python
Ik = np.eye(k, dtype='B')
P = np.array([[1,1,0],[1,0,1],[0,1,1],[1,1,1]], dtype='B')

G = np.hstack((Ik, P))

display(Math('G = ' + latex(Matrix(G))))
```

$\displaystyle G = \left[\begin{matrix}1 & 0 & 0 & 0 & 1 & 1 & 0\\0 & 1 & 0 & 0 & 1 & 0 & 1\\0 & 0 & 1 & 0 & 0 & 1 & 1\\0 & 0 & 0 & 1 & 1 & 1 & 1\end{matrix}\right]$

#### Матрица контроля чётности:

```python
Ir = np.eye(r, dtype='B')
H = np.vstack((P, Ir))

display(Math('H = ' + latex(Matrix(H))))
```

$\displaystyle H = \left[\begin{matrix}1 & 1 & 0\\1 & 0 & 1\\0 & 1 & 1\\1 & 1 & 1\\1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

#### Произвольный вектор с информационными битами: $\vec{D}$

```python
D = np.random.randint(0, 2, (1, k), dtype='B')

display(Math('\\vec{D} = ' + latex(Matrix(D))))
```

$\displaystyle \vec{D} = \left[\begin{matrix}0 & 1 & 1 & 0\end{matrix}\right]$

#### Вычисление кодовой последовательности: $\vec{C} = \vec{D} \times G$

```python
C = np.dot(D, G) % 2

display(Math('\\vec{C} = ' + latex(Matrix(C))))
```

$\displaystyle \vec{C} = \left[\begin{matrix}0 & 1 & 1 & 0 & 1 & 1 & 0\end{matrix}\right]$

#### Проверка кодовой последовательности, вычисление вектора синдромов: $\vec{S} = \vec{C} \times H$

```python
S = np.dot(C, H) % 2

display(Math('\\vec{S} = ' + latex(Matrix(S))))
```

$\displaystyle \vec{S} = \left[\begin{matrix}0 & 0 & 0\end{matrix}\right]$

---
#### Исправление ошибок в кодовой последовательности

Для обнаружения и исправления единичной ошибки в кодовой последовательности удобно распологать в ней биты в таком порядке, чтобы контрольные биты $p_{1..r}$ находились на позициях кратных степени двойки: $2^{x}, \{ x \in \mathbb{N}, 0 \leq x \lt r \}$.
В этом случае полученный при декодировании вектор синдромов $\langle S_1, S_2, S_3 \rangle$ будет являться двоичным представлением индекса ошибочного символа в кодовой последовательности.

Переставим столбцы в матрице-генераторе и строки в матрице контроля четности таким образом, чтобы результирующий вектор кодовой последовательности содержал биты в упомянутом порядке:

> $\langle d_1, d_2, d_3, d_4, p_1, p_2, p_3 \rangle \mapsto \langle p_1, p_2, d_1, p_3, d_2, d_3, d_4 \rangle$

```python
Gp = G[:,(4,5,0,6,1,2,3)]
Hp = H[(4,5,0,6,1,2,3),:]

display(Math('Gp = ' + latex(Matrix(Gp)) +'; ~~~ Hp = ' + latex(Matrix(Hp))))
```

$\displaystyle Gp = \left[\begin{matrix}1 & 1 & 1 & 0 & 0 & 0 & 0\\1 & 0 & 0 & 1 & 1 & 0 & 0\\0 & 1 & 0 & 1 & 0 & 1 & 0\\1 & 1 & 0 & 1 & 0 & 0 & 1\end{matrix}\right]; ~~~ Hp = \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\1 & 1 & 0\\0 & 0 & 1\\1 & 0 & 1\\0 & 1 & 1\\1 & 1 & 1\end{matrix}\right]$

Вычислим кодовую последовательность и синдром используя обновленные матрицы:

```python
Cp = np.dot(D, Gp) % 2
Sp = np.dot(Cp, Hp) % 2
display(Math('\\vec{Cp} = ' + latex(Matrix(Cp))))
display(Math('\\vec{Sp} = ' + latex(Matrix(Sp))))
```

$\displaystyle \vec{Cp} = \left[\begin{matrix}1 & 1 & 0 & 0 & 1 & 1 & 0\end{matrix}\right]$

$\displaystyle \vec{Sp} = \left[\begin{matrix}0 & 0 & 0\end{matrix}\right]$

Тест: Имитация единичной ошибки в кодовой последовательности и вычисление индекса ошибочного символа:

```python
for i in range(7):
    Cp[0,i] ^= 1
    Sp = np.dot(Cp, Hp) % 2
    Cp[0,i] ^= 1
    err = np.packbits(Sp, bitorder='little')
    display(Math("\\vec{Sp} = " + latex(Matrix(Sp)) + " \\implies \\text{ошибка в символе}~\\vec{Cp_{%d}}" % err))
```

$\displaystyle \vec{Sp} = \left[\begin{matrix}1 & 0 & 0\end{matrix}\right] \implies \text{ошибка в символе}~\vec{Cp_{1}}$

$\displaystyle \vec{Sp} = \left[\begin{matrix}0 & 1 & 0\end{matrix}\right] \implies \text{ошибка в символе}~\vec{Cp_{2}}$

$\displaystyle \vec{Sp} = \left[\begin{matrix}1 & 1 & 0\end{matrix}\right] \implies \text{ошибка в символе}~\vec{Cp_{3}}$

$\displaystyle \vec{Sp} = \left[\begin{matrix}0 & 0 & 1\end{matrix}\right] \implies \text{ошибка в символе}~\vec{Cp_{4}}$

$\displaystyle \vec{Sp} = \left[\begin{matrix}1 & 0 & 1\end{matrix}\right] \implies \text{ошибка в символе}~\vec{Cp_{5}}$

$\displaystyle \vec{Sp} = \left[\begin{matrix}0 & 1 & 1\end{matrix}\right] \implies \text{ошибка в символе}~\vec{Cp_{6}}$

$\displaystyle \vec{Sp} = \left[\begin{matrix}1 & 1 & 1\end{matrix}\right] \implies \text{ошибка в символе}~\vec{Cp_{7}}$