# Transformations, Eigenvectors, and Eigenvalues

Matrices and vectors are used together to manipulate spatial dimensions. This has a lot of applications, including the mathematical generation of 3D computer graphics, geometric modeling, and the training and optimization of machine learning algorithms. We're not going to cover the subject exhaustively here; but we'll focus on a few key concepts that are useful to know when you plan to work with machine learning.

## Linear Transformations
You can manipulate a vector by multiplying it with a matrix. The matrix acts a function that operates on an input vector to produce a vector output. Specifically, matrix multiplications of vectors are *linear transformations* that transform the input vector into the output vector.

For example, consider this matrix ***A*** and vector ***v***:

$$ A = \begin{bmatrix}2 & 3\\5 & 2\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\2\end{bmatrix}$$

We can define a transformation ***T*** like this:

$$ T(\vec{v}) = A\vec{v} $$

To perform this transformation, we simply calculate the dot product by applying the *RC* rule; multiplying each row of the matrix by the single column of the vector:

$$\begin{bmatrix}2 & 3\\5 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}8\\9\end{bmatrix}$$

Here's the calculation in Python:

```python
import numpy as np

v = np.array([1,2])
A = np.array([[2,3],
              [5,2]])

t = A@v
print (t)
```

In this case, both the input vector and the output vector have 2 components - in other words, the transformation takes a 2-dimensional vector and produces a new 2-dimensional vector; which we can indicate like this:

$$ T: \rm I\!R^{2} \to \rm I\!R^{2} $$

Note that the output vector may have a different number of dimensions from the input vector; so the matrix function might transform the vector from one space to another - or in notation, ${\rm I\!R}$n -> ${\rm I\!R}$m.

For example, let's redefine matrix ***A***, while retaining our original definition of vector ***v***:

$$ A = \begin{bmatrix}2 & 3\\5 & 2\\1 & 1\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\2\end{bmatrix}$$

Now if we once again define ***T*** like this:

$$ T(\vec{v}) = A\vec{v} $$

We apply the transformation like this:

$$\begin{bmatrix}2 & 3\\5 & 2\\1 & 1\end{bmatrix} \cdot  \begin{bmatrix}1\\2\end{bmatrix} = \begin{bmatrix}8\\9\\3\end{bmatrix}$$

So now, our transformation transforms the vector from 2-dimensional space to 3-dimensional space:

$$ T: \rm I\!R^{2} \to \rm I\!R^{3} $$

Here it is in Python:

```python
import numpy as np
v = np.array([1,2])
A = np.array([[2,3],
              [5,2],
              [1,1]])

t = A@v
print (t)
```

```python
import numpy as np
v = np.array([1,2])
A = np.array([[1,2],
              [2,1]])

t = A@v
print (t)
```

## Transformations of Magnitude and Amplitude

When you multiply a vector by a matrix, you transform it in at least one of the following two ways:
* Scale the length (*magnitude*) of the matrix to make it longer or shorter
* Change the direction (*amplitude*) of the matrix

For example consider the following matrix and vector:

$$ A = \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \;\;\;\; \vec{v} = \begin{bmatrix}1\\0\end{bmatrix}$$

As before, we transform the vector ***v*** by multiplying it with the matrix ***A***:

\begin{equation}\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}\end{equation}

In this case, the resulting vector has changed in length (*magnitude*), but has not changed its direction (*amplitude*).

Let's visualize that in Python:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,0],
              [0,2]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([t,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

The original vector ***v*** is shown in orange, and the transformed vector ***t*** is shown in blue - note that ***t*** has the same direction (*amplitude*) as ***v*** but a greater length (*magnitude*).

Now let's use a different matrix to transform the vector ***v***:
\begin{equation}\begin{bmatrix}0 & -1\\1 & 0\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}0\\1\end{bmatrix}\end{equation}

This time, the resulting vector has been changed to a different amplitude, but has the same magnitude.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[0,-1],
              [1,0]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'blue'], scale=10)
plt.show()
```

Now let's see change the matrix one more time:
\begin{equation}\begin{bmatrix}2 & 1\\1 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\1\end{bmatrix}\end{equation}

Now our resulting vector has been transformed to a new amplitude *and* magnitude - the transformation has affected both direction and scale.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,1],
              [1,2]])

t = A@v
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'blue'], scale=10)
plt.show()
```

### Afine Transformations
An Afine transformation multiplies a vector by a matrix and adds an offset vector, sometimes referred to as *bias*; like this:

$$T(\vec{v}) = A\vec{v} + \vec{b}$$

For example:

\begin{equation}\begin{bmatrix}5 & 2\\3 & 1\end{bmatrix} \cdot  \begin{bmatrix}1\\1\end{bmatrix} + \begin{bmatrix}-2\\-6\end{bmatrix} = \begin{bmatrix}5\\-2\end{bmatrix}\end{equation}

This kind of transformation is actually the basis of linear regression, which is a core foundation for machine learning. The matrix defines the *features*, the first vector is the *coefficients*, and the bias vector is the *intercept*.

here's an example of an Afine transformation in Python:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,1])
A = np.array([[5,2],
              [3,1]])
b = np.array([-2,-6])

t = A@v + b
print (t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'blue'], scale=15)
plt.show()
```

## Eigenvectors and Eigenvalues
So we can see that when you transform a vector using a matrix, we change its direction, length, or both. When the transformation only affects scale (in other words, the output vector has a different magnitude but the same amplitude as the input vector), the matrix multiplication for the transformation is the equivalent operation as some scalar multiplication of the vector.

For example, earlier we examined the following transformation that dot-mulitplies a vector by a matrix:

$$\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

You can achieve the same result by mulitplying the vector by the scalar value ***2***:

$$2 \times \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

The following python performs both of these calculation and shows the results, which are identical.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

v = np.array([1,0])
A = np.array([[2,0],
              [0,2]])

t1 = A@v
print (t1)
t2 = 2*v
print (t2)

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,v])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

In cases like these, where a matrix transformation is the equivelent of a scalar-vector multiplication, the scalar-vector pairs that correspond to the matrix are known respectively as eigenvalues and eigenvectors. We generally indicate eigenvalues using the Greek letter lambda (λ), and the formula that defines eigenvalues and eigenvectors with respect to a transformation is:

$$ T(\vec{v}) = \lambda\vec{v}$$

Where the vector ***v*** is an eigenvector and the value ***λ*** is an eigenvalue for transformation ***T***.

When the transformation ***T*** is represented as a matrix multiplication, as in this case where the transformation is represented by matrix ***A***:

$$ T(\vec{v}) = A\vec{v} = \lambda\vec{v}$$

Then  ***v*** is an eigenvector and ***λ*** is an eigenvalue of ***A***.

A matrix can have multiple eigenvector-eigenvalue pairs, and you can calculate them manually. However, it's generally easier to use a tool or programming language. For example, in Python you can use the ***linalg.eig*** function, which returns an array of eigenvalues and a matrix of the corresponding eigenvectors for the specified matrix.

Here's an example that returns the eigenvalue and eigenvector pairs for the following matrix:

$$A=\begin{bmatrix}2 & 0\\0 & 3\end{bmatrix}$$

```python
import numpy as np
A = np.array([[2,0],
              [0,3]])
eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

So there are two eigenvalue-eigenvector pairs for this matrix, as shown here:

$$ \lambda_{1} = 2, \vec{v_{1}} = \begin{bmatrix}1 \\ 0\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 3, \vec{v_{2}} = \begin{bmatrix}0 \\ 1\end{bmatrix} $$

Let's verify that multiplying each eigenvalue-eigenvector pair corresponds to the dot-product of the eigenvector and the matrix. Here's the first pair:

$$ 2 \times \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 3\end{bmatrix} \cdot \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix} $$

So far so good. Now let's check the second pair:

$$ 3 \times \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 3\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 3\end{bmatrix} \cdot \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 3\end{bmatrix} $$

So our eigenvalue-eigenvector scalar multiplications do indeed correspond to our matrix-eigenvector dot-product transformations.

Here's the equivalent code in Python, using the ***eVals*** and ***eVecs*** variables you generated in the previous code cell:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))
```

You can use the following code to visualize these transformations:

```python
t1 = lam1*vec1
print (t1)
t2 = lam2*vec2
print (t2)

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

Similarly, earlier we examined the following matrix transformation:

$$\begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot  \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

And we saw that you can achieve the same result by mulitplying the vector by the scalar value ***2***:

$$2 \times \begin{bmatrix}1\\0\end{bmatrix} = \begin{bmatrix}2\\0\end{bmatrix}$$

This works because the scalar value 2 and the vector (1,0) are an eigenvalue-eigenvector pair for this matrix.

Let's use Python to determine the eigenvalue-eigenvector pairs for this matrix:

```python
import numpy as np
A = np.array([[2,0],
              [0,2]])
eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

So once again, there are two eigenvalue-eigenvector pairs for this matrix, as shown here:

$$ \lambda_{1} = 2, \vec{v_{1}} = \begin{bmatrix}1 \\ 0\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 2, \vec{v_{2}} = \begin{bmatrix}0 \\ 1\end{bmatrix} $$

Let's verify that multiplying each eigenvalue-eigenvector pair corresponds to the dot-product of the eigenvector and the matrix. Here's the first pair:

$$ 2 \times \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}1 \\ 0\end{bmatrix} = \begin{bmatrix}2 \\ 0\end{bmatrix} $$

Well, we already knew that. Now let's check the second pair:

$$ 2 \times \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 2\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 0\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}0 \\ 1\end{bmatrix} = \begin{bmatrix}0 \\ 2\end{bmatrix} $$

Now let's use Pythonto verify and plot these transformations:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))

# Plot the resulting vectors
t1 = lam1*vec1
t2 = lam2*vec2

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

Let's take a look at one more, slightly more complex example. Here's our matrix:

$$\begin{bmatrix}2 & 1\\1 & 2\end{bmatrix}$$

Let's get the eigenvalue and eigenvector pairs:

```python
import numpy as np

A = np.array([[2,1],
              [1,2]])

eVals, eVecs = np.linalg.eig(A)
print(eVals)
print(eVecs)
```

This time the eigenvalue-eigenvector pairs are:

$$ \lambda_{1} = 3, \vec{v_{1}} = \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix}  \;\;\;\;\;\; \lambda_{2} = 1, \vec{v_{2}} = \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} $$

So let's check the first pair:

$$ 3 \times \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}2.12132034 \\ 2.12132034\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 1\\0 & 2\end{bmatrix} \cdot \begin{bmatrix}0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}2.12132034 \\ 2.12132034\end{bmatrix} $$

Now let's check the second pair:

$$ 1 \times \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}-0.70710678\\0.70710678\end{bmatrix}  \;\;\;and\;\;\; \begin{bmatrix}2 & 1\\1 & 2\end{bmatrix} \cdot \begin{bmatrix}-0.70710678 \\ 0.70710678\end{bmatrix} = \begin{bmatrix}-0.70710678\\0.70710678\end{bmatrix} $$

With more complex examples like this, it's generally easier to do it with Python:

```python
vec1 = eVecs[:,0]
lam1 = eVals[0]

print('Matrix A:')
print(A)
print('-------')

print('lam1: ' + str(lam1))
print ('v1: ' + str(vec1))
print ('Av1: ' + str(A@vec1))
print ('lam1 x v1: ' + str(lam1*vec1))

print('-------')

vec2 = eVecs[:,1]
lam2 = eVals[1]

print('lam2: ' + str(lam2))
print ('v2: ' + str(vec2))
print ('Av2: ' + str(A@vec2))
print ('lam2 x v2: ' + str(lam2*vec2))

# Plot the results
t1 = lam1*vec1
t2 = lam2*vec2

fig = plt.figure()
a=fig.add_subplot(1,1,1)
# Plot v and t1
vecs = np.array([t1,vec1])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
a=fig.add_subplot(1,2,1)
# Plot v and t2
vecs = np.array([t2,vec2])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['blue', 'orange'], scale=10)
plt.show()
```

## Eigendecomposition
So we've learned a little about eigenvalues and eigenvectors; but you may be wondering what use they are. Well, one use for them is to help decompose transformation matrices.

Recall that previously we found that a matrix transformation of a vector changes its magnitude, amplitude, or both. Without getting too technical about it, we need to remember that vectors can exist in any spatial orientation, or *basis*; and the same transformation can be applied in different *bases*.

We can decompose a matrix using the following formula:

$$A = Q \Lambda Q^{-1}$$

Where ***A*** is a trasformation that can be applied to a vector in its current base, ***Q*** is a matrix of eigenvectors that defines a change of basis, and ***Λ*** is a matrix with eigenvalues on the diagonal that defines the same linear transformation as ***A*** in the base defined by ***Q***.

Let's look at these in some more detail. Consider this matrix:

$$A=\begin{bmatrix}3 & 2\\1 & 0\end{bmatrix}$$

***Q*** is a matrix in which each column is an eigenvector of ***A***; which as we've seen previously, we can calculate using Python:

```python
import numpy as np

A = np.array([[3,2],
              [1,0]])

l, Q = np.linalg.eig(A)
print(Q)
```

So for matrix ***A***, ***Q*** is the following matrix:

$$Q=\begin{bmatrix}0.96276969 & -0.48963374\\0.27032301 & 0.87192821\end{bmatrix}$$

***Λ*** is a matrix that contains the eigenvalues for ***A*** on the diagonal, with zeros in all other elements; so for a 2x2 matrix, Λ will look like this:

$$\Lambda=\begin{bmatrix}\lambda_{1} & 0\\0 & \lambda_{2}\end{bmatrix}$$

In our Python code, we've already used the ***linalg.eig*** function to return the array of eigenvalues for ***A*** into the variable ***l***, so now we just need to format that as a matrix:

```python
L = np.diag(l)
print (L)
```

So ***Λ*** is the following matrix:

$$\Lambda=\begin{bmatrix}3.56155281 & 0\\0 & -0.56155281\end{bmatrix}$$

Now we just need to find ***Q-1***, which is the inverse of ***Q***:

```python
Qinv = np.linalg.inv(Q)
print(Qinv)
```

The inverse of ***Q*** then, is:

$$Q^{-1}=\begin{bmatrix}0.89720673 & 0.50382896\\-0.27816009 & 0.99068183\end{bmatrix}$$

So what does that mean? Well, it means that we can decompose the transformation of *any* vector multiplied by matrix ***A*** into the separate operations ***QΛQ-1***:

$$A\vec{v} = Q \Lambda Q^{-1}\vec{v}$$

To prove this, let's take vector ***v***:

$$\vec{v} = \begin{bmatrix}1\\3\end{bmatrix} $$

Our matrix transformation using ***A*** is:

$$\begin{bmatrix}3 & 2\\1 & 0\end{bmatrix} \cdot \begin{bmatrix}1\\3\end{bmatrix} $$

So let's show the results of that using Python:

```python
v = np.array([1,3])
t = A@v

print(t)

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'b'], scale=20)
plt.show()
```

And now, let's do the same thing using the ***QΛQ-1*** sequence of operations:

```python
import math
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

t = (Q@(L@(Qinv)))@v

# Plot v and t
vecs = np.array([v,t])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'b'], scale=20)
plt.show()
```

So ***A*** and ***QΛQ-1*** are equivalent.

If we view the intermediary stages of the decomposed transformation, you can see the transformation using ***A*** in the original base for ***v*** (orange to blue) and the transformation using ***Λ*** in the change of basis decribed by ***Q*** (red to magenta):

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

t1 = Qinv@v
t2 = L@t1
t3 = Q@t2

# Plot the transformations
vecs = np.array([v,t1, t2, t3])
origin = [0], [0]
plt.axis('equal')
plt.grid()
plt.ticklabel_format(style='sci', axis='both', scilimits=(0,0))
plt.quiver(*origin, vecs[:,0], vecs[:,1], color=['orange', 'red', 'magenta', 'blue'], scale=20)
plt.show()
```

So from this visualization, it should be apparent that the transformation ***Av*** can be performed by changing the basis for ***v*** using ***Q*** (from orange to red in the above plot) applying the equivalent linear transformation in that base using ***Λ*** (red to magenta), and switching back to the original base using ***Q-1*** (magenta to blue).

## Rank of a Matrix

The **rank** of a square matrix is the number of non-zero eigenvalues of the matrix. A **full rank** matrix has the same number of non-zero eigenvalues as the dimension of the matrix. A **rank-deficient** matrix has fewer non-zero eigenvalues as dimensions. The inverse of a rank deficient matrix is singular and so does not exist (this is why in a previous notebook we noted that some matrices have no inverse).

Consider the following matrix ***A***:

$$A=\begin{bmatrix}1 & 2\\4 & 3\end{bmatrix}$$

Let's find its eigenvalues (***Λ***):

```python
import numpy as np
A = np.array([[1,2],
              [4,3]])
l, Q = np.linalg.eig(A)
L = np.diag(l)
print(L)
```

$$\Lambda=\begin{bmatrix}-1 & 0\\0 & 5\end{bmatrix}$$

This matrix has full rank. The dimensions of the matrix is 2. There are two non-zero eigenvalues.

Now consider this matrix:

$$B=\begin{bmatrix}3 & -3 & 6\\2 & -2 & 4\\1 & -1 & 2\end{bmatrix}$$

Note that the second and third columns are just scalar multiples of the first column.

Let's examine it's eigenvalues:

```python
B = np.array([[3,-3,6],
              [2,-2,4],
              [1,-1,2]])
lb, Qb = np.linalg.eig(B)
Lb = np.diag(lb)
print(Lb)
```

$$\Lambda=\begin{bmatrix}3 & 0& 0\\0 & -6\times10^{-17} & 0\\0 & 0 & 3.6\times10^{-16}\end{bmatrix}$$

Note that matrix has only 1 non-zero eigenvalue. The other two eigenvalues are so extremely small as to be effectively zero. This is an example of a rank-deficient matrix; and as such, it has no inverse.

## Inverse of a Square Full Rank Matrix
You can calculate the inverse of a square full rank matrix by using the following formula:

$$A^{-1} = Q \Lambda^{-1} Q^{-1}$$

Let's apply this to matrix ***A***:

$$A=\begin{bmatrix}1 & 2\\4 & 3\end{bmatrix}$$

Let's find the matrices for ***Q***, ***Λ-1***, and ***Q-1***:

```python
import numpy as np
A = np.array([[1,2],
              [4,3]])

l, Q = np.linalg.eig(A)
L = np.diag(l)
print(Q)
Linv = np.linalg.inv(L)
Qinv = np.linalg.inv(Q)
print(Linv)
print(Qinv)
```

So:

$$A^{-1}=\begin{bmatrix}-0.70710678 & -0.4472136\\0.70710678 & -0.89442719\end{bmatrix}\cdot\begin{bmatrix}-1 & -0\\0 & 0.2\end{bmatrix}\cdot\begin{bmatrix}-0.94280904 & 0.47140452\\-0.74535599 & -0.74535599\end{bmatrix}$$

Let's calculate that in Python:

```python
Ainv = (Q@(Linv@(Qinv)))
print(Ainv)
```

That gives us the result:

$$A^{-1}=\begin{bmatrix}-0.6 & 0.4\\0.8 & -0.2\end{bmatrix}$$

We can apply the ***np.linalg.inv*** function directly to ***A*** to verify this:

```python
print(np.linalg.inv(A))
```

Vector and Matrices by Hiren/03-05-Transformations Eigenvectors and Eigenvalues.ipynb


# Explicit Methods for the Model Hyperbolic PDE

The one-dimensional wave equation or linear convection equation is given by the following partial differential equation.

$$
\frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} = 0
$$

When we solve this PDE numerically, we divide the spatial and temporal domains into a series of mesh points and time points. Assume our problem domain is of length $L$, and that we want to compute the solution of $u(x,t)$ between time equals to zero to some final time, $t_f$ for all values of $x$ between zero and $L$. The first step in solving our PDE numerically is to divide the domain, $0 \leq x \leq L$, into a discrete number of mesh points or nodes, $N_x$. Likewise, we also divide the time domain, $0 \leq t \leq t_f$, into a discrete number of time steps, $N_t$. If we do so uniformally in time and space, then the distance bewteen each mesh point is $\Delta x$, and the distance between each time step is $\Delta t$.

In other words, for a uniform mesh size and a uniform time step, the value of $x$ for the i-th node is

$$
x_i = i \Delta x
$$

and the time at each time step, $n$, is

$$
t_n = n \Delta t
$$

The goal is to approximate this PDE as a difference equation, meaning that we want to represent the PDE using only a discrete number of mesh points, $N_x$, and time points, $N_t$. To do this requires using a Taylor series expansion about each point in the mesh, both in time and space, to approximate the time and space derivatives of $u$ in terms of a difference formula. Since in our Taylor series expansion we choose only a finite number of terms, these difference formulas are known as finite-difference approximations.

Since there are many different finite-difference formulas, let use define a common nomenclature. Let the symbol $\mathcal{D}$ represent a difference approximation. The subscript of $\mathcal{D}$ shall represent the direction of the finite-difference, forward (+), backward (-), or central (0). Last, let the denominator represent the domain over which the finite-difference formula is used. For the Cartesian domain, let us use the $\Delta x$ to represent a finite-difference approximation in the $x$-direction. Likewise, $\Delta y$ would represent a finite-difference approximation in the $y$-direction.

## First-Derivative Approximations

Using this nomenclature described above, let use define a set of finite-difference approximations for the first directive of the variable $\phi$ with respect to $x$. In other words, what are the available finite-difference formulae for

$$
\frac{ \partial \phi }{ \partial x} = \textrm{Finite-difference Approximation} + \textrm{Truncation Error}
$$

These expressions can be derived using a Taylor series expansion. By keeping track of the order of the higher-order terms neglected or truncated from the Taylor series expansion, we can also provide an estimate of the truncation error. We say that a finite-difference approximation is first-order accurate if the truncation error is of order $\mathcal{O} (\Delta x)$. It is second-order accurate if the truncation error is of order $\mathcal{O} (\Delta x^2)$.

**Forward difference**, first-order accurate, $\mathcal{O} (\Delta x)$

- Equally spaced

$$
\frac{\mathcal{D}_{+}\cdot}{\Delta x} \phi_i = \frac{\phi_{i+1} - \phi_i}{\Delta x}
$$

- Nonequally spaced

$$
\frac{\mathcal{D}_{+}\cdot}{\Delta x} \phi_i = \frac{\phi_{i+1} - \phi_i}{x_{i+1} - x_i}
$$

**Backward difference**, first-order accurate, $\mathcal{O} (\Delta x)$

- Equally spaced

$$
\frac{\mathcal{D}_{-}\cdot}{\Delta x} \phi_i = \frac{\phi_{i} - \phi_{i-1}}{\Delta x}
$$

- Nonequally spaced

$$
\frac{\mathcal{D}_{-}\cdot}{\Delta x} \phi_i = \frac{\phi_{i} - \phi_{i-1}}{x_{i} - x_{i-1}}
$$

**Central difference**, second-order accurate, $\mathcal{O} (\Delta x^2)$

- Equally spaced

$$
\frac{\mathcal{D}_{0}\cdot}{\Delta x} \phi_i = \frac{\phi_{i+1} - \phi_{i-1}}{2\Delta x}
$$

- Nonequally spaced

$$
\frac{\mathcal{D}_{0}\cdot}{\Delta x} \phi_i = \frac{\phi_{i+1} - \phi_{i-1}}{x_{i+1} - x_{i-1}}
$$

## Difference Equations for the Model Hyperbolic PDE

Let us use the above finite-difference approximations to derive several difference equations for the model hyperbolic PDEs. We have free reign in this choice, and most importantly, we certainly not limited to those provided above. It is possible to derive, third and fourth-order finite-difference approximations. These higher-order methods can become increasingly complex. Note, however, that while we can freely choose the difference approximation, we still need to check that this difference equation is not only consistent but also stable. Different combinations of time and spatial difference approximations result in different numerically stability.

### Forward-Time, Backward Space

Using a forward-time, backward-space difference approximation, the partial differential equation,

$$
\frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} = 0
$$

is transformed into the following difference equation,

$$
\frac{\mathcal{D}_{+}}{\Delta t} \Big( u^n_i \Big) + c \frac{\mathcal{D}_{-}}{\Delta x} \Big( u^n_i \Big)  = 0,
$$

which we can then expand as

$$
\frac{ u^{n+1}_i -  u^n_i }{\Delta t} + c \frac{ u^n_i -  u^n_{i-1} }{\Delta x}  = 0.
$$

From the truncation error of the difference approximations, we can say that this method is first-order accurate in time and space, in other words the truncation error is of the order $\mathcal{O}(\Delta t, \Delta x)$. We can represent this numerical method using the following diagram.

The *stencil* diagram visually shows the mesh points and time points that are used in the difference equation.

### Forward-Time, Forward Space

Using a forward-time, forward-space difference approximation, the partial differential equation,

$$
\frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} = 0
$$

is transformed into the following difference equation,

$$
\frac{\mathcal{D}_{+}}{\Delta t} \Big( u^n_i \Big) + c \frac{\mathcal{D}_{+}}{\Delta x} \Big( u^n_i \Big)  = 0,
$$

which we can then expand as

$$
\frac{ u^{n+1}_i -  u^n_i }{\Delta t} + c \frac{ u^n_{i+1} -  u^n_{i} }{\Delta x}  = 0.
$$

From the truncation error of the difference approximations, we can say that this method is first-order accurate in time and space, in other words the truncation error is of the order $\mathcal{O}(\Delta t, \Delta x)$. We can represent this numerical method using the following diagram.

The *stencil* diagram visually shows the mesh points and time points that are used in the difference equation.

### Forward-Time, Central Space

Using a forward-time, central-space difference approximation, the partial differential equation,

$$
\frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} = 0
$$

is transformed into the following difference equation,

$$
\frac{\mathcal{D}_{+}}{\Delta t} \Big( u^n_i \Big) + c \frac{\mathcal{D}_{0}}{\Delta x} \Big( u^n_i \Big)  = 0,
$$

which we can then expand as

$$
\frac{ u^{n+1}_i -  u^n_i }{\Delta t} + c \frac{ u^n_{i+1} -  u^n_{i-1} }{2 \Delta x}  = 0.
$$

From the truncation error of the difference approximations, we can say that this method is first-order accurate in time and second-order in space, in other words the truncation error is of the order $\mathcal{O}(\Delta t, \Delta x^2)$. We can represent this numerical method using the following diagram.

The *stencil* diagram visually shows the mesh points and time points that are used in the difference equation.

## Are all these methods stable?

***We need to apply von Neumann stability analysis to each of these difference equations to determine under what conditions, i.e., for what values of $\Delta t$ and $\Delta x$, is the numerical method stable?***

# Summary of Numerical Methods

## Explicit, Forward-Time, Backward-Space (FTBS)

$$
\begin{align}
\textrm{Method :}\quad & u^{n+1}_i = u^n_i - c \Delta t \frac{\mathcal{D}_{-} \cdot}{\Delta x} u_i^n \\
\textrm{Stability Criteria :}\quad & c > 0, \quad \Delta t \le \frac{\Delta x}{|c|} \\
\textrm{Order of Accuracy :}\quad & \mathcal{O}\left(\Delta t, \Delta x\right)
\end{align}
$$

## Explicit, Forward-Time, Forward-Space (FTFS)

$$
\begin{align}
\textrm{Method :}\quad & u^{n+1}_i = u^n_i - c \Delta t \frac{\mathcal{D}_{+} \cdot}{\Delta x} u_i^n \\
\textrm{Stability Criteria :}\quad & c < 0, \quad \Delta t \le \frac{\Delta x}{|c|} \\
\textrm{Order of Accuracy :}\quad & \mathcal{O}\left(\Delta t, \Delta x\right)
\end{align}
$$

Note that this method is only stable for $c < 0$.

## Explicit, Forward-Time, Central-Space (FTCS)

$$
\begin{align}
\textrm{Method :}\quad & u^{n+1}_i = u^n_i - c \Delta t \frac{\mathcal{D}_{0} \cdot}{\Delta x} u_i^n \\
\textrm{Stability Criteria :}\quad & \textrm{Always unstable} \\
\textrm{Order of Accuracy :}\quad & \mathcal{O}\left(\Delta t, \Delta x^2\right)
\end{align}
$$

This method is *always* unstable.

## Lax

$$
\begin{align}
\textrm{Method :}\quad & u^{n+1}_i = \frac{u^n_{i-1} + u^n_{i+1}}{2} - c \Delta t \frac{\mathcal{D}_{0} \cdot}{\Delta x} u_i^n \\
\textrm{Stability Criteria :}\quad & \Delta t \le \frac{\Delta x}{|c|} \\
\textrm{Order of Accuracy :}\quad & \mathcal{O}\left(\Delta t, \frac{\Delta x^2 } { \Delta t}, \Delta x^2\right)
\end{align}
$$

The Lax algorithm is an *inconsistent* difference equation because the truncation error is not gaurenteed to go to zero. It only does so if $\Delta x^2$ goes to zero faster than $\Delta t$.

## Lax-Wendroff

$$
\begin{align}
\textrm{Method :}\quad & u^{n+1}_i = u^n_i - c \Delta t \frac{\mathcal{D}_{0} \cdot}{\Delta x} u_i^n + \frac{1}{2}c^2 \Delta t^2 \frac{\mathcal{D}_{+} \cdot}{\Delta x} \frac{\mathcal{D}_{-} \cdot}{\Delta x} u_i^n \\
\textrm{Stability Criteria :}\quad & \Delta t \le \frac{\Delta x}{|c|} \\
\textrm{Order of Accuracy :}\quad & \mathcal{O}\left(\Delta t^2, \Delta x^2\right)
\end{align}
$$

## MacCormack

$$
\begin{align}
\textrm{Method :}\quad & u^{\overline{n+1}}_i = u^n_i - c \Delta t \frac{\mathcal{D}_{+} \cdot}{\Delta x} u_i^n \\
                       & u^{n+1}_i = \frac{1}{2} \left[u^n_i + u^{\overline{n+1}}_i - c \Delta t \frac{\mathcal{D}_{-} \cdot}{\Delta x} u^{\overline{n+1}}_i \right] \\
\textrm{Stability Criteria :}\quad & \Delta t \le \frac{\Delta x}{|c|} \\
\textrm{Order of Accuracy :}\quad & \mathcal{O}\left(\Delta t^2, \Delta x^2\right)
\end{align}
$$

## Jameson

$$
\begin{align}
\textrm{Method :}\quad & u^{(0)} = u^n_i \\
                       & u^{(k)} = u^n_i - \alpha_k c \Delta t \frac{\mathcal{D}_{0} \cdot}{\Delta x} u_i^{(k-1)} \\
                       & \qquad \textrm{where} \,\, \alpha_k = \frac{1}{5 - k}, \,\, k = 1, 2, 3, 4 \\
                       & u^{n+1}_i = u^{(4)}_i \\
\textrm{Stability Criteria :}\quad & \Delta t \le \frac{2 \sqrt{2} \Delta x}{|c|} \\
\textrm{Order of Accuracy :}\quad & \mathcal{O}\left(\Delta t^4, \Delta x^2\right)
\end{align}
$$

## Warming-Beam

$$
\begin{align}
\textrm{Method :}\quad & u^{n + 1/2}_i = u^n_i - \frac{ c \Delta t }{2} \frac{\mathcal{D}_{-} \cdot}{\Delta x} u_i^n \\
                       & u^{n+1}_i = u^n_i - c \Delta t \frac{\mathcal{D}_{-} \cdot}{\Delta x}
                       \left[ u^{n+1/2}_i + \frac{\Delta x}{2} \frac{\mathcal{D}_{-} \cdot}{\Delta x} u^n_i \right] \\
\textrm{Stability Criteria :}\quad & \Delta t \le \frac{2 \Delta x}{|c|} \\
\textrm{Order of Accuracy :}\quad & \mathcal{O}\left(\Delta t^2, \Delta x^2\right)
\end{align}
$$

## More difference equations

Let us look more closely at the Lax-Wendroff algorithm. We see the term like the following

$$
\frac{\mathcal{D}_{+} \cdot}{\Delta x} \frac{\mathcal{D}_{-} \cdot}{\Delta x} u_i^n
$$

what does mean? How do we apply two difference approximations? *This operators can be linearly combined,* so we just need to apply the first difference approximation, and then apply the second difference approximation on each of the remaing terms. Here is what that looks like for the above expression.

$$
\frac{\mathcal{D}_{+} \cdot}{\Delta x} \left[ \frac{\mathcal{D}_{-} \cdot}{\Delta x} u_i^n \right] =
\frac{\mathcal{D}_{+} \cdot}{\Delta x} \left[ \frac{u^n_i - u^n_{i-1}}{\Delta x} \right] =
\frac{1}{\Delta x} \left[ \frac{\mathcal{D}_{+} \cdot}{\Delta x}\Big( u^n_i \Big) - \frac{\mathcal{D}_{+} \cdot}{\Delta x} \Big( u^n_{i-1} \Big) \right]
$$

Now applying the second difference operator, results in

$$
\frac{\mathcal{D}_{+} \cdot}{\Delta x} \frac{\mathcal{D}_{-} \cdot}{\Delta x} u_i^n =
\frac{1}{\Delta x} \left[ \frac{\mathcal{D}_{+} \cdot}{\Delta x}\Big( u^n_i \Big) - \frac{\mathcal{D}_{+} \cdot}{\Delta x} \Big( u^n_{i-1} \Big) \right] =
\frac{1}{\Delta x} \left[ \frac{u^n_{i+1} - u^n_i}{\Delta x} - \frac{u^n_{i} - u^n_{i-1}}{\Delta x} \right] =
\frac{u^n_{i+1} - 2 u^n_i + u^n_{i-1} }{\Delta x^2}
$$


# Simple Linear Regression with NumPy

In school, students are taught to draw lines like the following.

$$ y = 2 x + 1$$

They're taught to pick two values for $x$ and calculate the corresponding values for $y$ using the equation.
Then they draw a set of axes, plot the points, and then draw a line extending through the two dots on their axes.

```python
# Import matplotlib.
import matplotlib.pyplot as plt
```

```python
# Draw some axes.
plt.plot([-1, 10], [0, 0], 'k-')
plt.plot([0, 0], [-1, 10], 'k-')

# Plot the red, blue and green lines.
plt.plot([1, 1], [-1, 3], 'b:')
plt.plot([-1, 1], [3, 3], 'r:')

# Plot the two points (1,3) and (2,5).
plt.plot([1, 2], [3, 5], 'ko')
# Join them with an (extending) green lines.
plt.plot([-1, 10], [-1, 21], 'g-')

# Set some reasonable plot limits.
plt.xlim([-1, 10])
plt.ylim([-1, 10])

# Show the plot.
plt.show()
```

Simple linear regression is about the opposite problem - what if you have some points and are looking for the equation?
It's easy when the points are perfectly on a line already, but usually real-world data has some noise.
The data might still look roughly linear, but aren't exactly so.

***

## Example (contrived and simulated)

#### Scenario
Suppose you are trying to weigh your suitcase to avoid an airline's extra charges.
You don't have a weighing scales, but you do have a spring and some gym-style weights of masses 7KG, 14KG and 21KG.
You attach the spring to the wall hook, and mark where the bottom of it hangs.
You then hang the 7KG weight on the end and mark where the bottom of the spring is.
You repeat this with the 14KG weight and the 21KG weight.
Finally, you place your case hanging on the spring, and the spring hangs down halfway between the 7KG mark and the 14KG mark.
Is your case over the 10KG limit set by the airline?

#### Hypothesis
When you look at the marks on the wall, it seems that the 0KG, 7KG, 14KG and 21KG marks are evenly spaced.
You wonder if that means your case weighs 10.5KG.
That is, you wonder if there is a *linear* relationship between the distance the spring's hook is from its resting position, and the mass on the end of it.

#### Experiment
You decide to experiment.
You buy some new weights - a 1KG, a 2KG, a 3Kg, all the way up to 20KG.
You place them each in turn on the spring and measure the distance the spring moves from the resting position.
You tabulate the data and plot them.

#### Analysis
Here we'll import the Python libraries we need for or investigations below.

```python
# Make matplotlib show interactive plots in the notebook.
%matplotlib inline
```

```python
# numpy efficiently deals with numerical multi-dimensional arrays.
import numpy as np

# matplotlib is a plotting library, and pyplot is its easy-to-use module.
import matplotlib.pyplot as plt

# This just sets the default plot size to be bigger.
plt.rcParams['figure.figsize'] = (8, 6)
```

Ignore the next couple of lines where I fake up some data. I'll use the fact that I faked the data to explain some results later. Just pretend that w is an array containing the weight values and d are the corresponding distance measurements.

```python
w = np.arange(0.0, 21.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)
```

```python
# Let's have a look at w.
w
```

    array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
           13., 14., 15., 16., 17., 18., 19., 20.])

```python
# Let's have a look at d.
d
```

    array([  7.87555482,  21.59953558,  26.98707474,  23.32362417,
            28.61198227,  38.81904106,  40.87544464,  47.68660422,
            53.36700771,  52.8709924 ,  54.8670291 ,  67.09374109,
            72.57056963,  79.88756871,  83.51318718,  76.50407315,
            93.042945  ,  97.96230846,  94.0586369 , 100.01267012,
           109.05819117])

Let's have a look at the data from our experiment.

```python
# Create the plot.

plt.plot(w, d, 'k.')

# Set some properties for the plot.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')

# Show the plot.
plt.show()
```

#### Model
It looks like the data might indeed be linear.
The points don't exactly fit on a straight line, but they are not far off it.
We might put that down to some other factors, such as the air density, or errors, such as in our tape measure.
Then we can go ahead and see what would be the best line to fit the data.

#### Straight lines
All straight lines can be expressed in the form $y = mx + c$.
The number $m$ is the slope of the line.
The slope is how much $y$ increases by when $x$ is increased by 1.0.
The number $c$ is the y-intercept of the line.
It's the value of $y$ when $x$ is 0.

#### Fitting the model
To fit a straight line to the data, we just must pick values for $m$ and $c$.
These are called the parameters of the model, and we want to pick the best values possible for the parameters.
That is, the best parameter values *given* the data observed.
Below we show various lines plotted over the data, with different values for $m$ and $c$.

```python
# Plot w versus d with black dots.
plt.plot(w, d, 'k.', label="Data")

# Overlay some lines on the plot.
x = np.arange(0.0, 21.0, 1.0)
plt.plot(x, 5.0 * x + 10.0, 'r-', label=r"$5x + 10$")
plt.plot(x, 6.0 * x +  5.0, 'g-', label=r"$6x +  5$")
plt.plot(x, 5.0 * x + 15.0, 'b-', label=r"$5x + 15$")

# Add a legend.
plt.legend()

# Add axis labels.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')

# Show the plot.
plt.show()
```

#### Calculating the cost
You can see that each of these lines roughly fits the data.
Which one is best, and is there another line that is better than all three?
Is there a "best" line?

It depends how you define the word best.
Luckily, everyone seems to have settled on what the best means.
The best line is the one that minimises the following calculated value.

$$ \sum_i (y_i - mx_i - c)^2 $$

Here $(x_i, y_i)$ is the $i^{th}$ point in the data set and $\sum_i$ means to sum over all points.
The values of $m$ and $c$ are to be determined.
We usually denote the above as $Cost(m, c)$.

Where does the above calculation come from?
It's easy to explain the part in the brackets $(y_i - mx_i - c)$.
The corresponding value to $x_i$ in the dataset is $y_i$.
These are the measured values.
The value $m x_i + c$ is what the model says $y_i$ should have been.
The difference between  the value that was observed ($y_i$) and the value that the model gives ($m x_i + c$), is $y_i - mx_i - c$.

Why square that value?
Well note that the value could be positive or negative, and you sum over all of these values.
If we allow the values to be positive or negative, then the positive could cancel the negatives.
So, the natural thing to do is to take the absolute value $\mid y_i - m x_i - c \mid$.
Well it turns out that absolute values are a pain to deal with, and instead it was decided to just square the quantity instead, as the square of a number is always positive.
There are pros and cons to using the square instead of the absolute value, but the square is used.
This is usually called *least squares* fitting.

```python
# Calculate the cost of the lines above for the data above.
cost = lambda m,c: np.sum([(d[i] - m * w[i] - c)**2 for i in range(w.size)])

print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (5.0, 10.0, cost(5.0, 10.0)))
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (6.0,  5.0, cost(6.0,  5.0)))
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (5.0, 15.0, cost(5.0, 15.0)))
```

    Cost with m =  5.00 and c = 10.00:   364.91
    Cost with m =  6.00 and c =  5.00:  1911.26
    Cost with m =  5.00 and c = 15.00:   784.03

#### Minimising the cost
We want to calculate values of $m$ and $c$ that give the lowest value for the cost value above.
For our given data set we can plot the cost value/function.
Recall that the cost is:

$$ Cost(m, c) = \sum_i (y_i - mx_i - c)^2 $$

This is a function of two variables, $m$ and $c$, so a plot of it is three dimensional.
See the **Advanced** section below for the plot.

In the case of fitting a two-dimensional line to a few data points, we can easily calculate exactly the best values of $m$ and $c$.
Some of the details are discussed in the **Advanced** section, as they involve calculus, but the resulting code is straight-forward.
We first calculate the mean (average) values of our $x$ values and that of our $y$ values.
Then we subtract the mean of $x$ from each of the $x$ values, and the mean of $y$ from each of the $y$ values.
Then we take the *dot product* of the new $x$ values and the new $y$ values and divide it by the dot product of the new $x$ values with themselves.
That gives us $m$, and we use $m$ to calculate $c$.

Remember that in our dataset $x$ is called $w$ (for weight) and $y$ is called $d$ (for distance).
We calculate $m$ and $c$ below.

```python
# Calculate the best values for m and c.

# First calculate the means (a.k.a. averages) of w and d.
w_avg = np.mean(w)
d_avg = np.mean(d)

# Subtract means from w and d.
w_zero = w - w_avg
d_zero = d - d_avg

# The best m is found by the following calculation.
m = np.sum(w_zero * d_zero) / np.sum(w_zero * w_zero)
# Use m from above to calculate the best c.
c = d_avg - m * w_avg

print("m is %8.6f and c is %6.6f." % (m, c))
```

    m is 4.768029 and c is 12.823887.

Note that numpy has a function that will perform this calculation for us, called polyfit.
It can be used to fit lines in many dimensions.

```python
np.polyfit(w, d, 1)
```

    array([ 4.76802927, 12.82388744])

#### Best fit line
So, the best values for $m$ and $c$ given our data and using least squares fitting are about $4.95$ for $m$ and about $11.13$ for $c$.
We plot this line on top of the data below.

```python
# Plot the best fit line.
plt.plot(w, d, 'k.', label='Original data')
plt.plot(w, m * w + c, 'b-', label='Best fit line')

# Add axis labels and a legend.
plt.xlabel('Weight (KG)')
plt.ylabel('Distance (CM)')
plt.legend()

# Show the plot.
plt.show()
```

Note that the $Cost$ of the best $m$ and best $c$ is not zero in this case.

```python
print("Cost with m = %5.2f and c = %5.2f: %8.2f" % (m, c, cost(m, c)))
```

    Cost with m =  4.77 and c = 12.82:   318.14

### Summary
In this notebook we:
1. Investigated the data.
2. Picked a model.
3. Picked a cost function.
4. Estimated the model parameter values that minimised our cost function.

### Advanced
In the following sections we cover some of the more advanced concepts involved in fitting the line.

#### Simulating data
Earlier in the notebook we glossed over something important: we didn't actually do the weighing and measuring - we faked the data.
A better term for this is *simulation*, which is an important tool in research, especially when testing methods such as simple linear regression.

We ran the following two commands to do this:

```python
w = np.arange(0.0, 21.0, 1.0)
d = 5.0 * w + 10.0 + np.random.normal(0.0, 5.0, w.size)
```

The first command creates a numpy array containing all values between 1.0 and 21.0 (including 1.0 but not including 21.0) in steps of 1.0.

```python
 np.arange(0.0, 21.0, 1.0)
```

    array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12.,
           13., 14., 15., 16., 17., 18., 19., 20.])

The second command is more complex.
First it takes the values in the `w` array, multiplies each by 5.0 and then adds 10.0.

```python
5.0 * w + 10.0
```

    array([ 10.,  15.,  20.,  25.,  30.,  35.,  40.,  45.,  50.,  55.,  60.,
            65.,  70.,  75.,  80.,  85.,  90.,  95., 100., 105., 110.])

It then adds an array of the same length containing random values.
The values are taken from what is called the normal distribution with mean 0.0 and standard deviation 5.0.

```python
np.random.normal(0.0, 5.0, w.size)
```

    array([ 2.00118422,  8.76268533,  7.24193421, -2.89217859,  4.38408328,
            6.27869981, -6.30259698,  0.40051845,  3.62034207,  4.924415  ,
            1.57441724,  4.17091877, -0.3527485 ,  4.48634788, -4.92792773,
           -0.46885963, -4.57679121,  2.23673805, -2.89051011,  4.45334771,
            8.2654568 ])

The normal distribution follows a bell shaped curve.
The curve is centred on the mean (0.0 in this case) and its general width is determined by the standard deviation (5.0 in this case).

```python
# Plot the normal distrution.
normpdf = lambda mu, s, x: (1.0 / (2.0 * np.pi * s**2)) * np.exp(-((x - mu)**2)/(2 * s**2))

x = np.linspace(-20.0, 20.0, 100)
y = normpdf(0.0, 5.0, x)
plt.plot(x, y)

plt.show()
```

The idea here is to add a little bit of randomness to the measurements of the distance.
The random values are entered around 0.0, with a greater than 99% chance they're within the range -15.0 to 15.0.
The normal distribution is used because of the [Central Limit Theorem](https://en.wikipedia.org/wiki/Central_limit_theorem) which basically states that when a bunch of random effects happen together the outcome looks roughly like the normal distribution. (Don't quote me on that!)

#### Plotting the cost function
We can plot the cost function for a given set of data points.
Recall that the cost function involves two variables: $m$ and $c$, and that it looks like this:

$$ Cost(m,c) = \sum_i (y_i - mx_i - c)^2 $$

To plot a function of two variables we need a 3D plot.
It can be difficult to get the viewing angle right in 3D plots, but below you can just about make out that there is a low point on the graph around the $(m, c) = (\approx 5.0, \approx 10.0)$ point.

```python
# This code is a little bit involved - don't worry about it.
# Just look at the plot below.

from mpl_toolkits.mplot3d import Axes3D

# Ask pyplot a 3D set of axes.
ax = plt.figure().gca(projection='3d')

# Make data.
mvals = np.linspace(4.5, 5.5, 100)
cvals = np.linspace(0.0, 20.0, 100)

# Fill the grid.
mvals, cvals = np.meshgrid(mvals, cvals)

# Flatten the meshes for convenience.
mflat = np.ravel(mvals)
cflat = np.ravel(cvals)

# Calculate the cost of each point on the grid.
C = [np.sum([(d[i] - m * w[i] - c)**2 for i in range(w.size)]) for m, c in zip(mflat, cflat)]
C = np.array(C).reshape(mvals.shape)

# Plot the surface.
surf = ax.plot_surface(mvals, cvals, C)

# Set the axis labels.
ax.set_xlabel('$m$', fontsize=16)
ax.set_ylabel('$c$', fontsize=16)
ax.set_zlabel('$Cost$', fontsize=16)

# Show the plot.
plt.show()
```

#### Coefficient of determination
Earlier we used a cost function to determine the best line to fit the data.
Usually the data do not perfectly fit on the best fit line, and so the cost is greater than 0.
A quantity closely related to the cost is the *coefficient of determination*, also known as the *R-squared* value.
The purpose of the R-squared value is to measure how much of the variance in $y$ is determined by $x$.

For instance, in our example the main thing that affects the distance the spring is hanging down is the weight on the end.
It's not the only thing that affects it though.
The room temperature and density of the air at the time of measurment probably affect it a little.
The age of the spring, and how many times it has been stretched previously probably also have a small affect.
There are probably lots of unknown factors affecting the measurment.

The R-squared value estimates how much of the changes in the $y$ value is due to the changes in the $x$ value compared to all of the other factors affecting the $y$ value.
It is calculated as follows:

$$ R^2 = 1 - \frac{\sum_i (y_i - m x_i - c)^2}{\sum_i (y_i - \bar{y})^2} $$

Note that sometimes the [*Pearson correlation coefficient*](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used instead of the R-squared value.
You can just square the Pearson coefficient to get the R-squred value.

```python
# Calculate the R-squared value for our data set.
rsq = 1.0 - (np.sum((d - m * w - c)**2)/np.sum((d - d_avg)**2))

print("The R-squared value is %6.4f" % rsq)
```

    The R-squared value is 0.9822

```python
# The same value using numpy.
np.corrcoef(w, d)[0][1]**2
```

    0.9821506875952921

#### The minimisation calculations
Earlier we used the following calculation to calculate $m$ and $c$ for the line of best fit.
The code was:

```python
w_zero = w - np.mean(w)
d_zero = d - np.mean(d)

m = np.sum(w_zero * d_zero) / np.sum(w_zero * w_zero)
c = np.mean(d) - m * np.mean(w)
```

In mathematical notation we write this as:

$$ m = \frac{\sum_i (x_i - \bar{x}) (y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \qquad \textrm{and} \qquad c = \bar{y} - m \bar{x} $$

where $\bar{x}$ is the mean of $x$ and $\bar{y}$ that of $y$.

Where did these equations come from?
They were derived using calculus.
We'll give a brief overview of it here, but feel free to gloss over this section if it's not for you.
If you can understand the first part, where we calculate the partial derivatives, then great!

The calculations look complex, but if you know basic differentiation, including the chain rule, you can easily derive them.
First, we differentiate the cost function with respect to $m$ while treating $c$ as a constant, called a partial derivative.
We write this as $\frac{\partial m}{ \partial Cost}$, using $\delta$ as opposed to $d$ to signify that we are treating the other variable as a constant.
We then do the same with respect to $c$ while treating $m$ as a constant.
We set both equal to zero, and then solve them as two simultaneous equations in two variables.

###### Calculate the partial derivatives
$$
\begin{align}
Cost(m, c)                       &= \sum_i (y_i - mx_i - c)^2      \\[1cm]
\frac{\partial Cost}{\partial m} &= \sum 2(y_i - m x_i -c)(-x_i) \\
                                 &= -2 \sum x_i (y_i - m x_i -c) \\[0.5cm]
\frac{\partial Cost}{\partial c} & = \sum 2(y_i - m x_i -c)(-1) \\
                                 & = -2 \sum (y_i - m x_i -c)   \\
\end{align}
$$

###### Set to zero
$$
\begin{align}
& \frac{\partial Cost}{\partial m}                         = 0 \\[0.2cm]
& \Rightarrow -2 \sum x_i (y_i - m x_i -c)                 = 0 \\
& \Rightarrow \sum (x_i y_i - m x_i x_i - x_i c)           = 0 \\
& \Rightarrow \sum x_i y_i - \sum_i m x_i x_i - \sum x_i c = 0 \\
& \Rightarrow m \sum x_i x_i                               = \sum x_i y_i - c \sum x_i \\[0.2cm]
& \Rightarrow m                                            = \frac{\sum x_i y_i - c \sum x_i}{\sum x_i x_i} \\[0.5cm]
& \frac{\partial Cost}{\partial c}               = 0 \\[0.2cm]
& \Rightarrow -2 \sum (y_i - m x_i - c)          = 0 \\
& \Rightarrow \sum y_i - \sum_i m x_i - \sum c   = 0 \\
& \Rightarrow \sum y_i - m \sum_i x_i            = c \sum 1 \\
& \Rightarrow c = \frac{\sum y_i - m \sum x_i}{\sum 1} \\
& \Rightarrow c = \frac{\sum y_i}{\sum 1} - m \frac{\sum x_i}{\sum 1} \\[0.2cm]
& \Rightarrow c = \bar{y} - m \bar{x} \\
\end{align}
$$

###### Solve the simultaneous equations
Here we let $n$ be the length of $x$, which is also the length of $y$.

$$
\begin{align}
& m = \frac{\sum_i x_i y_i - c \sum_i x_i}{\sum_i x_i x_i} \\[0.2cm]
& \Rightarrow m = \frac{\sum x_i y_i - (\bar{y} - m \bar{x}) \sum x_i}{\sum x_i x_i} \\
& \Rightarrow m \sum x_i x_i = \sum x_i y_i - \bar{y} \sum x_i + m \bar{x} \sum x_i \\
& \Rightarrow m \sum x_i x_i - m \bar{x} \sum x_i = \sum x_i y_i - \bar{y} \sum x_i \\[0.3cm]
& \Rightarrow m = \frac{\sum x_i y_i - \bar{y} \sum x_i}{\sum x_i x_i - \bar{x} \sum x_i} \\[0.2cm]
& \Rightarrow m = \frac{\sum (x_i y_i) - n \bar{y} \bar{x}}{\sum (x_i x_i) - n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum (x_i y_i) - n \bar{y} \bar{x} - n \bar{y} \bar{x} + n \bar{y} \bar{x}}{\sum (x_i x_i) - n \bar{x} \bar{x} - n \bar{x} \bar{x} + n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum (x_i y_i) - \sum y_i \bar{x} - \sum \bar{y} x_i + n \bar{y} \bar{x}}{\sum (x_i x_i) - \sum x_i \bar{x} - \sum \bar{x} x_i + n \bar{x} \bar{x}} \\
& \Rightarrow m = \frac{\sum_i (x_i - \bar{x}) (y_i - \bar{y})}{\sum_i (x_i - \bar{x})^2} \\
\end{align}
$$

<br>

#### Using sklearn neural networks

***

```python
import sklearn.neural_network as sknn

# Expects a 2D array of inputs.
w2d = w.reshape(-1, 1)

# Train the neural network.
regr = sknn.MLPRegressor(max_iter=10000).fit(w2d, d)

# Show the predictions.
np.array([d, regr.predict(w2d)]).T
```

    array([[  7.87555482,   8.17960683],
           [ 21.59953558,  18.50355393],
           [ 26.98707474,  23.1972181 ],
           [ 23.32362417,  27.89146122],
           [ 28.61198227,  32.58570433],
           [ 38.81904106,  37.27994745],
           [ 40.87544464,  41.97419056],
           [ 47.68660422,  46.66843368],
           [ 53.36700771,  51.36216639],
           [ 52.8709924 ,  56.05536619],
           [ 54.8670291 ,  60.74856598],
           [ 67.09374109,  65.44176577],
           [ 72.57056963,  70.13496556],
           [ 79.88756871,  74.82816534],
           [ 83.51318718,  79.52119914],
           [ 76.50407315,  84.21423294],
           [ 93.042945  ,  88.90726673],
           [ 97.96230846,  93.60030053],
           [ 94.0586369 ,  98.29333432],
           [100.01267012, 102.99082487],
           [109.05819117, 107.97221093]])

```python
# The score.
regr.score(w2d, d)
```

    0.9838518616697404

#### End


#Solving circuits with sympy

This documents explains how to solve electrical circuits using the **sympy** symbolic math module.

# Imports

First we need to import the sympy module

```python
# Import the sympy module
import sympy
```

## Example DC circuit

The following circuit includes one voltage source, one current source and two resistors.

The objective is to obtain the output voltage **Vo** as function of the components.

The circuit will be solved using the **nodal** method.

First we need to locate the circuit **nodes**, assign one as **ground** and assign a number for the rest of them. As the circuit has three nodes and one is ground, we have two nodes left: 1 and 2.

We will first generate a set of sympy symbols. There will be:

* One symbol for each component: Vs, R1, R2, Is

* One current symbol for each power sypply: iVs

* One symbol for each measurement we want to obtain: Vo

* One symbol for each node voltage that is not ground: V1, V2

```python
# Create the circuit symbols
Vs,iVs,R1,R2,Is,Vo, V1, V2 = sympy.symbols('Vs,iVs,R1,R2,Is,Vo,V1,V2')
```

Then we can define the current equations on each node except ground.

The current equations add all the currents in the node from each component.

All equations we add, are supposed to have a result of **zero**.

```python
# Create an empty list of equations
equations = []

# Nodal equations
equations.append(iVs-(V1-V2)/R1)       # Node 1
equations.append(Is-(V2-V1)/R1-V2/R2)  # Node 2
```

Then we add one equation for each voltage source that associates its voltage with the node voltages.

If we want to use two sided equations, we can use the sympy **Eq** function that equates the two sides.

```python
# Voltage source equations
equations.append(sympy.Eq(Vs,V1))
```

Finally we add one equation for each measurement we want to obtain

```python
# Measurement equations
equations.append(sympy.Eq(Vo,V2))
```

Now we can define the unknows for the circuit.

The number of unknowns shall be equal to the number of equations.

The list includes:

* The node voltages: V1, V2

* The current on voltage sources: iVs

* The measurement values: Vo

```python
unknowns = [V1,V2,iVs,Vo]
```

We can see the equations and unknows before solving the circuit.

To ease reusing the code, we will define a **showCircuit** function that shows equations and unknowns

```python
# Define the function
def showCircuit():
    print('Equations')
    for eq in equations:
        print('    ',eq)
    print()
    print('Unknowns:',unknowns)
    print()

# Use the function
showCircuit()
```

    Equations
         iVs - (V1 - V2)/R1
         Is - V2/R2 - (-V1 + V2)/R1
         Eq(Vs, V1)
         Eq(Vo, V2)

    Unknowns: [V1, V2, iVs, Vo]

Now, we can solve the circuit.

The sympy **solve** function gets a list of equations and unknowns and return a **dictionary** with solved unknowns

The following code solves the circuit and list the solutions

```python
# Solve the circuit
solution = sympy.solve(equations,unknowns)

# List the solutions
print('Solutions')
for sol in solution:
    print('  ',sol,'=',solution[sol])
```

    Solutions
       V1 = Vs
       V2 = R2*(Is*R1 + Vs)/(R1 + R2)
       iVs = (-Is*R2 + Vs)/(R1 + R2)
       Vo = R2*(Is*R1 + Vs)/(R1 + R2)

Note that, in this case, the equation that includes **iVs** is only needed to obtain this unknown, so we can eliminate its equation and the **iVs** unknown if we don't need the **iVs** solution.

Note also that we can easily identify **V1** as **Vs**, so we can also eliminate the **Vs** equation if we use **Vs** as the voltage on node **V1**

Finally note that we can also identify **V2** as **Vo** so we can also eliminate the **Vo** equation.

The following code solves the circuit using only one equations

```python
solution = sympy.solve(Is-(Vo-Vs)/R1-Vo/R2,Vo)
print('Vo =',solution[0])
```

    Vo = R2*(Is*R1 + Vs)/(R1 + R2)

## Solve using the loop current method

Instead of using the **nodal** method, we could also use the complementary **loop current** method

In this method we assign a current to each loop in the circuit

We will first generate a set of sympy symbols. There will be:

* One symbol for each component: Vs, R1, R2, Is

* One voltage symbol for each current sypply: Vo

* One symbol for each measurement we want to obtain: Vo

* One symbol for each loop current: I1, I2

Note that, in this circuit, **Vo** appears two times, as the voltage on the **Is** source and as the value to measure. Logically we only define one **Vo** symbol.

```python
# Create the circuit symbols
Vs,R1,R2,Is,Vo,I1,I2 = sympy.symbols('Vs,R1,R2,Is,Vo,I1,I2')
```

Then we create a list of equations and add one equation for each loop that adds all voltages on the loop

```python
# New list of equations
equations = []

# Loop current equations
equations.append(Vs-R1*I1-R2*(I1-I2)) # Loop current 1
equations.append(-R2*(I2-I1)-Vo)      # Loop current 2
```

Then we create one equation for each current supply that relates it to the loop currents

```python
# Current source equations
equations.append(sympy.Eq(Is,-I2))   # Current source Is
```

Now we can define the unknows for the circuit.

The number of unknowns shall be equal to the number of equations.

The list includes:

* The loop currents: I1, I2

* The voltage on current sources: Vo

* The measurement values: Vo

```python
# Unknowns list
unknowns = [I1,I2,Vo]
```

We can see the equations and unknows before solving the circuit

```python
showCircuit()
```

    Equations
         -I1*R1 - R2*(I1 - I2) + Vs
         -R2*(-I1 + I2) - Vo
         Eq(Is, -I2)

    Unknowns: [I1, I2, Vo]

Now we can obtain the solution

```python
# Obtain solution
solution = sympy.solve(equations,unknowns)

print('Vo =',solution[Vo])
```

    Vo = R2*(Is*R1 + Vs)/(R1 + R2)

As in the **nodal** case, you could have used less equations. For instance, you could have used the **Is** current for the second loop.

```python
# Create the circuit symbols
Vs,R1,R2,Is,Vo,I1 = sympy.symbols('Vs,R1,R2,Is,Vo,I1')

# New list of equations
equations = []

# Loop current equations
equations.append(Vs-R1*I1-R2*(I1+Is)) # Loop current 1
equations.append(R2*(Is+I1)-Vo)       # Loop current 2

# Unknowns list
unknowns = [I1,Vo]

# Show equations and unknowns
showCircuit()

# Obtain solution
solution = sympy.solve(equations,unknowns)

print('Vo =',solution[Vo])
```

    Equations
         -I1*R1 - R2*(I1 + Is) + Vs
         R2*(I1 + Is) - Vo

    Unknowns: [I1, Vo]

    Vo = R2*(Is*R1 + Vs)/(R1 + R2)

## Circuit with current measurements

The following example is special because the circuit has current measurements **I1** and **I2** that we want to obtain

The circuit can be solved using four different methods

### Method #1 : Use nodal method and get currents from resistors

In this method we will just use the normal nodal methods and we will compute the currents using Ohm's law

Note that we don't need the current on **Vs** so there is no point in obtaining the equation on node 1

```python
# Symbols for the circuit
Vs,R1,R2,R3,V2,I1,I2 = sympy.symbols('Vs,R1,R2,R3,V2,I1,I2')

# Nodal equation only on node 2
equations = []
equations.append(-(V2-Vs)/R1-V2/R2-V2/R3)

# Equations for the currents using Ohm's law
equations.append(sympy.Eq(I1,V2/R2))
equations.append(sympy.Eq(I2,V2/R3))

# Unknowns
unknowns = [V2,I1,I2]

# Show equations and unknowns
showCircuit()

# Solve the circuit
solution = sympy.solve(equations,unknowns)

print('I1 =',solution[I1])
print('I2 =',solution[I2])
```

    Equations
         -V2/R3 - V2/R2 + (-V2 + Vs)/R1
         Eq(I1, V2/R2)
         Eq(I2, V2/R3)

    Unknowns: [V2, I1, I2]

    I1 = R3*Vs/(R1*R2 + R1*R3 + R2*R3)
    I2 = R2*Vs/(R1*R2 + R1*R3 + R2*R3)

### Method #2 : Use four nodes plus ground

We can split the node 2 in three nodes: 2, 3 and 4

That way we can use the current equations to obtain I1 and I2

As all three nodes have the same voltage, we can set them equal using two equations

As in the previous case, we don't need the equation in node 1

```python
# Symbols for the circuit
Vs,R1,R2,R3,V2,V3,V4,I1,I2 = sympy.symbols('Vs,R1,R2,R3,V2,V3,V4,I1,I2')

# Node equations
equations = []
equations.append(-(V2-Vs)/R1-I1-I2)  # Node 2
equations.append(I1-V3/R2)           # Node 3
equations.append(I2-V4/R3)           # Node 4

# In fact, nodes 2, 3 and 4 are the same
equations.append(sympy.Eq(V2,V3))
equations.append(sympy.Eq(V2,V4))

# Unknowns
unknowns = [V2,V3,V4,I1,I2]

# Show equations and unknowns
showCircuit()

# Solve the circuit
solution = sympy.solve(equations,unknowns)

print('I1 =',solution[I1])
print('I2 =',solution[I2])
```

    Equations
         -I1 - I2 + (-V2 + Vs)/R1
         I1 - V3/R2
         I2 - V4/R3
         Eq(V2, V3)
         Eq(V2, V4)

    Unknowns: [V2, V3, V4, I1, I2]

    I1 = R3*Vs/(R1*R2 + R1*R3 + R2*R3)
    I2 = R2*Vs/(R1*R2 + R1*R3 + R2*R3)

### Method #3 : Use the loop current method

In this case we will define two loop currents

* Ia goes on the first loop:  Vs -> R1 -> R2
* I2 goes on the second loop: R2 -> R3

```python
# Symbols for the circuit
Vs,R1,R2,R3,Ia,I1,I2 = sympy.symbols('Vs,R1,R2,R3,Ia,I1,I2')

# Loop equations
equations = []
equations.append(Vs-R1*Ia-R2*(Ia-I2))  # Loop Ia
equations.append(-R2*(I2-Ia)-R3*I2)     # Loop I2

# Define I1 from loop currents
equations.append(sympy.Eq(I1,Ia-I2))

# Unknowns
unknowns = [Ia,I1,I2]

# Show equations and unknowns
showCircuit()

# Solve the circuit
solution = sympy.solve(equations,unknowns)

print('I1 =',solution[I1])
print('I2 =',solution[I2])
```

    Equations
         -Ia*R1 - R2*(-I2 + Ia) + Vs
         -I2*R3 - R2*(I2 - Ia)
         Eq(I1, -I2 + Ia)

    Unknowns: [Ia, I1, I2]

    I1 = R3*Vs/(R1*R2 + R1*R3 + R2*R3)
    I2 = R2*Vs/(R1*R2 + R1*R3 + R2*R3)

### Method #4 : Use a modified loop current method

In this method we will use **I1** and **I2** as loop currents

* I1 goes Vs -> R1 -> R2
* I2 goes Vs -> R1 -> R3

```python
# Symbols for the circuit
Vs,R1,R2,R3,I1,I2 = sympy.symbols('Vs,R1,R2,R3,I1,I2')

# Loop equations
equations = []
equations.append(Vs-R1*(I1+I2)-R2*I1)  # Loop I1
equations.append(Vs-R1*(I1+I2)-R3*I2)  # Loop I2

# Unknowns
unknowns = [I1,I2]

# Show equations and unknowns
showCircuit()

# Solve the circuit
solution = sympy.solve(equations,unknowns)

print('I1 =',solution[I1])
print('I2 =',solution[I2])
```

    Equations
         -I1*R2 - R1*(I1 + I2) + Vs
         -I2*R3 - R1*(I1 + I2) + Vs

    Unknowns: [I1, I2]

    I1 = R3*Vs/(R1*R2 + R1*R3 + R2*R3)
    I2 = R2*Vs/(R1*R2 + R1*R3 + R2*R3)

## Controlled voltage source

The following circuit includes voltage controlled source

In this case we will treat the controlled voltage source as an independent voltage source, but, we will use $k \cdot V_m$ as its value.

That means that we will need to add an equation to add $V_m$ to the set of symbols

The following code defines and solves the circuit using the loop current method.

```python
# Symbols for the circuit
Vs,R1,R2,k,R3,I1,I2,Vm = sympy.symbols('Vs,R1,R2,k,R3,I1,I2,Vm')

# Loop equations
equations = []
equations.append(Vs-I1*R1-I1*R2)  # Loop I1
equations.append(k*Vm-I2*R3)      # Loop I2

# Equations for Vm and Vo
equations.append(sympy.Eq(Vm,I1*R2))
equations.append(sympy.Eq(Vo,I2*R3))

# Unknowns
unknowns = [I1,I2,Vm,Vo]

# Show equations and unknowns
showCircuit()

# Solve the circuit
solution = sympy.solve(equations,unknowns)

print('Vo =',solution[Vo])
```

    Equations
         -I1*R1 - I1*R2 + Vs
         -I2*R3 + Vm*k
         Eq(Vm, I1*R2)
         Eq(Vo, I2*R3)

    Unknowns: [I1, I2, Vm, Vo]

    Vo = R2*Vs*k/(R1 + R2)

The circuit could also be solved using the nodal method.

Remember that we don't need the nodal equations for nodes on grounded supplies if we don't need the supply current

```python
# Symbols for the circuit
Vs,Vm,Vo,R1,R2,k = sympy.symbols('Vs,Vm,Vo,R1,R2,k')

# Node equation
equations = []
equations.append(-(Vm-Vs)/R1-Vm/R2)

# Equation for Vo
equations.append(sympy.Eq(Vo,k*Vm))

# Unknowns
unknowns = [Vm,Vo]

# Show equations and unknowns
showCircuit()

# Solve the circuit
solution = sympy.solve(equations,unknowns)

print('Vo =',solution[Vo])
```

    Equations
         -Vm/R2 + (-Vm + Vs)/R1
         Eq(Vo, Vm*k)

    Unknowns: [Vm, Vo]

    Vo = R2*Vs*k/(R1 + R2)

## RC Circuit

Now we can also solve circuits with capacitor or inductors

You just need to define the **capacitors** currents and voltages as function of the **s** variable:

$\qquad i_C = V_C \cdot C \cdot s \qquad v_C = \frac{i_C}{C \cdot s}$

Also, for **inductors**:

$\qquad i_L = \frac{V_L}{L \cdot s} \qquad v_L = i_L \cdot L \cdot s$

We will use the following circuit as an example

The following code describes and solves the circuit using the current loop method

```python
# Symbols for the circuit
Vs,R1,R3,C1,C2,Vo,I1,I2,s = sympy.symbols('Vs,R1,R3,C1,C2,Vo,I1,I2,s')

# Loop equations
equations = []
equations.append(Vs-R1*I1-(I1-I2)/(C1*s))
equations.append(-(I2-I1)/(C1*s)-R3*I2-I2/(C2*s))

# Equation for Vo
equations.append(sympy.Eq(Vo,I2/(C2*s)))

# Unknowns
unknowns = [I1,I2,Vo]

# Show equations and unknowns
showCircuit()

# Solve the circuit
solution = sympy.solve(equations,unknowns)

Vo_s = solution[Vo]
print('Solution')
print('   Vo =',Vo_s)
```

    Equations
         -I1*R1 + Vs - (I1 - I2)/(C1*s)
         -I2*R3 - I2/(C2*s) + (I1 - I2)/(C1*s)
         Eq(Vo, I2/(C2*s))

    Unknowns: [I1, I2, Vo]

    Solution
       Vo = -C1*Vs/(C2 - (C1*R1*s + 1)*(C1*C2*R3*s + C1 + C2))

We can obtain a better solution equation using **simplify**, **expand** and **collect** from the sympy module

```python
# Use simplify, expand and collect to give a prettier equation
Vo_s = Vo_s.simplify().expand()          # Eliminate quotients of quotients
Vo_s = sympy.collect(Vo_s,s).simplify()  # Group s symbols and simplify

print('Prettier solution')
print('   Vo =',Vo_s)
```

    Prettier solution
       Vo = Vs/(C1*C2*R1*R3*s**2 + s*(C1*R1 + C2*R1 + C2*R3) + 1)

We can obtian a particular solution using numbers to substitute the literals

To substitute one symbol you can use:

>`expr.subs(oldSymbol,newSymbol)`

But, in order to substitute several symbols at once you can use a substitution dictionary:

>`expr.subs({old1:new1,old2:new2,....})`

Substituting in our example we get a **particular** solution

```python
H_s = Vo_s.subs({Vs:1,R1:1000,R3:100,C1:1e-6,C2:100e-9})
H_s = H_s.simplify()

print('Particular solution')
print('    H(s) = Vo(s)/Vs(s) =',H_s)
```

    Particular solution
        H(s) = Vo(s)/Vs(s) = 1/(1.0e-8*s**2 + 0.00111*s + 1)

We can also get the **poles**, the **zeros** and the **DC gain**

```python
numer,denom =H_s.as_numer_denom()
print('Num =',numer)
print('Den =',denom)
print()

zeros = sympy.roots(numer,s)
poles = sympy.roots(denom,s)
print('Zeros =',zeros)
print('Poles =',poles)
print()

print('DC gain =',H_s.subs(s,0).evalf())
```

    Num = 1
    Den = 1.0e-8*s**2 + 0.00111*s + 1

    Zeros = {}
    Poles = {-110091.666030631: 1, -908.333969368548: 1}

    DC gain = 1.00000000000000

## Opamp circuit

We can also solve operational amplifier circuits

The easiest solution is obtained if we can guarantee that the **virtual shortcircuit** holds

In this case, the opamp output is an unknown and the two input voltages are equal:

$\qquad V_{(+)}=V_{(-)}$

We will use the nodal method.

Remember that we don't need the node equations in nodes 1 and 3 because we don't need the source currents

```python
# Symbols for the circuit
Vs,Ri,Rf,Vo,V2 = sympy.symbols('Vs,Ri,Rf,Vo,V2')

# Node equation
equations = []
equations.append(-(V2-Vs)/Ri-(V2-Vo)/Rf)

# Virtual shortcircuit
equations.append(V2)  # V2 = V(+) = 0

# Unknowns
unknowns = [V2,Vo]

# Show equations and unknowns
showCircuit()

# Solve the circuit
solution = sympy.solve(equations,unknowns)

print('Vo =',solution[Vo])
```

    Equations
         (-V2 + Vs)/Ri - (V2 - Vo)/Rf
         V2

    Unknowns: [V2, Vo]

    Vo = -Rf*Vs/Ri

### Finite gain solution

If we don't want to use the **virtual short circuit** we can solve the opamp as a voltage controlled voltage source

In this case the opamp can be defined with two equations:

$\qquad V_d = V_{(+)}-V_{(-)}$

$\qquad  V_O = A \cdot V_d $

Then, the ideal case will be given when $A \rightarrow \infty$

```python
# Symbols for the circuit
Vs,Ri,Rf,Vo,V2,Vd,A = sympy.symbols('Vs,Ri,Rf,Vo,V2,Vd,A')

# Node equation
equations = []
equations.append(-(V2-Vs)/Ri-(V2-Vo)/Rf)

# Opamp equations
equations.append(sympy.Eq(Vd,0-V2))
equations.append(sympy.Eq(Vo,A*Vd))

# Unknowns
unknowns = [V2,Vo,Vd]

# Show equations and unknowns
showCircuit()

# Solve the circuit
solution = sympy.solve(equations,unknowns)

print('Solution as function of A')
print()
print('   Vo =',solution[Vo])
print()
print('Solution for A -> oo')
print()
print('   Vo =',sympy.limit(solution[Vo],A,sympy.oo))
```

    Equations
         (-V2 + Vs)/Ri - (V2 - Vo)/Rf
         Eq(Vd, -V2)
         Eq(Vo, A*Vd)

    Unknowns: [V2, Vo, Vd]

    Solution as function of A

       Vo = -A*Rf*Vs/(A*Ri + Rf + Ri)

    Solution for A -> oo

       Vo = -Rf*Vs/Ri

### Dominant pole solution

Having a solution as function of **A** enables us to obtain the response of the circuit using a dominant pole model for the operational amplifier.

Just susbtitute, in the solution **A** for the one pole model of the opamp

$\qquad A = \frac{Ao \cdot p1}{s+p1}$

```python
# New A, p1 and s symbols
Ao,p1,s = sympy.symbols('Ao,p1,s')

Vo_s = solution[Vo].subs(A,Ao*p1/(s+p1))

# Use simplify, expand and collect to give a prettier equation
Vo_s = Vo_s.simplify().expand()  # Eliminate quotients of quotients
Vo_s = sympy.collect(Vo_s,s)     # Group s symbols
Vo_s = sympy.collect(Vo_s,Ri)    # Group Ri symbols

print('Vo(s) =',Vo_s)
```

    Vo(s) = -Ao*Rf*Vs*p1/(Rf*p1 + Ri*(Ao*p1 + p1) + s*(Rf + Ri))

We can obtian, as in a previous example, a particular solution using numbers to substitute the literals

In our opamp circuit solution we substitute:

* Circuit resistors: R1, R2

* Opamp model: Ao, p1

We also substitute Vs for 1 to obtain the transfer function $H(s)$

$\qquad H(s)=\frac{V_O(s)}{V_s(s)}$

```python
H_s = Vo_s.subs({Vs:1,Ao:100000,Rf:100000,Ri:10000,p1:16})
print('H(s) =',H_s)
```

    H(s) = -160000000000/(110000*s + 16001760000)

Now you can also obtain the **poles** and **zeros** of $H(s)$

Also we can get the DC gain

We will use the **evalf()** method that evaluates a **sympy** expression to a floating point number

```python
numer,denom =H_s.as_numer_denom()
print('Num =',numer)
print('Den =',denom)
print()

zeros = sympy.roots(numer,s)
poles = sympy.roots(denom,s)
print('Zeros =',zeros)
print('Poles =',poles)
print()

print('DC gain =',H_s.subs(s,0).evalf())
```

    Num = -160000000000
    Den = 110000*s + 16001760000

    Zeros = {}
    Poles = {-1600176/11: 1}

    DC gain = -9.99890012098669

## Document information

Copyright © Vicente Jiménez (2018)

Last update: 26/3/2018

This work is licensed under a [Creative Common Attribution-ShareAlike 4.0 International license](http://creativecommons.org/licenses/by-sa/4.0/).

You can find the module [here](https://github.com/R6500/Python-bits/tree/master/Modules)

See my blogs [AIM65](http://aim65.blogspot.com.es/) (in spanish) and [R6500](http://r6500.blogspot.com.es/) (in english)

```python

```