 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

Note:  This material largely follows the text "Numerical Linear Algebra" by Trefethen and Bau (SIAM, 1997) and is meant as a guide and supplement to the material presented there.

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# QR Factorizations and Least Squares

## Projections

A **projector** is a square matrix $P$ that satisfies
$$
    P^2 = P.
$$

Why does this definition make sense?  Why do we require it to be square?

A projector comes from the idea that we want to project a vector $\mathbf{v}$ onto a lower dimensional subspace.  For example, suppose that $\mathbf{v}$ lies completely within the subspace, i.e. $\mathbf{v} \in \text{range}(P)$. If that is the case then $P \mathbf{v}$ should not change, or $P\mathbf{v} = \mathbf{v}$.  This motivates the definition above.

i.e.  if
$$
    P\mathbf{v} = \mathbf{v}
$$
then
$$
    P( P \mathbf{v} ) = P\mathbf{v} = \mathbf{v}.
$$
or $$P^2 = P$$

Note:
- Columns of $P$ serve as bases for $\mathbf{v}$?

As another example, take a vector $\mathbf{x} \notin \text{range}(P)$ and project it onto the subspace $P\mathbf{x} = \mathbf{v}$.  If we apply the projection again to $\mathbf{v}$ we now have

$$\begin{aligned}
    P\mathbf{x} &= \mathbf{v}, \\
    P^2 \mathbf{x} & = P \mathbf{v} = \mathbf{v} \\
    \Rightarrow P^2 &= P.
\end{aligned}$$

It is also important to keep in mind the following, given again $\mathbf{x} \notin \text{range}(P)$, if we look at the difference between the projection and the original vector $P\mathbf{x} - \mathbf{x}$ and apply the projection again we have
$$
    P(P\mathbf{x} - \mathbf{x}) = P^2 \mathbf{x} - P\mathbf{x} = 0
$$
which means the difference between the projected vector $P\mathbf{x} = \mathbf{v}$ lies in the null space of $P$, $\mathbf{v} \in \text{null}(P)$.

Note:
- $\mathbf{x}$ is in nullspace of $P$
- Therefore, $P\mathbf{x} = 0$

### Complementary Projectors

A projector also has a complement defined to be $I - P$.

Show that this complement is also a projector.

We can show that this a projector by examining a repeated application of $(I-P)$:
$$\begin{aligned}
    (I - P)^2 &= I - IP - IP + P^2 \\
    &= I - 2 P + P^2 \\
    &= I - 2P + P \\
    &= I - P.
\end{aligned}$$

It turns out that the complement projects exactly onto $\text{null}(P)$.

Take
$$
    \mathbf{x} \in \text{null}(P),
$$

then
$$
    (I - P) \mathbf{x} = \mathbf{x} - P \mathbf{x} = \mathbf{x}
$$

since $P \mathbf{x} = 0$ implying that $\mathbf{x} \in \text{range}(I - P)$.

We also know that
$$
    (I - P) \mathbf{x }\in \text{null}(P)
$$
as well.

This shows that the
$$
    \text{range}(I - P) \subseteq \text{null}(P)
$$
and
$$
    \text{range}(I - P) \supseteq \text{null}(P)
$$
implying that
$$
    \text{range}(I - P) = \text{null}(P)
$$
exactly.

Reflect on these subspaces and convince yourself that this all makes sense.

This result provides an important property of a projector and its complement, namely that they divide a space into two subspaces whose intersection is

$$
    \text{range}(I - P) \cap \text{range}(P) = \{0\}
$$

or

$$
    \text{null}(P) \cap \text{range}(P) = \{0\}
$$

These two spaces are called **complementary subspaces**.

Given this property we can take any $P \in \mathbb C^{m \times m}$ which will split $\mathbb C^{m \times m}$ into two subspaces $S$ and $V$, assume that $\mathbf{s}\in S = \text{range}(P)$, and $\mathbf{v} \in V = \text{null}(P)$.  If we have $\mathbf{x} \in \mathbb C^{m \times m}$ that we can split the vector $\mathbf{x}$ into components in $S$ and $V$ by using the projections
$$\begin{aligned}
    P \mathbf{x} = \mathbf{x}_S& &\mathbf{x}_s \in S \\
    (I - P) \mathbf{x} = \mathbf{x}_V& &\mathbf{x}_V \in V
\end{aligned}$$
which we can also observe adds to the original vector as
$$
    \mathbf{x}_S + \mathbf{x}_V = P \mathbf{x} + (I - P) \mathbf{x} = \mathbf{x}.
$$

Try constructing a projection matrix so that $P \in \mathbb R^3$ that projects a vector into one of the coordinate directions ($\mathbb R$).
 - What is the complementary projector?
 - What is the complementary subspace?

Note:
- Question is asking:
$$
P \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix} = \begin{bmatrix} x_1 \\ 0 \\ 0 \end{bmatrix}
$$

### Example:  A non-orthogonal non-linear projector

Given a vector of mols of $N$ chemical components

$$
\mathbf{n} = \begin{bmatrix} n_1 \\ n_2 \\ \vdots \\ n_N\end{bmatrix}
$$
where (e.g. $n_1$ is the number of moles of component $1$) and $n_i\geq 0$

Then we can define the mol fraction  of component $i$ as
$$
    x_i = \frac{n_i}{\mathbf{n}^T\mathbf{1}}
$$
and the "vector" of mole fractions
$$
\mathbf{x} = \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_N\end{bmatrix}
$$

In the homework you will show that

* $\mathbf{x}^T\mathbf{1} = 1$ (the sum of the mole fractions add to 1)
* mole fractions do not form a vector space or subspace
* There exists a non Orthogonal projector $f$ such that $f(\mathbf{n})=\mathbf{x}$, $f^2=f$
* $P$ is singular (like all projection matrices) such that if you know the mole-fractions you don't know how many moles you have.
* Find $N(P)$

### Orthogonal Projectors

An **orthogonal projector** is one that projects onto a subspace $S$ that is orthogonal to the complementary subspace $V$ (this is also phrased that $S$ projects along a space $V$).  Note that we are only talking about the subspaces (and their basis), not the projectors!

A **hermitian** matrix is one whose complex conjugate is itself, i.e.

$$
    P = P^\ast.
$$

With this definition we can then say:  *A projector $P$ is orthogonal if and only if $P$ is hermitian.*

**Quick Proof**

Show that if $P^2 = P$ and $P^\ast = P$, then
$$
\langle P\mathbf{x}, (I-P)\mathbf{x}\rangle=0
$$

### Projection with an Orthonormal Basis

We can also directly construct a projector that uses an orthonormal basis on the subspace $S$.  If we define another matrix $Q \in \mathbb C^{m \times n}$ which is unitary (its columns are orthonormal) we can construct an orthogonal projector as
$$
    P = Q Q^*.
$$
Note that the resulting matrix $P$ is in $\mathbb C^{m \times m}$ as we require.  This means also that the dimension of the subspace $S$ is $n$.

#### Example:  Orthonormal projection and Least-Squares Problems... A review

Consider the overdetermined problem $A\mathbf{x}=\mathbf{b}$ where $A\in\mathbb{R}^{3\times2}$ and $\mathbf{b}\in\mathbb{R}^3$ i.e.

$$
    \begin{bmatrix} | & | \\
                    \mathbf{a}_1 & \mathbf{a}_2\\
                     | & | \\
     \end{bmatrix} \begin{bmatrix} x_1 \\ x_2\\ \end{bmatrix}
         = \begin{bmatrix} |   \\
                    \mathbf{b} \\
                     |  \\
                     \end{bmatrix}
$$

and $\mathbf{a}_1$, $\mathbf{a}_2$ are linearly independent vectors that span a two-dimensional subspace of $\mathbb{R}^3$.

#### Geometry

Geometrically this problem looks like

Note:
- $C(A)$ is the columnspace/image of $A$
- Want to find $\hat{\mathbf{x}}$ such that $\left\Vert \mathbf{e} \right\Vert_2$ is minimized
- $\mathbf{a_1}^\top e = 0$ and $\mathbf{a_2}^\top e = 0$ (or $A^\top e = 0$), so $e$ is in left nullspace of $A$
$$
A^\top (\mathbf{b} - \mathbf{p}) = 0 \\
A^\top (\mathbf{b} - A\hat{\mathbf{x}}) = 0 \\
A^\top A \hat{\mathbf{x}} = A^\top \mathbf{b}
$$
- Resolves to normal equations:
$$
\begin{cases}
A\hat{\mathbf{x}} = \mathbf{b} \\
A\hat{\mathbf{x}} = \mathbf{p}
\end{cases}
$$
- If bases are orthogonal ($a_1 = q_1$ and $a_2 = q_2$ are orthogonal)
$$
\begin{align}
    P &= A (A^\top A)^{-1} A^\top \\
    & = Q (Q^\top Q)^{-1} Q^\top \\
    & = QQ^\top \\
    & = QQ^*
\end{align}
$$

If $\mathbf{b}\notin C(A)$, then there is clearly no solution to $A\mathbf{x}=\mathbf{b}$.  However, we can find the point $\mathbf{p}=A\hat{\mathbf{x}}\in C(A)$ that minimizes the length of the the error $\mathbf{e}=\mathbf{b}-\mathbf{p}$.  While we could resort to calculus to find the values of $\hat{\mathbf{x}}$ that minimizes $||\mathbf{e}||_2$.  It should be clear from the figure that the shortest error (in the $\ell_2$ norm) is the one that is perpendicular to every vector in $C(A)$.

But the sub-space of vectors orthogonal to $C(A)$ is the left-Null Space $N(A^T)$,  and therefore we simply seek solutions of

\begin{align}
    0 &= A^T\mathbf{e} \\
    &= A^T(\mathbf{b}-\mathbf{p})\\
    &= A^T(\mathbf{b} - A\hat{\mathbf{x}})\\
\end{align}

or we just need to solve the "Normal Equations"  $A^T A\hat{\mathbf{x}} = A^T\mathbf{b}$ for the least-squares solution
$$
    \hat{\mathbf{x}} = \left(A^T A\right)^{-1}A^T\mathbf{b}
$$

if we're actually interested in $\mathbf{p}$ which is the orthogonal projection of $\mathbf{b}$ onto $C(A)$ we get

$$
    \mathbf{p}= A\hat{\mathbf{x}} = A \left(A^T A\right)^{-1}A^T\mathbf{b} = P\mathbf{b}
$$
where
$$
    P = A \left(A^T A\right)^{-1}A^T
$$

is an orthogonal projection matrix (verify that $P^2 = P$ and $(I - P)\mathbf{b}\perp P\mathbf{b})$

For a general matrix $A$,  this form of the projection matrix is rather horrid to find,  however,  if the columns of $A$ formed an orthonormal basis for $C(A)$,  i.e. $A=Q$,  then the form of the projection matrix is much simpler as $Q^T Q=I$, therefore

$$
    P = QQ^T
$$

This is actually quite general. Given any orthonormal basis for a vector space $S=\mathrm{span}\langle \mathbf{q}_1,\mathbf{q}_2,\ldots,\mathbf{q}_N\rangle$.  If these vectors form the columns of $Q$,  then the orthogonal projector onto $S$ is always $QQ^T$ and the complement is always $I-QQ^T$.

**Example:  Construction of an orthonormal projector**

Take $\mathbb R^3$ and derive a projector that projects onto the x-y plane and is an orthogonal projector.

the simplest Orthonormal basis for the $x-y$ plane are the columns of
$$
    Q = \begin{bmatrix} 1 & 0 \\
                        0 & 1 \\
                        0 & 0
        \end{bmatrix}
$$
and an orthogonal projector onto the $x-y$ plane is simply
$$
    Q Q^\ast = \begin{bmatrix} 1 & 0 \\
                               0 & 1 \\
                               0 & 0
               \end{bmatrix}
               \begin{bmatrix} 1 & 0 & 0 \\
                               0 & 1 & 0
               \end{bmatrix} = \begin{bmatrix}
                               1 & 0 & 0 \\
                               0 & 1 & 0 \\
                               0 & 0 & 0
               \end{bmatrix}
$$

```python
Q = numpy.array([[1, 0],[0, 1],[0, 0]])
P = numpy.dot(Q, Q.T)
I = numpy.identity(3)

x = numpy.array([3, 4, 5])
x_S = numpy.dot(P, x)
x_V = numpy.dot(I - P, x)
print(x)
print(x_S)
print(x_V)
print(x_S + x_V)
```

    [3 4 5]
    [3 4 0]
    [0. 0. 5.]
    [3. 4. 5.]

Note:
- Above, $\mathbf{x_S} + \mathbf{x_V} = \mathbf{x}$
- $\mathbf{x_S}$ is projection of $\mathbf{x}$ on columnspace of $P$
- $\mathbf{x_V}$ is complementary projection
- Below approach is more numerically sensible, since matrix multiplication introduces more floating point error than matrix addition/subtraction (check?)

```python
# Numerically more sensible approach
Q = numpy.array([[1, 0],[0, 1],[0, 0]])

x = numpy.array([3, 4, 5])
x_S = Q.dot(Q.T.dot(x))
x_V = x - x_S
print(x)
print(x_S)
print(x_V)
print(x_S + x_V)
```

    [3 4 5]
    [3 4 0]
    [0 0 5]
    [3 4 5]

#### Example: Construction of a projector that eliminates a direction

Goal:  Eliminate the component of a vector in the direction $\mathbf{q}$.

Form the projector $P = \mathbf{q} \mathbf{q}^\ast \in \mathbb C^{m \times m}$.  The complement $I - P$ will then include everything **BUT** that direction.  If $||\mathbf{q}|| = 1$ we can then simply use $I - \mathbf{q} \mathbf{q}^\ast$.  If not we can write the projector in terms of the arbitrary vector $\mathbf{a}$ as
$$
    I - \frac{\mathbf{a} \mathbf{a}^\ast}{||\mathbf{a}||^2} = I - \frac{\mathbf{a} \mathbf{a}^\ast}{\mathbf{a}^\ast \mathbf{a}}
$$
Note that differences in the resulting dimensions between the two values in the fraction.  Also note that as we saw with the outer product, the resulting $\text{rank}(\mathbf{a} \mathbf{a}^\ast) = 1$.

Now again try to construct a projector in $\mathbb R^3$ that projects onto the $x$-$y$ plane.

```python
q = numpy.array([0, 0, 1])
P = numpy.outer(q, q.conjugate())
P_comp = numpy.identity(3) - P

x = numpy.array([3, 4, 5])
print(numpy.dot(P, x))
print(numpy.dot(P_comp, x))

a = numpy.array([0, 0, 3])
P = numpy.outer(a, a.conjugate()) / (numpy.dot(a, a.conjugate()))
P_comp = numpy.identity(3) - P
print(numpy.dot(P, x))
print(numpy.dot(P_comp, x))
```

    [0 0 5]
    [3. 4. 0.]
    [0. 0. 5.]
    [3. 4. 0.]

## Quick Review

* A projection matrix is any square matrix $P$ such that $P^2=P$
* $P$ projects onto a subspace $S=\mathrm{range}(P)$
* The complementary projection matrix $I-P$ projects onto $V=\mathrm{null}(P)$
* An *orthogonal* projector can always be constructed as $P=QQ^T$ where the columns of $Q$ form an *orthonormal* basis for $S$ and $S\perp V$
* Solutions of Least-squares problems $A\mathbf{x}=\mathbf{b}$ are essentially projection problems where we seek to solve $A\mathbf{x}=\mathbf{b}$ where $\mathbf{b}\notin C(A)$

### Solution of Least Squares problems by the Normal Equations

given $A\mathbf{x}=\mathbf{b}$ where $\mathbf{b}\notin C(A)$ we can always solve them using the Normal Equations

$$
A^T A\hat{\mathbf{x}} = A^T\mathbf{b}
$$

which actually solves $A\hat{\mathbf{x}} =\mathbf{p}$  where $\mathbf{p}$ is the orthogonal projection of $\mathbf{b}$ onto $C(A)$

* but there is a better way...

### Solution of Least Squares problems by the QR factorization

given any matrix $A$ that is full column rank, we will show that we can always factor it as

$$
    A=QR
$$

where $Q$ **is a unitary matrix** whose columns form an *orthonormal* basis for $C(A)$,  and $R$ **is an upper triangular matrix** that says how to reconstruct the columns of $A$ from the columns of $Q$

$$
    \begin{bmatrix}  &  &  \\  &  &  \\ \mathbf{a}_1 & \cdots & \mathbf{a}_n \\  &  &  \\  &  &  \end{bmatrix} =
    \begin{bmatrix}  &  &  \\  &  &  \\ \mathbf{q}_1 & \cdots & \mathbf{q}_n \\  &  &  \\  &  &  \end{bmatrix}
    \begin{bmatrix} r_{11} & r_{12} & \cdots & r_{1n} \\  & r_{22} &  &  \\  &  & \ddots & \vdots \\  &  &  & r_{nn} \end{bmatrix}.
$$
If we write this out as a matrix multiplication we have
$$\begin{aligned}
    \mathbf{a}_1 &= r_{11} \mathbf{q}_1 \\
    \mathbf{a}_2 &= r_{22} \mathbf{q}_2 + r_{12} \mathbf{q}_1 \\
    \mathbf{a}_3 &= r_{33} \mathbf{q}_3 + r_{23} \mathbf{q}_2 + r_{13} \mathbf{q}_1 \\
    &\vdots
\end{aligned}$$

given $A=QR$,  then solving

$$
    A\mathbf{x} = \mathbf{b}
$$

becomes

$$
    QR\mathbf{x} = \mathbf{b}
$$

or since $Q^T Q=I$

$$
    R\mathbf{x} = Q^T\mathbf{b}
$$

which can be **solve quickly by back-substitution as it is a triangular matrix**.

Multiplying both sides by $Q$ again shows that this problem is equivalent to

$$
    QR\mathbf{x} = QQ^T\mathbf{b}
$$

or $A\mathbf{x} = QQ^T\mathbf{b}$ i.e. we are just solving $A\mathbf{x}=\mathbf{p}$ where $\mathbf{p}$ is the orthogonal projection of  $\mathbf{b}$ onto $C(A)$

Note:
- $QQ^\top$ is projection matrix onto $C(A)$. If $Q$ is square, $QQ^\top = I$.

**Question**... how to find $QR$?

## QR Factorization

One of the most important ideas in linear algebra is the concept of factorizing an original matrix into different constituents that may have useful properties.  These properties can help us understand the matrix better and lead to numerical methods.  In numerical linear algebra one of the most important factorizations is the **QR factorization**.

The basic idea is that we want to break up $A$ into its successive spaces spanned by the columns of $A$.  If we have
$$
    A = \begin{bmatrix}  &  &  \\  &  &  \\ \mathbf{a}_1 & \cdots & \mathbf{a}_n \\  &  &  \\  &  &  \end{bmatrix}
$$
where the columns of $A$ are linearly independent then we want to construct the sequence
$$
    \text{span}(\mathbf{a}_1) \subseteq \text{span}(\mathbf{a}_1, \mathbf{a}_2) \subseteq \text{span}(\mathbf{a}_1, \mathbf{a}_2, \mathbf{a}_3) \subseteq \cdots \subseteq \text{span}(\mathbf{a}_1, \mathbf{a}_2, \ldots , \mathbf{a}_n)
$$
where here $\text{span}(\mathbf{v}_1,\mathbf{v}_2,\ldots,\mathbf{v}_m)$ indicates the subspace spanned by the vectors $\mathbf{v}_1, \mathbf{v}_2, \ldots, \mathbf{v}_m$.

QR factorization attempts to construct a set of orthonormal vectors $\mathbf{q}_i$ that span each of the subspaces, i.e.
$$
    \text{span}(\mathbf{a}_1, \mathbf{a}_2, \ldots, \mathbf{a}_j) = \text{span}(\mathbf{q}_1, \mathbf{q}_2, \ldots, \mathbf{q}_j).
$$

Before specifying a general procedure we go through the first few steps of the process to get a feel for what needs to be done. In particular we are given a sequence of vectors, $\mathbf{a_1}$, $\mathbf{a_2}$, $\ldots$, $\mathbf{a_n}$, and generate a sequence of orthonormal vectors  $\mathbf{q}_1$, $\mathbf{q}_2$, $\ldots$, and $\mathbf{q}_n$ whose spans are the same subspace.

First, for $\text{span}(\mathbf{a}_1)$ we can directly use $\mathbf{a}_1$ but normalize the vector such that
$$
    \mathbf{q}_1 = \frac{\mathbf{a}_1}{||\mathbf{a}_1||}.
$$

2. For $\text{span}(\mathbf{a}_1, \mathbf{a}_2)$ we already have $\mathbf{q}_1$ so we need to have a vector $\mathbf{q}_2$ that is orthogonal to $\mathbf{q}_1$, i.e.
$$
    \langle \mathbf{q}_1, \mathbf{q}_2 \rangle = \mathbf{q}_1^\ast \mathbf{q}_2 = 0
$$
and is again normalized.  We can accomplish this by modifying $\mathbf{a}_2$ such that
$$
    \mathbf{q}_2 = \frac{\mathbf{a}_2 - \langle \mathbf{q}_1, \mathbf{a}_2\rangle \mathbf{q}_1}{||\mathbf{a}_2 - \langle \mathbf{q}_1, \mathbf{a}_2\rangle\mathbf{q}_1||}.
$$
which we can show is orthogonal to $\mathbf{q}_1$ as
$$\begin{aligned}
    \langle \mathbf{q}_1, \mathbf{q}_2 \rangle &= \left(\langle  \mathbf{q}_1, \mathbf{a}_2 - \langle  \mathbf{q}_1, \mathbf{a}_2 \rangle\mathbf{q}_1 \rangle \right) \frac{1}{||\mathbf{a}_2 - \langle  \mathbf{q}_1, \mathbf{a}_2 \rangle \mathbf{q}_1||} \\
    &= \left(\langle  \mathbf{q}_1, \mathbf{a}_2 \rangle - \langle  \mathbf{q}_1, \mathbf{a}_2 \rangle\langle\mathbf{q}_1,\mathbf{q}_1\rangle\right) \frac{1}{||\mathbf{a}_2 - \langle  \mathbf{q}_1, \mathbf{a}_2 \rangle \mathbf{q}_1||} \\
    &= \left(\langle  \mathbf{q}_1, \mathbf{a}_2 \rangle - \langle  \mathbf{q}_1, \mathbf{a}_2 \rangle\right) \frac{1}{||\mathbf{a}_2 - \langle  \mathbf{q}_1, \mathbf{a}_2 \rangle \mathbf{q}_1||} \\
    &= 0.
\end{aligned}$$

Note:
- If columns of $A$ are not all linearly independent, the non-independent columns orthgonal $\mathbf{q}$ will be $0$.

These results suggest then that we may have a matrix factorization that has the following form:
$$
    \begin{bmatrix}  &  &  \\  &  &  \\ \mathbf{a}_1 & \cdots & \mathbf{a}_n \\  &  &  \\  &  &  \end{bmatrix} =
    \begin{bmatrix}  &  &  \\  &  &  \\ \mathbf{q}_1 & \cdots & \mathbf{q}_n \\  &  &  \\  &  &  \end{bmatrix}
    \begin{bmatrix} r_{11} & r_{12} & \cdots & r_{1n} \\  & r_{22} &  &  \\  &  & \ddots & \vdots \\  &  &  & r_{nn} \end{bmatrix}.
$$
If we write this out as a matrix multiplication we have
$$\begin{aligned}
    \mathbf{a}_1 &= r_{11} \mathbf{q}_1 \\
    \mathbf{a}_2 &= r_{22} \mathbf{q}_2 + r_{12} \mathbf{q}_1 \\
    \mathbf{a}_3 &= r_{33} \mathbf{q}_3 + r_{23} \mathbf{q}_2 + r_{13} \mathbf{q}_1 \\
    &\vdots
\end{aligned}$$

We can also identify at least the first couple of values of $\mathbf{r}$ as
$$\begin{aligned}
    r_{11} &= \left \Vert \mathbf{a}_1 \right \Vert \\
    r_{12} &= \langle \mathbf{q}_1, \mathbf{a}_2 \rangle \\
    r_{22} &= \left \Vert \mathbf{a}_2 - r_{12} \mathbf{q}_1 \right \Vert
\end{aligned}$$
It turns out we can generalize this as Gram-Schmidt orthogonalization.

### Gram-Schmidt Orthogonalization

As may have been suggestive we can directly construct the arrays $Q$ and $R$ via a process of successive orthogonalization.  We have already shown the first two iterations so lets now consider the $j$th iteration.

We want to subtract off the components of the vector $\mathbf{a}_j$ in the direction of the $\mathbf{q}_i$ vectors where $i=1,\ldots,j-1$.  This suggests that we define a vector $\mathbf{v}_j$ such that
$$\begin{aligned}
    \mathbf{v}_j &= \mathbf{a}_j - \langle \mathbf{q}_1, \mathbf{a}_j \rangle \mathbf{q}_1 - \langle \mathbf{q}_2, \mathbf{a}_j \rangle \mathbf{q}_2 - \quad \cdots \quad- \langle \mathbf{q}_{j-1}, \mathbf{a}_j \rangle \mathbf{q}_{i-1} \\
    &= \mathbf{a}_j - \sum^{j-1}_{i=1} \langle \mathbf{q}_i, \mathbf{a}_j \rangle \mathbf{q}_i.
\end{aligned}$$

We also need to normalize $\mathbf{v}_j$ which allows to define the $j$th column of $Q$ as
$$
    \mathbf{q}_j = \frac{\mathbf{a}_j - \sum^{j-1}_{i=1} \langle \mathbf{q}_i, \mathbf{a}_j \rangle \mathbf{q}_i}{\left \Vert \mathbf{a}_j - \sum^{j-1}_{i=1} \langle \mathbf{q}_i, \mathbf{a}_j \rangle \mathbf{q}_i \right \Vert}.
$$

Note:
- Can turn out unstable. There's another way called **modified Gram-Schmidt**

We can also discern what the entries of $R$ are as we can write the matrix multiplication as the sequence
$$\begin{aligned}
    \mathbf{q}_1 &= \frac{\mathbf{a}_1}{r_{11}} \\
    \mathbf{q}_2 &= \frac{\mathbf{a}_2 - r_{12} \mathbf{q}_1}{r_{22}} \\
    \mathbf{q}_3 &= \frac{\mathbf{a}_3 - r_{13} \mathbf{q}_1 - r_{23} \mathbf{q}_2}{r_{33}} \\
    &\vdots \\
    \mathbf{q}_n &= \frac{\mathbf{a}_n - \sum^{n-1}_{i=1} r_{in} \mathbf{q}_i}{r_{nn}}
\end{aligned}$$
leading us to define
$$
    r_{ij} = \left \{ \begin{aligned}
        &\langle \mathbf{q}_i, \mathbf{a}_j \rangle & &i \neq j \\
        &\left \Vert \mathbf{a}_j - \sum^{j-1}_{i=1} r_{ij} \mathbf{q}_i \right \Vert & &i = j
    \end{aligned} \right .
$$

This is called the **classical Gram-Schmidt** iteration.  Turns out that the procedure above is unstable because of rounding errors introduced.

```python
# Implement Classical Gram-Schmidt Iteration
def classic_GS(A):
    m = A.shape[0]
    n = A.shape[1]
    Q = numpy.empty((m, n))
    R = numpy.zeros((n, n))
    for j in range(n):
        v = A[:, j]
        for i in range(j):
            R[i, j] = numpy.dot(Q[:, i].conjugate(), A[:, j])
            v = v - R[i, j] * Q[:, i]
        R[j, j] = numpy.linalg.norm(v, ord=2)
        Q[:, j] = v / R[j, j]
    return Q, R
```

```python
A = numpy.array([[12, -51, 4], [6, 167, -68], [-4, 24, -41]], dtype=float)
Q, R = classic_GS(A)
print(A)
print(Q)
print(numpy.dot(Q.transpose(), Q))
print(R)
print(numpy.dot(Q, R) - A)
```

    [[ 12. -51.   4.]
     [  6. 167. -68.]
     [ -4.  24. -41.]]
    [[ 0.85714286 -0.39428571 -0.33142857]
     [ 0.42857143  0.90285714  0.03428571]
     [-0.28571429  0.17142857 -0.94285714]]
    [[ 1.00000000e+00 -4.18882616e-17 -8.63255046e-17]
     [-4.18882616e-17  1.00000000e+00 -8.63481622e-17]
     [-8.63255046e-17 -8.63481622e-17  1.00000000e+00]]
    [[ 14.  21. -14.]
     [  0. 175. -70.]
     [  0.   0.  35.]]
    [[ 0.00000000e+00 -7.10542736e-15  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00 -1.42108547e-14]
     [ 0.00000000e+00  3.55271368e-15  0.00000000e+00]]

#### Full vs. Reduced QR

If the original matrix $A \in \mathbb C^{m \times n}$ where $m \ge n$ then we can still define a QR factorization, called the **full QR factorization**, which appends columns full of zeros to $R$ to reproduce the full matrix.
$$
    A = Q R = \begin{bmatrix} Q_1 & Q_2 \end{bmatrix}
              \begin{bmatrix} R_1 \\
                              0
              \end{bmatrix} = Q_1 R_1
$$
The factorization $Q_1 R_1$ is called the **reduced** or **thin QR factorization** of $A$.

We require that the additional columns added $Q_2$ are an orthonormal basis that is orthogonal itself to $\text{range}(A)$.  If $A$ is full ranked then $Q_1$ and $Q_2$ provide a basis for $\text{range}(A)$ and $\text{null}(A^\ast)$ respectively.

Note:
- $\begin{bmatrix} Q_1 & Q_2 \end{bmatrix}$ is a square matrix
- Only care about $Q_1$ mostly

#### QR Existence and Uniqueness
Two important theorems exist regarding this algorithm which we state without proof:

1. *Every $A \in \mathbb C^{m \times n}$ with $m \geq n$ has a full QR factorization and therefore a reduced QR factorization.*

2. *Each $A \in \mathbb C^{m \times n}$ with $m \geq n$ of full rank has a unique reduced QR factorization $A = QR$ with $r_{jj} > 0$.*

#### Gram-Schmidt in Terms of Projections

To start lets rewrite classical Gram-Schmidt as a series of projections:
$$
    \mathbf{q}_1 = \frac{P_1 \mathbf{a}_1}{||P_1 \mathbf{a}_1||}, \quad \mathbf{q}_2 = \frac{P_2 \mathbf{a}_2}{||P_2 \mathbf{a}_2||}, \quad \cdots \quad \mathbf{q}_n = \frac{P_n \mathbf{a}_n}{||P_n \mathbf{a}_n||}
$$
where the $P_i$ are orthogonal projectors onto the $\mathbf{q}_1, \mathbf{q}_2, \ldots, \mathbf{q}_{i-1}$, in other words the complement of $\text{span}(\mathbf{a}_1, \mathbf{a}_2, \ldots, \mathbf{a}_{i-1})$.

How should we construct these projectors?

We saw before that we can easily construct an orthogonal projector onto the complement of a space by first constructing the projector onto the space itself via
$$
    \hat{\!Q}_{i-1} = \begin{bmatrix}
           &    &       &        \\
           &    &       &        \\
        \mathbf{q}_1 & \mathbf{q}_2 & \cdots & \mathbf{q}_{i-1} \\
           &    &       &        \\
           &    &       &
    \end{bmatrix}.
$$
Constructing the projection onto the space spanned by $\mathbf{q}_1$ through $\mathbf{q}_{i-1}$ is then
$$
    \hat{\!P}_{i-1} = \hat{\!Q}_{i-1} \hat{\!Q}^\ast_{i-1}
$$
and therefore the projections in Gram-Schmidt orthogonalization is
$$
    P_{i} = I - \hat{\!P}_{i-1} = I - \hat{\!Q}_{i-1} \hat{\!Q}^\ast_{i-1}.
$$

#### Modified Gram-Schmidt

One problem with the original Gram-Schmidt algorithm is it is not stable numerically.  Instead we can derive a modified method that is more numerically stable.

Recall that the basic piece of the original algorithm was to take the inner product of $\mathbf{a}_j$ and all the relevant $\mathbf{q}_i$.  Using the rewritten version of Gram-Schmidt in terms of projections we then have

$$
    \mathbf{v}_i = P_i \mathbf{a}_i.
$$

This projection is of rank $m - (i - 1)$ as we know that the resulting $\mathbf{v}_i$ are linearly independent by construction.  The modified version of Gram-Schmidt instead uses projections that are all of rank $m-1$.  To construct this projection remember that we can again construct the complement to a projection and perform the following sequence of projections

$$
    P_i = \hat{\!P}_{\mathbf{q}_{i-1}} \hat{\!P}_{\mathbf{q}_{i-2}} \cdots \hat{\!P}_{\mathbf{q}_{2}} \hat{\!P}_{\mathbf{q}_{1}}
$$

where $\hat{\!P}_{\mathbf{q}_{i}}$ projects onto the complement of the space spanned by $\mathbf{q}_i$.  Note that this performs mathematically the same job as $P_i \mathbf{a}_i$ however each of these projectors are of rank $m - 1$.

This leads to the following set of calculations:

$$\begin{aligned}
    1.\quad \mathbf{v}^{(1)}_i &= \mathbf{a}_i  \\
    2.\quad \mathbf{v}^{(2)}_i &= \hat{\!P}_{\mathbf{q}_1} \mathbf{v}_i^{(1)} = \mathbf{v}^{(1)}_i - \mathbf{q}_1 q_1^\ast \mathbf{v}^{(1)}_i \\
    3.\quad \mathbf{v}^{(3)}_i &= \hat{\!P}_{\mathbf{q}_2} \mathbf{v}_i^{(2)} = \mathbf{v}^{(2)}_i - \mathbf{q}_2 \mathbf{q}_2^\ast \mathbf{v}^{(2)}_i \\
    & \text{  } \vdots & &\\
    i.\quad \mathbf{v}^{(i)}_i &= \hat{\!P}_{\mathbf{q}_{i-1}} \mathbf{v}_i^{(i-1)} =  \mathbf{v}_i^{(i-1)} - \mathbf{q}_{i-1} \mathbf{q}_{i-1}^\ast \mathbf{v}^{(i-1)}_i
\end{aligned}$$

The reason why this approach is more stable is that we are not projecting with a possibly arbitrarily low-rank projector, instead we only take projectors that are high-rank.

**Example: Implementation of modified Gram-Schmidt**
Implement the modified Gram-Schmidt algorithm checking to make sure the resulting factorization has the required properties.

```python
# Implement Modified Gram-Schmidt Iteration
def mod_GS(A):
    m = A.shape[0]
    n = A.shape[1]
    Q = numpy.empty((m, n))
    R = numpy.zeros((n, n))
    v = A.copy()
    for i in range(n):
        R[i, i] = numpy.linalg.norm(v[:, i], ord=2)
        Q[:, i] = v[:, i] / R[i, i]
        for j in range(i + 1, n):
            R[i, j] = numpy.dot(Q[:, i].conjugate(), v[:, j])
            v[:, j] -= R[i, j] * Q[:, i]
    return Q, R
```

```python
A = numpy.array([[12, -51, 4], [6, 167, -68], [-4, 24, -41]], dtype=float)
print(A)
Q, R = mod_GS(A)
print(R)
print(numpy.dot(Q.transpose(), Q))
print("Modified = ")
print(numpy.dot(Q, R) - A)
```

    [[ 12. -51.   4.]
     [  6. 167. -68.]
     [ -4.  24. -41.]]
    [[ 14.  21. -14.]
     [  0. 175. -70.]
     [  0.   0.  35.]]
    [[ 1.00000000e+00 -4.18882616e-17 -3.08143533e-17]
     [-4.18882616e-17  1.00000000e+00 -5.30437372e-16]
     [-3.08143533e-17 -5.30437372e-16  1.00000000e+00]]
    Modified =
    [[ 0.00000000e+00 -7.10542736e-15  8.88178420e-16]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 0.00000000e+00  3.55271368e-15  0.00000000e+00]]

### Householder Triangularization

**Gram-Schmidt triangularization**

One way to also interpret Gram-Schmidt orthogonalization is as a series of multiplications by upper triangular matrices of the matrix A.  For instance the first step in performing the modified algorithm is to divide through by the norm $r_{11} = ||v_1||$ to give $q_1$:

$$
    \begin{bmatrix}
           &    &       &  \\
           &    &       &  \\
        \mathbf{v}_1 & \mathbf{v}_2 & \mathbf{v}_3 &  \cdots & \mathbf{v}_n \\
           &    &       &  \\
           &    &       &
    \end{bmatrix}
    \begin{bmatrix}
        \frac{1}{r_{11}}   &  &\cdots &       \\
           & 1  &          \\
         &  & \ddots &  \\
           &    &  & 1
    \end{bmatrix} =
    \begin{bmatrix}
           &    &       &        \\
           &    &       &        \\
        \mathbf{q}_1 & \mathbf{v}_2 &  \mathbf{v}_3 & \cdots & \mathbf{v}_n \\
           &    &       &        \\
           &    &       &
    \end{bmatrix}
$$

We can also perform all the step (2) evaluations by also combining the step that projects onto the complement of $q_1$ by add the appropriate values to the entire first row:

$$
    \begin{bmatrix}
           &    &       &  \\
           &    &       &  \\
        \mathbf{v}_1 & \mathbf{v}_2 &  \mathbf{v}_3 & \cdots & \mathbf{v}_n \\
           &    &       &  \\
           &    &       &
    \end{bmatrix}
    \begin{bmatrix}
        \frac{1}{r_{11}}   & -\frac{r_{12}}{r_{11}} & -\frac{r_{13}}{r_{11}} & \cdots      \\
           & 1   &         \\
         &  & \ddots &  \\
           &    &  & 1
    \end{bmatrix} =
    \begin{bmatrix}
           &    &       &        \\
           &    &       &        \\
        \mathbf{q}_1 & \mathbf{v}_2^{(2)} & \mathbf{v}_3^{(2)} & \cdots & \mathbf{v}_n^{(2)} \\
           &    &       &        \\
           &    &       &
    \end{bmatrix}
$$

The next step can then be placed into the second row:
$$
    \begin{bmatrix}
           &    &       &  \\
           &    &       &  \\
        \mathbf{v}_1 & \mathbf{v}_2 & \mathbf{v}_3 & \cdots & \mathbf{v}_n \\
           &    &       &  \\
           &    &       &
    \end{bmatrix}
    \cdot R_1 \cdot
    \begin{bmatrix}
        1 &  &  &  & \\
         & \frac{1}{r_{22}}   & -\frac{r_{23}}{r_{22}} & -\frac{r_{25}}{r_{22}} & \cdots      \\
         &   & 1   &         \\
         &  &  & \ddots &  \\
         &    &  & & 1
    \end{bmatrix} =
    \begin{bmatrix}
           &    &       &        \\
           &    &       &        \\
        \mathbf{q}_1 & \mathbf{q}_2 & \mathbf{v}_3^{(3)} & \cdots & \mathbf{v}_n^{(3)} \\
           &    &       &        \\
           &    &       &
    \end{bmatrix}
$$

If we identify the matrices as $R_1$ for the first case, $R_2$ for the second case and so on we can write the algorithm as

$$
    A \underbrace{R_1R_2 \quad \cdots \quad R_n}_{\hat{R}^{-1}} = \hat{\!Q}.
$$

This view of Gram-Schmidt is called **Gram-Schmidt triangularization**.

**Householder triangularization** is similar in spirit.  Instead of multiplying $A$ on the right Householder multiplies $A$ on the left by unitary matrices $Q_k$.  Remember that a unitary matrix (or an orthogonal matrix when strictly real) has as its inverse its adjoint (transpose when real) $Q^* = Q^{-1}$ so that $Q^* Q = I$.  We therefore have

$$
    Q_n Q_{n-1} \quad \cdots \quad Q_2 Q_1 A = R
$$

which if we identify $Q_n Q_{n-1} \text{  } \cdots \text{  } Q_2 Q_1 = Q^*$ and note that if $Q = Q^\ast_n Q^\ast_{n-1} \text{  } \cdots \text{  } Q^\ast_2 Q^\ast_1$ then $Q$ is also unitary.

We can then write this as
$$\begin{aligned}
    Q_n Q_{n-1} \text{  } \cdots \text{  } Q_2 Q_1 A &= R \\
    Q_{n-1} \cdots Q_2 Q_1 A &= Q^\ast_n R \\
    & \text{  } \vdots \\
    A &= Q^\ast_1 Q^\ast_2 \text{  } \cdots \text{  } Q^\ast_{n-1} Q^\ast_n R \\
    A &= Q R
\end{aligned}$$

Note:
- Unitary matrices ($Q$) do not change the lengths of vectors, so they are numerically stable

This was we can think of Householder triangularization as one of introducing zeros into $A$ via orthogonal matrices.

$$
    \begin{bmatrix}
        \text{x} & \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x} \\
    \end{bmatrix} \overset{Q_1}{\rightarrow}
    \begin{bmatrix}
        \text{x} & \text{x} & \text{x} \\
        0 & \text{x} & \text{x} \\
        0 & \text{x} & \text{x} \\
        0 & \text{x} & \text{x} \\
    \end{bmatrix} \overset{Q_2}{\rightarrow}
    \begin{bmatrix}
        \text{x} & \text{x} & \text{x} \\
        0 & \text{x} & \text{x} \\
        0 & 0 & \text{x} \\
        0 & 0 & \text{x} \\
    \end{bmatrix} \overset{Q_3}{\rightarrow}
    \begin{bmatrix}
        \text{x} & \text{x} & \text{x} \\
        0 & \text{x} & \text{x} \\
        0 & 0 & \text{x} \\
        0 & 0 & 0\\
    \end{bmatrix}
$$

Now the question is how do we construct the $Q_k$.  The construction is usually broken down into a matrix of the form

$$
    Q_k = \begin{bmatrix} I & 0 \\ 0 & F \end{bmatrix}
$$

where $I \in \mathbb C^{k-1 \times k-1}$ identity matrix and $F \in \mathbb C^{m - (k - 1) \times m - (k-1)}$ unitary matrix.  Note that this will leave the rows and columns we have already worked on alone and be unitary.

We now turn to the task of constructing the matrix $F$. Note that the definition of $Q_k$ implies that
$$
\begin{align}
Q_k \mathbf{x} &= \begin{bmatrix} I & 0 \\ 0 & F \end{bmatrix} \mathbf{x}
     &= \begin{bmatrix} I & 0 \\ 0 & F \end{bmatrix}
     \begin{bmatrix} \mathbf{x}_{top} \\ \mathbf{x}_{bottom} \end{bmatrix} \\
     &= \begin{bmatrix} \mathbf{x}_{top} \\ F\mathbf{x}_{bottom}  \end{bmatrix}. \\
\end{align}
$$
The vectors $\mathbf{x}_{top}$ and $\mathbf{x}_{bottom}$ are defined to be
$$
\begin{align}
     \mathbf{x}_{top} &= \begin{bmatrix} x_1  \\  x_2 \\ \vdots \\ x_{k-1}\end{bmatrix} \\
     \mathbf{x}_{bottom} &= \begin{bmatrix} x_k \\ x_{k+1} \\ \vdots \\ x_n \end{bmatrix} \\
\end{align}
$$
The task is to construct the matrix $F$ so that after multiplying, $F\mathbf{x}_{bottom}$  results in a vector that has some nonzero number in the top position and the rest of the numbers are all zeroes.

One way to think of this operation is to consider the transformation that reflects the vector $x$ over the plane $H$ so that $F x = v = ||x|| \hat{e}_1$:

or mathematically
$$
    \mathbf{x} = \begin{bmatrix}
        x_1 \\
        x_2 \\
        \vdots \\
        x_{n}
     \end{bmatrix} \overset{F}{\rightarrow}
    F\mathbf{x} = \begin{bmatrix}
        ||\mathbf{x}|| \\
        0 \\
        \vdots \\
        0
    \end{bmatrix} = ||\mathbf{x}|| \hat{e}_1.
$$
The unit vector $\hat{e}_1$ is defined to be
$$
\hat{x} = \begin{bmatrix}
        1 \\
        0 \\
        \vdots \\
        0
    \end{bmatrix}.
$$

This is of course the effect on only one vector.  Any other vector will be reflected across $H$ (technically a hyperplane) which is orthogonal to $v = ||\mathbf{x}|| \hat{e}_1 - \mathbf{x}$.  This has a similar construction as to the projector complements we were working with before.  Consider the projector defined as

$$
    P x = \left (I - \frac{\mathbf{v} \mathbf{v}^\ast}{\mathbf{v}^\ast \mathbf{v}}\right) \mathbf{x} = \mathbf{x} - \mathbf{x} \left(\frac{\mathbf{v}^\ast \mathbf{x}}{\mathbf{v}^\ast \mathbf{v}} \right),
$$

the complement of a projection in the direction of the vector $\mathbf{v}$, in other words in the direction of $H$ above.  Since we actually want to transform $\mathbf{x}$ to lie in the direction of $\hat{e}_1$ we need to go twice as far as just the projection onto $H$.  This allows us to identify the matrix $F$ as

$$
    F = I - 2 \frac{\mathbf{v} \mathbf{v}^\ast}{\mathbf{v}^\ast \mathbf{v}}.
$$

There is actually a non-uniqueness to which direction we reflect over since another definition of $\hat{H}$ which is orthogonal to the one we originally choose is available.  For numerical stability purposes we will choose the reflector that is the most different from $\mathbf{x}$.  This comes back to having difficulties numerically when the vector $\mathbf{x}$ is nearly aligned with $\hat{e}_1$ and therefore one of the $H$ specification.  By convention the $\mathbf{v}$ chosen is defined by

$$
    v = \text{sign}(\mathbf{x}_1)||\mathbf{x}|| \hat{e}_1 + \mathbf{x}.
$$

```python
# Implementation of Householder QR Factorization
def householder_QR(A, verbose=False):
    R = A.copy()
    v = numpy.empty(A.shape)
    m, n = A.shape
    for k in range(n):
        x = R[k:, k]
        e1 = numpy.zeros(x.shape)
        e1[0] = 1.0
        v[k:, k] = numpy.sign(x[0]) * numpy.linalg.norm(x, ord=2) * e1 + x
        v[k:, k] = v[k:, k] / numpy.linalg.norm(v[k:, k], ord=2)
        #FIXME:  What's wrong with the next line?
        R[k:, k:] -= 2.0 * numpy.dot(numpy.outer(v[k:, k], v[k:, k]), R[k:, k:])

    # Form Q
    m, n = A.shape
    Q = numpy.zeros(A.shape)
    for i in range(n):
        en = numpy.zeros(m)
        en[i] = 1.0
        for j in range(n - 1, -1, -1):
            en[j:m] -= 2.0 * numpy.dot(numpy.outer(v[j:, j], v[j:, j]), en[j:m])
        Q[:, i] = en

    return Q, R

A = numpy.array([[12, -51, 4], [6, 167, -68], [-4, 24, -41]], dtype=float)
print("Matrix A = ")
print(A)
m, n = A.shape
Q, R = householder_QR(A, verbose=False)
print("Householder (reduced) Q = ")
print(Q)
print("Householder (full) R = ")
print(R)

print("Check to see if factorization worked...")
print(numpy.abs(A - numpy.dot(Q, R[:n, :n])))
```

    Matrix A =
    [[ 12. -51.   4.]
     [  6. 167. -68.]
     [ -4.  24. -41.]]
    Householder (reduced) Q =
    [[-0.85714286  0.39428571 -0.33142857]
     [-0.42857143 -0.90285714  0.03428571]
     [ 0.28571429 -0.17142857 -0.94285714]]
    Householder (full) R =
    [[-1.4000000e+01 -2.1000000e+01  1.4000000e+01]
     [ 8.8817842e-16 -1.7500000e+02  7.0000000e+01]
     [-4.4408921e-16  0.0000000e+00  3.5000000e+01]]
    Check to see if factorization worked...
    [[1.77635684e-15 3.55271368e-14 1.19904087e-14]
     [8.88178420e-16 8.52651283e-14 4.26325641e-14]
     [4.44089210e-16 3.55271368e-15 0.00000000e+00]]

In the above algorithm we do not need to explicitly form the matrix $Q$ to save memory and computation.  If we wanted to for instance solve $A \mathbf{x} = \mathbf{b}$ we again have

$$\begin{aligned}
    A \mathbf{x} &= \mathbf{b} \\
    Q R \mathbf{x} &= \mathbf{b} \\
    R\mathbf{x} &= Q^\ast \mathbf{b}.
\end{aligned}$$

This requires then the multiplication $Q^\ast \mathbf{b}$.  To do this we just need to recognize that the vectors $\mathbf{v}$ we save specify the $Q_k$ so we can form the matrices $F$ and multiply the vector directly with the $\mathbf{v}_k$s:

#### Example 1:   Random Matrix QR

Consider a matrix $A$ with a random eigenspace and widely varying eigenvalues.  The values along the diagonal of $R$ gives us some idea of the size of the projections as we go, i.e. the larger the values the less effective we are in constructing othorgonal directions.

```python
N = 80
# construct a random matrix with known singular values
U, X = numpy.linalg.qr(numpy.random.random((N, N)))
V, X = numpy.linalg.qr(numpy.random.random((N, N)))
S = numpy.diag(2.0**numpy.arange(-1.0, -(N + 1), -1.0))
A = numpy.dot(U, numpy.dot(S, V))

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
Q, R = classic_GS(A)
axes.semilogy(numpy.diag(R), 'bo', label="Classic")
Q, R = mod_GS(A)
axes.semilogy(numpy.diag(R), 'ro', label="Modified")
Q, R = householder_QR(A)
axes.semilogy(numpy.diag(R), 'ko', label="Householder")

axes.set_xlabel("Index", fontsize=16)
axes.set_ylabel("$R_{ii}$", fontsize=16)
axes.legend(loc=3, fontsize=14)
axes.plot(numpy.arange(0, N), numpy.ones(N) * numpy.sqrt(numpy.finfo(float).eps), 'k--')
axes.plot(numpy.arange(0, N), numpy.ones(N) * numpy.finfo(float).eps, 'k--')

plt.show()
```

#### Example 2:  Comparing Orthogonality

Consider
$$
    A = \begin{bmatrix}
        0.70000 & 0.70711 \\ 0.70001 & 0.70711
    \end{bmatrix}.
$$
Check that the matrix $Q$ is really unitary given this matrix.

```python
%precision 16

A = numpy.array([[0.7, 0.70711], [0.70001, 0.70711]])

Q, R = classic_GS(A)
print("Classic: ", numpy.linalg.norm(numpy.dot(Q.transpose(), Q) - numpy.eye(2)))

Q, R = mod_GS(A)
print("Modified: ", numpy.linalg.norm(numpy.dot(Q.transpose(), Q) - numpy.eye(2)))

Q, R = householder_QR(A)
print("Householder:", numpy.linalg.norm(numpy.dot(Q.transpose(), Q) - numpy.eye(2)))

Q, R = numpy.linalg.qr(A)
print("Numpy: ", numpy.linalg.norm(numpy.dot(Q.transpose(), Q) - numpy.eye(2)))
```

    Classic:  3.254726094493924e-11
    Modified:  3.254726094493924e-11
    Householder: 1.111052298468932e-16
    Numpy:  2.3411870786352597e-16

### Applications of QR

#### Solving $A\mathbf{x} = \mathbf{b}$ with QR

Suppose we want to solve the system $A\mathbf{x} = \mathbf{b}$ where $A \in \mathbb C^{m \times m}$.  See if you can figure out how to use a QR factorization to help with this.

Say we have found the QR factorization of $A$, then
$$\begin{aligned}
    A \mathbf{x} &= \mathbf{b} \\
    QR \mathbf{x} & = \mathbf{b} \\
    Q^\ast Q R \mathbf{x} &= Q^\ast \mathbf{b} \\
    R \mathbf{x} &= Q^\ast \mathbf{b}.
\end{aligned}$$
Given that $R$ is upper triangular we can use back-substitution to find $\mathbf{b}$.

#### Applications: Least Squares Problems

Least squares problems have already been introduced but lets consider how our QR factorizations might help us.  As before the least squares problem is characterized by wanting to find the $\mathbf{b}$ such that $||\mathbf{b} - A\mathbf{x}||_2$ is minimized over $\mathbf{x} \in \mathbb C^n$.

Since we are using the $\ell_2$ norm and know this is equivalent to the Euclidean norm we know that there is a geometric interpretation to this goal, find the vector $\mathbf{x}$ that gives the minimum distance between the vector $\mathbf{b}$ and $A \mathbf{x}$.  This can be interpreted as a projection:

where
$$
    \mathbf{r} = \mathbf{b} - A\mathbf{x}
$$
and
$$
    \mathbf{y} = A\mathbf{x} = P\mathbf{b}.
$$
The vector $\mathbf{r}$ is called the residual (and the thing we are trying to minimize).  $P$ represents the orthogonal projector onto the $\text{range}(A)$.

QR factorization plays a role similar to the ideas we saw from Householder triangularization.  Define the orthogonal projector $P = Q Q^\ast$ based on the reduced QR factorization of $A$.  We know then that the projector projects onto the span of column space of $A$, $\text{span}(A)$.  Using this QR factorization we know that the least-squares formulation then becomes
$$\begin{aligned}
    A^\ast A \mathbf{x} &= A^\ast \mathbf{b} \\
    R^\ast Q^\ast Q R \mathbf{x} &= R^\ast Q^\ast \mathbf{b} \\
    R^\ast R \mathbf{x} & = R^\ast Q^\ast \mathbf{b} \\
    R \mathbf{x} & = Q^\ast \mathbf{b}
\end{aligned}$$
reducing the least-squares calculation to one of finding the QR factorization and backwards substitution.

```python

```


## Probabilidad

### Intro

La probabilidad es una medida de la certidumbre o posibilidad asociada a un evento o suceso y se expresa como un número entre 0 y 1.

Es importante conocer las distribuciones de probabilidades y sus aplicaciones para trabajar eficazmente en problemas de ciencia de datos:

* Las distribuciones de probabilidad nos permiten reconocer patrones en variables aleatorias.

* La mayoría de las técnicas de data science o machine learning se basan en ciertos supuestos sobre las distribuciones de probabilidad de nuestros datos.
Por lo tanto, saber qué distribución tienen los datos con los que estamos trabajando, puede ayudarnos a identificar qué técnicas o modelos nos conviene usar.

* Usamos *inferencia* para predecir o analizar tendencias a partir de datos. La *inferencia estadística* se basa en la distribución de probabilidad de estos datos.

* Una tarea muy frecuente en ciencia de datos es la *imputación de valores faltantes*, en general conocer la distribución de los datos con los que estamos trabajando nos permiten imputar de manera menos arbitraria.

A continuación vamos a revisar / presentar varias distribuciones de probabilidad de variables aleatorias discretas y continuas.

El objetivo es que se familiaricen con ellas y hacia el final las usemos en algunos ejercicios.

Una **variable aleatoria** es una función que asigna un valor, usualmente numérico, al resultado de un experimento aleatorio, es decir los resultados que se presentan al azar en cualquier evento o experimento.

La **distribución de probabilidad de una variable aleatoria X**, también llamada **función de distribución** de X es la función $F_X(x)$, que asigna a cada evento definido sobre la variable aleatoria $X$ una probabilidad dada por la siguiente expresión:

\begin{equation}
  F_X(x) = P(X \le x)
\end{equation}

Entonces la distribución de probabilidad está completamente especificada por la función de distribución, cuyo valor en cada $x$ real es la probabilidad de que la variable aleatoria sea menor o igual que $x$.

La **función de densidad de probabilidad de una variable aleatoria continua**, función de densidad, o, simplemente, densidad describe la probabilidad relativa, representada comúnmente como $f(x)$, nos permite conocer cómo se distribuyen las probabilidades de un suceso o evento en relación al resultado.
La función de distribución de probabilidad $F(x)$ es la integral de la función de densidad $f(x)$

\begin{equation}
  F_X(x) =  \int^x_{-\infty}f(t)dt
\end{equation}

La probabilidad de que la variable aleatoria caiga en una región específica del espacio de posibilidades estará dada por la integral de la densidad de esta variable entre uno y otro límite de dicha región.
La función de densidad de probabilidad es positiva en todo su dominio y su integral sobre todo el espacio es de valor 1.

### Variables Aleatorias Discretas

Cuando los valores que toma una variable aleatoria son **finitos o infinitos numerables**, se dice que es **discreta**

Ejemplo:

X es el resultado obtenido al lanzar un dado {1, 2, 3, 4, 5, 6}

### Variables Aleatorias Continuas

Cuando los valores que toma una variable aleatoria pueden ser **cualquier valor en un intervalo continuo dado**, se dice que es **continua**

Ejemplo:

X es la altura (sin redondear) de un alumno del curso de Data Science

___

### Distribuciones de Probabilidad

Estudiaremos ahora algunas distribuciones de probabilidad particulares.

Empezaremos con las distribuciones de tipo discreto y continuaremos con las de tipo continuo.

Observaremos además que las distribuciones pueden depender de uno o más parámetros, es decir, para cada valor de estos parámetros se tiene una distribución de probabilidad diferente. A veces se usa el término “familias de distribuciones paramétricas” para referirse a estas distribuciones.

[Distribución Uniforme](./4.1_uniforme.ipynb)

[Distribución Bernoulli](./4.2_bernoulli.ipynb)

[Distribución Binomial](./4.3_binomial.ipynb)

[Distribución Poisson](./4.4_poisson.ipynb)

[Distribución Normal](./4.5_normal.ipynb)

[Distribución Exponencial](./4.6_exponencial.ipynb)

___

### Relaciones entre Distribuciones

#### Relación entre Bernoulli y Binomial

La distribución de Bernoulli es un caso particular de la distribucón Binomial con una única repetición.

Si cada evento Bernoulli es **independiente**, entonces el número de éxitos en una serie de observaciones de una variable aleatoria con distribución Bernoulli tiene distribución Binomial. La distribución Bernoulli puede definirse entonces como una Binomial con n = 1.

#### Relación entre Poisson y Binomial

La distribución de Poisson es una distribución Binomial con las siguientes condiciones:

* El número de repeticiones es infinitamente grande $n \rightarrow \infty$

* La probabilidad de éxito de cada repetición es la misma e infinitamente pequeña $p \rightarrow 0$

* $np = \lambda$, es finito.

```python
import numpy as np
random_generator = np.random.default_rng()
```

```python
import seaborn as sns
def distribution_plotter(data, label, bins=None):
    sns.set(rc={"figure.figsize": (10, 7)})
    sns.set_style("white")
    dist = sns.distplot(data, bins= bins, hist_kws={'alpha':0.2}, kde_kws={'linewidth':5})
    dist.set_title('Distribucion de ' + label + '\n', fontsize=16)
```

```python
n = 1000000
p = .01
lambda_value = n * p

random_poisson_data = random_generator.poisson(lam = lambda_value, size=n)
random_binomial_data = random_generator.binomial(n, p, size=n)

distribution_plotter(random_poisson_data, "poisson")
distribution_plotter(random_binomial_data, "poisson y binomial")
```

Prueben ahora generando datos para p = 0.1, 0.5 y 0.7 y comparen los gráficos. ¿Qué pueden concluir?

#### Relación entre Normal y Binomial

La distribución normal es el límite de una distribución Binomial con las siguientes condiciones:

* El número de repeticiones en infinitamente grande $n \rightarrow \infty$

* p (probabilidad de éxito) y q (probabilidad de fracaso) no son infinitamente pequeñas simultaneamente

La distribución normal a la que tiende tiene parámetros $\mu = n . p$ y $\sigma^2 = n . p . (1-p)$

```python
n = 1000000
p = .01

media=n*p
varianza = n*p*(1-p)

random_binomial_data = random_generator.binomial(n, p, size=n)
random_normal_data = random_generator.normal(loc=media, scale=np.sqrt(varianza), size=n)

distribution_plotter(random_binomial_data, "binomial")
distribution_plotter(random_normal_data, "binomial y normal")

```

Prueben ahora generando datos para n = 1000, 10000, 100000 y comparen los gráficos. ¿Qué pueden concluir?

Observen también que pasa cuando varían los valores de p.

#### Relación entre Normal y Poisson

La distribución normal es el límite de una distribución Poisson con el parámetro $\lambda \rightarrow \infty$

```python
n = 10000

lambda_value = 10000
madia = lambda_value
varianza = lambda_value

random_poisson_data = random_generator.poisson(lam = lambda_value, size=n)
random_normal_data = random_generator.normal(loc=media, scale=np.sqrt(varianza), size=n)

distribution_plotter(random_poisson_data, "poisson")
distribution_plotter(random_normal_data, "poisson y normal")
```

Prueben ahora generando datos para lambda = 1000, 5000 y comparen los gráficos.

#### Relación entre Exponencial y Poisson

Si el tiempo entre eventos sigue una distribución exponencial de parámetro $\lambda$, entonces el número total de eventos en un período de tiempo t sigue una distribución Poisson con parámetro $\lambda{t}$

```python
n = 100000
lambda_value = 2
t = 10

beta_value = 1.0/lambda_value

random_exponential_data = random_generator.exponential(scale=beta_value, size=n)
random_poisson_data = random_generator.poisson(lam = lambda_value*t, size=n)

distribution_plotter(random_exponential_data, "exponencial")
distribution_plotter(random_poisson_data, "tiempo exponencial, número total de eventos poisson")
```

---

### Conclusiones

Hay muchas distribuciones de probabilidad, cada una de ellas fue diseñada para analizar datos de diversos orígenes y que tienen ciertas características. En la práctica, estas distribuciones se utilizan para analizar datos reales de un tipo similar.

Las distribuciones de probabilidad son modelos estadísticos probabilísticos cuyas propiedades se utilizan para analizar datos empíricos: Los datos se recopilan y comparan con cualquier distribución teórica conocida. Si son similares, las propiedades del modelo teórico se transfieren a datos empíricos con las conclusiones correspondientes.

Este es el enfoque subyacente a los métodos clásicos asociados con la prueba de hipótesis estadísticas (cálculo de intervalos de confianza, comparación de valores promedio, verificación de la importancia de los parámetros, etc.).

Si los datos disponibles no corresponden a ninguna distribución conocida (que muchas veces ocurre en la práctica), no se recomienda utilizar estos modelos estadísticos.

Para resolver este problema, existen otros enfoques asociados con el uso de estadística no paramétrica.


## Surfinpy

#### Tutorial 1 - Bulk Phase diagrams

In this tutorial we learn how to generate a basic bulk phase diagram from DFT energies.  This enables the comparison of the thermodynamic stability of various different bulk phases under different chemical potentials giving valuable insight in to the syntheis of solid phases.  This example will consider a series of bulk phases which can be defined through a reaction scheme across all phases, thus for this example including an example bulk phase, H2O and CO2 as reactions and A as a generic product.

\begin{align}
x\text{Bulk} + y\text{H}_2\text{O} + z\text{CO}_2 \rightarrow \text{A}
\end{align}

##### Methodology

The system is in equilibrium when the chemical potentials of the reactants and product are equal; i.e. the change in Gibbs free energy is $\delta G_{T,p} = 0$.

\begin{align}
	\delta G_{T,p} = \mu_A - x\mu_{\text{Bulk}} - y\mu_{\text{H}_2\text{O}} - z\mu_{\text{CO}_2} = 0
\end{align}

Assuming that H2O and CO2 are gaseous species, $\mu_{CO_2}$ and $\mu_{H_2O}$ can be written as

\begin{align}
	\mu_{\text{H}_2\text{O}} = \mu^0_{\text{H}_2\text{O}} + \Delta\mu_{\text{H}_2\text{O}}
\end{align}
and
\begin{align}
	\mu_{\text{CO}_2} = \mu^0_{\text{CO}_2} + \Delta\mu_{\text{CO}_2}
\end{align}

The chemical potential $\mu^0_x$ is the partial molar free energy of any reactants or products (x) in their standard states, in this example we assume all solid components can be expressed as

\begin{align}
\mu_{\text{component}} = \mu^0_{\text{component}}
\end{align}

Hence, we can now rearrange the equations to produce;

\begin{align}
	\mu^0_A - x\mu^0_{\text{Bulk}} - y\mu^0_{\text{H}_2\text{O}} - z\mu^0_{\text{CO}_2} = y\Delta\mu_{\text{H}_2\text{O}} + z\Delta\mu_{\text{CO}_2}
\end{align}

As $\mu^0_A$  corresponds to the partial molar free energy of product A, we can replace the left side with the Gibbs free energy ($\Delta G_{\text{f}}^0$).

\begin{align}
	\delta G_{T,p} = \Delta G_{\text{f}}^0 - y\Delta\mu_{\text{H}_2\text{O}} - z\Delta\mu_{\text{CO}_2}
\end{align}

At equilibrium $\delta G_{T,p} = 0$, and hence

\begin{align}
	\Delta G_{\text{f}}^0 = y\Delta\mu_{\text{H}_2\text{O}} + z\Delta\mu_{\text{CO}_2}
\end{align}

Thus, we can find the values of $\Delta\mu_{\text{H}_2\text{O}}$ and $\Delta\mu_{\text{CO}_2}$ (or $(p_{\text{H}_2\text{O}})^y$ and $(p_{\text{CO}_2})^z$) when solid phases are in thermodynamic equilibrium; i.e. they are more or less stable than bulk.  This procedure can then be applied to all phases to identify which is the most stable, provided that the free energy $\Delta G_f^0$ is known for each phase.

The free energy can be calculated using

\begin{align}
\Delta G^{0}_{f} = \sum\Delta G_{f}^{0,\text{products}} - \sum\Delta G_{f}^{0,\text{reactants}}
\end{align}

Where for this tutorial the free energy (G) is equal to the calculated DFT energy (U0).

```python
import matplotlib.pyplot as plt
from surfinpy import bulk_mu_vs_mu as bmvm
from surfinpy import utils as ut
from surfinpy import data
```

The first thing to do is input the data that we have generated from our DFT simulations. The input data needs to be contained within a `surfinpy.data` object. First we have created a `surfinpy.data.ReferenceDataSet` object for the bulk data (reference data), where `cation` is the number of cations, `anion` is the number of anions, `energy` is the DFT energy and `funits` is the number of formula units.

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 1, energy = -92.0, funits = 10)
```

Next we create the bulk `surfinpy.data.DataSet` objects - one for each surface or "phase". `cation` is the number of cations, `x` is in this case the number of oxygen species (corresponding to the X axis of the phase diagram), `y` is the number of in this case water molecules (corresponding to the Y axis of our phase diagram), `energy` is the DFT energy, `label` is the label for the phase (appears on the phase diagram) and finally `nSpecies` is the number of adsorbin species.

```python
Bulk = data.DataSet(cation = 10, x = 0, y = 0, energy = -92.0, label = "Bulk")
A = data.DataSet(cation = 10, x = 5, y = 20, energy = -468.0, label = "A")
B = data.DataSet(cation = 10, x = 0, y = 10, energy = -228.0, label = "B")
C = data.DataSet(cation = 10, x = 10, y = 30, energy = -706.0, label = "C")
D = data.DataSet(cation = 10, x = 10, y = 0, energy = -310.0, label = "D")
E = data.DataSet(cation = 10, x = 10, y = 50, energy = -972.0, label = "E")
F = data.DataSet(cation = 10, x = 8, y = 10, energy = -398.0, label = "F")

```

Next we need to create a list of our data. Don't worry about the order, surfinpy will sort that out for you.

```python
data = [Bulk, A, B, C,  D, E, F]
```

We now need to generate our X and Y axis, or more appropriately, our chemical potential values. These exist in a dictionary. 'Range' corresponds to the range of chemcial potential values to be considered and 'Label' is the axis label.  Additionally, the x and y energy need to be specified.

```
    deltaX = {'Range': Range of Chemical Potential,
              'Label': Species Label}

```

```python
deltaX = {'Range': [ -3, 2],  'Label': 'CO_2'}
deltaY = {'Range': [ -3, 2], 'Label': 'H_2O'}
x_energy=-20.53412969
y_energy=-12.83725889
```

And finally we can generate our plot using these 6 variables of data.

```python
system = bmvm.calculate(data, bulk, deltaX, deltaY, x_energy, y_energy)

ax = system.plot_phase(figsize=(6, 4.5))
plt.show()
```

```python

```


# Tutorial 1: Gradient Descent and AutoGrad
**Week 1, Day 2: Linear Deep Learning**

**By Neuromatch Academy**

__Content creators:__ Saeed Salehi, Vladimir Haltakov, Andrew Saxe

__Content reviewers:__ Polina Turishcheva, Antoine De Comite, Kelson Shilling-Scrivo

__Content editors:__ Anoop Kulkarni, Spiros Chavlis

__Production editors:__ Khalid Almubarak, Spiros Chavlis

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
#Tutorial Objectives

Day 2 Tutorial 1 will continue on buiding PyTorch skillset and motivate its core functionality, Autograd. In this notebook, we will cover the key concepts and ideas of:

* Gradient descent
* PyTorch Autograd
* PyTorch nn module

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in this tutorial
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/3qevp/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

This a GPU-Free tutorial!

```python
# @title Install dependencies
!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet

from evaltools.airtable import AirtableForm
# init airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W1D2_T1','https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303')
```

```python
# Imports
import torch
import numpy as np
from torch import nn
from math import pi
import matplotlib.pyplot as plt
```

```python
# @title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")
```

```python
# @title Plotting functions

from mpl_toolkits.axes_grid1 import make_axes_locatable

def ex3_plot(model, x, y, ep, lss):
  f, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))
  ax1.set_title("Regression")
  ax1.plot(x, model(x).detach().numpy(), color='r', label='prediction')
  ax1.scatter(x, y, c='c', label='targets')
  ax1.set_xlabel('x')
  ax1.set_ylabel('y')
  ax1.legend()

  ax2.set_title("Training loss")
  ax2.plot(np.linspace(1, epochs, epochs), losses, color='y')
  ax2.set_xlabel("Epoch")
  ax2.set_ylabel("MSE")

  plt.show()

def ex1_plot(fun_z, fun_dz):
  """Plots the function and gradient vectors

  """
  x, y = np.arange(-3, 3.01, 0.02), np.arange(-3, 3.01, 0.02)
  xx, yy = np.meshgrid(x, y, sparse=True)
  zz = fun_z(xx, yy)
  xg, yg = np.arange(-2.5, 2.6, 0.5), np.arange(-2.5, 2.6, 0.5)
  xxg, yyg = np.meshgrid(xg, yg, sparse=True)
  zxg, zyg = fun_dz(xxg, yyg)

  plt.figure(figsize=(8, 7))
  plt.title("Gradient vectors point towards steepest ascent")
  contplt = plt.contourf(x, y, zz, levels=20)
  plt.quiver(xxg, yyg, zxg, zyg, scale=50, color='r', )
  plt.xlabel('$x$')
  plt.ylabel('$y$')
  ax = plt.gca()
  divider = make_axes_locatable(ax)
  cax = divider.append_axes("right", size="5%", pad=0.05)
  cbar = plt.colorbar(contplt, cax=cax)
  cbar.set_label('$z = h(x, y)$')
  plt.show()
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("GPU is not enabled in this notebook. \n"
          "If you want to enable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `GPU` from the dropdown menu")
  else:
    print("GPU is enabled in this notebook. \n"
          "If you want to disable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `None` from the dropdown menu")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

    Random seed 2021 has been set.
    GPU is not enabled in this notebook.
    If you want to enable it, in the menu under `Runtime` ->
    `Hardware accelerator.` and select `GPU` from the dropdown menu

---
# Section 0: Introduction

Today, we will go through 3 tutorials. Starting with Gradient Descent, the workhorse of deep learning algorithms, in this tutorial. The second tutorial will help us build a better intuition about neural networks and basic hyper-parameters. Finally, in tutorial 3, we learn about the learning dynamics, what the (a good) deep network is learning, and why sometimes they may perform poorly.

```python
# @title Video 0: Introduction
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Qf4y1578t", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"i7djAv2jnzY", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 0:Introduction')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

---
# Section 1: Gradient Descent Algorithm

*Time estimate: ~30-45 mins*

Since the goal of most learning algorithms is **minimizing the risk (also known as the cost or loss) function**, optimization is often the core of most machine learning techniques! The gradient descent algorithm, along with its variations such as stochastic gradient descent, is one of the most powerful and popular optimization methods used for deep learning. Today we will introduce the basics, but you will learn much more about Optimization in the coming days (Week 1 Day 4).

## Section 1.1: Gradients & Steepest Ascent

```python
# @title Video 1: Gradient Descent
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Pq4y1p7em", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"UwgA_SgG0TM", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 1: Gradient Descent')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Before introducing the gradient descent algorithm, let's review a very important property of gradients. The gradient of a function always points in the direction of the steepest ascent. The following exercise will help clarify this.

### Analytical Exercise 1.1: Gradient vector (Optional)

Given the following function:

\begin{equation}
z = h(x, y) = \sin(x^2 + y^2)
\end{equation}

find the gradient vector:

\begin{equation}
  \begin{bmatrix}
  \dfrac{\partial z}{\partial x} \\ \\ \dfrac{\partial z}{\partial y}
  \end{bmatrix}
\end{equation}

*hint: use the chain rule!*

**Chain rule**: For a composite function $F(x) = g(h(x)) \equiv (g \circ h)(x)$:

\begin{equation}
F'(x) = g'(h(x)) \cdot h'(x)
\end{equation}

or differently denoted:

\begin{equation}
\frac{dF}{dx} = \frac{dg}{dh} ~ \frac{dh}{dx}
\end{equation}

---
#### Solution:
We can rewrite the function as a composite function:

\begin{equation}
z = f\left( g(x,y) \right), ~~ f(u) = \sin(u), ~~ g(x, y) = x^2 + y^2
\end{equation}

Using chain rule:

\begin{align}
\dfrac{\partial z}{\partial x} &= \dfrac{\partial f}{\partial g} \dfrac{\partial g}{\partial x} = \cos(g(x,y)) ~ (2x) = \cos(x^2 + y^2) \cdot 2x \\ \\
\dfrac{\partial z}{\partial y} &= \dfrac{\partial f}{\partial g} \dfrac{\partial g}{\partial y} = \cos(g(x,y)) ~ (2y) = \cos(x^2 + y^2) \cdot 2y
\end{align}

### Coding Exercise 1.1: Gradient Vector

Implement (complete) the function which returns the gradient vector for $z=\sin(x^2 + y^2)$.

```python
def fun_z(x, y):
  """Function sin(x^2 + y^2)

  Args:
    x (float, np.ndarray): variable x
    y (float, np.ndarray): variable y

  Return:
    z (float, np.ndarray): sin(x^2 + y^2)
  """
  z = np.sin(x**2 + y**2)
  return z

def fun_dz(x, y):
  """Function sin(x^2 + y^2)

  Args:
    x (float, np.ndarray): variable x
    y (float, np.ndarray): variable y

  Return:
    (tuple): gradient vector for sin(x^2 + y^2)
  """
  #################################################
  ## Implement the function which returns gradient vector
  ## Complete the partial derivatives dz_dx and dz_dy
  # Complete the function and remove or comment the line below
  # raise NotImplementedError("Gradient function `fun_dz`")
  #################################################
  dz_dx = np.cos(x**2 + y**2) * 2 * x
  dz_dy = np.cos(x**2 + y**2) * 2 * y
  return (dz_dx, dz_dy)

#add event to airtable
atform.add_event('Coding Exercise 1.1: Gradient Vector')

## Uncomment to run
ex1_plot(fun_z, fun_dz)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D2_LinearDeepLearning/solutions/W1D2_Tutorial1_Solution_0c8e3872.py)

*Example output:*

We can see from the plot that for any given $x_0$ and $y_0$, the gradient vector $\left[ \dfrac{\partial z}{\partial x}, \dfrac{\partial z}{\partial y}\right]^{\top}_{(x_0, y_0)}$ points in the direction of $x$ and $y$ for which $z$ increases the most. It is important to note that gradient vectors only see their local values, not the whole landscape! Also, length (size) of each vector, which indicates the steepness of the function, can be very small near local plateaus (i.e. minima or maxima).

Thus, we can simply use the aforementioned formula to find the local minima.

In 1847, Augustin-Louis Cauchy used **negative of gradients**  to develop the Gradient Descent algorithm as an **iterative** method to **minimize** a **continuous** and (ideally) **differentiable function** of **many variables**.

```python
# @title Video 2: Gradient Descent - Discussion
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Rf4y157bw", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"8s22ffAfGwI", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 2: Gradient Descent ')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Section 1.2: Gradient Descent Algorithm

Let $f(\mathbf{w}): \mathbb{R}^d \rightarrow \mathbb{R}$ be a differentiable function. Gradient Descent is an iterative algorithm for minimizing the function $f$, starting with an initial value for variables $\mathbf{w}$, taking steps of size $\eta$ (learning rate) in the direction of the negative gradient at the current point to update the variables $\mathbf{w}$.

\begin{equation}
\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \eta \nabla f (\mathbf{w}^{(t)})
\end{equation}

where $\eta > 0$ and $\nabla f (\mathbf{w})= \left( \frac{\partial f(\mathbf{w})}{\partial w_1}, ..., \frac{\partial f(\mathbf{w})}{\partial w_d} \right)$. Since negative gradients always point locally in the direction of steepest descent, the algorithm makes small steps at each point **towards** the minimum.

**Vanilla Algorithm**

---
> **inputs**: initial guess $\mathbf{w}^{(0)}$, step size $\eta > 0$, number of steps $T$

> *For* $t = 0, 2, \dots , T-1$ *do* \
$\qquad$ $\mathbf{w}^{(t+1)} = \mathbf{w}^{(t)} - \eta \nabla f (\mathbf{w}^{(t)})$\
*end*

> *return*: $\mathbf{w}^{(t+1)}$

---

Hence, all we need is to calculate the gradient of the loss function with respect to the learnable parameters (i.e. weights):

\begin{equation}
\dfrac{\partial Loss}{\partial \mathbf{w}} = \left[ \dfrac{\partial Loss}{\partial w_1}, \dfrac{\partial Loss}{\partial w_2} , ..., \dfrac{\partial Loss}{\partial w_d} \right]^{\top}
\end{equation}

### Analytical Exercise 1.2: Gradients

Given $f(x, y, z) = \tanh \left( \ln \left[1 + z \frac{2x}{sin(y)} \right] \right)$, how easy is it to derive $\dfrac{\partial f}{\partial x}$, $\dfrac{\partial f}{\partial y}$ and $\dfrac{\partial f}{\partial z}$? (*hint: you don't have to actually calculate them!*)

## Section 1.3: Computational Graphs and Backprop

```python
# @title Video 3: Computational Graph
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1c64y1B7ZG", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"2z1YX5PonV4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 3: Computational Graph ')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

 *Exercise 1.2* is an example of how overwhelming the derivation of gradients can get, as the number of variables and nested functions increases. This function is still extraordinarily simple compared to the loss functions of modern neural networks. So how can we (as well as PyTorch and similar frameworks) approach such beasts?

Let’s look at the function again:

\begin{equation}
f(x, y, z) = \tanh \left(\ln \left[1 + z \frac{2x}{sin(y)} \right] \right)
\end{equation}

We can build a so-called computational graph (shown below) to break the original function into smaller and more approachable expressions.

Starting from $x$, $y$, and $z$ and following the arrows and expressions, you would see that our graph returns the same function as $f$. It does so by calculating intermediate variables $a,b,c,d,$ and $e$. This is called the **forward pass**.

Now, let’s start from $f$, and work our way against the arrows while calculating the gradient of each expression as we go. This is called the **backward pass**, from which the **backpropagation of errors** algorithm gets its name.

By breaking the computation into simple operations on intermediate variables, we can use the chain rule to calculate any gradient:

\begin{equation}
\dfrac{\partial f}{\partial x} = \dfrac{\partial f}{\partial e}~\dfrac{\partial e}{\partial d}~\dfrac{\partial d}{\partial c}~\dfrac{\partial c}{\partial a}~\dfrac{\partial a}{\partial x} = \left( 1-\tanh^2(e) \right) \cdot \frac{1}{d+1}\cdot z \cdot \frac{1}{b} \cdot 2
\end{equation}

Conveniently, the values for $e$, $b$, and $d$ are available to us from when we did the forward pass through the graph. That is, the partial derivatives have simple expressions in terms of the intermediate variables $a,b,c,d,e$ that we calculated and stored during the forward pass.

### Analytical Exercise 1.3: Chain Rule (Optional)
For the function above, calculate the $\dfrac{\partial f}{\partial y}$ using the computational graph and chain rule.

---
#### Solution:
\begin{equation}
\dfrac{\partial f}{\partial y} = \dfrac{\partial f}{\partial e}~\dfrac{\partial e}{\partial d}~\dfrac{\partial d}{\partial c}~\dfrac{\partial c}{\partial b}~\dfrac{\partial b}{\partial y} = \left( 1-\tanh^2(e) \right) \cdot \frac{1}{d+1}\cdot z \cdot \frac{-a}{b^2} \cdot \cos(y)
\end{equation}

For more: [Calculus on Computational Graphs: Backpropagation](https://colah.github.io/posts/2015-08-Backprop/)

---
# Section 2: PyTorch AutoGrad

*Time estimate: ~30-45 mins*

```python
# @title Video 4: Auto-Differentiation
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1UP4y1s7gv", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"IBYFCNyBcF8", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 4: Auto-Differentiation ')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

Deep learning frameworks such as PyTorch, JAX, and TensorFlow come with a very efficient and sophisticated set of algorithms, commonly known as Automatic Differentiation. AutoGrad is PyTorch's automatic differentiation engine. Here we start by covering the essentials of AutoGrad, and you will learn more in the coming days.

## Section 2.1: Forward Propagation

Everything starts with the forward propagation (pass). PyTorch tracks all the instructions, as we declare the variables and operations, and it builds the graph when we call the `.backward()` pass. PyTorch rebuilds the graph every time we iterate or change it (or simply put, PyTorch uses a dynamic graph).

For gradient descent, it is only required to have the gradients of cost function with respect to the variables we wish to learn. These variables are often called "learnable / trainable parameters" or simply "parameters" in PyTorch. In neural nets, weights and biases are often the learnable parameters.

### Coding Exercise 2.1: Buiding a Computational Graph

In PyTorch, to indicate that a certain tensor contains learnable parameters, we can set the optional argument `requires_grad` to `True`. PyTorch will then track every operation using this tensor while configuring the computational graph. For this exercise, use the provided tensors to build the following graph, which implements a single neuron with scalar input and output.

```python
#add event to airtable
atform.add_event('Coding Exercise 2.1: Computational Graph ')

class SimpleGraph:
  def __init__(self, w, b):
    """Initializing the SimpleGraph

    Args:
      w (float): initial value for weight
      b (float): initial value for bias
    """
    assert isinstance(w, float)
    assert isinstance(b, float)
    self.w = torch.tensor([w], requires_grad=True)
    self.b = torch.tensor([b], requires_grad=True)

  def forward(self, x):
    """Forward pass

    Args:
      x (torch.Tensor): 1D tensor of features

    Returns:
      torch.Tensor: model predictions
    """
    assert isinstance(x, torch.Tensor)
    #################################################
    ## Implement the the forward pass to calculate prediction
    ## Note that prediction is not the loss, but the value after `tanh`
    # Complete the function and remove or comment the line below
    # raise NotImplementedError("Forward Pass `forward`")
    #################################################
    prediction = torch.tanh((x * self.w) + self.b)
    return prediction

def sq_loss(y_true, y_prediction):
  """L2 loss function

  Args:
    y_true (torch.Tensor): 1D tensor of target labels
    y_prediction (torch.Tensor): 1D tensor of predictions

  Returns:
    torch.Tensor: L2-loss (squared error)
  """
  assert isinstance(y_true, torch.Tensor)
  assert isinstance(y_prediction, torch.Tensor)
  #################################################
  ## Implement the L2-loss (squred error) given true label and prediction
  # Complete the function and remove or comment the line below
  # raise NotImplementedError("Loss function `sq_loss`")
  #################################################
  loss = (y_true - y_prediction)**2
  return loss

feature = torch.tensor([1])  # input tensor
target = torch.tensor([7])  # target tensor
## Uncomment to run
simple_graph = SimpleGraph(-0.5, 0.5)
print(f"initial weight = {simple_graph.w.item()}, "
      f"\ninitial bias = {simple_graph.b.item()}")
prediction = simple_graph.forward(feature)
square_loss = sq_loss(target, prediction)
print(f"for x={feature.item()} and y={target.item()}, "
      f"prediction={prediction.item()}, and L2 Loss = {square_loss.item()}")
```

    initial weight = -0.5,
    initial bias = 0.5
    for x=1 and y=7, prediction=0.0, and L2 Loss = 49.0

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D2_LinearDeepLearning/solutions/W1D2_Tutorial1_Solution_6668feea.py)

It is important to appreciate the fact that PyTorch can follow our operations as we arbitrarily go through classes and functions.

## Section 2.2: Backward Propagation

Here is where all the magic lies. In PyTorch, `Tensor` and `Function` are interconnected and build up an acyclic graph, that encodes a complete history of computation. Each variable has a `grad_fn` attribute that references a function that has created the Tensor (except for Tensors created by the user - these have `None` as `grad_fn`).  The example below shows that the tensor `c = a + b` is created by the `Add` operation and the gradient function is the object `<AddBackward...>`. Replace `+` with other single operations (e.g., `c = a * b` or `c = torch.sin(a)`) and examine the results.

```python
a = torch.tensor([1.0], requires_grad=True)
b = torch.tensor([-1.0], requires_grad=True)
c = a * b
print(f'Gradient function = {c.grad_fn}')
```

    Gradient function = <MulBackward0 object at 0x7ff2de0a0bd0>

For more complex functions, printing the `grad_fn` would only show the last operation, even though the object tracks all the operations up to that point:

```python
print(f'Gradient function for prediction = {prediction.grad_fn}')
print(f'Gradient function for loss = {square_loss.grad_fn}')
```

    Gradient function for prediction = <TanhBackward object at 0x7ff2de0a0550>
    Gradient function for loss = <PowBackward0 object at 0x7ff2de0a0610>

Now let's kick off the backward pass to calculate the gradients by calling `.backward()` on the tensor we wish to initiate the backpropagation from. Often, `.backward()` is called on the loss, which is the last node on the graph. Before doing that, let's calculate the loss gradients by hand:

$$\frac{\partial{loss}}{\partial{w}} = - 2 x (y_t - y_p)(1 - y_p^2)$$

$$\frac{\partial{loss}}{\partial{b}} = - 2 (y_t - y_p)(1 - y_p^2)$$

Where $y_t$ is the target (true label), and $y_p$ is the prediction (model output). We can then compare it to PyTorch gradients, which can be obtained by calling `.grad` on the relevant tensors.

**Important Notes**
* Learnable parameters (i.e. `reguires_grad` tensors) are "contagious". Let's look at a simple example: `Y = W @ X`, where `X` is the feature tensors and `W` is the weight tensor (learnable parameters, `reguires_grad`), the newly generated output tensor `Y` will be also `reguires_grad`. So any operation that is applied to `Y` will be part of the computational graph. Therefore, if we need to plot or store a tensor that is `reguires_grad`, we must first `.detach()` it from the graph by calling the `.detach()` method on that tensor.

* `.backward()` accumulates gradients in the leaf nodes (i.e., the input nodes to the node of interest). We can call `.zero_grad()` on the loss or optimizer to zero out all `.grad` attributes (see [autograd.backward](https://pytorch.org/docs/stable/autograd.html#torch.autograd.backward) for more).

* Recall that in python we can access variables and associated methods with `.method_name`. You can use the command `dir(my_object)` to observe all variables and associated methods to your object, e.g., `dir(simple_graph.w)`.

```python
# analytical gradients (remember detaching)
ana_dloss_dw = - 2 * feature * (target - prediction.detach())*(1 - prediction.detach()**2)
ana_dloss_db = - 2 * (target - prediction.detach())*(1 - prediction.detach()**2)

if simple_graph.w.grad is not None:
  simple_graph.w.grad.data.zero_()
  simple_graph.b.grad.data.zero_()
  prediction = simple_graph.forward(feature)
  square_loss = sq_loss(target, prediction)
square_loss.backward()  # first we should call the backward to build the graph
autograd_dloss_dw = simple_graph.w.grad  # we calculate the derivative w.r.t weights
autograd_dloss_db = simple_graph.b.grad  # we calculate the derivative w.r.t bias

print(ana_dloss_dw == autograd_dloss_dw)
print(ana_dloss_db == autograd_dloss_db)
```

    tensor([True])
    tensor([True])

References and more:
* [A GENTLE INTRODUCTION TO TORCH.AUTOGRAD](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)

* [AUTOMATIC DIFFERENTIATION PACKAGE - TORCH.AUTOGRAD](https://pytorch.org/docs/stable/autograd.html)

* [AUTOGRAD MECHANICS](https://pytorch.org/docs/stable/notes/autograd.html)

* [AUTOMATIC DIFFERENTIATION WITH TORCH.AUTOGRAD](https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html)

---
# Section 3: PyTorch's Neural Net module (`nn.Module`)

*Time estimate: ~30 mins*

```python
# @title Video 5: PyTorch `nn` module
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1MU4y1H7WH", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"jzTbQACq7KE", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 5: PyTorch `nn` module')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

PyTorch provides us with ready-to-use neural network building blocks, such as layers (e.g. linear, recurrent, ...), different activation and loss functions, and much more, packed in the [`torch.nn`](https://pytorch.org/docs/stable/nn.html) module. If we build a neural network using `torch.nn` layers, the weights and biases are already in `requires_grad` mode and will be registered as model parameters.

For training, we need three things:

*   **Model parameters** - Model parameters refer to all the learnable parameters of the model, which are accessible by calling `.parameters()` on the model. Please note that NOT all the `requires_grad` tensors are seen as model parameters. To create a custom model parameter, we can use [`nn.Parameter`](https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html) (*A kind of Tensor that is to be considered a module parameter*).

*   **Loss function** - The loss that we are going to be optimizing, which is often combined with regularization terms (conming up in few days).

*   **Optimizer** - PyTorch provides us with many optimization methods (different versions of gradient descent). Optimizer holds the current state of the model and by calling the `step()` method, will update the parameters based on the computed gradients.

You will learn more details about choosing the right model architecture, loss function, and optimizer later in the course.

## Section 3.1: Training loop in PyTorch

We use a regression problem to study the training loop in PyTorch.

The task is to train a wide nonlinear (using $\tanh$ activation function) neural net for a simple $\sin$ regression task. Wide neural networks are thought to be really good at generalization.

```python
# @markdown #### Generate the sample dataset
set_seed(seed=SEED)
n_samples = 32
inputs = torch.linspace(-1.0, 1.0, n_samples).reshape(n_samples, 1)
noise = torch.randn(n_samples, 1) / 4
targets = torch.sin(pi * inputs) + noise
plt.figure(figsize=(8, 5))
plt.scatter(inputs, targets, c='c')
plt.xlabel('x (inputs)')
plt.ylabel('y (targets)')
plt.show()
```

Let's define a very wide (512 neurons) neural net with one hidden layer and `Tanh` activation function.

```python
## A Wide neural network with a single hidden layer
class WideNet(nn.Module):

  def __init__(self):
    """Initializing the WideNet
    """
    n_cells = 512
    super().__init__()
    self.layers = nn.Sequential(
        nn.Linear(1, n_cells),
        nn.Tanh(),
        nn.Linear(n_cells, 1),
    )

  def forward(self, x):
    """Forward pass

    Args:
      x (torch.Tensor): 2D tensor of features

    Returns:
      torch.Tensor: model predictions
    """
    return self.layers(x)
```

We can now create an instance of our neural net and print its parameters.

```python
# creating an instance
set_seed(seed=SEED)
wide_net = WideNet()
print(wide_net)
```

    Random seed 2021 has been set.
    WideNet(
      (layers): Sequential(
        (0): Linear(in_features=1, out_features=512, bias=True)
        (1): Tanh()
        (2): Linear(in_features=512, out_features=1, bias=True)
      )
    )

```python
# Create a mse loss function
loss_function = nn.MSELoss()

# Stochstic Gradient Descent optimizer (you will learn about momentum soon)
lr = 0.003  # learning rate
sgd_optimizer = torch.optim.SGD(wide_net.parameters(), lr=lr, momentum=0.9)
```

The training process in PyTorch is interactive - you can perform training iterations as you wish and inspect the results after each iteration.

Let's perform one training iteration. You can run the cell multiple times and see how the parameters are being updated and the loss is reducing. This code block is the core of everything to come: please make sure you go line-by-line through all the commands and discuss their purpose with the pod.

```python
# Reset all gradients to zero
sgd_optimizer.zero_grad()

# Forward pass (Compute the output of the model on the features (inputs))
prediction = wide_net(inputs)

# Compute the loss
loss = loss_function(prediction, targets)
print(f'Loss: {loss.item()}')

# Perform backpropagation to build the graph and compute the gradients
loss.backward()

# Optimizer takes a tiny step in the steepest direction (negative of gradient)
# and "updates" the weights and biases of the network
sgd_optimizer.step()
```

    Loss: 0.4475176930427551

### Coding Exercise 3.1: Training Loop
Using everything we've learned so far, we ask you to complete the `train` function below.

```python
def train(features, labels, model, loss_fun, optimizer, n_epochs):

  """Training function

  Args:
    features (torch.Tensor): features (input) with shape torch.Size([n_samples, 1])
    labels (torch.Tensor): labels (targets) with shape torch.Size([n_samples, 1])
    model (torch nn.Module): the neural network
    loss_fun (function): loss function
    optimizer(function): optimizer
    n_epochs (int): number of training iterations

  Returns:
    list: record (evolution) of training losses
  """
  loss_record = []  # keeping recods of loss

  for i in range(n_epochs):
    #################################################
    ## Implement the missing parts of the training loop
    # Complete the function and remove or comment the line below
    # raise NotImplementedError("Training loop `train`")
    #################################################
    optimizer.zero_grad()  # set gradients to 0
    predictions = model(features)  # Compute model prediction (output)
    loss = loss_fun(predictions, labels)  # Compute the loss
    loss.backward()  # Compute gradients (backward pass)
    optimizer.step()  # update parameters (optimizer takes a step)

    loss_record.append(loss.item())
  return loss_record

#add event to airtable
atform.add_event('Coding Exercise 3.1: Training Loop')

set_seed(seed=2021)
epochs = 1847 # Cauchy, Exercices d'analyse et de physique mathematique (1847)
## Uncomment to run
losses = train(inputs, targets, wide_net, loss_function, sgd_optimizer, epochs)
ex3_plot(wide_net, inputs, targets, epochs, losses)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D2_LinearDeepLearning/solutions/W1D2_Tutorial1_Solution_5204c053.py)

*Example output:*

---
# Summary

In this tutorial we covered one of the most basic concepts of deep learning; the computational graph and how a network learns via gradient descent and the backpropagation algorithm. We have seen all of these using PyTorch modules and we compared the analytical solutions with the ones provided directly by the PyTorch module.

```python
# @title Video 6: Tutorial 1 Wrap-up
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Pg41177VU", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"TvZURbcnXc4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 6: Tutorial 1 Wrap-up')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

```python
# @title Airtable Submission Link
from IPython import display
display.HTML(
    f"""
  <div>
    <a href= "{atform.url()}" target="_blank">
    </a>
    </div>""" )
```

  <a href= "https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303?data=eyJmb3JtX2lkIjogImFwcG43VmRQUnNlU29NWEVHIiwgInRhYmxlX25hbWUiOiAiVzFEMl9UMSIsICJhbnN3ZXJzIjoge30sICJldmVudHMiOiBbeyJldmVudCI6ICJpbml0IiwgInRzIjogMTYyODAwODkyMi45MDg1NzM0fSwgeyJldmVudCI6ICJWaWRlbyAwOkludHJvZHVjdGlvbiIsICJ0cyI6IDE2MjgwMDg5MjMuODk5NzMxNH0sIHsiZXZlbnQiOiAiVmlkZW8gMTogR3JhZGllbnQgRGVzY2VudCIsICJ0cyI6IDE2MjgwMDg5MjQuMjA0ODMzfSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgMS4xOiBHcmFkaWVudCBWZWN0b3IiLCAidHMiOiAxNjI4MDA4OTI0LjI0MjYxMzh9LCB7ImV2ZW50IjogIlZpZGVvIDI6IEdyYWRpZW50IERlc2NlbnQgIiwgInRzIjogMTYyODAwODkyNS40NTg2OTE2fSwgeyJldmVudCI6ICJWaWRlbyAzOiBDb21wdXRhdGlvbmFsIEdyYXBoICIsICJ0cyI6IDE2MjgwMDg5MjUuNzY2ODQ1Mn0sIHsiZXZlbnQiOiAiVmlkZW8gNDogQXV0by1EaWZmZXJlbnRpYXRpb24gIiwgInRzIjogMTYyODAwODkyNi4wNjMxODk1fSwgeyJldmVudCI6ICJDb2RpbmcgRXhlcmNpc2UgMi4xOiBDb21wdXRhdGlvbmFsIEdyYXBoICIsICJ0cyI6IDE2MjgwMDg5MjYuMTIwMjU5OH0sIHsiZXZlbnQiOiAiVmlkZW8gNTogUHlUb3JjaCBgbm5gIG1vZHVsZSIsICJ0cyI6IDE2MjgwMDg5MzYuNjY3MDUwOH0sIHsiZXZlbnQiOiAiQ29kaW5nIEV4ZXJjaXNlIDMuMTogVHJhaW5pbmcgTG9vcCIsICJ0cyI6IDE2MjgwMDk2NDMuMjU1MzM5Nn0sIHsiZXZlbnQiOiAiVmlkZW8gNjogVHV0b3JpYWwgMSBXcmFwLXVwIiwgInRzIjogMTYyODAwOTY1NC42MjQ1ODl9LCB7ImV2ZW50IjogInVybCBnZW5lcmF0ZWQiLCAidHMiOiAxNjI4MDA5NjU5LjEwNjQ1OTZ9XX0%3D" target="_blank">

```python

```