 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

Note:  This material largely follows the text "Numerical Linear Algebra" by Trefethen and Bau (SIAM, 1997) and is meant as a guide and supplement to the material presented there.

```python
%matplotlib inline
%precision 3
import numpy
import matplotlib.pyplot as plt
```

# Numerical Linear Algebra

Numerical methods for linear algebra problems lies at the heart of many numerical approaches and is something we will spend some time on.  Roughly we can break down problems that we would like to solve into two general problems, solving a system of equations

$$A \mathbf{x} = \mathbf{b}$$

and solving the eigenvalue problem

$$A \mathbf{v} = \lambda \mathbf{v}.$$

We examine each of these problems separately and will evaluate some of the fundamental properties and methods for solving these problems. We will be careful in deciding how to evaluate the results of our calculations and try to gain some understanding of when and how they fail.

## General Problem Specification

The number and power of the different tools made available from the study of linear algebra makes it an invaluable field of study. Before we dive in to numerical approximations we first consider some of the pivotal problems that numerical methods for linear algebra are used to address.

For this discussion we will be using the common notation $m \times n$ to denote the dimensions of a matrix $A$.  The $m$ refers to the number of rows and $n$ the number of columns.  If a matrix is square, i.e. $m = n$, then we will use the notation that $A$ is $m \times m$.

### Systems of Equations

The first type of problem is to find the solution to a linear system of equations.  If we have $m$ equations for $m$ unknowns it can be written in matrix/vector form,

$$A \mathbf{x} = \mathbf{b}.$$

For this example $A$ is an $m \times m$ matrix, denoted as being in $\mathbb{R}^{m\times m}$, and $\mathbf{x}$ and $\mathbf{b}$ are column vectors with $m$ entries, denoted as $\mathbb{R}^m$.

#### Example: Vandermonde Matrix

We have data $(x_i, y_i), ~~ i = 1, 2, \ldots, m$ that we want to fit a polynomial of order $m-1$.  Solving the linear system $A \mathbf{p} = \mathbf{y}$ does this for us where

$$A = \begin{bmatrix}
    1 & x_1 & x_1^2 & \cdots & x_1^{m-1} \\
    1 & x_2 & x_2^2 & \cdots & x_2^{m-1} \\
    \vdots & \vdots & \vdots & & \vdots \\
    1 & x_m & x_m^2 & \cdots & x_m^{m-1}
\end{bmatrix} \quad \quad \mathbf{y} = \begin{bmatrix}
y_1 \\ y_2 \\ \vdots \\ y_m
\end{bmatrix}$$

and $\mathbf{p}$ are the coefficients of the interpolating polynomial $\mathcal{P}_N(x) = p_0 + p_1 x + p_2 x^2 + \cdots + p_m x^{m-1}$. The solution to this system satisfies $\mathcal{P}_N(x_i)=y_i$ for $i=1, 2, \ldots, m$.

#### Example: Linear least squares 1

In a similar case as above, say we want to fit a particular function (could be a polynomial) to a given number of data points except in this case we have more data points than free parameters.  In the case of polynomials this could be the same as saying we have $m$ data points but only want to fit a $n - 1$ order polynomial through the data where $n - 1 \leq m$.  One of the common approaches to this problem is to minimize the "least-squares" error between the data and the resulting function:
$$
    E = \left( \sum^m_{i=1} |y_i - f(x_i)|^2 \right )^{1/2}.
$$

But how do we do this if our matrix $A$ is now $m \times n$ and looks like
$$
    A = \begin{bmatrix}
        1 & x_1 & x_1^2 & \cdots & x_1^{n-1} \\
        1 & x_2 & x_2^2 & \cdots & x_2^{n-1} \\
        \vdots & \vdots & \vdots & & \vdots \\
        1 & x_m & x_m^2 & \cdots & x_m^{n-1}
    \end{bmatrix}?
$$

Turns out if we solve the system

$$A^T A \mathbf{x} = A^T \mathbf{b}$$

we can guarantee that the error is minimized in the least-squares sense[1](#footnoteRegression). (Although we will also show that this is not the most numerically stable way to solve this problem)

#### Example:  Linear least squares 2

Fitting a line through data that has random noise added to it.

```python
# Linear Least Squares Problem

# First define the independent and dependent variables.
N = 20
x = numpy.linspace(-1.0, 1.0, N)
y = x + numpy.random.random((N))

# Define the Vandermonde matrix based on our x-values
A = numpy.array([ numpy.ones(x.shape), x]).T
A = numpy.array([ numpy.ones(x.shape), x, x**2]).T
A
```

```python
# Determine the coefficients of the polynomial that will
# result in the smallest sum of the squares of the residual.
p = numpy.linalg.solve(numpy.dot(A.T, A), numpy.dot(A.T, y))
print("Error in slope = %s, y-intercept = %s" % (numpy.abs(p[1] - 1.0), numpy.abs(p[0] - 0.5)))
print(p)
```

```python
# Plot it out, cuz pictures are fun!
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

f = numpy.zeros(x.shape)
for i in range(len(p)):
    f += p[i] * x**i
axes.plot(x, y, 'ko')
axes.plot(x, f, 'r')
axes.set_title("Least Squares Fit to Data")
axes.set_xlabel("$x$")
axes.set_ylabel("$f(x)$ and $y_i$")
axes.grid()

plt.show()
```

### Eigenproblems

Eigenproblems come up in a variety of contexts and often are integral to many problem of scientific and engineering interest. It is such a powerful idea that it is not uncommon for us to take a problem and convert it into an eigenproblem. Here we introduce the idea and give some examples.

As a review, if $A \in \mathbb{C}^{m\times m}$ (a square matrix with complex values), a non-zero vector $\mathbf{v}\in\mathbb{C}^m$ is an **eigenvector** of $A$ with a corresponding **eigenvalue** $\lambda \in \mathbb{C}$ if

$$A \mathbf{v} = \lambda \mathbf{v}.$$

One way to interpret the eigenproblem is that we are attempting to ascertain the "action" of the matrix $A$ on some subspace of $\mathbb{C}^m$ where this action acts like scalar multiplication.  This subspace is called an **eigenspace**.

### General idea of EigenProblems

Rewriting the standard Eigen problem $A\mathbf{v}=\lambda\mathbf{v}$ for $A \in \mathbb{C}^{m\times m}$, $\mathbf{v}\in\mathbb{C}^m$ as

$$
    (A - \lambda I)\mathbf{v} = 0
$$

it becomes clear that for $\mathbf{v}$ to be non-trivial (i.e. $\neq \mathbf{0}$), requires that the matrix $(A-\lambda I)$ be singular,

This is equivalent to finding all values of $\lambda$ such that $|A-\lambda I| = 0$ (the determinant of singular matrices is always zero).  However, it can also be shown that

$$
   | A-\lambda I| = P_m(\lambda)
$$

which is a $m$th order polynomial in $\lambda$.  Thus $P_m(\lambda)=0$ implies the eigenvalues are the $m$ roots of $P$, and the **eigenspace** corresponding to $\lambda_i$ is just $N(A-\lambda_i I)$

### Solving EigenProblems

The temptation (and what) we usually teach in introductory linear algebra is to simply find the roots of $P_m(\lambda)$.  However that would be **wrong**.  The best algorithms for finding Eigenvalues are completely unrelated to rootfinding as we shall see.

#### Example

Compute the eigenspace of the matrix
$$
    A = \begin{bmatrix}
        1 & 2 \\
        2 & 1
    \end{bmatrix}
$$

Recall that we can find the eigenvalues of a matrix by computing $\det(A - \lambda I) = 0$.

In this case we have
$$\begin{aligned}
    A - \lambda I &= \begin{bmatrix}
        1 & 2 \\
        2 & 1
    \end{bmatrix} - \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} \lambda\\
    &= \begin{bmatrix}
        1 - \lambda & 2 \\
        2 & 1 - \lambda
    \end{bmatrix}.
\end{aligned}$$
The determinant of the matrix is
$$\begin{aligned}
\begin{vmatrix}
    1 - \lambda & 2 \\
    2 & 1 - \lambda
\end{vmatrix} &= (1 - \lambda) (1 - \lambda) - 2 \cdot 2 \\
&= 1 - 2 \lambda + \lambda^2 - 4 \\
&= \lambda^2 - 2 \lambda - 3.
\end{aligned}$$
This result is sometimes referred to as the characteristic equation of the matrix, $A$.

Setting the determinant equal to zero we can find the eigenvalues as
$$\begin{aligned}
    & \\
    \lambda &= \frac{2 \pm \sqrt{4 - 4 \cdot 1 \cdot (-3)}}{2} \\
    &= 1 \pm 2 \\
    &= -1 \mathrm{~and~} 3
\end{aligned}$$

The eigenvalues are used to determine the eigenvectors. The eigenvectors are found by going back to the equation $(A - \lambda I) \mathbf{v}_i = 0$ and solving for each vector.  A trick that works some of the time is to normalize each vector such that the first entry is 1 ($\mathbf{v}_1 = 1$):

$$
    \begin{bmatrix}
        1 - \lambda & 2 \\
        2 & 1 - \lambda
    \end{bmatrix} \begin{bmatrix} 1 \\ v_2 \end{bmatrix} = 0
$$

$$\begin{aligned}
    1 - \lambda + 2 v_2 &= 0 \\
    v_2 &= \frac{\lambda - 1}{2}
\end{aligned}$$

We can check this by
$$\begin{aligned}
    2 + \left(1- \lambda \frac{\lambda - 1}{2}\right) & = 0\\
    (\lambda - 1)^2 - 4 &=0
\end{aligned}$$

which by design is satisfied by our eigenvalues.  Another sometimes easier approach is to plug-in the eigenvalues to   find the Null space of $A-\lambda I$ where the eigenvectors will be a basis for $N(A-\lambda I)$.  The eigenvectors are therefore

$$\mathbf{v} = \begin{bmatrix}1 \\ -1 \end{bmatrix}, \begin{bmatrix}1 \\ 1 \end{bmatrix}.$$

Note that these are linearly independent (and because $A^T = A$, also orthogonal)

## Fundamentals

### Matrix-Vector Multiplication

One of the most basic operations we can perform with matrices is to multiply them be a vector.  This matrix-vector product $A \mathbf{x} = \mathbf{b}$ is defined as
$$
    b_i = \sum^n_{j=1} a_{ij} x_j \quad \text{where}\quad i = 1, \ldots, m
$$

### row picture
In addition to index form, we can consider matrix-vector as a sequence of inner products (dot-products between the rows of $A$ and the vector $\mathbf{x}$.
\begin{align}
    \mathbf{b} &= A \mathbf{x}, \\
     &=
    \begin{bmatrix}  \mathbf{a}_1^T \mathbf{x} \\ \mathbf{a}_2^T \mathbf{x} \\ \vdots \\ \mathbf{a}_m^T \mathbf{x}\end{bmatrix}
\end{align}
where $\mathbf{a}_i^T$ is the $i$th **row** of $A$

#### Operation Counts
This view is convenient for calculating the **Operation counts** required for to compute $A\mathbf{x}$.  If $A\in\mathbb{C}^{m\times n}$ and $\mathbf{x}\in\mathbb{C}^n$.  Then just counting the number of multiplications involved to compute $A\mathbf{x}$ is $O(??)$

### Column picture

An alternative (and entirely equivalent way) to write the matrix-vector product is as a linear combination of the columns of $A$ where each column's weighting is $x_j$.

$$
\begin{align}
    \mathbf{b} &= A \mathbf{x}, \\
    &=
    \begin{bmatrix}  &  &  &  \\  &  &  &  \\ \mathbf{a}_1 & \mathbf{a}_2 & \cdots & \mathbf{a}_n \\  &  &  &   \\  &  &  &  \end{bmatrix}
    \begin{bmatrix} x_1 \\ x_2 \\ \vdots \\ x_n \end{bmatrix}, \\
  &= x_1 \mathbf{a}_1 + x_2 \mathbf{a}_2 + \cdots + x_n \mathbf{a}_n.
\end{align}
$$

This view will be useful later when we are trying to interpret various types of matrices.

One important property of the matrix-vector product is that is a **linear** operation, also known as a **linear operator**.  This means that the for any $\mathbf{x}, \mathbf{y} \in \mathbb{C}^n$ and any $c \in \mathbb{C}$ we know that

1. $A (\mathbf{x} + \mathbf{y}) = A\mathbf{x} + A\mathbf{y}$
1. $A\cdot (c\mathbf{x}) = c A \mathbf{x}$

#### Example:  Vandermonde Matrix

In the case where we have $m$ data points and want $m - 1$ order polynomial interpolant the matrix $A$ is a square, $m \times m$, matrix as before.  Using the above interpretation the polynomial coefficients $p$ are the weights for each of the monomials that give exactly the $y$ values of the data.

#### Example:  Numerical matrix-vector multiply

Write a matrix-vector multiply function and check it with the appropriate `numpy` routine.  Also verify the linearity of the matrix-vector multiply.

```python
#A x = b
#(m x n) (n x 1) = (m x 1)
def matrix_vector_product(A, x):
    m, n = A.shape
    b = numpy.zeros(m)
    for i in range(m):
        for j in range(n):
            b[i] += A[i, j] * x[j]
    return b

m = 4
n = 3
A = numpy.random.uniform(size=(m,n))
x = numpy.random.uniform(size=(n))
y = numpy.random.uniform(size=(n))
c = numpy.random.uniform()
b = matrix_vector_product(A, x)
print(numpy.allclose(b, numpy.dot(A, x)))
print(numpy.allclose(matrix_vector_product(A, (x + y)), matrix_vector_product(A, x) + matrix_vector_product(A, y)))
print(numpy.allclose(matrix_vector_product(A, c * x), c*matrix_vector_product(A, x)))
```

### Matrix-Matrix Multiplication

The matrix product with another matrix $A B = C$ is defined as
$$
    c_{ij} = \sum^m_{k=1} a_{ik} b_{kj} = \mathbf{a}_i^T\mathbf{b}_j
$$

i.e. each component of $C$ is a dot-product between the $i$th row of $A$ and the $j$th column of $B$

As with matrix-vector multiplication, Matrix-matrix multiplication can be thought of multiple ways

* $m\times p$ dot products
* $A$ multiplying the columns of $B$
$$
    C = AB = \begin{bmatrix}
                A\mathbf{b}_1 & A\mathbf{b}_2 & \ldots & A\mathbf{b}_p\\
             \end{bmatrix}
$$
* Linear combinations of the rows of $B$
$$
C = AB = \begin{bmatrix}
                \mathbf{a}_1^T B \\ \mathbf{a}_2^T B \\ \vdots \\ \mathbf{a}_m^T B\\
             \end{bmatrix}
$$

### Questions
* What are the dimensions of $A$ and $B$ so that the multiplication works?
* What are the Operations Counts for Matrix-Matrix Multiplication?
* Comment on the product $\mathbf{c}=(AB)\mathbf{x}$ vs. $\mathbf{d} = A(B\mathbf{x})$

#### Example:  Outer Product

The product of two vectors $\mathbf{u} \in \mathbb{C}^m$ and $\mathbf{v} \in \mathbb{C}^n$ is a $m \times n$ matrix where the columns are the vector $u$ multiplied by the corresponding value of $v$:
$$
\begin{align}
    \mathbf{u} \mathbf{v}^T &=
    \begin{bmatrix}  u_1 \\ u_2 \\ \vdots \\ u_n  \end{bmatrix}
    \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}, \\
    & = \begin{bmatrix} v_1u_1 & \cdots & v_n u_1 \\ \vdots &  & \vdots \\ v_1 u_m & \cdots & v_n u_m \end{bmatrix}.
\end{align}
$$

It is useful to think of these as operations on the column vectors, and an equivalent way to express this relationship is
$$
\begin{align}
    \mathbf{u} \mathbf{v}^T &=
    \begin{bmatrix}  \\ \mathbf{u} \\ \\  \end{bmatrix}
    \begin{bmatrix} v_1 & v_2 & \cdots & v_n \end{bmatrix}, \\
    &=
    \begin{bmatrix}  &  &  &  \\  &  &  &  \\ \mathbf{u}v_1  & \mathbf{u} v_2  & \cdots & \mathbf{u} v_n  \\  &  &  & \\  &  &  &  \end{bmatrix}, \\
    & = \begin{bmatrix} v_1u_1 & \cdots & v_n u_1 \\ \vdots &  & \vdots \\ v_1 u_m & \cdots & v_n u_m \end{bmatrix}.
\end{align}
$$

### rank 1 updates

We call any matrix of the form $\mathbf{u}\mathbf{v}^T$ a "rank one matrix"  ( because its rank r=?).  These sort of matrix operations are very common in numerical algorithms for orthogonalization, eigenvalues and the original page-rank algorithm for google.  Again, the order of operations is critical.

Comment on the difference in values and operation counts between

$$
    \mathbf{y} = (\mathbf{u}\mathbf{v}^T)\mathbf{x}
$$

and
$$
    \mathbf{y}' = \mathbf{u}(\mathbf{v}^T\mathbf{x})
$$
for $\mathbf{u}$, $\mathbf{v}$, $\mathbf{x}$, $\mathbf{y}$, $\mathbf{y}'\in\mathbb{R}^n$,

#### Example:  Upper Triangular Multiplication

Consider the multiplication of a matrix $A \in \mathbb{C}^{m\times n}$ and the **upper-triangular** matrix $R$ defined as the $n \times n$ matrix with entries $r_{ij} = 1$ for $i \leq j$ and $r_{ij} = 0$ for $i > j$.  The product can be written as
$$
    \begin{bmatrix}  \\  \\ \mathbf{b}_1 & \cdots & \mathbf{b}_n \\ \\ \\ \end{bmatrix} = \begin{bmatrix} \\ \\  \mathbf{a}_1 & \cdots & \mathbf{a}_n \\ \\ \\ \end{bmatrix} \begin{bmatrix} 1 & \cdots & 1 \\  & \ddots & \vdots \\  &  & 1 \end{bmatrix}.
$$

The columns of $B$ are then
$$
    \mathbf{b}_j = A \mathbf{r}_j = \sum^j_{k=1} \mathbf{a}_k
$$
so that $\mathbf{b}_j$ is the sum of the first $j$ columns of $A$.

#### Example: Write Matrix-Matrix Multiplication

Write a function that computes matrix-matrix multiplication and demonstrate the following properties:
1. $A (B + C) = AB + AC$ (for square matrices))
1. $A (cB) = c AB$ where $c \in \mathbb{C}$
1. $AB \neq BA$ in general

```python
def matrix_matrix_product(A, B):
    C = numpy.zeros((A.shape[0], B.shape[1]))
    for i in range(A.shape[0]):
        for j in range(B.shape[1]):
            for k in range(A.shape[1]):
                C[i, j] += A[i, k] * B[k, j]
    return C

m = 4
n = 4
p = 4
A = numpy.random.uniform(size=(m, n))
B = numpy.random.uniform(size=(n, p))
C = numpy.random.uniform(size=(m, p))
c = numpy.random.uniform()
print(numpy.allclose(matrix_matrix_product(A, B), numpy.dot(A, B)))
print(numpy.allclose(matrix_matrix_product(A, (B + C)), matrix_matrix_product(A, B) + matrix_matrix_product(A, C)))
print(numpy.allclose(matrix_matrix_product(A, c * B), c*matrix_matrix_product(A, B)))
print(numpy.allclose(matrix_matrix_product(A, B), matrix_matrix_product(B, A)))
```

#### NumPy Products

NumPy and SciPy contain routines that are optimized to perform matrix-vector and matrix-matrix multiplication.  Given two `ndarray`s you can take their product by using the `dot` function.

```python
n = 10
m = 5
# Matrix vector with identity
A = numpy.identity(n)
x = numpy.random.random(n)
print(numpy.allclose(x, numpy.dot(A, x)))
print(x-A.dot(x))
print(A*x)

# Matrix vector product
A = numpy.random.random((m, n))
print(numpy.dot(A, x))

# Matrix matrix product
B = numpy.random.random((n, m))
print(numpy.dot(A, B))

```

### Range and Null-Space

#### Range
- The **range** of a matrix $A \in \mathbb R^{m \times n}$ (similar to any function), denoted as $\text{range}(A)$, is the set of vectors that can be expressed as $A x$ for $x \in \mathbb R^n$.
- We can also then say that that $\text{range}(A)$ is the space **spanned** by the columns of $A$.  In other words the columns of $A$ provide a basis for $\text{range}(A)$, also called the **column space** of the matrix $A$.
- $C(A)$ controls the **existence** of solutions to $A\mathbf{x}=\mathbf{b}$

#### Null-Space
- Similarly the **null-space** of a matrix $A$, denoted $\text{null}(A)$ is the set of vectors $x$ that satisfy $A x = 0$.
- $N(A)$ controls the **uniqueness** of solutions to $A\mathbf{x}=\mathbf{b}$
- A similar concept is the **rank** of the matrix $A$, denoted as $\text{rank}(A)$, is the dimension of the column space.  A matrix $A$ is said to have **full-rank** if $\text{rank}(A) = \min(m, n)$.  This property also implies that the matrix mapping is **one-to-one**.

### Inverse

A **non-singular** or **invertible** matrix is characterized as a matrix with full-rank.  This is related to why we know that the matrix is one-to-one, we can use it to transform a vector $x$ and using the inverse, denoted $A^{-1}$, we can map it back to the original matrix.  The familiar definition of this is
\begin{align*}
    A \mathbf{x} &= \mathbf{b}, \\
    A^{-1} A \mathbf{x} & = A^{-1} \mathbf{b}, \\
    x &=A^{-1} \mathbf{b}.
\end{align*}
Since $A$ has full rank, its columns form a basis for $\mathbb{R}^m$ and the vector $\mathbf{b}$ must be in the column space of $A$.

There are a number of important properties of a non-singular matrix A.  Here we list them as the following equivalent statements
1. $A$ has an inverse $A^{-1}$
1. $\text{rank}(A) = m$
1. $\text{range}(A) = \mathbb{C}^m$
1. $\text{null}(A) = {0}$
1. 0 is not an eigenvalue of $A$
1. $\text{det}(A) \neq 0$

#### Example:  Properties of invertible matrices

Show that given an invertible matrix that the rest of the properties hold.  Make sure to search the `numpy` packages for relevant functions.

```python
m = 3
for n in range(100):
    A = numpy.random.uniform(size=(m, m))
    if numpy.linalg.det(A) != 0:
        break

print(numpy.dot(numpy.linalg.inv(A), A))
print(numpy.linalg.matrix_rank(A))
print("N(A)= {}".format(numpy.linalg.solve(A, numpy.zeros(m))))
print("Eigenvalues = {}".format(numpy.linalg.eigvals(A)))
```

### Orthogonal Vectors and Matrices

Orthogonality is a very important concept in linear algebra that forms the basis of many of the modern methods used in numerical computations.

Two vectors are said to be *orthogonal* if their **inner-product** or **dot-product** defined as
$$
    < \mathbf{x}, \mathbf{y} > \equiv (\mathbf{x}, \mathbf{y}) \equiv \mathbf{x}^T\mathbf{y} \equiv \mathbf{x} \cdot \mathbf{y} = \sum^m_{i=1} x_i y_i = 0
$$
Here we have shown the various notations you may run into (the inner-product is in-fact a general term for a similar operation for mathematical objects such as functions).

If $\langle \mathbf{x},\mathbf{y} \rangle = 0$ then we say $\mathbf{x}$ and $\mathbf{y}$ are orthogonal.  The reason we use this terminology is that the inner-product of two vectors can also be written in terms of the angle between them where
$$
    \cos \theta = \frac{\langle \mathbf{x}, \mathbf{y} \rangle}{||\mathbf{x}||_2~||\mathbf{y}||_2}
$$
and $||\mathbf{x}||_2$ is the Euclidean ($\ell^2$) norm of the vector $\mathbf{x}$.

We can write this in terms of the inner-product as well as
$$
    ||\mathbf{x}||_2^2 = \langle \mathbf{x}, \mathbf{x} \rangle = \mathbf{x}^T\mathbf{x} = \sum^m_{i=1} |x_i|^2.
$$

$$
    ||\mathbf{x}||_2 = \sqrt{\langle \mathbf{x}, \mathbf{x} \rangle}
$$

The generalization of the inner-product to complex spaces is defined as
$$
    \langle x, y \rangle = \sum^m_{i=1} x_i^* y_i
$$
where $x_i^*$ is the complex-conjugate of the value $x_i$.

#### Orthonormality

Taking this idea one step further we can say a set of vectors $\mathbf{x} \in X$ are orthogonal to $\mathbf{y} \in Y$ if $\forall \mathbf{x},\mathbf{y}$ $< \mathbf{x}, \mathbf{y} > = 0$.  If $\forall \mathbf{x},\mathbf{y}$ $||\mathbf{x}|| = 1$ and $||\mathbf{y}|| = 1$ then they are also called orthonormal. Note that we dropped the 2 as a subscript to the notation for the norm of a vector. Later we will explore other ways to define a norm of a vector other than the Euclidean norm defined above.

Another concept that is related to orthogonality is linear-independence.  A set of vectors $\mathbf{x} \in X$ are **linearly independent** if $\forall \mathbf{x} \in X$ that each $\mathbf{x}$ cannot be written as a linear combination of the other vectors in the set $X$.

An equivalent statement is that given a set of $n$ vectors $\mathbf{x}_i$,  the only  set of scalars $c_i$ that satisfies
$$
    \sum_{i=1}^n c_i\mathbf{x}_i = \mathbf{0}
$$
is if $c_i=0$ for all $i\in[1,n]$

This can be related directly through the idea of projection.  If we have a set of vectors $\mathbf{x} \in X$ we can project another vector $\mathbf{v}$ onto the vectors in $X$ by using the inner-product.  This is especially powerful if we have a set of **orthogonal** vectors $X$, which are said to **span** a space (or provide a **basis** for a space), s.t. any vector in the space spanned by $X$ can be expressed as a linear combination of the basis vectors $X$
$$
    \mathbf{v} = \sum^n_{i=1} \, \langle \mathbf{v}, \mathbf{x}_i \rangle \, \mathbf{x}_i.
$$
Note if $\mathbf{v} \in X$ that
$$
    \langle \mathbf{v}, \mathbf{x}_i \rangle = 0 \quad \forall \mathbf{x}_i \in X \setminus \mathbf{v}.
$$

Looping back to matrices, the column space of a matrix is spanned by its linearly independent columns.  Any vector $v$ in the column space can therefore be expressed via the equation above.  A special class of matrices are called **unitary** matrices when complex-valued and **orthogonal** when purely real-valued if the columns of the matrix are orthonormal to each other.  Importantly this implies that for a unitary matrix $Q$ we know the following

1. $Q^* = Q^{-1}$
1. $Q^*Q = I$

where $Q^*$ is called the **adjoint** of $Q$.  The adjoint is defined as the transpose of the original matrix with the entries being the complex conjugate of each entry as the notation implies.

As an example if we have the matrix
$$
\begin{aligned}
    Q &= \begin{bmatrix} q_{11} & q_{12} \\ q_{21} & q_{22} \\ q_{31} & q_{32} \end{bmatrix} \quad \text{then} \\
    Q^* &= \begin{bmatrix} q^*_{11} & q^*_{21} & q^*_{31} \\ q^*_{12} & q^*_{22} & q^*_{32} \end{bmatrix}
\end{aligned}
$$
The important part of being an unitary matrix is that the projection onto the column space of the matrix $Q$ preserves geometry in an Euclidean sense, i.e. preserves the Cartesian distance.

### Vector Norms

Norms (and also measures) provide a means for measure the "size" or distance in a space.  In general a norm is a function, denoted by $||\cdot||$, that maps $\mathbb{C}^m \rightarrow \mathbb{R}$.  In other words we stick in a multi-valued object and get a single, real-valued number out the other end.  All norms satisfy the properties:

1. $~~~~||\mathbf{x}|| \geq 0$
1. $~~~~||\mathbf{x}|| = 0$ only if $\mathbf{x} = \mathbf{0}$
1. $$||\mathbf{x} + \mathbf{y}||\leq ||\mathbf{x}|| + ||\mathbf{y}||$$ (triangle inequality)
1. $~~~||c \mathbf{x}|| = |c| ~ ||\mathbf{x}||$ where $c \in \mathbb{C}$

There are a number of relevant norms that we can define beyond the Euclidean norm, also know as the 2-norm or $\ell_2$ norm:

1. $\ell_1$ norm:
$$
    ||\mathbf{x}||_1 = \sum^m_{i=1} |x_i|,
$$
1. $\ell_2$ norm:
$$
    ||\mathbf{x}||_2 = \left( \sum^m_{i=1} |x_i|^2 \right)^{1/2},
$$

3. $\ell_p$ norm:
$$
    ||\mathbf{x}||_p = \left( \sum^m_{i=1} |x_i|^p \right)^{1/p}, \quad \quad 1 \leq p < \infty,
$$
1. $\ell_\infty$ norm:
$$
    ||\mathbf{x}||_\infty = \max_{1\leq i \leq m} |x_i|,
$$
1. weighted $\ell_p$ norm:
$$
    ||\mathbf{x}||_{W_p} = \left( \sum^m_{i=1} |w_i x_i|^p \right)^{1/p}, \quad \quad 1 \leq p < \infty,
$$

These are also related to other norms denoted by capital letters ($L_2$ for instance).  In this case we use the lower-case notation to denote finite or discrete versions of the infinite dimensional counterparts.

#### Example:  Comparisons Between Norms

Compute the norms given some vector $\mathbf{x}$ and compare their values.  Verify the properties of the norm for one of the norms.

```python
def pnorm(x, p):
    """ return the vector p norm of a vector
    parameters:
    -----------
    x: numpy array
        vector
    p: float or numpy.inf
        value of p norm such that ||x||_p = (sum(|x_i|^p))^{1/p} for p< inf
        for infinity norm return max(abs(x))
    returns:
    --------
    pnorm: float
        pnorm of x
    """
    if p == numpy.inf:
        norm = numpy.max(numpy.abs(x))
    else:
        norm = numpy.sum(numpy.abs(x)**p)**(1./p)
    return norm
```

```python
m = 10
p = 4
x = numpy.random.uniform(size=m)
ell_1 = pnorm(x, 1)
ell_2 = pnorm(x, 2)
ell_p = pnorm(x, p)
ell_infty = pnorm(x, numpy.inf)

print('x = {}'.format(x))
print()
print("L_1   = {}\nL_2   = {}\nL_{}   = {}\nL_inf = {}".format(ell_1, ell_2, p, ell_p, ell_infty))

y = numpy.random.uniform(size=m)
print()
print("Properties of norms:")

print('y = {}\n'.format(y))
p = 2
print('||x+y||_{p}         = {nxy}\n||x||_{p} + ||y||_{p} = {nxny}'.format(
    p=p,nxy=pnorm(x+y, p), nxny=pnorm(x, p) + pnorm(y, p)))
c = 0.1
print('||c x||_{} = {}'.format(p,pnorm(c * x, p)))
print(' c||x||_{} = {}'.format(p,c * pnorm(x, p)))

```

### Matrix Norms

The most direct way to consider a matrix norm is those induced by a vector-norm.  Given a vector norm, we can define a matrix norm as the smallest number $C$ that satisfies the inequality
$$
    ||A \mathbf{x}||_{m} \leq C ||\mathbf{x}||_{n}.
$$
or as the supremum of the ratios so that
$$
    C = \sup_{\mathbf{x}\in\mathbb{C}^n ~ \mathbf{x}\neq\mathbf{0}} \frac{||A \mathbf{x}||_{m}}{||\mathbf{x}||_n}.
$$

Noting that $||A \mathbf{x}||$ lives in the column space and $||\mathbf{x}||$ on the domain we can think of the matrix norm as the "size" of the matrix that maps the domain to the range.  Also noting that if $||\mathbf{x}||_n = 1$ we also satisfy the condition we can write the induced matrix norm as
$$
    ||A||_{(m,n)} = \sup_{\mathbf{x} \in \mathbb{C}^n ~ ||\mathbf{x}||_{n} = 1} ||A \mathbf{x}||_{m}.
$$

This definition has a **geometric interpretation**.  The set of all $\mathbf{x}$ such that $||\mathbf{x}||_n = 1$ is the "unit sphere" in $\mathbb{C}^n$.  So the induced matrix norm is the largest vector in the deformed "sphere" and measures how much the matrix distorts the unit sphere.

#### Example: Induced Matrix Norms

Consider the matrix
$$
    A = \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix}.
$$
Compute the induced-matrix norm of $A$ for the vector norms $\ell_2$ and $\ell_\infty$.

$\ell^2$: For both of the requested norms the unit-length vectors $[1, 0]$ and $[0, 1]$ can be used to give an idea of what the norm might be and provide a lower bound.

$$
    ||A||_2 = \sup_{x \in \mathbb{R}^n} \left( ||A \cdot [1, 0]^T||_2, ||A \cdot [0, 1]^T||_2 \right )
$$

computing each of the norms we have

$$\begin{aligned}
    \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix} &= \begin{bmatrix} 1 \\ 0 \end{bmatrix} \\
    \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix} \cdot \begin{bmatrix} 0 \\ 1 \end{bmatrix} &= \begin{bmatrix} 2 \\ 2 \end{bmatrix}
\end{aligned}$$

which translates into the norms $||A \cdot [1, 0]^T||_2 = 1$ and $||A \cdot [0, 1]^T||_2 = 2 \sqrt{2}$.  This implies that the $\ell_2$ induced matrix norm of $A$ is at least $||A||_{2} = 2 \sqrt{2} \approx 2.828427125$.

The exact value of $||A||_2$ can be computed using the spectral radius defined as
$$
    \rho(A) = \max_{i} |\lambda_i|,
$$
where $\lambda_i$ are the eigenvalues of $A$.  With this we can compute the $\ell_2$ norm of $A$ as
$$
    ||A||_2 = \sqrt{\rho(A^\ast A)}
$$

Computing the norm again here we find
$$
    A^\ast A = \begin{bmatrix} 1 & 0 \\ 2 & 2 \end{bmatrix} \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix} = \begin{bmatrix} 1 & 2 \\ 2 & 8 \end{bmatrix}
$$
which has eigenvalues
$$
    \lambda = \frac{1}{2}\left(9 \pm \sqrt{65}\right )
$$
so $||A||_2 \approx 2.9208096$.

The actual  induced 2-norm of a matrix can be derived using the Singular Value Decomposition (SVD) and is simply the largest singular value $\sigma_1$.

**Proof**:
Given that every Matrix $A\in\mathbb{C}^{m\times n}$ can be factored into its SVD (see notebook 10.1):

$$
    A = U\Sigma V^*
$$

where $U\in\mathbb{C}^{m\times m}$ and $V\in\mathbb{C}^{n\times n}$ are unitary matrices with the property $U^*U=I$ and $V^*V=I$ (of their respective sizes) and $\Sigma$ is a real diagonal matrix of singular values $\sigma_1 \geq\sigma_2\geq...\sigma_n\geq 0$.

Then the 2-norm squared of a  square matrix is
$$
    ||A||^2_2 = \sup_{\mathbf{x} \in \mathbb{C}^n ~ ||\mathbf{x}||_2 = 1} ||A \mathbf{x}||_2^2 = \mathbf{x}^TA^*A\mathbf{x}
$$
but $A^*A = V\Sigma^2V^*$ so

\begin{align}
    ||A \mathbf{x}||_2^2 &= \mathbf{x}^*V\Sigma^2V^*\mathbf{x} \\
                         &= \mathbf{y}^*\Sigma^2\mathbf{y} \quad\mathrm{where}\quad \mathbf{y}=V^*\mathbf{x}\\
                         &= \sum_{i=1}^n \sigma_i^2|y_i|^2\\
                         &\leq \sigma_1^2\sum_{i=1}^n |y_i|^2 = \sigma_i^2||\mathbf{y}||_2\\
\end{align}

but if $||\mathbf{x}||_2 = 1$ (i.e. is a unit vector), then so is $\mathbf{y}$ because unitary matrices don't change the length of vectors.  So it follows that
$$
    ||A||_2 = \sigma_1
$$

```python
A = numpy.array([[1, 2], [0, 2]])

#calculate the SVD(A)
U, S, Vt = numpy.linalg.svd(A)

print('Singular_values = {}'.format(S))
print('||A||_2 = {}'.format(S.max()))
print('||A||_2 = {}'.format(numpy.linalg.norm(A, ord=2)))

# more fun facts about the SVD
#print(U.T.dot(U))
#print(Vt.T.dot(Vt))
#print(A - numpy.dot(U,numpy.dot(numpy.diag(S),Vt)))

```

####  Other useful norms of a  Matrix

The 2-norm of a matrix can be expensive to compute,  however there are other norms that are equivalent that can be directly computed from  the components of $A$.  For example

* The induced 1-norm is simply max of the 1-norm of the **columns** of $A$

$$
||A \mathbf{x}||_1 = || \sum^n_{j=1} x_j \mathbf{a}_j ||_1 \leq \sum^n_{j=1} |x_j| ||\mathbf{a}_j||_1 \leq \max_{1\leq j\leq n} ||\mathbf{a}_j||_1 ||\mathbf{x}||_1 =  \max_{1\leq j\leq n} ||\mathbf{a}_j||_1
$$

* The induce $\infty$-norm is simply the max of the 1-norm of **rows** of $A$

$$
||A \mathbf{x}||_\infty = \max_{1 \leq i \leq m} | \mathbf{a}^*_i \mathbf{x} | \leq \max_{1 \leq i \leq m} ||\mathbf{a}^*_i||_1
$$
because the largest unit vector on the unit sphere in the $\infty$ norm is a vector of 1's.

####  Example:

$$
    A = \begin{bmatrix} 1 & 2 \\ 0 & 2 \end{bmatrix}.
$$

$$ ||A||_1 = 4, \quad ||A||_\infty = 3$$

```python
# Calculate the 1-norm of A
normA_1 = numpy.max(numpy.sum(numpy.abs(A), axis=0))
print('||A||_1 = {}'.format(normA_1))
print('||A||_1 = {}'.format(numpy.linalg.norm(A, ord=1)))
# calculate the infinity norm of A
normA_inf = numpy.max(numpy.sum(numpy.abs(A), axis=1))
print('||A||_inf = {}'.format(normA_inf))
print('||A||_inf = {}'.format(numpy.linalg.norm(A, ord=numpy.inf)))
```

One of the most useful ways to think about matrix norms is as a transformation of a unit-ball to an ellipse.  Depending on the norm in question, the norm will be some combination of the resulting ellipse.

```python
A = numpy.array([[1, 2], [0, 2]])

```

#### 2-Norm

```python
# ============
# 2-norm

# Unit-ball
fig = plt.figure()
fig.suptitle("2-Norm: $||A||_2 = ${:3.4f}".format(numpy.linalg.norm(A,ord=2)),fontsize=16)
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
axes.add_artist(plt.Circle((0.0, 0.0), 1.0, edgecolor='r', facecolor='none'))
draw_unit_vectors(axes, numpy.eye(2))
axes.set_title("Unit Ball")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)

# Image
# Compute some geometry
u, s, v = numpy.linalg.svd(A)
theta = numpy.empty(A.shape[0])
ellipse_axes = numpy.empty(A.shape)
theta[0] = numpy.arccos(u[0][0]) / numpy.linalg.norm(u[0], ord=2)
theta[1] = theta[0] - numpy.pi / 2.0
for i in range(theta.shape[0]):
    ellipse_axes[0, i] = s[i] * numpy.cos(theta[i])
    ellipse_axes[1, i] = s[i] * numpy.sin(theta[i])

axes = fig.add_subplot(1, 2, 2, aspect='equal')
axes.add_artist(patches.Ellipse((0.0, 0.0), 2 * s[0], 2 * s[1], theta[0] * 180.0 / numpy.pi,
                                edgecolor='r', facecolor='none'))
for i in range(A.shape[0]):
    axes.arrow(0.0, 0.0, ellipse_axes[0, i] - head_length * numpy.cos(theta[i]),
                         ellipse_axes[1, i] - head_length * numpy.sin(theta[i]),
                         head_width=head_width, color='k')
draw_unit_vectors(axes, A, head_width=0.2)
axes.set_title("Images Under A")
axes.set_xlim((-s[0] + 0.1, s[0] + 0.1))
axes.set_ylim((-s[0] + 0.1, s[0] + 0.1))
axes.grid(True)

plt.show()
```

#### 1-Norm

```python
# Note: that this code is a bit fragile to angles that go beyond pi
# due to the use of arccos.

import matplotlib.patches as patches

def draw_unit_vectors(axes, A, head_width=0.1):
    head_length = 1.5 * head_width
    image_e = numpy.empty(A.shape)
    angle = numpy.empty(A.shape[0])
    image_e[:, 0] = numpy.dot(A, numpy.array((1.0, 0.0)))
    image_e[:, 1] = numpy.dot(A, numpy.array((0.0, 1.0)))
    for i in range(A.shape[0]):
        angle[i] = numpy.arccos(image_e[0, i] / numpy.linalg.norm(image_e[:, i], ord=2))
        axes.arrow(0.0, 0.0, image_e[0, i] - head_length * numpy.cos(angle[i]),
                             image_e[1, i] - head_length * numpy.sin(angle[i]),
                             head_width=head_width, color='b', alpha=0.5)

head_width = 0.2
head_length = 1.5 * head_width
# ============
# 1-norm
# Unit-ball
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
fig.suptitle("1-Norm: $||A||_1 = {}$".format(numpy.linalg.norm(A,ord=1)), fontsize=16)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
axes.plot((1.0, 0.0, -1.0, 0.0, 1.0), (0.0, 1.0, 0.0, -1.0, 0.0), 'r')
draw_unit_vectors(axes, numpy.eye(2))
axes.set_title("Unit Ball")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)

# Image
axes = fig.add_subplot(1, 2, 2, aspect='equal')
axes.plot((1.0, 2.0, -1.0, -2.0, 1.0), (0.0, 2.0, 0.0, -2.0, 0.0), 'r')
draw_unit_vectors(axes, A, head_width=0.2)

axes.set_title("Images Under A")
axes.grid(True)

plt.show()
```

#### $\infty$-Norm

```python
# ============
# infty-norm
# Unit-ball
fig = plt.figure()
fig.suptitle("$\infty$-Norm: $||A||_\infty = {}$".format(numpy.linalg.norm(A,ord=numpy.inf)),fontsize=16)
fig.set_figwidth(fig.get_figwidth() * 2)

axes = fig.add_subplot(1, 2, 1, aspect='equal')
axes.plot((1.0, -1.0, -1.0, 1.0, 1.0), (1.0, 1.0, -1.0, -1.0, 1.0), 'r')
draw_unit_vectors(axes, numpy.eye(2))
axes.set_title("Unit Ball")
axes.set_xlim((-1.1, 1.1))
axes.set_ylim((-1.1, 1.1))
axes.grid(True)

# Image
# Geometry - Corners are A * ((1, 1), (1, -1), (-1, 1), (-1, -1))
# Symmetry implies we only need two.  Here we just plot two
u = numpy.empty(A.shape)
u[:, 0] = numpy.dot(A, numpy.array((1.0, 1.0)))
u[:, 1] = numpy.dot(A, numpy.array((-1.0, 1.0)))
theta[0] = numpy.arccos(u[0, 0] / numpy.linalg.norm(u[:, 0], ord=2))
theta[1] = numpy.arccos(u[0, 1] / numpy.linalg.norm(u[:, 1], ord=2))

axes = fig.add_subplot(1, 2, 2, aspect='equal')
axes.plot((3, 1, -3, -1, 3), (2, 2, -2, -2, 2), 'r')
for i in range(A.shape[0]):
    axes.arrow(0.0, 0.0, u[0, i] - head_length * numpy.cos(theta[i]),
                         u[1, i] - head_length * numpy.sin(theta[i]),
                         head_width=head_width, color='k')

draw_unit_vectors(axes, A, head_width=0.2)
axes.set_title("Images Under A")
axes.set_xlim((-4.1, 4.1))
axes.set_ylim((-3.1, 3.1))
axes.grid(True)

plt.show()
```

#### Cauchy-Schwarz and Hölder Inequalities

Computing matrix norms where $p \neq 1$ or $\infty$ is more difficult unfortunately.  We have a couple of tools that can be useful however.

 - **Cauchy-Schwarz Inequality**:  For the special case where $p=q=2$, for any vectors $\mathbf{x}$ and $\mathbf{y}$
$$
    |\mathbf{x}^*\mathbf{y}| \leq ||\mathbf{x}||_2 ||\mathbf{y}||_2
$$
 - **Hölder's Inequality**:  Turns out this holds in general if given a $p$ and $q$ that satisfy $1/p + 1/q = 1$ with $1 \leq p, q \leq \infty$

$$
    |\mathbf{x}^*\mathbf{y}| \leq ||\mathbf{x}||_p ||\mathbf{y}||_q.
$$

**Note**: this is essentially what we used in the proof of the $\infty-$norm with $p=1$ and $q=\infty$

#### General Matrix Norms (induced and  non-induced)

In general matrix-norms have the following properties whether they are induced from a vector-norm or not:
1. $||A|| \geq 0$ and $||A|| = 0$ only if $A = 0$
1. $||A + B|| \leq ||A|| + ||B||$ (Triangle Inequality)
1. $||c A|| = |c| ||A||$

The most widely used matrix norm not induced by a vector norm is the **Frobenius norm** defined by
$$
    ||A||_F = \left( \sum^m_{i=1} \sum^n_{j=1} |A_{ij}|^2 \right)^{1/2}.
$$

#### Invariance under unitary multiplication

One important property of the matrix 2-norm (and Frobenius norm) is that multiplication by a unitary matrix does not change the product (kind of like multiplication by 1).  In general for any $A \in \mathbb{C}^{m\times n}$ and unitary matrix $Q \in \mathbb{C}^{m \times m}$ we have
\begin{align*}
    ||Q A||_2 &= ||A||_2 \\ ||Q A||_F &= ||A||_F.
\end{align*}

1 http://www.utstat.toronto.edu/~brunner/books/LinearModelsInStatistics.pdf


```python
import numpy
```

8. Out of a random sample of 50 Americans, zero report having ever held political office. From this information, give a 95% confidence interval for the proportion of Americans who have ever held political office.

#### Response

Use the Agresti-Coull interval, with adjustment *z = 2*

$
\begin{align}
\tilde{n} = n + z^2
\end{align}
$

$
\begin{align}
p* = \dfrac{1}{\tilde{n}}(X + \dfrac{z^2}{2})
\end{align}
$

$
\begin{align}
adjustedse = 1.96 * \sqrt{\dfrac{\hat{p}_*(1-\hat{p}_*)}{\tilde{n}}}
\end{align}
$

```python
ssstar = 50 + 2**2 # add z**2, where z=2
ac = 0 + 2 # add 2
se = np.sqrt(((ssstar - ac)/ss * (ac)/ss) / ss)
# Between 0 and 10%
max(0.04 - 1.96 * se, 0), 0.04 + 1.96 * se
```

    (0, 0.09653504399927534)

```python

```


# Exact Wald `GiRaFFEfood` Initial Data for `GiRaFFE`

## Author: Zach Etienne & Patrick Nelson
### Formatting improvements courtesy Brandon Clark

[comment]: <> (Abstract: TODO)

### NRPy+ Source Code for this module: [GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Exact_Wald.py](../../edit/in_progress/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Exact_Wald.py)

**Notebook Status:**  Validated

**Validation Notes:** This tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation1). The initial data has validated against the original `GiRaFFE`, as documented [here](Tutorial-Start_to_Finish_UnitTest-GiRaFFEfood_NRPy.ipynb).

## Introduction:
With the `GiRaFFE` evolution thorn constructed, we now need to "feed" our giraffe with initial data to evolve. There are several different choices of initial data we can use here; here, we will only be implementing the "Exact Wald" initial data, given by Table 3 in [the original paper](https://arxiv.org/pdf/1704.00599.pdf):
\begin{align}
A_{\phi} &= \frac{C_0}{2} r^2 \sin^2 \theta \\
E_{\phi} &= 2 M C_0 \left( 1+ \frac {2M}{r} \right)^{-1/2} \sin^2 \theta \\
\end{align}
(the unspecified components are set to 0). Here, $C_0$ is a constant that we will set to $1$ in our simulations. Now, to use this initial data scheme, we need to transform the above into the quantities actually tracked by `GiRaFFE` and HydroBase: $A_i$, $B^i$, $\tilde{S}_i$, $v^i$, and $\Phi$. Of these quantities, `GiRaFFEfood` will only set $A_i$, $v^i$, and $\Phi=0$, then call a separate function to calculate  $\tilde{S}_i$; `GiRaFFE` itself will call a function to set $B^i$ before the time-evolution begins. This can be done with eqs. 16 and 18, here given in that same order:
\begin{align}
v^i &= \alpha \frac{\epsilon^{ijk} E_j B_k}{B^2} -\beta^i \\
B^i &= \frac{[ijk]}{\sqrt{\gamma}} \partial_j A_k \\
\end{align}
In the simulations, $B^i$ will be calculated numerically from $A_i$; however, it will be useful to analytically calculate $B^i$ to use calculating the initial $v^i$.

# Table of Contents:
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Import core NRPy+ modules and set NRPy+ parameters
1. [Step 2](#set_a_i): Set the vector $A_i$
1. [Step 3](#set_vi): Calculate $v^i$ from $A_i$ and $E_i$
1. [Step 4](#code_validation1): Code Validation against `GiRaFFEfood_NRPy.GiRaFFEfood_NRPy` NRPy+ Module
1. [Step 5](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Import core NRPy+ modules and set NRPy+ parameters \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

Here, we will import the NRPy+ core modules, set the reference metric to Cartesian, and set commonly used NRPy+ parameters. We will also set up a parameter to determine what initial data is set up, although it won't do much yet.

```python
# Step 0: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

# Step 0.a: Import the NRPy+ core modules and set the reference metric to Cartesian
import sympy as sp               # SymPy: The Python computer algebra package upon which NRPy+ depends
import NRPy_param_funcs as par   # NRPy+: Parameter interface
import indexedexp as ixp         # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_Common_Functions as gfcf # Some useful functions for GiRaFFE initial data.

import reference_metric as rfm
par.set_parval_from_str("reference_metric::CoordSystem","Cartesian")
rfm.reference_metric()

# Step 1a: Set commonly used parameters.
thismodule = "GiRaFFEfood_NRPy_Exact_Wald"
```

# Step 2: Set the vector $A_i$ \[Back to [top](#toc)\]
$$\label{set_a_i}$$

We will first build the fundamental vectors $A_i$ and $E_i$ in spherical coordinates (see [Table 3](https://arxiv.org/pdf/1704.00599.pdf)). Note that we use reference_metric.py to set $r$ and $\theta$ in terms of Cartesian coordinates; this will save us a step later when we convert to Cartesian coordinates. Since $C_0 = 1$,
$$A_{\phi} = \frac{1}{2} r^2 \sin^2 \theta.$$

```python
# Step 2: Set the vectors A and E in Spherical coordinates
def Ar_EW(r,theta,phi, **params):
    return sp.sympify(0)

def Ath_EW(r,theta,phi, **params):
    return sp.sympify(0)

def Aph_EW(r,theta,phi, **params):
    # 1/2 r^2 \sin^2 \theta
    return sp.Rational(1,2) * (r * r * sp.sin(theta)**2)
```

# Step 3: Calculate $v^i$ from $A_i$ and $E_i$ \[Back to [top](#toc)\]
$$\label{set_vi}$$

Next, we will code up the Valencia velocity, which will require us to first code the electric and magnetic fields. The magnetic field is simply $$B^i = \frac{[ijk]}{\sqrt{\gamma}} \partial_j A_k.$$

The electric field is
$$E_{\phi} = 2 M \left( 1+ \frac {2M}{r} \right)^{-1/2} \sin^2 \theta,$$\
Where the other components are $0$.

We can then calculate the the velocity as $$v_{(n)}^i = \frac{\epsilon^{ijk} E_j B_k}{B^2}.$$

```python
#Step 3: Compute v^i from A_i and E_i
def ValenciavU_func_EW(**params):
    M = params["M"]
    gammaDD = params["gammaDD"] # Note that this must use a Cartesian basis!
    sqrtgammaDET = params["sqrtgammaDET"]
    KerrSchild_radial_shift = params["KerrSchild_radial_shift"]
    r     = rfm.xxSph[0] + KerrSchild_radial_shift # We are setting the data up in Shifted Kerr-Schild coordinates
    theta = rfm.xxSph[1]

    LeviCivitaTensorUUU = ixp.LeviCivitaTensorUUU_dim3_rank3(sqrtgammaDET)

    AD = gfcf.Axyz_func_spherical(Ar_EW,Ath_EW,Aph_EW,False,**params)
    # For the initial data, we can analytically take the derivatives of A_i
    ADdD = ixp.zerorank2()
    for i in range(3):
        for j in range(3):
            ADdD[i][j] = sp.simplify(sp.diff(AD[i],rfm.xx_to_Cart[j]))

    BU = ixp.zerorank1()
    for i in range(3):
        for j in range(3):
            for k in range(3):
                BU[i] += LeviCivitaTensorUUU[i][j][k] * ADdD[k][j]

    EsphD = ixp.zerorank1()
    # 2 M ( 1+ 2M/r )^{-1/2} \sin^2 \theta
    EsphD[2] = 2 * M * sp.sin(theta)**2 / sp.sqrt(1+2*M/r)

    ED = rfm.basis_transform_vectorD_from_rfmbasis_to_Cartesian(gfcf.Jac_dUrfm_dDCartUD, EsphD)

    return gfcf.compute_ValenciavU_from_ED_and_BU(ED, BU, gammaDD)
```

# Step 4: Code Validation against `GiRaFFEfood_NRPy.GiRaFFEfood_NRPy` NRPy+ module  \[Back to [top](#toc)\]
$$\label{code_validation1}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for the `GiRaFFE` Exact Wald initial data equations  we intend to use between
1. this tutorial and
2. the NRPy+ [GiRaFFEfood_NRPy.GiRaFFEfood_NRPy_Exact_Wald](../edit/GiRaFFEfood_NRPy/GiRaFFEfood_NRPy_Exact_Wald.py) module.

```python
import GiRaFFEfood_NRPy.GiRaFFEfood_NRPy as gf
import BSSN.ShiftedKerrSchild as sks
sks.ShiftedKerrSchild(True)
import reference_metric as rfm   # NRPy+: Reference metric support
par.set_parval_from_str("reference_metric::CoordSystem","Cartesian")
rfm.reference_metric()

gammaSphDD = ixp.zerorank2()
for i in range(3):
    for j in range(3):
        gammaSphDD[i][j] += sks.gammaSphDD[i][j].subs(sks.r,rfm.xxSph[0]).subs(sks.th,rfm.xxSph[1])

gammaDD = rfm.basis_transform_tensorDD_from_rfmbasis_to_Cartesian(gfcf.Jac_dUrfm_dDCartUD,gammaSphDD)
unused_gammaUU,gammaDET = ixp.symm_matrix_inverter3x3(gammaDD)
sqrtgammaDET = sp.sqrt(gammaDET)

A_ewD = gfcf.Axyz_func_spherical(Ar_EW,Ath_EW,Aph_EW,stagger_enable = True,KerrSchild_radial_shift=sks.r0)
Valenciav_ewD = ValenciavU_func_EW(M=sks.M,KerrSchild_radial_shift=sks.r0,gammaDD=gammaDD,sqrtgammaDET=sqrtgammaDET)
gf.GiRaFFEfood_NRPy_generate_initial_data(ID_type = "ExactWald", stagger_enable = True,M=sks.M,KerrSchild_radial_shift=sks.r0,gammaDD=gammaDD,sqrtgammaDET=sqrtgammaDET)

print("Consistency check between GiRaFFEfood_NRPy tutorial and NRPy+ module:")

def consistency_check(quantity1,quantity2,string):
    if quantity1-quantity2==0:
        print(string+" is in agreement!")
    else:
        print(string+" does not agree!")
        sys.exit(1)

for i in range(3):
    consistency_check(Valenciav_ewD[i],gf.ValenciavU[i],"ValenciavU"+str(i))
    consistency_check(A_ewD[i],gf.AD[i],"AD"+str(i))
```

    Consistency check between GiRaFFEfood_NRPy tutorial and NRPy+ module:
    ValenciavU0 is in agreement!
    AD0 is in agreement!
    ValenciavU1 is in agreement!
    AD1 is in agreement!
    ValenciavU2 is in agreement!
    AD2 is in agreement!

# Step 5: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-GiRaFFEfood_NRPy.pdf](Tutorial-GiRaFFEfood_NRPy.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-GiRaFFEfood_NRPy-Exact_Wald",location_of_template_file=os.path.join(".."))
```

    Created Tutorial-GiRaFFEfood_NRPy-Exact_Wald.tex, and compiled LaTeX file
        to PDF file Tutorial-GiRaFFEfood_NRPy-Exact_Wald.pdf


# Variational Auto-Encoder (VAE)

### Zhenwen Dai (2019-05-29)

Variational auto-encoder (VAE) is a latent variable model that uses a latent variable to generate data represented in vector form. Consider a latent variable $x$ and an observed variable $y$. The plain VAE is defined as
\begin{align}
p(x) =& \mathcal{N}(0, I) \\
p(y|x) =& \mathcal{N}(f(x), \sigma^2I)
\end{align}
where $f$ is the deep neural network (DNN), often referred to as the decoder network.

The variational posterior of VAE is defined as
\begin{align}
q(x) = \mathcal{N}\left(g_{\mu}(y), \sigma^2_x I)\right)
\end{align}
where $g_{\mu}$ is the encoder networks that generate the mean of the variational posterior of $x$. For simplicity, we assume that all the data points share the same variance in the variational posteior. This can be extended by generating the variance also from the encoder network.

```python
import warnings
warnings.filterwarnings('ignore')
import mxfusion as mf
import mxnet as mx
import numpy as np
import mxnet.gluon.nn as nn
import mxfusion.components
import mxfusion.inference
%matplotlib inline
from pylab import *
```

## Load a toy dataset

```python
import GPy
data = GPy.util.datasets.oil_100()
Y = data['X']
label = data['Y'].argmax(1)
```

```python
N, D = Y.shape
```

## Model Defintion

We first define that the encoder and decoder DNN with MXNet Gluon blocks. Both DNNs have two hidden layers with tanh non-linearity.

```python
Q = 2
```

```python
H = 50
encoder = nn.HybridSequential(prefix='encoder_')
with encoder.name_scope():
    encoder.add(nn.Dense(H, in_units=D, activation="tanh", flatten=False))
    encoder.add(nn.Dense(H, in_units=H, activation="tanh", flatten=False))
    encoder.add(nn.Dense(Q, in_units=H, flatten=False))
encoder.initialize(mx.init.Xavier(magnitude=3))
```

```python
H = 50
decoder = nn.HybridSequential(prefix='decoder_')
with decoder.name_scope():
    decoder.add(nn.Dense(H, in_units=Q, activation="tanh", flatten=False))
    decoder.add(nn.Dense(H, in_units=H, activation="tanh", flatten=False))
    decoder.add(nn.Dense(D, in_units=H, flatten=False))
decoder.initialize(mx.init.Xavier(magnitude=3))
```

Then, we define the model of VAE in MXFusion. Note that for simplicity in implementation, we use scalar normal distributions defined for individual entries of a Matrix instead of multivariate normal distributions with diagonal covariance matrices.

```python
from mxfusion.components.variables.var_trans import PositiveTransformation
from mxfusion import Variable, Model, Posterior
from mxfusion.components.functions import MXFusionGluonFunction
from mxfusion.components.distributions import Normal
from mxfusion.components.functions.operators import broadcast_to

m = Model()
m.N = Variable()
m.decoder = MXFusionGluonFunction(decoder, num_outputs=1,broadcastable=True)
m.x = Normal.define_variable(mean=broadcast_to(mx.nd.array([0]), (m.N, Q)),
                             variance=broadcast_to(mx.nd.array([1]), (m.N, Q)), shape=(m.N, Q))
m.f = m.decoder(m.x)
m.noise_var = Variable(shape=(1,), transformation=PositiveTransformation(), initial_value=mx.nd.array([0.01]))
m.y = Normal.define_variable(mean=m.f, variance=broadcast_to(m.noise_var, (m.N, D)),
                             shape=(m.N, D))
print(m)
```

    Model (37a04)
    Variable (b92c2) = BroadcastToOperator(data=Variable noise_var (a50d4))
    Variable (39c2c) = BroadcastToOperator(data=Variable (e1aad))
    Variable (b7150) = BroadcastToOperator(data=Variable (a57d4))
    Variable x (53056) ~ Normal(mean=Variable (b7150), variance=Variable (39c2c))
    Variable f (ad606) = GluonFunctionEvaluation(decoder_input_0=Variable x (53056), decoder_dense0_weight=Variable (b9b70), decoder_dense0_bias=Variable (d95aa), decoder_dense1_weight=Variable (73dc2), decoder_dense1_bias=Variable (b85dd), decoder_dense2_weight=Variable (7a61c), decoder_dense2_bias=Variable (eba91))
    Variable y (23bca) ~ Normal(mean=Variable f (ad606), variance=Variable (b92c2))

We also define the variational posterior following the equation above.

```python
q = Posterior(m)
q.x_var = Variable(shape=(1,), transformation=PositiveTransformation(), initial_value=mx.nd.array([1e-6]))
q.encoder = MXFusionGluonFunction(encoder, num_outputs=1, broadcastable=True)
q.x_mean = q.encoder(q.y)
q.x.set_prior(Normal(mean=q.x_mean, variance=broadcast_to(q.x_var, q.x.shape)))
print(q)
```

    Posterior (4ec05)
    Variable x_mean (86d22) = GluonFunctionEvaluation(encoder_input_0=Variable y (23bca), encoder_dense0_weight=Variable (51b3d), encoder_dense0_bias=Variable (c0092), encoder_dense1_weight=Variable (ad9ef), encoder_dense1_bias=Variable (83db0), encoder_dense2_weight=Variable (78b82), encoder_dense2_bias=Variable (b856d))
    Variable (6dc84) = BroadcastToOperator(data=Variable x_var (19d07))
    Variable x (53056) ~ Normal(mean=Variable x_mean (86d22), variance=Variable (6dc84))

## Variational Inference

Variational inference is done via creating an inference object and passing in the stochastic variational inference algorithm.

```python
from mxfusion.inference import BatchInferenceLoop, StochasticVariationalInference, GradBasedInference

observed = [m.y]
alg = StochasticVariationalInference(num_samples=3, model=m, posterior=q, observed=observed)
infr = GradBasedInference(inference_algorithm=alg, grad_loop=BatchInferenceLoop())
```

SVI is a gradient-based algorithm. We can run the algorithm by providing the data and specifying the parameters for the gradient optimizer (the default gradient optimizer is Adam).

```python
infr.run(max_iter=2000, learning_rate=1e-2, y=mx.nd.array(Y), verbose=True)
```

    Iteration 200 loss: 1720.556396484375
    Iteration 400 loss: 601.11962890625
    Iteration 600 loss: 168.620849609375
    Iteration 800 loss: -48.67474365234375
    Iteration 1000 loss: -207.34835815429688
    Iteration 1200 loss: -354.17742919921875
    Iteration 1400 loss: -356.26409912109375
    Iteration 1600 loss: -561.263427734375
    Iteration 1800 loss: -697.8665161132812
    Iteration 2000 loss: -753.83203125				8

## Plot the training data in the latent space

Finally, we may be interested in visualizing the latent space of our dataset. We can do that by calling encoder network.

```python
from mxfusion.inference import TransferInference

q_x_mean = q.encoder.gluon_block(mx.nd.array(Y)).asnumpy()
```

```python
for i in range(3):
    plot(q_x_mean[label==i,0], q_x_mean[label==i,1], '.')
```


```python
from logicqubit.logic import *
from cmath import *
import numpy as np
import sympy as sp
import scipy
from random import randrange
from scipy.optimize import *
import matplotlib.pyplot as plt
```

```python
gates = Gates(1)

ID = gates.ID()
X = gates.X()
Y = gates.Y()
Z = gates.Z()
```

```python
III = ID.kron(ID).kron(ID)
XXX = X.kron(X).kron(X)
YYY = Y.kron(Y).kron(Y)
ZZZ = Z.kron(Z).kron(Z)
IZZ = ID.kron(Z).kron(Z)
ZZI = ID.kron(Z).kron(ID)
sig_izz = [IZZ.get()[i,i] for i in range(len(IZZ.get()))]
sig_zzi = [ZZI.get()[i,i] for i in range(len(ZZI.get()))]
sig_izz
```

    [1, -1, -1, 1, 1, -1, -1, 1]

```python
H = III*2 + YYY*4 + IZZ*3 + ZZZ*2

min(scipy.linalg.eig(H.get())[0])
```

    (-5.47213595499958+0j)

```python
def _ansatz(reg, params):
    n_qubits = len(reg)
    depth = n_qubits
    for i in range(depth):
        reg[1].CNOT(reg[0])
        for j in range(n_qubits):
            reg[i].RY(params[j])

def ansatz(reg, params):
    n_qubits = len(reg)
    depth = n_qubits
    for i in range(depth):
        for j in range(n_qubits):
            if(j < n_qubits-1):
                reg[j+1].CNOT(reg[j])
            reg[i].RY(params[j])

def ansatz_3q(q1, q2, q3, params):
    q1.RY(params[0])
    q2.RY(params[1])
    q3.RY(params[2])
    q2.CNOT(q1)
    q3.CNOT(q2)
    q1.RX(params[3])
    q2.RX(params[4])
    q3.RX(params[5])
    q2.CNOT(q1)
    q3.CNOT(q2)
    q1.RY(params[6])
    q2.RY(params[7])
    q3.RY(params[8])
    q2.CNOT(q1)
    q3.CNOT(q2)
```

```python
def expectation_3q(params):
    logicQuBit  = LogicQuBit(3)
    q1 = Qubit()
    q2 = Qubit()
    q3 = Qubit()

    ansatz_3q(q1,q2,q3,params)
    #ansatz([q1,q2,q3],params)
    psi = logicQuBit.getPsi()

    return (psi.adjoint()*H*psi).get()[0][0]

minimum = minimize(expectation_3q, [0,0,0,0,0,0,0,0,0], method='Nelder-Mead', options={'xtol': 1e-10, 'ftol': 1e-10})
print(minimum)
```

     final_simplex: (array([[ 0.12921139, -1.57086317,  1.57080437, -1.12420308, -3.03704134,
             1.45367023, -0.29225308,  1.95750013,  1.5707652 ],
           [ 0.12924501, -1.57078486,  1.57088293, -1.1242658 , -3.0369472 ,
             1.4536376 , -0.29225486,  1.95729195,  1.57079917],
           [ 0.12916145, -1.5708624 ,  1.57083772, -1.12424803, -3.03717848,
             1.45370456, -0.29226349,  1.95760535,  1.57082513],
           [ 0.12919184, -1.57076394,  1.57080683, -1.12432742, -3.03703452,
             1.45363379, -0.29218685,  1.95743054,  1.57083746],
           [ 0.12922316, -1.5708506 ,  1.57084737, -1.12431734, -3.03698036,
             1.45369964, -0.29223934,  1.95744884,  1.57076656],
           [ 0.1291532 , -1.57080607,  1.57081857, -1.12425794, -3.03700681,
             1.45368743, -0.29227974,  1.95751193,  1.57072828],
           [ 0.12924718, -1.57072935,  1.5707414 , -1.12422313, -3.03688703,
             1.45354606, -0.29216359,  1.95727596,  1.57078718],
           [ 0.12910942, -1.57073622,  1.57084631, -1.12426268, -3.03708219,
             1.45366151, -0.29229058,  1.95749856,  1.57078757],
           [ 0.12925603, -1.57089954,  1.57074983, -1.12423081, -3.03709666,
             1.45364761, -0.29216064,  1.95753102,  1.57083743],
           [ 0.12913261, -1.57076602,  1.57069762, -1.12420655, -3.03696704,
             1.45362587, -0.29222644,  1.95754186,  1.57069371]]), array([-5.47213594, -5.47213594, -5.47213594, -5.47213593, -5.47213593,
           -5.47213593, -5.47213593, -5.47213591, -5.47213591, -5.47213589]))
               fun: -5.472135937489735
           message: 'Maximum number of function evaluations has been exceeded.'
              nfev: 1801
               nit: 1237
            status: 1
           success: False
                 x: array([ 0.12921139, -1.57086317,  1.57080437, -1.12420308, -3.03704134,
            1.45367023, -0.29225308,  1.95750013,  1.5707652 ])

```python
#ZZZ
#000 =  1
#001 = -1
#010 = -1
#011 =  1
#100 = -1
#101 =  1
#110 =  1
#111 = -1

def expectation_value(measurements, base = np.array([1,-1,-1,1,-1,1,1,-1])):
    probabilities = np.array(measurements)
    expectation = np.sum(base * probabilities)
    return expectation

def sigma_xxx(params):
    logicQuBit  = LogicQuBit(3, first_left = False)
    q1 = Qubit()
    q2 = Qubit()
    q3 = Qubit()

    ansatz_3q(q1,q2,q3,params)

    # medidas em XX
    q1.RY(-pi/2)
    q2.RY(-pi/2)
    q3.RY(-pi/2)

    result = logicQuBit.Measure([q1,q2,q3])
    result = expectation_value(result)
    return result

def sigma_yyy(params):
    logicQuBit  = LogicQuBit(3, first_left = False)
    q1 = Qubit()
    q2 = Qubit()
    q3 = Qubit()

    ansatz_3q(q1,q2,q3,params)

    # medidas em YY
    q1.RX(pi/2)
    q2.RX(pi/2)
    q3.RX(pi/2)

    result = logicQuBit.Measure([q1,q2,q3])
    result = expectation_value(result)
    return result

def sigma_zzz(params):
    logicQuBit  = LogicQuBit(3, first_left = False)
    q1 = Qubit()
    q2 = Qubit()
    q3 = Qubit()

    ansatz_3q(q1,q2,q3,params)

    result = logicQuBit.Measure([q1,q2,q3])
    zzz = expectation_value(result)
    izz = expectation_value(result, sig_izz) # [zzz, izz] = 0
    return zzz, izz

def expectation_energy(params):
    xxx =  sigma_xxx(params)
    yyy =  sigma_yyy(params)
    zzz, izz =  sigma_zzz(params)

    result = 2 + 4*yyy + 3*izz + 2*zzz
    return result
```

```python
#initial_values = [random.random() for _ in range(3)]
minimum = minimize(expectation_energy, [0,0,0,0,0,0,0,0,0], method='Nelder-Mead', options={'xtol': 1e-10, 'ftol': 1e-10})
print(minimum)
```

     final_simplex: (array([[-1.02400892,  1.28055431,  0.05946761,  1.33300369,  0.08342998,
             2.19934652, -6.29492554,  1.83668327,  0.42674918],
           [-1.0045284 ,  1.23991259,  0.10521914,  1.37250857,  0.10442777,
             2.12314711, -6.24443198,  1.80795255,  0.41251561],
           [-0.96316889,  1.28037413, -0.02258307,  1.37967455,  0.087312  ,
             2.19233154, -6.29865919,  1.84036414,  0.42136336],
           [-1.01088839,  1.26968456,  0.02772341,  1.47440312,  0.11925323,
             2.16213358, -6.28100343,  1.81227668,  0.37292877],
           [-0.93486007,  1.27946439,  0.01883379,  1.40376204,  0.10724767,
             2.13684427, -6.31317357,  1.84579082,  0.39146723],
           [-1.01355085,  1.25750614,  0.099545  ,  1.45014256,  0.12627078,
             2.12002938, -6.27548029,  1.81035501,  0.36927212],
           [-1.07176354,  1.24202417,  0.03454186,  1.48001232,  0.11766726,
             2.1641785 , -6.1354307 ,  1.75219463,  0.36242693],
           [-0.96319626,  1.2278143 ,  0.03577467,  1.41344896,  0.10253174,
             2.13218961, -6.27560476,  1.81349595,  0.43456215],
           [-0.99780333,  1.2340127 ,  0.00854529,  1.45831798,  0.10591945,
             2.16471894, -6.27091469,  1.80239979,  0.42345559],
           [-0.91029119,  1.2947923 ,  0.07297279,  1.40533249,  0.11793036,
             2.11702152, -6.44888308,  1.89306445,  0.39471469]]), array([-5.17369093, -5.15754491, -5.15406618, -5.14643163, -5.14535456,
           -5.1451482 , -5.14303204, -5.13943651, -5.13679564, -5.13092301]))
               fun: -5.173690931894985
           message: 'Maximum number of function evaluations has been exceeded.'
              nfev: 1800
               nit: 1247
            status: 1
           success: False
                 x: array([-1.02400892,  1.28055431,  0.05946761,  1.33300369,  0.08342998,
            2.19934652, -6.29492554,  1.83668327,  0.42674918])

```python
def gradient(params, evaluate):
    n_params = params.shape[0]
    shift = pi/2
    gradients = np.zeros(n_params)

    for i in range(n_params):
        #parameter shift rule
        shift_vect = np.array([shift if j==i else 0 for j in range(n_params)])
        shift_right = params + shift_vect
        shift_left = params - shift_vect

        expectation_right = evaluate(shift_right)
        expectation_left = evaluate(shift_left)

        gradients[i] = expectation_right - expectation_left

    return gradients
```

```python
params = np.random.uniform(-np.pi, np.pi, 9)
last_params = np.zeros(9)
```

```python
lr = 0.1
err = 1
while err > 1e-3:
    grad = gradient(params, expectation_energy)
    params = params - lr*grad
    err = abs(sum(params - last_params))
    last_params = np.array(params)
    print(err)
```

    /tmp/ipykernel_12923/1088512639.py:15: ComplexWarning: Casting complex values to real discards the imaginary part
      gradients[i] = expectation_right - expectation_left

    2.269557430510688
    0.43148743285348523
    0.3173515038192558
    0.1843803492700199
    0.04327682033010166
    0.0205755822034282
    0.02982539400493711
    0.0242148016084765
    0.01835772396267603
    0.013352718021686673
    0.009994564789378567
    0.007462727872315965
    0.005675856339475829
    0.004299655543890113
    0.00327869541767134
    0.002487918781161013
    0.0018904697303590463
    0.0014302939341544803
    0.001081493614157969
    0.00081494892435412

```python
expectation_energy(params)
```

    (-5.472130682938133+0j)

```python

```


# Elementary greenhouse models

This notebook is part of [The Climate Laboratory](https://brian-rose.github.io/ClimateLaboratoryBook) by [Brian E. J. Rose](http://www.atmos.albany.edu/facstaff/brose/index.html), University at Albany.

____________

## 1. A single layer atmosphere
____________

We will make our first attempt at quantifying the greenhouse effect in the simplest possible greenhouse model: a single layer of atmosphere that is able to absorb and emit longwave radiation.

### Assumptions

- Atmosphere is a single layer of air at temperature $T_a$
- Atmosphere is **completely transparent to shortwave** solar radiation.
- The **surface** absorbs shortwave radiation $(1-\alpha) Q$
- Atmosphere is **completely opaque to infrared** radiation
- Both surface and atmosphere emit radiation as **blackbodies** ($\sigma T_s^4, \sigma T_a^4$)
- Atmosphere radiates **equally up and down** ($\sigma T_a^4$)
- There are no other heat transfer mechanisms

We can now use the concept of energy balance to ask what the temperature need to be in order to balance the energy budgets at the surface and the atmosphere, i.e. the **radiative equilibrium temperatures**.

### Energy balance at the surface

$$
\begin{align*}
\text{energy in} &= \text{energy out} \\
(1-\alpha) Q + \sigma T_a^4 &= \sigma T_s^4 \\
\end{align*}
$$

The presence of the atmosphere above means there is an additional source term: downwelling infrared radiation from the atmosphere.

We call this the **back radiation**.

### Energy balance for the atmosphere

$$
\begin{align}
\text{energy in} &= \text{energy out} \\
\sigma T_s^4 &= A\uparrow + A\downarrow = 2 \sigma T_a^4 \\
\end{align}
$$

which means that
$$ T_s = 2^\frac{1}{4} T_a \approx 1.2 T_a $$

So we have just determined that, in order to have a purely **radiative equilibrium**, we must have $T_s > T_a$.

*The surface must be warmer than the atmosphere.*

### Solve for the radiative equilibrium surface temperature

Now plug this into the surface equation to find

$$ \frac{1}{2} \sigma T_s^4 = (1-\alpha) Q  $$

and use the definition of the emission temperature $T_e$ to write

$$ (1-\alpha) Q = \sigma T_e^4 $$

*In fact, in this model, $T_e$ is identical to the atmospheric temperature $T_a$, since all the OLR originates from this layer.*

Solve for the surface temperature:

$$ T_s = 2^\frac{1}{4} T_e $$

From the observed outgoing longwave radiation that we studied last week, $T_e = 255$ K. Thus you can solve for the surface temperature...

```python
def surface(Te = 255):
    Ts = 2**(1/4)*Te
    return Ts

surface()
```

    303.24781432569387

This model is one small step closer to reality: surface is warmer than atmosphere, emissions to space generated in the atmosphere, atmosphere heated from below and helping to keep surface warm.

### Why does this model overpredict the surface temperature

Our model now overpredicts the surface temperature by about 15ºC (303 K versus the observed 288 K).

Why is this?  Let's discuss.

- Te is not the same as Ta.
- The atmosphere is not actually completely opaque to infrared radiation
- The atmosphere has a vertical structure

____________

## 2. Introducing the two-layer grey gas model
____________

Let's generalize the above model just a little bit to build a slighly more realistic model of longwave radiative transfer.

We will address two shortcomings of our single-layer model:
1. No vertical structure
2. 100% longwave opacity

Relaxing these two assumptions gives us what turns out to be a very useful prototype model for **understanding how the greenhouse effect works**.

### Assumptions

- The atmosphere is **transparent to shortwave radiation** (still)
- Divide the atmosphere up into **two layers of equal mass** (the dividing line is thus at 500 hPa pressure level)
- Each layer **absorbs only a fraction $\epsilon$** of whatever longwave radiation is incident upon it.
- We will call the fraction $\epsilon$ the **absorptivity** of the layer.
- Assume $\epsilon$ is the same in each layer

This is called the **grey gas** model, where **grey** here means the emission and absorption have **no spectral dependence** (same at every wavelength).

We can think of this model informally as a "leaky greenhouse".

Note that the assumption that $\epsilon$ is the same in each layer is appropriate if the absorption is actually carried out by a gas that is **well-mixed** in the atmosphere.

Out of our two most important absorbers:

- CO$_2$ is well mixed
- H$_2$O is not (mostly confined to lower troposphere due to strong temperature dependence of the saturation vapor pressure).

But we will ignore this aspect of reality for now.

### Kirchoff's Law

In order to build our model, we need to introduce one additional piece of physics known as **Kirchoff's Law**:

$$ \text{absorptivity} = \text{emissivity} $$

So **if a layer of atmosphere at temperature $T$ absorbs a fraction $\epsilon$** of incident longwave radiation, it must **emit**

$$ \epsilon ~\sigma ~T^4 $$

both up and down.

### A sketch of the radiative fluxes in the 2-layer atmosphere

- Surface temperature is $T_s$
- Atm. temperatures are $T_0, T_1$ where $T_0$ is closest to the surface.
- absorptivity of atm layers is $\epsilon$
- Surface emission is $\sigma T_s^4$
- Atm emission is $\epsilon \sigma T_0^4, \epsilon \sigma T_1^4$ (up and down)
- Absorptivity = emissivity for atmospheric layers
- a fraction $(1-\epsilon)$ of the longwave beam is **transmitted** through each layer

____________
## 3. Tracing the upwelling beam of longwave radiation
____________

Let's think about the upwelling beam of longwave radiation, which we denote $U$.

### Surface to layer 0

We start at the surface. The upward flux **from the surface to layer 0** is

$$U_0 = \sigma T_s^4$$

(just the emission from the suface).

### Layer 0 to layer 1

Now **following this beam upward**, we first recognize that a fraction $\epsilon$ of this beam is **absorbed** in layer 0.

The upward flux from layer 0 to layer 1 consists of the sum of two parts:

1. The **transmitted part** of whatever is incident from below (i.e. the part that is **not absorbed**)
2. **New upward emissions** from layer 0

We can write this upward flux from layer 0 to layer 1 as:

$$U_1 = (1-\epsilon) \sigma T_s^4 + \epsilon \sigma T_0^4$$

### Beyond layer 1

Continuing to follow the same beam, we follow the same logic! A fraction $\epsilon$ of $U_1$ is absorbed in layer 1, and therefore the transmitted part is $(1-\epsilon) U_1$.

Including new emissions from layer 1, the upwelling flux above layer 1 is

$$U_2 = (1-\epsilon) U_1 + \epsilon \sigma T_1^4$$

### Outgoing Longwave Radiation

Since there is **no more atmosphere above layer 1**, this upwelling beam is our OLR for this model:

$$\text{OLR} = U_2 = (1-\epsilon) U_1 + \epsilon \sigma T_1^4$$

which, plugging in the above expression for $U_1$, works out to

$$\text{OLR} = (1-\epsilon)^2 \sigma T_s^4 + \epsilon(1-\epsilon)\sigma T_0^4 + \epsilon \sigma T_1^4$$

Here the three terms represent **contributions to the total OLR** that **originate from each of the three levels**

### Limits of large and small absorptivity/emissivity

Think about the following two questions:

- What happens to this expression if $\epsilon=1$? *What does this represent physically?*
- What about $\epsilon=0$?

By allowing the atmosphere to partially absorb emissions from other levels, we now see that the Outgoing Longwave Radiation to space **includes emissions from every level** - and therefore **affected by temperature at every level**!

____________

## 4. Tuning the grey gas model to observations
____________

In building our new model we have introduced exactly **one parameter**, the absorptivity $\epsilon$. We need to choose a value for $\epsilon$.

We will tune our model so that it **reproduces the observed global mean OLR** given **observed global mean temperatures**.

### Global mean air temperature observations

To get appropriate temperatures for $T_s, T_0, T_1$, let's revisit the global, annual mean lapse rate plot from NCEP Reanalysis data we first encountered in the [Radiation notes](https://brian-rose.github.io/ClimateLaboratoryBook/courseware/radiation.html).

```python
#  This code is used just to create the skew-T plot of global, annual mean air temperature
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import xarray as xr
from metpy.plots import SkewT
ncep_url = "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/"
ncep_air = xr.open_dataset( ncep_url + "pressure/air.mon.1981-2010.ltm.nc", use_cftime=True)
#  Take global, annual average and convert to Kelvin
coslat = np.cos(np.deg2rad(ncep_air.lat))
weight = coslat / coslat.mean(dim='lat')
Tglobal = (ncep_air.air * weight).mean(dim=('lat','lon','time'))
fig = plt.figure(figsize=(9, 9))
skew = SkewT(fig, rotation=30)
skew.plot(Tglobal.level, Tglobal, color='black', linestyle='-', linewidth=2, label='Observations')
skew.ax.set_ylim(1050, 10)
skew.ax.set_xlim(-75, 45)
# Add the relevant special lines
skew.plot_dry_adiabats(linewidth=0.5)
skew.plot_moist_adiabats(linewidth=0.5)
#skew.plot_mixing_lines()
skew.ax.legend()
skew.ax.set_xlabel(r'Temperature ($^{\circ} C$)')
skew.ax.set_ylabel('Atmos. pressure (hPa)')
skew.ax.set_title('Global, annual mean sounding from NCEP Reanalysis', fontsize = 16);
```

### Target temperatures for our model tuning

First, we set
$$T_s = 288 \text{ K}  $$

From the lapse rate plot, an average temperature for the layer between 1000 and 500 hPa is

$$ T_0 = 275 \text{ K}$$

Defining an average temperature for the layer between 500 and 0 hPa is more ambiguous because of the lapse rate reversal at the tropopause. We will choose

$$ T_1 = 230 \text{ K}$$

From the graph, this is approximately the observed global mean temperature at 275 hPa or about 10 km.

### OLR

From the observed global energy budget (see previous weeks) we set

$$\text{OLR} = 238.5 \text{ W m}^{-2}$$

### Solving for $\epsilon$

We wrote down the expression for OLR as a function of temperatures and absorptivity in our model above.

All we need to do is plug the observed values into the above expression for OLR, and solve for $\epsilon$.

It is a **quadratic equation** for the unknown $\epsilon$. We could work out the exact solution using the quadratic formula.

But let's do it **graphically**, using Python!

### Exercise: graphical solution to find the best fit value of $\epsilon$

The OLR formula for the leaky greenhouse that we derived above is

$$\text{OLR} = (1-\epsilon)^2 \sigma T_s^4 + \epsilon(1-\epsilon)\sigma T_0^4 + \epsilon \sigma T_1^4$$

Do the following:

- Write a Python function that implements this formula
- The function should accept **four input parameters**:
    - The three temperatures $T_s, T_0, T_1$
    - The emissivity $\epsilon$
- Using this function, make a **graph of OLR vs. $\epsilon$** for the observed temperature values $T_s = 288, T_0 = 275, T_1 = 230$
- For the graph, $\epsilon$ should range between 0 and 1.
- From your graph, find the approximate value of $\epsilon$ that gives $OLR = 238.5$

```python
sigma=5.67E-8
def OLR(Ts, T0, T1, epsilon):
    fromsurf = ((1-epsilon)**2)*sigma*(Ts**4)
    fromL0 = epsilon*(1-epsilon)*sigma*(T0**4)
    fromL1 = epsilon*sigma*(T1**4)
    return fromsurf+fromL0+fromL1
```

```python
eps = np.linspace(0,1,num=20)
olrtests = OLR(Ts=288, T0=275, T1=230, epsilon=eps)
```

```python
fig, ax=plt.subplots()
ax.plot(eps, olrtests)
ax.axhline(238.5, color='k', ls=':')
ax.set(xlabel=r'$\epsilon$', ylabel='OLR [W/m2]')
```

Note if you solve the quadratic equation algebraically you will get two solutions:

- $\epsilon \approx 0.586$
- $\epsilon \approx 3.93$

(for details, see [the advanced notes here](https://brian-rose.github.io/ClimateLaboratoryBook/courseware/sympy-greenhouse.html))

**Why is the second solution not physically meaningful?**

Hopefully your graph shows that $\epsilon = 0.586$ gives the correct value of OLR.

This is the absorptivity that guarantees that **our model reproduces the observed OLR given the observed temperatures**.

____________

## 5. Level of emission
____________

### Contributions from each level to the outgoing radiation

Now that we have tuned up our model, we can see exactly how strongly each level contributes to the OLR.

The three components of the OLR are

$$
\begin{align*}
\text{OLR}_s &= (1-\epsilon)^2 \sigma T_s^4  \\
\text{OLR}_0 &= \epsilon(1-\epsilon)\sigma T_0^4 \\
\text{OLR}_1 &= \epsilon \sigma T_1^4
\end{align*}
$$

which of course add up to the total OLR we wrote down above.

### Exercise: calculate contributions to OLR

**Write some simple Python code to calculate each term in the OLR using the observed temperatures and the tuned value $\epsilon = 0.586$. Fill out the list below using your calculated numbers.**

```python
def OLR(e=0.586, sigma=5.67E-8, Ts=288, T0=275, T1=230):
    OLRs = ((1-e)**2)*sigma*Ts**4
    OLR0 = e*(1-e)*sigma*T0**4
    OLR1 = e*sigma*T1**4
    return OLRs, OLR0, OLR1

OLR()
```

    (66.85804791878124, 78.67047843703125, 92.980530342)

**Contributions to the OLR originating from each level, in W/m2:**

- Surface: 66.86
- Level 0: 78.67
- Level 1: 92.98

```python
Temp=sum(OLR())
print(Temp)
```

    238.50905669781247

Notice that the largest single contribution is coming from the top layer.

*This is in spite of the fact that the emissions from this layer are weak, because it is so cold.*

###  Changing the level of emission by adding absorbers

Adding some **extra greenhouse absorbers** will mean that a **greater fraction** of incident longwave radiation is **absorbed in each layer**.

Thus **$\epsilon$ must increase** as we add greenhouse gases.

Suppose we have $\epsilon$ initially, and the absorptivity increases to $\epsilon_2 = \epsilon + \Delta \epsilon$.

Suppose further that this increase happens **abruptly** so that there is no time for the temperatures to respond to this change. **We hold the temperatures fixed** in the column and ask how the radiative fluxes change.

**Question: Do you expect the OLR to increase or decrease?**

### Calculating the change in level of emission

Let's use our two-layer leaky greenhouse model to investigate the answer.

The components of the OLR before the perturbation are

$$
\begin{align*}
\text{OLR}_s &= (1-\epsilon)^2 \sigma T_s^4  \\
\text{OLR}_0 &= \epsilon(1-\epsilon)\sigma T_0^4 \\
\text{OLR}_1 &= \epsilon \sigma T_1^4
\end{align*}
$$

and after the perturbation we have

$$
\begin{align*}
\text{OLR}_s &= (1-\epsilon - \Delta \epsilon)^2 \sigma T_s^4  \\
\text{OLR}_0 &= (\epsilon + \Delta \epsilon)(1-\epsilon - \Delta \epsilon)\sigma T_0^4 \\
\text{OLR}_1 &= (\epsilon + \Delta \epsilon) \sigma T_1^4
\end{align*}
$$

Let's subtract off the original components to get the contributions to the **change in OLR** from each layer:

$$
\begin{align*}
\Delta \text{OLR}_s &= \left[(1-\epsilon - \Delta \epsilon)^2 - (1-\epsilon)^2\right]\sigma T_s^4  \\
\Delta \text{OLR}_0 &= \left[(\epsilon + \Delta \epsilon)(1-\epsilon - \Delta \epsilon) - \epsilon(1-\epsilon)  \right] \sigma T_0^4 \\
\Delta \text{OLR}_1 &= \left[(\epsilon + \Delta \epsilon) - \epsilon \right] \sigma T_1^4
\end{align*}
$$

Now expand this out, but to make things easier to deal with, neglect term in $\Delta \epsilon^2$ (very small - we will be considering changes of less than 10% in $\epsilon$):

$$
\begin{align*}
\Delta \text{OLR}_s &\approx (\Delta \epsilon) \left[ -2(1-\epsilon) \right] \sigma T_s^4  \\
\Delta \text{OLR}_0 &\approx (\Delta \epsilon) (1 - 2 \epsilon) \sigma T_0^4 \\
\Delta \text{OLR}_1 &\approx (\Delta \epsilon) \sigma T_1^4
\end{align*}
$$

Now look at the **sign** of each term. Recall that $0 < \epsilon < 1$. **Which terms in the OLR go up and which go down?**

**THIS IS VERY IMPORTANT, SO STOP AND THINK ABOUT IT.**

The contribution from the **surface** must **decrease**, while the contribution from the **top layer** must **increase**.

**When we add absorbers, the average level of emission goes up!**

____________

## 6. Radiative forcing in the 2-layer grey gas model
____________

### Definition of Radiative Forcing

We now define a very important quantity:

**"Radiative forcing" is the change in total radiative flux at TOA after adding absorbers**

In this model, **only the longwave flux can change**, so we calculate the radiative forcing as

$$ R = - \Delta \text{OLR} $$

(with the minus sign so that $R$ is **positive when the climate system is gaining extra energy**).

### Connection between radiative forcing and level of emission

We just worked out that whenever we **add some extra absorbers**, the **emissions to space** (on average) will originate from **higher levels** in the atmosphere.

What does this mean for OLR? Will it increase or decrease?

To get the answer, we just have to sum up the three contributions we wrote above:

$$
\begin{align*}
R &= -\Delta \text{OLR}_s - \Delta \text{OLR}_0 - \Delta \text{OLR}_1  \\
  &= -\Delta \epsilon \left[  -2(1-\epsilon) \sigma T_s^4 + (1 - 2 \epsilon) \sigma T_0^4  + \sigma T_1^4 \right]
\end{align*}
$$

Is this a positive or negative number? The key point is this:

**It depends on the temperatures, i.e. on the lapse rate.**

### Greenhouse effect for an isothermal atmosphere

Stop and think about this question:

If the **surface and atmosphere are all at the same temperature**, does the OLR go up or down when $\epsilon$ increases (i.e. we add more absorbers)?

Understanding this question is key to understanding how the greenhouse effect works.

#### Let's solve the isothermal case

We will just set $T_s = T_0 = T_1$ in the above expression for the radiative forcing.

What do you get?

```python
R = 0
```

#### The answer is $R=0$

For an isothermal atmosphere, there is **no change** in OLR when we add extra greenhouse absorbers. Hence, no radiative forcing and no greenhouse effect.

Why?

The level of emission still must go up. But since the temperature at the upper level is the **same** as everywhere else, the emissions are exactly the same.

### The radiative forcing (change in OLR) depends on the lapse rate!

For a more realistic example of radiative forcing due to an increase in greenhouse absorbers, we use our observed temperatures and the tuned value for $\epsilon$.

We'll express the answer in W m$^{-2}$ for a 2% increase in $\epsilon$:

$$ \Delta \epsilon = 0.02 \times 0.58 $$

```python
epsilon = 0.586041150248834
delta_epsilon = 0.02 * epsilon
delta_epsilon
```

    0.011720823004976679

Calculate the three components of the radiative forcing:

```python
sigma = 5.67E-8
Ts = 288.
T0 = 275.
T1 = 230.
```

```python
#  Component originating from the surface
Rs = -delta_epsilon * (-2*(1-epsilon)*sigma * Ts**4)
Rs
```

    3.78528239476332

```python
#  Component originating from level 0
R0 = -delta_epsilon * (1-2*epsilon) * sigma * (T0**4)
print(R0)
```

    0.6540457761108388

```python
#  Component originating from level 1
R1 = -delta_epsilon * sigma*T1**4
print(R1)
```

    -1.8597411929137297

So just add them up to get the total radiative forcing:

```python
R = Rs + R0 + R1
R
```

    2.579586977960429

So in our example, **the OLR decreases by 2.6 W m$^{-2}$**, or equivalently, the **radiative forcing is +2.6 W m$^{-2}$.**

What we have just calculated is this:

**Given the observed lapse rates, a small increase in absorbers will cause a small decrease in OLR.**

The **greenhouse effect** thus gets **stronger**, and energy will begin to accumulate in the system -- which will eventually **cause temperatures to increase** as the system adjusts to a new equilibrium.

____________

## 7. Summary
____________

### Key physical lessons

- Putting a **layer of longwave absorbers** above the surface keeps the **surface substantially warmer**, because of the **backradiation** from the atmosphere (greenhouse effect).
- The **grey gas** model assumes that each layer absorbs and emits a fraction $\epsilon$ of its blackbody value, independent of wavelength.

- With **incomplete absorption** ($\epsilon < 1$), there are contributions to the OLR from every level and the surface (there is no single **level of radiative emission**)
- Adding more absorbers means that **contributions to the OLR** from **upper levels** go **up**, while contributions from the surface go **down**.

- The **radiative forcing** caused by an increase in absorbers **depends on the lapse rate** (decrease in temperature with altitude in the atmosphere).
- For an **isothermal atmosphere** the radiative forcing is zero and there is **no greenhouse effect**
- The radiative forcing is positive for our atmosphere **because tropospheric temperatures tend to decrease with height**.

____________

## Credits

This notebook is based on a chapter of [The Climate Laboratory](https://brian-rose.github.io/ClimateLaboratoryBook), an open-source textbook developed and maintained by [Brian E. J. Rose](http://www.atmos.albany.edu/facstaff/brose/index.html), University at Albany.

It is licensed for free and open consumption under the
[Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/) license.

Development of these notes and the [climlab software](https://github.com/brian-rose/climlab) is partially supported by the National Science Foundation under award AGS-1455071 to Brian Rose. Any opinions, findings, conclusions or recommendations expressed here are mine and do not necessarily reflect the views of the National Science Foundation.
____________