## Topological Data Analysis - Part 4 - Persistent Homology

This is Part 4 in a series on topological data analysis.
See Part 1 | Part 2 | Part 3 | Part 5

In this part we learn more about how linear algebra can help us calculate topological features in a computationally efficient manner.

### Linear algebra saves the day

You might have noticed that calculating the homology groups and Betti numbers by hand can be very tedious and impractical for anything larger than the simple examples we've considered thus far. Fortunately, there are better ways. In particularly, we can represent most of the computation of homology groups in terms of vectors and matrices and computers are very efficient when working with vectors and matrices.

Now, we already went over what a vector is (an element in a vector space) but what is a matrix? You probably think of a matrix has a kind of 2-dimensional grid of numbers, and you know you can multiply matrices by other matrices and vectors and what not. Well a grid of numbers is certainly a convenient notation for matrices, but that's not what they _are_.

##### What's a matrix?

By this point, you should be comfortable with the idea of a function or map. These are both ways of translating one type of mathematical structure into another (or at least mapping one element in a structure to a different element in the same structure). In particular, we've spent a good amount of time working with boundary maps that mapped a higher-dimensional chain group to a lower dimensional chain group that preserved the structure of the original group in some way (it is a homomorphism).

So just like we can have a map between two groups, we can have a map between two vector spaces. And we call (linear) maps between vector spaces, __matrices__. A matrix basically applies a linear transformation on a vector space (or individual vector element) producing new vector space. A _linear_ transformation just implies that we can only transform the vector space via the normal vector operations of scaling by a constant and addition of a constant vector.

>__Definition (Linear Tansformation)__

More precisely, a linear transformation $M\ :\ V_1 \rightarrow V_2$ is a map $M$ from the vector spaces $V_1$ to $V_2$ such that $M(V_1 + V_2) = M(V_1) + M(V_2)$ and $M(aV_1) = aM(V_1)$ where $a$ is a scalar.

Let's say we want to map the real-valued volume $\mathbb R^3$ to the plane $\mathbb R^2$.

$$
\begin{aligned}
V_1 &= span\{(1,0,0),(0,1,0),(0,0,1)\} \\
V_2 &= span\{(1,0),(0,1)\}
\end{aligned}
$$

Now, what if I want to map $V_1$ to $V_2$, that is, I want to send every point in $V_1$ to a point in $V_2$. There are many reasons why I might want to do something like this. If I'm making a graphics application, for example, I want to offer an option to rotate images drawn, and this is simply a matter of applying a linear transformation on the pixels.

So we want a map $M : V_1 \rightarrow V_2$, and we'll call this map a matrix. Notice that $V_1$ has a basis with 3 elements, and $V_2$ has a basis with 2 elements. In order to map from one space to the other, we just need to map one basis set to another basis set. And remember, since this is a linear map, all we are allowed to do to a basis is multiply it by a scalar or add another vector to it; we can't do exotic things like square it or take the logarithm.

Let's call the 3 basis elements of $V_1: B_1, B_2, B_3$. Hence, $V_1 = \langle{B_1, B_2, B_3}\rangle$.
Similarly, we'll call the 2 basis elements of $V_2: \beta_1, \beta_2$. Hence, $V_2 = \langle{\beta_1, \beta_2}\rangle$. (Remember the angle brackets $\langle\rangle$ mean span, i.e. the set of all linear combinations of those elements). We can setup equations such that any vector in $V_1$ can be mapped to a vector in $V_2$ by using the fact that each vector space can be defined by their bases.

>_New Notation (Vector)_

To prevent confusion between symbols that refer to scalars and symbols that refer vectors, I will henceforth add a little arrow over every vector $\vec{v}$ to denote it is a vector, not a scalar. Remember, the scalar is just a single element from the underlying field $F$ over which the vector space is defined.

We can define our map $M(V_1) = V_2$ in this way:

$$
\begin{aligned}
M(B_1) &= a\vec \beta_1 + b\vec \beta_2 \mid a,b \in \mathbb R \\
M(B_2) &= c\vec \beta_1 + d\vec \beta_2 \mid c,d \in \mathbb R \\
M(B_3) &= e\vec \beta_1 + f\vec \beta_2 \mid e,f \in \mathbb R \\
\end{aligned}
$$

That is, the map from each basis in $V_1$ is setup as a linear combination of the basis elements in $V_2$. This requires us to define a total of 6 new pieces of data: $a,b,c,d,e,f \in \mathbb R$ required for our mapping. We just have to keep track of the fact that $a,b$ are for mapping to $\beta_1$ and $d,e,f$ are for mapping to $\beta_2$. What's a convenient way to keep track of all of that? Oh I know, a matrix!

$$
M =
\begin{pmatrix}
a & c & e \\
b & d & f \\
\end{pmatrix}
$$

That is a very convenient way to represent our map $M\ :\ V_1 \rightarrow V_2$ indeed. Notice that each _column_ of this matrix corresponds to the "mapping equation" coefficients for each M(B_n). Also notice that the dimensions of this matrix, $2\times3$, corresponds to the dimensions of the two vector spaces we're mapping to and from. That is, any map $\mathbb R^n \rightarrow \mathbb R^m$ will be represented as an $m\times n$ matrix. It is important to keep in mind that since the linear map (and hence the matrix) depend on coefficients applied to a basis, then the matrix elements will change if one uses a different basis.

Knowing this we can easily see how vector-matrix multiplication _should_ work and why the dimensions of a matrix and vector have to correspond. Namely, a $n \times m$ vector/matrix multiplied by a $j \times k$ vector/matrix must produce a $n \times k$ vector/matrix, and for it to work at all, $m = j$.

This is how we can multiply our matrix map $M$ by any vector in $V_1$ to produce the mapped-to vector in $V_2$:

$$
M(\vec v\ \in\ V_1)
=
\underbrace{
\begin{bmatrix}
a & c & e \\
b & d & f \\
\end{bmatrix}}_{M:V_1\rightarrow V_2}
\underbrace{
\begin{pmatrix}
x \\ y \\ z \\
\end{pmatrix}}_{\vec v\ \in\ V_1}
=
\underbrace{
\begin{bmatrix}
a * x & c * y & e * z \\
b * x & d * y & f * z \\
\end{bmatrix}}_{M:V_1\rightarrow V_2}
=
\begin{pmatrix}
a * x + c * y + e * z \\
b * x + d * y + f * z \\
\end{pmatrix}
$$

Ok now we know a matrix is a linear map between two vector spaces. But what happens if you multiply two matrices together? Well that is just a composition of maps. For example, we have three vector spaces $T, U, V$ and two linear maps $m_1, m_2$:

$$ T \stackrel{m_1}{\rightarrow} U \stackrel{m_2}{\rightarrow} V$$

To get from $T$ to $V$, we need to apply both maps: $m_2(m_1(T)) = V$. Hence, multiplying two matrices together gives us a composition of maps $m_2 \circ m_1$. The _identity_ matrix is an identity map (i.e. it doesn't change the input) that takes the form where 1s are along the diagonal and 0s everywhere else, e.g.:

$$
m=
\begin{bmatrix}
\ddots & 0 & 0 & 0 & ⋰ \\
0 & 1 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 \\
0 & 0 & 0 & 1 & 0 \\
⋰ & 0 & 0 & 0 & \ddots \\
\end{bmatrix}
\\
m \vec V = \vec V
$$

#### Back to simplicial homology (again)

We learned all of that so we can represent the boundary map $\partial(C_n)$ as a __matrix__ so we can apply the tools of linear algebra. This makes sense since we already know that the chain groups $C_n$ can be viewed as vector spaces when we allow scalar multiplication, so then a linear map between the chain vector spaces is the boundary map, that we can represent as a matrix.

We represent an $n$-boundary map, i.e. $\partial(C_n)$, where $n$ is the dimension of the chain group, $k$ is the number of simplices in $C_n$ and $l$ is the number of simplices in $C_{n-1}$, as a matrix with $k$ columns and $l$ rows. Thus each column represents a simplex in $C_n$ and each row represents a simplex in $C_{n-1}$. We put a $1$ in a cell of the matrix if the simplex in that column maps to the simplex in that row. For example, $\partial([a,b]) = a - b$ if the field is $\mathbb Z$, so we will put a $1$ in the row for $a$ and $b$ since the 1-simplex $[a,b]$ maps to those two 0-simplices.

Let's try calculating the homology groups of the previous simplicial complex (depicted below again) using matrices and vectors. We're going to go back to using $\mathbb Z_2$ as our field (so simplex orientation can be ignored) because it is computationally more efficient to do so.

$$ S = \text{ {[a], [b], [c], [d], [a, b], [b, c], [c, a], [c, d], [d, b], [a, b, c]} } $$

Since we're using the (very small) finite field $\mathbb Z_2$ then we can actually list out all the vectors in our chain (group) vector space. We have 3 chain groups, namely the group of 0-simplices (vertices), 1-simplices (edges), and 2-simplices (triangle).

In our example, we only have a single 2-simplex: [a,b,c], thus the group it generates over the field $\mathbb Z_2$ is only $\{0, [a,b,c]\}$ which is isomorphic to $\mathbb Z_2$. Recall, in general, the group generated by the number $n$ of $p$-simplices in a simplicial complex is isomorphic to $\mathbb Z^n_2$. For a computer to understand, we can encode the group elements just using their coefficients 1 or 1. So, for example, the group generated by $[a,b,c]$ can just be represented as $\{0,1\}$. Or the group generated by the 0-simplices $\{a, b, c, d\}$ can be represented by 4-dimensional vectors, for example, if a group element is $a+b+c$ then we encode this as $(1, 1, 1, 0)$ where each position represents the presence or abscence of $(a, b, c, d)$, respectively.

Here are all the chain groups represented as vectors with just coefficients (I didn't list all elements for $C_1$ since there are so many [32]):

$$
\begin{align}
C_0
&=
\left\{
\begin{array}{ll}
(0,0,0,0) & (1,0,0,0) & (0,1,1,0) & (0,1,0,1) \\
(0,1,0,0) & (0,0,1,0) & (0,0,1,1) & (0,1,1,1) \\
(0,0,0,1) & (1,1,0,0) & (1,0,0,1) & (1,0,1,1) \\
(1,1,1,0) & (1,1,1,1) & (1,0,1,0) & (1,1,0,1) \\
\end{array}
\right.
& \cong \mathbb Z^4_2
\\
C_1
&=
\left\{
\begin{array}{ll}
(0,0,0,0,0) & (1,0,0,0,0) & (0,1,1,0,0) & (0,1,0,1,0) \\
(0,1,0,0,0) & (0,0,1,0,0) & (0,0,1,1,0) & (0,1,1,1,0) \\
\dots
\end{array}
\right.
& \cong \mathbb Z^5_2
\\
C_2
&=
\left\{
\begin{array}{ll}
0 & 1
\end{array}
\right.
& \cong \mathbb Z_2
\end{align}
$$

To represent the boundary map (which is a linear map) of the group of $p$-simplices as a matrix, we set the columns to represent each $p$-simplex in the group, and the rows represent each $(p-1)$-simplex. We put a $1$ in each position of the matrix if the $(p-1)$-simplex row is a _face_ of the $p$-simplex column.

We index rows and columns as an ordered pair $(i, j)$ respectively. Thus the element $a_{2,3}$ is the element in the 2nd row (from the top) and the 3rd column (from the left).

The generic boundary matrix is thus (each column is a $p$-simplex, each row is a $(p-1)$-simplex):

$$ \begin{align}
\partial_p
&=
\begin{pmatrix}
a_{1,1} & a_{1,2} & a_{1,3} & \cdots & a_{1,j}  \\
a_{2,1} & a_{2,2} & a_{2,3} & \cdots & a_{2,j}  \\
a_{3,1} & a_{3,2} & a_{3,3} & \cdots & a_{3,j}  \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
a_{i,1} & a_{i,2} & a_{i,3} & \cdots & a_{i,j}
\end{pmatrix}
\end{align}
$$

We'll start by representing the boundary map $\partial(C_2)$ as a matrix. There's only one 2-simplex in $C_2$ so there is only one column, but there are five 1-simplices in $C_1$ so there are 5 rows.

$$
\partial_2
=
\begin{array}{c|lcr}
\partial & [a,b,c] \\
\hline
[a,b] & 1 \\
[b,c] & 1 \\
[c,a] & 1 \\
[c,d] & 0 \\
[d,b] & 0 \\
\end{array}
$$

We put a $1$ if each row-element was a face of the simplex $[a,b,c]$. This matrix makes sense as a linear map because if we multiply it by a vector element in $C_2$ (there's only 1, besides the 0 element) we get what we expect:

$$
\begin{align}
\begin{pmatrix}
1 \\
1 \\
1 \\
0 \\
0 \\
\end{pmatrix} *
0 \qquad
&=
\qquad
\begin{pmatrix}
0 \\
0 \\
0 \\
0 \\
0 \\
\end{pmatrix} \\
\begin{pmatrix}
1 \\
1 \\
1 \\
0 \\
0 \\
\end{pmatrix} *
1 \qquad
&=
\qquad
\begin{pmatrix}
1 \\
1 \\
1 \\
0 \\
0 \\
\end{pmatrix}
\end{align}
$$

Okay, let's move on to building the boundary matrix $\partial(C_1)$:

$$
\partial_1 =
\begin{array}{c|lcr}
\partial & [a,b] & [b,c] & [c,a] & [c,d] & [d,b] \\
\hline
a & 1 & 0 & 1 & 0 & 0 \\
b & 1 & 1 & 0 & 0 & 1 \\
c & 0 & 1 & 1 & 1 & 0 \\
d & 0 & 0 & 0 & 1 & 1 \\
\end{array}
$$

Does this make sense? Let's check with some python/numpy. Let's take an arbitrary element from the group of 1-chains, namely: $[a,b]+[c,a]+[c,d]$ which we've encoded as $(1,0,1,1,0)$ and apply the boundary matrix and see what we get.

```python
import numpy as np
b1 = np.matrix([[1,0,1,0,0],[1,1,0,0,1],[0,1,1,1,0],[0,0,0,1,1]]) #boundary matrix C_1
el = np.matrix([1,0,1,1,0]) #random element from C_1
np.fmod(b1 * el.T, 2) # we want integers modulo 2
```

    matrix([[0],
            [1],
            [0],
            [1]])

$\require{cancel}$

Recall that $(0,1,0,1)$ translates to $b+d$. By hand we can calculate the boundary and compare:
$$\partial([a,b]+[c,a]+[c,d]) = a+b+c+a+c+d = \cancel{a}+b+\cancel{c}+\cancel{a}+\cancel{c}+d = b+d = (0,1,0,1)$$

It works!

Lastly, we need the boundary matrix for $C_0$ which is trivial since the boundary of $0$-simplices always maps to $0$, so $$
\partial_0 =
\begin{pmatrix}
0 & 0 & 0 & 0 \\
\end{pmatrix}
$$

Okay, now we have our three boundary matrices, how do we calculate the Betti numbers? Well recall that sequence of subgroups of a chain group: $ B_n \leq Z_n \leq C_n $ which are the group of boundaries, group of cycles, and chain group, respectively.

Also recall that Betti $b_n = dim(Z_n\ /\ B_n)$. But that's when things were represented as just sets with group structure, now everything is represented as vectors and matrices, so instead we define the Betti number $b_n = rank(Z_n)\ -\ rank(B_n)$. What does __rank__ mean? Rank and dimension are related but not the same. If we think of the columns of a matrix as a set basis vectors: $\beta_1, \beta_2, \dots \beta_k$ then the dimension of the span of those column vectors $\langle \beta_1, \beta_2, \dots \beta_k \rangle$ is the rank of the matrix. It turns out that you can also use the rows and it will have the same result. Importantly, however, dimension is defined on the smallest set of basis elements, i.e. the basis elements that are linearly independent.

The boundary matrix $\partial_n$ contains the information for the chain group and cycles subgroup, and the $B_{n-1}$ boundary subgroup, all the information we need to calculate the Betti number. Unfortunately, in general, our naïve approach of building the boundary matrix is not in a form where the group and subgroup information is readily accessible. We need to modify the boundary matrix, without disturbing the mapping information it contains, into a new form called __Smith normal form__. Basically, the smith normal form of a matrix will have $1$s along the diagonal starting from the top left of the matrix and $0$s everywhere else.

For example,
$$
\begin{align}
\text{Smith normal form}
&:\
\begin{pmatrix}
1 & 0 & 0 & \cdots & 0  \\
0 & 1 & 0 & \cdots & 0  \\
0 & 0 & 1 & \cdots & 0  \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
0 & 0 & 0 & \cdots & ?
\end{pmatrix}
\end{align}
$$

Notice the $1$s along the diagonal do not necessarily need to extend all the way down to the bottom right. And here's the information available once it's in Smith normal form (the red diagonal box indicates the $1$s):

(Source: "COMPUTATIONAL TOPOLOGY" by Edelsbrunner and Harer, pg. 104)

So how do we get a matrix into Smith normal form? We do so by playing a game involving manipulating the matrix according to some rules. Here are the two allowed operations on the matrix:
1. You can swap any two columns or any two rows in the matrix.
2. You can add a column to another column, or a row to another row.

Now you just need to apply these operations until you get the matrix in Smith normal form. I should point out that this process is alot easier when we use the field $\mathbb Z_2$. Let's try it out on the boundary matrix for $C_1$.

$$
\partial_1 =
\begin{array}{c|lcr}
\partial & [a,b] & [b,c] & [c,a] & [c,d] & [d,b] \\
\hline
a & 1 & 0 & 1 & 0 & 0 \\
b & 1 & 1 & 0 & 0 & 1 \\
c & 0 & 1 & 1 & 1 & 0 \\
d & 0 & 0 & 0 & 1 & 1 \\
\end{array}
$$

We already have 1s across the diagonal, but we have a lot of 1s not along the diagonal.

Steps: Add column 3 to 5, then add column 4 to 5, then add column 5 to 1, then swap columns 1 and 5:
$$
\partial_1 =
\begin{pmatrix}
1 & 0 & 1 & 0 & 0 \\
1 & 1 & 0 & 0 & 0 \\
0 & 1 & 1 & 1 & 0 \\
0 & 0 & 0 & 1 & 0 \\
\end{pmatrix}
$$

Steps: Add column 1 to 3, add column 2 to 3, swap columns 3 and 4, add row 1 to 2, add row 4 to 2, add row 3 to 2, add row 4 to 3, swap rows 3 and 2, swap rows 4 and 3. Stop.
$$
\text{Smith normal form: }
\begin{pmatrix}
1 & 0 & 0 & 0 & 0 \\
0 & 1 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 \\
0 & 0 & 0 & 0 & 0 \\
\end{pmatrix}
$$
Once we have the matrix in Smith normal form, we don't do any more operations. We could of course continue adding rows and columns together until we get a matrix of only 0s, but that wouldn't be very helpful! I sort of randomly added rows/columns to get it into the Smith normal form, but there really is an algorithm that can do it relatively efficiently.

Rather than walk through the detailed implementation of the Smith normal form algorithm, I will merely use an existing algorithm:

```python
def reduce_matrix(matrix):
    #Returns [reduced_matrix, rank, nullity]
    if np.size(matrix)==0:
        return [matrix,0,0]
    m=matrix.shape[0]
    n=matrix.shape[1]

    def _reduce(x):
        #We recurse through the diagonal entries.
        #We move a 1 to the diagonal entry, then
        #knock out any other 1s in the same  col/row.
        #The rank is the number of nonzero pivots,
        #so when we run out of nonzero diagonal entries, we will
        #know the rank.
        nonzero=False
        #Searching for a nonzero entry then moving it to the diagonal.
        for i in range(x,m):
            for j in range(x,n):
                if matrix[i,j]==1:
                    matrix[[x,i],:]=matrix[[i,x],:]
                    matrix[:,[x,j]]=matrix[:,[j,x]]
                    nonzero=True
                    break
            if nonzero:
                break
        #Knocking out other nonzero entries.
        if nonzero:
            for i in range(x+1,m):
                if matrix[i,x]==1:
                    matrix[i,:] = np.logical_xor(matrix[x,:], matrix[i,:])
            for i in range(x+1,n):
                if matrix[x,i]==1:
                    matrix[:,i] = np.logical_xor(matrix[:,x], matrix[:,i])
            #Proceeding to next diagonal entry.
            return _reduce(x+1)
        else:
            #Run out of nonzero entries so done.
            return x
    rank=_reduce(0)
    return [matrix, rank, n-rank]

# Source: < https://triangleinequality.wordpress.com/2014/01/23/computing-homology/ >
```

```python
reduce_matrix(b1)
#Returns the matrix in Smith normal form as well as rank(B_n-1) and rank(Z_n)
```

    [matrix([[1, 0, 0, 0, 0],
             [0, 1, 0, 0, 0],
             [0, 0, 1, 0, 0],
             [0, 0, 0, 0, 0]]), 3, 2]

As you can see we got the same result by hand, but surely the algorithm was more efficient.

Since each boundary map gives us $Z_n$ (cycles) and $B_{n-1}$ (boundary for (n-1)-chain group) we need both $\partial_n$ and $\partial_{n+1}$ in order to calculate the Betti number for chain group $n$. Remember, we now calculate Betti numbers as

Betti $b_n = rank(Z_n) - rank(B_n)$

Let's start calculating those Betti numbers.

```python
#Initialize boundary matrices
boundaryMap0 = np.matrix([[0,0,0,0]])
boundaryMap1 = np.matrix([[1,0,1,0,0],[1,1,0,0,1],[0,1,1,1,0],[0,0,0,1,1]])
boundaryMap2 = np.matrix([[1,1,1,0,0]])

#Smith normal forms of the boundary matrices
smithBM0 = reduce_matrix(boundaryMap0)
smithBM1 = reduce_matrix(boundaryMap1)
smithBM2 = reduce_matrix(boundaryMap2)

#Calculate Betti numbers
betti0 = (smithBM0[2] - smithBM1[1])
betti1 = (smithBM1[2] - smithBM2[1])
betti2 = 0  #There is no n+1 chain group, so the Betti is 0

print(smithBM0)
print(smithBM1)
print(smithBM2)
print("Betti #0: %s \n Betti #1: %s \n Betti #2: %s" % (betti0, betti1, betti2))
```

    [matrix([[0, 0, 0, 0]]), 0, 4]
    [matrix([[1, 0, 0, 0, 0],
            [0, 1, 0, 0, 0],
            [0, 0, 1, 0, 0],
            [0, 0, 0, 0, 0]]), 3, 2]
    [matrix([[1, 0, 0, 0, 0]]), 1, 4]
    Betti #0: 1
     Betti #1: 1
     Betti #2: 0

Great it worked!

But we skipped an important step. We designed the boundary matrices by hand initially, in order to algorithm-ize the entire process from building a simplicial complex over data to computing Betti numbers, we need an algorithm that takes a simplicial complex and builds the boundary matrices. Let's tackle that now.

```python
#return the n-simplices in a complex
def nSimplices(n, complex):
    nchain = []
    for simplex in complex:
        if len(simplex) == (n+1):
            nchain.append(simplex)
    if (nchain == []): nchain = [0]
    return nchain

#check if simplex is a face of another simplex
def checkFace(face, simplex):
    if simplex == 0:
        return 1
    elif set(face) < set(simplex): #if face is a subset of simplex
        return 1
    else:
        return 0

#build boundary matrix for dimension n ---> (n-1) = p
def boundaryMatrix(nchain, pchain):
    bmatrix = np.zeros((len(nchain),len(pchain)))
    i = 0
    for nSimplex in nchain:
        j = 0
        for pSimplex in pchain:
            bmatrix[i, j] = checkFace(pSimplex, nSimplex)
            j += 1
        i += 1
    return bmatrix.T
```

Those are very simple helper functions that we'll use to build the boundary matrix and then use the previously described reduction algorithm to get it into Smith normal form. Remember, the simplicial complex example we're using looks like this:

I've just replaced {a,b,c,d} with {0,1,2,3} so Python can understand it.

```python
S = [{0}, {1}, {2}, {3}, {0, 1}, {1, 2}, {2, 0}, {2, 3}, {3, 1}, {0, 1, 2}] #this is our simplex from above

chain2 = nSimplices(1, S)
chain1 = nSimplices(0, S)
reduce_matrix(boundaryMatrix(chain2, chain1))
```

    [array([[ 1.,  0.,  0.,  0.,  0.],
            [ 0.,  1.,  0.,  0.,  0.],
            [ 0.,  0.,  1.,  0.,  0.],
            [ 0.,  0.,  0.,  0.,  0.]]), 3, 2]

Now let's put everything together and make a function that will return all the Betti numbers of a simplicial complex.

```python
def betti(complex):
    max_dim = len(max(complex, key=len)) #get the maximum dimension of the simplicial complex, 2 in our example
    betti_array = np.zeros(max_dim) #setup array to store n-th dimensional Betti numbers
    z_n = np.zeros(max_dim) #number of cycles (from cycle group)
    b_n = np.zeros(max_dim) #b_(n-1) boundary group
    for i in range(max_dim): #loop through each dimension starting from maximum to generate boundary maps
        bm = 0 #setup n-th boundary matrix
        chain2 = nSimplices(i, complex) #n-th chain group
        if i==0: #there is no n+1 boundary matrix in this case
            bm = 0
            z_n[i] = len(chain2)
            b_n[i] = 0
        else:
            chain1 = nSimplices(i-1, complex) #(n-1)th chain group
            bm = reduce_matrix(boundaryMatrix(chain2, chain1))
            z_n[i] = bm[2]
            b_n[i] = bm[1] #b_(n-1)

    for i in range(max_dim): #Calculate betti number: Z_n - B_n
        if (i+1) < max_dim:
            betti_array[i] = z_n[i] - b_n[i+1]
        else:
            betti_array[i] = z_n[i] - 0 #if there are no higher simplices, the boundary group of this chain is 0

    return betti_array
```

Alright, no we should have everything we need to calculate the set of Betti numbers on any arbitrary simplicial complex given in the right format. Keep in mind that all this code is for learning purposes so I've kept it intentionally simple. It is not production ready. It has basically no safety checks so it will just fail if it gets something even slightly unexpected.

But let's see how versatile our procedure is by trying it out on various simplicial complexes.

Let $H = \text{ { {0}, {1}, {2}, {3}, {4}, {5}, {4, 5}, {0, 1}, {1, 2}, {2, 0}, {2, 3}, {3, 1}, {0, 1, 2} } }$

As you can tell this is the same simplicial complex we've been working with except now it has a disconnected edge on the right. Thus we should get Betti=2 for dimension 0 since there are 2 connect components.

```python
H = [{0}, {1}, {2}, {3}, {4}, {5}, {4, 5}, {0, 1}, {1, 2}, {2, 0}, {2, 3}, {3, 1}, {0, 1, 2}]
betti(H)
```

    array([ 2.,  1.,  0.])

Let's try another one, now with 2 cycles and 2 connect components.

Let $Y_1 = \text{ { {0}, {1}, {2}, {3}, {4}, {5}, {6}, {0, 6}, {2, 6}, {4, 5}, {0, 1}, {1, 2}, {2, 0}, {2, 3}, {3, 1}, {0, 1, 2} } }$

```python
Y1 = [{0}, {1}, {2}, {3}, {4}, {5}, {6}, {0, 6}, {2, 6}, {4, 5}, {0, 1}, {1, 2}, {2, 0}, {2, 3}, {3, 1}, {0, 1, 2}]
betti(Y1)
```

    array([ 2.,  2.,  0.])

Here's another. I just added a stranded vertex:

Let $Y_2 = \text{ { {0}, {1}, {2}, {3}, {4}, {5}, {6}, {7}, {0, 6}, {2, 6}, {4, 5}, {0, 1}, {1, 2}, {2, 0}, {2, 3}, {3, 1}, {0, 1, 2} } }$

```python
Y2 = [{0}, {1}, {2}, {3}, {4}, {5}, {6}, {7}, {0, 6}, {2, 6}, {4, 5}, {0, 1}, {1, 2}, {2, 0}, {2, 3}, {3, 1}, {0, 1, 2}]
betti(Y2)
```

    array([ 3.,  2.,  0.])

One last one. This is a hollow tetrahedron:

```python
D = [{0}, {1}, {2}, {3}, {0,1}, {1,3}, {3,2}, {2,0}, {2,1}, {0,3}, {0,1,3}, {0,1,2}, {2,0,3}, {1,2,3}]
betti(D)
```

    array([ 1.,  0.,  1.])

Exactly what we expect! Okay, it looks like we can reliably calculate Betti numbers for any arbitrary simplicial complex.

###### What's next?
These first 4 posts were all just exposition on the math and concepts behind persistent homology, but so far all we've done is (non-persistent) homology. Remember back in part 2 where we wrote an algorithm to build a simplicial complex from data? Well recall that we needed to arbitrarily choose a parameter $\epsilon$ that determined whether or not two vertices were close enough to connect with an edge. If we set a small $\epsilon$ then we'd have a very dense graph with a lot of edges, if we chose a large $\epsilon$ then we'd get a more sparse graph.

The problem is we have no way of knowing what the "correct" $\epsilon$ value should be. We will get dramatically different simplicial complexes (and thus different homology groups and Betti numbers) with varying levels of $\epsilon$. Persistent homology basically says: let's just continuously scale $\epsilon$ from 0 to the maximal value (where all vertices are edge-wise connected) and see which topological features _persist_ the longest. We then believe that topological features (e.g. connected components, cycles) that are short-lived across scaling $\epsilon$ are noise whereas those that are long-lived (i.e. persistent) are _real_ features of the data. So next time we will work on modifying our algorithms to be able to continuously vary $\epsilon$ while tracking changes in the calculated homology groups.

#### References (Websites):

1. http://dyinglovegrape.com/math/topology_data_1.php
2. http://www.math.uiuc.edu/~r-ash/Algebra/Chapter4.pdf
3. https://en.wikipedia.org/wiki/Group_(mathematics)
4. https://jeremykun.com/2013/04/03/homology-theory-a-primer/
5. http://suess.sdf-eu.org/website/lang/de/algtop/notes4.pdf
6. http://www.mit.edu/~evanchen/napkin.html
7. https://triangleinequality.wordpress.com/2014/01/23/computing-homology

#### References (Academic Publications):
7. Basher, M. (2012). On the Folding of Finite Topological Space. International Mathematical Forum, 7(15), 745–752. Retrieved from http://www.m-hikari.com/imf/imf-2012/13-16-2012/basherIMF13-16-2012.pdf

8. Day, M. (2012). Notes on Cayley Graphs for Math 5123 Cayley graphs, 1–6.

9. Doktorova, M. (2012). CONSTRUCTING SIMPLICIAL COMPLEXES OVER by, (June).

10. Edelsbrunner, H. (2006). IV.1 Homology. Computational Topology, 81–87. Retrieved from http://www.cs.duke.edu/courses/fall06/cps296.1/

11. Erickson, J. (1908). Homology. Computational Topology, 1–11.

12. Evan Chen. (2016). An Infinitely Large Napkin.

13. Grigor’yan, A., Muranov, Y. V., & Yau, S. T. (2014). Graphs associated with simplicial complexes. Homology, Homotopy and Applications, 16(1), 295–311. http://doi.org/10.4310/HHA.2014.v16.n1.a16

14. Kaczynski, T., Mischaikow, K., & Mrozek, M. (2003). Computing homology. Homology, Homotopy and Applications, 5(2), 233–256. http://doi.org/10.4310/HHA.2003.v5.n2.a8

15. Kerber, M. (2016). Persistent Homology – State of the art and challenges 1 Motivation for multi-scale topology. Internat. Math. Nachrichten Nr, 231(231), 15–33.

16. Khoury, M. (n.d.). Lecture 6 : Introduction to Simplicial Homology Topics in Computational Topology : An Algorithmic View, 1–6.

17. Kraft, R. (2016). Illustrations of Data Analysis Using the Mapper Algorithm and Persistent Homology.

18. Lakshmivarahan, S., & Sivakumar, L. (2016). Cayley Graphs, (1), 1–9.

19. Liu, X., Xie, Z., & Yi, D. (2012). A fast algorithm for constructing topological structure in large data. Homology, Homotopy and Applications, 14(1), 221–238. http://doi.org/10.4310/HHA.2012.v14.n1.a11

20. Naik, V. (2006). Group theory : a first journey, 1–21.

21. Otter, N., Porter, M. A., Tillmann, U., Grindrod, P., & Harrington, H. A. (2015). A roadmap for the computation of persistent homology. Preprint ArXiv, (June), 17. Retrieved from http://arxiv.org/abs/1506.08903

22. Semester, A. (2017). § 4 . Simplicial Complexes and Simplicial Homology, 1–13.

23. Singh, G. (2007). Algorithms for Topological Analysis of Data, (November).

24. Zomorodian, A. (2009). Computational Topology Notes. Advances in Discrete and Computational Geometry, 2, 109–143. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.50.7483

25. Zomorodian, A. (2010). Fast construction of the Vietoris-Rips complex. Computers and Graphics (Pergamon), 34(3), 263–271. http://doi.org/10.1016/j.cag.2010.03.007

26. Symmetry and Group Theory 1. (2016), 1–18. http://doi.org/10.1016/B978-0-444-53786-7.00026-5


## Histograms of Oriented Gradients (HOG)

As we saw with the ORB algorithm, we can use keypoints in images to do keypoint-based matching to detect objects in images. These type of algorithms work great when you want to detect objects that have a lot of consistent internal features that are not affected by the background. For example, these algorithms work well for facial detection because faces have a lot of consistent internal features that don’t get affected by the image background, such as the eyes, nose, and mouth. However, these type of algorithms don’t work so well when attempting to do more general object recognition, say for example, pedestrian detection in images. The reason is that people don’t have consistent internal features, like faces do, because the body shape and style of every person is different (see Fig. 1). This means that every person is going to have a different set of internal features, and so we need something that can more generally describe a person.

Fig. 1. - Pedestrians.

One option is to try to detect pedestrians by their contours instead.  Detecting objects in images by their contours (boundaries) is very challenging because we have to deal with the difficulties brought about by the contrast between the background and the foreground. For example, suppose you wanted to detect a pedestrian in an image that is walking in front of a white building and she is wearing a white coat and black pants (see Fig. 2). We can see in Fig. 2, that since the background of the image is mostly white, the black pants are going to have a very high contrast, but the coat, since it is white as well, is going to have very low contrast. In this case, detecting the edges of pants is going to be easy but detecting the edges of the coat is going to be very difficult. This is where **HOG** comes in. HOG stands for **Histograms of Oriented Gradients** and it was first introduced by Navneet Dalal and Bill Triggs in 2005.

Fig. 2. - High and Low Contrast.

The HOG algorithm works by creating histograms of the distribution of gradient orientations in an image and then normalizing them in a very special way. This special normalization is what makes HOG so effective at detecting the edges of objects even in cases where the contrast is very low. These normalized histograms are put together into a feature vector, known as the HOG descriptor, that can be used to train a machine learning algorithm, such as a Support Vector Machine (SVM), to detect objects in images based on their boundaries (edges). Due to its great success and reliability, HOG has become one of the most widely used algorithms in computer vison for object detection.

In this notebook, you will learn:

* How the HOG algorithm works
* How to use OpenCV to create a HOG descriptor
* How to visualize the HOG descriptor.

# The HOG Algorithm

As its name suggests, the HOG algorithm, is based on creating histograms from the orientation of image gradients. The HOG algorithm is implemented in a series of steps:

1. Given the image of particular object, set a detection window (region of interest) that covers the entire object in the image (see Fig. 3).

2. Calculate the magnitude and direction of the gradient for each individual pixel in the detection window.

3. Divide the detection window into connected *cells* of pixels, with all cells being of the same size (see Fig. 3). The size of the cells is a free parameter and it is usually chosen so as to match the scale of the features that want to be detected. For example, in a 64 x 128 pixel detection window, square cells 6 to 8 pixels wide are suitable for detecting human limbs.

4. Create a Histogram for each cell, by first grouping the gradient directions of all pixels in each cell into a particular number of orientation (angular) bins; and then adding up the gradient magnitudes of the gradients in each angular bin (see Fig. 3). The number of bins in the histogram is a free parameter and it is usually set to 9 angular bins.

5. Group adjacent cells into *blocks* (see Fig. 3). The number of cells in each block is a free parameter and all blocks must be of the same size. The distance between each block (known as the stride) is a free parameter but it is usually set to half the block size, in which case you will get overlapping blocks (*see video below*).  The HOG algorithm has been shown empirically to work better with overlapping blocks.

6. Use the cells contained within each block to normalize the cell histograms in that block (see Fig. 3). If you have overlapping blocks this means that most cells will be normalized with respect to different blocks (*see video below*). Therefore, the same cell may have several different normalizations.

7. Collect all the normalized histograms from all the blocks into a single feature vector called the HOG descriptor.

8. Use the resulting HOG descriptors from many images of the same type of object to train a machine learning algorithm, such as an SVM, to detect those type of objects in images. For example, you could use the HOG descriptors from many images of pedestrians to train an SVM to detect pedestrians in images. The training is done with both positive a negative examples of the object you want detect in the image.

9. Once the SVM has been trained, a sliding window approach is used to try to detect and locate objects in images.  Detecting an object in the image entails finding the part of the image that looks similar to the HOG pattern learned by the SVM.

Fig. 3. - HOG Diagram.

Vid. 1. - HOG Animation.

# Why The HOG Algorithm Works

As we learned above, HOG creates histograms by adding the magnitude of the gradients in particular orientations in localized portions of the image called *cells*.  By doing this we guarantee that stronger gradients will contribute more to the magnitude of their respective angular bin, while the effects of weak and randomly oriented gradients resulting from noise are minimized. In this manner the histograms tell us the dominant gradient orientation of each cell.

### Dealing with contrast

Now, the magnitude of the dominant orientation can vary widely due to variations in local illumination and the contrast between the background and the foreground.

To account for the background-foreground contrast differences, the HOG algorithm tries to detect edges locally.  In order to do this, it defines groups of cells, called **blocks**, and normalizes the histograms using this local group of cells. By normalizing locally, the HOG algorithm can detect the edges in each block very reliably; this is called **block normalization**.

In addition to using block normalization, the HOG algorithm also uses overlapping blocks to increase its performance. By using overlapping blocks, each cell contributes several independent components to the final HOG descriptor, where each component corresponds to a cell being normalized with respect to a different block. This may seem redundant but, it has been shown empirically that by normalizing each cell several times with respect to different local blocks, the performance of the HOG algorithm increases dramatically.

### Loading Images and Importing Resources

The first step in building our HOG descriptor is to load the required packages into Python and to load our image.

We start by using OpenCV to load an image of a triangle tile. Since, the `cv2.imread()` function loads images as BGR we will convert our image to RGB so we can display it with the correct colors. As usual we will convert our BGR image to Gray Scale for analysis.

```python
import cv2
import numpy as np
import matplotlib.pyplot as plt

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Load the image
image = cv2.imread('./images/triangle_tile.jpeg')

# Convert the original image to RGB
original_image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Convert the original image to gray scale
gray_image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

# Print the shape of the original and gray scale images
print('The original image has shape: ', original_image.shape)
print('The gray scale image has shape: ', gray_image.shape)

# Display the images
plt.subplot(121)
plt.imshow(original_image)
plt.title('Original Image')
plt.subplot(122)
plt.imshow(gray_image, cmap='gray')
plt.title('Gray Scale Image')
plt.show()
```

    The original image has shape:  (250, 250, 3)
    The gray scale image has shape:  (250, 250)

# Creating The HOG Descriptor

We will be using OpenCV’s `HOGDescriptor` class to create the HOG descriptor. The parameters of the HOG descriptor are setup using the `HOGDescriptor()` function. The parameters of the `HOGDescriptor()` function and their default values are given below:

`cv2.HOGDescriptor(win_size = (64, 128),
                  block_size = (16, 16),
                  block_stride = (8, 8),
                  cell_size = (8, 8),
                  nbins = 9,
                  win_sigma = DEFAULT_WIN_SIGMA,
                  threshold_L2hys = 0.2,
                  gamma_correction = true,
                  nlevels = DEFAULT_NLEVELS)`

Parameters:

* **win_size** – *Size*
Size of detection window in pixels (*width, height*). Defines the region of interest. Must be an integer multiple of cell size.

* **block_size** – *Size*
Block size in pixels (*width, height*). Defines how many cells are in each block. Must be an integer multiple of cell size and it must be smaller than the detection window. The smaller the block the finer detail you will get.

* **block_stride** – *Size*
Block stride in pixels (*horizontal, vertical*). It must be an integer multiple of cell size. The `block_stride` defines the distance between adjecent blocks, for example, 8 pixels horizontally and 8 pixels vertically. Longer `block_strides` makes the algorithm run faster (because less blocks are evaluated) but the algorithm may not perform as well.

* **cell_size** – *Size*
Cell size in pixels (*width, height*). Determines the size fo your cell. The smaller the cell the finer detail you will get.

* **nbins** – *int*
Number of bins for the histograms. Determines the number of angular bins used to make the histograms. With more bins you capture more gradient directions. HOG uses unsigned gradients, so the angular bins will have values between 0 and 180 degrees.

* **win_sigma** – *double*
Gaussian smoothing window parameter. The performance of the HOG algorithm can be improved by smoothing the pixels near the edges of the blocks by applying a Gaussian spatial window to each pixel before computing the histograms.

* **threshold_L2hys** – *double*
L2-Hys (Lowe-style clipped L2 norm) normalization method shrinkage. The L2-Hys method is used to normalize the blocks and it consists of an L2-norm followed by clipping and a renormalization. The clipping limits the maximum value of the descriptor vector for each block to have the value of the given threshold (0.2 by default). After the clipping the descriptor vector is renormalized as described in *IJCV*, 60(2):91-110, 2004.

* **gamma_correction** – *bool*
Flag to specify whether the gamma correction preprocessing is required or not. Performing gamma correction slightly increases the performance of the HOG algorithm.

* **nlevels** – *int*
Maximum number of detection window increases.

As we can see, the  `cv2.HOGDescriptor()`function supports a wide range of parameters. The first few arguments (`block_size, block_stride, cell_size`, and `nbins`) are probably the ones you are most likely to change. The other parameters can be safely left at their default values and you will get good results.

In the code below, we will use the `cv2.HOGDescriptor()`function to set the cell size, block size, block stride, and the number of bins for the histograms of the HOG descriptor. We will then use `.compute(image)`method to compute the HOG descriptor (feature vector) for the given `image`.

```python
# Specify the parameters for our HOG descriptor

# Cell Size in pixels (width, height). Must be smaller than the size of the detection window
# and must be chosen so that the resulting Block Size is smaller than the detection window.
cell_size = (6, 6)

# Number of cells per block in each direction (x, y). Must be chosen so that the resulting
# Block Size is smaller than the detection window
num_cells_per_block = (2, 2)

# Block Size in pixels (width, height). Must be an integer multiple of Cell Size.
# The Block Size must be smaller than the detection window
block_size = (num_cells_per_block[0] * cell_size[0],
              num_cells_per_block[1] * cell_size[1])

# Calculate the number of cells that fit in our image in the x and y directions
x_cells = gray_image.shape[1] // cell_size[0]
y_cells = gray_image.shape[0] // cell_size[1]

# Horizontal distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (x_cells - num_cells_per_block[0]) / h_stride = integer.
h_stride = 1

# Vertical distance between blocks in units of Cell Size. Must be an integer and it must
# be set such that (y_cells - num_cells_per_block[1]) / v_stride = integer.
v_stride = 1

# Block Stride in pixels (horizantal, vertical). Must be an integer multiple of Cell Size
block_stride = (cell_size[0] * h_stride, cell_size[1] * v_stride)

# Number of gradient orientation bins
num_bins = 9

# Specify the size of the detection window (Region of Interest) in pixels (width, height).
# It must be an integer multiple of Cell Size and it must cover the entire image. Because
# the detection window must be an integer multiple of cell size, depending on the size of
# your cells, the resulting detection window might be slightly smaller than the image.
# This is perfectly ok.
win_size = (x_cells * cell_size[0] , y_cells * cell_size[1])

# Print the shape of the gray scale image for reference
print('\nThe gray scale image has shape: ', gray_image.shape)
print()

# Print the parameters of our HOG descriptor
print('HOG Descriptor Parameters:\n')
print('Window Size:', win_size)
print('Cell Size:', cell_size)
print('Block Size:', block_size)
print('Block Stride:', block_stride)
print('Number of Bins:', num_bins)
print()

# Set the parameters of the HOG descriptor using the variables defined above
hog = cv2.HOGDescriptor(win_size, block_size, block_stride, cell_size, num_bins)

# Compute the HOG Descriptor for the gray scale image
hog_descriptor = hog.compute(gray_image)
```

    The gray scale image has shape:  (250, 250)

    HOG Descriptor Parameters:

    Window Size: (246, 246)
    Cell Size: (6, 6)
    Block Size: (12, 12)
    Block Stride: (6, 6)
    Number of Bins: 9

# Number of Elements In The HOG Descriptor

The resulting HOG Descriptor (feature vector), contains the normalized histograms from all cells from all blocks in the detection window concatenated in one long vector. Therefore, the size of the HOG feature vector will be given by the total number of blocks in the detection window, multiplied by the number of cells per block, times the number of orientation bins:

\begin{equation}
\mbox{total_elements} = (\mbox{total_number_of_blocks})\mbox{ } \times \mbox{ } (\mbox{number_cells_per_block})\mbox{ } \times \mbox{ } (\mbox{number_of_bins})
\end{equation}

If we don’t have overlapping blocks (*i.e.* the `block_stride`equals the `block_size`), the total number of blocks can be easily calculated by dividing the size of the detection window by the block size. However, in the general case we have to take into account the fact that we have overlapping blocks. To find the total number of blocks in the general case (*i.e.* for any `block_stride` and `block_size`), we can use the formula given below:

\begin{equation}
\mbox{Total}_i = \left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right)\left( \frac{\mbox{window_size}_i}{\mbox{block_size}_i} \right) - \left [\left( \frac{\mbox{block_size}_i}{\mbox{block_stride}_i} \right) - 1 \right]; \mbox{  for  } i = x,y
\end{equation}

Where Total$_x$, is the total number of blocks along the width of the detection window, and Total$_y$, is the total number of blocks along the height of the detection window. This formula for Total$_x$ and Total$_y$, takes into account the extra blocks that result from overlapping. After calculating Total$_x$ and Total$_y$, we can get the total number of blocks in the detection window by multiplying Total$_x$  $\times$ Total$_y$. The above formula can be simplified considerably because the `block_size`, `block_stride`, and `window_size`are all defined in terms of the `cell_size`. By making all the appropriate substitutions and cancelations the above formula reduces to:

\begin{equation}
\mbox{Total}_i = \left(\frac{\mbox{cells}_i - \mbox{num_cells_per_block}_i}{N_i}\right) + 1\mbox{  }; \mbox{  for  } i = x,y
\end{equation}

Where cells$_x$ is the total number of cells along the width of the detection window, and cells$_y$, is the total number of cells along the height of the detection window. And $N_x$ is the horizontal block stride in units of `cell_size` and $N_y$ is the vertical block stride in units of `cell_size`.

Let's calculate what the number of elements for the HOG feature vector should be and check that it matches the shape of the HOG Descriptor calculated above.

```python
# Calculate the total number of blocks along the width of the detection window
tot_bx = np.uint32(((x_cells - num_cells_per_block[0]) / h_stride) + 1)

# Calculate the total number of blocks along the height of the detection window
tot_by = np.uint32(((y_cells - num_cells_per_block[1]) / v_stride) + 1)

# Calculate the total number of elements in the feature vector
tot_els = (tot_bx) * (tot_by) * num_cells_per_block[0] * num_cells_per_block[1] * num_bins

# Print the total number of elements the HOG feature vector should have
print('\nThe total number of elements in the HOG Feature Vector should be: ',
      tot_bx, 'x',
      tot_by, 'x',
      num_cells_per_block[0], 'x',
      num_cells_per_block[1], 'x',
      num_bins, '=',
      tot_els)

# Print the shape of the HOG Descriptor to see that it matches the above
print('\nThe HOG Descriptor has shape:', hog_descriptor.shape)
print()
```

    The total number of elements in the HOG Feature Vector should be:  40 x 40 x 2 x 2 x 9 = 57600

    The HOG Descriptor has shape: (57600, 1)

# Visualizing The HOG Descriptor

We can visualize the HOG Descriptor by plotting the histogram associated with each cell as a collection of vectors. To do this, we will plot each bin in the histogram as a single vector whose magnitude is given by the height of the bin and its orientation is given by the angular bin that its associated with. Since any given cell might have multiple histograms associated with it, due to the overlapping blocks, we will choose to average all the histograms for each cell to produce a single histogram for each cell.

OpenCV has no easy way to visualize the HOG Descriptor, so we have to do some manipulation first in order to visualize it. We will start by reshaping the HOG Descriptor in order to make our calculations easier.  We will then compute the average histogram of each cell and finally we will convert the histogram bins into vectors. Once we have the vectors, we plot the corresponding vectors for each cell in an image.

The code below produces an interactive plot so that you can interact with the figure. The figure contains:
* the grayscale image,
* the HOG Descriptor (feature vector),
* a zoomed-in portion of the HOG Descriptor, and
* the histogram of the selected cell.

**You can click anywhere on the gray scale image or the HOG Descriptor image to select a particular cell**. Once you click on either image a *magenta* rectangle will appear showing the cell you selected. The Zoom Window will show you a zoomed in version of the HOG descriptor around the selected cell; and the histogram plot will show you the corresponding histogram for the selected cell. The interactive window also has buttons at the bottom that allow for other functionality, such as panning, and giving you the option to save the figure if desired. The home button returns the figure to its default value.

**NOTE**: If you are running this notebook in the Udacity workspace, there is around a 2 second lag in the interactive plot. This means that if you click in the image to zoom in, it will take about 2 seconds for the plot to refresh.

```python
%matplotlib notebook

import copy
import matplotlib.patches as patches

# Set the default figure size
plt.rcParams['figure.figsize'] = [9.8, 9]

# Reshape the feature vector to [blocks_y, blocks_x, num_cells_per_block_x, num_cells_per_block_y, num_bins].
# The blocks_x and blocks_y will be transposed so that the first index (blocks_y) referes to the row number
# and the second index to the column number. This will be useful later when we plot the feature vector, so
# that the feature vector indexing matches the image indexing.
hog_descriptor_reshaped = hog_descriptor.reshape(tot_bx,
                                                 tot_by,
                                                 num_cells_per_block[0],
                                                 num_cells_per_block[1],
                                                 num_bins).transpose((1, 0, 2, 3, 4))

# Print the shape of the feature vector for reference
print('The feature vector has shape:', hog_descriptor.shape)

# Print the reshaped feature vector
print('The reshaped feature vector has shape:', hog_descriptor_reshaped.shape)

# Create an array that will hold the average gradients for each cell
ave_grad = np.zeros((y_cells, x_cells, num_bins))

# Print the shape of the ave_grad array for reference
print('The average gradient array has shape: ', ave_grad.shape)

# Create an array that will count the number of histograms per cell
hist_counter = np.zeros((y_cells, x_cells, 1))

# Add up all the histograms for each cell and count the number of histograms per cell
for i in range (num_cells_per_block[0]):
    for j in range(num_cells_per_block[1]):
        ave_grad[i:tot_by + i,
                 j:tot_bx + j] += hog_descriptor_reshaped[:, :, i, j, :]

        hist_counter[i:tot_by + i,
                     j:tot_bx + j] += 1

# Calculate the average gradient for each cell
ave_grad /= hist_counter

# Calculate the total number of vectors we have in all the cells.
len_vecs = ave_grad.shape[0] * ave_grad.shape[1] * ave_grad.shape[2]

# Create an array that has num_bins equally spaced between 0 and 180 degress in radians.
deg = np.linspace(0, np.pi, num_bins, endpoint = False)

# Each cell will have a histogram with num_bins. For each cell, plot each bin as a vector (with its magnitude
# equal to the height of the bin in the histogram, and its angle corresponding to the bin in the histogram).
# To do this, create rank 1 arrays that will hold the (x,y)-coordinate of all the vectors in all the cells in the
# image. Also, create the rank 1 arrays that will hold all the (U,V)-components of all the vectors in all the
# cells in the image. Create the arrays that will hold all the vector positons and components.
U = np.zeros((len_vecs))
V = np.zeros((len_vecs))
X = np.zeros((len_vecs))
Y = np.zeros((len_vecs))

# Set the counter to zero
counter = 0

# Use the cosine and sine functions to calculate the vector components (U,V) from their maginitudes. Remember the
# cosine and sine functions take angles in radians. Calculate the vector positions and magnitudes from the
# average gradient array
for i in range(ave_grad.shape[0]):
    for j in range(ave_grad.shape[1]):
        for k in range(ave_grad.shape[2]):
            U[counter] = ave_grad[i,j,k] * np.cos(deg[k])
            V[counter] = ave_grad[i,j,k] * np.sin(deg[k])

            X[counter] = (cell_size[0] / 2) + (cell_size[0] * i)
            Y[counter] = (cell_size[1] / 2) + (cell_size[1] * j)

            counter = counter + 1

# Create the bins in degress to plot our histogram.
angle_axis = np.linspace(0, 180, num_bins, endpoint = False)
angle_axis += ((angle_axis[1] - angle_axis[0]) / 2)

# Create a figure with 4 subplots arranged in 2 x 2
fig, ((a,b),(c,d)) = plt.subplots(2,2)

# Set the title of each subplot
a.set(title = 'Gray Scale Image\n(Click to Zoom)')
b.set(title = 'HOG Descriptor\n(Click to Zoom)')
c.set(title = 'Zoom Window', xlim = (0, 18), ylim = (0, 18), autoscale_on = False)
d.set(title = 'Histogram of Gradients')

# Plot the gray scale image
a.imshow(gray_image, cmap = 'gray')
a.set_aspect(aspect = 1)

# Plot the feature vector (HOG Descriptor)
b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
b.invert_yaxis()
b.set_aspect(aspect = 1)
b.set_facecolor('black')

# Define function for interactive zoom
def onpress(event):

    #Unless the left mouse button is pressed do nothing
    if event.button != 1:
        return

    # Only accept clicks for subplots a and b
    if event.inaxes in [a, b]:

        # Get mouse click coordinates
        x, y = event.xdata, event.ydata

        # Select the cell closest to the mouse click coordinates
        cell_num_x = np.uint32(x / cell_size[0])
        cell_num_y = np.uint32(y / cell_size[1])

        # Set the edge coordinates of the rectangle patch
        edgex = x - (x % cell_size[0])
        edgey = y - (y % cell_size[1])

        # Create a rectangle patch that matches the the cell selected above
        rect = patches.Rectangle((edgex, edgey),
                                  cell_size[0], cell_size[1],
                                  linewidth = 1,
                                  edgecolor = 'magenta',
                                  facecolor='none')

        # A single patch can only be used in a single plot. Create copies
        # of the patch to use in the other subplots
        rect2 = copy.copy(rect)
        rect3 = copy.copy(rect)

        # Update all subplots
        a.clear()
        a.set(title = 'Gray Scale Image\n(Click to Zoom)')
        a.imshow(gray_image, cmap = 'gray')
        a.set_aspect(aspect = 1)
        a.add_patch(rect)

        b.clear()
        b.set(title = 'HOG Descriptor\n(Click to Zoom)')
        b.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 5)
        b.invert_yaxis()
        b.set_aspect(aspect = 1)
        b.set_facecolor('black')
        b.add_patch(rect2)

        c.clear()
        c.set(title = 'Zoom Window')
        c.quiver(Y, X, U, V, color = 'white', headwidth = 0, headlength = 0, scale_units = 'inches', scale = 1)
        c.set_xlim(edgex - cell_size[0], edgex + (2 * cell_size[0]))
        c.set_ylim(edgey - cell_size[1], edgey + (2 * cell_size[1]))
        c.invert_yaxis()
        c.set_aspect(aspect = 1)
        c.set_facecolor('black')
        c.add_patch(rect3)

        d.clear()
        d.set(title = 'Histogram of Gradients')
        d.grid()
        d.set_xlim(0, 180)
        d.set_xticks(angle_axis)
        d.set_xlabel('Angle')
        d.bar(angle_axis,
              ave_grad[cell_num_y, cell_num_x, :],
              180 // num_bins,
              align = 'center',
              alpha = 0.5,
              linewidth = 1.2,
              edgecolor = 'k')

        fig.canvas.draw()

# Create a connection between the figure and the mouse click
fig.canvas.mpl_connect('button_press_event', onpress)
plt.show()
```

    The feature vector has shape: (57600, 1)
    The reshaped feature vector has shape: (40, 40, 2, 2, 9)
    The average gradient array has shape:  (41, 41, 9)

    <IPython.core.display.Javascript object>

# Understanding The Histograms

Let's take a look at a couple of snapshots of the above figure to see if the histograms for the selected cell make sense. Let's start looking at a cell that is inside a triangle and not near an edge:

Fig. 4. - Histograms Inside a Triangle.

In this case, since the triangle is nearly all of the same color there shouldn't be any dominant gradient in the selected cell. As we can clearly see in the Zoom Window and the histogram, this is indeed the case. We have many gradients but none of them clearly dominates over the other.

Now let’s take a look at a cell that is near a horizontal edge:

Fig. 5. - Histograms Near a Horizontal Edge.

Remember that edges are areas of an image where the intensity changes abruptly. In these cases, we will have a high intensity gradient in some particular direction. This is exactly what we see in the corresponding histogram and Zoom Window for the selected cell.  In the Zoom Window, we can see that the dominant gradient is pointing up, almost at 90 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 90-degree bin in the histogram to dominate strongly over the others. This is in fact what we see.

Now let’s take a look at a cell that is near a vertical edge:

Fig. 6. - Histograms Near a Vertical Edge.

In this case we expect the dominant gradient in the cell to be horizontal, close to 180 degrees, since that’s the direction in which there is a sharp change in intensity. Therefore, we should expect to see the 170-degree bin in the histogram to dominate strongly over the others. This is what we see in the histogram but we also see that there is another dominant gradient in the cell, namely the one in the 10-degree bin. The reason for this, is because the HOG algorithm is using unsigned gradients, which means 0 degrees and 180 degrees are considered the same. Therefore, when the histograms are being created, angles between 160 and 180 degrees, contribute proportionally to both the 10-degree bin and the 170-degree bin. This results in there being two dominant gradients in the cell near the vertical edge instead of just one.

To conclude let’s take a look at a cell that is near a diagonal edge.

Fig. 7. - Histograms Near a Diagonal Edge.

To understand what we are seeing, let’s first remember that gradients have an *x*-component, and a *y*-component, just like vectors. Therefore, the resulting orientation of a gradient is going to be given by the vector sum of its components. For this reason, on vertical edges the gradients are horizontal, because they only have an x-component, as we saw in Figure 4. While on horizontal edges the gradients are vertical, because they only have a y-component, as we saw in Figure 3. Consequently, on diagonal edges, the gradients are also going to be diagonal because both the *x* and *y* components are non-zero.  Since the diagonal edges in the image are close to 45 degrees, we should expect to see a dominant gradient orientation in the 50-degree bin. This is in fact what we see in the histogram but, just like in Figure 4., we see there are two dominant gradients instead of just one. The reason for this is that when the histograms are being created, angles that are near the boundaries of bins, contribute proportionally to the adjacent bins.  For example, a gradient with an angle of 40 degrees, is right in the middle of the 30-degree and 50-degree bin. Therefore, the magnitude of the gradient is split evenly into the 30-degree and 50-degree bin. This results in there being two dominant gradients in the cell near the diagonal edge instead of just one.

Now that you know how HOG is implemented, in the workspace you will find a notebook named *Examples*. In there, you will be able set your own paramters for the HOG descriptor for various images. Have fun!


---
## 02. La Magneto-HidroDinámica (MHD)

Eduard Larrañaga (ealarranaga@unal.edu.co)

---

### Resumen

Se presentan las generalidades de las ecuaciones de la **M**agneto-**H**idro**D**inámica (MHD).

---

## Enfoques Euleriano y Lagrangiano en el estudio de Fluidos

- **Enfoque Euleriano.** En este enfoque, el elemento de volumen esta fijo espacialmente en el sistema de referencia del laboratorio.  Por ello, el fluido puede ingresar y salr de este elemento a través de la superficie que lo rodea.

- **Enfoque Lagrangiano.** La superficie del elemento de volumen se mueve junto el fluido (i.e. es co-móvil) en el sistema de referencia del fluido. De esta manera NO hay flujo a través de la superficie que rodea el elemento y la masa contenida dentro de su interior es constante. (El elemento puede deformarse, de pendiendo de como es el movimiento del fluido).

---
## Ecuación de Continuidad

La ecuación de continuidad manifiesta la conservación de la masa en la mecánica de fluidos. En el enfoque Euleriano es

\begin{equation}
\frac{\partial \rho}{\partial t} + \vec{\nabla} \cdot (\rho \vec{v}) = 0
\end{equation}

donde $\rho$ es la densidad de masa y $\vec{v}$ es el campo de velocidades del fluido.

En el enfoque Lagrangiano, la ecuación de continuidad toma la forma

\begin{equation}
\frac{d \rho}{d t} + \rho \vec{\nabla} \cdot \vec{v} = 0
\end{equation}

en la cual se ha utilizado el operador de **derivada Lagrangiana** o **derivada temporal total**,

\begin{equation}
\frac{d }{d t}  = \frac{\partial }{\partial t} +  \vec{v} \cdot \vec{\nabla}
\end{equation}

---
## Ecuación de Movimiento

La ecuación de movimiento del fluido (conservación del momento) en el enfoque Lagrangiano es

\begin{equation}
\rho \frac{d \vec{v}}{d t} = - \vec{\nabla} P + \vec{\nabla} \cdot \overset{\text{$\leftrightarrow$}}{\sigma}+ \vec{J}\times \vec{B} + \vec{f}
\end{equation}

donde

- $P$ es la presión del fluido

- $\overset{\text{$\leftrightarrow$}}{\sigma}$ es el tensor de esfuerzos de viscosidad

- $\vec{J}$ es la densidad de corrientes eléctricas y $\vec{B}$ es el campo magnetico, con lo cual el término $\vec{J}\times \vec{B}$ representa la fuerza de Lorentz (No aparece un término de fuerza eléctrica ($\rho_q \vec{E}$) porque, en el caso de fluidos no-relativistas, éste tiene una magnitud despreciable).

- $\vec{f}$ representa otras fuerzas volumétricas presentes. Por ejemplo, aqui puede incluirse la fuerza gravitacional (por unidad de volumen), $\rho \vec{g} = - \rho \vec{\nabla} \phi$.

En el enfoque Euleriano, la ecuación de movimiento es

\begin{equation}
\rho \left( \frac{\partial \vec{v}}{\partial t} +  \vec{v} \cdot \vec{\nabla} \vec{v}\right) = - \vec{\nabla} P + \vec{\nabla} \cdot \overset{\text{$\leftrightarrow$}}{\sigma}+ \vec{J}\times \vec{B} + \vec{f}
\end{equation}

---
## Ecuación de la Energía

La conservación de la energía se establece, en el enfoque Lagrangiano, a través de la ecuación

\begin{equation}
\rho \frac{d e}{d t} = - P \vec{\nabla} \cdot \vec{v} - \vec{\nabla} \cdot \vec{q} + \vec{f} \cdot \vec{v} - \vec{\nabla} \cdot \vec{F}_{rad} + Q^{(vis)} + \eta_e J^2
\end{equation}

en la cual:

- $e$ es la energía interna del fluido

- $- P \vec{\nabla} \cdot \vec{v}$ representa el trabajo hecho por la presión.

- $- \vec{\nabla} \cdot \vec{q}$ es la razón de transporte de energía térmica dentro del gas (por movimiento aleatorio de particulas)

- $\vec{f} \cdot \vec{v}$ es el trabajo hecho por la fuerza volumétrica $\vec{f}$

- $- \vec{\nabla} \cdot \vec{F}_{rad}$ es la razón con la que la energía de radiación se pierde por emisión (o aumentada por absorción)

- $Q^{(vis)}$ es la razón de calentamiento (volumétrico) por viscosidad

- $\eta_e J^2$ es la razón de calentamiento (volumétrico) Ohmico. La función $\eta_e$ es la resistividad eléctrica del fluido.

Utilizando el enfoque Euleriano, la eucaión de la energía toma la forma

\begin{equation}
\rho \frac{\partial (\rho e)}{\partial t} +  \vec{\nabla} \cdot (\rho e \vec{v}) = - P \vec{\nabla} \cdot \vec{v} - \vec{\nabla} \cdot \vec{q} + \vec{f} \cdot \vec{v} - \vec{\nabla} \cdot \vec{F}_{rad} + Q^{(vis)} + \eta_e J^2
\end{equation}

---

## Electrodinámica

Las ecuaciones que describen el campo electromagnético son las ecuaciones de Maxwell,

\begin{align}
\vec{\nabla} \cdot \vec{E} = &\frac{\rho_q}{\epsilon_0}\\
\vec{\nabla} \cdot \vec{B} = &0 \\
\vec{\nabla} \times \vec{E} = &- \frac{\partial \vec{B}}{\partial t} \\
\vec{\nabla} \times \vec{B} = &\mu_0 \vec{J} +  \epsilon_0 \mu_0 \frac{\partial \vec{E}}{\partial t}
\end{align}

donde $\vec{E}$ y $\vec{B}$ son los campos eléctrico y magnético, $\rho_q$ es la denisdad de carga eléctrica, $\vec{J}$ es la denisad de corriente, $\epsilon_0$ es la permitividad eléctrica del vacio y $\mu_0$ es la permeabilidad magnética del vacio.

Al considerar sistemas no-relativistas (esto quiere decir considerando bajas velocidades, $v\ll c$, o equivalentemente bajas frecuencias), estas ecuaciones se aproximan, a primer orden en $\frac{v}{c}$ en la forma

\begin{align}
\vec{\nabla} \cdot \vec{E} = &\frac{\rho_q}{\epsilon_0}\\
\vec{\nabla} \cdot \vec{B} = &0 \\
\vec{\nabla} \times \vec{E} = &- \frac{\partial \vec{B}}{\partial t} \\
\vec{\nabla} \times \vec{B} = &\mu_0 \vec{J}
\end{align}

y además, se puede incorporar la **ley de Ohm**,

\begin{equation}
\vec{E} + \vec{v} \times \vec{B} = \eta_e \vec{J},
\end{equation}

con $\eta_e$ la resistividad eléctrica del fluido.

## Ecuaciones de la MHD y Grados de Libertad

Reuniendo las anteriores ecuaciones se tiene el siguiente sistema diferencial de PDEs,

\begin{align}
\frac{\partial \rho}{\partial t} + \vec{\nabla} \cdot (\rho \vec{v}) = &0\\
\rho \left( \frac{\partial \vec{v}}{\partial t} +  \vec{v} \cdot \vec{\nabla} \vec{v}\right) = &- \vec{\nabla} P + \vec{\nabla} \cdot \overset{\text{$\leftrightarrow$}}{\sigma}+ \vec{J}\times \vec{B} + \vec{f}\\
\rho \frac{\partial (\rho e)}{\partial t} +  \vec{\nabla} \cdot (\rho e \vec{v}) = &- P \vec{\nabla} \cdot \vec{v} - \vec{\nabla} \cdot \vec{q} + \vec{f} \cdot \vec{v} - \vec{\nabla} \cdot \vec{F}_{rad} + Q^{(vis)} + \eta_e J^2\\
\frac{\partial \vec{B}}{\partial t} = &\vec{\nabla} \left( \vec{v} \times \vec{B} - \frac{\eta_e}{\mu_0} \vec{\nabla} \times \vec{B} \right)\\
\vec{\nabla} \cdot \vec{B} = &0
\end{align}

en donde se han re-escrito las ecuaciones de la electrodinámica para evitar algunos de los grados de libertad del sistema. Este sistema de PDEs consta entonces de 9 ecuaciones con un total de 22 variables:
- $e$ :[1]
- $\rho$ :[1]
- $P$ :[1]
- $\vec{v}$ :[3]
- $\vec{B}$ :[3]
- $\overset{\text{$\leftrightarrow$}}{\sigma}$ :[6] (Tensor simétrico de orden 2 en tres dimensiones)
- $\vec{q}$ :[3]
- $\eta_e$ :[1]
- $F_{rad}$ :[3].

De esta forma, el sistema requiere de 13 ecuaciones adicionales para poder ser resuelto.

---
## Ecuaciones de Cierre

Las **ecuaciones de cierre** son relaciones adicionales que permiten completar el sistema para poder solucionarlo.

1. El vector de flujo radiativo se puede expresar como

\begin{equation}
\vec{F}_{rad} = \int d\nu \int d\Omega \hat{n} I_\nu (\hat{n}, \vec{r})
\end{equation}

donde $I_\nu (\hat{n}, \vec{r})$ es la intensidad específica de la radiación en el punto $\vec{r}$ y en dirección $\hat{n}$. Esta intensidad está dada por la ecuación de conservación para el campo de radiación. De esta forma, el flujo radiativo puede escribirse, en casi todos los casos del interés, como una función de la densidad, $\rho$, y la temperatura del gas, $T$; o de la densidad y la presión, $P$. Esta relación constituye **3** de las ecuaciones de cierre requeridas,

\begin{equation}
\vec{F}_{rad} = \vec{F}_{rad} (\rho, P).
\end{equation}

2. El flujo conductivo de calor a través de las fronteras del elemento, $\vec{q}$, mide la razón con la que los movimientos aleatorios (principalmente de electrones) transportan energía dentro del gas y suavizando los gradientes de temperatura. De esta forma, a partir de la teoría cinética de los gases, es posible obtener una dependencia con la densidad,la presión y el campo magnético. Esta relación constituye otras **3** ecuaciones de cierre,

\begin{equation}
\vec{q} = \vec{q} (\rho, P, \vec{B}).
\end{equation}

3. El tensor de esfuerzos, $\overset{\text{$\leftrightarrow$}}{\sigma}$, se construye a partir de la viscosidad y/o de las interacciones con el campo magnético. Por ejemplo, en el caso de un fluido des-magnetizado (i.e. en la Hidrodinámica), se tienen unas componentes del tensor expresadoas en términos de los gradientes de velocidad,

\begin{equation}
\sigma_{ij} = 2\eta \tau_{ij}
\end{equation}

con

\begin{equation}
\tau_{ij} = \frac{1}{2} \left( \partial_i v_j + \partial_j v_i - \frac{2}{3} \partial_k v_k \delta_{ij}\right)
\end{equation}

en donde $\eta = \eta(\rho, P)$ es una propiedad del medio denominada *coeficiente de viscosidad dinámica*.

De esta manera, la expresión para el tensor de esfuerzos constituye **6** ecuaciones de cierre adicionales,

\begin{equation}
\overset{\text{$\leftrightarrow$}}{\sigma} = \overset{\text{$\leftrightarrow$}}{\sigma} (\rho, \vec{v}, \vec{B}).
\end{equation}

De esta manera, ya se tienen 12 de las ecuaciones requeridas para *cerrar* el sistema diferencial. La ecuación final, se denomina la **Ecuación de Estado**.

---
##  Ecuación de Estado

La ecuación de cierre final es una relación entre la densidad y al presión,

\begin{equation}
P = P(\rho).
\end{equation}

La ecuación de estado más común es la relación para los gases ideales, la cual permite modelar muy bien los fluidos en el contexto astrofísico. Esta ecuación es

\begin{equation}
P = \frac{k_B T}{\mu m_H}\rho,
\end{equation}

donde:

- $k_B$ es la constante de
- $T$ es la temperatura del gas
- $m_H \sim m_P$ es la masa del átomo de Hidrogeno (que se puede aproximar a la masa del protón)
- $\mu$ es el peso molecular medio (i.e. masa promedio por partícula de gas medida en unidades de $m_H$ y por lo tanto equivale al inveros del número de partículas en una masa $m_H$ de gas). Este factor vale $\mu=1$ para Hidrógeno neutro y $\mu = 1/2$ para Hidrógeno completamente ionizado. En general, se tiene que $\frac{1}{2} \leq \mu \leq 1$ dependiendo del estado ionizado del gas.

### Ecuación Politrópica

Al considerar un fluido desmagnetizado adiabático (i.e. no existen los términos de intercambio de calor ni contribuciones del campo magnético), la ecuación del gas ideal en combinación con la ecuación de energías da como resultado la **ecuación politrópica**,

\begin{equation}
P \propto \rho^\gamma
\end{equation}

donde $\gamma$ es el **índice adiabático** (razón de calores específicos). Para gases monoatómicos (como lo sería un gas completamente ionizado), el índice adiabático suele tomar valores en el rango $1\leq \gamma \leq \frac{5}{3}$, donde algunos valores de interés a nivel astrofísico son:

- $\gamma = 1$ : Flujo isotermo
- $\gamma = \frac{5}{3}$ : Flujo adiabático. También para un **gas no-relativista completamente degenerado**.
- $\gamma = \frac{4}{3}$ : Gas dominado por presión de radiación. También para un **gas relativista completamente degenerado**.

10. PDE I. Generalidades. Diferencias Finitas/02. MHD.ipynb


Imports.

```python
import sys
sys.path.insert(0, 'utils/')
```

```python
import datetime
import time
import os
import numpy as np
import matplotlib.pyplot as plt
import numba
from numba import njit
from scipy.integrate import quad

import tensorflow as tf
# because of tf version many warnings will occur
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)
print('TF version: {}'.format(tf.__version__))
print('GPU support enabled: {}'.format(tf.test.is_built_with_gpu_support()))
print('CUDA support enabled: {}'.format(tf.test.is_built_with_cuda()))
import deepxde as dde
```

    TF version: 1.15.0
    GPU support enabled: True
    CUDA support enabled: True

Figures configuration for journal papers.

```python
from plotting import latexconfig, figsize
latexconfig()
```

# Pennes Bioheat Transfer Equation

**Ref:** Analytical Study on Bioheat Transfer Problems with Spatial or Transient Heating on Skin Surface or Inside Biological Bodies; Zhong-Shan Deng, Jing Liu; DOI: 10.1115/1.1516810

The 1-D Pennes Bioheat Transfer Equation:

\begin{equation}
k \frac{d^2 T(x)}{dx^2} + \omega_b \cdot T_a - \omega_b \cdot T(x) + Q_m + Q_{ext} = 0
\end{equation}

$k$ is the thermal conductivity of the tissue; $\omega_b$ is the blood perfusion; $T_a$ is the arterial temperature and $T$ stands for the tissue temperature. $Q_m$ is the metabolic heat generation and $Q_{ext}$ is the heat generated from an external source.

Mixed boundary conditions are defined; Robins on the skin surface:

\begin{equation}
  -k \frac{dT_0(x)}{dx}=h_0[T_f - T_0(x)] \hspace{10mm} x=0
\end{equation}

and Dirichlet at maximum heat penetration depth:

\begin{equation}
  T_L(x) = T_c \hspace{10mm} x=L
\end{equation}

where $h_0$, $T_c$ and $T_f$ are the convection coefficient, the body core temperature and the atmospheric temperature, respectively.

****

In this study, time-invariant Pennes' bioheat transfer equation is solved using a novel neural network approach, where calculations were performed for no external source of heating, $Q_{ext} = 0$.

```python
# consts
c_b = 4200
rho_b = 1e3
om_b = 5e-4
c = 4200
h_f = 100;
T_c = 37
L = 3e-2

# random variables
k = 0.5
w_b = c_b*rho_b*om_b
Q_m = 33800
h_0 = 10
T_f = 25
T_a = 37
rvs = [k, w_b, T_a, Q_m, h_0, T_f]
rvs = np.array(rvs).reshape(1, -1)

# solution domain
x = np.linspace(0, L, num=49)
```

## Analytical solution

```python
def analyticSol(x):
    A = np.sqrt(w_b/k)
    lterm = (T_c - T_a - Q_m/w_b) * (A * np.cosh(A * x)  +\
                +  (h_0/k) * np.sinh(A * x)) / (A * np.cosh(A * L) + (h_0/k) * np.sinh(A * L))
    rterm = h_0/k * (T_f - T_a - Q_m/w_b) * np.sinh(A * (L * np.ones(shape=x.shape) - x)) / ( A * np.cosh(A * L) + (h_0/k) * np.sinh(A * L))

    return  T_a + Q_m/w_b + lterm + rterm
```

```python
T_anal = analyticSol(x)

fig = plt.figure(figsize=figsize(1))
ax = fig.add_subplot(111)
ax.plot(x, T_anal, 'r-', label='$T(x)$')
ax.set_xlabel('$x$ [m]')
ax.set_ylabel('$T$ [°C]')
plt.grid()
plt.legend(loc='best')
plt.show()
fig.savefig('{}.pdf'.format('figs/analytical-solution'), bbox_inches='tight')
```

## Monte Carlo Uncertainty Analysis

```python
@njit
def monteCarloSim(rvs, coef_var=0.2, num_samples=1000, num_iters=1000):
    sims_global = np.empty(shape=(num_iters, num_samples, rvs.shape[1]))
    for i in range(num_iters):
        xi = np.random.rand(num_samples, len(rvs)) # random values
        sims = np.zeros(shape=xi.shape)            # empty simulations matrix

        rvs_min = rvs - coef_var*rvs
        rvs_max = rvs + coef_var*rvs

        sims = rvs_min + (rvs_max - rvs_min) * xi
        sims_global[i, :, :] = sims
    return sims_global

@njit
def monteCarloT(x, K, w_b, T_a, Q_m, h_0, T_f):
    A = np.sqrt(w_b/K)
    lterm = (T_c - T_a - Q_m/w_b) * (A * np.cosh(A * x)  +\
                +  (h_0/K) * np.sinh(A * x)) / (A * np.cosh(A * L) + (h_0/K) * np.sinh(A * L))
    rterm = h_0/K * (T_f - T_a - Q_m/w_b) * np.sinh(A * (L * np.ones(shape=x.shape) - x)) / ( A * np.cosh(A * L) + (h_0/K) * np.sinh(A * L))

    return T_a + Q_m/w_b + lterm + rterm

@njit
def monteCarloResample(x, mc):
    num_iters, num_samples, n_rv = mc.shape

    T_mc = np.empty(shape=(num_iters, num_samples, x.shape[0]))

    for it in range(num_iters):
        for sample in range(num_samples):
            T_mc[it, sample, :] = monteCarloT(x, mc[it, sample, 0], mc[it, sample, 1], mc[it, sample, 2], mc[it, sample, 3], mc[it, sample, 4], mc[it, sample, 5])
    return T_mc
```

```python
num_samples = 100
num_iters = 100
coef_var = 0.2

start = time.time()
mc = monteCarloSim(rvs, coef_var, num_samples, num_iters)
T_mc = monteCarloResample(x, mc)
print('Simulation time: {}s'.format(time.time() - start))
```

    Simulation time: 0.04871392250061035s

```python
# single sample
_sample_exmp = 0
T_mc0 = T_mc[_sample_exmp]
T_mc0_mean = np.mean(T_mc0, axis=0)
T_mc0_std = np.std(T_mc0, axis=0)
T_mc0_lower_bound = T_mc0_mean - T_mc0_std
T_mc0_upper_bound = T_mc0_mean + T_mc0_std

fig = plt.figure(figsize=figsize(1, 2))
ax = fig.add_subplot(211)
ax.plot(x, T_mc0_mean, 'r-', label='$<T(x)>$')
ax.plot(x, T_mc0_lower_bound, 'k--', label='$<T(x)> \pm \sigma_T$')
ax.plot(x, T_mc0_upper_bound, 'k--')
ax.fill_between(x, T_mc0_lower_bound, T_mc0_upper_bound, alpha=0.2)
ax.set_xlabel('$x$ [m]')
ax.set_ylabel('$<T>$ [°C]')
plt.legend(loc='best')
# plt.title('Single MC sample')
plt.grid()

# full simulation
T_mc_mean = np.mean(T_mc, axis=1)
T_mean = np.mean(T_mc_mean, axis=0)
T_std = np.std(T_mc_mean, axis=0)
T_lower_bound = T_mc_mean[np.argmin(T_mc_mean[:, 0], axis=0)]
T_upper_bound = T_mc_mean[np.argmax(T_mc_mean[:, 0], axis=0)]

ax = fig.add_subplot(212)
ax.plot(x, T_mean, 'r-', label='MCM ({} samples, {} epochs) mean'.format(num_samples, num_iters))
ax.plot(x, T_lower_bound, 'k--', label='MCM min')
ax.plot(x, T_upper_bound, 'k--', label='MCM max')
ax.fill_between(x, T_lower_bound, T_upper_bound, alpha=0.2)
ax.set_xlabel('$x$ [m]')
ax.set_ylabel('$<T>$ [°C]')
plt.legend(loc='best')
# plt.title('MC num. of epochs: {}'.format(num_iters))
plt.grid()

fig.savefig('{}-{}epochs-{}samples.pdf'.format('figs/mc-simulation', num_iters, num_samples), bbox_inches='tight')
plt.show()
```

```python
# distribution of sample means
def ecdf(data):
    """Compute ECDF for a one-dimensional array of measurements."""
    n = len(data)
    x = np.sort(data)
    y = np.arange(1, n+1) / n
    return x, y

# distribution
_exmp = 0
x_cdf, y_cdf = ecdf(T_mc_mean[:, _exmp])

# theretical gaussian distribution based on the MC simualted data
samples = np.random.normal(np.mean(T_mc_mean[:, _exmp]), np.std(T_mc_mean[:, _exmp]), 10000)
x_theor, y_theor = ecdf(samples)

fig = plt.figure(figsize=figsize(1))
ax = fig.add_subplot(111)
ax.plot(x_cdf, y_cdf, marker='o', alpha=0.6, label='mean of samples on surface \n($x=0$)')
ax.plot(x_theor, y_theor, 'r-', linewidth=2, label='Gaussian fit')
ax.set_xlabel('$T$ [°C]')
ax.set_ylabel('CDF')
plt.grid()
plt.legend(loc='best')

fig.savefig('{}.pdf'.format('figs/mc-distribution'), bbox_inches='tight')
plt.show()
```

## Finite Element Numerical Solution

```python
def assemble_l(lhss, lhs_glob):
    if lhs_glob.shape[0]!=lhs_glob.shape[1]:
        raise Exception("Global matrix is not square matrix!")
    else:
        for i, lhs in enumerate(lhss):
            lhs_glob[i:i+lhs.shape[0], i:i+lhs.shape[1]] += lhs
    return lhs_glob

def assemble_r(rhss, rhs_glob):
    for i, rhs in enumerate(rhss):
        rhs_glob[i:i+rhs.shape[0], ] += rhs
    return rhs_glob
```

```python
# domain
L = 3e-2
x = np.linspace(0, L, 100)

# fem
N = 20 # number of elements

# solution
lhs_glob = np.zeros((N+1, N+1))
T = np.zeros((N+1, 1))
rhs_glob = np.zeros((N+1, 1))
```

```python
# base function combinations
n1 = lambda x, a, b:     (w_b*T_a + Q_m) * (b-x)/(b-a)
n2 = lambda x, a, b:     (w_b*T_a + Q_m) * (x-a)/(b-a)

nn11 = lambda x, a, b:    w_b * ((b-x)/(b-a))**2
nn12 = lambda x, a, b:    w_b * ((x-a)*(b-x))/((b-a)**2)
nn21 = lambda x, a, b:    w_b * ((x-a)*(b-x))/((b-a)**2)
nn22 = lambda x, a, b:    w_b * ((x-a)/(b-a))**2

nn_x11 = lambda x, a, b:  k/((b-a)**2)
nn_x12 = lambda x, a, b:  -k/((b-a)**2)
nn_x21 = lambda x, a, b:  -k/((b-a)**2)
nn_x22 = lambda x, a, b:  k/((b-a)**2)
```

```python
a = 0
dx = L/N
b = dx
lhs = np.zeros((2, 2))
rhs = np.zeros((2, 1))
lhss = []
rhss = []
for i in range(N):
    print('{}. element: integration domain [{}, {}]'.format(i+1, a, b))
    lhs[0,0] = -(quad(nn11, a, b, args=(a, b))[0] + quad(nn_x11, a, b, args=(a, b))[0])
    lhs[0,1] = -(quad(nn12, a, b, args=(a, b))[0] + quad(nn_x12, a, b, args=(a, b))[0])
    lhs[1,0] = -(quad(nn21, a, b, args=(a, b))[0] + quad(nn_x21, a, b, args=(a, b))[0])
    lhs[1,1] = -(quad(nn22, a, b, args=(a, b))[0] + quad(nn_x22, a, b, args=(a, b))[0])
    lhss.append(lhs)

    rhs[0, 0] = -(quad(n1, a, b, args=(a, b))[0])
    rhs[1, 0] = -(quad(n2, a, b, args=(a, b))[0])
    rhss.append(rhs)

    a = a + dx
    b = b + dx
```

    1. element: integration domain [0, 0.0015]
    2. element: integration domain [0.0015, 0.003]
    3. element: integration domain [0.003, 0.0045000000000000005]
    4. element: integration domain [0.0045000000000000005, 0.006]
    5. element: integration domain [0.006, 0.0075]
    6. element: integration domain [0.0075, 0.009]
    7. element: integration domain [0.009, 0.010499999999999999]
    8. element: integration domain [0.010499999999999999, 0.011999999999999999]
    9. element: integration domain [0.011999999999999999, 0.013499999999999998]
    10. element: integration domain [0.013499999999999998, 0.014999999999999998]
    11. element: integration domain [0.014999999999999998, 0.016499999999999997]
    12. element: integration domain [0.016499999999999997, 0.018]
    13. element: integration domain [0.018, 0.0195]
    14. element: integration domain [0.0195, 0.021]
    15. element: integration domain [0.021, 0.022500000000000003]
    16. element: integration domain [0.022500000000000003, 0.024000000000000004]
    17. element: integration domain [0.024000000000000004, 0.025500000000000005]
    18. element: integration domain [0.025500000000000005, 0.027000000000000007]
    19. element: integration domain [0.027000000000000007, 0.028500000000000008]
    20. element: integration domain [0.028500000000000008, 0.03000000000000001]

```python
# assemble in global matrices
lhs_glob = assemble_l(lhss, lhs_glob)
rhs_glob = assemble_r(rhss, rhs_glob)
```

```python
# boundary conds
# DBC
T[-1, 0] = T_c
lhs_glob = lhs_glob[:-1, :-1]
rhs_glob = rhs_glob[:-1, ]

# RBC
lhs_glob[0, 0] -= h_0
rhs_glob[0, 0] -= h_0 * T_f
rhs_glob[-1, 0] -= T_c * lhs_glob[-1, -2]
```

```python
# T = B \ A
T[:-1, ] = np.linalg.inv(lhs_glob) @ rhs_glob
```

```python
x_anal = np.linspace(0, L, num=T_anal.shape[0])
x = np.linspace(0, L, num=T.shape[0])

fig = plt.figure(figsize=figsize(1))
ax = fig.add_subplot(111)
ax.axhline(y=T_c, color='black', linestyle=':', label='body/arterial temperature')
ax.axvline(x=x[np.where(np.max(T)==T)[0]], color='black', linestyle='-.', label='peak temperature depth')
ax.plot(x_anal, T_anal, linewidth=2, color='red', label='analytical')
ax.plot(x, T, linestyle='None', color='red', marker='o', markersize=7, label='FEM ({} elements)'.format(N))
ax.set_xlabel('$x$ [m]')
ax.set_ylabel('$T$ [°C]')
plt.grid()
plt.legend(loc='best')
# plt.title('{} finite elements'.format(N))

fig.savefig('{}-{}felems.pdf'.format('figs/FEM', N), bbox_inches='tight')
plt.show()
```

## Physics Informed Neural Network

Ref 1: [Data-driven Solutions of Nonlinear Partial Differential Equations, M. Raissi et al.](https://arxiv.org/abs/1711.10561)

Ref 2: [DeepXDE: A deep learning library for solving differential equations, Lu Lu et al.](https://arxiv.org/abs/1907.04502)

Data-driven solution of time-independent PDE using Physics-informed Neural Network approach. Instead of dealing with it numerically, the solution is transformed into optimization problem:

$$
k \cdot \frac{d^2 T}{d x^2} - w_b \cdot T(x) + w_b \cdot T_a + Q_m  = 0
$$

$$ \Downarrow $$

$$
J(\theta) = [k \cdot \frac{d^2 \hat{T}}{d x^2} - w_b \cdot \hat T(x) + w_b \cdot T_a + Q_m]^2 + [\hat T(0) - T_0]^2 + [\hat T(L) - T_L]^2
$$

$$
\mbox{Optimization: } \fbox{ $\min_{\theta} J(\theta)$}
$$

where:
* $J(\theta)$ is the cost function;
* $\hat{T}(x)$ is the neural network's output for every input x and
* $T_0$ and $T_L$ are boundary conditions.

Neural network configuration:
* activation: tanh
* initializer: Glorot uniform
* architecture: input (1), 3 hidden (50), output (1)
* optimization: Adam
* metrics: l2 relative error

```python
def pde(x, y):
    dy_x = tf.gradients(y, x)[0]
    dy_xx = tf.gradients(dy_x, x)[0]
    return k*dy_xx - w_b*y + (w_b*T_a + Q_m)

def boundary_l(x, on_boundary):
    return on_boundary and np.isclose(x[0], 0)

def boundary_r(x, on_boundary):
    return on_boundary and np.isclose(x[0], L)
```

```python
geom = dde.geometry.Interval(0, L)
#bc_l = dde.RobinBC(geom, lambda x, y: (h_0/k)*y - (h_0*T_f/k), boundary_l)
bc_l = dde.DirichletBC(geom, lambda x: analyticSol(x), boundary_l)
bc_r = dde.DirichletBC(geom, lambda x: analyticSol(x), boundary_r)
data = dde.data.PDE(geom, 1, pde, [bc_l, bc_r], 20, 2, func=analyticSol, num_test=100)

layer_size = [1] + [50] * 3 + [1]
activation = "tanh"
initializer = "Glorot uniform"
net = dde.maps.FNN(layer_size, activation, initializer)
net.outputs_modify(lambda x, y: y*10)

t = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
new_model_path = 'model/{}'.format(t)
if not os.path.exists(new_model_path):
    os.makedirs(new_model_path)
model = dde.Model(data, net)
model.compile("adam", lr=0.001, metrics=["l2 relative error"], loss_weights=[1e-7, 1e-2, 1])
losshistory, train_state = model.train(epochs=10000, uncertainty=True)

dde.postprocessing.save_loss_history(losshistory, 'model/{}/loss.dat'.format(t))
dde.postprocessing.save_best_state(train_state, 'model/{}/train.dat'.format(t), 'model/{}/test.dat'.format(t))
```

    Building feed-forward neural network...
    'build' took 0.053797 s

    Compiling model...
    'compile' took 0.561302 s

    Initializing variables...
    Training model...

    Step      Train loss                        Test loss                         Test metric
    0         [1.24e+03, 1.87e+01, 1.37e+03]    [1.24e+03, 0.00e+00, 0.00e+00]    [1.00e+00]
    1000      [1.86e+01, 2.09e+00, 1.64e+00]    [1.87e+01, 0.00e+00, 0.00e+00]    [1.57e-01]
    2000      [1.32e-01, 4.69e-01, 3.29e-05]    [1.18e-01, 0.00e+00, 0.00e+00]    [7.57e-02]
    3000      [8.10e-02, 4.55e-01, 9.36e-06]    [7.30e-02, 0.00e+00, 0.00e+00]    [7.44e-02]
    4000      [6.80e-02, 3.87e-01, 9.21e-05]    [6.05e-02, 0.00e+00, 0.00e+00]    [6.87e-02]
    5000      [2.29e-02, 1.02e-02, 5.54e-06]    [1.86e-02, 0.00e+00, 0.00e+00]    [1.12e-02]
    6000      [1.07e-02, 1.19e-03, 3.02e-07]    [7.86e-03, 0.00e+00, 0.00e+00]    [3.83e-03]
    7000      [5.46e-03, 4.99e-04, 1.87e-03]    [3.89e-03, 0.00e+00, 0.00e+00]    [2.34e-03]
    8000      [2.41e-03, 1.85e-04, 2.96e-05]    [1.70e-03, 0.00e+00, 0.00e+00]    [1.53e-03]
    9000      [1.15e-03, 6.02e-05, 2.69e-08]    [7.96e-04, 0.00e+00, 0.00e+00]    [8.57e-04]
    10000     [8.28e-04, 2.37e-05, 4.09e-08]    [5.92e-04, 0.00e+00, 0.00e+00]    [5.34e-04]

    Best model at step 10000:
      train loss: 8.52e-04
      test loss: 5.92e-04
      test metric: [5.34e-04]
      Uncertainty:
        l2: 0.0028827
        l_infinity: 0.000614162
        max uncertainty location: [0.00393939]

    'train' took 35.853998 s

    Saving loss history to model/20200326-170340/loss.dat ...
    Saving training data to model/20200326-170340/train.dat ...
    Saving test data to model/20200326-170340/test.dat ...

```python
X_train, y_train, X_test, y_test, best_y, best_ystd = train_state.packed_data()
y_dim = y_train.shape[1]
idx = np.argsort(X_test[:, 0])
X = X_test[idx, 0]

# regression plot
fig = plt.figure(figsize=figsize(1, nplots=1))
ax = fig.add_subplot(111)
for i in range(y_dim):
    ax.plot(X_train[:, 0], y_train[:, i], 'ok', label='True points')
    ax.plot(X, best_y[idx, i], "-r", label='Prediction')
ax.set_xlabel('$x$ [m]')
ax.set_ylabel('$T$ [°C]')
plt.grid()
plt.legend(loc='best')

fig.savefig('{}.pdf'.format('figs/regression-mean'), bbox_inches='tight')
plt.show()

fig = plt.figure(figsize=figsize(1, nplots=1))
ax = fig.add_subplot(111)
for i in range(y_dim):
    #ax.plot(X_train[:, 0], y_train[:, i], 'ok', label='True points')
    if best_ystd is not None:
        ax.plot(X, best_y[idx, i] + 2 * best_ystd[idx, i], '-b', label='95\% confidence interval')
        ax.plot(X, best_y[idx, i] - 2 * best_ystd[idx, i], '-b')
    ax.plot(X, best_y[idx, i], "--r", label='Prediction')
ax.set_xlabel('$x$ [m]')
ax.set_ylabel('$T$ [°C]')
plt.grid()
plt.legend(loc='best')

fig.savefig('{}.pdf'.format('figs/regression-uncertainty'), bbox_inches='tight')
plt.show()
```

```python
loss_train = np.sum(
    np.array(losshistory.loss_train) * losshistory.loss_weights, axis=1
)
loss_test = np.sum(
    np.array(losshistory.loss_test) * losshistory.loss_weights, axis=1
)

fig = plt.figure(figsize=figsize(1))
ax = fig.add_subplot(111)
ax.semilogy(losshistory.steps, loss_train, color='C0', label="Train loss")
ax.semilogy(losshistory.steps, loss_test, color='red', label="Test loss")
#for i in range(len(losshistory.metrics_test[0])):
    #ax.semilogy(losshistory.steps, np.array(losshistory.metrics_test)[:, i], color='black', label="Test metric",)
ax.set_xlabel('Number of epochs')
ax.set_ylabel(r'$J(\theta)$')
plt.grid()
plt.legend(loc='best')

fig.savefig('{}.pdf'.format('figs/loss'), bbox_inches='tight')
plt.show()
```

```python
# random variables
rvs = [k, w_b, T_a, Q_m, h_0]
rvs = np.array(rvs).reshape(1, -1)
num_samples = 5
num_iters = 5
rvs_data = monteCarloSim(rvs, coef_var=0.2, num_samples=num_samples, num_iters=num_iters)
rvs_data.shape
```

    (5, 5, 5)

```python
# train_state_ensemble = []
# for epoch in range(num_iters):
#     train_state_full = []
#     for sample in range(num_samples):
#         k = rvs_data[epoch, sample, 0]
#         w_b = rvs_data[epoch, sample, 1]
#         T_a = rvs_data[epoch, sample, 2]
#         Q_m = rvs_data[epoch, sample, 3]
#         h_0 = rvs_data[epoch, sample, 4]

#         geom = dde.geometry.Interval(0, L)
#         bc_l = dde.DirichletBC(geom, lambda x: analyticSol(x), boundary_l)
#         bc_r = dde.DirichletBC(geom, lambda x: analyticSol(x), boundary_r)
#         data = dde.data.PDE(geom, 1, pde, [bc_l, bc_r], 50, 2, func=analyticSol, num_test=100)

#         layer_size = [1] + [50] * 3 + [1]
#         activation = "tanh"
#         initializer = "Glorot uniform"
#         net = dde.maps.FNN(layer_size, activation, initializer)
#         net.outputs_modify(lambda x, y: y*10)

#         model = dde.Model(data, net)
#         model.compile("adam", lr=0.001, metrics=["l2 relative error"], loss_weights=[1e-7, 1e-2, 1])
#         losshistory, train_state = model.train(epochs=10000, uncertainty=True, display_every=10000)
#         train_state_full.append(train_state)
#     train_state_ensemble.append(train_state_full)

# X_train = np.zeros(shape=(num_iters, num_samples, 54))
# y_train = np.zeros(shape=(num_iters, num_samples, 54))
# X_test = np.zeros(shape=(num_iters, num_samples, 100))
# y_test = np.zeros(shape=(num_iters, num_samples, 100))
# best_y = np.zeros(shape=(num_iters, num_samples, 100))
# best_ystd = np.zeros(shape=(num_iters, num_samples, 100))

# for i, train_state_full in enumerate(train_state_ensemble):
#     for j, train_state in enumerate(train_state_full):
#         unpacked = train_state.packed_data()
#         X_train[i, j, :] = unpacked[0].ravel()
#         y_train[i, j, :] = unpacked[1].ravel()
#         X_test[i, j, :] = unpacked[2].ravel()
#         y_test[i, j, :] = unpacked[3].ravel()
#         best_y[i, j, :] = unpacked[4].ravel()
#         best_ystd[i, j, :] = unpacked[5].ravel()

# np.save('data/X_train.npy', X_train)
# np.save('data/y_train.npy', y_train)
# np.save('data/X_test.npy', X_test)
# np.save('data/y_test.npy', y_test)
# np.save('data/best_y.npy', best_y)
# np.save('data/best_ystd.npy', best_ystd)
```

```python
X_train = np.load('data/X_train.npy')
y_train = np.load('data/y_train.npy')
X_test = np.load('data/X_test.npy')
y_test = np.load('data/y_test.npy')
best_y = np.load('data/best_y.npy')
best_ystd = np.load('data/best_ystd.npy')

best_y_mc_mean = np.mean(best_y, axis=1)
best_y_mean = np.mean(best_y_mc_mean, axis=0)
best_y_std = np.std(best_y_mean, axis=0)
best_y_lower_bound = best_y_mc_mean[np.argmin(best_y_mc_mean[:, 0], axis=0)]
best_y_upper_bound = best_y_mc_mean[np.argmax(best_y_mc_mean[:, 0], axis=0)]

fig = plt.figure(figsize=figsize(1))
ax = fig.add_subplot(111)
ax.plot(X, best_y_mean, 'r-', label='MC Neural Network mean')
ax.plot(X, best_y_lower_bound, 'k--', label='MC lower bound')
ax.plot(X, best_y_upper_bound, 'k--', label='MC upper bound')
ax.fill_between(X, best_y_lower_bound, best_y_upper_bound, alpha=0.2)
ax.set_xlabel('$x$ [m]')
ax.set_ylabel('$<T>$ [°C]')
plt.legend(loc='best')
plt.grid()
fig.savefig('{}.pdf'.format('figs/uncertainty-mcnn-{}epochs{}samples'.format(num_iters, num_samples)), bbox_inches='tight')
plt.show()
```