```python
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
```

# Unsupervised learning: Hierarchical and density-based clustering algorithms

In a previous notebook, "08 Unsupervised Learning - Clustering.ipynb", we introduced one of the essential and widely used clustering algorithms, K-means. One of the advantages of K-means is that it is extremely easy to implement, and it is also computationally very efficient compared to other clustering algorithms. However, we've seen that one of the weaknesses of K-Means is that it only works well if the data can be grouped into a globular or spherical shape. Also, we have to assign the number of clusters, *k*, *a priori* -- this can be a problem if we have no prior knowledge about how many clusters we expect to find.

In this notebook, we will take a look at 2 alternative approaches to clustering, hierarchical clustering and density-based clustering.

# Hierarchical Clustering

One nice feature of hierachical clustering is that we can visualize the results as a dendrogram, a hierachical tree. Using the visualization, we can then decide how "deep" we want to cluster the dataset by setting a "depth" threshold. Or in other words, we don't need to make a decision about the number of clusters upfront.

**Agglomerative and divisive hierarchical clustering**

Furthermore, we can distinguish between 2 main approaches to hierarchical clustering: Divisive clustering and agglomerative clustering. In agglomerative clustering, we start with a single sample from our dataset and iteratively merge it with other samples to form clusters -- we can see it as a bottom-up approach for building the clustering dendrogram.
In divisive clustering, however, we start with the whole dataset as one cluster, and we iteratively split it into smaller subclusters -- a top-down approach.

In this notebook, we will use **agglomerative** clustering.

**Single and complete linkage**

Now, the next question is how we measure the similarity between samples. One approach is the familiar Euclidean distance metric that we already used via the K-Means algorithm. As a refresher, the distance between 2 m-dimensional vectors $\mathbf{p}$ and $\mathbf{q}$ can be computed as:

\begin{align} \mathrm{d}(\mathbf{q},\mathbf{p}) & = \sqrt{(q_1-p_1)^2 + (q_2-p_2)^2 + \cdots + (q_m-p_m)^2} \\[8pt]
& = \sqrt{\sum_{j=1}^m (q_j-p_j)^2}.\end{align}

However, that's the distance between 2 samples. Now, how do we compute the similarity between subclusters of samples in order to decide which clusters to merge when constructing the dendrogram? I.e., our goal is to iteratively merge the most similar pairs of clusters until only one big cluster remains. There are many different approaches to this, for example single and complete linkage.

In single linkage, we take the pair of the most similar samples (based on the Euclidean distance, for example) in each cluster, and merge the two clusters which have the most similar 2 members into one new, bigger cluster.

In complete linkage, we compare the pairs of the two most dissimilar members of each cluster with each other, and we merge the 2 clusters where the distance between its 2 most dissimilar members is smallest.

To see the agglomerative, hierarchical clustering approach in action, let us load the familiar Iris dataset -- pretending we don't know the true class labels and want to find out how many different follow species it consists of:

```python
from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data[:, [2, 3]]
y = iris.target
n_samples, n_features = X.shape

plt.scatter(X[:, 0], X[:, 1], c=y);
```

First, we start with some exploratory clustering, visualizing the clustering dendrogram using SciPy's `linkage` and `dendrogram` functions:

```python
from scipy.cluster.hierarchy import linkage
from scipy.cluster.hierarchy import dendrogram

clusters = linkage(X,
                   metric='euclidean',
                   method='complete')

dendr = dendrogram(clusters)

plt.ylabel('Euclidean Distance');
```

Next, let's use the `AgglomerativeClustering` estimator from scikit-learn and divide the dataset into 3 clusters. Can you guess which 3 clusters from the dendrogram it will reproduce?

```python
from sklearn.cluster import AgglomerativeClustering

ac = AgglomerativeClustering(n_clusters=3,
                             affinity='euclidean',
                             linkage='complete')

prediction = ac.fit_predict(X)
print('Cluster labels: %s\n' % prediction)
```

    Cluster labels: [2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
     2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 1
     1 1 1 0 1 1 1 1 1 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 1 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0]

```python
plt.scatter(X[:, 0], X[:, 1], c=prediction);
```

# Density-based Clustering - DBSCAN

Another useful approach to clustering is *Density-based Spatial Clustering of Applications with Noise* (DBSCAN). In essence, we can think of DBSCAN as an algorithm that divides the dataset into subgroup based on dense regions of point.

In DBSCAN, we distinguish between 3 different "points":

- Core points: A core point is a point that has at least a minimum number of other points (MinPts) in its radius epsilon.
- Border points: A border point is a point that is not a core point, since it doesn't have enough MinPts in its neighborhood, but lies within the radius epsilon of a core point.
- Noise points: All other points that are neither core points nor border points.

A nice feature about DBSCAN is that we don't have to specify a number of clusters upfront. However, it requires the setting of additional hyperparameters such as the value for MinPts and the radius epsilon.

```python
from sklearn.datasets import make_moons
X, y = make_moons(n_samples=400,
                  noise=0.1,
                  random_state=1)
plt.scatter(X[:,0], X[:,1])
plt.show()
```

```python
from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.2,
            min_samples=10,
            metric='euclidean')
prediction = db.fit_predict(X)

print("Predicted labels:\n", prediction)

plt.scatter(X[:, 0], X[:, 1], c=prediction);
```

# Exercise

    EXERCISE:

      Using the following toy dataset, two concentric circles, experiment with the three different clustering algorithms that we used so far: `KMeans`, `AgglomerativeClustering`, and `DBSCAN`.

Which clustering algorithms reproduces or discovers the hidden structure (pretending we don't know `y`) best?

Can you explain why this particular algorithm is a good choice while the other 2 "fail"?

```python
from sklearn.datasets import make_circles

X, y = make_circles(n_samples=1500,
                    factor=.4,
                    noise=.05)

plt.scatter(X[:, 0], X[:, 1], c=y);
```

```python
from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN

for n in range(1,5):
    kmeans = KMeans(n, random_state=1234)
    y_pred = kmeans.fit_predict(X)
    plt.scatter(X[:,0], X[:,1], c=y_pred)
    plt.show()
```

```python
for n in range(1,5):
    agglo = AgglomerativeClustering(n)
    y_pred = agglo.fit_predict(X)
    plt.scatter(X[:,0], X[:,1], c=y_pred)
    plt.show()
```

```python
dbscan = DBSCAN(eps=0.2, min_samples=10)
y_pred = dbscan.fit_predict(X)
plt.scatter(X[:,0], X[:,1], c=y_pred)
```

```python
# %load solutions/20_clustering_comparison.py
from sklearn.datasets import make_circles
from sklearn.cluster import KMeans, AgglomerativeClustering, DBSCAN

X, y = make_circles(n_samples=1500,
                    factor=.4,
                    noise=.05)

km = KMeans(n_clusters=2)
plt.figure()
plt.scatter(X[:, 0], X[:, 1], c=km.fit_predict(X))

ac = AgglomerativeClustering(n_clusters=2, affinity='euclidean', linkage='complete')
plt.figure()
plt.scatter(X[:, 0], X[:, 1], c=ac.fit_predict(X))

db = DBSCAN(eps=0.2)
plt.figure()
plt.scatter(X[:, 0], X[:, 1], c=db.fit_predict(X));

```

```python

```


This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also http://splines.readthedocs.io/.

# Finite Difference Splines

Idea: average velocity between the two adjoining line segments:

\begin{equation}
\boldsymbol{\dot{x}}_i = \frac{1}{2} \left(
\frac{\boldsymbol{x}_i - \boldsymbol{x}_{i-1}}{t_i - t_{i-1}} +
\frac{\boldsymbol{x}_{i + 1} - \boldsymbol{x}_i}{t_{i + 1} - t_i}
\right)
\end{equation}

According to [Wikipedia](https://en.wikipedia.org/wiki/Cubic_Hermite_spline#Finite_difference), this is called "finite difference" or "three-point difference".

In the uniform case, this can be simplified to

\begin{equation}
\boldsymbol{\dot{x}}_\text{0,uniform} = \frac{\boldsymbol{x}_1 - \boldsymbol{x}_{-1}}{2}
\end{equation}

... which is equivalent to uniform [Catmull--Rom splines](catmull-rom.ipynb).

```python
%matplotlib inline
import sympy as sp
sp.init_printing()
```

```python
from utility import NamedExpression, NamedMatrix
```

```python
x_1, x0, x1, x2 = sp.symbols('xbm_-1 xbm:3')
```

```python
t, t_1, t0, t1, t2 = sp.symbols('t t_-1 t:3')
```

```python
control_values_FD = sp.Matrix([x_1, x0, x1, x2])
control_values_FD
```

```python
xd0 = NamedExpression('xdotbm0', ((x0 - x_1) / (t0 - t_1) + (x1 - x0) / (t1 - t0)) / 2)
xd0
```

```python
xd1 = NamedExpression('xdotbm1', ((x1 - x0) / (t1 - t0) + (x2 - x1) / (t2 - t1)) / 2)
xd1
```

```python
control_values_H = sp.Matrix(sp.symbols('xbm:2 xdotbm:2'))
control_values_H
```

```python
M_H = NamedMatrix(
    r'{M_\text{H}}',
    sp.S('Matrix([[2, -2, Delta0, Delta0], [-3, 3, -2*Delta0, -Delta0], [0, 0, Delta0, 0], [1, 0, 0, 0]])'))
M_H
```

```python
M_FDtoH = NamedMatrix(r'{M_{\text{FD$\to$H}}}', 4, 4)
```

```python
M_FD = NamedMatrix(r'{M_\text{FD}}', M_H.name * M_FDtoH.name)
M_FD
```

```python
sp.Eq(control_values_H, M_FDtoH.name * control_values_FD)
```

```python
M_FDtoH.expr = sp.Matrix([[expr.expand().coeff(cv) for cv in control_values_FD]
                          for expr in control_values_H.subs([xd0.args, xd1.args])])
M_FDtoH
```

```python
deltas = [
    (t_1, -sp.Symbol('Delta_-1')),
    (t0, 0),
    (t1, sp.Symbol('Delta0')),
    (t2, sp.Symbol('Delta0') + sp.Symbol('Delta1'))
]
```

```python
M_FDtoH.subs(deltas)
```

```python
M_HtoFD = NamedMatrix(r'{M_{\text{H$\to$FD}}}', M_FDtoH.I.expr)
M_HtoFD.subs(deltas).expand()
```

```python
M_FD = M_FD.subs([M_H, M_FDtoH]).doit()
M_FD.subs(deltas).expand()
```

```python
M_FD.subs(deltas).I.expand()
```

```python
uniform = [
    (sp.Symbol('Delta_-1'), 1),
    (sp.Symbol('Delta0') , 1),
    (sp.Symbol('Delta1') , 1),
    #(M_FD.name, sp.Symbol('{M_\text{FD,uniform}}')),
]
```

```python
M_FD.subs(deltas).subs(uniform).pull_out(sp.S.Half).expr
```

```python

```


```python
import numpy as np
import pandas as pd
from scipy.sparse import coo_matrix, eye
import networkx as nx
import matplotlib.pyplot as plt
import grblas
from grblas import Matrix, Vector, Scalar
from grblas import descriptor
from grblas import unary, binary, monoid, semiring, op
from grblas import io as gio
```

## Create and visualize a Matrix

```python
# 23   // The input matrix A must be symmetric.  Self-edges (diagonal entries) are
# 24   // OK, and are ignored.  The values and type of A are ignored; just its
# 25   // pattern is accessed.
row_col = np.array(
    [
        [0, 0, 0, 1, 2, 2, 3, 6, 6, 9, 9],
        [1, 2, 3, 2, 4, 5, 4, 7, 8, 10, 11],
    ]
)
rows, cols = row_col
data = np.full_like(rows, fill_value=1)
```

```python
A = coo_matrix((data, (rows, cols)), shape=(12, 12)).tolil()
A[cols, rows] = A[rows, cols]  # symmetrize matrix
A = A.tocoo()
```

```python
# Draw A using spring layout which may even reveal the connected components
G = nx.convert_matrix.from_scipy_sparse_matrix(A)
layout = nx.drawing.layout.spring_layout(G, k=0.6, scale=1, threshold=1e-10)
nx.draw_networkx(G, with_labels=True, node_size=500, font_color="w", pos=layout)
```

```python
A = gio.from_scipy_sparse_matrix(A, name="A")
# Size of the sparse matrix is 12x12 with 22 non-zero elements of type INT64
```

```python
A
# This is an adjacency matrix
# Reading along a row shows the out-nodes of a vertex
# Reading along a column shows the in-nodes of a vertex
```

<tt>A</tt>

grblas.Matrix

nvals

nrows

ncols

dtype

format

    22
    12
    12
    INT32
    bitmapr (iso)

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      0

      1
      1
      1

      1
      1

      1

      2
      1
      1

      1
      1

      3
      1

      1

      4

      1
      1

      5

      1

      6

      1
      1

      7

      1

      8

      1

      9

      1
      1

      10

      1

      11

      1

```python
# grblas.io.draw could do with a few more tunable options to improve pretty display
gio.draw(A)
```

## Connected Components

https://github.com/GraphBLAS/LAGraph/blob/reorg/src/algorithm/LAGraph_ConnectedComponents.c

Sections of the C-code found at the above link are reproduced here in comments and translated into python

```python
# 10    // Code is based on the algorithm described in the following paper
# 11    // Zhang, Azad, Hu. FastSV: FastSV: A Distributed-Memory Connected Component
# 12    // Algorithm with Fast Convergence (SIAM PP20)
# 13
# 14    // A subsequent update to the algorithm is here (which might not be reflected
# 15    // in this code):
# 16    //
# 17    // Yongzhe Zhang, Ariful Azad, Aydin Buluc: Parallel algorithms for finding
# 18    // connected components using linear algebra. J. Parallel Distributed Comput.
# 19    // 144: 14-27 (2020).
```

```python
# 342      GrB_TRY (GrB_Matrix_nrows (&n, S)) ;
# 343      GrB_TRY (GrB_Matrix_nvals (&nnz, S)) ;
n = A.nrows
nnz = A.nvals
```

```python
# 370      // vectors
# 371      GrB_TRY (GrB_Vector_new (&f,      GrB_UINT32, n)) ;
# 372      GrB_TRY (GrB_Vector_new (&gp_new, GrB_UINT32, n)) ;
# 373      GrB_TRY (GrB_Vector_new (&mod,    GrB_BOOL,   n)) ;
dtype = np.uint32
f = Vector.new(dtype=dtype, size=n, name="parents")  # parent of each vertex
gp_new = Vector.new(dtype=dtype, size=n, name="grandparents")  # grandparent of each vertex
mod = Vector.new(dtype=bool, size=n, name="modified?")  # boolean flag for each vertex
f
```

<tt>parents</tt>

grblas.Vector

nvals

size

dtype

format

    0
    12
    UINT32
    sparse

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

```python
mod
```

<tt>modified?</tt>

grblas.Vector

nvals

size

dtype

format

    0
    12
    BOOL
    sparse

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

```python
# 387      GrB_TRY (GrB_Vector_build (f, I, V32, n, GrB_PLUS_UINT32)) ;
# 388      GrB_TRY (GrB_Vector_dup (&gp,   f)) ;
# 389      GrB_TRY (GrB_Vector_dup (&mngp, f)) ;
I = np.arange(n)
V32 = I.astype(dtype)
f.build(I, V32)  # The parent of each vertex is initialized to be the vertex itself
gp = f.dup()  # grandparent of each vertex initialized to parent
mngp = f.dup(name="Minimum grandparent")  # minimum grandparent of each vertex belonging to a star
```

```python
f
# The parent of each vertex is initialized to the vertex itself
```

<tt>parents</tt>

grblas.Vector

nvals

size

dtype

format

    12
    12
    UINT32
    full

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

```python
change = Scalar.new(dtype=bool, name="changed?")  # flag to terminate FastSV algorithm
```

This uses the ***min_second*** semiring with the *GrB_mxv()* function where *min* returns the minimum of its two inputs and *second* returns its second input.

```python
# 703         // hooking & shortcutting
# 704         GrB_TRY (GrB_mxv (mngp, NULL, GrB_MIN_UINT32,
# 705                           GrB_MIN_SECOND_SEMIRING_UINT32, T, gp, NULL)) ;
mngp(binary.min) << op.min_second(A @ gp)
mngp
```

<tt>Minimum grandparent</tt>

grblas.Vector

nvals

size

dtype

format

    12
    12
    UINT32
    full

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      0
      0
      0
      0
      2
      2
      6
      6
      6
      9
      9
      9

It is not yet clear to me if the function ***Reduce_assign32*** (described in the C-code) instead of ***GrB_assign*** is really required for the algorithm to work, as it is not referred to in any of the authors' papers.  Nevertheless, I'm choosing ***GrB_assign***, in accordance with authors' papers.  This seems to work anyway for the example graph used here.

```python
# 706        GrB_TRY (Reduce_assign32 (&f, &mngp, V32, n, nthreads, ht_key,
# 707                                  ht_val, &seed, msg)) ;
#
#
# 139   //------------------------------------------------------------------------------
# 140   // Reduce_assign32:  w (index) += s, using MIN as the "+=" accum operator
# 141   //------------------------------------------------------------------------------
# 142
# 143   // mask = NULL, accumulator = GrB_MIN_UINT32, descriptor = NULL.
# 144   // Duplicates are summed with the accumulator, which differs from how
# 145   // GrB_assign works.  GrB_assign states that the presence of duplicates results
# 146   // in undefined behavior.  GrB_assign in SuiteSparse:GraphBLAS follows the
# 147   // MATLAB rule, which discards all but the first of the duplicates.
# 148
# 149   // todo: add this to GraphBLAS as a variant of GrB_assign, either as
# 150   // GxB_assign_accum (or another name), or as a GxB_* descriptor setting.
#       et cetera
#
#
f(binary.min)[V32] << mngp
```

```python
# 708        GrB_TRY (GrB_eWiseAdd (f, NULL, GrB_MIN_UINT32, GrB_MIN_UINT32,
# 709                               mngp, gp, NULL)) ;
f(binary.min) << op.min(mngp | gp)
```

```python
# 710        // calculate grandparent
# 711        // fixme: NULL parameter is SS:GrB extension
# 712        GrB_TRY (GrB_Vector_extractTuples (NULL, V32, &n, f)) ; // fixme
_, V32 = f.to_values()
V32
```

    array([0, 0, 0, 0, 2, 2, 6, 6, 6, 9, 9, 9], dtype=uint32)

```python
I = V32.astype(I.dtype)
```

```python
# 719        GrB_TRY (GrB_extract (gp_new, NULL, NULL, f, I, n, NULL)) ;
gp_new << f[I]
```

```python
# 721        // check termination
# 722        GrB_TRY (GrB_eWiseMult (mod, NULL, NULL, GrB_NE_UINT32, gp_new, gp,
# 723            NULL)) ;
# 724        GrB_TRY (GrB_reduce (&change, NULL, GrB_LOR_MONOID_BOOL, mod, NULL)) ;
mod << gp_new.ewise_mult(gp, binary.ne)
change << mod.reduce(binary.lor)
```

```python
mod
```

<tt>modified?</tt>

grblas.Vector

nvals

size

dtype

format

    12
    12
    BOOL
    full

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      False
      True
      True
      True
      True
      True
      False
      True
      True
      False
      True
      True

```python
change
```

<tt>changed?</tt>

grblas.Scalar

value

dtype

    True
    BOOL

```python
change.value
```

    True

```python
# 726        // swap gp and gp_new
# 727        GrB_Vector t = gp ; gp = gp_new ; gp_new = t ;
gp, gp_new = gp_new, gp
```

The algorithm repeats until a new computation is the same as the previous result.

Here is the full python listing updated using changes from authors' paper: Yongzhe Zhang et al., J. Parallel Distributed Comput. 19 144: 14-27 (2020)):

```python
def fastSV(A):
    n = A.nrows
    I = np.arange(n)
    # The parent of each vertex is initialized to be the vertex itself:
    f = Vector.from_values(I, I, name="parents")
    gp = f.dup()  # grandparent of each vertex initialized to parent
    gp_dup = gp.dup()  # duplicate grandparents
    mngp = f.dup()  # minimum grandparent of each star-vertex

    # boolean flag for each vertex
    mod = Vector.new(dtype=bool, size=n, name="modified?")
    # flag to terminate FastSV algorithm
    change = Scalar.from_value(True, dtype=bool, name="changed?")

    while change:
        # Step 1: Hooking phase
        mngp << op.min_second(A @ gp)
        f(binary.min)[I] << mngp
        f << op.min(f | mngp)
        # Step 2: Shortcutting
        f << op.min(f | gp)
        # Step 3: Calculate grandparents
        _, I = f.to_values()
        gp << f[I]
        # Check termination
        mod << op.ne(gp_dup & gp)
        change << mod.reduce(binary.lor)
        gp_dup << gp

    return f
```

```python
connected_components = fastSV(A)
connected_components
```

<tt>parents</tt>

grblas.Vector

nvals

size

dtype

format

    12
    12
    INT32
    full

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      0
      0
      0
      0
      0
      0
      6
      6
      6
      9
      9
      9

*connected_components* gives the label of the component to which each vertex belongs.

Compare with the graph drawing to check result:

```python
nx.draw_networkx(G, with_labels=True, node_size=500, font_color="w", pos=layout)
```

Each component has been identified and labeled with the least vertex ID in that component.

### And that's FastSV in essentially 10 very readable lines of Python, thanks to GraphBLAS

Now let's test the algorithm a bit further by applying a random permutation to the vertex labels of the graph:

```python
p = np.random.permutation(A.shape[0])
```

```python
p
```

    array([10,  3,  2, 11,  4,  6,  7,  5,  8,  0,  1,  9])

The permutation $\mathsf{p}$ can be viewed not only as a rearrangement of the vertex labels, but also as a bijection
$$p: V \rightarrow V $$
from the set of vertices $V \subset \mathbb{Z}$ to itself.  So, for example,
$$p(0) = \mathsf{p[0]}\mbox{,}\;\;p(1) = \mathsf{p[1]}\mbox{, ...} $$

I do not know if GraphBLAS provides primitives for permuting vertex labels.  It might be worthwhile to check.  Here I'll try using grblas:

Let's build the above permutation's matrix $\mathbf{P}$ whose components are defined by:
$$P_{i\,j} \equiv \delta_{p(i)\, j},$$
where $\delta_{i\,j} = 1$ when $i=j$, otherwise $\delta_{i\,j} = 0$.
Note that,
$$ \sum_{j} j\,P_{i\,j} = \sum_{j} j\,\delta_{p(i)\, j} = p(i). $$
Also, it can be shown that
$$ \sum_{j} P^{\phantom{\mathrm{T}}}_{i\,j}P^{\mathrm{T}}_{j\,k} = \sum_{j} P_{i\,j}P_{k\,j} = \delta_{i\, k},$$
where $\mathrm{T}$ denotes the matrix transpose, and
$$P_{i\,j} = \delta_{p(i)\, j}  \iff P_{i\,p(j)} = \delta_{i\, j}.$$
We will now use the last equation above to build $\mathbf{P}$:

```python
rows, cols = np.arange(p.size), p
data = np.full_like(rows, fill_value=1)
P = Matrix.from_values(rows, cols, data, name="P")
```

Check from the definition, $P_{i\,j} \equiv \delta_{p(i)\, j}$, that the nonzero matrix elements are indeed correctly placed:

```python
P
```

<tt>P</tt>

grblas.Matrix

nvals

nrows

ncols

dtype

format

    12
    12
    12
    INT32
    csr (iso)

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      0

      1

      1

      1

      2

      1

      3

      1

      4

      1

      5

      1

      6

      1

      7

      1

      8

      1

      9
      1

      10

      1

      11

      1

Now let us transform the adjacency matrix $\mathbf{A}$, using the permutation matrix $\mathbf{P}$, into
$$\mathbf{A}' = \mathbf{P}^{\mathrm{T}} \cdot \mathbf{A} \cdot \mathbf{P},$$
which ensures that the graph edges are preserved after permutation, that is,
\begin{equation}
\boxed{A_{i\,j} = A'_{p(i)\,p(j)}}
\end{equation}
for all $i$, $j$.

```python
AA = A.dup(name="AA")
AA << P.T @ A @ P
```

Let's redraw the graph with the new labels and compare with the permutation array and graph-drawing above.

```python
A_sci = gio.to_scipy_sparse_matrix(AA, format="csr")
G_perm = nx.convert_matrix.from_scipy_sparse_matrix(A_sci)
layout_perm = {p[k]: layout[k] for k in layout}
nx.draw_networkx(G_perm, with_labels=True, node_size=500, font_color="w", pos=layout_perm)
```

Now let's re-apply the algorithm:

```python
connected_components_perm = fastSV(AA)
connected_components_perm
```

<tt>parents</tt>

grblas.Vector

nvals

size

dtype

format

    12
    12
    INT32
    full

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      0
      0
      2
      2
      2
      5
      2
      5
      5
      0
      2
      2

It looks like once again the algorithm worked as expected.  Let's confirm this programmatically by undoing the permutation on the result:

```python
_, components_perm = connected_components_perm.to_values()
```

```python
_, components = connected_components.to_values()
```

```python
def assert_components_equal(components, components_perm, p):
    """
    This function undoes the vertex-label permutation p in
    components_perm and compares the result to the original
    components obtained before the permutation was applied.
    """
    # Undo the permutation in components_perm:
    components_unperm_not_min = components_perm[p]
    # Note that the resulting component-labels are not
    # necessarily the minimum vertex-labels for each
    # component.
    # Extract minimum vertex-label for each component:
    non_min_vertices, min_vertices = np.unique(components_unperm_not_min, return_index=True)
    # create a mapping from the non-minimum to minimum
    # component labels:
    q = p.copy()
    q[non_min_vertices] = min_vertices
    # apply the map:
    components_perm_undone = q[components_unperm_not_min]
    assert np.all(components == components_perm_undone)
```

```python
assert_components_equal(components, components_perm, p)
```

To further test this assertion function, let us apply a second random permutation $p_2$ to the previous permutation:

```python
p2 = np.random.permutation(A.shape[0])
p2
```

    array([ 7,  1, 10,  4, 11,  0,  8,  5,  6,  3,  9,  2])

```python
rows, cols = np.arange(p2.size), p2
data = np.full_like(rows, fill_value=1)
P = Matrix.from_values(rows, cols, data)
```

```python
AAA = A.dup()
AAA << P.T @ AA @ P
```

```python
AAA_sci = gio.to_scipy_sparse_matrix(AAA, format="csr")
G_perm2 = nx.convert_matrix.from_scipy_sparse_matrix(AAA_sci)
layout_perm2 = {p2[k]: layout_perm[k] for k in layout_perm}
nx.draw_networkx(G_perm2, with_labels=True, node_size=500, font_color="w", pos=layout_perm2)
```

```python
connected_components_perm2 = fastSV(AAA)
connected_components_perm2
```

<tt>parents</tt>

grblas.Vector

nvals

size

dtype

format

    12
    12
    INT32
    full

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11

      0
      1
      2
      1
      2
      0
      0
      1
      2
      2
      2
      2

```python
_, components_perm2 = connected_components_perm2.to_values()
```

```python
assert_components_equal(components_perm, components_perm2, p2)
```

```python

```

notebooks/Connected Components -- FastSV.ipynb


# Introduction aux outils numériques

Nous allons faire un survol de quelques commandes que nous utiliserons dans les autres TP numériques.

## Objectifs de cette séance

Comme vous avez tous une formation de base en Matlab, nous allons nous appuyer sur cela pour réaliser quelques opérations simples que nous employerons sur des problèmes d'aérodynamique par la suite.

Réalisez les exercices avec Matlab et dans ce notebook pour vous familiariser avec les commandes python.

## Modules

Comme Matlab, Python est langage interprété. On s'appuie sur des fonctions pré-compilées pour rendre les opérations de calcul efficace.

Contrairement à Matlab, les modules ne sont pas chargés automatiquement. Il faut donc toujours importer ce dont on a besoin. Dans le cadre de ce cours, nous aurons besoin essentiellement de `numpy` et `matplotlib`.

[NumPy](https://numpy.org) est une bibliothèque d'objets et de fonctions adaptés au calcul vectoriel c'est-à-dire à des opérations sur des listes ou tableaux (N-dimensions) homogènes de nombres (_array_). Grace à cette bibliothèque, les opérations principales que l'on réalise avec Matlab sont réalisable sous Python. Souvent les fonctions ont le même nom.

[MatplotLib](https://matplotlib.org/) est une bibliothèque d'objets et de fonctions pour réaliser des représentations graphiques 2D comme dans Matlab (et certainement même plus !). On utilise le sous-module `pyplot` qui contient tout ce dont on a besoin.

Pour charger un module, on utilise la fonction `import` et on lui donne un petit nom pour raccourcir nos appels.

```python
import numpy as np
import matplotlib.pyplot as plt
```

Pour avoir des graphiques de bonne qualité, nous utilisons les paramètres suivants.

```python
plt.rcParams['figure.autolayout'] = True
plt.rcParams['figure.dpi'] = 100
plt.rcParams['font.size'] = 12
```

Voilà vous êtes prêt !

## Exercice 1 - Intersection de deux droites

On souhaite calculer le point d'intersection de deux droites définies par les équations suivantes :
\begin{equation}
y = 2 x -5 \hspace{2cm} \text{et} \hspace{2cm}
y = -x + 3
\end{equation}

1. Construisez les tableaux suivants :
  * `x` de 20 points uniformément répartis de -5 à 5. Utilisez la fonction `linspace` de `numpy`.
  * `y1` et `y2` les ordonnées des points aux abscisses `x` des points sur les droites 1 et 2.

```python

```

2. Tracez les deux droites ainsi définies :
  * La droite 1 en bleu, la droite 2 en rouge et en pointillés.
  * Affichez une grille, une légende et des labels sur les axes.

```python

```

3. Définissez le problème matriciel $M X = B$ à résoudre pour trouver l'intersection des droites et créez les tableaux numpy `M` et `B`. Utilisez la fonction `array`, vérifiez les formes avec `shape` et utilisez `reshape` pour corriger.

```python

```

4. Résolvez le problème en utilisant le sous-module d'algèbre linéaire [`linalg`](https://numpy.org/doc/stable/reference/routines.linalg.html#solving-equations-and-inverting-matrices) de `numpy`. Le produit matriciel (`dot`) est implémenté avec le symbole `@`

```python

```

5. Reprennez la figure de la question 2 et ajoutez le point d'intersection avec un symbole vert. Si l'intersection n'est pas correcte, vérifiez vos matrices et leurs formes. Enregistrez la figure sous un format image.

```python

```

## Exercice 2 - Les profils NACA 4 chiffres

Les profils NACA sont des formes d'aile testées par le [National Advisory Committee for Aeronautics](https://fr.wikipedia.org/wiki/National_Advisory_Committee_for_Aeronautics). C'est des profils dont la forme correspond à une équation mathématique dont les formes sont paramétrées et dont les coefficients aérodynamiques en fonction de l'incidence (angle d'attaque) ont été mesurés de manière systématique en soufflerie.

Ici on s'intéressera aux profils dits 4 chiffres symmétrique `NACA00XX`. Il sont définis par 2 paramètres :
  * La corde du profil $c$ (distance bord d'attaque/bord de fuite).
  * L'épaisseur maximale du profil est caractérisée par le paramètre $t$ qui est un pourcentage de la corde :
\begin{equation}
t = \frac{\texttt{XX}}{100}
\end{equation}

On définit la coordoonnée $x_c$ qui varie de 0 à 1 le long de la corde. La demi-épaisseur du profil $y_t$ (distance entre ses deux côtés) est alors paramétrée en fonction de cette coordonnée par l'équation suivante :
\begin{equation}
y_t = 5 t \left( 0.2969 \sqrt{x_c} - 0.1260 \,  x_c - 0.3516 \, x_c^2 + 0.2843 \, x_c^3 - 0.1036 \, x_c^4\right)
\end{equation}

Ainsi un profil dont la forme pourra être obtenue grace au deux courbes (redimmensionnées):

\begin{align}
x_{up} &= x_c \, c   & et \hspace{1cm}  y_{up} &= + y_t \, c \\
x_{down} &= x_c \, c  & et \hspace{1cm}  y_{down} &= - y_t \, c
\end{align}

Pour les questions suivantes, n'hésitez pas à contacter l'enseignant. Les questions sont choisies pour vous faire réfléchir et découvrir des options des modules.

1. Construisez et tracez le profil NACA007 de corde 12 cm avec 15 points de chaque côté. Déterminez l'épaisseur maximum obtenue la position sur la corde où ce point est atteint. Que pouvez vous dire ?

```python
# Parametres du profil

# Parametre de discretisation

```

```python
# Coordonnee de corde

# Epaisseur

```

```python
# Extrados

# Intrados

```

```python
# Epaisseur max

```

```python
# Trace dimensionnel et adimensionne par la corde

```

2. Créez une fonction qui permette d'obtenir les coordonnées dimensionnées à partir des paramètres du profil. Vous pourrez utiliser un distionnaire pour retourner les tableaux nécessaires. Tracez sur une même graphique les profils `NACA0005`, `NACA0012` et `NACA0017` de corde 10 cm, 5 cm et 7.5 cm respectivement.

```python
def NACA00_shape(chiffres34,corde,npts=15):

    return
```

```python
# Traces
```

3. Que peut-on faire pour rendre les tracés pour lisse ? Réalisez un zoom sur les 5 premiers millimètres de corde et commentez.

```python

```

```python
# Nouveau trace
```

4. La forme du bord d'attaque et du bord de fuite ont un rôle crucial sur la portance d'un profil aérodynamique. Il est parfois utile d'utiliser une distribution de points non uniforme pour calculer les efforts aérodynamiques.
Reprennez la fonction de la question 2 et ajoutez une option pour avoir une distribution cosinusoidale selon :
\begin{align}
x_c = \frac{1}{2}\left(1 - \cos{\zeta}\right) \hspace{1cm} \text{avec : } \hspace{0.2cm} \zeta \in [0, \pi]
\end{align}

```python
def NACA00_shape_improved(chiffres34,corde,npts=15,distri='cos'):

    return
```

5. Pour les 3 mêmes profils de la question 2, avec une discretisation cosinusoidale de 41 points, réalisez un zoom sur les 5 premiers millimètres de corde et comparez à la distribution uniforme avec le même nombre de points. Exportez la figure en image.

```python

```

```python
# Traces cote a cote
fig, ax = plt.subplots(1,2,constrained_layout=True,tight_layout=False,figsize=(8,4))

plt.show()
```

## Exercice 3 - Calcul symbolique

Si vous n'avez pas de TI, sachez que le module [SymPy](https://www.sympy.org) vous permet de faire du calcul symbolique. Nous allons voir quelques fonction de base qui pourront vous être pour la théorie 2D et 3D des profils.

À nouveau tout d'abord il faut l'importer. On active l'affichage sous forme formel.

```python
import sympy as sb
sb.init_printing()
```

On souhaite calculer la dérivée et la primitive de la fonction suivante :
\begin{align}
 \int{{\left(\cos{\theta}\right)}^3 \, d\theta}
\end{align}

1. Les variables formelles sont des objets de type `Symbol`. Définissez le symbole $\theta$.

```python
theta = sb.Symbol('theta')
```

```python
theta
```

2. Calculez la dérivée première et seconde de ${\left(\cos{\theta}\right)}^3$  à l'aide de la fonction `diff`

```python
f = sb.cos(theta)**3
```

```python
fp = sb.diff(f,theta)
fp
```

```python
fpp = sb.diff(fp,theta)
fpp
```

```python
fpp_2 = sb.diff(f,theta,2)
fpp_2
```

3. Calculez la primitive de ${\left(\cos{\theta}\right)}^3$ à l'aide de la fonction `integrate`

```python
I = sb.integrate(f,theta)
I
```


## Plotting Vector Fields

In this note, we will build a couple of different vector-field plots using `Quiver` library of `Matplotlib`.

**Reference**
 1. [Quiver plots](https://pythonforundergradengineers.com/quiver-plot-with-matplotlib-and-jupyter-notebooks.html)

```python
# Math and plotting libraries:
#import sympy
#from sympy.physics.vector import *
#from sympy import Curve, line_integrate, E, ln, diff
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

The `quiver()` method takes four positional arguments:

> `quiver(x_pos, y_pos, x_direct, y_direct)`

where `x_pos` & `y_pos` are the arrow starting positions and `x_direct` & `y_direct`
are the arrow directions.

**Example 1** Plotting the vector field $\vec{F}(x,y) = \vec{i}-\vec{j}$, where
$\vec{i} = <1, 0>$ and $\vec{j} = <0, 1>$ are standard unit vecotrs in 2D.

```python
# Establish coordinates for the arrow starting positions as a set of arrays
x,y = np.meshgrid(np.linspace(-5,5,10), np.linspace(-5,5,10)) # 10x10 = 100 positions

# Define the quiver arrow direction arrays in a vector field format.
u = 1
v = -1

# Plot the vector field
plt.figure(figsize=(7,7))

plt.quiver(x,y,u,v)

plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

**Example 2** Plotting the vector field $\vec{F}(x,y) = <\cos(x), \sin(y)>$.

```python
x,y = np.meshgrid(np.arange(0,2.2,0.2), np.arange(0,2.2,0.2)) # 10x10 = 100 positions

u = np.cos(x)*y
v = np.sin(y)*y

plt.figure(figsize=(7,7))

plt.quiver(x,y,u,v)

plt.xlabel('x')
plt.ylabel('y')
plt.title("$F=<\cos(x),\sin(y)>$")
plt.show()
```

## Plotting Gradient Vector Fields

Next, let's build a quiver plot of the gradient vector $\vec{\nabla} f$ of a scalar-valued function $f(x,y)$. We will use numpy's `np.gradient()` function to apply $\vec{\nabla} f$ based on every arrow's $x, y$.

**Example 3** Plotting the gradient vector field of $f(x,y) = xe^{-x^2-y^2}$.

```python
x,y = np.meshgrid(np.arange(-2,2.2,0.2), np.arange(-2,2.2,0.2))

z = x*np.exp(-x**2-y**2)
dx, dy = np.gradient(z)

plt.figure(figsize=(7,7))

plt.quiver(x,y,dx,dy)

#plt.xticks([])
#plt.yticks([])
plt.title("Gradient vector Field of $f(x,y) = xe^{-x^2-y^2}$")
plt.show()
```

Next, let's add some color to the quiver plots. The `quiver()` function has an optional fifth positional argument that specifies the quiver arrow color. The quiver arrow color argument needs to have the same dimensions as the position and direction arrays.

**Example 4** Let's add some colors to the gradient vector plot in Example 3.

```python
x,y = np.meshgrid(np.arange(-2,2.2,0.2), np.arange(-2,2.2,0.2))

z = x*np.exp(-x**2-y**2)
dx, dy = np.gradient(z)

n = -2
color_array = np.sqrt(((dx-n)/2)**2 + ((dy-n)/2)**2)

plt.figure(figsize=(7,7))

plt.quiver(x,y,dx,dy, color_array)

#plt.xticks([])
#plt.yticks([])
plt.title("Gradient vector Field of $f(x,y) = xe^{-x^2-y^2}$")
plt.show()
```

## 3D Vector Fields

```python
%reset -f # Reset all previously defined variables
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
import numpy as np

%matplotlib notebook # To generate interactive plot

fig = plt.figure(figsize=(8,8))
ax = fig.gca(projection='3d')

x, y, z = np.meshgrid(np.arange(-0.8, 1, 0.2),
                      np.arange(-0.8, 1, 0.2),
                      np.arange(-0.8, 1, 0.2))

u = np.sin(np.pi * x) * np.cos(np.pi * y) * np.cos(np.pi * z)
v = -np.cos(np.pi * x) * np.sin(np.pi * y) * np.cos(np.pi * z)
w = (np.sqrt(2.0 / 3.0) * np.cos(np.pi * x) * np.cos(np.pi * y) *
     np.sin(np.pi * z))

ax.quiver(x, y, z, u, v, w, length=0.1)

ax.set_xlabel('X axis')
ax.set_ylabel('Y axis')
ax.set_zlabel('Z axis')

#plt.show()
```

    <IPython.core.display.Javascript object>

    Text(0.5, 0, 'Z axis')

## Exercises

1. Plot the vector fields:

   (a) $\vec{F}(x,y) = \frac{x}{\sqrt{x^2+y^2}}\vec{i} + \frac{y}{\sqrt{x^2+y^2}}\vec{j}$

   (b) $\vec{F}(x,y) = -\frac{y}{\sqrt{x^2+y^2}}\vec{i} + \frac{x}{\sqrt{x^2+y^2}}\vec{j}$

   (c) $\vec{F}(x,y) = \sin(x)\cos(y)\vec{i} - \cos(x)\sin(y)\vec{j}$

```python

```


```python
%matplotlib inline
```

```python
import numpy as np
import numpy.polynomial.polynomial as p

import matplotlib.pyplot as plt
```

# Basic Algebra Exercise
## Functions, Polynomials, Complex Numbers. Applications of Abstract Algebra

### Problem 1. Polynomial Interpolation
We know that if we have a set of $n$ data points with coordinates $(x_1; y_1), (x_2; y_2), \dots, (x_n; y_n)$, we can try to figure out what function may have generated these points.

Please note that **our assumptions about the data** will lead us to choosing one function over another. This means that our results are as good as our data and assumptions. Therefore, it's extremely important that we write down our assumptions (which sometimes can be difficult as we sometimes don't realize we're making them). It will be better for our readers if they know what those assumptions and models are.

In this case, we'll state two assumptions:
1. The points in our dataset are generated by a polynomial function
2. The points are very precise, there is absolutely no error in them. This means that the function should pass **through every point**

This method is called *polynomial interpolation* (*"polynomial"* captures assumption 1 and *"interpolation"* captures assumption 2).

It can be proved (look at [Wikipedia](https://en.wikipedia.org/wiki/Polynomial_interpolation) for example) that if we have $n$ data points, there is only one polynomial of degree $n-1$ which passes through them. In "math speak": "the vector spaces of $n$ points and polynomials of degree $n-1$ are isomorphic (there exists a bijection mapping one to the other)".

There are a lot of ways to do interpolation. We can also write the function ourselves if we want but this requires quite a lot more knowledge than we already covered in this course. So we'll use a function which does this for us. `numpy.polyfit()` is one such function. It accepts three main parameters (there are others as well, but they are optional): a list of $x$ coordinates, a list of $y$ coordinates, and a polynomial degree.

Let's say we have these points:
```python
points = np.array([(0, 0), (1, 0.8), (2, 0.9), (3, 0.1), (4, -0.8), (5, -1.0)])
```

First, we need to "extract" the coordinates:
```python
x = points[:, 0]
y = points[:, 1]
```

Then, we need to calculate the interpolating polynomial. For the degree, we'll set $n-1$:
```python
coefficients = np.polyfit(x, y, len(points) - 1)
poly = np.poly1d(coefficients)
```

After that, we need to plot the function. To do this, we'll create a range of $x$ values and evaluate the polynomial at each value:
```python
plot_x = np.linspace(np.min(x), np.max(x), 1000)
plot_y = poly(plot_x)
```

Finally, we need to plot the result. We'll plot both the fitting polynomial curve (using `plt.plot()`) and the points (using `plt.scatter`). It's also nice to have different colors to make the line stand out from the points.
```python
plt.plot(plot_x, plot_y, c = "green")
plt.scatter(x, y)
plt.xlabel("x")
plt.ylabel("y")
plt.show()
```
Don't forget to label the axes!

Your task now is to **wrap the code in a function**. It should accept a list of points, the polynomial degree, min and max value of $x$ used for plotting. **Be extremely careful to ensure that the function uses its parameters!**

We'll use this function to try some other cases.

```python
points = np.array([(0, 0), (1, 0.8), (2, 0.9), (3, 0.1), (4, -0.8), (5, -1.0)])
```

```python
def plot_points(points):
    x=points[:,0]
    y=points[:,1]

    plt.scatter(x,y,c="g")
    plt.show()
```

```python
plot_points(points)
```

```python
def interpolate_polynomial(points, degree, min_x, max_x):
    x=points[:,0]
    y=points[:,1]

    poly=np.polyfit(x,y,degree)
    polynomial_function=np.poly1d(poly)

    x_polynomial=np.linspace(min_x,max_x,5000)
    y_polynomial=polynomial_function(x_polynomial)

    plt.scatter(x,y,c="g")
    plt.plot(x_polynomial,y_polynomial,c="r")
    plt.show()
```

```python

```

```python

interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

We see this is a very nice fit. This is expected, of course. Let's try to expand our view a little. Let's try to plot other values of $x$, further than the original ones. This is **extrapolation**.

```python
interpolate_polynomial(points, len(points) - 1, -5, 10)
```

Hmmm... it seems our polynomial goes a little wild outside the original range. This is to show how **extrapolation can be quite dangerous**.

Let's try a lower polynomial degree now. We used 4, how about 3, 2 and 1?
**Note:** We can add titles to every plot so that we know what exactly we're doing. Te title may be passed as an additional parameter to our function.

```python
interpolate_polynomial(points, 3, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 2, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

We see the fitting curves (or line in the last case) struggle more and more and they don't pass through every point. This breaks our assumptions but it can be very useful.

Okay, one more thing. How about increasing the degree? Let's try 5, 7 and 10. Python might complain a little, just ignore it, everything is fine... sort of :).

```python
interpolate_polynomial(points, 5, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 7, np.min(points[:, 0]), np.max(points[:, 0]))
interpolate_polynomial(points, 10, np.min(points[:, 0]), np.max(points[:, 0]))
```

Those graphs look pretty much the same. But that's the point exactly. I'm being quite sneaky here. Let's try to expand our view once again and see what our results really look like.

```python
interpolate_polynomial(points, 5, -10, 10)
interpolate_polynomial(points, 7, -10, 10)
interpolate_polynomial(points, 10, -10, 10)
```

Now we see there are very wild differences. Even though the first two plots look quite similar, look at the $y$ values - they're quite different.

So, these are the dangers of interpolation. Use a too high degree, and you get "the polynomial wiggle". These are all meant to represent **the same** data points but they look insanely different. Here's one more comparison.

```python
interpolate_polynomial(points, len(points) - 1, -2, 7)
interpolate_polynomial(points, len(points) + 1, -2, 7)
```

Now we can see what big difference even a small change in degree can make. This is why we have to choose our interpolating functions very carefully. Generally, a lower degree means a simpler function, which is to be preferred. See [Occam's razor](https://en.wikipedia.org/wiki/Occam%27s_razor).

And also, **we need to be very careful about our assumptions**.

```python
points = np.array([(-5, 0.03846), (-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882), (5, 0.03846)])
interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

This one definitely looks strange. Even stranger, if we remove the outermost points... ($x = \pm 5$), we get this

```python
points = np.array([(-4, 0.05882), (-3, 0.1), (-2, 0.2), (-1, 0.5), (0, 1), (1, 0.5), (2, 0.2), (3, 0.1), (4, 0.05882)])
interpolate_polynomial(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

This is because the generating function is not a polynomial. It's actually:
$$ y = \frac{1}{1 + x^2} $$

Plot the polynomial interpolation and the real generating function **on the same plot**. You may need to modify the original plotting function or just copy its contents.

```python
# Write your code here
def interpolate_polynomial2(points, degree, min_x, max_x):
    x=points[:,0]
    y=points[:,1]

    poly=np.polyfit(x,(1/(1+x**2)),degree)
    polynomial_function=np.poly1d(poly)

    x_polynomial=np.linspace(min_x,max_x,5000)
    y_polynomial=polynomial_function(x_polynomial)

    plt.scatter(x,y,c="g")
    plt.plot(x_polynomial,y_polynomial,c="r")
    plt.show()
```

```python
interpolate_polynomial2(points, len(points) - 1, np.min(points[:, 0]), np.max(points[:, 0]))
```

### Problem 2. Complex Numbers as Vectors
We saw that a complex number $z = a + bi$ is equivalent to (and therefore can be represented as) the ordered tuple $(a; b)$, which can be plotted in a 2D space. So, complex numbers and 2D points are equivalent. What is more, we can draw a vector from the origin of the coordinate plane to our point. This is called a point's **radius-vector**.

Let's try plotting complex numbers as radius vectors. Don't forget to label the real and imaginary axes. Also, move the axes to the origin. Hint: These are called "spines"; you'll need to move 2 of them to the origin and remove the other 2 completely. Hint 2: You already did this in the previous lab.

We can use `plt.quiver()` to plot the vector. It can behave a bit strangely, so we'll need to set the scale of the vectors to be the same as the scale on the graph axes:
```python
plt.quiver(0, 0, z.real, z.imag, angles = "xy", scale_units = "xy", scale = 1)
```

Other than that, the main parameters are: $x_{begin}$, $y_{begin}$, $x_{length}$, $y_{length}$ in that order.

Now, set the aspect ratio of the axes to be equal. Also, add grid lines. Set the axis numbers (called ticks) to be something like `range(-3, 4)` for now.
```python
plt.xticks(range(-3, 4))
plt.yticks(range(-3, 4))

```

If you wish to, you can be a bit more clever with the tick marks. Find the minimal and maximal $x$ and $y$ values and set the ticks according to them. It's a good practice not to jam the plot too much, so leave a little bit of space. That is, if the actual x-range is $[-2; 2]$, set the plotting to be $[-2.5; 2.5]$ for example. Otherwise, the vector heads (arrows) will be "jammed" into a corner or side of the plot.

```python
def plot_complex_number(z):
    """
    Plots the complex number z as a radius vector in the 2D space
    """
    # Write your code here
    pass

plot_complex_number(2 + 3j)
```

How about many numbers? We'll need to get a little bit more creative. First, we need to create a 2D array, each element of which will be a 4-element array: `[0, 0, z.real, z.imag]`. Next, `plt.quiver()` can accept a range of values. Look at [this StackOverflow post](https://stackoverflow.com/questions/12265234/how-to-plot-2d-math-vectors-with-matplotlib) for details and adapt your code.

```python
def plot_complex_numbers(numbers, colors):
    """
    Plots the given complex numbers as radius vectors in the 2D space
    """
    zeroes=[0 for number in numbers]
    real_parts=[number.real for number in numbers]
    imaginary_parts=[number.imag for number in numbers]

    plt.xlim(-4,4)
    plt.ylim(-4, 4)
    plt.quiver(zeroes, zeroes, real_parts, imaginary_parts, angles = "xy", scale_units = "xy", scale = 1, color=colors)
    plt.show()
plot_complex_numbers([2 + 3j, -2 - 1j, -3, 2j], ["green", "red", "blue", "orange"])
```

Now let's see what the operations look like. Let's add two numbers and plot the result.

```python
z1 = 2 + 3j
z2 = 1 - 1j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

We can see that adding the complex numbers is equivalent to adding vectors (remember the "parallelogram rule"). As special cases, let's try adding pure real and pure imaginary numbers:

```python
z1 = 2 + 3j
z2 = 2 + 0j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

```python
z1 = 2 + 3j
z2 = 0 + 2j
plot_complex_numbers([z1, z2, z1 + z2], ["red", "blue", "green"])
```

How about multiplication? First we know that multiplying by 1 gives us the same vector and mulpiplying by -1 gives us the reversed version of the same vector. How about multiplication by $\pm i$?

```python
z = 2 + 3j
plot_complex_numbers([z, z * 1], ["red", "blue"])
plot_complex_numbers([z, z * -1], ["red", "blue"])
plot_complex_numbers([z, z * 1j], ["red", "blue"])
plot_complex_numbers([z, z * -1j], ["red", "blue"])
```

So, multiplication by $i$ is equivalent to 90-degree rotation. We can actually see the following equivalence relationships between multiplying numbers and rotation about the origin:

| Real | Imaginary | Result rotation |
|------|-----------|-----------------|
| 1    | 0         | $0^\circ$       |
| 0    | 1         | $90^\circ$      |
| -1   | 0         | $180^\circ$     |
| 0    | -1        | $270^\circ$     |

Once again, we see the power of abstraction and algebra in practice. We know that complex numbers and 2D vectors are equivalent. Now we see something more: addition and multiplication are equivalent to translation (movement) and rotation!

Let's test the multiplication some more. We can see the resulting vector is the sum of the original vectors, but *scaled and rotated*:

```python
z1 = 2 + 3j
z2 = 1 - 2j
plot_complex_numbers([z1, z2, z1 * z2], ["red", "blue", "green"])
```

### Problem 3. Recursion and Fractals

> "To understand recursion, you first need to understand recursion."

There are three main parts to a recursive function:
1. Bottom - when the recursion should finish
2. Operation - some meaningful thing to do
3. Recursive call - calling the same function
4. Clean-up - returning all data to its previous state (this reverses the effect of the operation)

Let's do one of the most famous recursion examples. And I'm not talking about Fibonacci here. Let's draw a tree using recursive functions.

The figure we're going to draw is called a **fractal**. It's self-similar, which means that if you zoom in on a part of it, it will look the same. You can see fractals everywhere in nature, with broccoli being one of the prime examples. Have a look:

First, we need to specify the recursive part. In order to draw a tree, we need to draw a line of a given length (which will be the current branch), and then draw two more lines to the left and right. By "left" and "right", we should mean "rotation by a specified angle".

So, this is how to draw a branch: draw a line and prepare to draw two more branches to the left and right. This is going to be our recursive call.

To make things prettier, more natural-looking (and have a natural end to our recursion), let's draw each "sub-branch" a little shorter. If the branch becomes too short, it won't have "child branches". This will be the bottom of our recursion.

There's one more important part of recursion, and this is **"the clean-up"**. After we did something in the recursive calls, it's very important to return the state of everything as it was **before** we did anything. In this case, after we draw a branch, we go back to our starting position.

Let's first import the most import-ant (no pun intended...) Python drawing library: `turtle`! In order to make things easier, we'll import all methods directly.
```python
from turtle import *
```

You can look up the docs about turtle if you're more interested. The basic things we're going to use are going forward and backward by a specified number of pixels and turning left and right by a specified angle (in degrees).

Let's now define our recursive function:
```python
def draw_branch(branch_length, angle):
    if branch_length > 5:
        forward(branch_length)
        right(angle)
        draw_branch(branch_length - 15, angle)
        left(2 * angle)
        draw_branch(branch_length - 15, angle)
        right(angle)
        backward(branch_length)
```

And let's call it:
```python
draw_branch(100, 20)
```

We need to start the tree not at the middle, but toward the bottom of the screen, so we need to make a few more adjustments. We can wrap the setup in another function and call it. Let's start one trunk length below the center (the trunk length is the length of the longest line).

**Note:** It's important to call `done()` after the drawing is finished. If you miss it, the window with the turtle drawing will freeze and throw an exception.

```python
def draw_tree(trunk_length, angle):
    speed("fastest")
    left(90)
    up()
    backward(trunk_length)
    down()
    draw_branch(trunk_length, angle)
    done()
```

Note that the graphics will show in a separate window. Also note that sometimes you might get bugs. If you do, go to Kernel > Restart.

```python
# Write your code here
```

```python
draw_tree(100, 20)
```

Experiment with different lengths and angles. Especially interesting angles are $30^\circ$, $45^\circ$, $60^\circ$ and $90^\circ$.

```python
draw_tree(100, 30)
```

```python
draw_tree(100, 45)
```

```python
draw_tree(100, 90)
```

Now modify the original function a little. Draw the lines with different thickness. Provide the trunk thickness at the initial call. Similar to how branches go shorter, they should also go thinner.

```python
# Write your code here
```

#### * Optional problem
Try to draw another kind of fractal graphic using recursion and the `turtle` library. Two very popular examples are the "Koch snowflake" and the "Sierpinski triangle". You can also modify the original tree algorithm to create more natural-looking trees. You can, for example, play with angles, number of branches, lengths, and widths. The Internet has a lot of ideas about this :). Hint: Look up **"L-systems"**.

### Problem 4. Run-length Encoding
One application of algebra and basic math can be **compression**. This is a way to save data in less space than it originally takes. The most basic form of compression is called [run-length encoding](https://en.wikipedia.org/wiki/Run-length_encoding).

Write a function that encodes a given text. Write another one that decodes.

We can see that RLE is not very useful in the general case. But it can be extremely useful if we have very few symbols. An example of this can be DNA and protein sequences. DNA code, for example, has only 4 characters.

Test your encoding and decoding functions on a DNA sequence (you can look up some on the Internet). Measure how much your data is compressed relative to the original.

```python
def encode(text):
    """
    Returns the run-length encoded version of the text
    (numbers after symbols, length = 1 is skipped)
    """
    pass

def decode(text):
    """
    Decodes the text using run-length encoding
    """
    pass
```

```python
# Tests
# Test that the functions work on their own
assert encode("AABCCCDEEEE") == "A2BC3DE4"
assert decode("A2BC3DE4") == "AABCCCDEEEE"

# Test that the functions really invert each other
assert decode(encode("AABCCCDEEEE")) == "AABCCCDEEEE"
assert encode(decode("A2BC3DE4")) == "A2BC3DE4"
```

### * Problem 5. Function Invertibility and Cryptography
As we already saw, some functions are able to be inverted. That is, if we know the output, we can see what input generated it directly. This is true if the function is **one-to-one correspondence** (bijection).

However, not all functions are created the same. Some functions are easy to compute but their inverses are extremely difficult. A very important example is **number factorization**. It's relatively easy (computationally) to multiply numbers but factoring them is quite difficult. Let's run an experiment.

We'll need a function to generate random n-bit numbers. One such number can be found in the `Crypto` package
```python
from Crypto.Util import number
random_integer = number.getRandomNBitInteger(n_bits)
```

We could, of course, write our factorization by hand but we'll use `sympy`
```python
from sympy.ntheory import factorint
factorint(1032969399047817906432668079951) # {3: 2, 79: 1, 36779: 1, 7776252885493: 1, 5079811103: 1}
```

This function returns a `dict` where the keys are the factors, and the values - how many times they should be multiplied.

We'll also need a tool to accurately measure performance. Have a look at [this one](https://docs.python.org/3/library/time.html#time.time) for example.

Specity a sequence of bit lengths, in increasing order. For example, you might choose something like `[10, 20, 25, 30, 32, 33, 35, 38, 40]`. Depending on your computer's abilities you can go as high as you want. For each bit length, generate a number. See how much time it takes to factor it. Then see how much time it takes to multiply the factors. Be careful how you measure these. You shouldn't include the number generation (or any other external functions) in your timing.

In order to have better accuracy, don't do this once per bit length. Do it, for example, five times, and average the results.

Plot all multiplication and factorization times as a function of the number of bits. You should see that factorization is much, much slower. If you don't see this, just try larger numbers :D.

```python
# Write your code here
```

### * Problem 6. Diffie - Hellman Simulation
As we already saw, there are functions which are very easy to compute in the "forward" direction but really difficult (computationally) to invert (that is, determine the input from the output). There is a special case: the function may have a hidden "trap door". If you know where that door is, you can invert the function easily. This statement is at the core of modern cryptography.

Look up **Diffie - Hellman key exchange** (here's a [video](https://www.youtube.com/watch?v=cM4mNVUBtHk) on that but feel free to use anything else you might find useful).

Simulate the algorithm you just saw. Generate large enough numbers so the difference is noticeable (say, factoring takes 10-15 seconds). Simulate both participants in the key exchange. Simulate an eavesdropper.

First, make sure after both participants run the algotihm, they have *the same key* (they generate the same number).

Second, see how long it takes for them to exchange keys.

Third, see how long it takes the eavesdropper to arrive at the correct shared secret.

You should be able to see **the power of cryptography**. In this case, it's not that the function is irreversible. It can be reversed, but it takes a really long time (and with more bits, we're talking billions of years). However, if you know something else (this is called a **trap door**), the function becomes relatively easy to invert.

```python
# Write your code here
```

### ** Problem 7. The Galois Field in Cryptography
Research about the uses of the Galois field. What are its properties? How can it be used in cryptography? Write a simple cryptosystem based on the field.

You can use the following questions to facilitate your research:
* What is a field?
* What is GF(2)? Why is it an algebraic field?
* What is perfect secrecy? How does it relate to the participants in the conversation, and to the outside eavesdropper?
* What is symmetrical encryption?
* How to encrypt one-bit messages?
* How to extend the one-bit encryption system to many buts?
* Why is the system decryptable? How do the participants decrypt the encrypted messages?
* Why isn't the eavesdropper able to decrypt?
* What is a one-time pad?
    * How does the one-time pad achieve perfect secrecy?
* What happens if we try to use a one-time pad many times?
    * Provide an example where you break the "many-time pad" security
* What are some current enterprise-grade applications of encryption over GF(2)?
* Implement a cryptosystem based on GF(2). Show correctness on various test cases

### ** Problem 8. Huffman Compression Algorithm
Examine and implement the **Huffman algorithm** for compressing data. It's based on information theory and probiability theory. Document your findings and provide your implementation.

This algorithm is used for **lossless compression**: compressing data without loss of quality. You can use the following checklist:

* What is the difference betwenn lossless and lossy compression?
* When can we get away with lossy compression?
* What is entropy?
* How are Huffman trees constructed?
    * Provide a few examples
* How can we get back the uncompressed data from the Huffman tree?
* How and where are Huffman trees stored?
* Implement the algorithm. Add any other formulas / assumptions / etc. you might need.
* Test the algorithm. A good meaure would be percentage compression: $$\frac{\text{compressed}}{\text{uncompressed}} * 100\%$$
* How well does Huffman's algorithm perform compared to other compression algorithms (e.g. LZ77)?

01.Math_Concept_For_Devs/04. Basic-Algebra-Exercise/Basic Algebra Exercise.ipynb


# Introduction to Metropolis MCMC

Metropolis Markov chain Monte Carlo is a method for sampling from a distribution where direct sampling is difficult. Such difficulties usually come about in fields with high dimensions and with prior distributions in Bayesian work-flow.  The Metropolis algorithm is as follows.

1. Create a random initialization point $x_0$ and decide on candidate distribution function $g(x'|x)$. The Metropolis algorithm requires that $g$ must be symmetric, so $g(x'|x)=g(x|x')$ for all $x$ and $x'$.

2. For each iteration of $t$.
    1. Generate a candidate $x'$ from $g(x'|x_t)$
    2. Calculate the acceptance ratio $\alpha=\frac{P(x'|y)}{P(x|y}$. Given that Log Probabilities, $LP(x|y)$, are often used,
    \begin{equation}
    \alpha=e^{LP(x'|y)-LP(x|y)}
    \end{equation}
    3. If $\alpha\ge 1$ then $x_{t+1}=x'$ else
       Pull a random number number from a Uniform distribution between 0 and 1 called $u$.
       \begin{equation}
       x_{t+1} = \begin{cases} x' & u\ge \alpha\\
       x_t & otherwise
       \end{cases}
       \end{equation}

The Metropolis algorithm is a special case of the Metropolis-Hastings algorithm. The Metropolis-Hastings algorithm includes $g(x'|x)$ in the calculation of $\alpha$, which allows for asymmetric candidate distributions.

If $P(x'|y)$ is greater than $P(x|y)$, the candidate is always accepted. This encourages the algorithm to turn back when it needs to, but it might discourage it from venturing far enough to properly describe the space. This can be especially problematic in the case of a multimodal distribution, where it might be hard to venture from one mode to the next. Multiple chains with different starting points can solve this issue, but the more modes there are, the more chains will be needed. On top of that, the mirage of a multimodal distributions might be a product of highly correlated data, where one parameter takes precedent over another and vice versa, rather than one large distribution.

Even with these problems, the Metropolis algorithm is useful one for many purposes. Let's take a look at the code.

# Tensorflow Code

As usual, we start by loading the packages and setting the seed.

```python
import tensorflow as tf
import tensorflow.distributions as tfd
from tensorflow import contrib
import numpy as np
import matplotlib.pyplot as plt

tf.random.set_random_seed(78)
```

The data is created with numpy so that it can be easily analyzed and reused. There will be a thousand points created froma standard normal distribution and the mean will be moved to 1.2.

```python
np.random.seed(78)
y_data = np.random.randn(1000) + 1.2
print("Mean: {}".format(np.mean(y_data)))
plt.hist(y_data)
plt.show
plt.show()
```

The data will be imported into Tensorflow by a placeholder called x. The jumps will be pre-decided. Our generative function will be $g(\mu'|\mu_t)=N(\mu_t, 0.1)$. Given that we don't have each $\mu_t$ yet we can make an array of jumps, $J\sim N(0, 0.1)$, so
\begin{equation}
g(\mu'|\mu_t) = \mu_t + j_t \sim N(0+\mu_t, 0.1) = N(\mu_t, 0.1)
\end{equation}

This is far from efficient if you have many dimensions and many samples, but the demands of this program won't come close to pushing the limits of a basic computer.

```python
y = tf.placeholder(tf.float32)
J = tf.random_normal([1000], stddev=.1, dtype=tf.float32)
```

In this case, the initialization will be decided and calculated outside of the eventual jump function. Normally it would be randomly chosen by the computer, but this was defined for the sake of analysis.

```python
init_mu = tf.constant(5., dtype=tf.float32)
init_dist = tfd.Normal(loc=init_mu, scale=1.)
init_lp = tf.reduce_sum(init_dist.log_prob(y))
```

Next, the jump function is defined. It takes two inputs: the prior output and the next jump. The prior output will be a tuple that includes the $x_t$ in the first slot and the log probability of $x_t$ in the second spot. The next jump will be our $j_t$ number.

The candidate is produced from the sum of the jump and the first element of the output, which is $x_t$. The candidate distribution and the candidate log probability are calculated from our candidate. In this case, we know the standard deviation is 1; we're only searching for the mean of this distribution.

Finally, the ratio is calculated and a flip is made. In theory, you could pre-calculate the flips along with the jumps. The jump function then returns the result of a condition. In TensorFlow, the outputs of a condition have to be functions.

```python
def jump(output, next):
    candidate = output[0] + next
    cand_dist = tfd.Normal(loc=candidate, scale=1.)
    cand_lp = tf.reduce_sum(cand_dist.log_prob(y))
    ratio = tf.exp(cand_lp - output[1])
    flip = tf.random_uniform([1])[0]
    f1 = lambda: (candidate, cand_lp)
    f2 = lambda: (output[0], output[1])
    return tf.cond(flip < ratio, f1, f2)
```

A chain is created through a scan function. A scan function is usually used for recurrent neural networks, where a function uses the prior output in combination with the next input. This works perfectly for the needs of this experiment.

```python
with tf.Session() as sess:
    test_one = sess.run(tf.scan(jump, J, initializer=(init_mu, init_lp)), feed_dict={y: y_data})
```

A quick look at the results. Only the last 500 results will be considered. The first 500 are a warmup leading into the final value.

```python
def summarise_result(result, warmup_ratio=.5, title='$\mu$'):
    result=result[int(warmup_ratio*result.shape[0]):]
    print('Mean: {}'.format(np.mean(result)))
    plt.hist(result)
    plt.title(title)
    return plt.show()

summarise_result(test_one[0])
```

A step graph function is create to quickly analyze our chain.

```python
def step_graph(data, label=None, warmup=False, warmup_ratio=.5):
    if not warmup:
        data=data[int(warmup_ratio*data.shape[0]):]
    plt.plot(data)
    plt.title(label)
    return plt.show()
```

For starters, we'll look to see how the chain warmed up into its distribution, and then we'll take a closer look at the post warmup phase.

```python
step_graph(test_one[0], label='$\mu$', warmup=True)
step_graph(test_one[1], label='Log Probability', warmup=True)
```

```python
step_graph(test_one[0], label='$\mu$')
step_graph(test_one[1], label='Log Probability')
```

Lastly, we'll take a look at the comparison of the draws to the log probability. To no surprise, given that there is only one variable in a normal distribution, the graph displays a concave down quadratic function with it's peak at the mean of the data.

```python
plt.plot(test_one[0][500:], test_one[1][500:], '.')
plt.xlabel('$\mu$')
plt.ylabel('Log Probablity')
plt.show()
```

In fact, if you run a quadratic regression from x to the log probability and find the Vertex, the x portion of the vertex is close to the mean of the data.

```python
quadratic_x = np.stack([test_one[0], np.square(test_one[0]), np.ones(1000)])
quadratic_y = test_one[1]
lin_result = np.linalg.inv(quadratic_x @ quadratic_x.T) @ quadratic_x @ quadratic_y
print('Vertex of mu: {}'.format(-lin_result[0] / (2 * lin_result[1])))
```

    Vertex of mu: 1.1923796565842126

## Changing The Jump Size & Shape

The size and shapes of the jumps can change the outcomes quite a bit. In this case, the standard devation of our normal jump will be increased and decreased, and then the shape will be changed to a uniform.

```python
J_two = tf.random_normal([1000], stddev=.5, dtype=tf.float32)

with tf.Session() as sess:
    test_two = sess.run(tf.scan(jump, J_two, initializer=(init_mu, init_lp)), feed_dict={y: y_data})

summarise_result(test_two[0])
step_graph(test_two[0], label='$\mu$')
step_graph(test_two[1], label='Log Probability')
step_graph(test_two[0], label='$\mu$', warmup=True)
step_graph(test_two[1], label='Log Probability', warmup=True)
```

The chain converges on the center quicker but has a tendency to sit, which creates an uneven distribution with certain portions having for more samples than they should. For instance, here are the number of unique variables out of the 500.

```python
print("# of unique samples:", np.unique(test_two[0]).shape[0])
```

    # of unique samples: 96

```python
J_three = tf.random_normal([2000], stddev=.01, dtype=tf.float32)

with tf.Session() as sess:
    test_three = sess.run(tf.scan(jump, J_three, initializer=(init_mu, init_lp)), feed_dict={y: y_data})

summarise_result(test_three[0])
step_graph(test_three[0], label='$\mu$')
step_graph(test_three[1], label='Log Probability')
step_graph(test_three[0], label='$\mu$', warmup=True)
step_graph(test_three[1], label='Log Probability', warmup=True)
```

In the case of a small jump, the number of jumps had to be doubles to make up for the amount of time it takes to converge on the mean, but, once it gets there, it does a good job of sampling the space. Finally, here are the results of a uniform jump. The results are far from inadequate.

```python
J_four = tf.random_uniform([1000], minval=-.1, maxval=.1, dtype=tf.float32)

with tf.Session() as sess:
    test_four = sess.run(tf.scan(jump, J_four, initializer=(init_mu, init_lp)), feed_dict={y: y_data})

summarise_result(test_four[0])
step_graph(test_four[0], label='$\mu$')
step_graph(test_four[1], label='Log Probability')
step_graph(test_four[0], label='$\mu$', warmup=True)
step_graph(test_four[1], label='Log Probability', warmup=True)
```

## Linear Regression

Next up, a sample from a simple linear regression model will be taken. Once again, the standard deviation, the error in this case, will be known.

The linear regression model will look like so
\begin{equation}
y \sim N(2.7 + 0.4x, 1)
\end{equation}

The sampling will be taken from
\begin{equation}
y \sim N(\alpha + \beta x, 1)
\end{equation}

Specifically, the $\alpha$ and $\beta$ parameters.

For starters, new data has to be created an x placeholder will also have to be added.

```python
np.random.seed(555)
x_data = (np.random.rand(1000) * 20).astype(np.float32)
y_data_two = 2.7 + x_data * .4 + np.random.randn(1000)

plt.plot(x_data, y_data_two, '.')
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

```python
x = tf.placeholder(tf.float32, name='x')
```

Next, the jumps are made in the for of a 1000 X 2 matrix. The first column will be the jump for the $\alpha$, and the second column will be the jump for the $\beta$. If a jump fails, both the $\alpha$ and the $\beta$ remain the same even if one should have moved.

```python
J_five = tf.random_uniform([1000, 2], minval=-.09, maxval=.09, dtype=tf.float32)
```

The initialized now includes two inputs. One of the inputs will be much further away from its result than the other. We will see why we are doing this later. The calculation of the $\mu$ inside of the normal function was described earlier. Instead of having one $\mu$ that answers for all y, there is a single $\mu$ for each y. There is essentially 1000 Normal distributions with a mean of $\mu=\alpha + \beta x$ and standard deviation of 1. A column of ones could have been stacked onto the x vector, and the second initialization $\mu$ could have been matrix multiplied against that, but it was written this way for readability.

```python
init_mu_two = tf.constant([5., -1.3], dtype=tf.float32)
init_dist_two = tfd.Normal(loc=init_mu_two[0] + init_mu_two[1] * x, scale=1.)
init_lp_two = tf.reduce_sum(init_dist_two.log_prob(y))
```

The only thing that has chaged in our jump function is the calculation of the $\mu=\alpha + \beta x$ in the Normal distribution. All other things are the same.

```python
def jump_two(output, next):
    candidate = output[0] + next
    cand_dist = tfd.Normal(loc=candidate[0] + candidate[1] * x, scale=1.)
    cand_lp = tf.reduce_sum(cand_dist.log_prob(y))
    ratio = tf.exp(cand_lp - output[1])
    flip = tf.random_uniform([1])[0]
    f1 = lambda: (candidate, cand_lp)
    f2 = lambda: (output[0], output[1])
    return tf.cond(flip < ratio, f1, f2)
```

```python
with tf.Session() as sess:
    test_five = sess.run(tf.scan(jump_two, J_five, initializer=(init_mu_two, init_lp_two)), feed_dict={y: y_data_two, x: x_data})
```

A quick look at the data:

```python
summarise_result(test_five[0][:,0], title=r'$\alpha$')
summarise_result(test_five[0][:,1], title=r'$\beta$')
```

The data is skewed. The reason for this can be seen in the Step graph that includes the warm-up. It takes longer for the $\alpha$ parameter to reach its destination since it is further away than the $\beta$ parameter is from its destination.

```python
step_graph(test_five[0][:,0], warmup=True, label=r"$ \alpha $")
step_graph(test_five[0][:,1], warmup=True, label=r'$beta$')
```

Even when it reaches its desitination, it doesn't move around much.

```python
step_graph(test_five[0][:,0], warmup=False, label=r'$\alpha$')
```

To solve this, the jumps are going to start out much larger and then convert to much smaller jumps. Specifically the first 400 jumps will have a uniform distribution from -1 to 1. The next 1600 jumps, 600 of which will still be warmups, will have a uniform distribution from -0.02 to 0.02. All other things will be left the same.

```python
J_six = tf.concat([tf.random_uniform([400, 2], minval=-1., maxval=1., dtype=tf.float32), tf.random_uniform([1600, 2], minval=-.02, maxval=.02)], 0)

with tf.Session() as sess:
    test_six = sess.run(tf.scan(jump_two, J_six, initializer=(init_mu_two, init_lp_two)), feed_dict={y: y_data_two, x: x_data})
```

Now the jumps quickly find their mark well before the 400 mark and have room for any necessary easing before the 1000 mark. In this case, data could be used from the last 1500 jumps, but the last 1000 will be more than sufficient enough.

```python
step_graph(test_six[0][:,0], warmup=True, label=r'$\alpha$')
step_graph(test_six[0][:,1], warmup=True, label=r'$\beta$')
step_graph(test_six[0][:,0], label=r'$\alpha$')
step_graph(test_six[0][:,1], label=r'$\beta$')
```

```python
summarise_result(test_six[0][:,0], title=r"$\alpha$")
summarise_result(test_six[0][:,1], title=r"$\beta$")
```

The Random Walk Metropolis algorithm does a good enough job of exploring the space but sometimes gets carried away in the direction it's going. After the algorithm gets to a certain point, it's unlikely for it to continue on, but there is still the possibility of the algorithm turning around twice and exploring a little too much of one side of the space. This can Usually, this can be solved by having multiple chains. It is likely that one of the chains will explore the other end of the state space. This assumes the distribution isn't skewed to begin with.

Lastly, a bivariate graph to see how the space was explored by both. Overall, it centers around what would be expected, but there is definitely some missing information that can be found with multiple chains.

```python
import seaborn as sns
sns.jointplot(test_six[0][:,0][500:], test_six[0][:,1][500:], kind='kde').plot_joint(plt.scatter, color='w', marker='+').set_axis_labels(r'$\alpha$', r'$\beta$')
```

## Conclusions

The Metropolis algorithm, like any algorithm, has its flaws. There is a lot of work when it comes to trying to find the right jump distance, and there is no guarantee that the algorithm explores everything that it needs to. In fact, there is no denying that there are much better algorithms, but the Metropolis algorithm sets the stage for many of those and is an important and interesting study.


## Exercises

```python
import numpy as np
import graphviz as gz
import itertools
import sys
sys.path.append('../..')
from utils.graph_utils import digraph_from_numpy_array
```

### Exercise 3.1
The joint distribution corresponding to the given BBN is the following
$$
p(A, H, D, U, P) = p(A \mid U)p(U \mid P, D)p(H \mid P)p(P)p(D).
$$
Then, we can calculate $p(P = tr)$ as follows:
$$
p(P = tr) = \sum_{A,H,U,D} p(A, H, U, D, P = tr) = \sum_{A,H,U,D} p(A \mid U)p(U \mid P, D)p(H \mid P)p(P)p(D).
$$

To do so, we define the conditional probability tables in the following format:
1. A 1D array is used to store the probabilities.
2. For a conditional distribution $P(A, B \mid C, D)$ the array is indexed with the value
$$
2^{(A = tr) + (B = tr) + (C = tr) + (D = tr)} - 1
$$
in the same order as the variables are listed in the distribution.
3. According to this indexing, for an array $A$ of length $L$, $A[i] + A[i + L/2] = 1$ for $i < L/2$.

```python
pAgU = np.array([0.5, 0.05, 0.5, 0.95])
pUgPaD = np.array([0.99, 0.1, 0.1, 0.0001, 0.01, 0.9, 0.9, 0.999])
pHgP = np.array([0.8, 0.1, 0.2, 0.9])
pP = np.array([0.8, 0.2])
pD = np.array([0.6, 0.4])

prob_sum = 0
P = '1'
for A in '01':
    for H in '01':
        for U in '01':
            for D in '01':
                prod = 1
                prod *= pAgU[int(A + U, 2)]
                prod *= pUgPaD[int(U + P + D, 2)]
                prod *= pHgP[int(H + P, 2)]
                prod *= pP[int(P, 2)]
                prod *= pD[int(D, 2)]
                prob_sum += prod
prob_sum
```

    0.19992799999999999

### Exercise 3.2
The distribution $p(a, b, c) = p(c \mid a, b)p(a)p(b)$ corresponds to the following BBN:

```python
G = gz.Digraph()
G.edges(['ac', 'bc'])
G
```

#### Part (i)
If we have no information about $c$, $a$ and $b$ are independent. Hence, $a \ci b \mid \varnothing$.

#### Part (ii)
If we condition on $c$, we have some information about both $a$ and $b$, and we know that $a$ and $b$ combine to result in $c$. Hence, $a$ and $b$ are not independent conditioned on $c$.

### Exercise 3.3

```python
G = gz.Digraph()
G.edges(['at', 'te', 'ex', 'le', 'sl', 'sb', 'bd', 'ed'])
G
```

#### $t \ci s \mid d$ ?
In both paths from $t$ to $s$ there isn't a blocking node. Hence, both paths induce graphical dependence.

Recall that a node on a path is blocking if it is either
1. a node who is a collider and neither it nor any of its descendants are in the condition set, or
2. a node who is not a collider and it is in the condition set.

#### $l \ci b \mid s$ ?
Both paths from $l$ to $b$ are blocked.

1. Upper path is blocked because of $s$. $s$ is not a collider and it is in the condition set.
2. Lower path is blocked because of $d$. $d$ is a collider and neither it nor any of its descendants are in the condition set.

#### $a \ci s \mid l$ ?
Both paths from $a$ to $s$ are blocked.

1. $a-t-e-l-s$ is blocked because of $l$. $l$ is not a collider and it is in the condition set.
2. $a-t-e-d-b-s$ is blocked because of $d$. $d$ is a collider and neither it nor any of its descendants are in the condition set.

#### $a \ci s \mid l, d$ ?
In this case, $a-t-e-d-b-s$ is not blocked, and hence $a$ is not conditionally independent from $l$. This path is not blocked because now $d$ is a collider and it is in the condition set. Hence, $d$ doesn't block the path.

### Exercise 3.4
$$
p(\cdot) = p(x \mid e)p(d \mid e, b)p(e \mid t, l)p(t \mid a)p(l \mid s)p(b \mid s)p(a)p(s)
$$

```python
pXgE = np.array([0.95, 0.02, 0.05, 0.98])
pDgEaB = np.array([0.9, 0.2, 0.3, 0.1, 0.1, 0.8, 0.7, 0.9])
pEgTaL = np.array([1, 0, 0, 0, 0, 1, 1, 1])
pTgA = np.array([0.99, 0.95, 0.01, 0.05])
pLgS = np.array([0.99, 0.9, 0.01, 0.1])
pBgS = np.array([0.7, 0.4, 0.3, 0.6])
pA = np.array([0.99, 0.01])
pS = np.array([0.5, 0.5])

pD = np.array([0., 0.])
pDgS = np.array([0., 0., 0., 0.])
for d in '01':
    for s in '01':
        for x in '01':
            for e in '01':
                for t in '01':
                    for l in '01':
                        for a in '01':
                            for b in '01':
                                prod = 1
                                prod *= pXgE[int(x + e, 2)]
                                prod *= pDgEaB[int(d + e + b, 2)]
                                prod *= pEgTaL[int(e + t + l, 2)]
                                prod *= pTgA[int(t + a, 2)]
                                prod *= pLgS[int(l + s, 2)]
                                prod *= pBgS[int(b + s, 2)]
                                prod *= pA[int(a, 2)]
                                prod *= pS[int(s, 2)]
                                pD[int(d, 2)] += prod
                                pDgS[int(d + s, 2)] += prod/pS[int(s, 2)]
```

```python
pDgS
```

    array([ 0.6808668,  0.447192 ,  0.3191332,  0.552808 ])

* $p(D = tr) = $ {{pD[1]}}
* $p(D = fa) = $ {{pD[0]}}

* $p(D = tr \mid S = tr) = $ {{pDgS[3]}}
* $p(D = fa \mid S = tr) = $ {{pDgS[1]}}
* $p(D = tr \mid S = fa) = $ {{pDgS[2]}}
* $p(D = fa \mid S = fa) = $ {{pDgS[0]}}

### Exercise 3.5
If we interpret curing bronchitis causally, then we intervene the network at node $b$ and hence, edge $s - b$ should be removed. This would correspond to an experiment in which the doctor first cures his/her patient's bronchitis, and then conducts tests to check shortness of breath. Hence, the system would correspond to

$$
p(a, t, e, x, l, s, d \mid do(b = fa)) = \prod_{x \in X \setminus b} p(x \mid pa(x)).
$$

This differs from $p(d = tr \mid b = fa)$ non-casual interpretation because in the non-casual interpretation, not having bronchitis also depends on smoking status. If we assume that there is a really strong dependence between smoking and bronchitis, $p(d = tr \mid b = fa)$ would sample from non-smoking part of the population. On the other hand, a casual interpretation of curing the bronchitis would allow us to sample all persons regardless of their smoking status.

### Exercise 3.7
$p(A, S, C) = p(C \mid A, S)p(A)p(S)$ corresponds to

```python
G = gz.Digraph()
G.edges(['AC', 'SC'])
G
```

#### $A \ci S \mid \varnothing$ ?
True, because $C$ is a collider and it is not in the condition set.

#### $A \ci S \mid C$ ?
False, because $C$ cannot block the dependence path between $A$ and $S$. Hence, $A$ and $S$ are d-connected, and they are not conditionally independent

#### How to incorporate building industry variable?
$$
p(A, B, C, S) = p(C \mid A, S)p(A \mid B)p(S \mid B)p(B).
$$
which corresponds to the diamond shape belief network:

```python
G.edges(['BA', 'BS'])
G
```

### Exercise 3.8

#### Part (1)
$
\begin{align}
p(B = tr \mid W = tr) &= \frac{p(B = tr, W = tr)}{p(W = tr)} \\
		      &= \frac{\sum_{A, G}p(B = tr, W = tr, A, G)}{\sum_{A, B, G}p(B, W = tr, A, G)} \\
		      &= \frac{p(B = tr)\sum_{A, G} p(W = tr \mid A)p(G \mid A)p(A \mid B = tr)}{\sum_{A, B, G} p(W = tr \mid A)p(G \mid A)p(A \mid B)p(B)} \\
\end{align}
$

$
\begin{align}
p(B = tr \mid W = tr, G = fa) &= \frac{p(B = tr, W = tr, G = fa)}{p(W = tr, G = fa)} \\
			      &= \frac{\sum_A p(W = tr \mid A)p(G = fa \mid A)p(A \mid B = tr)p(B = tr)}{\sum_{A, B} p(W = tr \mid A)p(G = fa \mid A)p(A \mid B)p(B)}
\end{align}
$

#### Part (2)
In this case, we have the following information
$
\begin{align}
p(G = fa \mid \tilde{G}) &= 0.9 \\
p(G = tr \mid \tilde{G}) &= 0.1 \\
p(W = tr \mid \tilde{W}) &= 0.3 \\
p(W = tr \mid \tilde{W}) &= 0.7 \\
\end{align}
$
Since we have albeit uncertain evidence about $G$ and $W$ separately, we can do the following derivation
$
\begin{align}
p(G, W \mid \tilde{G}, \tilde{W}) &= p(W \mid G, \tilde{G}, \tilde{W})p(G \mid \tilde{G}, \tilde{W}) \\
				 &= p(W \mid G, \tilde{W})p(G \mid \tilde{G}) && \\
				 &= p(W \mid \tilde{W})p(G \mid \tilde{G}) \\
\end{align}
$
because of the following statements:
1. Evidence about $W$ doesn't directly tell anything about $G$, and vice versa,
2. If we have evidence about $W$, we completely specify the distribution of $W$ regardless of any dependencies it has with other variables. Hence, $p(W \mid X_1, \dots, X_n, \tilde{W}) = p(W \mid \tilde{W})$

Afterwards, we can compute the desired distributions using Jeffrey's rule as follows:
$$
p(B = tr \mid \tilde{W}) = \sum_W p(B = tr \mid W)p(W \mid \tilde{W}).
$$
$
\begin{align}
p(B = tr \mid \tilde{W}, \tilde{G}) &= \sum_{G, W} p(B = tr \mid G, W)p(G, W \mid \tilde{G}, \tilde{W}) \\
				    &= \sum_{G, W} p(B = tr \mid G, W)p(G \mid \tilde{G})p(W \mid \tilde{W})
\end{align}
$

### Exercise 3.9

```python
G = gz.Digraph()
G.edges(['AD', 'GD', 'AR', 'DR', 'GR'])
G
```

$
\begin{align}
p(R = tr \mid D = tr) &= \frac{p(R = tr, D = tr)}{p(D = tr)} \\
		      &= \frac{\sum_{A, G} p(R = tr \mid D = tr, A, G)p(D = tr \mid A, G)p(A)p(G)}{\sum_{A, G, R} p(R \mid D = tr, A, G)p(D = tr \mid A, G)p(A)p(G)}
\end{align}
$

If we intervene on node $D$ and set it to $tr$, then the new belief network becomes:

```python
G = gz.Digraph()
G.edges(['AR', 'GR'])
G.node('D = tr', color='green', style='filled')
G.edge('D = tr', 'R')
G
```

Let $\tilde{p}$ represent the distribution of this new network. Then, $p(R = tr \mid do(D = tr), A = yo) = \tilde{p}(R = tr \mid D = tr, A = yo)$. Hence
$
\begin{align}
\tilde{p}(R = tr \mid D = tr, A = yo) &= \frac{p(D = tr)p(A = yo) \sum_G p(R = tr, D = tr, A = yo, G)p(G)}{p(D = tr)p(A = yo) \sum_{R, G} p(R, D = tr, A = yo, G)p(G)} \\
				      &= \frac{\sum_G p(R = tr, D = tr, A = yo, G)p(G)}{\sum_{R, G} p(R, D = tr, A = yo, G)p(G)}
\end{align}
$

### Exercise 3.12

```python
def skeleton(A):
    """
    Given a (directed/undirected) graph A, this function returns the
    skeleton of A.

    Skeleton of a graph is obtained by making every edge in A
    bi-directional and setting its value to 1.

    Parameters
    ----------
    A : numpy.array
        Adjacency matrix of the graph represented by A.

    Returns
    -------
    skeleton : numpy.array
        Adjacency matrix of the skeleton of A.

    Complexity
    ----------
    B(n) : n^2 for nxn A.
    W(n) : n^2 for nxn A.
    A(n) : n^2 for nxn A.
    """
    skeleton = np.copy(A)
    n = len(skeleton)

    skeleton[skeleton != 0] = 1
    for i in range(n):
        for j in range(n):
            skeleton[i, j] |= skeleton[j, i]

    return skeleton
```

```python
def immorality_set(A):
    """
    Given a graph A, this function returns the set of immoralities
    in A.

    An immorality is defined as a pair ({m, n}, k) such that
    k is a child of both m and m with no connection between m and n.

    Parameters
    ----------
    A : numpy.array
        Adjacency matrix of the graph represented by A.

    Returns
    -------
    immoral_set : set
        Set of pairs ({m, n}, k) whose first element is a set of nodes and
        the second element the target of the immorality.

    Complexity
    ----------
    B(n) : n^4 for nxn A.
    W(n) : n^4 for nxn A.
    A(n) : n^4 for nxn A.

    TODO : Come up with an efficient algorithm to find immoralities.
    """
    n = len(A)

    immoral_set = set()
    for i in range(n):
        for j in range(i):
            for k in range(n):
                for m in range(n):
                    if A[i, m] and A[j, m] and not (A[i, j] or A[j, i]):
                        immoral_set.add((frozenset((i, j)), m))

    return immoral_set
```

```python
def markov_equiv(A, B):
    """
    Given two belief networks represented as DAGs with associated matrices A and B,
    this function returns True if A and B are Markov equivalent; and False otherwise.

    Parameters
    ----------
    A : numpy.array
        Adjacency matrix of the first DAG.
    B : numpy.array
        Adjacency matrix of the second DAG.

    Returns
    -------
    equiv : bool
        True if A and B are Markov equivalent; false, otherwise.

    Complexity
    ----------
    B(n) : n^4 for nxn A.
    W(n) : n^4 for nxn A.
    A(n) : n^4 for nxn A.
    """
    if A.shape != B.shape:
        return False

    if not np.array_equal(skeleton(A), skeleton(B)):
        return False

    return immorality_set(A) == immorality_set(B)
```

### Exercise 3.13

```python
A = np.array([
    [0, 0, 1, 1, 0, 1, 0, 0, 0],
    [0, 0, 1, 0, 1, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 1, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 1, 1],
    [0, 0, 1, 0, 0, 0, 1, 0, 0],
    [0, 0, 0, 1, 0, 0, 0, 1, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 1],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
])
B = np.array([
    [0, 0, 1, 1, 0, 0, 0, 0, 0],
    [0, 0, 1, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 1, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 1, 1],
    [0, 1, 1, 0, 0, 0, 1, 0, 0],
    [1, 0, 0, 1, 0, 0, 0, 1, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 1],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
])
print('Graph of A')
display(digraph_from_numpy_array(A))
print('Graph of B')
display(digraph_from_numpy_array(B))
```

    Graph of A

    Graph of B

```python
print('Skeleton of A')
print(skeleton(A))
print('Skeleton of B')
print(skeleton(B))
print('skeleton(A) == skeleton(B) : {}'.format(np.array_equal(skeleton(A), skeleton(B))))
print()
print('Immoralities in A')
print(immorality_set(A))
print('Immoralities in B')
print(immorality_set(B))
print('Same immoralities : {}'.format(immorality_set(A) == immorality_set(B)))
print()
print('Markov Equivalent(A, B) : {}'.format(markov_equiv(A, B)))
```

    Skeleton of A
    [[0 0 1 1 0 1 0 0 0]
     [0 0 1 0 1 0 0 0 0]
     [1 1 0 0 1 0 1 0 0]
     [1 0 0 0 0 1 0 1 1]
     [0 1 1 0 0 0 1 0 0]
     [1 0 0 1 0 0 0 1 0]
     [0 0 1 0 1 0 0 0 1]
     [0 0 0 1 0 1 0 0 0]
     [0 0 0 1 0 0 1 0 0]]
    Skeleton of B
    [[0 0 1 1 0 1 0 0 0]
     [0 0 1 0 1 0 0 0 0]
     [1 1 0 0 1 0 1 0 0]
     [1 0 0 0 0 1 0 1 1]
     [0 1 1 0 0 0 1 0 0]
     [1 0 0 1 0 0 0 1 0]
     [0 0 1 0 1 0 0 0 1]
     [0 0 0 1 0 1 0 0 0]
     [0 0 0 1 0 0 1 0 0]]
    skeleton(A) == skeleton(B) : True

    Immoralities in A
    {(frozenset({3, 6}), 8), (frozenset({0, 1}), 2), (frozenset({0, 4}), 2)}
    Immoralities in B
    {(frozenset({3, 6}), 8), (frozenset({0, 1}), 2), (frozenset({0, 4}), 2)}
    Same immoralities : True

    Markov Equivalent(A, B) : True

### Exercise 3.14
#### The graph of the computer network
First, we draw the graph of the computer network. Note that this graph has nothing to do with Bayes Nets.

```python
adj_mat = np.reshape(np.ones(9), (3, 3))
adj_mat[np.diag_indices(3)] = 0
digraph_from_numpy_array(adj_mat, labels=['C1', 'C2', 'C3'], edge_values=False)
```

#### Describing the Bayes Net
1. When we consider the $1$ timestep paths, each $C_{ij}$ is independent for $i \neq j$. Hence, if we draw the corresponding Bayes Net, none of $C_{ij}$s would have direct links between them.
2. When we consider the $2$ timestep paths, direct dependencies get a little more complicated. However, before describing these dependencies, note that $C_{ij}(2)$ and $C_{ik}(2)$ or $C_{ij}(2)$ and $C_{kj}(2)$ **do not directly depend on each other**. In general, knowing one can alter our beliefs about the other one through $C_{ij}$ terms, i.e. $1$ timestep probabilities. Hence, each $C_{ij}(2)$ would not have direct links between them either for $i \neq j$.
3. $C_{ij}(2) = 1 \implies C_{ij} = 1 \lor (C_{ik} = 1 \land C_{kj} = 1)$ for $k = \{1, 2, 3\} \setminus \{i, j\}$.
Hence, $C_{ij}$ directly affects the value of $C_{ij}(2), C_{ik}(2), C_{kj}(2)$.

According to these statements we draw the Bayesian Network corresponding to the probabilistic system described in the question.

```python
G = gz.Digraph()
computer_set = {1, 2, 3}
roots = ('C{}{}'.format(*perm) for perm in itertools.permutations(computer_set))
edges = []
for root in roots:
    i, j = [int(c) for c in root[1:]]
    k = (computer_set - {i, j}).pop()
    edges.append((root, '{}(2)'.format(root)))
    edges.append((root, 'C{}{}(2)'.format(i, k)))
    edges.append((root, 'C{}{}(2)'.format(k, j)))

G.edges(edges)
G
```

#### $p(C_{12} = 1 \mid C_{12}(2) = 1, C_{23}(2) = 0)$
$$
    p(C_{12} = 1 \mid C_{12}(2) = 1, C_{23}(2) = 0) = \frac{p(C_{12}(2) = 1, C_{23}(2) = 0 \mid C_{12} = 1)p(C_{12} = 1)}{p(C_{12}(2) = 1, C_{23}(2) = 0)}
$$
##### $p(C_{12}(2) = 1, C_{23}(2) = 0 \mid C_{12} = 1)$
Note that $p(C_{ij}(2) = 1 \mid C_{ij} = 1) = 1$ according to the definition of $2$ timestep definition. Hence,
$
\begin{align}
p(C_{12}(2) = 1, C_{23}(2) = 0 \mid C_{12} = 1) &= p(C_{23}(2) = 0 \mid C_{12} = 1) \\
						&= p(C_{23} = 0)(1 - p(C_{21} = 1, C_{13} = 1)) \\
						&= 0.9(1 - 0.1 \cdot 0.1) \\
						&= 0.891.
\end{align}
$
##### $p(C_{12}(2) = 1, C_{23}(2) = 0)$
To calculate this joint probability, we don't need to marginalize over the whole distribution due to the structure of our Bayes Net. Although $C_{12}(2)$ and $C_{23}(2)$ are not (unconditionally) independent of each other, **they are independent given $C_{13}$**, i.e. $C_{12}(2) \ci C_{23}(2) \mid C_{13}$. This can be easily verified with the following two observations
1. If $C_{13}$ is given, then $C_{12}(2) - C_{13} - C_{23}(2)$ path is blocked since $C_{13}$ is not a collider and is in the condition set.
2. Any other indirect path between $C_{12}(2)$ and $C_{23}(2)$ is blocked since each such path must contain a $C_{ij}(2)$ node different than $C_{12}(2)$ and $C_{23}(2)$, and each such node must be a collider without being in the condition set.

After this conditional independence realization, we can calculate this probability as follows:

$
\begin{align}
p(C_{12}(2) = 1, C_{23}(2) = 0) &= \sum_{C_{13}} p(C_{12}(2) = 1, C_{23}(2) = 0 \mid C_{13})p(C_{13}) \\
				&= \sum_{C_{13}} p(C_{12}(2) = 1 \mid C_{13})p(C_{23}(2) = 0 \mid C_{13})p(C_{13}) \\
				&= (0.1 + 0.1)(0.9 \cdot 0.9)0.1 + 0.1 \cdot 0.9 \cdot 0.9 \\
				&= 0.0972.
\end{align}
$
##### Final Result
$$
    p(C_{12} = 1 \mid C_{12}(2) = 1, C_{23}(2) = 0) = \frac{0.891 \cdot 0.1}{0.0972} = 0.916.
$$
We see that our posterior belief about $C_{12} = 1$ increases drastically. This makes sense since our evidence suggests that $C_{12}(2) = 1$.

#### $p(C_{32} = 1 \mid C_{12}(2) = 1, C_{23}(2) = 0)$
In the first question, computing the likelihood was straightforward due to the given variable. In this case, the likelihood is $p(C_{12}(2) = 1, C_{23}(2) = 0 \mid C_{32} = 1)$. We cannot get rid of any variable directly, or separate the joint probability into multiplicatives. Again, we need to look for conditional independence cases to efficiently calculate the likelihood. Again, we use $C_{13}$. Note that $\{C_{12}(2), C_{32}\} \ci C_{23}(2) \mid C_{13}$ since all the paths from one set to the other is blocked. Then, we can calculate the likelihood as follows:

$
\begin{align}
p(C_{12}(2) = 1, C_{23}(2) = 0 \mid C_{32} = 1) &= \frac{p(C_{12}(2) = 1, C_{23}(2) = 0, C_{32} = 1)}{p(C_{32} = 1)} \\
						&= 10p(C_{12}(2) = 1, C_{23}(2) = 0, C_{32} = 1).
\end{align}
$
where
$
\begin{align}
p(C_{12}(2) = 1, C_{23}(2) = 0, C_{32} = 1) &= \sum_{C_{13}} p(C_{12}(2) = 1, C_{23}(2) = 0, C_{32} = 1 \mid C_{13})p(C_{13}) \\
					    &= \sum_{C_{13}} p(C_{12}(2) = 1, C_{32} = 1 \mid C_{13})p(C_{23}(2) = 0 \mid C_{13}) \\
					    &= \sum_{C_{13}} p(C_{12}(2) = 1 \mid C_{32} = 1, C_{13})p(C_{32} = 1)p(C_{23}(2) = 0 \mid C_{13}) \\
					    &= (p(C_{12}(2) = 1 \mid C_{32} = 1, C_{13} = 1)p(C_{32} = 1)p(C_{23}(2) = 0 \mid C_{13} = 1)) + (p(C_{12}(2) = 1 \mid C_{32} = 1, C_{13} = 0)p(C_{32} = 1)p(C_{23}(2) = 0 \mid C_{13} = 0)) \\
                        &= 1 \cdot 0.1 \cdot 0.9 \cdot 0.9 + 0.1 \cdot 0.1 \cdot 0.9 \\
                        &= 0.09.
\end{align}
$
With this likelihood, $p(C_{32} = 1 \mid C_{12}(2) = 1, C_{23}(2) = 0) = 0.092$. In this example, we see that our posterior belief did not change that much from our prior belief. This also makes sense because we did not get a game-changing evidence about $C_{32}$.