```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

import matplotlib
import seaborn as sns
```

```python
matplotlib.rcParams['figure.figsize'] = (10, 8) # set default figure size, 8in by 6in
%matplotlib inline
```

# Decision Trees

Like SVMs, *Decision Trees* are versatile machine learning algorithms that can perform both classification
and regression tasks.  They are very powreful algorithms, capable of fitting complex datasets.

Decision Trees are also the fundamental components of *Random Forests*, which are basically a collection, an
ensemble if you will, of several separately trained decision trees.

# Training and Visualizing a Decision Tree

To understand Decision Trees, let's just jump into building and training on.  The following code trains a
`DecisionTreeClassifier` from the `scikit-learn` library.  We will train the Decision Tree on the
iris dataset (see [here](https://en.wikipedia.org/wiki/Iris_flower_data_set) for more details about
this dataset if you are not familiar with it).

```python
# just fyi, get a sense of the data set
iris = sns.load_dataset('iris')
iris.head()
```

      sepal_length
      sepal_width
      petal_length
      petal_width
      species

      0
      5.1
      3.5
      1.4
      0.2
      setosa

      1
      4.9
      3.0
      1.4
      0.2
      setosa

      2
      4.7
      3.2
      1.3
      0.2
      setosa

      3
      4.6
      3.1
      1.5
      0.2
      setosa

      4
      5.0
      3.6
      1.4
      0.2
      setosa

```python
# also fyi, visualize the 4 parameters
sns.pairplot(iris, hue='species', height=3.0);
```

```python
# just get the petal length and width, we reload the data from sklearn, which already has the labels
# converted to integers
from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data[:, 2:] # petal length and width
y = iris.target
```

```python
# create a Decision Tree classifier and fit it to the data
from sklearn.tree import DecisionTreeClassifier

tree_clf = DecisionTreeClassifier(max_depth=2)
tree_clf.fit(X, y)
```

    DecisionTreeClassifier(max_depth=2)

We can visualize the trained Decision Tree by first using the export_graphviz() method to output a graph
definition file called `iris_tree.dot`

```python
from sklearn.tree import export_graphviz
export_graphviz(
   tree_clf,
   out_file = '../../figures/iris_tree.dot',
   feature_names = iris.feature_names[2:],
   class_names = iris.target_names,
   rounded=True,
   filled=True)
```

You can convert this `.dot` file to a variety of formats such as PDF or PNG using the dot command-line tool
from the `graphviz` package.  (From conda try 'conda install -c anaconda graphviz'). The following command line command converts the `.dot` file to a `.png` image file:

The following line will not work unless you have the `graphviz` package installed and the `dot` command line
tool available.  However, the image is in your repository, so you should be able to see the one generated for you
even if you can't generate it for yourself.

```python
!dot -Tpng ../../figures/iris_tree.dot -o ../../figures/iris_tree.png
```

The decision tree should looks like the following

# Making Predictions
The decision tree is a binary tree.  At each node a feature is tested, and based on the test you either make a decision
to proceed to the left or right in the tree.   For example, the root decision (at the top of the tree) tests if the
petal length is less than 2.45 (cm).  If this is true, we proceed to the left.  All of the colored nodes in the
figure represent leaf nodes, and when you reach a leaf node then a decision is made.  If you proceed to the left,
the decision tree will classifiy the input as being in the `setosa` class.

If however the petal length is greater than 2.45, you would instead move to the right.  Here we have not yet reached
a leaf node, so instead a second decision is made, in this case by testing the petal width.  Based on that decision,
you would proceed either left or right to the leaf nodes representing decisions for `versicolor` and `virginica`
classes respectively.

**Note**: One nice quality about the way a decision tree works is that they require very little data preparation.
In particular, they do not require feature scaling or centering at all.  This is because, unlike for logistic regression and SVM, we are not multiplying the features times a $\theta$ parameter.  This multiplication of feature times a weighted parameter means that feautres on a larger scale might have much larger effects on the prediction than small scaled features.  This does not happen for decision trees becase each decision is simply
a if/then split of the data.

In the figure, a nodes samples attribute counts how many training instances it applies to. For example, 50 training
samples have a petal length <= 2.45 cm, and 100 samples have a petal length > 2.45.  Also the value attribute
shown in the figure gives some valuable information.  For example, for the green `versicolor` leaf node, the
`value = [0, 49, 5]` attribute means that this leaf node applies to 54 training samples, of which it
classifies 49 of the training samples correctly as `versicolor` and 5 of the training samples incorrectly
as `virginica`.

Finally a nodes `gini` attribute measures the nodes *impurity*: a node is "pure" (gini = 0) if all training instances
it applies to belong to the same class.  For example, since the gold leaf node only applies to `setosa` training
instances, it is pure and its gini score is 0.

The Gini impurity equation is defined using this equaiton:

$$
G_i = 1 - \sum_{k=1}^{n} p_{i,k}^2
$$

Where $p_{i,k}$ is the ratio of class k instances among the training instances in the $i^{th}$ node.  So for example,
the green node that categorizes as `versicolor` has a Gini coefficient of

$$
G_i = 1 - \Big( \frac{0}{54} \Big)^2 - \Big( \frac{49}{54} \Big)^2 - \Big( \frac{5}{54} \Big)^2 \approx 0.168
$$

We will later come back to the Gini impurity measure, but we present it here because it is the value that is used when
actually building/fitting the decision tree.

We had previously introduced the concept of decision boundaries for a classifier, and visualized the decision boundaries.
The decision boundaries created by our simple decision tree so far can be visualized as this:

```python
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
h = .02  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

Z = tree_clf.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(figsize=(12,8))
plt.contourf(xx, yy, Z, cmap=plt.cm.Pastel2, alpha=0.8);

# plot the original data
setosa_indexes = np.where(y==0)[0]
versicolor_indexes = np.where(y==1)[0]
virginica_indexes = np.where(y==2)[0]

plt.plot(X[setosa_indexes, 0], X[setosa_indexes, 1], 'yo', label='setosa')
plt.plot(X[versicolor_indexes, 0], X[versicolor_indexes, 1], 'bs', label='versicolor')
plt.plot(X[virginica_indexes, 0], X[virginica_indexes, 1], 'g^', label='virginica')
plt.xlabel('petal length')
plt.ylabel('petal width')
plt.legend();
```

When we created the decision tree classifier initialy we had specified a `max_depth` of 2, meaning that
the decision tree stops splitting/createing nodes at depth 2.  The root of the tree is considered to be at depth
0, and it makes the split based on petal length being less or greater than 2.45.  Then at dept 1 the second
decision is made based on a petal width of 1.45.  The tree is depth 2 because all nodes are depth 2 or shallower.

However, as long as the Gini impurity score is not 0, the decision tree classifier can continue making more
finer grained decisions.  For example, if we set the `max_depth` to 3, we would get the following tree and decision
boundaries.

```python
# create tree classifier with max depth of 3
tree_clf = DecisionTreeClassifier(max_depth=3)
tree_clf.fit(X, y)

# export tree graph of the new tree
export_graphviz(
   tree_clf,
   out_file = '../../figures/iris_tree_depth3.dot',
   feature_names = iris.feature_names[2:],
   class_names = iris.target_names,
   rounded=True,
   filled=True)

# make figure from the .dot file graph
!dot -Tpng ../../figures/iris_tree_depth3.dot -o ../../figures/iris_tree_depth3.png
```

```python
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
h = .02  # step size in the mesh
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

Z = tree_clf.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(figsize=(12,8))
plt.contourf(xx, yy, Z, cmap=plt.cm.Pastel2, alpha=0.8);

# plot the original data
setosa_indexes = np.where(y==0)[0]
versicolor_indexes = np.where(y==1)[0]
virginica_indexes = np.where(y==2)[0]

plt.plot(X[setosa_indexes, 0], X[setosa_indexes, 1], 'yo', label='setosa')
plt.plot(X[versicolor_indexes, 0], X[versicolor_indexes, 1], 'bs', label='versicolor')
plt.plot(X[virginica_indexes, 0], X[virginica_indexes, 1], 'g^', label='virginica')
plt.xlabel('petal length')
plt.ylabel('petal width')
plt.legend();
```

Here you can see the additional decision boundary associated with the depth 2 decision.

## Model Interpretaiton: White Box vs. Black Box

As you can see from the decision boundary and tree diagrams, a decision tree classifier is fairly intuitive and
their decisions are easy to interpret.  Such models are often called *white box models*.  In contrast, models
such as SVM, or as we will see random forests are generally considered *black box models*.  They make
good predictions, and you can easily check the calculations they perform to make these predictions.  However, unlike
for decision trees, it is much harder to give a simple explanation for why a prediection was made the way it was.
Decision trees provide nice and simple classification rules that can be applied manually if needed.

# Estimating Class Probabilities

A decision tree can also estimate the probability that an instance belongs to a particular class $k$.
Basically the ratio of the training instances (the `value` parameter in the tree diagrams) of the class $k$
is an estimate of how sure or likely we are that the current decision is correct for an unknown instance.

For example, suppose you have found a flower whose petals are 5cm long and 1.5cm wide.  If we use the
original `max_depth=2` decision tree, the corresponding leaf node is the depth-2 left node, so the decision
tree should output the following probabilities: 0% Iris-Setosa (0/54), 90.7% Iris-Versicolor (49/54) and
9.3% Iris-Virginica (5/54).  And if you ask to predict the class it should output Iris-Versicolor (class 1)
since it has the highest probability.

For example, we can use the `scikit-learn` `predict()` and `predict_proba()` functions to perform these predictions.

```python
# go back to tree classifier with max depth of 2
tree_clf = DecisionTreeClassifier(max_depth=2)
tree_clf.fit(X, y)

print('predicted class    : ', tree_clf.predict([[5, 1.5]]))
print('class probabilities: ', tree_clf.predict_proba([[5, 1.5]]))
```

    predicted class    :  [1]
    class probabilities:  [[0.         0.90740741 0.09259259]]

# The CART Training Algorithm

So far we have been using trained decision tree classifiers, but have not yet discused how the tree is automatically
constructed from a given set of training data.  By inspection, you can see that basically something must pick
a feature, and pick a decision boundary on that feature, to try and make "good" decision boundaries given the
data we are trying to create a decision tree classifier for.

`scikit-learn` uses the *Classification and Regression Tree* (CART) algorithm to train decision trees
(also called "growing" trees).  The idea is relatively simple: the algorithm first splits the training set into two subsets using a single feature $k$ and a threshold on that feature $t_k$ (e.g. petal length $\le$ 2.45 cm).
The big question then is "How does it choose a good feature $k$ and threshold $t_k$?"

The CART algorithms searches for pairs $(k, t_k)$ that produce the purest subsets (weighted by their size),
where "purity" is measured by the Gini impurity measure we introduced before.  So this allows us to define
a formal cost function that we can minimise.  The cost function that the algorithm tries to minimize is:

\begin{equation}
J(k, t_k) = \frac{m_\text{left}}{m} G_\text{left} + \frac{m_\text{right}}{m} G_\text{right}
\end{equation}

where

\begin{equation}
G_{\text{left} / \text{right} } \text{measures the impurity of the left/right subset,}\\
m_{\text{left} / \text{right} } \text{is the number of instances in the left/right subset.}
\end{equation}

One it has successfully split the training set in two, it splits the subsets using the same logic,
then the sub-subsets and so on recursivly.  It stops recursing once it reaches the maximum depth, or if it cannot
find a split that will reduce the $J$ cost, which is a measure of the impurity remaining.

The CART algorithm is a *greedy algorithm*.  It greedily searches for the optimum split it can
find at the top level, then repeats the process at each depth level of the tree.  It does not check
whether or not the split will lead to the lowes possible impurity over severl levels.  A greedy algorithm
often produces a reasonably good solution, but it is not guaranteed to be the optimal solution.

Unfortunately an optimal cost function over all levels is impractical for the CART algorith, as finding
the optimal tree is known to be an *NP-Complete* problem: it requires $\mathcal{O}(e^m)$ time, e.g. it is
exponential in the number of training instances $m$, making the problem intractable even for fairly small
training sets.  This is why we must instead settle for a "reasonably good" algorithm using a greedy
split at each level rather than trying to find an overall minimal cost tree.

# Computational Complexity

Making predictions requires traversing the decision tree from root to leaf.  Decision trees are generally approximately balanced, so traversing the decision tree requires going through roughly $\mathcal{O}(\log_2 m)$
nodes.  Since each node only requires checking the value of 1 feature, the overall prediction complexity
is also just $\mathcal{O}(\log_2 m))$, independent of the number of features.  So predictions are fast, even when
dealing with large training sets.

However the training algorithm compares all features on all samples at each node.  This results in a training
complexity of $\mathcal{O}(n \times m \log m)$.  Basically to determine the best split for each node, we have to
consider all $n$ features, and consider many possible splits for each feature, computing the cost function for each,
to find an optimum.

# Gini Impurity or Entropy?

By default for `scikit-learn` the Gini impurity measure is used to calculate the cost function of the decision
tree splits.  But you can select a different also commonly used measure known as the *entropy* impurity measure
instead by setting the `criterion` hyperparameter to "entropy".

A sets entropy is zero when it contains instances of only one class, thus it is a similar concept to the Gini impurity measure.  e.g. both will be 0 when all instances in a set are of 1 class.  The entropy
of the $i^{\text{th}}$ node is given by:

\begin{equation}
H_i = -\sum_{k=1 \\ p_{i,k} \ne 0 }^n p_{i,k} \log ( p_{i,k} )
\end{equation}

Thus it is the sum of these whenever the ratio $p_{i,k}$ is not 0.  For the example depth-2 left node
we calculated before, it has an entropy impurity measure of

\begin{equation}
H_i = -\frac{49}{54} log (\frac{49}{54}) - \frac{5}{54} \log (\frac{5}{54}) \approx 0.31
\end{equation}

So which should you use regularly?  Usually it does not make a big difference, they should lead to
similar trees since the measures are similar.  Gini impurity is slightly faster to compute, so it is
a good default.  However, when they differ, the Gini impurity tends to isolate the most frequent class in
its own branch of the tree, while entropy tends to produce slightly more balanced trees (which
is a good thing).

# Regularization Hyperparameters

Decision trees make very few assumptions about the training data.  If left unconstrained, the tree structure can
adapt itself very closely to the training data, most likely badly overfitting it.  Such a model is known as
a *nonparametric model*, not because it does not have any parameters (it often has a lot), but because
the number of parameters is not determined prior to training.  This results in a training that just keeps adding
parameters (decision nodes in this case) untill the impurity is reduced to 0, meaning the classifier will perfectly
predict the training data, but is most likely badly overfit and will perform poorly on unseen data.

In contrast a *parametric model* has a predetermined number of parameters.  Linear regression, for example, is a
parametric models, the number of parameters is fixed beforehand, e.g. we assume a linear relationship, thus we
have only 2 parameters, an intercept parameter and a slope parameter to define the line we try and fit to the data.

To avoid overfitting when using decision trees, we need to restrict their freedom during training.  Decision
trees in `scikit-learn` support several regularization methods and hyperparameters to prevent this overfitting.
Limiting the maximum depth of the resulting decision tree is one example of regularizing and constraining the
model parameters.

Some other parameters you can use to regularize a decision tree in `scikit-learn` include:

- `min_samples_split` The minimum number of samples a node must have before it can be split
- `min_samples_leaf` The minum number of samples a leaf node must have
- `max_leaf_nodes` The maximum number of leaf nodes that can end up in the decision tree
- `max_features` The maximum nunber of feathres that are evaluated for splitting each node.

# Regression

Though at first glance it might not seem that decision trees can perform regression tasks, since we
are proposing discrete threshold boundaries to separate data based on features, it is actually the
case that you can also use decision trees for regression problems.  The best way to think of this is that, over
a particular range, you learn a linear approximation for a feature(s) on that range.

For example, we can train a decision tree on a noisy quadratic data set (like similar to what we used in a previous
assignment).  The `DecisionTreeRegressor` class is used for creating decision trees on regression data.

A decision tree with a `max_depth=2` looks like the following:

```python
def make_quadratic_dataset(m, a=4.0, b=-4.0, c=1.2, mu=0.0, sigma=0.15):
    """Make a randomly generated artifical quadratic regression data set.  We generate and return 2 arrays,
    the x samples, and the y regression targets.  x values are sampled randomly in the interval from
    [-1.0, 1.0].  The y regression is a cubic polynomial function of the form y = ax^3 + bx^2 + cx + d
    In addition, random gaussian noise is added to each sampled point with mean of mu, and standard deviation
    of sigma

    Parameters
    ----------
    m - Number of random samples to artificially generate (an integer value >= 1)
    a, b, c - terms of the quadratic polynomial used to generate the random data set
    mu, sigma - terms controlling the amount of random noise added to each artifically sampled point, where
      mu is the mean of the gaussian noise to generate and sigma is the standard deviation of the noise to
      be generated and added to the samples.

    Returns
    -------
    x - A NumPy array of shape (m,) (a vector).  The m samples are all in the range from [0.0, 10.0]
    y - A NumPy array of shape (m,) (a vector).  The regression targets generated by the underlying cubic
        function plus some added noise
    """
    # step 1, generate m random values for the x independent feature, in the range [-1.0, 1.0]
    x = np.random.rand(m)

    # step 2, generate y from the described cubic function and the a,b,c,d parameters
    y = a * x**2 + b * x + c

    # step 3, generate m samples of gaussian noise with a mean of mu and a standard deviation of sigma
    y = y + np.random.normal(mu, sigma, size=(m,))

    # step 4, return the x,y (independent variable, dependent variable) random sample pairs
    return x, y
```

```python
x,y = make_quadratic_dataset(250)
X = x.reshape((250,1))
```

```python
from sklearn.tree import DecisionTreeRegressor

tree_reg = DecisionTreeRegressor(max_depth=2)
tree_reg.fit(X, y)
```

    DecisionTreeRegressor(max_depth=2)

```python
plt.figure(figsize=(10,8))

# plot the raw noisy data
plt.plot(x,y, 'bo', alpha=0.25);

# plot the prediciton/decisions
X_pred = np.linspace(0.0, 1.0, 500).reshape((500,1))
y_pred = tree_reg.predict(X_pred)
plt.plot(X_pred, y_pred, 'r-', label='$\hat{y}$')
plt.title('max_depth=2')
plt.legend();
```

The model's predictions will be linear segments with decision boundaries, in this case controlled by the
`max_depth=2` parameter.  If we increase the to `max_depth=3` we can of course improve the fit.

```python
tree_reg = DecisionTreeRegressor(max_depth=3)
tree_reg.fit(X, y)
```

    DecisionTreeRegressor(max_depth=3)

```python
plt.figure(figsize=(10,8))

# plot the raw noisy data
plt.plot(x,y, 'bo', alpha=0.25);

# plot the prediciton/decisions
X_pred = np.linspace(0.0, 1.0, 500).reshape((500,1))
y_pred = tree_reg.predict(X_pred)
plt.plot(X_pred, y_pred, 'r-', label='$\hat{y}$')
plt.title('max_depth=3')
plt.legend();
```

The CART algorithm works mostly the same was as described earlier, except that instead of trying to
split the training set in a way that minimizes impurity, it now tries to split the training set in a way
that minimizes the MSE.

Just like for classification tasks, decision trees are prone to overfitting when dealing with regression tasks.  Without
any regularization (using default hyperparameters), you get predictions like the following (with no restrictions
on depth).

```python
tree_reg = DecisionTreeRegressor()
tree_reg.fit(X, y)
```

    DecisionTreeRegressor()

```python
plt.figure(figsize=(10,8))

# plot the raw noisy data
plt.plot(x,y, 'bo', alpha=0.25);

# plot the prediciton/decisions
X_pred = np.linspace(0.0, 1.0, 500).reshape((500,1))
y_pred = tree_reg.predict(X_pred)
plt.plot(X_pred, y_pred, 'r-', label='$\hat{y}$')
plt.title('No restrictions')
plt.legend();
```

If it is not obvious, this is badly overfit, with one decision now for every one of the input training samples.

If we set `min_samples_leaf=10` we get a much more reasonable regression model:

```python
tree_reg = DecisionTreeRegressor(min_samples_leaf=10)
tree_reg.fit(X, y)
```

    DecisionTreeRegressor(min_samples_leaf=10)

```python
plt.figure(figsize=(10,8))

# plot the raw noisy data
plt.plot(x,y, 'bo', alpha=0.25);

# plot the prediciton/decisions
X_pred = np.linspace(0.0, 1.0, 500).reshape((500,1))
y_pred = tree_reg.predict(X_pred)
plt.plot(X_pred, y_pred, 'r-', label='$\hat{y}$')
plt.title('min_samples_leaf=10')
plt.legend();
```

```python
import sys
sys.path.append("../../src") # add our class modules to the system PYTHON_PATH

from ml_python_class.custom_funcs import version_information
version_information()
```

                  Module   Versions
    --------------------   ------------------------------------------------------------
             matplotlib:   ['3.3.0']
                  numpy:   ['1.18.5']
                 pandas:   ['1.0.5']
                seaborn:   ['0.10.1']


## Optimization problem to be studied

We will start studying functions of multiple variables by studying unconstrained optimization problems
$$
\begin{align}
\min \quad &f(x)\\
\text{s.t.}\quad &x\in \mathbb R^n
\end{align}
$$

As an example, we study the optimization problem
$$
\begin{align}
\min \quad & (x_1-10)^2+(x_2+5)^2+x_1^2\\
\text{s.t.}\quad &x_1,x_2\in\mathbb R
\end{align}
$$
This problem is unconstrained, because there are no constraints.

Now we need to define a function in Python that evaluates that function. That is, we define a two-variable function
$$f:(x_1,x_2)\to (x_1-10)^2+(x_2+5)^2+x_1^2$$

```python
def f_simple(x):
    return (x[0] - 10.0)**2 + (x[1] + 5.0)**2+x[0]**2
```

Now we can call that function

```python
print "At point (5,-5) the value of the function is ",f_simple([5,-5])
```

    At point (5,-5) the value of the function is  50.0

We can also plot that function

```python
%matplotlib
import numpy as np
from pylab import meshgrid
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

def plot_2d_function(lb1,lb2,ub1,ub2,f):
    x = np.arange(lb1,ub1,0.1)
    y = np.arange(lb2,ub2,0.1)
    X,Y = meshgrid(x, y) # grid of point
    Z = [f([x,y]) for (x,y) in zip (X,Y)] # evaluation of the function on the grid
    fig = plt.figure()
    ax = fig.gca(projection='3d')
    surf = ax.plot_surface(X, Y, Z)
    return plt
```

    Using matplotlib backend: Qt4Agg

```python
plot_2d_function(3,-7,7,-3,f_simple)
```

    /usr/lib64/python2.7/site-packages/matplotlib/collections.py:590: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
      if self._edgecolors == str('face'):

    <module 'matplotlib.pyplot' from '/usr/lib64/python2.7/site-packages/matplotlib/pyplot.pyc'>

# Direct search methods

Direct search methods (also called pattern search methods) just rely on the function values to find a (local) optimum. Direct search methods consist of a set of
1. exploratory moves that acquire information about the function $f$ in th eneighbourhood of the current solution, and
2. pattern moves that attempt to speed up the search using the information acquired in the exploratory moves.

## Hooke&Jeeves

**input:** a minimum step length $L>0$, initial step length $\epsilon_0$, constant $0<\delta<1$ for reducing the step length, exploratory step multiplier $\gamma>1$, and starting solution $x_0$
**output:** an approximation of a local optimum (no guarantees of quality in general cases)
```
set epsilon as the initial step length epsilon0
set x as the starting solution
while epsilon is greater than L:
    for each coordinate direction i:
        find the smallest of function values by incrementing and reducing the variable value in that coordinate by epsilon, let this value be xi*
    if x is the same as (x1*,...,xn*):
        reduce epsilon to delta*epsilon
    else:
        if the value of f at (x1*,...,xn*) is smaller than at x+gamma*((x1*,...,xn*)-x):
            set x as (x1*,...xn*)
        else:
            set x as x+gamma*((x1*,...,xn*)-x)
return x

```
Thus, the exploratory step of Hooke&Jeeves is performed by incrementing and reducing the variable to each coordinate direction and the pattern move is just a multiplication of the exploratory move.

```python
import copy #Copying vectors
import numpy as np #Import vector calculus and much more!
def hookejeeves(L,epsilon0,delta,gamma,x0,f):
    epsilon = epsilon0
    x = np.array(x0)
    while epsilon>L:
        xtest = np.zeros(len(x))
        for coordinate in range(len(x)):
            exp_points = [copy.copy(x) for _ in range(3)] #points to be explored
            exp_points[0][coordinate]-=epsilon
            exp_points[1][coordinate]+=epsilon
            f_exp_points = [f(exp_point) for exp_point in exp_points]
            min_value = min(f_exp_points)
            xtest[coordinate] = exp_points[f_exp_points.index(min_value)][coordinate] #The coordinate value is the one where the minimum is attained
        if all(xtest==x):
            epsilon = delta*epsilon
        else:
            if f(xtest)<f(x+gamma*(xtest-x)):
                x = xtest
            else:
                x = x+gamma*(xtest-x)
    return x
```

```python
%timeit hookejeeves(0.001,1,0.1,2,[0,0],f_simple)
```

    10000 loops, best of 3: 172 µs per loop

## Powell's method

Powell's method is similar to Hooke&Jeeves, but the first step in exploratory moves is taken to the direction of the last pattern move. This speeds up the convergence in most cases.

```python
import copy
import numpy as np
def powell(L,epsilon0,delta,gamma,x0,f):
    epsilon = epsilon0
    exp_direction = np.array([0,1])
    x = np.array(x0)
    while epsilon>L:
        xtest = np.zeros(len(x))
        exp_direction2 = np.array([exp_direction[1],-exp_direction[0]]) #Only works in 2d!!
        exp_points = [x,x+exp_direction,x-exp_direction,x+exp_direction2,x-exp_direction2]
        f_exp_points = [f(exp_point) for exp_point in exp_points]
        min_value = min(f_exp_points)
        xtest = exp_points[f_exp_points.index(min_value)] #Comparing among all exploratory points
        if all(xtest==x):
            epsilon = delta*epsilon
        else:
            if f(xtest)<f(x+gamma*(xtest-x)):
                x = xtest
            else:
                x = x+gamma*(xtest-x)
                exp_direction = (xtest-x)/np.linalg.norm(xtest-x)
    return x
```

```python
%timeit powell(0.001,1,0.1,2,[0,0],f_simple)
```

    1000 loops, best of 3: 210 µs per loop

Lecture 3, Direct search methods; Hooke&Jeeves and Powell's method.ipynb


# First example

## Learn : Linear Model for NOT

$f^{*}(0) = 1$
$f^{*}(1) = 0$

No activation function this time.

### Diagram:

### Function:

$$f(x, w, b) = w \cdot x + b$$

### Error function:
$$MSE=\frac{1}{n}\sum_{x=0}^{1}{(f(x, w, b) - f^*(x))^2}$$
$n$ is equal to the number of entries, in our case: 2

**Objective:** we want to minimize this error function

**How:** With calculus, finding the derivative for minimizing the error

### Mean Squared Error (MSE):

$$ MSE = \frac{1}{2} \cdot (\,f(0, w, b) - f^*(0)\,)^2 + \frac{1}{2} \cdot (\,f(1, w, b) - f^*(1)\,)^2  $$
$$ MSE = \frac{1}{2} \cdot (\,b-1\,)^2 + \frac{1}{2} \cdot (\,w+b - 0\,)^2  $$
$$ MSE = b^2 - b + \frac{1}{2} + \frac{w^2}{2} + w \cdot b $$

### Partial derivatives

Now, calculating the partial derivatives to find the gradient descedent:

 $$\frac{\partial MSE}{\partial w} = w + b$$

 $$\frac{\partial MSE}{\partial b} = 2 \cdot b - 1 + w$$

### Iteration:
$$w \leftarrow w- \eta \cdot \frac{\partial MSE}{\partial w}$$

$$b \leftarrow b- \eta \cdot \frac{\partial MSE}{\partial b}$$

```python
import random
import itertools

# generic mse
def mse(f, f_star, w, b, n):
    return sum([(f(x, w, b) - f_star(x)) ** 2 for x in itertools.product([0, 1], repeat=n)])
```

```python
def not_func(x):
    if x[0] == 0:
        return 1
    if x[0] == 1:
        return 0

    raise ValueError('Incorrect input, it should be 0 or 1')

def not_model(x, w, b):
    return w * x[0] + b

def grad_w(w, b, eta):
    return w - eta * (w + b)

def grad_b(w, b, eta):
    return b - eta * (2 *
                      b - 1 + w)

def train_not(model, objective, n, num_interactions=500, eta=0.1):
    w = random.uniform(-1, 1)
    b = random.uniform(-1, 1)

    j = 0
    for i in range(num_interactions):
        mse_value = mse(model, objective, w, b, n)
        w = grad_w(w, b, eta)
        b = grad_b(w, b, eta)

        if j == 0:
            print(mse_value)
        elif j == 10:
            j = -1

        j += 1

    def trained_model(x):
        return model(x, w, b)

    return trained_model
```

```python
trained_model = train_not(not_model, not_func, n=2)
```

    3.1273125908836406
    0.09097695296660298
    0.03567994424746805
    0.014551440351822587
    0.0059280827849091285
    0.0024148165342546637
    0.0009836745834271108
    0.0004006993048506026
    0.00016322463896691278
    6.648946578546263e-05
    2.7084446857500127e-05
    1.1032834343084863e-05
    4.494218925064842e-06
    1.8307175761297194e-06
    7.457417850426648e-07
    3.037775008061611e-07
    1.2374359576856902e-07
    5.040688481898796e-08
    2.0533216457561165e-08
    8.364194288274525e-09
    3.407149885006487e-09
    1.387900608083934e-09
    5.653605397308239e-10
    2.3029930099176767e-10
    9.381229199556604e-11
    3.821438489630135e-11
    1.556660840372668e-11
    6.341049263512728e-12
    2.583022885592576e-12
    1.052192934008835e-12
    4.2861020560916524e-13
    1.7459412849918662e-13
    7.112082098637322e-14
    2.8971026821594625e-14
    1.1801331653804728e-14
    4.807265885281481e-15
    1.9582370865126027e-15
    7.976867920247288e-16
    3.249372720017398e-16
    1.3236301388442732e-16
    5.3918000378717415e-17
    2.1963467976054934e-17
    8.946807103117877e-18
    3.644476357878606e-18
    1.484575451602423e-18
    6.047410942713969e-19

```python
for x in itertools.product([0, 1], repeat=1):
    print('input: {}, output: {}'.format(x, trained_model(x)))
```

    input: (0,), output: 0.9999999996269427
    input: (1,), output: 2.4870483450456504e-10

# Second example: Linear Model for XOR
## Learn XOR

 x1 | x2 | XOR
----|----|----
 0  | 0  | 0
 0  | 1  | 1
 1  | 0  | 1
 1  | 1  | 0

*No activation function.*

### Diagram:

### Function:

$$ f(x, w, b) = w_1 \cdot x_1 + w_2 \cdot x_2 + b$$

### MSE:

$$ MSE = \frac{1}{4} \cdot (b^2 + (w_2+1-1)^2+(w_1+b-1)+(w_1+b-1)^2+(w_1+w_2+b)^2)$$

### Partial derivatives:

$$\frac{\partial MSE}{\partial w_1} = w_1 + b - \frac{1}{2} + \frac{w_2}{2}$$

$$\frac{\partial MSE}{\partial w_2} = w_2 + b - \frac{1}{2} + \frac{w_1}{2}$$

$$\frac{\partial MSE}{\partial b} = 2b + w_1 + w_2 - 1$$

```python
def xor_func(x):
    return int(x[0] != x[1])

def xor_model(x, w, b):
    return x[0] * w[0] + x[1] * w[1] + b

def grad_w_1(w, b, eta):
    return w[0] - eta * (w[0] + b - 0.5 + 0.5 * w[1])

def grad_w_2(w, b, eta):
    return w[1] - eta * (w[1] + b - 0.5 + 0.5 * w[0])

def grad_b(w, b, eta):
    return b - eta * (2 * b + w[0] + w[1] - 1)

def train_xor(model, objective, n, num_interactions=500, eta=0.1):
    w = [random.uniform(-1, 1) for _ in range(2)]
    b = random.uniform(-1, 1)

    j = 0
    for i in range(num_interactions):
        mse_value = mse(model, objective, w, b, n)
        w[0] = grad_w_1(w, b, eta)
        w[1] = grad_w_2(w, b, eta)
        b = grad_b(w, b, eta)

        if j == 0:
            print(mse_value)
        elif j == 10:
            j = -1

        j += 1

    def trained_model(x):
        return model(x, w, b)

    return trained_model
```

```python
trained_model = train_xor(xor_model, xor_func, n=4)
```

    74.21251615645126
    4.416213022532111
    4.138188488536622
    4.050881249927394
    4.019597506970198
    4.00790416439515
    4.003325607279144
    4.001449798328364
    4.000649764838388
    4.000297169776201
    4.000137846871886
    4.000064553535356
    4.00003041838905
    4.000014389975611
    4.00000682395414
    4.000003240696449
    4.000001540265461
    4.000000732383601
    4.000000348308851
    4.000000165658091
    4.000000078786001
    4.00000003746763
    4.0000000178166735
    4.000000008471465
    4.000000004027666
    4.000000001914761
    4.0000000009102195
    4.000000000432664
    4.000000000205652
    4.000000000097744
    4.000000000046457
    4.000000000022078
    4.000000000010494
    4.000000000004986
    4.00000000000237
    4.000000000001125
    4.000000000000535
    4.000000000000253
    4.000000000000121
    4.000000000000057
    4.0000000000000275
    4.000000000000013
    4.000000000000007
    4.0000000000000036
    4.000000000000002
    4.000000000000001

```python
for x in itertools.product([0, 1], repeat=2):
    print('input: {}, output: {}'.format(x, trained_model(x)))
```

    input: (0, 0), output: 0.4999999913242938
    input: (0, 1), output: 0.4999999991588299
    input: (1, 0), output: 0.4999999984537919
    input: (1, 1), output: 0.5000000062883281

# Improved model for XOR

### Activation function: ReLU
$$\sigma(x) = max(0, x)$$

### Derivative:
\begin{cases}
    x,& \text{if } x\geq 1\\
    0,              & \text{otherwise}
\end{cases}

### MSE:

$$ MSE = \frac{1}{4} \sum_x (f^{(2)}(f^{(1)}(x)) - f^{*}(x))^2$$

$$ MSE = \frac{1}{4} \sum_x (w_1 \cdot f^{(1)}(x_1) + w_2 \cdot f^{(1)}(x_2) + b - f^{*}(x))^2$$

$$ MSE = \frac{1}{4} \sum_{x_1=0}^{1} \sum_{x_2=0}^{1} (w_1 \cdot max(0, w_{1,1}x_1 + w_{1,2}x_2 + b_1) + w_2 \cdot max(0, w_{2,1}x_1 + w_{2,2}x_2 + b_2) + b - f^{*}(x))^2$$

```python
import sympy
import random
sympy.init_printing(use_latex='mathjax')

class Xor(sympy.Function):
    @classmethod
    def eval(cls, x1, x2):
        if x1.is_Number and x2.is_Number:
            return int(x1 != x2)

class Var:
    def __init__(self, symbol, value):
        self.s = sympy.Symbol(symbol)
        self.v = value

    @staticmethod
    def symbols(variables):
        return (Var(symbol, random.uniform(-1, 1)) for symbol in variables.split(' '))

    def set_diff(self, expr):
        self.diff = sympy.diff(expr, self.s)

    def set_rand(self):
        self.v = random.uniform(-1, 1)

def subs(expr, variables):
    return expr.subs([
        (var.s, var.v) for var in variables
    ])

def diff(variables, eta):
    for var in variables:
        var.v -= eta * subs(var.diff, variables)
```

```python
variables = list(Var.symbols('x_1 x_2 w_1 w_2 w_11 w_12 w_21 w_22 b_1 b_2 b'))
x_1, x_2, w_1, w_2, w_11, w_12, w_21, w_22, b_1, b_2, b = variables

model = (
    w_1.s * sympy.Max(0, w_11.s * x_1.s + w_12.s * x_2.s + b_1.s) +
    w_2.s * sympy.Max(0, w_21.s * x_1.s + w_22.s * x_2.s + b_2.s) + b.s
)
expr = (model - Xor(x_1.s, x_2.s)) ** 2
expr = sympy.Rational(1, 4) * sympy.Sum(sympy.Sum(expr, (x_1.s, 0, 1)), (x_2.s, 0, 1))

expr
```

$$\frac{1}{4} \sum_{\substack{0 \leq x_{1} \leq 1\\0 \leq x_{2} \leq 1}} \left(b + w_{1} \max\left(0, b_{1} + w_{11} x_{1} + w_{12} x_{2}\right) + w_{2} \max\left(0, b_{2} + w_{21} x_{1} + w_{22} x_{2}\right) - x_{1} \veebar x_{2}\right)^{2}$$

```python
# do the sum and simplify
expr = expr.doit().simplify()
expr
```

$$\frac{1}{4} \left(b + w_{1} \max\left(0, b_{1}\right) + w_{2} \max\left(0, b_{2}\right)\right)^{2} + \frac{1}{4} \left(b + w_{1} \max\left(0, b_{1} + w_{11} + w_{12}\right) + w_{2} \max\left(0, b_{2} + w_{21} + w_{22}\right)\right)^{2} + \frac{1}{4} \left(b + w_{1} \max\left(0, b_{1} + w_{11}\right) + w_{2} \max\left(0, b_{2} + w_{21}\right) - 1\right)^{2} + \frac{1}{4} \left(b + w_{1} \max\left(0, b_{1} + w_{12}\right) + w_{2} \max\left(0, b_{2} + w_{22}\right) - 1\right)^{2}$$

```python
# set variables derivatives
for var in variables:
    var.set_diff(expr)
```

```python
# train based on number of iterations with learning rate eta
def train(iterations=500, eta=0.05):
    for var in variables:
        var.set_rand()

    j = 0
    for _ in range(iterations):
        mse = subs(expr, variables)
        diff(variables, eta)

        if j == 0:
            print(mse)
        elif j == 10:
            j = -1

        j += 1

    def trained_model(x):
        return (
            w_1.v * max(0, w_11.v * x[0] + w_12.v * x[1] + b_1.v) +
            w_2.v * max(0, w_21.v * x[0] + w_22.v * x[1] + b_2.v) + b.v
        )

    return trained_model
```

```python
trained_model = train()
```

    0.265646446065377
    0.224858254011097
    0.206991861276136
    0.190704149182786
    0.181199984040078
    0.174393451491669
    0.167328360539738
    0.163526028868485
    0.154924305564428
    0.146407625120289
    0.134448533946257
    0.121845513677617
    0.110361501188768
    0.0983061437253321
    0.0861120076772425
    0.0736537068454568
    0.0617838854835395
    0.0503109833076007
    0.0381953811393830
    0.0281041646397589
    0.0209350569412672
    0.0154467266138095
    0.0111496916915322
    0.00778730172723044
    0.00530334174726595
    0.00376824501705862
    0.00258942186143672
    0.00189570651881333
    0.00126690378478501
    0.000820948277708715
    0.000578653842596901
    0.000397286773700105
    0.000276810189056663
    0.000181024255154999
    0.000117368107852272
    8.05656907870844e-5
    5.49147629686018e-5
    3.69530191406392e-5
    2.36920184911874e-5
    1.63506687844512e-5
    1.13638812082042e-5
    7.30898060985584e-6
    4.78150729183676e-6
    3.16891761858790e-6
    2.18022378660339e-6
    1.48938479073708e-6

```python
for x in itertools.product([0, 1], repeat=2):
    print('input: {}, output: {}'.format(x, trained_model(x)))
```

    input: (0, 0), output: 0.000702514318965952
    input: (0, 1), output: 0.998703406850084
    input: (1, 0), output: 0.998724427574680
    input: (1, 1), output: 0.00102738435601935


# Question:

why `ConvTranspose2d` can be seen as the gradient of `Conv2d` with respect to its input?

# Answer:

see blog article: [Blog Link](https://simonjisu.github.io/datascience/2019/10/27/convtranspose2d.html)

## Reference:
- https://arxiv.org/abs/1603.07285
- https://en.wikipedia.org/wiki/Toeplitz_matrix

```python
import sympy
from sympy import Symbol, MatrixSymbol, Matrix
import numpy as np

def convolution(x, w, K, N):
    res = []
    for i in range(N-K+1):
        for j in range(N-K+1):
            a = sum(Matrix(x)[i:(K+i), j:(K+j)].multiply_elementwise(Matrix(w)))
            res.append(a)
    return Matrix(res).reshape(N-K+1, N-K+1)
```

```python
x_input = MatrixSymbol("x^{(l)}", 4, 4)
x_output = MatrixSymbol("x^{(l+1)}", 2, 2)
w = MatrixSymbol("w", 3, 3)
```

```python
o = convolution(x_input, w, K=3, N=4)
o
```

$\displaystyle \left[\begin{matrix}w_{0, 0} x^{(l)}_{0, 0} + w_{0, 1} x^{(l)}_{0, 1} + w_{0, 2} x^{(l)}_{0, 2} + w_{1, 0} x^{(l)}_{1, 0} + w_{1, 1} x^{(l)}_{1, 1} + w_{1, 2} x^{(l)}_{1, 2} + w_{2, 0} x^{(l)}_{2, 0} + w_{2, 1} x^{(l)}_{2, 1} + w_{2, 2} x^{(l)}_{2, 2} & w_{0, 0} x^{(l)}_{0, 1} + w_{0, 1} x^{(l)}_{0, 2} + w_{0, 2} x^{(l)}_{0, 3} + w_{1, 0} x^{(l)}_{1, 1} + w_{1, 1} x^{(l)}_{1, 2} + w_{1, 2} x^{(l)}_{1, 3} + w_{2, 0} x^{(l)}_{2, 1} + w_{2, 1} x^{(l)}_{2, 2} + w_{2, 2} x^{(l)}_{2, 3}\\w_{0, 0} x^{(l)}_{1, 0} + w_{0, 1} x^{(l)}_{1, 1} + w_{0, 2} x^{(l)}_{1, 2} + w_{1, 0} x^{(l)}_{2, 0} + w_{1, 1} x^{(l)}_{2, 1} + w_{1, 2} x^{(l)}_{2, 2} + w_{2, 0} x^{(l)}_{3, 0} + w_{2, 1} x^{(l)}_{3, 1} + w_{2, 2} x^{(l)}_{3, 2} & w_{0, 0} x^{(l)}_{1, 1} + w_{0, 1} x^{(l)}_{1, 2} + w_{0, 2} x^{(l)}_{1, 3} + w_{1, 0} x^{(l)}_{2, 1} + w_{1, 1} x^{(l)}_{2, 2} + w_{1, 2} x^{(l)}_{2, 3} + w_{2, 0} x^{(l)}_{3, 1} + w_{2, 1} x^{(l)}_{3, 2} + w_{2, 2} x^{(l)}_{3, 3}\end{matrix}\right]$

```python
C = o.reshape(1, 4).diff(Matrix(x_input).reshape(16, 1)).reshape(16, 4)
C
```

$\displaystyle \left[\begin{matrix}w_{0, 0} & 0 & 0 & 0\\w_{0, 1} & w_{0, 0} & 0 & 0\\w_{0, 2} & w_{0, 1} & 0 & 0\\0 & w_{0, 2} & 0 & 0\\w_{1, 0} & 0 & w_{0, 0} & 0\\w_{1, 1} & w_{1, 0} & w_{0, 1} & w_{0, 0}\\w_{1, 2} & w_{1, 1} & w_{0, 2} & w_{0, 1}\\0 & w_{1, 2} & 0 & w_{0, 2}\\w_{2, 0} & 0 & w_{1, 0} & 0\\w_{2, 1} & w_{2, 0} & w_{1, 1} & w_{1, 0}\\w_{2, 2} & w_{2, 1} & w_{1, 2} & w_{1, 1}\\0 & w_{2, 2} & 0 & w_{1, 2}\\0 & 0 & w_{2, 0} & 0\\0 & 0 & w_{2, 1} & w_{2, 0}\\0 & 0 & w_{2, 2} & w_{2, 1}\\0 & 0 & 0 & w_{2, 2}\end{matrix}\right]$

```python
# forward
(Matrix(x_input).reshape(1, 16) @ C).reshape(2, 2)
```

$\displaystyle \left[\begin{matrix}w_{0, 0} x^{(l)}_{0, 0} + w_{0, 1} x^{(l)}_{0, 1} + w_{0, 2} x^{(l)}_{0, 2} + w_{1, 0} x^{(l)}_{1, 0} + w_{1, 1} x^{(l)}_{1, 1} + w_{1, 2} x^{(l)}_{1, 2} + w_{2, 0} x^{(l)}_{2, 0} + w_{2, 1} x^{(l)}_{2, 1} + w_{2, 2} x^{(l)}_{2, 2} & w_{0, 0} x^{(l)}_{0, 1} + w_{0, 1} x^{(l)}_{0, 2} + w_{0, 2} x^{(l)}_{0, 3} + w_{1, 0} x^{(l)}_{1, 1} + w_{1, 1} x^{(l)}_{1, 2} + w_{1, 2} x^{(l)}_{1, 3} + w_{2, 0} x^{(l)}_{2, 1} + w_{2, 1} x^{(l)}_{2, 2} + w_{2, 2} x^{(l)}_{2, 3}\\w_{0, 0} x^{(l)}_{1, 0} + w_{0, 1} x^{(l)}_{1, 1} + w_{0, 2} x^{(l)}_{1, 2} + w_{1, 0} x^{(l)}_{2, 0} + w_{1, 1} x^{(l)}_{2, 1} + w_{1, 2} x^{(l)}_{2, 2} + w_{2, 0} x^{(l)}_{3, 0} + w_{2, 1} x^{(l)}_{3, 1} + w_{2, 2} x^{(l)}_{3, 2} & w_{0, 0} x^{(l)}_{1, 1} + w_{0, 1} x^{(l)}_{1, 2} + w_{0, 2} x^{(l)}_{1, 3} + w_{1, 0} x^{(l)}_{2, 1} + w_{1, 1} x^{(l)}_{2, 2} + w_{1, 2} x^{(l)}_{2, 3} + w_{2, 0} x^{(l)}_{3, 1} + w_{2, 1} x^{(l)}_{3, 2} + w_{2, 2} x^{(l)}_{3, 3}\end{matrix}\right]$

```python
# backward
(Matrix(x_output).reshape(1, 4) @ C.transpose()).reshape(4, 4)
```

$\displaystyle \left[\begin{matrix}w_{0, 0} x^{(l+1)}_{0, 0} & w_{0, 0} x^{(l+1)}_{0, 1} + w_{0, 1} x^{(l+1)}_{0, 0} & w_{0, 1} x^{(l+1)}_{0, 1} + w_{0, 2} x^{(l+1)}_{0, 0} & w_{0, 2} x^{(l+1)}_{0, 1}\\w_{0, 0} x^{(l+1)}_{1, 0} + w_{1, 0} x^{(l+1)}_{0, 0} & w_{0, 0} x^{(l+1)}_{1, 1} + w_{0, 1} x^{(l+1)}_{1, 0} + w_{1, 0} x^{(l+1)}_{0, 1} + w_{1, 1} x^{(l+1)}_{0, 0} & w_{0, 1} x^{(l+1)}_{1, 1} + w_{0, 2} x^{(l+1)}_{1, 0} + w_{1, 1} x^{(l+1)}_{0, 1} + w_{1, 2} x^{(l+1)}_{0, 0} & w_{0, 2} x^{(l+1)}_{1, 1} + w_{1, 2} x^{(l+1)}_{0, 1}\\w_{1, 0} x^{(l+1)}_{1, 0} + w_{2, 0} x^{(l+1)}_{0, 0} & w_{1, 0} x^{(l+1)}_{1, 1} + w_{1, 1} x^{(l+1)}_{1, 0} + w_{2, 0} x^{(l+1)}_{0, 1} + w_{2, 1} x^{(l+1)}_{0, 0} & w_{1, 1} x^{(l+1)}_{1, 1} + w_{1, 2} x^{(l+1)}_{1, 0} + w_{2, 1} x^{(l+1)}_{0, 1} + w_{2, 2} x^{(l+1)}_{0, 0} & w_{1, 2} x^{(l+1)}_{1, 1} + w_{2, 2} x^{(l+1)}_{0, 1}\\w_{2, 0} x^{(l+1)}_{1, 0} & w_{2, 0} x^{(l+1)}_{1, 1} + w_{2, 1} x^{(l+1)}_{1, 0} & w_{2, 1} x^{(l+1)}_{1, 1} + w_{2, 2} x^{(l+1)}_{1, 0} & w_{2, 2} x^{(l+1)}_{1, 1}\end{matrix}\right]$

```python

```


```python
import numpy as np
from sympy import *
from mpmath import mp

import matplotlib.pyplot as plt
```

## Problem 1

```python
x, y = symbols('x y')
```

```python
eq = Eq(y, sympify('ln(x)'))
eq
```

    Eq(y, log(x))

```python
x = solve(eq, x)[0]
x
```

    exp(y)

```python
plot(x, (y, 1, 2), xlim=(0, 3), ylim=(0, 10))
```

```python
ans = integrate(mp.pi*x**2, (y, 1, 2))
ans
```

    -1.5707963267949*exp(2) + 1.5707963267949*exp(4)

```python
ans.evalf()
```

    74.1558713431779

### Arrived Ans:

```python
(mp.pi*(exp(4)-exp(2))/2).evalf()
```

    74.1558713431779

## Problem 2

```python
x, y0, y1, pi = symbols('x y0 y1 pi')
```

```python
y0 = 6 - x**2
y0
```

    -x**2 + 6

```python
y1 = 2
y1
```

    2

```python
eq = Eq(y0, y1)
eq
```

    Eq(-x**2 + 6, 2)

```python
bounds = solve(eq, x)
bounds
```

    [-2, 2]

```python
plot((y0, (x, *bounds)), (y1, (x, *bounds)), xlim=(-3, 3), ylim=(0, 10))
```

```python
ans = integrate(pi*(y0**2-y1**2), (x, -2, 2))
ans
```

    384*pi/5

```python
ans.subs(pi, mp.pi)
```

    241.274315795696

## Problem 3

```python
x, y0, y1, pi, r0, r1 = symbols('x y0 y1 pi r0 r1')
```

```python
y0 = x
y0
```

    x

```python
y1 = x**2
y1
```

    x**2

```python
eq = Eq(y0, y1)
solve(eq, x)
```

    [0, 1]

```python
plot(y0, y1, xlim=(-1, 3), ylim=(0, 4))
```

```python
r0 = 2 - y0
r0
```

    -x + 2

```python
r1 = 2 - y1
r1
```

    -x**2 + 2

```python
(r1**2-r0**2).simplify()
```

    x*(x**3 - 5*x + 4)

```python
integrate(pi*(r1**2-r0**2), x)
```

    pi*x**5/5 - 5*pi*x**3/3 + 2*pi*x**2

```python
integrate(pi*(r1**2-r0**2), (x, 0, 1))
```

    8*pi/15

```python
integrate(pi*(r0**2-r1**2), x).subs(x, 1)
```

    -8*pi/15

```python
integrate(pi*(r0**2-r1**2), (x, 1, sqrt(2)))
```

    -52*pi/15 + 38*sqrt(2)*pi/15

```python
integrate(pi*r0**2, x).subs(x, sqrt(2))
```

    -4*pi + 14*sqrt(2)*pi/3

```python
ans = integrate(pi*(r1**2-r0**2), (x, 0, 1)) + integrate(pi*(r0**2-r1**2), (x, 1, sqrt(2))) + integrate(pi*r0**2, (x, sqrt(2), 2))
ans
```

    -32*sqrt(2)*pi/15 + 56*pi/15

```python
ans.subs(pi, mp.pi).evalf()
```

    2.25046230533071

```python

```


# A Simple Illustration of a 2-Dimensional Groundwater Model
## By: Lejo Flores

This is an example of a 2-dimensional groundwater model in which the domain is square, heterogeneous, and isotropic (i.e., there is one value of hydraulic conductivity $K_{sat}$ that is valid everywhere in the domain and in all directions). The purpose of this Jupyter notebook is to illustrate how solution of the groundwater flow equations (conservation of mass and Darcy's Law) is largely an extension of the previously reviewed 1D examples into a second dimension.

Conceptually this model is really quite simple - in addition to gradients in pressure head in the $x$ direction we also now need to compute pressure head gradients in the $y$ direction. We will reduce this problem to a suite of discrete linear relationships and use linear algebra, matrix-vector multiplication to solve the system of equations. This requires us to represent, or "flatten," our 2D domain into a column vector. However, where this introduces complications in the numerics is in the "bookkeeping" of keeping track of where in our column vector the cells that were north and south of a central pixel now are after we "flatten" our domain. I will actually illustrate how this works out using a "100s chart" from my daughters 1st grade classroom, because that's the simplist way to figure out how this works in a general way.

## A Review of the Maths

\begin{equation}
\frac{\partial^2h}{\partial x^2}+\frac{\partial^2h}{\partial y^2}=\frac{S}{T}\frac{\partial h}{\partial t} - \frac{R(x,y,t)}{T}
\end{equation}

```python
import numpy as np
import matplotlib.pyplot as plt

hi = 80.0
hf = 10.0
K = 16.0

ti = 0.0
tf = 500.0
dt = 0.1

OutputInterval = 50.0

Nt = (tf - ti)/dt + 1

Noutput = (tf - ti)/OutputInterval + 1

xMax = yMax = 500.0
xMin = yMin = 0.0
Nx = Ny = 50

x = np.linspace(xMin,xMax,Nx)
y = np.linspace(yMin,yMax,Ny)

dx = x[1] - x[0]
dy = y[1] - y[0]

X, Y = np.meshgrid(x,y)
```

## Plot the X and Y grid meshes

```python
# Plot X grid
plt.figure(1,figsize=(5,3), dpi=150)
plt.title("X grid $(X_{i,j})_{i,j}$")
plt.xlabel(r"$j$")
plt.ylabel(r"$i$")
plt.imshow(X,cmap="jet")
plt.colorbar()
plt.show()

# Plot Y grid
plt.figure(2,figsize=(5,3), dpi=150)
plt.title("Y grid $(Y_{i,j})_{i,j}$")
plt.xlabel(r"$j$")
plt.ylabel(r"$i$")
plt.imshow(Y,cmap="jet")
plt.colorbar()
plt.show()

```

## Compute the stability variable

```python
alpha = K*dt/dx**2
print("alpha = ",alpha)
print("dt = ",dt)
print("dx = ",dx)
if alpha >= 0.5:
    raise Exception("Stability criteria error! Either decrease dt or increase dx!")
```

    alpha =  0.0153664
    dt =  0.1
    dx =  10.2040816327

## Set up the solver matrix

```python
N = Nx*Ny

ind = np.linspace(0,N-1,N)
Ind = np.linspace(0,N-1,N).reshape((Ny,Nx))

Delta2 = np.matrix(np.diagflat(-4.0*np.ones((int(N),1)),k=0) + np.diagflat(np.ones((int(N-1),1)),k=1) \
    + np.diagflat(np.ones((int(N-1),1)),k=-1) + np.diagflat(np.ones((int(N-Nx),1)),k=int(Nx)) \
    + np.diagflat(np.ones((int(N-Ny),1)),k=-(int(Ny))))

plt.figure(3,figsize=(6,4), dpi=150)
plt.title("Incidence matrix")
plt.imshow(Delta2[0:(2*int(Ny)),0:(2*int(Nx))],cmap="seismic")
plt.colorbar()
plt.show()
```

## Setting up the boundary conditions

```python
NBound = np.array(Ind[0,:])
SBound = np.array(Ind[-1,:])
WBound = np.array(Ind[:,0])
EBound = np.array(Ind[:,-1])

hNEt = np.linspace(hi,hf,num=Nt,endpoint=True)

# Now compute the boundary conditions on the North and East side of the domain as the temporal evolution of a !D
# profile in response to a lowering of head on 1-side of the domain

Delta2_1D = np.matrix(np.diagflat(-2.0*np.ones((int(Nx),1)),k=0) + np.diagflat(np.ones((int(Nx-1),1)),k=1) \
    + np.diagflat(np.ones((int(Nx-1),1)),k=-1))

h0_1D = hi*np.ones(shape=(int(Nx),1))

H_1D = np.matrix(np.zeros(shape=(int(Nx),int(Nt))))

H_1D[:,0] = np.matrix(h0_1D)

for i in range(1,int(Nt)):
    hinit_1D = H_1D[:,i-1]

    hnext_1D = hinit_1D + alpha*Delta2_1D*hinit_1D
    hnext_1D[0] = hi
    hnext_1D[-1] = hNEt[i]

    H_1D[:,i] = hnext_1D[:,0]

```

## Running the model

```python
import matplotlib as mpl

h0 = hi*np.ones(shape=(int(N),1))
hnext = np.matrix(h0)

figcounter = 1

for i in range(1,int(Nt)):
    hinit = hnext

    hnext = hinit + alpha*Delta2*hinit
    hnext[NBound.astype(int)] = H_1D[:,-1]
    hnext[EBound.astype(int)] = np.flipud(H_1D[:,-1])
    hnext[SBound.astype(int)] = hi
    hnext[WBound.astype(int)] = hi

    if((i*dt) % OutputInterval == 0):
        H = np.reshape(hnext,(Ny,Nx))
        fig = plt.figure(figcounter, figsize=(3,3), dpi=150)
        plt.title("Pressure head $h_{i,j}$ at t = %s"%(i*dt), fontsize=9)
        plt.xlabel('$x$', fontsize=9)
        plt.ylabel('$y$', fontsize=9)
        plt.xticks(np.arange(xMin,xMax+1.0,(xMax-xMin)/5))
        plt.yticks(np.arange(yMin,yMax+1.0,(yMax-yMin)/5))
        mpl.rcParams['xtick.labelsize'] = 9
        mpl.rcParams['ytick.labelsize'] = 9
        plt.imshow(H,cmap="jet",extent=[xMin,xMax,yMin,xMax],vmin=hf,vmax=hi)
        cb = plt.colorbar()
        cb.ax.tick_params(labelsize=9)
        plt.show()
        figcounter = figcounter + 1

```

```python

```


```python
# Models from Introduction to Algorithmic Marketing
# https://algorithmicweb.wordpress.com/
#
# Latent Semantic Analysis (LSA) is a method of text analysis
# that helps to identify concepts represented in the text as related words
# In this example, we both calculate document representation in the
# concept space and score documents againt the query using a distance metric in
# this space
```

```python
%matplotlib inline
import sympy as sy
import numpy as np
import matplotlib.pyplot as plt
from itertools import chain
from tabulate import tabulate
from collections import Counter

def tabprint(msg, A):
    print(msg)
    print(tabulate(A, tablefmt="fancy_grid"))
```

```python
docs = [
"chicago chocolate retro candies made with love",
"chocolate sweets and candies collection with mini love hearts",
"retro sweets from chicago for chocolate lovers"]
```

```python
# Basic analyzer:
# - split documents into words
# - remove stop words
# - apply a simple stemmer
analyzer = {
    "with": None,
    "for": None,
    "and": None,
    "from": None,
    "lovers": "love",
    "hearts": "heart"
}
bag_of_words_docs = [list(filter(None, [analyzer.get(word, word) for word in d.split()])) for d in docs]
```

```python
# Create term frequency matrix
unique_words = list(set(chain.from_iterable(bag_of_words_docs)))
word_freq = [Counter(d) for d in bag_of_words_docs]
A = np.array([[freq.get(word, 0) for freq in word_freq] for word in unique_words])
for i, word in enumerate(unique_words):
    print("%10s %s" % (word, str(A[i])))
```

       candies [1 1 0]
    collection [0 1 0]
     chocolate [1 1 1]
         heart [0 1 0]
       chicago [1 0 1]
        sweets [0 1 1]
          made [1 0 0]
         retro [1 0 1]
          love [1 1 1]
          mini [0 1 0]

```python
# Perform truncated SVD decomposition
U, s, V = np.linalg.svd(A, full_matrices=False)
truncate_rank = 2
Ut = U[:, 0:truncate_rank]
Vt = V[0:truncate_rank, :]
St = np.diag(s[0:truncate_rank])
reconstruction = np.dot(Ut, np.dot(St, Vt))
tabprint("Ut =", Ut)
tabprint("St =", St)
tabprint("Vt =", Vt)
tabprint("Ut x St x Vt =", np.round(reconstruction))
```

    Ut =
    ╒═══════════╤════════════╕
    │ -0.333888 │ -0.148947  │
    ├───────────┼────────────┤
    │ -0.167762 │ -0.4059    │
    ├───────────┼────────────┤
    │ -0.485799 │  0.0183087 │
    ├───────────┼────────────┤
    │ -0.167762 │ -0.4059    │
    ├───────────┼────────────┤
    │ -0.318037 │  0.424208  │
    ├───────────┼────────────┤
    │ -0.319674 │ -0.238644  │
    ├───────────┼────────────┤
    │ -0.166126 │  0.256953  │
    ├───────────┼────────────┤
    │ -0.318037 │  0.424208  │
    ├───────────┼────────────┤
    │ -0.485799 │  0.0183087 │
    ├───────────┼────────────┤
    │ -0.167762 │ -0.4059    │
    ╘═══════════╧════════════╛
    St =
    ╒═════════╤═════════╕
    │ 3.56192 │ 0       │
    ├─────────┼─────────┤
    │ 0       │ 1.96588 │
    ╘═════════╧═════════╛
    Vt =
    ╒═══════════╤═══════════╤═══════════╕
    │ -0.591727 │ -0.597555 │ -0.541097 │
    ├───────────┼───────────┼───────────┤
    │  0.505138 │ -0.79795  │  0.328805 │
    ╘═══════════╧═══════════╧═══════════╛
    Ut x St x Vt =
    ╒════╤════╤═══╕
    │  1 │  1 │ 1 │
    ├────┼────┼───┤
    │ -0 │  1 │ 0 │
    ├────┼────┼───┤
    │  1 │  1 │ 1 │
    ├────┼────┼───┤
    │ -0 │  1 │ 0 │
    ├────┼────┼───┤
    │  1 │  0 │ 1 │
    ├────┼────┼───┤
    │  0 │  1 │ 0 │
    ├────┼────┼───┤
    │  1 │ -0 │ 0 │
    ├────┼────┼───┤
    │  1 │  0 │ 1 │
    ├────┼────┼───┤
    │  1 │  1 │ 1 │
    ├────┼────┼───┤
    │ -0 │  1 │ 0 │
    ╘════╧════╧═══╛

```python
# Project a query to the concept space and score documents
query = "chicago"
q = [int(query == word) for word in unique_words]
qs = np.dot(q, np.dot(Ut, np.linalg.inv(St)))

def score(query_vec, doc_vec):
    return np.dot(query_vec, doc_vec) / ( np.linalg.norm(query_vec) * np.linalg.norm(doc_vec) )

for d in range(len(docs)):
    print("Document %s score: %s" % (d, score(qs, Vt[:, d])))
```

    Document 0 score: 0.890730150933
    Document 1 score: -0.51043666768
    Document 2 score: 0.806592806364


```python
import numpy as np
import scipy.misc
from scipy.fftpack import dct, idct
import sys
from PIL import Image
import matplotlib
import matplotlib.pyplot as plt
import random
from tqdm._tqdm_notebook import tqdm_notebook
from scipy.fftpack import dct, idct
import seaborn as sns
from skimage.metrics import structural_similarity as ssim
import pandas as pd
import sympy

%matplotlib inline

class ImageLoader:
    def __init__(self, FILE_PATH):
        self.img = np.array(Image.open(FILE_PATH))
        # 行数
        self.row_blocks_count = self.img.shape[0] // 8
        # 列数
        self.col_blocks_count = self.img.shape[1] // 8

    def get_points(self, POINT):
        Row = random.randint(0, len(self.img) - POINT - 1)
        Col = random.randint(0, len(self.img) - 1)
        return self.img[Row : Row + POINT, Col]

    def get_block(self, col, row):
        return self.img[col * 8 : (col + 1) * 8, row * 8 : (row + 1) * 8]

# plt.rcParams['font.family'] ='sans-serif'#使用するフォント
# plt.rcParams["font.sans-serif"] = "Source Han Sans"
plt.rcParams["font.family"] = "Source Han Sans JP"  # 使用するフォント
plt.rcParams["xtick.direction"] = "in"  # x軸の目盛線が内向き('in')か外向き('out')か双方向か('inout')
plt.rcParams["ytick.direction"] = "in"  # y軸の目盛線が内向き('in')か外向き('out')か双方向か('inout')
plt.rcParams["xtick.major.width"] = 1.0  # x軸主目盛り線の線幅
plt.rcParams["ytick.major.width"] = 1.0  # y軸主目盛り線の線幅
plt.rcParams["font.size"] = 12  # フォントの大きさ
plt.rcParams["axes.linewidth"] = 1.0  # 軸の線幅edge linewidth。囲みの太さ
matplotlib.font_manager._rebuild()

MONO_DIR_PATH = "../../Mono/"
AIRPLANE = ImageLoader(MONO_DIR_PATH + "airplane512.bmp")
BARBARA = ImageLoader(MONO_DIR_PATH + "barbara512.bmp")
BOAT = ImageLoader(MONO_DIR_PATH + "boat512.bmp")
GOLDHILL = ImageLoader(MONO_DIR_PATH + "goldhill512.bmp")
LENNA = ImageLoader(MONO_DIR_PATH + "lenna512.bmp")
MANDRILL = ImageLoader(MONO_DIR_PATH + "mandrill512.bmp")
MILKDROP = ImageLoader(MONO_DIR_PATH + "milkdrop512.bmp")
SAILBOAT = ImageLoader(MONO_DIR_PATH + "sailboat512.bmp")

IMAGES = [
    AIRPLANE,
    BARBARA,
    BOAT,
    GOLDHILL,
    LENNA,
    MANDRILL,
    MILKDROP,
    SAILBOAT
]
```

```python
class DMLCT:
    def __init__(self, n_bar, N):
        self.n_bar = n_bar
        self.N = N

        self.x_l = (2 * np.arange(N) + 1) / (2 * N)
        self.s_l = np.arange(n_bar) / (n_bar - 1)
        self.xi = (np.arange(n_bar + 1) - 0.5) / (n_bar - 1)

        self.lambda_kh = self.get_lambda_kh(self.n_bar)

        self.w_k_j = self.get_w_k_j(self.n_bar, self.N)
        self.W_L_k_kh = self.get_W_L_k_kh(self.n_bar, self.N)
        self.W_k_kh = self.get_W_k_kh(self.n_bar, self.N)
        self.W_R_k_kh = self.get_W_R_k_kh(self.n_bar, self.N)

    def Lagrange_j(self, j):
        x = sympy.Symbol("x")
        L_x = 1.0
        for l in range(self.n_bar):
            if l != j:
                L_x *= (x - self.s_l[l]) / (self.s_l[j] - self.s_l[l])
        return sympy.integrate(L_x)

    def get_lambda_kh(self, n_bar):
        lambda_kh = np.ones(n_bar)
        lambda_kh[0] = np.sqrt(1 / 2)
        return lambda_kh

    def get_w_k_j(self, n_bar, N):
        L_j = np.zeros((n_bar, N))
        x = sympy.Symbol("x")
        for j in range(n_bar):
            temp = []
            Lj = self.Lagrange_j(j)
            for k in range(N):
                temp.append(Lj.subs(x, self.x_l[k]))
            L_j[j] = np.array(temp)

        w_k_j = np.zeros((n_bar, N))
        for j in range(n_bar):
            w_k_j[j] = scipy.fftpack.dct(L_j[j], norm="ortho")
        return w_k_j

    def get_W_L_k_kh(self, n_bar, N):
        W_L_k_kh = np.zeros((n_bar - 1, N))
        lambda_kh = self.get_lambda_kh(n_bar)

        for kh in range(n_bar - 1):
            W_L_k_kh[kh] = (
                (1 - n_bar)
                * np.sqrt(2 / N)
                * lambda_kh[kh]
                * np.cos(np.pi * kh * (self.xi[0] + 1))
                * self.w_k_j[0]
            )
        return W_L_k_kh

    def get_W_k_kh(self, n_bar, N):
        W_k_kh = np.zeros((n_bar - 1, N))
        for kh in range(n_bar - 1):
            sum_sin = np.zeros(N)
            for j in range(1, n_bar - 2 + 1):
                sum_sin += np.sin(np.pi * kh * self.s_l[j]) * self.w_k_j[j]

            W_k_kh[kh] = (
                (n_bar - 1)
                * np.sqrt(2 / N)
                * self.lambda_kh[kh]
                * (
                    np.cos(np.pi * kh * self.xi[1])
                    * (self.w_k_j[0] - (-1) ** (kh) * self.w_k_j[n_bar - 1])
                    - 2 * np.sin((np.pi * kh) / (2 * (n_bar - 1))) * sum_sin
                )
            )
        return W_k_kh

    def get_W_R_k_kh(self, n_bar, N):
        W_R_k_kh = np.zeros((n_bar - 1, N))
        for kh in range(n_bar - 1):
            W_R_k_kh[kh] = (
                (n_bar - 1)
                * np.sqrt(2 / N)
                * self.lambda_kh[kh]
                * np.cos(np.pi * kh * (self.xi[n_bar] - 1))
                * self.w_k_j[n_bar - 1]
            )
        return W_R_k_kh
```

```python
def get_F_L_k_horizontal(arr, N, row, col):
    # w
    if col == 0:
        w_block = np.zeros(N)
    else:
        w_block = arr[row, (col - 1) * N : col * N]

    return w_block
```

```python
def get_F_R_k_horizontal(arr, N, row, col):
    # e
    if col == arr.shape[1] // N - 1:
        e_block = np.zeros(N)
    else:
        e_block = arr[row, (col + 1) * N : (col + 2) * N]

    return e_block
```

```python
def get_F_L_k_vertical(arr, N, row, col):
    # n
    if row == 0:
        n_block = np.zeros(N)
    else:
        n_block = arr[(row - 1) * N : row * N, col]

    return n_block
```

```python
def get_F_R_k_vertical(arr, N, row, col):
    # s
    if row == arr.shape[0] // N - 1:
        s_block = np.zeros(N)
    else:
        s_block = arr[(row + 1) * N : (row + 2) * N, col]

    return s_block
```

```python
n_bar = 4
N = 8
```

```python
dmlct = DMLCT(n_bar, N)
```

```python
IMG = LENNA
```

```python
Fk = np.zeros(IMG.img.shape)
```

# 順変換

```python
def DMLCT_forward(IMG,n_bar,N):
    Fk = np.zeros(IMG.img.shape)
    ## 縦方向
    ### DCT
    for row in range(IMG.img.shape[0] // N):
        for col in range(IMG.img.shape[1]):
            eight_points = IMG.img[N * row : N * (row + 1), col]
            c = scipy.fftpack.dct(eight_points, norm="ortho")
            Fk[N * row : N * (row + 1), col] = c
    ### 残差
    dmlct = DMLCT(n_bar,N)
    for row in range(IMG.img.shape[0] // N):
        for col in range(IMG.img.shape[1]):
            # ビューなら直接いじっちゃう
            F = Fk[N * row : N * (row + 1), col]
            F_L = get_F_L_k_vertical(Fk, N, row, col)
            F_R = get_F_R_k_vertical(Fk, N, row, col)

            U_k_n_bar = np.zeros(N)
            for kh in range(n_bar - 2 + 1):
                U_k_n_bar += (
                    F_L[kh] * dmlct.W_L_k_kh[kh]
                    + F[kh] * dmlct.W_k_kh[kh]
                    + F_R[kh] * dmlct.W_R_k_kh[kh]
                )

            # n_bar = 4 なら 0,1,2は残す 3,4,5,6,7を書き換える
            F[n_bar - 2 + 1 :] -= U_k_n_bar[n_bar - 2 + 1 :]
    # 0を残す
    for k in reversed(range(1, n_bar - 2 + 1)):
        dmlct = DMLCT(k+1, N)
        for row in range(IMG.img.shape[0] // N):
            for col in range(IMG.img.shape[1]):
                # ビューなら直接いじっちゃう
                F = Fk[N * row : N * (row + 1), col]
                F_L = get_F_L_k_vertical(Fk, N, row, col)
                F_R = get_F_R_k_vertical(Fk, N, row, col)

                U_k_n_bar = np.zeros(N)

                for kh in range((k + 1) - 2 + 1):
                    U_k_n_bar += (
                        F_L[kh] * dmlct.W_L_k_kh[kh]
                        + F[kh] * dmlct.W_k_kh[kh]
                        + F_R[kh] * dmlct.W_R_k_kh[kh]
                    )

                F[k] -= U_k_n_bar[k]
    ## 横方向
    ### DCT
    for row in range(Fk.shape[0]):
        for col in range(Fk.shape[1] // N):
            eight_points = Fk[row, N * col : N * (col + 1)]
            c = scipy.fftpack.dct(eight_points, norm="ortho")
            Fk[row, N * col : N * (col + 1)] = c

    ### 残差
    dmlct = DMLCT(n_bar,N)
    for row in range(IMG.img.shape[0]):
        for col in range(IMG.img.shape[1] // N):
            F = Fk[row, N * col : N * (col + 1)]
            F_L = get_F_L_k_horizontal(Fk, N, row, col)
            F_R = get_F_R_k_horizontal(Fk, N, row, col)

            U_k_n_bar = np.zeros(N)
            for kh in range(n_bar - 2 + 1):
                U_k_n_bar += (
                    F_L[kh] * dmlct.W_L_k_kh[kh]
                    + F[kh] * dmlct.W_k_kh[kh]
                    + F_R[kh] * dmlct.W_R_k_kh[kh]
                )

            # n_bar = 4 なら 0,1,2は残す 3,4,5,6,7を書き換える
            F[n_bar - 2 + 1 :] -= U_k_n_bar[n_bar - 2 + 1 :]
    # 0を残す
    for k in reversed(range(1, n_bar - 2 + 1)):
        dmlct = DMLCT(k+1, N)
        for row in range(IMG.img.shape[0]):
            for col in range(IMG.img.shape[1] // N):
                F = Fk[row, N * col : N * (col + 1)]
                F_L = get_F_L_k_horizontal(Fk, N, row, col)
                F_R = get_F_R_k_horizontal(Fk, N, row, col)

                U_k_n_bar = np.zeros(N)

                for kh in range((k + 1) - 2 + 1):
                    U_k_n_bar += (
                        F_L[kh] * dmlct.W_L_k_kh[kh]
                        + F[kh] * dmlct.W_k_kh[kh]
                        + F_R[kh] * dmlct.W_R_k_kh[kh]
                    )

                F[k] -= U_k_n_bar[k]

    return Fk
```

```python
# DCT係数の平均を求める
```

```python
Fk_values = np.zeros((512,512))
```

```python
for IMG in tqdm_notebook(IMAGES):
    values = np.zeros((25,4))
    Fk = np.zeros(IMG.img.shape)
    for row in range(IMG.img.shape[0] // N):
        for col in range(IMG.img.shape[1] // N):
            block = IMG.img[row * N : (row + 1) * N, col * N : (col + 1) * N]
            c = scipy.fftpack.dct(
                scipy.fftpack.dct(block, axis=0, norm="ortho"), axis=1, norm="ortho"
            )
            Fk[row * N : (row + 1) * N, col * N : (col + 1) * N] = c

    Fk_values += np.abs(Fk)
```

    HBox(children=(IntProgress(value=0, max=8), HTML(value='')))

```python
Fk_values /= len(IMAGES)
pd.DataFrame(Fk_values).to_csv("DCT_coef_ave.csv",header=False,index=False)
```

```python
# 各n_barの残差係数の平均を求める
```

```python
Vk_values = np.zeros((512,512))
```

```python
for n_bar in tqdm_notebook(range(2,6)):
    Vk_values = np.zeros((512,512))
    for IMG in IMAGES:
        Fk = DMLCT_forward(IMG,n_bar,N)
        Vk_values += np.abs(Fk)

    pd.DataFrame(Vk_values / len(IMAGES)).to_csv("DMLCT_" + str(n_bar) + "_coef_ave.csv",header=False,index=False)
```

```python
# DCT係数を読み込む
```

```python
Fk_values = pd.read_csv("DCT_coef_ave.csv",header=None).values
```

```python
# 残差係数を読み込む
```

```python
n_bar = 5
Vk_values_arr = []
for i in range(2,n_bar+1,1):
    Vk_values_arr.append(pd.read_csv("DMLCT_" + str(i) + "_coef_ave.csv",header=None).values)
```

```python
# NxNブロック1個当たりの係数の平均を求める
```

```python
Fk_block_ave_values = np.zeros((N,N))
for row in range(IMG.img.shape[0] // N):
    for col in range(IMG.img.shape[1] // N):
        if col == 0:
            continue
        if col == IMG.img.shape[1] // N -1:
            continue
        if row == 0:
            continue
        if row == IMG.img.shape[0] // N -1:
            continue
        block = Fk_values[row * N : (row + 1) * N, col * N : (col + 1) * N]
        Fk_block_ave_values += np.abs(block)

Fk_block_ave_values /= (IMG.img.shape[0]//N)**2
```

```python
Vk_block_ave_values_arr = []
for Vk_values in Vk_values_arr:
    Vk_block_ave_values = np.zeros((N,N))
    for row in range(IMG.img.shape[0] // N):
        for col in range(IMG.img.shape[1] // N):
            if col == 0:
                continue
            if col == IMG.img.shape[1] // N -1:
                continue
            if row == 0:
                continue
            if row == IMG.img.shape[0] // N -1:
                continue
            block = Vk_values[row * N : (row + 1) * N, col * N : (col + 1) * N]
            Vk_block_ave_values += np.abs(block)

    Vk_block_ave_values /= (IMG.img.shape[0]//N)**2
    Vk_block_ave_values_arr.append(Vk_block_ave_values)
```

```python
columns = []
for n in reversed(range(2,n_bar+1,1)):
    columns.append("n=" + str(n))
df = pd.DataFrame(columns=columns)
```

```python
k_max = 5
for index in range(1,k_max,1):
    for i in range(index):
        Gk1k2_1 = []
        Gk1k2_2 = []
        for n in reversed(range(2,n_bar+1,1)):
            Vk_block = Vk_block_ave_values_arr[n-2]
            Vk = Vk_block[i,index]
            Fk = Fk_block_ave_values[i,index]
            Gk1k2_1.append(100 * (1 - Vk/Fk))

            Vk = Vk_block[index,i]
            Fk = Fk_block_ave_values[index,i]
            Gk1k2_2.append(100 * (1 - Vk/Fk))

        df.loc["(" + str(i) + "," + str(index) + ")"] = Gk1k2_1
        df.loc["(" + str(index) + "," + str(i) + ")"] = Gk1k2_2

    Gk1k2 = []
    for n in reversed(range(2,n_bar+1,1)):
        Vk_block = Vk_block_ave_values_arr[n-2]
        Vk = Vk_block[index,index]
        Fk = Fk_block_ave_values[index,index]
        Gk1k2.append(100 * (1 - Vk/Fk))

    df.loc["(" + str(index) + "," + str(index) + ")"] = Gk1k2

Gk1k2 = []
for n in reversed(range(2,n_bar+1,1)):
    Vk_block = Vk_block_ave_values_arr[n-2]

    Vk_sum = 0
    Fk_sum = 0
    for row in range(N):
        for col in range(N):
            if row > k_max-1 or col > k_max-1:
                Vk_sum += Vk_block[row,col]
                Fk_sum += Fk_block_ave_values[row,col]
    Gk1k2.append(100 * (1 - Vk_sum/Fk_sum))
df.loc["others"] = Gk1k2
```

```python
df.to_csv("DMLCT_high_freq_comp.csv")
df
```

      n=5
      n=4
      n=3
      n=2

      (0,1)
      20.717614
      20.717614
      20.717614
      20.717614

      (1,0)
      22.109009
      22.109009
      22.109009
      22.109009

      (1,1)
      13.555398
      13.555398
      13.555398
      13.555398

      (0,2)
      16.276007
      16.276007
      16.276007
      8.275768

      (2,0)
      15.410603
      15.410603
      15.410603
      8.012282

      (1,2)
      11.685421
      11.685421
      11.685421
      7.123901

      (2,1)
      11.454613
      11.454613
      11.454613
      7.019498

      (2,2)
      10.841250
      10.841250
      10.841250
      5.345241

      (0,3)
      10.217886
      10.217886
      7.196623
      3.656807

      (3,0)
      10.071757
      10.071757
      6.772187
      3.114785

      (1,3)
      6.239762
      6.239762
      4.049929
      1.617404

      (3,1)
      7.735845
      7.735845
      5.350861
      2.765466

      (2,3)
      7.700865
      7.700865
      6.180695
      2.214539

      (3,2)
      8.141687
      8.141687
      6.139296
      2.276848

      (3,3)
      6.603818
      6.603818
      3.897633
      0.711983

      (0,4)
      6.779308
      5.540569
      5.211938
      2.864035

      (4,0)
      6.865616
      5.124495
      5.021251
      2.731048

      (1,4)
      3.771901
      2.730691
      2.833273
      1.524135

      (4,1)
      4.666098
      3.256033
      3.501182
      2.072189

      (2,4)
      4.390676
      3.602220
      3.860522
      1.734659

      (4,2)
      5.691351
      4.372579
      4.741983
      2.280640

      (3,4)
      5.576304
      4.825351
      3.322126
      0.849770

      (4,3)
      5.769750
      4.488810
      3.454942
      1.008925

      (4,4)
      4.035814
      2.083875
      2.590941
      1.003001

      others
      0.856398
      2.018130
      1.425437
      0.522690

```python

```