### Example 1b: Linear convection in 2D, revisited

We will now revisit the first example of this tutorial with an example that is better suited to the numerical scheme used in Devito. As a reminder, the governing equation is:

$$\frac{\partial u}{\partial t}+c\frac{\partial u}{\partial x} + c\frac{\partial u}{\partial y} = 0$$

We then discretized this using forward differences in time and backward differences in space:

$$u_{i,j}^{n+1} = u_{i,j}^n-c \frac{\Delta t}{\Delta x}(u_{i,j}^n-u_{i-1,j}^n)-c \frac{\Delta t}{\Delta y}(u_{i,j}^n-u_{i,j-1}^n)$$

In the previous example, the system was initialised with a hat function. As easy as this example seems, it actually highlights a few limitations of finite differences and related methods:
 - The governing equation above contains spatial derivatives ($\frac{\partial u}{\partial x}$ and $\frac{\partial u}{\partial y}$). The hat, with its sharp corners, is discontinuous and therefore non-smooth, meaning that the derivatives do not exist at the corners of the hat. This means that the governing equation has no solution in the strict sense for this problem. Mathematically, this problem can be overcome by introducing weak solutions, which still exist in the presence of discontinuities, as long as the problem is smooth almost everywhere. The Finite Volume (FV), Finite Element (FEM) and related schemes are based on this weak form.
 - The finite differences method only works well if finite differences are a good approximation of the derivatives. With the chosen discretization above, this requires that $\frac{u_{i,j}^n-u_{i-1,j}^n}{\Delta x} \approx \frac{\partial u}{\partial x}$ and $\frac{u_{i,j}^n-u_{i,j-1}^n}{\Delta y} \approx \frac{\partial u}{\partial y}$. This is the case for systems with a smooth solution if $\Delta x$ and $\Delta y$ are sufficiently small. But if the solution is non-smooth, as in this example, then we can't expect much regardless of the grid size.
 - First-order methods, such as the backward differences that we have used in this example, are known to create artificial diffusion. Higher-order schemes, such as central differences, avoid this problem. However, in the presence of discontinuities these methods introduce so-called spurious oscillations. These oscillations may even build up (grow infinitely) and cause the computation to diverge.
 - Discontinuities can appear by themselves for some equations (such as the Burgers equation that we discuss next), even if the initial condition is smooth. In CFD, discontinuities appear for example as shocks in the simulation of transonic flow. For this reason, numerical schemes that behave well in the presence of discontinuities have been a research subject for a long time. A thorough discussion is beyond the scope of this tutorial, but can be found in [R. LeVeque (1992): Numerical Methods for Conservation Laws, 2nd ed., Birkhäuser Verlag, pp. 8-13].

In the remainder of this example, we will reproduce the results from the previous example, only this time with a smooth initial condition. This lets us observe Devito in a setting for which it is better equipped.

```python
from examples.cfd import plot_field, init_hat, init_smooth
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 81
ny = 81
nt = 100
c = 1.
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .2
dt = sigma * dx
```

Let us now initialise the field with an infinitely smooth bump, as given by [J.A. Krakos (2012): Unsteady Adjoint Analysis for Output Sensitivity
and Mesh Adaptation, PhD thesis, p. 68] as $$
f(r)=
\begin{cases}
\frac{1}{A}e^{-1/(r-r^2)} &\text{ for } 0 < r < 1,\\
0 &\text{ else.}
\end{cases}
$$
We use this with $A=100$, and define the initial condition in two dimensions as $$u^0(x,y)=1+f\left(\frac{2}{3}x\right)*f\left(\frac{2}{3}y\right).$$

```python
#NBVAL_IGNORE_OUTPUT

# Create field and assign initial conditions
u = np.empty((nx, ny))
init_smooth(field=u, dx=dx, dy=dy)

# Plot initial condition
plot_field(u, zmax=4)
```

Solving this will move the bump again.

```python
# Repeat initialisation, so we can re-run the cell
init_smooth(field=u, dx=dx, dy=dy)

for n in range(nt + 1):
    # Copy previous result into a new buffer
    un = u.copy()

    # Update the new result with a 3-point stencil
    u[1:, 1:] = (un[1:, 1:] - (c * dt / dy * (un[1:, 1:] - un[1:, :-1])) -
                              (c * dt / dx * (un[1:, 1:] - un[:-1, 1:])))

    # Apply boundary conditions.
    # Note: -1 here is the last index in the array, not the one at x=-1 or y=-1.
    u[0, :] = 1.  # left
    u[-1, :] = 1. # right
    u[:, 0] = 1.  # bottom
    u[:, -1] = 1. # top
```

```python
#NBVAL_IGNORE_OUTPUT

# A small sanity check for auto-testing
assert (u[45:55, 45:55] > 1.8).all()
u_ref = u.copy()

plot_field(u, zmax=4.)
```

Hooray, the wave moved! It looks like the solver works much better for this example: The wave has not noticeably changed its shape.

#### Devito implementation
Again, we can re-create this via a Devito operator. Let's fill the initial buffer with smooth data and look at it:

```python
#NBVAL_IGNORE_OUTPUT
from devito import Grid, TimeFunction

grid = Grid(shape=(nx, ny), extent=(2., 2.))
u = TimeFunction(name='u', grid=grid)
init_smooth(field=u.data[0], dx=dx, dy=dy)

plot_field(u.data[0])
```

```python
u.data.shape
```

    (2, 81, 81)

We create again the discretized equation as shown below. Note that the equation is still the same, only the initial condition has changed.

```python
from devito import Eq

eq = Eq(u.dt + c*u.dxl + c*u.dyl)

print(eq)
```

    Eq(Derivative(u(t, x, y), x) + Derivative(u(t, x, y), y) + Derivative(u(t, x, y), t), 0)

SymPy can re-organise this equation just like in the previous example.

```python
from devito import solve
from sympy import nsimplify, pprint

stencil = solve(eq, u.forward)

pprint(nsimplify(stencil))
```

       ⎛  ∂                ∂                u(t, x, y)⎞
    dt⋅⎜- ──(u(t, x, y)) - ──(u(t, x, y)) + ──────────⎟
       ⎝  ∂x               ∂y                   dt    ⎠

We can now use this stencil expression to create an operator to apply to our data object:

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

# Reset our initial condition in both buffers.
# This is required to avoid 0s propagating into
# our solution, which has a background value of 1.
init_smooth(field=u.data[0], dx=dx, dy=dy)
init_smooth(field=u.data[1], dx=dx, dy=dy)

# Apply boundary conditions.
# Note that as the u.data method is from numpy, we can use the
#  -1 syntax to represent the last item in the array.
u.data[:, 0, :] = 1.
u.data[:, -1, :] = 1.
u.data[:, :, 0] = 1.
u.data[:, :, -1] = 1.

# Create an Operator that updates the forward stencil
# point in the interior subdomain only.
op = Operator(Eq(u.forward, stencil, subdomain=grid.interior))

# Apply the operator for a number of timesteps
op(time=nt, dt=dt)

plot_field(u.data[0])

# Some small sanity checks for the testing framework
assert (u.data[0, 45:55, 45:55] > 1.8).all()
assert np.allclose(u.data[0], u_ref, rtol=3.e-2)
```

Again, this looks just like the result from NumPy. Since this example is just like the one before, the low-level treatment of boundaries is also unchanged.

```python
#NBVAL_IGNORE_OUTPUT

# Reset our data field and ICs in both buffers
init_smooth(field=u.data[0], dx=dx, dy=dy)
init_smooth(field=u.data[1], dx=dx, dy=dy)

# For defining BCs, we generally to explicitly set rows/columns
# in our field using an expression. We can use Devito's "indexed"
# notation to do this. A u in this instance is a sympy function
# we cannot use the numpy shortcut u[-1] to refer to the last location:
x, y = grid.dimensions
t = grid.stepping_dim
bc_left = Eq(u[t + 1, 0, y], 1.)
bc_right = Eq(u[t + 1, nx-1, y], 1.)
bc_top = Eq(u[t + 1, x, ny-1], 1.)
bc_bottom = Eq(u[t + 1, x, 0], 1.)

# Now combine the BC expressions with the stencil to form operator.
expressions = [Eq(u.forward, stencil, subdomain=grid.interior)]
expressions += [bc_left, bc_right, bc_top, bc_bottom]
op = Operator(expressions=expressions, opt=None, openmp=False)  # <-- Turn off performance optimisations
op(time=nt, dt=dt)

plot_field(u.data[0])

# Some small sanity checks for the testing framework
assert (u.data[0, 45:55, 45:55] > 1.8).all()
assert np.allclose(u.data[0], u_ref, rtol=3.e-2)
```

The C code of the Kernel is also still the same.

```python
print(op.ccode)
```

    #define _POSIX_C_SOURCE 200809L
    #define START_TIMER(S) struct timeval start_ ## S , end_ ## S ; gettimeofday(&start_ ## S , NULL);
    #define STOP_TIMER(S,T) gettimeofday(&end_ ## S, NULL); T->S += (double)(end_ ## S .tv_sec-start_ ## S.tv_sec)+(double)(end_ ## S .tv_usec-start_ ## S .tv_usec)/1000000;

    #include "stdlib.h"
    #include "math.h"
    #include "sys/time.h"

    struct dataobj
    {
      void *restrict data;
      int * size;
      int * npsize;
      int * dsize;
      int * hsize;
      int * hofs;
      int * oofs;
    } ;

    struct profiler
    {
      double section0;
      double section1;
      double section2;
    } ;

    int Kernel(const float dt, const float h_x, const float h_y, struct dataobj *restrict u_vec, const int i0x_ltkn, const int i0x_rtkn, const int i0y_ltkn, const int i0y_rtkn, const int time_M, const int time_m, const int x_M, const int x_m, const int y_M, const int y_m, struct profiler * timers)
    {
      float (*restrict u)[u_vec->size[1]][u_vec->size[2]] __attribute__ ((aligned (64))) = (float (*)[u_vec->size[1]][u_vec->size[2]]) u_vec->data;

      for (int time = time_m, t0 = (time)%(2), t1 = (time + 1)%(2); time <= time_M; time += 1, t0 = (time)%(2), t1 = (time + 1)%(2))
      {
        /* Begin section0 */
        START_TIMER(section0)
        for (int i0x = i0x_ltkn + x_m; i0x <= -i0x_rtkn + x_M; i0x += 1)
        {
          for (int i0y = i0y_ltkn + y_m; i0y <= -i0y_rtkn + y_M; i0y += 1)
          {
            u[t1][i0x + 1][i0y + 1] = dt*(-(-u[t0][i0x][i0y + 1]/h_x + u[t0][i0x + 1][i0y + 1]/h_x) - (-u[t0][i0x + 1][i0y]/h_y + u[t0][i0x + 1][i0y + 1]/h_y) + u[t0][i0x + 1][i0y + 1]/dt);
          }
        }
        STOP_TIMER(section0,timers)
        /* End section0 */

        /* Begin section1 */
        START_TIMER(section1)
        for (int y = y_m; y <= y_M; y += 1)
        {
          u[t1][1][y + 1] = 1.00000000000000F;

          u[t1][81][y + 1] = 1.00000000000000F;
        }
        STOP_TIMER(section1,timers)
        /* End section1 */

        /* Begin section2 */
        START_TIMER(section2)
        for (int x = x_m; x <= x_M; x += 1)
        {
          u[t1][x + 1][81] = 1.00000000000000F;

          u[t1][x + 1][1] = 1.00000000000000F;
        }
        STOP_TIMER(section2,timers)
        /* End section2 */
      }

      return 0;
    }


# 素数の判定

```python
import itertools as it
from kanren import isvar, membero, var, run, eq
from kanren.core import success, fail, condeseq
from sympy.ntheory.generate import prime, isprime
```

```python
# 素数かチェックする関数
def check_prime(x):
    if isvar(x):
        return condeseq([eq(x, p)] for p in map(prime, it.count(1)))
    else:
        return success if isprime(x) else fail

x = var()

list_nums = (23, 4, 27, 17, 13, 10, 21, 29, 3, 32, 11, 19)
print('List of primes in the list:')
print(set(run(0, x, (membero, x, list_nums), (check_prime, x))))
```

    List of primes in the list:
    {3, 11, 13, 17, 19, 23, 29}

`prime`は素数のこと

```python
print('List of first 7 prime numbers:')
print(set(run(13, x, (check_prime, x))))
```

    List of first 7 prime numbers:
    {2, 3, 5, 37, 7, 41, 11, 13, 17, 19, 23, 29, 31}

```python

```


# Notebook 7: Logistic Regression and SoftMax for MNIST

## Learning Goal

The goal of this notebook is to familiarize the reader with SoftMax regression (a generalization of logistic regression to more than two categories), categorical predictions, and the MNIST handwritten dataset. The reader will understand how to use the Scikit Logistic regression package and visualize learned weights.

## Overview
### The MNIST dataset:
The MNIST classification problem is one of the classical ML problems for learning classification on high-dimensional data with a fairly sizable number of examples (60000). Yann LeCun and collaborators collected and processed $70000$ handwritten digits (60000 are used for training and 10000 for testing) to produce what became known as one of the most widely used datasets in ML: the [MNIST](http://yann.lecun.com/exdb/mnist/) dataset. Each handwritten digit comes in a grayscale square image in the shape of a $28\times 28$ pixel grid. Every pixel takes a value in the range $[0,255]$, representing $256$ nuances of the gray color. The problem of image classification finds applications in a wide range of fields and is important for numerous industry applications of ML.
### SoftMax regression:
We will use SoftMax regression, which can be thought of as a statistical model which assigns a probability that a given input image corresponds to any of the 10 handwritten digits. The model is a generalization of the logistic regression and reads:
\begin{align}
p(y=i|\boldsymbol{x};W) = \frac{e^{\boldsymbol{w}_i^T \boldsymbol{x}}}{\sum_{j=0}^9 e^{\boldsymbol{w}_j^T}},
\end{align}
Where $p(y=i|\boldsymbol{x};W)$ is the probability that input $\boldsymbol{x}$ is the $i$-th digit, $i\in[0,9]$.
The model also has 10 weight vectors $\boldsymbol{w}_i$ which we will train below. Finally, one can use this information for prediction by taking the value of $y$ for which this probability is maximized:
\begin{align}
y_{pred}=\arg\max_i p(y=i|\boldsymbol{x})
\end{align}

## Numerical Experiments

The reader is invited to check out the code below to build up their intuition about SoftMax regression. The following notebook is a slight modification of [this Scikit tutorial](http://scikit-learn.org/dev/auto_examples/linear_model/plot_sparse_logistic_regression_mnist.html) by Arthur Mensch on studying the MNIST problem using Logistic Regression.

```python
import time
import numpy as np

from sklearn.datasets import fetch_openml # MNIST data
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.utils import check_random_state

print(__doc__)

# Turn down for faster convergence
t0 = time.time()
train_size = 50000
test_size = 10000

### load MNIST data from https://www.openml.org/d/554
X, y = fetch_openml('mnist_784', version=1, return_X_y=True)

# shuffle data
random_state = check_random_state(0)
permutation = random_state.permutation(X.shape[0])
X = X[permutation]
y = y[permutation]
X = X.reshape((X.shape[0], -1))

# pick training and test data sets
X_train, X_test, y_train, y_test = train_test_split(X,y,train_size=train_size,test_size=test_size)

# scale data to have zero mean and unit variance [required by regressor]
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# apply logistic regressor with 'sag' solver, C is the inverse regularization strength
clf = LogisticRegression(C=1e5,
                         multi_class='multinomial',
                         penalty='l2', solver='sag', tol=0.1)
# fit data
clf.fit(X_train, y_train)
# percentage of nonzero weights
sparsity = np.mean(clf.coef_ == 0) * 100
# compute accuracy
score = clf.score(X_test, y_test)

#display run time
run_time = time.time() - t0
print('Example run in %.3f s' % run_time)

print("Sparsity with L2 penalty: %.2f%%" % sparsity)
print("Test score with L2 penalty: %.4f" % score)
```

    Automatically created module for IPython interactive environment
    Example run in 47.833 s
    Sparsity with L2 penalty: 9.18%
    Test score with L2 penalty: 0.8948

```python
%matplotlib inline
import matplotlib.pyplot as plt

# plot weights vs the pixel position
coef = clf.coef_.copy()
plt.figure(figsize=(10, 5))
scale = np.abs(coef).max()
for i in range(10):
    l2_plot = plt.subplot(2, 5, i + 1)
    l2_plot.imshow(coef[i].reshape(28, 28), interpolation='nearest',
                   cmap=plt.cm.Greys, vmin=-scale, vmax=scale)
    l2_plot.set_xticks(())
    l2_plot.set_yticks(())
    l2_plot.set_xlabel('Class %i' % i)
plt.suptitle('classification weights vector $w_j$ for digit class $j$')

plt.show()
```

### Exercises: ###

 The snippet above invoked the "sag" solver which only deals with $L2$ regularization. Try another solver, e.g. 'liblinear', in scikit that supports $L1$ regularization. You can find more details on  http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html.

 Visualize the misclassified samples. Analyze on how logistic regression performs on MNIST, and note this since we will study the MNIST problem later on using deep neural nets.

 Try to vary the size of the training set and see how that affects the accuracy.

  [Theory] In this notebook, we considered a $K=10$ multi-class logistic regression (MLR) also known as the SoftMax regression. Recall that the our model reads:
$$
p(y=i|\boldsymbol{x};W) = \frac{e^{\boldsymbol{w}_i^T \boldsymbol{x}}}{\sum_{j=0}^9 e^{\boldsymbol{w}_j^T \boldsymbol{x}}}
$$
Use this to construct the likelihood function $P(y|\boldsymbol{x})$.

 [Theory] Now consider we're given a set of i.i.d. data $\mathcal{D}=\{(\boldsymbol{x}_1,y_1),\cdots, (\boldsymbol{x}_n,y_n)\}$. As we argued in the main text, the cross-entropy error used in logistic regression is motivated by maximizing the likelihood (or equivalently, minimizing the negative log-likelihood), find the corresponding error function $E(\boldsymbol{w}_0,\cdots,\boldsymbol{w}_9)$ in MLR using the hypothesis above.

 [Theory] Using the error function derived above, calculate its gradient $\frac{\partial E}{\partial \boldsymbol{w}_k}$. Can you find an elegant way of writing the gradient?

```python

```


## Simple qubit rotation with Pennylane and TFQ

In this jupyter file we define a variational quantum circuit $V(\theta)$ that rotates an initial state $|0000\rangle$ into a target state with equal superposition $\frac{1}{\sqrt{|\Sigma|}}\sum_{\sigma_i} | \sigma_i \rangle$. The aim is that  $\frac{1}{\sqrt{|\Sigma|}}\sum_{\sigma_i} \langle \sigma_i | V(\theta) | 0000\rangle = 1$.

```python
import pennylane as qml
from pennylane import numpy as np
from tqdm.notebook import tqdm
```

## Pennylane version

Define the device `default.qubit` and a circuit where one layer contains a general rotation $R(\phi, \theta, \omega) = R_z(\omega)R_x(\theta)R_z(\phi)$ on each qubit, followed by entangling gates. We apply 2 layers. The $R(\phi, \theta, \omega)$ gate is a native in pennylane `qml.Rot()`. We use 4 qubits.

```python
dev1 = qml.device("default.qubit", wires=4)
```

```python
target_state = np.ones(2**4)/np.sqrt(2**4)
density = np.outer(target_state, target_state)

@qml.qnode(dev1)
def circuit(params):
    for j in range(2): # 2 layers
        for i in range(4): # 4 qubits
            qml.Rot(*params[j][i], wires=i)
        qml.CNOT(wires=[0,1])
        qml.CNOT(wires=[2,3])
        qml.CNOT(wires=[1,2])
    return qml.expval(qml.Hermitian(density, wires=[0,1,2,3]))
```

Define a cost function. In our case we want the overlap of the circuit output to be maximal with the targe_state. Therefore we minimize $1-\frac{1}{\sqrt{|\Sigma|}}\sum_{\sigma_i}\langle \sigma_i | V(\theta) | 0000\rangle$

```python
def cost(var):
    return 1-circuit(var)
```

Initialize the parameters randomly. The shape of the parametrs is $(layers, number of qubits, 3)$ because for each layer and qubit we have 3 paramters.

```python
init_params = np.random.rand(2, 4, 3) # 2 layers, 4 qubits, 3 parameters per rotation
print(cost(init_params))
```

    0.9791545794006542

### Training

For the training we define a gradient descent optimizer and continuously update the parameters

```python
# initialise the optimizer
opt = qml.GradientDescentOptimizer(stepsize=0.4) # stepsize is the learning rate

# set the number of steps
steps = 100
# set the initial parameter values
params = init_params

for i in tqdm(range(steps)):
    # update the circuit parameters
    params = opt.step(cost, params)

    if (i + 1) % 10 == 0:
        print("Cost after step {:5d}: {: .7f}".format(i + 1, cost(params)))

print("Optimized rotation angles: {}".format(params))
```

    /home/apozas/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:9: TqdmDeprecationWarning: This function will be removed in tqdm==5.0.0
    Please use `tqdm.notebook.tqdm` instead of `tqdm.tqdm_notebook`
      if __name__ == '__main__':

    HBox(children=(FloatProgress(value=0.0), HTML(value='')))

    Cost after step    10:  0.7926067
    Cost after step    20:  0.1511231
    Cost after step    30:  0.0142782
    Cost after step    40:  0.0049359
    Cost after step    50:  0.0024388
    Cost after step    60:  0.0013136
    Cost after step    70:  0.0007371
    Cost after step    80:  0.0004231
    Cost after step    90:  0.0002461
    Cost after step   100:  0.0001442

    Optimized rotation angles: [[[ 4.37680865e-01  5.83405123e-09  3.20036600e-01]
      [ 5.87685497e-01 -5.85645201e-08 -2.78093370e-02]
      [ 9.77501871e-02 -4.70612985e-02  5.59492498e-01]
      [ 3.37501386e-01  1.05926966e+00 -7.16822962e-02]]

     [[ 9.64584837e-01  1.57079632e+00  3.82648191e-09]
      [ 6.43859415e-01  1.57079627e+00 -2.28028241e-08]
      [ 1.10680485e+00  1.56630413e+00  4.68466476e-02]
      [ 2.36741061e-01  5.17397955e-01 -1.43807908e-01]]]

If we check the final state we see that appart from a global pahse we find the target state.

```python
circuit(params)
dev1.state
```

    array([-0.21085093-0.13604769j, -0.20637046-0.13970451j,
           -0.20998474-0.13732926j, -0.20530324-0.14079939j,
           -0.20998475-0.13732924j, -0.20530324-0.14079937j,
           -0.21085091-0.13604769j, -0.20637045-0.1397045j ,
           -0.21085091-0.13604769j, -0.20637044-0.1397045j ,
           -0.20998474-0.13732924j, -0.20530324-0.14079937j,
           -0.20998474-0.13732926j, -0.20530323-0.14079939j,
           -0.21085093-0.13604769j, -0.20637046-0.1397045j ])

## TFQ version

```python
import tensorflow as tf
import tensorflow_quantum as tfq
import tensorflow.keras as keras

import cirq
import sympy
import numpy as np

# visualization tools
%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit
```

### Generate symbols

I did not figure out, how I can do the gradients in TFQ without using symbols, this seems to be mandatory for tfq. I don't reallay see the advantage so far. Especially the evaluation with the resolver function seems a bit odd and unnecessary.

The definition of the circuit is almost the same as in Pennylane.

There is no option to define a projections operator to calculate the overlap with a target state because they are not unitary. This gives a bit less room to play with TFQ. I assume the idea was, that these gates are not really feasible on a real quantum device.

% Instead of defining a hermitian matrix that gives the overlap with the target state, we can simply measure the operator $M = 1/4*(X_1 + X_2 + X_3 + X_4)$ and minimize the loss $1-\langle M \rangle$.

```python
def generate_circuit(nr_of_qubits, layers):
    qubits = cirq.GridQubit.rect(1, nr_of_qubits) # Define qubit grid. In this case
    nr_parameters = 3*nr_of_qubits*layers # 3 params for each qubit and layer

    symb = sympy.symbols('theta0:'+str(nr_parameters))
    symbols = np.array(symb)
    symbols = symbols.reshape(layers, nr_of_qubits, 3)
    circuit = cirq.Circuit()

    for l in range(layers):
        # Add a series of single qubit rotations.
        for i, qubit in enumerate(qubits):
            circuit += cirq.rz(symbols[l][i][0])(qubit)
            circuit += cirq.rx(symbols[l][i][1])(qubit)
            circuit += cirq.rz(symbols[l][i][2])(qubit)

        circuit += cirq.CZ(qubits[0], qubits[1])
        circuit += cirq.CZ(qubits[2], qubits[3])
        circuit += cirq.CZ(qubits[1], qubits[2])

    op = 1/4*(cirq.X(qubits[0]) + cirq.X(qubits[1]) + cirq.X(qubits[2]) + cirq.X(qubits[3]))
    return circuit, op, list(symb)

nr_of_qubits = 4
layers = 2
tf_circuit, op, (symbols) = generate_circuit(nr_of_qubits, layers)
SVGCircuit(tf_circuit)
```

### Training

One can leverage all the Tensorflow machinery for training quantum circuits. We will now insert the previous circuit in a ``tf.keras`` model in order to train it.

First of all, the circuit must be converted into a layer so it can be inserted in a model. The most direct choice is the ``PQC`` (which stands for Parameterized Quantum Circuit) layer. This layer requires as additional specifications the operator we are going to measure in the end, the number of evaluations, and the way the gradients are going to be computed.

```python
outputs = tfq.layers.PQC(tf_circuit,         # Circuit to be transformed into tf layer
                         1-op,               # Output of the layer
                         repetitions=1024,
                         differentiator=tfq.differentiators.ParameterShift())
```

Next, we can instantiate a model, taking an arbitrary input and outputting the result of the measurement of $M$

```python
model = tf.keras.Sequential([
    keras.layers.Input(shape=(), dtype=tf.string),
    outputs
])
```

In order to compile and fit the model, one needs to define a loss function (the quantity to optimize) and an optimizer. We want to optimize the expectation value of $M$, this is, the output of the model. Tensorflow needs this, however, as a function ``f(real_values,predictions)``

```python
loss = lambda real, pred: pred    # Forget about any "real" values, just consider as loss the model's output

model.compile(loss=loss,
              optimizer=keras.optimizers.SGD(learning_rate=0.4)) # Same optimizer as the Pennylane case
```

Finally, the ``fit`` function is designed for classification problems, and thus it needs of some inputs and corresponding "true" outputs. Our variational quantum circuit has none of these, so we just insert empty attributes: the input will be an empty quantum circuit, which corresponds to setting the initial state to $|0000\rangle$; the output will be an empty array (you can choose anything you want, since the loss function will discard whatever you set)

```python
dummy_input  = tfq.convert_to_tensor([cirq.Circuit()])
dummy_output = np.array([[]])
```

An it is finally time to train!

```python
model.fit(dummy_input, dummy_output, epochs=steps)
```

    WARNING:tensorflow:AutoGraph could not transform <bound method ParameterShift.differentiate_sampled of <tensorflow.python.eager.function.TfMethodTarget object at 0x7ff2e854d450>> and will run it as-is.
    Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
    Cause: unexpected indent (<unknown>, line 77)

    Train on 1 samples
    Epoch 1/100
    WARNING: AutoGraph could not transform <bound method ParameterShift.differentiate_sampled of <tensorflow.python.eager.function.TfMethodTarget object at 0x7ff2e854d450>> and will run it as-is.
    Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
    Cause: unexpected indent (<unknown>, line 77)
    1/1 [==============================] - 7s 7s/sample - loss: 1.2593
    Epoch 2/100
    1/1 [==============================] - 7s 7s/sample - loss: 1.1460
    Epoch 3/100
    1/1 [==============================] - 8s 8s/sample - loss: 1.0508
    Epoch 4/100
    1/1 [==============================] - 7s 7s/sample - loss: 0.9326
    Epoch 5/100
    1/1 [==============================] - 7s 7s/sample - loss: 0.8359
    Epoch 6/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.7749
    Epoch 7/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.7402
    Epoch 8/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.7217
    Epoch 9/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.7085
    Epoch 10/100
    1/1 [==============================] - 7s 7s/sample - loss: 0.7012
    Epoch 11/100
    1/1 [==============================] - 7s 7s/sample - loss: 0.6929
    Epoch 12/100
    1/1 [==============================] - 7s 7s/sample - loss: 0.6855
    Epoch 13/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.6724
    Epoch 14/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.6670
    Epoch 15/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.6606
    Epoch 16/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.6538
    Epoch 17/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.6460
    Epoch 18/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.6372
    Epoch 19/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.6265
    Epoch 20/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.6167
    Epoch 21/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.6050
    Epoch 22/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.5884
    Epoch 23/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.5806
    Epoch 24/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.5713
    Epoch 25/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.5571
    Epoch 26/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.5430
    Epoch 27/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.5332
    Epoch 28/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.5229
    Epoch 29/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.5059
    Epoch 30/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.4883
    Epoch 31/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.4717
    Epoch 32/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.4580
    Epoch 33/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.4399
    Epoch 34/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.4233
    Epoch 35/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.4072
    Epoch 36/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.3921
    Epoch 37/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.3726
    Epoch 38/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.3511
    Epoch 39/100
    1/1 [==============================] - 7s 7s/sample - loss: 0.3335
    Epoch 40/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.3086
    Epoch 41/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.2837
    Epoch 42/100
    1/1 [==============================] - 11s 11s/sample - loss: 0.2695
    Epoch 43/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.2505
    Epoch 44/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.2305
    Epoch 45/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.2085
    Epoch 46/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.1870
    Epoch 47/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.1733
    Epoch 48/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.1528
    Epoch 49/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.1382
    Epoch 50/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.1270
    Epoch 51/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.1167
    Epoch 52/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.1011
    Epoch 53/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0903
    Epoch 54/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0767
    Epoch 55/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0640
    Epoch 56/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0532
    Epoch 57/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0449
    Epoch 58/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0386
    Epoch 59/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0327
    Epoch 60/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0273
    Epoch 61/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0225
    Epoch 62/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0181
    Epoch 63/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0132
    Epoch 64/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0107
    Epoch 65/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0098
    Epoch 66/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0078
    Epoch 67/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0063
    Epoch 68/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0049
    Epoch 69/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0044
    Epoch 70/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0044
    Epoch 71/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0029
    Epoch 72/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0020
    Epoch 73/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0020
    Epoch 74/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0020
    Epoch 75/100
    1/1 [==============================] - 6s 6s/sample - loss: 9.7656e-04
    Epoch 76/100
    1/1 [==============================] - 5s 5s/sample - loss: 9.7656e-04
    Epoch 77/100
    1/1 [==============================] - 6s 6s/sample - loss: 9.7656e-04
    Epoch 78/100
    1/1 [==============================] - 5s 5s/sample - loss: 4.8828e-04
    Epoch 79/100
    1/1 [==============================] - 6s 6s/sample - loss: 4.8828e-04
    Epoch 80/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0000e+00
    Epoch 81/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0000e+00
    Epoch 82/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0000e+00
    Epoch 83/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0000e+00
    Epoch 84/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0000e+00
    Epoch 85/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 86/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 87/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 88/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 89/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 90/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 91/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 92/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 93/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 94/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 95/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 96/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 97/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 98/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 99/100
    1/1 [==============================] - 5s 5s/sample - loss: 0.0000e+00
    Epoch 100/100
    1/1 [==============================] - 6s 6s/sample - loss: 0.0000e+00

    <tensorflow.python.keras.callbacks.History at 0x7ff2cd65ab10>

Like in the Hello World example we can extract the wave function we see that we get the equal superposition state with some global phase.

```python
simulator = cirq.Simulator()
dictionary = {symb: model.trainable_variables[0].numpy()[i] for i, symb in enumerate(symbols)}
resolver = cirq.ParamResolver(dictionary)
resolved_circuit = cirq.resolve_parameters(tf_circuit, resolver)
output_state_vector = simulator.simulate(tf_circuit, resolver).final_state
output_state_vector
```

    array([-0.00286266+0.01872098j, -0.00046927+0.2587476j ,
           -0.0138061 +0.10716106j,  0.0847271 -0.19069251j,
            0.05543072-0.0915546j ,  0.00039238-0.29273975j,
           -0.02523318+0.16849318j,  0.265944  -0.46552533j,
            0.08174167+0.0482669j ,  0.17641312+0.06576671j,
            0.11405597+0.04105367j, -0.3545978 -0.23970872j,
            0.03512595-0.02338716j,  0.09215477-0.29585007j,
           -0.06562621+0.1250625j ,  0.2509636 -0.20192285j], dtype=complex64)

```python

```


# Introdução - movimento Browniano geométrico

Um processo estocástico $\{S_t\}_{t \geq 0}$ pode ser modelado como um mBg (movimento Browniano geométrico) se satisfazer a  SDE:

$$dS_t = \mu S_t dt + \sigma S_t dB_t$$

Neste caso, a solução da SDE acima é dada por $S_t = S_0 e^{X(t)}$, onde $X(t) = \left(\mu - \frac{\sigma^2}{2} \right)t + \sigma B_t$ e $B_t$ é um movimento Browniano, ou seja, $X(t)$ é um movimento Browniano com drift $\left(\mu - \frac{\sigma^2}{2}\right)$.

## Propriedade

$\quad$ i. $\lim_{t \to +\infty} \mathbb{P}\left(S_t = +\infty \mid \mu - \frac{\sigma^2}{2} > 0\right) = 1$.

# Estratégia

O backtest abaixo tem a seguinte ideia, mensalmente iremos estimar os parâmetros do modelo movimento Browniano geométrico e iremos rankear os ativos do maior para o menor com base na seguinte fórmula:

$$\frac{\mu}{\sigma^2}$$

A ideia é que queremos os ativos com maior drift e menor volatilidade e aproveitar-se da propriedade i citada anteriormente na introdução. Obsreve que $\mu$ e $\sigma$ são os parâmetros do modelo. Selecionaremos o top n ativos com maior ranking e aplicaremos a fronteira eficiente nesse conjunto, a estratégia fará o rebalanceamento dos ativos mensalmente.

```python
from __future__ import (absolute_import, division, print_function,
                        unicode_literals)
from collections import OrderedDict

#library for filtering signals
import statsmodels.api as sm

#Numerical operations
import numpy as np
from numpy import log as ln
from math import floor, isnan

#Symbolic operations
import sympy as sy
from sympy import symbols, lambdify
from sympy.functions.elementary.exponential import exp

#Datetime operations
import datetime

#Import backtrader platform
import backtrader as bt
import backtrader.analyzers as btanalyzers

#Module to analyze portfolio
import pyfolio as pf

#Ploting library
import matplotlib.pyplot as plt

#operating system library
import os

#Optimizations
from scipy.optimize import minimize, Bounds, LinearConstraint
from scipy.stats import norm as N

#portfolio analyzer
import quantstats as qs

#data analysis library
import pandas as pd

#debugging library
from ipdb import set_trace

class Portfolio():
    """
        Class portfolio, retrieve optimal portfolio weights based on Markowitz efficient frontier model.

        Inputs:
            μ - assets log return vector
            Σ - assets variance-covariance matrix
            minimize_method - method to optimize (std = standard deviation, neg_sharpe = negative sharpe)
            long_only(True/False) - allows short position if False (not implemented)
    """

    def __init__(self, μ, Σ, premiumRisk=0, minimize_method='std', long_only=True):

        n = len(μ)

        self.n = n
        self.μ = μ
        self.Σ = Σ
        self.premiumRisk = 0
        self.n_days = 1
        self.minimize_method = minimize_method
        self.premiumRisk = premiumRisk
        self.set_long_only = long_only

    def n_bound(self,n):
        return [(0,None) for i in range(n)]

    def std(self, w, μ, Σ):
        return (np.dot(w.T, np.dot(Σ, w)))**0.5

    def neg_sharpe(self, w, μ, Σ):
        return -(np.dot(w.T,μ))/((np.dot(w.T, np.dot(Σ, w)))**0.5)

    def optimize(self):
        μ = self.μ
        Σ = self.Σ
        n = self.n
        if self.minimize_method=='std':
            f=self.std
        if self.minimize_method=='neg_sharpe':
            f=self.neg_sharpe

        #Initial Guess
        w = np.random.rand(n)
        w /= np.sum(w)

        #Model Constraints

        bounds = self.n_bound(n)
        constraints = LinearConstraint(np.ones(n), lb=1, ub=1)

        #Optimization
        optimization = minimize(f, w, args=(μ, Σ),
                        method='SLSQP',
                        bounds=bounds, constraints=constraints)

        #Risk, Return Calculations
        optW = optimization['x']
        optReturn = np.dot(optW.T,self.μ)
        optRisk = np.sqrt(np.dot(optW.T, np.dot(self.Σ, optW)))

        self.optRisk = optRisk
        self.optReturn = optReturn

        return optW, optRisk, optReturn

    def plot(self):
        #Creation of a markowitz graph representation
        n = self.n
        PortfolioReturn = []
        PortfolioRisk = []
        for t in range(10000):
            ω = np.random.rand(n)
            ω /= (np.sum(ω)*1)
            Return = np.dot(ω.T, self.μ)
            Risk = np.sqrt(np.dot(ω.T, np.dot(self.Σ, ω)))
            PortfolioReturn.append(Return)
            PortfolioRisk.append(Risk)
        fig, ax = plt.subplots(figsize=(16,10), facecolor=(0.75, 0.75, 0.75))

        #plot
        ax.plot(PortfolioRisk, PortfolioReturn, 'o', label='Portfolios')
        ax.plot(self.optRisk, self.optReturn, 'bo', label='Optimal Point')

        #aesthetics
        ax.set_facecolor((0.1, 0.1, 0.1))
        ax.grid(True)
        ax.set_ylabel('Expected Return')
        ax.set_xlabel('Standard Deviation')
        ax.legend()

        plt.show()

class gBm(bt.Indicator):
    """
        Geometric Brownian Motion model.
                dS_t = μ S_t dt + σ S_t dB_t

        Crescimento Exponencial.
        o modelo tem um fit melhor para growth stocks como AAPL, TSLA, MGLU3, PRIO3 etc i.e.
        para ativos que tem um Crescimento Exponencial
    """
    lines = ('m', 'σ', 'μ', 'var', 'sl')
    params = dict(size=2**9+1,
                  sl_quantil=0.1,
                  sl_look_forward=5)

    def __init__(self):
        self.addminperiod(self.p.size)

    def next(self):
        self.S = self.data.close.get(0, size=self.p.size)
        self.estimateParams()

        self.lines.m[0] = self.m
        self.lines.σ[0] = self.σ
        self.lines.μ[0] = self.μ
        self.lines.var[0] = self.Var(self.p.sl_look_forward)
        self.lines.sl[0] = self.q(self.p.sl_quantil, self.p.sl_look_forward)

    def estimateParams(self):
        """
        Referência:
        "Estimation of Geometric Brownian Motion Parameters for Oil Price Analysis" C. Jakob et al.
        """
        S = self.S
        X = np.diff(np.log(S), n=1)
        m = X.mean() #mean
        σ = X.std() #standard deviation
        μ = m + ((σ**2)/2) #drift
        n = len(S)

        self.m = m
        self.σ = σ
        self.μ = μ
        self.n = n

    def E(self, t):
        """
        Referência:
        Ross, Sheldon M. (2014). "Variations on Brownian Motion".
        Introduction to Probability Models (11th ed.).
        """
        S = self.S
        S0 = S[-1]
        μ = self.μ
        return S0*np.exp(μ*t)

    def Var(self, t):
        """
        Referência:
        Ross, Sheldon M. (2014). "Variations on Brownian Motion".
        Introduction to Probability Models (11th ed.).
        """
        S = self.S
        S0 = S[-1]
        μ = self.μ
        σ = self.σ
        return (S0**2)*np.exp(2*μ*t)*(np.exp((σ**2)*t) - 1)

    def q(self, p, t):
        """
         quantil de St/S0 o qual é definido como:
                q(p) = exp( (μ - σ**2/2)*t + σ*np.sqrt(t)*inv_Φ(p))
         p ∈ (0,1)
        """
        #assert p>0 and p<1
        #assert type(t)==int

        σ = self.σ
        μ = self.μ

        mean = (μ - (σ**2/2))*t
        var = σ**2*t
        return np.exp(mean +  np.sqrt(var)*N.ppf(p, 0, 1))

class momentumStrat(bt.Strategy):
    """
        strategy class
    """
    params = dict(gBmSize=2**8+1, #number of observations to use in backtest to estimate parameters
                  enableLog=True, #enable log of buy and sell of assets
                  exposure=1, #exposure to the market
                  rebal_monthday=1, #which day in month to do the rebalancing
                  stocks_holding=10) #Max number of stocks holding

    def __init__(self):
        self.order = None #variable to track pending orders
        acoes = self.datas

        #construct line objects of gBm data
        gBms = [gBm(d, size=self.p.gBmSize) for d in acoes]
        driftAcoes = [gBm.μ for gBm in gBms]
        volatAcoes = [gBm.σ for gBm in gBms]

        #construct a line object of ranks
        self.rank = {d:drift/volat for d, drift, volat in zip(acoes, driftAcoes, volatAcoes)}

        self.driftAcoes = {d:drift for d, drift in zip(acoes, driftAcoes)}
        self.volatAcoes = {d:volat for d, volat in zip(acoes, volatAcoes)}

        #create a timer to execute the strategy montly
        self.add_timer(
            when=bt.Timer.SESSION_START,
            monthdays=[self.p.rebal_monthday],
            monthcarry=True  # if a day isn't there, execute on the next
        )

    def notify_order(self, order):
        '''Notify if an orde was executed/submited or not'''

        if order.status in [order.Submitted, order.Accepted]:
            # Buy/Sell order submitted/accepted to/by broker - Nothing to do
            return

        # Check if an order has been completed
        # Attention: broker could reject order if not enough cash
        if order.status in [order.Completed]:
            if order.isbuy():
                self.log(
                    'BUY EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                    (order.executed.price,
                     order.executed.value,
                     order.executed.comm))

                self.buyprice = order.executed.price
                self.buycomm = order.executed.comm

            else:  # Sell

                self.log('SELL EXECUTED, Price: %.2f, Cost: %.2f, Comm %.2f' %
                         (order.executed.price,
                          order.executed.value,
                          order.executed.comm))

            self.bar_executed = len(self)

        elif order.status==order.Canceled:
            self.log('Order Canceled. Price: {}'.format(order.price))
        elif order.status==order.Margin:
            self.log('Order Margin. Price: {}'.format(order.price))
        elif order.status==order.Rejected:
            self.log('Order Rejected. Price: {}'.format(order.price))

        self.order = None

    def log(self, txt, dt=None, doprint=False):
        '''Logging function fot this strategy'''
        if self.params.enableLog or doprint:
            dt = dt or self.datas[0].datetime.date(0)
            print('%s, %s' % (dt.isoformat(), txt))

    def getPortfolioParameters(self, datas:list):
        '''Get portfolio assets log returns vector and assets variance-covariance matrix'''
        S = np.column_stack(tuple(np.array(d).T for d in map(lambda x: x.close.get(0, self.p.gBmSize), datas)))
        S0 = S[0]
        X = np.diff(np.log(np.divide(S,S0)), n=1, axis=0)
        μ = X.mean(axis=0)
        Σ = np.cov(X, rowvar=False)

        n = len(datas)

        return μ, Σ

    def notify_timer(self, timer, when, *args, **kwargs):
        '''this function is called monthly, it is our strategy logic'''
        l = len(self)
        if l < self.p.gBmSize or self.order:
            return

        rank = {d:rank for (d, rank) in self.rank.items() if d.close.get(0, self.p.gBmSize)}

        ranks = sorted(
            rank.items(),  # get the (d, rank), pair
            key=lambda x: x[1][0],  # use rank (elem 1) and current time "0"
            reverse=True,  # highest ranked 1st ... please
        )

        top_ranks = ranks[:self.p.stocks_holding]
        datas = [data for (data, rank) in top_ranks]
        μ, Σ = self.getPortfolioParameters(datas)
        portfolio = Portfolio(μ, Σ, minimize_method='neg_sharpe')
        w, Risk, Return = portfolio.optimize()
        #portfolio.plot()

        opt_portfolio = [(data,weight) for (data,weight) in zip(datas,w)]
        current_positions = [d for d, pos in self.getpositions().items() if pos]

        #if we're not in market
        if len(current_positions)==0:
            for (d,weight) in opt_portfolio:
                if weight!=0:
                    self.order = self.order_target_percent(d, target=weight)
                    self.log('Buying: {} /---/ Rank: {:.4f} /---/ Weight: {:.3f}'.format(d._name,
                                                                                rank[d][0],
                                                                                weight))
            return

        #if we're in market
        else:

            # remove those no longer top ranked
            # do this first to issue sell orders and free cash
            for d in (d for d in current_positions if d not in datas):
                self.log('Closing: {} /---/ Rank: {:.4f}'.format(d._name, rank[d][0]))
                self.order_target_percent(d, target=0.0)

            # rebalance those already top ranked and still there
            for d in (d for d in current_positions if d in datas):
                i = datas.index(d)
                self.log('Rebalancing: {} /---/ Rank: {:.4f} /---/ Weight: {:.3f}'.format(d._name,
                                                                                rank[d][0],
                                                                                w[i]))
                self.order_target_percent(d, target=w[i])
                del datas[i]  # remove it, to simplify next iteration
                w = np.delete(w, i)

            # issue a target order for the newly top ranked stocks
            # do this last, as this will generate buy orders consuming cash
            for i,d in enumerate(datas):

                if w[i]!=0:
                    self.log('Buying: {} /---/ Rank: {:.4f} /---/ Weight: {:.3f}'.format(d._name,
                                                                                rank[d][0],
                                                                                w[i]))
                    self.order_target_percent(d, target=w[i])
                else:
                    continue

    def stop(self):
        self.log('Ending Value %.2f' %
                 (self.broker.getvalue()), doprint=True)

def runStrat(in_start, in_end, wf=False, best_set=None):
    '''create cerebro instance, set cash, add data, add strategy and run it'''
    cerebro = bt.Cerebro()

    #Set cash value
    cerebro.broker.set_cash(100000)

    #Set commission modelo
    cerebro.broker.setcommission(commission=0)

    cerebro.broker.set_checksubmit(False)

    data_font = 'data'

    #Adding data
    start=datetime.datetime.strptime(in_start, '%Y-%m-%d')
    end=datetime.datetime.strptime(in_end, '%Y-%m-%d')

    files = os.listdir(data_font)

    for file in files:
        #print(file)
        datapath='{}//'.format(data_font)+file
        data = bt.feeds.YahooFinanceCSVData(dataname=datapath,
                                            fromdate=start,
                                            todate=end,
                                            reverse=False)
        cerebro.adddata(data)

    #Add analyzers
    cerebro.addanalyzer(bt.analyzers.PyFolio, _name='pyfolio')
    cerebro.addanalyzer(btanalyzers.SharpeRatio, _name='sharpe')

    #Add Strategy
    if wf:
        cerebro.addstrategy(momentumStrat, wf=wf)
    else:
        cerebro.addstrategy(momentumStrat)

    #Run Algo
    results = cerebro.run(maxcpus=1)

    #Plot results
    #cerebro.plot()

    return results

def analyzeStrat(results, live):
    '''analyze strategy results'''
    strat = results[0]
    pyfoliozer = strat.analyzers.getbyname('pyfolio')
    returns, positions, transactions, gross_lev = pyfoliozer.get_pf_items()

    # extend pandas functionality with metrics, etc.
    qs.extend_pandas()
    returns.index = pd.to_datetime(returns.index.astype(str).str[:10])
    qs.reports.full(returns, '^BVSP')

if __name__=='__main__':
    live = '2021-01-01'
    r = runStrat('2005-01-03', '2021-06-02')
    analyzeStrat(r, live)
```


# Lecture 22: Transformations, Log-Normal, Convolutions, Proving Existence

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## Variance of Hypergeometric, con't

Returning to where we left off in Lecture 21, recall that we are considering $X \sim \operatorname{HGeom}(w, b, n)$ where $p = \frac{w}{w+b}$ and $w + b = N$.

\begin{align}
  Var\left( \sum_{j=1}^{n} X_j \right) &= \operatorname{Var}(X_1) + \dots + \operatorname{Var}(X_n) + 2 \, \sum_{i<j} \operatorname{Cov}(X_i, X_j) \\
  &= n \, Var(X_1) + 2 \, \binom{n}{2} \operatorname{Cov} (X_1, X_2) & \quad \text{symmetry, amirite?} \\
  &= n \, p \, (1-p) + 2 \, \binom{n}{2} \left( \frac{w}{w+b} \, \frac{w-1}{w+b-1} - p^2  \right) \\
  &= \frac{N-n}{N-1} \, n \, p \, (1-p) \\
  \\
  \text{where } \frac{N-n}{N-1} &\text{  is known as the finite population correction}
\end{align}

Note how this closely resembles the variance for a binomial distribution, except for scaling by that finite population correction.

Let's idiot-check this:

\begin{align}
  \text{let } n &= 1 \\
  \\
  \operatorname{Var}(X) &= \frac{N-1}{N-1} 1 \, p \, (1-p) \\
  &= p \, (1-p) & \quad \text{ ... just a Bernoulli, since we only sample once!} \\
  \\
  \text{let } N &\gg n \\
  \Rightarrow \frac{N-n}{N-1} &= 1
  \\
  \operatorname{Var}(X) &= \frac{N-n}{N-1} n \, p \, (1-p) \\
  &= n \, p \, (1-p) & \quad \text{ ... Binomial, we probably never sample same element twice!} \\
\end{align}

## Transformations

### Or a change of variables

A function of an r.v. is itself an r.v., and we can use LOTUS to find mean and variance. But what if we want more than just the mean and variance? What if we want to know the entire distribution (PDF)?

### Theorem

> Let $X$ be a continuous r.v. with PDF $f_X, Y = g(X)$.
> Given that $g$ is differentiable, and strictly increasing
> (at least on the region in which we are interested),
> then the PDF of $Y$ is given by
>
> \begin\{align\}
>   f_Y(y) &= f_X(x) \, \frac{dx}{dy} & \quad \text{ where } y = g(x) \text{ , } x = g^{-1}(y)
> \end\{align\}
>

And since we know from the [Chain Rule](https://en.wikipedia.org/wiki/Chain_rule) that $\frac{dx}{dy} = \left( \frac{dy}{dx} \right)^{-1}$, you can substitute $\left( \frac{dy}{dx} \right)^{-1}$ for $\frac{dx}{dy}$ if that makes things easier.

#### Proof

\begin{align}
  &\text{starting from the CDF...} \\
  \\
  F_Y(y) &= P(Y \le y) \\
  &= P \left(g(x) \le y \right) \\
  &= P \left(X \le g^{-1}(y) \right) \\
  &= F_X \left( g^{-1}(y) \right) \\
  &= F_X(x) \\
  \\
  &\text{and now differentiating to get the PDF...} \\
  \\
  \Rightarrow f_{Y}(y) &=  f_{X}(x) \frac{dx}{dy}
\end{align}

### Log-Normal

Now let's try applying what we now know about transformations to get the PDF of a [Log-Normal distribution](https://en.wikipedia.org/wiki/Log-normal_distribution).

Given the log-normal distribution $Y = e^{z}$, where $Z \sim (0,1)$, find the PDF.

Note that $\frac{dy}{dz} = e^z = y$.

\begin{align}
  f_Y(y) &= f_Z{z} \, \frac{dz}{dy} \\
  &= \frac{1}{\sqrt{2\pi}} \, e^{-\frac{lny^2}{2}} \, \frac{1}{y} & \quad \text{where }y \gt 0
\end{align}

## Transformations in $\mathbb{R}^n$

### Multi-dimensional Example

Here's a multi-dimensional example.

Given the distribution $\vec{Y} = g(\vec{X})$, where $g \colon \mathbb{R}^n \rightarrow \mathbb{R}^n$, with continuous joint PDF $\vec{X} = \{ X_1, \dots , X_n \}$.

What is the joint PDF of $Y$ in terms of the joint PDF $X$?

\begin{align}
  f_Y(\vec{y}) &= f_X(\vec{x}) \, | \frac{d\vec{x}}{d\vec{y}} | \\
  \\
  \text{where } \frac{d\vec{x}}{d\vec{y}} &=
    \begin{bmatrix}
      \frac{\partial x_1}{\partial y_1} & \cdots & \frac{\partial x_1}{\partial y_n} \\
      \vdots&\ddots&\vdots \\
      \frac{\partial x_n}{\partial y_1}& \cdots &\frac{\partial x_n}{\partial y_n}
    \end{bmatrix} & \text{... is the Jacobian} \\
    \\
  \text{and }  | \frac{d\vec{x}}{d\vec{y}} | &= \left| \, det \, \frac{d\vec{x}}{d\vec{y}} \, \right| & \quad \text{... absolute value of determinant of Jacobian}
\end{align}

Similar to the previous explanation on transformations,  you can substitute $\left( | \, \frac{d\vec{y}}{d\vec{x}} \, | \right)^{-1}$ for $\frac{d\vec{x}}{d\vec{y}}$ if that makes things easier.

## Convolutions

### Distribution for a Sum of Random Variables

Let $T = X + Y$, where $X,Y$ are independent.

\begin{align}
  P(T=t) &= \sum_{x} P(X=x) \, P(Y=t-x) & \quad \text{discrete case}\\
  \\
  f_T(t) &= \int_{-\infty}^{\infty} f_X(x) \, f_Y(t-x) \, dx & \quad \text{continuous case} \\
\end{align}

#### Proof of continuous case

\begin{align}
  &\text{starting from the CDF...} \\
  \\
  F_T(t) &= P(T \le t) \\
  &= \int_{-\infty}^{\infty} P(X + Y \le t \, | \, X=x) \, f_X(x) \, dx & \quad \text{ law of total probability} \\
  &= \int_{-\infty}^{\infty} P(Y \le t - x) \, f_X(x) \, dx \\
  &= \int_{-\infty}^{\infty} F_Y(t-x) \, f_X(x) \, dx \\
  \\
  &\text{and now differentiating w.r.t. } T \text{ ...} \\
  \\
  \Rightarrow f_{T}(t) &= \int_{-\infty}^{\infty} f_Y(t-x) \, f_X(x) \, dx
\end{align}

## Proving Existence

### Using Probability to Prove the Existence of Object with Desired Properties

Let us say that $A$ is our desired property.

Can we show that $P(A) \gt 0$ for a _random object_? For if $P(A) \gt 0$, it follows that there should be _at least one object with property $A$_.

Suppose each object has some associated "score". We can pick a random object, and use that to compute the average score. From there, we can reason that there must be an object where this score is $\ge \mathbb{E}(X)$

Suppose we have:

* 100 people
* 15 committees
* each committee has 20 people
* assume that each person is on 3 committees

Show that there exists 2 committees where a group of 3 people are on both committees (overlap $\ge 3$).

Rather than try to enumerate all possible committee permutations, find the average overlap of 2 _random_ committees using indicator random variables.

#### Proof

\begin{align}
  \text{let } \, I_1 &= \text{person 1 on both the randomly chosen committees} \\
  \\
  \text{then } \, P(I_1) &= \frac{\binom{3}{2}}{\binom{15}{2}} \\
  \\
  \mathbb{E}(overlap) &= 100 \, \frac{\binom{3}{2}}{\binom{15}{2}} & \quad \text{... by symmetry} \\
  &= 100 \, \frac{3}{105} \\
  &= \frac{20}{7} \\
  &= 2.857142857142857 \\
\end{align}

But if the average overlap is $\frac{20}{7}$, since overlap must be an integer, we can safely round up and assume that average overlap is 3. And so we conclude that there must be at least one pair of committees where the overlap $\ge 3$.

This is similar to how Shannon proved his theory on channel capacity.

----

View [Lecture 22: Transformations and Convolutions | Statistics 110](http://bit.ly/2wRz77T) on YouTube.


# Model Validation

## References

[1] [Information Theoretic Model Selection for Pattern Analysis](https://proceedings.mlr.press/v27/buhmann12a/buhmann12a.pdf)

[2] [Approximated sorting](https://link.springer.com/content/pdf/10.1007%2F978-3-642-40602-7_15.pdf)

```python
import numpy as np
import numpy
import matplotlib.pylab as plt
import time

import sklearn as skl
from functools import reduce

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import scipy.linalg as linalg
import scipy.stats as ss

# Make sure to install treelib in the slt-ce conda environment: conda install treelib
import treelib as tl

from sklearn.utils.validation import check_is_fitted
import sklearn.model_selection as model_selection
import sklearn.cluster as cluster
import sklearn.metrics as metrics

import sklearn.manifold as man
```

Main idea behind the Model Validation:

Weighted Approximation Set Coding (wASC) takes two sample sets $\mathbf X^{(1)}$ and $\mathbf X^{(2)}$ with the same inherent structure but different noise realizations. When optimizing a cost function on these data sets (learning) their global minima will typically be different (instability of solutions). Therefore, wASC introduces approximation weights to tackle this problem. Non-optimal solutions with large approximation weights $w_\beta(c,\mathbf X)\ge1-\varepsilon, \varepsilon \ll 1$ can be accepted as substitutes of the global minimizers. Therefore the inverse computational tenperature $\beta$ must be chosen suitably based on the noise level that is inherent in the data.

In a learning-theoretic sense, the set of solutions with large weights generalizes significantly better than the set of global minimizers, provided that $\beta$ is suitably chosen. wASC is a concept developed to determine such an appropriate scale $\beta$. The two data sets $\mathbf X^{(q)},q\in\{ 1,2 \}$ define two weight sets $w_\beta (c,\mathbf X^{(q)})$. These weights give rise to the two weight sums $\mathcal Z_q$ and the joint weight sum $\mathcal Z_{12}$:

\begin{equation}
\mathcal Z_q := \mathcal Z(\mathbf X^{(q)}) = \sum_{c \in \mathcal C(\mathbf X^{(q)})} \exp \left( -\beta \Delta R(c,\mathbf X^{(q)}) \right), \quad q=1,2
\end{equation}
\begin{equation}
\mathcal Z_{12} := \mathcal Z(\mathbf X^{(1)},\mathbf X^{(2)}) = \sum_{c \in \mathcal C(\mathbf X^{(2)})} \exp \left( -\beta \left( \Delta R(c,\mathbf X^{(1)}) + \Delta R(c,\mathbf X^{(2)}) \right) \right)
\end{equation}

where the combined weight sum quantifies how well the solution $c$ minimizes costs on both datasets. $\mathcal Z_q$ is the number of statistically indistinguishable data patterns that approximate the minimum of the objective function. While the global optimum $c^\perp(\mathbf X)$ can change whenever another random subset of the data is taken, the set of weights $\{w_\beta(c,\mathbf X)\}$ remains approximately invariant for a well-tuned $\beta$. Now the question is how one should choose such a $\beta$ so that  identifiability of $\{ w_\beta : w_\beta (c,\mathbf X) \ge 1 - \varepsilon \}$ is ensure in the presence of data fluctuations.

Based on a hypothetical communication scenario, [1] derive the optimal $\beta$ as the maximizer of the mutual information $\mathcal I_\beta(\tau_s\hat \tau)$. This result is derived from the error analysis of the coding protocol for the proposed communication scenario.

With two data sets $\mathbf X^{(1)}$ and $\mathbf X^{(2)}$ drawn from the same source, the cost function of a parametric clustering model with $K$ clusters reads

\begin{equation}
R(c,\mathbf X)=\sum_{i=1}^n \varepsilon_{i,c(i)} \quad \mathrm{with} \quad \forall i, c(i) \in \{ 1,...,K \},
\end{equation}

where $\varepsilon_{i,c(i)}$ indicates the cost of assigning object $i$ to cluster $c(i)$. For k-means we have $\varepsilon_{i,c(i)}=\| x_i - y_{c(i)} \|^2$. Hence, the mutual information $\mathcal I_\beta(\tau_s\hat \tau)$ for clustering is given by

\begin{equation}
\mathcal I_{\beta}
= \frac{1}{n} \log |\{ \tau_s \}
+ \frac{1}{n} \sum_{i=1}^N
\left(
\log \sum_{k=1}^K e^{-\beta \left( \varepsilon_{i,k}^{(1)} + \varepsilon_{i,k}^{(2)} \right) } - \log \sum_{k=1}^N e ^{-\beta \varepsilon_{i,k}^{(1)}} \sum_{k'=1}^K e^{-\beta \varepsilon_{i,k'}^{(2)} }.
\right)
\end{equation}

Eventually, the approximation capacity (mutual information of the best model, i.e. the solution of the model selection problem) is numerically determined as the maximum of $\mathcal I_{\beta}$ over $\beta$.

In general, the procedure is as follows:
- Randomly split the given data set into two subsets $\mathbf X^{(1)}$ and $\mathbf X^{(1)}$
- Compute the mutual information for each candidate cost function $R(c,\mathbf X) \in \mathcal R$ and maximize it with respect to $\beta$
- Select the cost function that achieves the highest capacity at the best resilution $\beta^\star$

For the special case of model selection for clustering we have:
- Compute the cardinality $| \{ \tau_s \} |$ of the set of possible transformations $\tau_s$ (hypothesis space). This is the space of all possible assignments of objects to sources - i.e. the permutation of objects. $| \{ \tau_s \} |$ is the number of all distinct clusterings on $\mathbf X^{(1)}$
- Calculate the induvidual and joint weight sums $\mathcal Z_1$, $\mathcal Z_2$ and $\mathcal Z_{12}$
- Maximize $\mathcal I_\beta$ with respect to $\beta$

We use the following data source:

$$p(\textbf{x})=\frac{1}{2}\mathcal{N}(\textbf{x} | \textbf{0}, \sigma_1^2\mathbb{I}) +
\frac{1}{2}\mathcal{N}(\textbf{x} | r\textbf{v},\sigma_2^2\mathbb{I}) $$

with $\mathbf{x}, \mathbf{v} \in \mathbb{R}^d$, $\lVert \mathbf{v} \rVert=1$ and $r\in\mathbb{R}_+$.
The following function generates data from this distribution:

```python
def generate_data(var_1, var_2, r, v, num_samples, plot_hist=False):
    """Generate data from the described model using provided parameters:
    Args:
            var_1 (number): varience of the first cluster
            var_2 (number): varience of the second cluster
            r (number): distance r to the center of the second cluster
            v (np.ndarray): direction v of the center of the second cluster
            num_samples (number): number of points to be generated

    Returns:
            data (np.ndarray): generated points
            labels (np.ndarray): true cluster assigment for the generated points
    """

    # Set-up
    n_dim = v.size
    n_components = 2
    numpy.random.seed(0x5eed)

    # Parameters of the mixture components
    cov = np.zeros((n_dim,n_dim,n_components))
    cov[:,:,0] = var_1 * np.identity(n_dim)
    cov[:,:,1] = var_2 * np.identity(n_dim)
    mean = np.zeros((n_dim,n_components))
    mean[:,0] = np.zeros(n_dim)
    mean[:,1] = r * v

    # Weight of each component, in this case all of them are 1/3
    weights = np.ones(num_components, dtype=np.float64) / float(num_components)

    # A stream of indices from which to choose the component
    labels = numpy.random.choice(num_components, size=num_samples, replace=True, p=weights)

    # Y is the multivariate mixture sample
    data = np.zeros((n_dim,len(labels)))
    counter = 0
    for i in labels:
        data[:,counter] = np.random.multivariate_normal(mean[:,i],cov[:,:,i],1)
        counter += 1

    # Plot histogram per dimension
    if plot_hist:
        for i in range(v.size):
            y = data[i,:]
            # Theoretical PDF plotting - generate the x and y plotting positions
            xs = np.linspace(y.min(), y.max(), 200)
            ys = np.zeros_like(xs)
            # Theoretical distribution
            for j in range(num_components):
                ys += ss.norm.pdf(xs, loc=mean[i,j], scale=np.sqrt(cov[i,i,j])) * weights[j]
            plt.plot(xs, ys)
            plt.hist(y, normed=True, bins="fd")
            plt.xlabel("x_{}".format(i))
            plt.ylabel("f(x)")
            plt.show()

    return data, labels

```

```python
# Test data generation

var_1 = 1
var_2 = 1
r = 3
v = np.array([1,1])
num_samples = 10000
num_components = 2
plot_hist = True
data, labels = generate_data(var_1, var_2, r, v, num_samples, plot_hist)

```

```python
# Generate and plot 2D points for different combinations of  σ1 ,  σ2  and  r.
# Colors indicate the cluster assigment.

plt.figure(figsize=(15,8))
colors = labels #+ 1
plt.scatter(data[0],data[1],c = colors)
plt.show()

```

In order to reproduce the results described in the reference paper [1], section 4.1, we need to calculate mutual information of two clusterings. See section 3 in [1] for more details.

```python
def mutual_information(X_1, X_2, y_1, y_2, t):
    """Compute mutual information of two clusterings
    Args:
        x_1 (np.ndarray): first dataset
        x_2 (np.ndarray): second dataset
        y_1 (np.ndarray): cluster centers for the first dataset
        y_2 (np.ndarray): cluster centers for the second dataset
        t (float): temperature T
    Returns:
        mutual_information (float): mutual information of x_1 and x_2
    """

    N_1 = X_1.shape[1]
    N_2 = X_2.shape[1]
    K_1 = y_1.shape[1]
    K_2 = y_2.shape[1]
    eps_1 = np.zeros((N_1,K_1))
    eps_2 = np.zeros((N_1,K_1))

    beta = 1/t
    card_tau_s = float(N_1**K_1)

    # Compute the cost matrices
    for i in range(N_1):
        for k in range(K_1):
            eps_1[i,k] = np.linalg.norm(X_1[:,i] - y_1[:,k])**2

    for i in range(N_2):
        for k in range(K_2):
            eps_2[i,k] = np.linalg.norm(X_2[:,i] - y_2[:,k])**2

    sum_data_logs = 0
    for i in range(N_1): # N_1 = N_2
        sum_cluster_exp_data12_i = 0
        sum_cluster_exp_data1_i = 0
        sum_cluster_exp_data2_i = 0
        for k in range(K_1): # K_1 = K_2
            sum_cluster_exp_data12_i += np.exp(-beta*(eps_1[i,k]+eps_2[i,k]))
            sum_cluster_exp_data1_i += np.exp(-beta*eps_1[i,k])
            sum_cluster_exp_data2_i += np.exp(-beta*eps_2[i,k])
        sum_data_logs += np.log2(sum_cluster_exp_data12_i) - np.log2(sum_cluster_exp_data1_i * sum_cluster_exp_data2_i)

    I_beta = np.log2(card_tau_s)/N_1
    I_beta += 1/N_1 * sum_data_logs

    return I_beta
```

```python
# Test of mutual information implementation (with two identical data sets)

var_1 = 1
var_2 = 1
r = 1
v = np.array([1, 1])
n_samples = 100
data, labels = generate_data(var_1, var_2, r, v, n_samples, False)
centers = np.array([[0, 0], [r, r]])

MI = mutual_information(data, data, centers, centers, 1)
print(MI)

```

    3.0995036748662224
    9.210340371976184
    -69.31471805599465
    -0.6010437768401846

We implement an extended Deterministic Annealing class from notebook 2 in this folder so that it logs the MI score for every evaluated temperature T. Note that the mutual information score is defined for a hard assignment while deterministic annealing results in a probabilistic one.

```python
class DeterministicAnnealingClustering(skl.base.BaseEstimator, skl.base.TransformerMixin):
    """Template class for DAC

    Attributes:
        cluster_centers_ (np.ndarray): Cluster centroids y_i (n_clusters, n_features)
        cluster_probabs_ (np.ndarray): Assignment probability vectors p(y_i | x) for each sample
                                       (n_samples, n_clusters)
        bifurcation_tree_ (treelib.Tree): Tree object that contains information about cluster evolution during
                                          annealing.

    Parameters:
        n_clusters (int): Maximum number of clusters returned by DAC.
        random_state (int): Random seed.
    """

    def __init__(self, n_clusters=8, random_state=42, metric="euclidian", record = True):
        self.n_clusters = n_clusters
        self.random_state = random_state
        self.metric = metric
        # Add more parameters, if necessary.
        self.record = record
        self.bifurcation_tree_ = tl.Tree()
        self.cluster_history = []
        self.eff_clusters = [1]
        self.expected_distortions = []
        self.temperatures = []

    def fit(self, X):
        """Compute DAC for input vectors X

        Preferred implementation of DAC as described in reference [1].
        Consider to use initialization and reseeding as in sklearn k-means for improved performance.

         Args:
            X (np.ndarray): Input array with shape (samples, n_features)

        Returns:
            self
        """
        #Initialization
        np.random.seed(self.random_state)
        rate = 0.95 #0.95 #alpha
        iterations = 130#100
        temp_sum = np.zeros((len(X[0]),len(X[0])))
        centroids = [(np.sum(X,axis=0)/len(X)).T]
        for j in range(len(X)): #Initial covariance matrix
            tc = np.reshape(X[j] - centroids[0],(-1,1)) @ np.reshape(X[j] - centroids[0],(1,-1))
            temp_sum += tc
        C = temp_sum/(len(X))
        eigen_value = linalg.eigvalsh(C, eigvals=(len(X[0])-1,len(X[0])-1))[0]
        T = eigen_value*3 #Start temperature, 3 > 2
        self.temperatures.append(T)
        t_min = T*(rate**iterations)
        K = 1
        probs = [1]
        self.cluster_history.append(centroids[0])
        self.bifurcation_tree_.create_node(tag = ".0", data = [[centroids[0]],[0],"+"], identifier = ".0")
        parent = [""]
        if self.metric == "euclidian":
            e_distort = [np.dot(X[row]-centroids[0],X[row]-centroids[0]) for row in range(len(X))]
            self.expected_distortions.append(np.sum(e_distort)/len(X))
            while T > t_min:
                probscop = probs.copy()
                squares = np.zeros((len(X),K))
                for w in range(len(X)):
                        for j in range(K):
                            squares[w][j] = np.dot(X[w]-centroids[j],X[w]-centroids[j])
                for i in range(K):
                    pyx = np.zeros(len(X))
                    for j in range(len(X)):
                        b = np.min(squares[j])/T # Shift to ensure no overflow
                        temp = probscop[i]*np.exp(-(squares[j][i]/T)+b) # enumrator of p(y|X_i)
                        Z = np.dot(probscop, np.exp(-squares[j]/T + b)) # Denominator
                        pyx[j] = temp/Z # P(y|x)
                    probs[i] = np.sum(pyx)/len(X)
                    temp_sum = np.matmul(pyx,X) # sum over x*p(y|x)
                    centroids[i] = temp_sum/(probs[i]*len(X)) # multiply by p(x) = 1/n
                T *= rate
                if K < self.n_clusters:
                    new_cs = 0 # New clusters added at this temperature
                    squares = np.zeros((len(X),K))
                    zx = np.zeros(len(X)) # Denominator of P(y|x_i) for each x_i
                    for r in range(len(X)):
                        for j in range(K):
                            squares[r][j] = np.dot(X[r]-centroids[j],X[r]-centroids[j])
                        b = np.min(squares[r])/T
                        zx[r] = np.dot(probs,np.exp(-squares[r]/T+b))
                    for i in range(K):
                        temp_sum = np.zeros((len(centroids[0]),len(centroids[0])))
                        for j in range(len(X)):
                            b = np.min(squares[j])/T
                            tc = np.reshape(X[j] - centroids[i],(-1,1)) @ np.reshape(X[j] - centroids[i],(1,-1))
                            expo = np.exp((-squares[j][i]/T)+b)
                            temp_sum += (expo/zx[j])*tc
                        C = probs[i]*temp_sum/(len(X))
                        eigen_value = linalg.eigvalsh(C, eigvals=(len(X[0])-1,len(X[0])-1))[0]
                        if T <= 2*eigen_value and K + new_cs < self.n_clusters:
                            centroids.append(centroids[i] + np.random.normal(0,0.001,size=(len(X[0]))))
                            probs[i] /= 2
                            probs.append(probs[i])
                            new_cs += 1
                            if self.record:
                                parent.append(parent[i] + "." + str(i))
                                parent[i] += "." + str(i)
                                self.bifurcation_tree_.create_node(tag = parent[i] + "." + str(i),
                                                                data = [[],[],"+"],
                                                                identifier = (parent[i] + "." + str(i)),
                                                                parent = parent[i])
                                self.bifurcation_tree_.create_node(tag = parent[i] + "." + str(len(centroids)-1),
                                                               data = [[],[],"-"],
                                                               identifier = (parent[i]  + "." + str(len(centroids)-1)),
                                                               parent = parent[i])
                    K += new_cs

                if self.record:
                    for node in range(K):
                        current_node = self.bifurcation_tree_.get_node(parent[node] + "." + str(node))
                        current_node.data[0].append(centroids[node])
                        if current_node.identifier == ".0":
                            current_node.data[1].append(0)
                        else:
                            node_parent = self.bifurcation_tree_.get_node(parent[node])
                            current_node.data[1].append(np.sqrt(np.dot(centroids[node]-node_parent.data[0][-1],centroids[node]-node_parent.data[0][-1])))

                    self.cluster_history.append(centroids.copy())
                    self.eff_clusters.append(K)

                    e_distort = 0
                    squares = np.zeros((len(X),K))
                    for i in range(len(X)):
                        for j in range(K):
                            squares[i][j] = np.dot(X[i]-centroids[j],X[i]-centroids[j])
                    for row in range(len(X)):
                        temp = []
                        dist = []
                        b = np.min(squares[row])/T
                        for col in range(K):
                            square = squares[row][col]
                            temp.append(probs[col]*np.exp(-square/T+b)/np.dot(probs,np.exp(-squares[row]/T + b)))
                            dist.append(square)
                        e_distort += np.dot(dist,temp)
                    e_distort /= len(X)
                    self.expected_distortions.append(e_distort)
                    self.temperatures.append(T)

            # T = 0 STEP

            squares = np.zeros((len(X),K))
            for i in range(len(X)):
                for j in range(K):
                    squares[i][j] = np.dot(X[i]-centroids[j],X[i]-centroids[j])
            closest_cluster = np.zeros(len(X))
            for row in range(len(squares)):
                closest_cluster[row] = np.argmin(squares[row])
            temp_clusters = np.zeros((K,len(X[0])))
            count_y = np.zeros(K)
            for closest in range(len(closest_cluster)):
                temp_clusters[int(closest_cluster[closest])] += X[closest]
                count_y[int(closest_cluster[closest])] += 1
            for cluster in range(K):
                temp_clusters[cluster] /= count_y[cluster]

            centroids = temp_clusters

        elif self.metric == "ratioscale":
            def distance(x,y):
                n = len(x)
                temp_sum = 0
                for i in range(n):
                    for j in range(n):
                        temp_sum += (x[i]*y[j])/(x[j]*y[i])
                return temp_sum/(n**2)

            def gradient(x,y):
                enumerator = np.zeros(len(x))
                denominator = 0
                for i in range(len(x)):
                        for j in range(len(x)):
                            denominator += x[i]*y[j]/(x[j]*y[i])
                for k in range(len(x)):
                    summand_1 = 0
                    summand_2 = 0
                    for i in range(len(x)):
                        if i != k:
                            qoutient = (x[i]/y[i])
                            summand_1 += qoutient
                            summand_2 += (1/qoutient)
                    enumerator[k] = (1/x[k])*summand_1 - (x[k]/(y[k]**2))*summand_2
                return enumerator/denominator
            while T > t_min:
                probscop = probs.copy()
                pyx = np.zeros((len(X),K))
                z = np.zeros(len(X))
                for row in range(len(X)):
                    for j in range(K):
                        pyx[row][j] = (distance(X[row],centroids[j])**(-1/T))
                    z[row] = np.sum(pyx[row])
                for i in range(K):
                    probs[i] = sum([pyx[row][i]/z[row] for row in range(len(X))])/len(X)
                    echelons = 30
                    learning_rate = 0.02
                    #Gradient Descent
                    for iteration in range(echelons):
                        grad = np.zeros(len(X[0]))
                        samp = np.random.choice(len(X), 125, replace=False)
                        for j in samp:
                            grad += (pyx[j][i]/z[j])*gradient(X[j],centroids[i])
                        centroids[i] -= learning_rate*grad/len(X)
                        learning_rate *= 0.975
                        for element in range(len(X[0])):
                            if centroids[i][element] <= 0:
                                centroids[i][element] = 0.00000001
                        for row in samp:
                            pyx[row][i] = (distance(X[row],centroids[i])**(-1/T))
                            z[row] = np.sum(pyx[row])
                        probs[i] = sum([pyx[r][i]/z[r] for r in range(len(X))])/len(X)
                if K < self.n_clusters:
                    new_cs = 0 #New clusters added at this temperature
                    pyx = np.zeros((len(X),K))
                    for row in range(len(X)):
                        for j in range(K):
                            pyx[row][j] = ((distance(X[row],centroids[j])**(-1/T)))
                        pyx[row] /= np.sum(pyx[row])
                    for i in range(K):
                        temp_sum = np.zeros((len(centroids[0]),len(centroids[0])))
                        for j in range(len(X)):
                            tc = np.reshape(X[j] - centroids[i],(-1,1)) @ np.reshape(X[j] - centroids[i],(1,-1))
                            temp_sum += pyx[j][i]*tc
                        C = temp_sum/(probs[i]*len(X))
                        eigen_value = linalg.eigvalsh(C, eigvals=(len(X[0])-1,len(X[0])-1))[-1]
                        if T <= 2*eigen_value and K + new_cs < self.n_clusters:
                            centroids.append(centroids[i] + np.abs(np.random.normal(0,1,size=(len(X[0])))))
                            probs[i] /= 2
                            probs.append(probs[i])
                            new_cs += 1
                    K += new_cs
                self.cluster_history.append(centroids.copy())
                self.eff_clusters.append(K)

                e_distort = 0

                for row in range(len(X)):
                    temp = []
                    dist = []
                    for col in range(K):
                        square = distance(X[row],centroids[col])
                        temp.append(square**(-1/T))
                        dist.append(np.log(square))
                    e_distort += (np.dot(dist,temp)/sum(temp))
                e_distort /= len(X)
                self.expected_distortions.append(e_distort)
                self.temperatures.append(T)
                T *= rate
            pyx = np.zeros((len(X),K))
            for row in range(len(X)):
                mini = 100000000
                classi = ""
                for i in range(K):
                    disti = distance(X[row], centroids[i])
                    if disti < mini:
                        mini = disti
                        classi = i
                pyx[row][classi] += 1
            for i in range(K):
                probs[i] = sum([pyx[row][i] for row in range(len(X))])/len(X)
                echelons = 40
                learning_rate = 0.02
                #Gradient Descent
                for iteration in range(echelons):
                    grad = np.zeros(len(X[0]))
                    #samp = np.random.choice(len(X), 75, replace=False)
                    for j in range(len(X)):
                        grad += pyx[j][i]*gradient(X[j],centroids[i])
                    centroids[i] -= learning_rate*grad/len(X)
                    learning_rate *= 0.975

            # code for extension
        if not(self.record):
            self.eff_clusters.append(K)
        self.cluster_centers_ = centroids
        self.T_min = T
        return self

    def predict(self, X):
        """Predict assignment probability vectors for each sample in X.

        Args:
            X (np.ndarray): Input array with shape (new_samples, n_features)

        Returns:
            P (np.ndarray): Assignment probability vectors (new_samples, n_clusters)
        """
        probs = []
        for row in range(len(X)):
            temp = np.zeros(self.eff_clusters[-1])
            for j in range(self.eff_clusters[-1]):
                temp[j] = np.exp(-np.sqrt(np.dot(X[row] - self.cluster_centers_[j],X[row] - self.cluster_centers_[j]))/self.T_min)
            probs.append(temp/sum(temp))

        P = np.array(probs)

        return P

    def transform(self, X):
        """Transform X to a cluster-distance space.

        In the new space, each dimension is the distance to the cluster centers.

        Args:
            X (np.ndarray): Input array with shape (new_samples, n_features)

        Returns:
            Y (np.ndarray): Cluster-distance vectors (new_samples, n_clusters)
        """
        check_is_fitted(self, ["cluster_centers_"])
        Y = np.zeros((len(X),self.eff_clusters[-1]))
        def distance(x,y):
            n = len(x)
            temp_sum = 0
            for i in range(n):
                for j in range(n):
                    temp_sum += (x[i]*y[j])/(x[j]*y[i])
            return np.log((1/(n**2)*temp_sum))
        for row in range(len(X)):
            for j in range(self.eff_clusters[-1]):
                if self.metric == "euclidian":
                    Y[row][j] = np.sqrt(np.dot(X[row]-self.cluster_centers_[j],X[row]-self.cluster_centers_[j]))
                elif self.metric == "ratioscale":
                    Y[row][j] = distance(X[row],self.cluster_centers_[j])

        return Y

    def plot_bifurcation(self):
        """Show the evolution of cluster splitting"""
        check_is_fitted(self, ["bifurcation_tree_"])

        paths = self.bifurcation_tree_.paths_to_leaves()
        dists = []
        y_axis = [np.log(self.temperatures[0]/self.temperatures[i]) for i in range(len(self.temperatures))]
        used = set()
        for path in paths:
            temp = []
            for node in path:
                current_node = self.bifurcation_tree_.get_node(node)
                new_distances = current_node.data[1]
                if len(temp) > 0:
                    if current_node.data[2] == "+":
                        temp.extend([temp[-1] + di for di in new_distances])
                    else:
                        temp.extend([temp[-1] - di for di in new_distances])
                else:
                    temp.extend(new_distances)
            dists.append(temp)
        plt.figure()
        for di in dists:
            plt.plot(di,y_axis)

        return None

def postprocess(y_DAC, y_kmeans):
    """
    Calculates the most likely cluster assignement of DAC

    Transform the cluster label to match kmeans cluster labeling.

    The method is needed, so one can compare kmeans with DAC using the same labeling of the clusters.
    Args:
        y_DAC (np.array): DAC's probability of assigning to cluster
        y_kmeans (np.array): cluster assignment of kmeans
    Returns:
        Y (np.array): Cluster assignement of DAC, with same cluster labeling as kmeans
    """
    y_DAC_hard = np.argmax(y_DAC, axis=1) #which cluster is most likely
    n_clusters = len(np.unique(y_DAC_hard)) #how many clusters exists
    dac2kmeans = []
    for cluster in range(n_clusters):
        argmax = np.argmax(y_DAC[:, cluster]) #Most likely to belong to cluster according to DAC
        dac2kmeans.append(y_kmeans[argmax]) #aligning cluster labeling with kmeans, assuming most likely DAC matches kmeans
    y_DAC_new = []
    for dac_label in y_DAC_hard:
        y_DAC_new.append(dac2kmeans[dac_label]) #Appending kmeans cluster labeling based on DAC labeling
    return np.array(y_DAC_new)

```

Now we fix the parameters $\sigma_1=\sigma_2=\sigma$ and produce a plot of the mutual information (MI) vs. temperature for different $r$ (as in their figure 2b).

```python
%%time

var_1 = 1
var_2 = 1
v = np.array([1, 1])
n_samples = 100

plt.figure(figsize=(15,8))
for r in [1.1,1.5,2,2.5]:

    print("\nr = {}".format(r))

    data, labels = generate_data(var_1, var_2, r, v, n_samples, False)
    centers = np.array([[0, 0], [r, r]])

    X_1 = np.transpose(data[:,:int(n_samples/2)])
    X_2 = np.transpose(data[:,int(n_samples/2):])
    labels_1 = labels[:int(n_samples/2)]
    labels_2 = labels[int(n_samples/2):]

    DAC_1 = DeterministicAnnealingClustering(n_clusters=2, random_state=42, record = True)
    DAC_1.fit(X_1)
    labels_DAC_1 = DAC_1.predict(X_1)

    DAC_2 = DeterministicAnnealingClustering(n_clusters=2, random_state=42, record = True)
    DAC_2.fit(X_2)
    labels_DAC_2 = DAC_2.predict(X_2)

    num_temperatures_1 = len(DAC_1.cluster_history)
    num_temperatures_2 = len(DAC_2.cluster_history)
    num_temperatures = np.minimum(num_temperatures_1,num_temperatures_2)

    mi_history = np.zeros(num_temperatures)
    temp_history = np.zeros(num_temperatures)

    for k in range(num_temperatures):

        centers_1 = np.transpose(np.vstack(DAC_1.cluster_history[k]))
        centers_2 = np.transpose(np.vstack(DAC_2.cluster_history[k]))
        temp = DAC_1.temperatures[k]

        mi = mutual_information(np.transpose(X_1), np.transpose(X_2), np.transpose(centers_1), np.transpose(centers_2), temp)

        if np.isnan(mi):
            mi = -100
        mi_history[k] = mi
        temp_history[k] = temp

    mi_history = mi_history[2:-10]
    temp_history = temp_history[2:-10]

    mi_max_ind = np.argmax(mi_history)
    mi_max = mi_history[mi_max_ind]
    temp_max = temp_history[mi_max_ind]

    print("mi_max = {}".format(mi_max))
    print("temp_max = {}".format(temp_max))
    pp = plt.plot(temp_history,mi_history,label="r = {}".format(r))
    plt.plot([temp_max,temp_max],[-0.8,mi_max],color=pp[0].get_color(),linestyle='--',marker="x")

    mat_1 = metrics.confusion_matrix(labels_1, postprocess(labels_DAC_1, labels_1))
    mat_2 = metrics.confusion_matrix(labels_2, postprocess(labels_DAC_2, labels_2))
    print("Confusion matrices: ")
    print(mat_1)
    print(mat_2)

plt.ylim(bottom=-0.8,top=0.0)
plt.xlim(right=10.0)
plt.xlabel("T")
plt.ylabel("I")
plt.legend(loc='upper right')
plt.show()

```

Now we plot of the maximum MI vs. the distance $r$.

```python
var_1 = 1
var_2 = 1
v = np.array([1, 1])
n_samples = 1000

plt.figure(figsize=(15,8))
#for r in [1.1,1.5,2,2.5]:

for n_samples in [50,100,500,1000]:

    mi_max_list = list()
    r_list = np.arange(1,5,0.2)

    for r in r_list:

        data, labels = generate_data(var_1, var_2, r, v, n_samples, False)
        centers = np.array([[0, 0], [r, r]])

        X_1 = np.transpose(data[:,:int(n_samples/2)])
        X_2 = np.transpose(data[:,int(n_samples/2):])
        labels_1 = labels[:int(n_samples/2)]
        labels_2 = labels[int(n_samples/2):]

        DAC_1 = DeterministicAnnealingClustering(n_clusters=2, random_state=42, record = True)
        DAC_1.fit(X_1)
        labels_DAC_1 = DAC_1.predict(X_1)

        DAC_2 = DeterministicAnnealingClustering(n_clusters=2, random_state=42, record = True)
        DAC_2.fit(X_2)
        labels_DAC_2 = DAC_2.predict(X_2)

        num_temperatures_1 = len(DAC_1.cluster_history)
        num_temperatures_2 = len(DAC_2.cluster_history)
        num_temperatures = np.minimum(num_temperatures_1,num_temperatures_2)

        mi_history = np.zeros(num_temperatures)
        temp_history = np.zeros(num_temperatures)

        for k in range(num_temperatures):

            centers_1 = np.transpose(np.vstack(DAC_1.cluster_history[k]))
            centers_2 = np.transpose(np.vstack(DAC_2.cluster_history[k]))
            temp = DAC_1.temperatures[k]

            mi = mutual_information(np.transpose(X_1), np.transpose(X_2), np.transpose(centers_1), np.transpose(centers_2), temp)

            if np.isnan(mi):
                mi = -100
            mi_history[k] = mi
            temp_history[k] = temp

        mi_history = mi_history[2:-10]
        temp_history = temp_history[2:-10]

        mi_max_ind = np.argmax(mi_history)
        mi_max = mi_history[mi_max_ind]
        temp_max = temp_history[mi_max_ind]

        mi_max_list.append(mi_max)

    plt.plot(r_list,mi_max_list,label="n_samples = {}".format(n_samples))

plt.xlabel("r")
plt.ylabel("max(I) (CAP)")
plt.legend(loc='upper left')
plt.show()

```

We analyze the behavior of the maximal MI as a function of the number of data samples $n$ and the data dimension $d$. To do so, we plot several curves MI vs. $r$ for different $n$ and $d$.

```python
var_1 = 1
var_2 = 1
v = np.array([1, 1])
n_samples = 100

plt.figure(figsize=(15,8))
for n_dim in [3,4,5,6]:

    v = np.ones(n_dim)

    mi_max_list = list()
    r_list = np.arange(1,5,0.5)

    for r in r_list:

        data, labels = generate_data(var_1, var_2, r, v, n_samples, False)

        X_1 = np.transpose(data[:,:int(n_samples/2)])
        X_2 = np.transpose(data[:,int(n_samples/2):])
        labels_1 = labels[:int(n_samples/2)]
        labels_2 = labels[int(n_samples/2):]

        DAC_1 = DeterministicAnnealingClustering(n_clusters=2, random_state=42, record = True)
        DAC_1.fit(X_1)
        labels_DAC_1 = DAC_1.predict(X_1)

        DAC_2 = DeterministicAnnealingClustering(n_clusters=2, random_state=42, record = True)
        DAC_2.fit(X_2)
        labels_DAC_2 = DAC_2.predict(X_2)

        num_temperatures_1 = len(DAC_1.cluster_history)
        num_temperatures_2 = len(DAC_2.cluster_history)
        num_temperatures = np.minimum(num_temperatures_1,num_temperatures_2)

        mi_history = -1000*np.ones(num_temperatures)
        temp_history = np.zeros(num_temperatures)

        for k in range(num_temperatures):

            clusters_t_1 = DAC_1.cluster_history[k]

            if len(clusters_t_1) == 2:

                centers_1 = np.transpose(np.vstack(DAC_1.cluster_history[k]))
                centers_2 = np.transpose(np.vstack(DAC_2.cluster_history[k]))
                temp = DAC_1.temperatures[k]

                n_clusters = len(DAC_1.cluster_history[k])

                mi = mutual_information(np.transpose(X_1), np.transpose(X_2), np.transpose(centers_1), np.transpose(centers_2), temp)

                if np.isnan(mi):
                    mi = -100
                mi_history[k] = mi
                temp_history[k] = temp

        mi_history = mi_history[2:-10]
        temp_history = temp_history[2:-10]

        mi_max_ind = np.argmax(mi_history)
        mi_max = mi_history[mi_max_ind]
        temp_max = temp_history[mi_max_ind]

        print(mi_max)
        mi_max_list.append(mi_max)

    plt.plot(r_list,mi_max_list,label="n_dim = {}".format(n_dim))

plt.xlabel("r")
plt.ylabel("max(I) (CAP)")
plt.legend(loc='upper right')
plt.show()

```


```python
import numpy as np
import sympy as sym
import pydae.build as db
```

https://ctms.engin.umich.edu/CTMS/index.php?aux=Extras_Leadlag

```python
params_dict = {'T_wo':5}  # parameters with default values

u_ini_dict = {'u_wo':0.0}  # input for the initialization problem
u_run_dict = {'u_wo':0.0}                  # input for the running problem, its value is updated

x_list = ['x_wo']  # dynamic states
y_ini_list = ['z_wo']          # algebraic states for the initialization problem
y_run_list = ['z_wo']        # algebraic for the running problem

sys_vars = {'params':params_dict,
            'u_list':u_run_dict,
            'x_list':x_list,
            'y_list':y_run_list}

exec(db.sym_gen_str())  # exec to generate the required symbolic varables and constants
```

```python
dx_wo = (u_wo - x_wo)/T_wo
g_wo =  (u_wo - x_wo) - z_wo
```

```python
sys = {'name':'washout',
       'params_dict':params_dict,
       'f_list':[dx_wo],
       'g_list':[g_wo],
       'x_list':[x_wo],
       'y_ini_list':y_ini_list,
       'y_run_list':y_run_list,
       'u_run_dict':u_run_dict,
       'u_ini_dict':u_ini_dict,
       'h_dict':{'u_wo':u_wo}}

sys = db.system(sys)
db.sys2num(sys)
```

    jacobians respect u = 0

```python

```

```python

```