### Lab 3: Expectation Maximization and Variational Autoencoder

### Machine Learning 2 (2019)

* The lab exercises can be done in groups of two people, or individually.
* The deadline is Tuesday, October 15th at 17:00.
* Assignment should be submitted through Canvas! Make sure to include your and your teammates' names with the submission.
* Attach the .IPYNB (IPython Notebook) file containing your code and answers. Naming of the file should be "studentid1\_studentid2\_lab#", for example, the attached file should be "12345\_12346\_lab1.ipynb". Only use underscores ("\_") to connect ids, otherwise the files cannot be parsed.

Notes on implementation:

* You should write your code and answers in an IPython Notebook: http://ipython.org/notebook.html. If you have problems, please ask.
* Use __one cell__ for code and markdown answers only!
    * Put all code in the cell with the ```# YOUR CODE HERE``` comment and overwrite the ```raise NotImplementedError()``` line.
    * For theoretical questions, put your solution using LaTeX style formatting in the YOUR ANSWER HERE cell.
* Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
* Large parts of you notebook will be graded automatically. Therefore it is important that your notebook can be run completely without errors and within a reasonable time limit. To test your notebook before submission, select Kernel -> Restart \& Run All.
$\newcommand{\bx}{\mathbf{x}} \newcommand{\bpi}{\mathbf{\pi}} \newcommand{\bmu}{\mathbf{\mu}} \newcommand{\bX}{\mathbf{X}} \newcommand{\bZ}{\mathbf{Z}} \newcommand{\bz}{\mathbf{z}}$

### Installing PyTorch

In this lab we will use PyTorch. PyTorch is an open source deep learning framework primarily developed by Facebook's artificial-intelligence research group. In order to install PyTorch in your conda environment go to https://pytorch.org and select your operating system, conda, Python 3.6, no cuda. Copy the text from the "Run this command:" box. Now open a terminal and activate your 'ml2labs' conda environment. Paste the text and run. After the installation is done you should restart Jupyter.

### MNIST data

In this Lab we will use several methods for unsupervised learning on the MNIST dataset of written digits. The dataset contains digital images of handwritten numbers $0$ through $9$. Each image has 28x28 pixels that each take 256 values in a range from white ($= 0$) to  black ($=1$). The labels belonging to the images are also included.
Fortunately, PyTorch comes with a MNIST data loader. The first time you run the box below it will download the MNIST data set. That can take a couple of minutes.
The main data types in PyTorch are tensors. For Part 1, we will convert those tensors to numpy arrays. In Part 2, we will use the torch module to directly work with PyTorch tensors.

```python
%pylab inline
import torch
from torchvision import datasets, transforms

train_dataset = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ]))

train_labels = train_dataset.train_labels.numpy()
train_data = train_dataset.train_data.numpy()
# For EM we will use flattened data
train_data = train_data.reshape(train_data.shape[0], -1)

```

    Populating the interactive namespace from numpy and matplotlib

    C:\Users\kztod\Anaconda3\envs\ml2labs\lib\site-packages\torchvision\datasets\mnist.py:43: UserWarning: train_labels has been renamed targets
      warnings.warn("train_labels has been renamed targets")
    C:\Users\kztod\Anaconda3\envs\ml2labs\lib\site-packages\torchvision\datasets\mnist.py:53: UserWarning: train_data has been renamed data
      warnings.warn("train_data has been renamed data")

## Part 1: Expectation Maximization
We will use the Expectation Maximization (EM) algorithm for the recognition of handwritten digits in the MNIST dataset. The images are modelled as a Bernoulli mixture model (see Bishop $\S9.3.3$):
$$
p(\bx|\bmu, \bpi) = \sum_{k=1}^K  \pi_k \prod_{i=1}^D \mu_{ki}^{x_i}(1-\mu_{ki})^{(1-x_i)}
$$
where $x_i$ is the value of pixel $i$ in an image, $\mu_{ki}$ represents the probability that pixel $i$ in class $k$ is black, and $\{\pi_1, \ldots, \pi_K\}$ are the mixing coefficients of classes in the data. We want to use this data set to classify new images of handwritten numbers.

### 1.1 Binary data (5 points)
As we like to apply our Bernoulli mixture model, write a function `binarize` to convert the (flattened) MNIST data to binary images, where each pixel $x_i \in \{0,1\}$, by thresholding at an appropriate level.

```python
def binarize(X):
    # YOUR CODE HERE
    return 1. * (X >= 0.5)
```

```python
# Test test test!
bin_train_data = binarize(train_data)
assert bin_train_data.dtype == np.float
assert bin_train_data.shape == train_data.shape

```

Sample a few images of digits $2$, $3$ and $4$; and show both the original and the binarized image together with their label.

```python
# YOUR CODE HERE
def visualize_digit(digit, n_samples=4):
    idxs = np.random.choice(np.where(train_labels == digit)[0], n_samples)
    fig, ax = plt.subplots(nrows=1, ncols=2*n_samples, figsize=(20, 20))

    i=0
    for idx in idxs:
        img = train_data[idx].reshape(28, 28)
        bin_img = binarize(img)
        ax[i].imshow(img, cmap='gray')
        ax[i].set_title(str(digit), fontsize=16)
        ax[i].axis('off')
        ax[i+1].imshow(bin_img, cmap='gray')
        ax[i+1].set_title('Binarized '+str(digit), fontsize=16)
        ax[i+1].axis('off')
        i+=2

for digits in [2, 3, 4]:
    visualize_digit(digits)

```

### 1.2 Implementation (40 points)
You are going to write a function ```EM(X, K, max_iter)``` that implements the EM algorithm on the Bernoulli mixture model.

The only parameters the function has are:
* ```X``` :: (NxD) array of input training images
* ```K``` :: size of the latent space
* ```max_iter``` :: maximum number of iterations, i.e. one E-step and one M-step

You are free to specify your return statement.

Make sure you use a sensible way of terminating the iteration process early to prevent unnecessarily running through all epochs. Vectorize computations using ```numpy``` as  much as possible.

You should implement the `E_step(X, mu, pi)` and `M_step(X, gamma)` separately in the functions defined below. These you can then use in your function `EM(X, K, max_iter)`.

```python
def E_step(X, mu, pi):
    # YOUR CODE HERE
    eps = 1e-10 # for stability in log operations
    gamma = np.exp(np.log(pi+eps) + X @ np.log(mu.T+eps) + (1 - X) @ np.log(1 - mu.T + eps))
    gamma  /= gamma.sum(axis=1)[:, np.newaxis]
    return gamma

```

```python
# Let's test on 5 datapoints
n_test = 5
X_test = bin_train_data[:n_test]
D_test, K_test = X_test.shape[1], 10

np.random.seed(2018)
mu_test = np.random.uniform(low=.25, high=.75, size=(K_test,D_test))
pi_test = np.ones(K_test) / K_test

gamma_test = E_step(X_test, mu_test, pi_test)
assert gamma_test.shape == (n_test, K_test)

```

```python
def M_step(X, gamma):
    # YOUR CODE HERE
    N_m = gamma.sum(axis=0)
    pi = N_m / X.shape[0]
    mu = gamma.T @ X / N_m[:, np.newaxis]
    return mu, pi

```

```python
# Oh, let's test again
mu_test, pi_test = M_step(X_test, gamma_test)

assert mu_test.shape == (K_test,D_test)
assert pi_test.shape == (K_test, )

```

```python
def EM(X, K, max_iter, threshold=1e-3, mu=None, pi=None):
    # YOUR CODE HERE
    if mu is None: mu = np.random.uniform(low=.15, high=.85, size=(K, shape(X)[1]))
    if pi is None: pi = np.ones(K) / K

    for i in range(1, max_iter+1):
        mu_old, pi_old = mu, pi

        gamma = E_step(X, mu, pi)
        mu, pi = M_step(X, gamma)

        delta_mu, delta_pi = np.linalg.norm(mu-mu_old), np.linalg.norm(pi-pi_old)
        if i%10 == 0:
            print('Epoch [{:4d}/{:4d}] | delta mu: {:6.4f} | delta pi: {:6.4f}'.format(
                i, max_iter, delta_mu, delta_pi))

        if delta_mu < threshold and delta_pi < threshold:
            print("\nConverged at iteration {}.".format(i))
            return gamma, mu, pi

    return gamma, mu, pi

```

### 1.3 Three digits experiment (10 points)
In analogue with Bishop $\S9.3.3$, sample a training set consisting of only __binary__ images of written digits $2$, $3$, and $4$. Run your EM algorithm and show the reconstructed digits.

```python
# YOUR CODE HERE
def sample_labels(labels, epochs=100, K=None, mu=None, pi=None, true_pi_init=False):
    if K is None: K = len(labels)
    print('-'*60+'\nTraining EM on labels {} and number of clusters K = {}\n'.format(labels, K)+'-'*60+'\n')

    labels_idxs = np.isin(train_labels, labels)
    subset_train_labels = train_labels[labels_idxs]
    subset_train_data = bin_train_data[labels_idxs]

    pi_true = []
    for label in labels:
        n_labels = np.isin(train_labels, label)
        pi_true.append(n_labels.sum())
    print('True mixing coefficients: {}'.format(np.array(pi_true)/np.array(pi_true).sum()))
    if true_pi_init: pi = np.array(pi_true)

    print('\nTraining Progress')
    gamma, mu, pi = EM(X=subset_train_data, K=K, max_iter=epochs, mu=mu, pi=pi)

    fig, ax = plt.subplots(nrows=1, ncols=K, figsize=(15, 15), dpi=100)
    for i in range(K):
        ax[i].imshow(mu[i].reshape(28,28), cmap='gray')
        ax[i].set_title('Parameters class: {}\n pi = {:0.3f}'.format(i, pi[i]), fontsize=K**(-1)//0.02)
        ax[i].axis('off')
    plt.show()

sample_labels([2, 3, 4])
```

Can you identify which element in the latent space corresponds to which digit? What are the identified mixing coefficients for digits $2$, $3$ and $4$, and how do these compare to the true ones?

#### YOUR ANSWER HERE

Looking at the results, we can clearly seperate the distinguish the digits between the elements in the latent space. They illustrated as 'gost' digits, which is expected, as we plot the mean distribution of every latent space.

Moreover, the mixing coefficients (appeared in the second row of the title of every class figure), follow a uniform distribution, imitating the original one, since it appears that we train our model on a balanced dataset.

### 1.4 Experiments (20 points)
Perform the follow-up experiments listed below using your implementation of the EM algorithm. For each of these, describe/comment on the obtained results and give an explanation. You may still use your dataset with only digits 2, 3 and 4 as otherwise computations can take very long.

#### 1.4.1 Size of the latent space (5 points)
Run EM with $K$ larger or smaller than the true number of classes. Describe your results.

```python
# YOUR CODE HERE
for K in [7, 2]:
    sample_labels(labels=[2, 3, 4], K=K)

```

#### YOUR ANSWER HERE

When we use a bigger number of class, the EM seems to further distinguishes the digits accoring to their variations. For example, it classifies differently the digits `2` where their bottom part is rounded from the standard one. Anothere case it the digit `4`, where it distinguishes thouse who are skewing to the left from all the others.

When we run the experiment with a lower number of class, specifically for `K = 2`, we observed that the digit `4` always make it as a seperate class, and the two others are merged. As the EM seems `2` and `3` closser (similarity) than any combination of the previous with `4`, it makes the algorithm treat `2` as a special case of `3` and it desides to merge them together into one class. Intuitively, one can arguee that the digit `3` is dominand over the `2`, as one can see the `3` cover a big area of `2`.
This can also be justified by the mixing coefficients of the latend space, as the class where `4` is visible has half the probability than the other (~ 0.35 and 0.65 = 2*0.325). Finally, on the class that `4` is not visible, we can see that shape is the digit `3`, with a more washed-out connected line from the middle part to the bottom, which is the part of `2` which makes it different from the former.

#### 1.4.2 Identify misclassifications (10 points)
How can you use the data labels to assign a label to each of the clusters/latent variables? Use this to identify images that are 'misclassified' and try to understand why they are. Report your findings.

```python
# YOUR CODE HERE

def sample_misclassifications(labels, epochs=100, K=None, n_mcls=7):
    if K is None: K = len(labels)
    print('-'*60+'\nTraining EM on labels {} and number of clusters K = {}\n'.format(labels, K)+'-'*60+'\n')

    labels_idxs = np.isin(train_labels, labels)
    subset_train_labels = train_labels[labels_idxs]
    subset_train_data = bin_train_data[labels_idxs]

    print('Training Progress')
    gamma, mu, pi = EM(subset_train_data, K, epochs)

    assigned_cluster = np.argmax(gamma, axis=1)
    cluster2class = {assigned_cluster[0]:4, assigned_cluster[1]:2, assigned_cluster[2]:3}
    pred_labels = np.array([cluster2class[i] for i in assigned_cluster])

    misclas_idxs = pred_labels!=subset_train_labels
    misclas_imgs = subset_train_data[misclas_idxs]
    misclas_labels = pred_labels[misclas_idxs]

    n_mcls = min(n_mcls, misclas_idxs.shape[0])
    rand_miss_indxs = np.random.choice(misclas_labels.shape[0]-1, n_mcls)

    fig, ax = plt.subplots(nrows=1, ncols=n_mcls, figsize=(15, 15), dpi=100)
    for i,miss_indx in enumerate(rand_miss_indxs):
        ax[i].imshow(misclas_imgs[miss_indx].reshape(28,28), cmap='gray')
        ax[i].set_title('Missclassified as: {}'.format(misclas_labels[miss_indx]), fontsize=n_mcls**(-1)//0.02)
        ax[i].axis('off')
    plt.show()

sample_misclassifications([2, 3, 4])
```

#### YOUR ANSWER HERE

We can seperate the reasons of misclassification into two classes: _type 1_ and _type 2_.

__Type 1__ refers to thoughs missclassifications that are equally hard for humans. These consist of digits that are result of poor handwriting. These types of errors are more or less expected.

__Type 2__ consists thouse digits that are disdinguisible from humans but the algorithm fails to cluster them correctly. These digits are rare variatios that the algorithm did not see ofter in the training set. This can be solved by putting more examples of these into the training set.

#### 1.4.3 Initialize with true values (5 points)
Initialize the three classes with the true values of the parameters and see what happens. Report your results.

```python
# YOUR CODE HERE
labels = [2, 3, 4]
sample_labels(labels, true_pi_init=True)
```

#### YOUR ANSWER HERE

By initializing the classes with the mixing coefficients, we are able to __converge much faster__ , compared with the uniform initialization. In our experiments, the improvement is of about half the number of iterations required. This is of course expected, since we are starting the optimization with a proposed solution with a higher quality than just random. Besides, the final mixture coefficients are close to the true mixture

## Part 2: Variational Auto-Encoder

A Variational Auto-Encoder (VAE) is a probabilistic model $p(\bx, \bz)$ over observed variables $\bx$ and latent variables and/or parameters $\bz$. Here we distinguish the decoder part, $p(\bx | \bz) p(\bz)$ and an encoder part $p(\bz | \bx)$ that are both specified with a neural network. A lower bound on the log marginal likelihood $\log p(\bx)$ can be obtained by approximately inferring the latent variables z from the observed data x using an encoder distribution $q(\bz| \bx)$ that is also specified as a neural network. This lower bound is then optimized to fit the model to the data.

The model was introduced by Diederik Kingma (during his PhD at the UVA) and Max Welling in 2013, https://arxiv.org/abs/1312.6114.

Since it is such an important model there are plenty of well written tutorials that should help you with the assignment. E.g: https://jaan.io/what-is-variational-autoencoder-vae-tutorial/.

In the following, we will make heavily use of the torch module, https://pytorch.org/docs/stable/index.html. Most of the time replacing `np.` with `torch.` will do the trick, e.g. `np.sum` becomes `torch.sum` and `np.log` becomes `torch.log`. In addition, we will use `torch.FloatTensor()` as an equivalent to `np.array()`. In order to train our VAE efficiently we will make use of batching. The number of data points in a batch will become the first dimension of our data tensor, e.g. A batch of 128 MNIST images has the dimensions [128, 1, 28, 28]. To check check the dimensions of a tensor you can call `.size()`.

### 2.1 Loss function
The objective function (variational lower bound), that we will use to train the VAE, consists of two terms: a log Bernoulli loss (reconstruction loss) and a Kullback–Leibler divergence. We implement the two terms separately and combine them in the end.
As seen in Part 1: Expectation Maximization, we can use a multivariate Bernoulli distribution to model the likelihood $p(\bx | \bz)$ of black and white images. Formally, the variational lower bound is maximized but in PyTorch we are always minimizing therefore we need to calculate the negative log Bernoulli loss and Kullback–Leibler divergence.

### 2.1.1 Negative Log Bernoulli loss (5 points)
The negative log Bernoulli loss is defined as,

\begin{align}
loss = - (\sum_i^D \bx_i \log \hat{\bx_i} + (1 − \bx_i) \log(1 − \hat{\bx_i})).
\end{align}

Write a function `log_bernoulli_loss` that takes a D dimensional vector `x`, its reconstruction `x_hat` and returns the negative log Bernoulli loss. Make sure that your function works for batches of arbitrary size.

```python
def log_bernoulli_loss(x_hat, x):
    # YOUR CODE HERE
    return torch.nn.functional.binary_cross_entropy(x_hat, x, reduction='sum')

```

```python
### Test test test
x_test = torch.FloatTensor([[0.1, 0.2, 0.3, 0.4], [0.5, 0.6, 0.7, 0.8], [0.9, 0.9, 0.9, 0.9]])
x_hat_test = torch.FloatTensor([[0.11, 0.22, 0.33, 0.44], [0.55, 0.66, 0.77, 0.88], [0.99, 0.99, 0.99, 0.99]])

assert log_bernoulli_loss(x_hat_test, x_test) > 0.0
assert log_bernoulli_loss(x_hat_test, x_test) < 10.0

```

### 2.1.2 Negative Kullback–Leibler divergence (10 Points)
The variational lower bound (the objective to be maximized) contains a KL term $D_{KL}(q(\bz)||p(\bz))$ that can often be calculated analytically. In the VAE we assume $q = N(\bz, \mu, \sigma^2I)$ and $p = N(\bz, 0, I)$. Solve analytically!

\begin{align*}
    \mathcal{KL}(q(\mathbf z)||p(\mathbf z))
    &= \int q(\mathbf z) \ln{\frac{q(\mathbf z)}{p(\mathbf z)}}d\mathbf x=\\
    &= \int q(\mathbf z) \left(\ln{q(\mathbf z)} - \ln{p(\mathbf z)}\right)d\mathbf x=\\
    &= \frac{1}{2} \int q(\mathbf z) (-\log |\sigma^2 I | + z^T z - (z-\mu)^T (\sigma^2 I )^{-1}(z-\mu) ) d\mathbf x \\
    &=\frac{1}{2} (-\log |\sigma^2 I | + \mu^T \mu + Tr(\sigma^2 I ) - Tr(I^{-1}I) ) \\
    &=- \frac{1}{2}\sum_i^D \log \sigma_i^2 + \frac{1}{2} \mu^T \mu + \frac{1}{2} \sum_i^D \sigma_i^2 - \frac{D}{2}
\end{align*}

Write a function `KL_loss` that takes two J dimensional vectors `mu` and `logvar` and returns the negative Kullback–Leibler divergence. Where `logvar` is $\log(\sigma^2)$. Make sure that your function works for batches of arbitrary size.

```python
def KL_loss(mu, logvar):
    # YOUR CODE HERE
    return - 0.5*torch.sum(1 + logvar - mu.pow(2) - logvar.exp())

```

```python
### Test test test
mu_test = torch.FloatTensor([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]])
logvar_test = torch.FloatTensor([[0.01, 0.02], [0.03, 0.04], [0.05, 0.06]])

assert KL_loss(mu_test, logvar_test) > 0.0
assert KL_loss(mu_test, logvar_test) < 10.0

```

### 2.1.3 Putting the losses together (5 points)
Write a function `loss_function` that takes a D dimensional vector `x`, its reconstruction `x_hat`, two J dimensional vectors `mu` and `logvar` and returns the final loss. Make sure that your function works for batches of arbitrary size.

```python
def loss_function(x_hat, x, mu, logvar):
    # YOUR CODE HERE
    return KL_loss(mu, logvar) + log_bernoulli_loss(x_hat, x)

```

```python
x_test = torch.FloatTensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
x_hat_test = torch.FloatTensor([[0.11, 0.22, 0.33], [0.44, 0.55, 0.66], [0.77, 0.88, 0.99]])
mu_test = torch.FloatTensor([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]])
logvar_test = torch.FloatTensor([[0.01, 0.02], [0.03, 0.04], [0.05, 0.06]])

assert loss_function(x_hat_test, x_test, mu_test, logvar_test) > 0.0
assert loss_function(x_hat_test, x_test, mu_test, logvar_test) < 10.0

```

### 2.2 The model
Below you see a data structure for the VAE. The modell itself consists of two main parts the encoder (images $\bx$ to latent variables $\bz$) and the decoder (latent variables $\bz$ to images $\bx$). The encoder is using 3 fully-connected layers, whereas the decoder is using fully-connected layers. Right now the data structure is quite empty, step by step will update its functionality. For test purposes we will initialize a VAE for you. After the data structure is completed you will do the hyperparameter search.

```python
from torch import nn
from torch.nn import functional as F

class VAE(nn.Module):
    def __init__(self, fc1_dims, fc21_dims, fc22_dims, fc3_dims, fc4_dims):
        super(VAE, self).__init__()

        self.fc1 = nn.Linear(*fc1_dims)
        self.fc21 = nn.Linear(*fc21_dims)
        self.fc22 = nn.Linear(*fc22_dims)
        self.fc3 = nn.Linear(*fc3_dims)
        self.fc4 = nn.Linear(*fc4_dims)

    def encode(self, x):
        # To be implemented
        raise Exception('Method not implemented')

    def reparameterize(self, mu, logvar):
        # To be implemented
        raise Exception('Method not implemented')

    def decode(self, z):
        # To be implemented
        raise Exception('Method not implemented')

    def forward(self, x):
        # To be implemented
        raise Exception('Method not implemented')

VAE_test = VAE(fc1_dims=(784, 4), fc21_dims=(4, 2), fc22_dims=(4, 2), fc3_dims=(2, 4), fc4_dims=(4, 784))

```

### 2.3 Encoding (10 points)
Write a function `encode` that gets a vector `x` with 784 elements (flattened MNIST image) and returns `mu` and `logvar`. Your function should use three fully-connected layers (`self.fc1()`, `self.fc21()`, `self.fc22()`). First, you should use `self.fc1()` to embed `x`. Second, you should use `self.fc21()` and `self.fc22()` on the embedding of `x` to compute `mu` and `logvar` respectively. PyTorch comes with a variety of activation functions, the most common calls are `F.relu()`, `F.sigmoid()`, `F.tanh()`. Make sure that your function works for batches of arbitrary size.

```python
def encode(self, x):
    embeddings = self.fc1.forward(x)

    embeddings = F.relu(embeddings)
    mu = self.fc21.forward(embeddings)

    logvar = self.fc22.forward(embeddings)
    return mu, logvar

```

```python
### Test, test, test
VAE.encode = encode

x_test = torch.ones((5,784))
mu_test, logvar_test = VAE_test.encode(x_test)

assert np.allclose(mu_test.size(), [5, 2])
assert np.allclose(logvar_test.size(), [5, 2])

```

### 2.4 Reparameterization (10 points)
One of the major question that the VAE is answering, is 'how to take derivatives with respect to the parameters of a stochastic variable?', i.e. if we are given $\bz$ that is drawn from a distribution $q(\bz|\bx)$, and we want to take derivatives. This step is necessary to be able to use gradient-based optimization algorithms like SGD.
For some distributions, it is possible to reparameterize samples in a clever way, such that the stochasticity is independent of the parameters. We want our samples to deterministically depend on the parameters of the distribution. For example, in a normally-distributed variable with mean $\mu$ and standard deviation $\sigma$, we can sample from it like this:

\begin{align}
\bz = \mu + \sigma \odot \epsilon,
\end{align}

where $\odot$ is the element-wise multiplication and $\epsilon$ is sampled from $N(0, I)$.

Write a function `reparameterize` that takes two J dimensional vectors `mu` and `logvar`. It should return $\bz = \mu + \sigma \odot \epsilon$.

```python
def reparameterize(self, mu, logvar):
    # YOUR CODE HERE
    epsilon = torch.randn_like(mu)
    return mu + torch.exp(0.5*logvar)*epsilon

```

```python
### Test, test, test
VAE.reparameterize = reparameterize
VAE_test.train()

mu_test = torch.FloatTensor([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]])
logvar_test = torch.FloatTensor([[0.01, 0.02], [0.03, 0.04], [0.05, 0.06]])

z_test = VAE_test.reparameterize(mu_test, logvar_test)

assert np.allclose(z_test.size(), [3, 2])
assert z_test[0][0] < 5.0
assert z_test[0][0] > -5.0

```

### 2.5 Decoding (10 points)
Write a function `decode` that gets a vector `z` with J elements and returns a vector `x_hat` with 784 elements (flattened MNIST image). Your function should use two fully-connected layers (`self.fc3()`, `self.fc4()`). PyTorch comes with a variety of activation functions, the most common calls are `F.relu()`, `F.sigmoid()`, `F.tanh()`. Make sure that your function works for batches of arbitrary size.

```python
def decode(self, z):
    # YOUR CODE HERE
    embeddings = self.fc3.forward(z)
    embeddings = F.relu(embeddings)
    x_hat = self.fc4.forward(embeddings)
    x_hat = F.sigmoid(x_hat)

    return x_hat

```

```python
# test test test
VAE.decode = decode

z_test = torch.ones((5,2))
x_hat_test = VAE_test.decode(z_test)

assert np.allclose(x_hat_test.size(), [5, 784])
assert (x_hat_test <= 1).all()
assert (x_hat_test >= 0).all()

```

    C:\Users\kztod\Anaconda3\envs\ml2labs\lib\site-packages\torch\nn\functional.py:1351: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
      warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")

### 2.6 Forward pass (10)
To complete the data structure you have to define a forward pass through the VAE. A single forward pass consists of the encoding of an MNIST image $\bx$ into latent space $\bz$, the reparameterization of $\bz$ and the decoding of $\bz$ into an image $\bx$.

Write a function `forward` that gets a a vector `x` with 784 elements (flattened MNIST image) and returns a vector `x_hat` with 784 elements (flattened MNIST image), `mu` and `logvar`.

```python
def forward(self, x):
    x = x.view(-1, 784)

    mu, logvar = self.encode(x)
    z = self.reparameterize(mu, logvar)
    x_hat = self.decode(z)

    return x_hat, mu, logvar

```

```python
# test test test
VAE.forward = forward

x_test = torch.ones((5,784))
x_hat_test, mu_test, logvar_test = VAE_test.forward(x_test)

assert np.allclose(x_hat_test.size(), [5, 784])
assert np.allclose(mu_test.size(), [5, 2])
assert np.allclose(logvar_test.size(), [5, 2])

```

### 2.7 Training (15)
We will now train the VAE using an optimizer called Adam, https://arxiv.org/abs/1412.6980. The code to train a model in PyTorch is given below.

```python
from torch.autograd import Variable

def train(epoch, train_loader, model, optimizer):
    model.train()
    train_loss = 0
    for batch_idx, (data, _) in enumerate(train_loader):
        data = Variable(data)
        optimizer.zero_grad()
        recon_batch, mu, logvar = model(data)
        loss = loss_function(recon_batch, data.view(-1, 784), mu, logvar)
        loss.backward()
        train_loss += loss.data
        optimizer.step()
        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader),
                loss.data / len(data)))

    print('====> Epoch: {} Average loss: {:.4f}'.format(
          epoch, train_loss / len(train_loader.dataset)))

```

Let's train. You have to choose the hyperparameters. Make sure your loss is going down in a reasonable amount of epochs (around 10).

```python
# Hyperparameters
def get_fc_dims(input_dim, hidden_dim, latent_dim):

    fc1_dims = (input_dim,hidden_dim)
    fc21_dims = (hidden_dim, latent_dim)
    fc22_dims = (hidden_dim, latent_dim)
    fc3_dims = (latent_dim, hidden_dim)
    fc4_dims = (hidden_dim, input_dim)

    return fc1_dims, fc21_dims, fc22_dims, fc3_dims, fc4_dims

fc1_dims, fc21_dims, fc22_dims, fc3_dims, fc4_dims = get_fc_dims(784, 512, 20)
lr = 1e-3
batch_size = 128
epochs = 10

```

```python
# This cell contains a hidden test, please don't delete it, thx
```

Run the box below to train the model using the hyperparameters you entered above.

```python
from torchvision import datasets, transforms
from torch import nn, optim

# Load data
train_data = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.ToTensor())

train_loader = torch.utils.data.DataLoader(train_data,
                                           batch_size=batch_size, shuffle=True, **{})

# Init model
VAE_MNIST = VAE(fc1_dims=fc1_dims, fc21_dims=fc21_dims, fc22_dims=fc22_dims, fc3_dims=fc3_dims, fc4_dims=fc4_dims)

# Init optimizer
optimizer = optim.Adam(VAE_MNIST.parameters(), lr=lr)

# Train
for epoch in range(1, epochs + 1):
    train(epoch, train_loader, VAE_MNIST, optimizer)

```

    Train Epoch: 1 [0/60000 (0%)]	Loss: 549.420288
    Train Epoch: 1 [12800/60000 (21%)]	Loss: 165.154419
    Train Epoch: 1 [25600/60000 (43%)]	Loss: 148.408249
    Train Epoch: 1 [38400/60000 (64%)]	Loss: 142.493729
    Train Epoch: 1 [51200/60000 (85%)]	Loss: 130.366180
    ====> Epoch: 1 Average loss: 160.5637
    Train Epoch: 2 [0/60000 (0%)]	Loss: 124.422745
    Train Epoch: 2 [12800/60000 (21%)]	Loss: 122.737106
    Train Epoch: 2 [25600/60000 (43%)]	Loss: 117.952682
    Train Epoch: 2 [38400/60000 (64%)]	Loss: 117.588188
    Train Epoch: 2 [51200/60000 (85%)]	Loss: 118.087013
    ====> Epoch: 2 Average loss: 119.4647
    Train Epoch: 3 [0/60000 (0%)]	Loss: 114.580696
    Train Epoch: 3 [12800/60000 (21%)]	Loss: 111.336197
    Train Epoch: 3 [25600/60000 (43%)]	Loss: 114.804596
    Train Epoch: 3 [38400/60000 (64%)]	Loss: 112.073029
    Train Epoch: 3 [51200/60000 (85%)]	Loss: 112.831673
    ====> Epoch: 3 Average loss: 113.3405
    Train Epoch: 4 [0/60000 (0%)]	Loss: 113.186501
    Train Epoch: 4 [12800/60000 (21%)]	Loss: 108.525803
    Train Epoch: 4 [25600/60000 (43%)]	Loss: 109.851463
    Train Epoch: 4 [38400/60000 (64%)]	Loss: 111.450203
    Train Epoch: 4 [51200/60000 (85%)]	Loss: 112.557625
    ====> Epoch: 4 Average loss: 110.7277
    Train Epoch: 5 [0/60000 (0%)]	Loss: 108.033661
    Train Epoch: 5 [12800/60000 (21%)]	Loss: 109.020096
    Train Epoch: 5 [25600/60000 (43%)]	Loss: 107.202797
    Train Epoch: 5 [38400/60000 (64%)]	Loss: 108.098267
    Train Epoch: 5 [51200/60000 (85%)]	Loss: 110.236450
    ====> Epoch: 5 Average loss: 109.2143
    Train Epoch: 6 [0/60000 (0%)]	Loss: 112.094574
    Train Epoch: 6 [12800/60000 (21%)]	Loss: 108.448944
    Train Epoch: 6 [25600/60000 (43%)]	Loss: 107.784309
    Train Epoch: 6 [38400/60000 (64%)]	Loss: 108.773636
    Train Epoch: 6 [51200/60000 (85%)]	Loss: 110.752167
    ====> Epoch: 6 Average loss: 108.0780
    Train Epoch: 7 [0/60000 (0%)]	Loss: 110.189331
    Train Epoch: 7 [12800/60000 (21%)]	Loss: 108.849312
    Train Epoch: 7 [25600/60000 (43%)]	Loss: 106.374802
    Train Epoch: 7 [38400/60000 (64%)]	Loss: 105.631760
    Train Epoch: 7 [51200/60000 (85%)]	Loss: 106.702667
    ====> Epoch: 7 Average loss: 107.3552
    Train Epoch: 8 [0/60000 (0%)]	Loss: 101.984901
    Train Epoch: 8 [12800/60000 (21%)]	Loss: 102.954422
    Train Epoch: 8 [25600/60000 (43%)]	Loss: 112.939835
    Train Epoch: 8 [38400/60000 (64%)]	Loss: 110.186157
    Train Epoch: 8 [51200/60000 (85%)]	Loss: 108.644188
    ====> Epoch: 8 Average loss: 106.7992
    Train Epoch: 9 [0/60000 (0%)]	Loss: 104.180588
    Train Epoch: 9 [12800/60000 (21%)]	Loss: 108.961700
    Train Epoch: 9 [25600/60000 (43%)]	Loss: 107.066277
    Train Epoch: 9 [38400/60000 (64%)]	Loss: 106.925812
    Train Epoch: 9 [51200/60000 (85%)]	Loss: 104.646286
    ====> Epoch: 9 Average loss: 106.2447
    Train Epoch: 10 [0/60000 (0%)]	Loss: 108.739685
    Train Epoch: 10 [12800/60000 (21%)]	Loss: 106.443497
    Train Epoch: 10 [25600/60000 (43%)]	Loss: 105.563385
    Train Epoch: 10 [38400/60000 (64%)]	Loss: 108.337875
    Train Epoch: 10 [51200/60000 (85%)]	Loss: 109.859657
    ====> Epoch: 10 Average loss: 105.8726

Run the box below to check if the model you trained above is able to correctly reconstruct images.

```python
### Let's check if the reconstructions make sense
# Set model to test mode
VAE_MNIST.eval()

# Reconstructed
train_data_plot = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.ToTensor())

train_loader_plot = torch.utils.data.DataLoader(train_data_plot,
                                           batch_size=1, shuffle=False, **{})

for batch_idx, (data, _) in enumerate(train_loader_plot):
    x_hat, mu, logvar = VAE_MNIST(data)
    plt.imshow(x_hat.view(1,28,28).squeeze().data.numpy(), cmap='gray')
    plt.title('%i' % train_data.train_labels[batch_idx])
    plt.show()
    if batch_idx == 3:
        break

```

### 2.8 Visualize latent space (20 points)
Now, implement the auto-encoder now with a 2-dimensional latent space, and train again over the MNIST data. Make a visualization of the learned manifold by using a linearly spaced coordinate grid as input for the latent space, as seen in  https://arxiv.org/abs/1312.6114 Figure 4.

```python
fc1_dims, fc21_dims, fc22_dims, fc3_dims, fc4_dims = get_fc_dims(784, 512, 2)

# Load data
train_data = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.ToTensor())

train_loader = torch.utils.data.DataLoader(train_data,
                                           batch_size=batch_size, shuffle=True, **{})

# Init model
VAE_MNIST = VAE(fc1_dims=fc1_dims, fc21_dims=fc21_dims, fc22_dims=fc22_dims, fc3_dims=fc3_dims, fc4_dims=fc4_dims)

# Init optimizer
optimizer = optim.Adam(VAE_MNIST.parameters(), lr=lr)

# Train
for epoch in range(1, epochs + 1):
    train(epoch, train_loader, VAE_MNIST, optimizer)

from scipy.stats import norm
from torchvision.utils import make_grid

def generate_manifold(model: VAE, rows: int = 20):
    grid = torch.linspace(0, 1, rows)
    samples = [torch.erfinv(2 * torch.tensor([x, y]) - 1)
               * np.sqrt(2) for x in grid for y in grid]

    samples = torch.stack(samples)
    manifold = model.decode(samples).view(-1, 1, 28, 28)
    image_grid = make_grid(manifold, nrow=rows).detach().numpy().transpose(1, 2, 0)

    figure(figsize=(20, 20))
    ax = plt.gca()
    ax.xaxis.set_visible(False)
    ax.yaxis.set_visible(False)
    ax.imshow(image_grid)
    plt.show()

generate_manifold(VAE_MNIST)
```

### 2.8 Amortized inference (10 points)
What is amortized inference? Where in the code of Part 2 is it used? What is the benefit of using it?

Amortized inference is the idea that we are introducing a parameterized function which serves as a mapping from the observation space to the parameters of the approximate posterior distribution. This allows us to share some of the parameters between the datapoints instead of using a different set every time.

It is used in the Variational Auto-encoder where the Decoder part uses the same parameters for any datapoint.

The main benefit in using it is the now constant amount of parameters that we have with respect to the data size. It is also smaller than original and thus reduces the computation cost of the algorithm.


```python
import numpy as np
import scipy.misc
from scipy.fftpack import dct, idct
import sys
from PIL import Image
import matplotlib
import matplotlib.pyplot as plt
import random
from tqdm._tqdm_notebook import tqdm_notebook
from scipy.fftpack import dct, idct
import seaborn as sns
from skimage.metrics import structural_similarity as ssim
import pandas as pd
import sympy

%matplotlib inline

class ImageLoader:
    def __init__(self, FILE_PATH):
        self.img = np.array(Image.open(FILE_PATH))
        # 行数
        self.row_blocks_count = self.img.shape[0] // 8
        # 列数
        self.col_blocks_count = self.img.shape[1] // 8

    def get_points(self, POINT):
        Row = random.randint(0, len(self.img) - POINT - 1)
        Col = random.randint(0, len(self.img) - 1)
        return self.img[Row : Row + POINT, Col]

    def get_block(self, col, row):
        return self.img[col * 8 : (col + 1) * 8, row * 8 : (row + 1) * 8]

# plt.rcParams['font.family'] ='sans-serif'#使用するフォント
# plt.rcParams["font.sans-serif"] = "Source Han Sans"
plt.rcParams["font.family"] = "Source Han Sans JP"  # 使用するフォント
plt.rcParams["xtick.direction"] = "in"  # x軸の目盛線が内向き('in')か外向き('out')か双方向か('inout')
plt.rcParams["ytick.direction"] = "in"  # y軸の目盛線が内向き('in')か外向き('out')か双方向か('inout')
plt.rcParams["xtick.major.width"] = 1.0  # x軸主目盛り線の線幅
plt.rcParams["ytick.major.width"] = 1.0  # y軸主目盛り線の線幅
plt.rcParams["font.size"] = 12  # フォントの大きさ
plt.rcParams["axes.linewidth"] = 1.0  # 軸の線幅edge linewidth。囲みの太さ
matplotlib.font_manager._rebuild()

MONO_DIR_PATH = "../../Mono/"
AIRPLANE = ImageLoader(MONO_DIR_PATH + "airplane512.bmp")
BARBARA = ImageLoader(MONO_DIR_PATH + "barbara512.bmp")
BOAT = ImageLoader(MONO_DIR_PATH + "boat512.bmp")
GOLDHILL = ImageLoader(MONO_DIR_PATH + "goldhill512.bmp")
LENNA = ImageLoader(MONO_DIR_PATH + "lenna512.bmp")
MANDRILL = ImageLoader(MONO_DIR_PATH + "mandrill512.bmp")
MILKDROP = ImageLoader(MONO_DIR_PATH + "milkdrop512.bmp")
SAILBOAT = ImageLoader(MONO_DIR_PATH + "sailboat512.bmp")
```

```python
n_bar = 5
N = 16
```

# MSDS

```python
def msds(N,arr):
    w_e = 0
    e_e = 0
    n_e = 0
    s_e = 0
    nw_e = 0
    ne_e = 0
    sw_e = 0
    se_e = 0
    for row in range(arr.shape[0] // N):
        for col in range(arr.shape[1] // N):
            f_block = arr[row * N : (row + 1) * N, col * N : (col + 1) * N]
            # w
            if col == 0:
                w_block = np.fliplr(f_block)
            else:
                w_block = arr[row * N : (row + 1) * N, (col - 1) * N : col * N]
            # e
            if col == arr.shape[1] // N - 1:
                e_block = np.fliplr(f_block)
            else:
                e_block = arr[row * N : (row + 1) * N, (col + 1) * N : (col + 2) * N]
            # n
            if row == 0:
                n_block = np.flipud(f_block)
            else:
                n_block = arr[(row - 1) * N : row * N, col * N : (col + 1) * N]
            # s
            if row == arr.shape[0] // N - 1:
                s_block = np.flipud(f_block)
            else:
                s_block = arr[(row + 1) * N : (row + 2) * N, col * N : (col + 1) * N]

            w_d1 = f_block[:, 0] - w_block[:, N-1]
            e_d1 = f_block[:, N-1] - e_block[:, 0]
            n_d1 = f_block[0, :] - n_block[N-1, :]
            s_d1 = f_block[N-1, :] - s_block[0, :]

            w_d2 = (w_block[:, N-1] - w_block[:, N-2] + f_block[:, 1] - f_block[:, 0]) / 2
            e_d2 = (e_block[:, 1] - e_block[:, 0] + f_block[:, N-1] - f_block[:, N-2]) / 2
            n_d2 = (n_block[N-1, :] - n_block[N-2, :] + f_block[1, :] - f_block[0, :]) / 2
            s_d2 = (s_block[1, :] - s_block[0, :] + f_block[N-1, :] - f_block[N-2, :]) / 2

            w_e += np.sum((w_d1 - w_d2) ** 2 )
            e_e += np.sum((e_d1 - e_d2) ** 2 )
            n_e += np.sum((n_d1 - n_d2) ** 2)
            s_e += np.sum((s_d1 - s_d2) ** 2)

            # nw
            if row == 0 or col == 0:
                nw_block = np.flipud(np.fliplr(f_block))
            else:
                nw_block = arr[(row - 1) * N : row * N, (col - 1) * N : col * N]
            # ne
            if row == 0 or col == arr.shape[1] // N - 1:
                ne_block = np.flipud(np.fliplr(f_block))
            else:
                ne_block = arr[(row-1) * N : row * N, (col + 1) * N : (col + 2) * N]
            # sw
            if row == arr.shape[0] // N -1 or col == 0:
                sw_block = np.flipud(np.fliplr(f_block))
            else:
                sw_block = arr[row * N : (row+1) * N, (col-1) * N : col * N]
            # se
            if  row == arr.shape[0]//N-1 or col == arr.shape[0] // N -1:
                se_block = np.flipud(np.fliplr(f_block))
            else:
                se_block = arr[(row + 1) * N : (row + 2) * N, (col+1) * N : (col + 2) * N]

            nw_g1 = f_block[0, 0] - nw_block[N-1, N-1]
            ne_g1 = f_block[0, N-1] - ne_block[N-1, 0]
            sw_g1 = f_block[N-1, 0] - sw_block[0, N-1]
            se_g1 = f_block[N-1, N-1] - se_block[0, 0]

            nw_g2 = (nw_block[N-1,N-1] - nw_block[N-2,N-2] + f_block[1,1] - f_block[0,0])/2
            ne_g2 = (ne_block[N-1,0] - ne_block[N-2,1] + f_block[1,N-2] - f_block[0,N-1])/2
            sw_g2 = (sw_block[0,N-1] - nw_block[1,N-2] + f_block[N-2,1] - f_block[N-1,0])/2
            se_g2 = (nw_block[0,0] - nw_block[1,1] + f_block[N-2,N-2] - f_block[N-1,N-1])/2

            nw_e += (nw_g1 - nw_g2) ** 2
            ne_e += (ne_g1 - ne_g2) ** 2
            sw_e += (sw_g1 - sw_g2) ** 2
            se_e += (se_g1 - se_g2) ** 2

    MSDSt = (w_e + e_e + n_e + s_e + nw_e + ne_e + sw_e + se_e)/ ((arr.shape[0]/N)**2)
    MSDS1 = (w_e + e_e + n_e + s_e)/ ((arr.shape[0]/N)**2)
    MSDS2 = (nw_e + ne_e + sw_e + se_e)/ ((arr.shape[0]/N)**2)
    return MSDSt, MSDS1, MSDS2
```

```python
class DMLCT:
    def __init__(self, n_bar, N):
        self.n_bar = n_bar
        self.N = N

        self.x_l = (2 * np.arange(N) + 1) / (2 * N)
        self.s_l = np.arange(n_bar) / (n_bar - 1)
        self.xi = (np.arange(n_bar + 1) - 0.5) / (n_bar - 1)

        self.lambda_kh = self.get_lambda_kh(self.n_bar)

        self.w_k_j = self.get_w_k_j(self.n_bar, self.N)
        self.W_L_k_kh = self.get_W_L_k_kh(self.n_bar, self.N)
        self.W_k_kh = self.get_W_k_kh(self.n_bar, self.N)
        self.W_R_k_kh = self.get_W_R_k_kh(self.n_bar, self.N)

    def Lagrange_j(self, j):
        x = sympy.Symbol("x")
        L_x = 1.0
        for l in range(self.n_bar):
            if l != j:
                L_x *= (x - self.s_l[l]) / (self.s_l[j] - self.s_l[l])
        return sympy.integrate(L_x)

    def get_lambda_kh(self, n_bar):
        lambda_kh = np.ones(n_bar)
        lambda_kh[0] = np.sqrt(1 / 2)
        return lambda_kh

    def get_w_k_j(self, n_bar, N):
        L_j = np.zeros((n_bar, N))
        x = sympy.Symbol("x")
        for j in range(n_bar):
            temp = []
            Lj = self.Lagrange_j(j)
            for k in range(N):
                temp.append(Lj.subs(x, self.x_l[k]))
            L_j[j] = np.array(temp)

        w_k_j = np.zeros((n_bar, N))
        for j in range(n_bar):
            w_k_j[j] = scipy.fftpack.dct(L_j[j], norm="ortho")
        return w_k_j

    def get_W_L_k_kh(self, n_bar, N):
        W_L_k_kh = np.zeros((n_bar - 1, N))
        lambda_kh = self.get_lambda_kh(n_bar)

        for kh in range(n_bar - 1):
            W_L_k_kh[kh] = (
                (1 - n_bar)
                * np.sqrt(2 / N)
                * lambda_kh[kh]
                * np.cos(np.pi * kh * (self.xi[0] + 1))
                * self.w_k_j[0]
            )
        return W_L_k_kh

    def get_W_k_kh(self, n_bar, N):
        W_k_kh = np.zeros((n_bar - 1, N))
        for kh in range(n_bar - 1):
            sum_sin = np.zeros(N)
            for j in range(1, n_bar - 2 + 1):
                sum_sin += np.sin(np.pi * kh * self.s_l[j]) * self.w_k_j[j]

            W_k_kh[kh] = (
                (n_bar - 1)
                * np.sqrt(2 / N)
                * self.lambda_kh[kh]
                * (
                    np.cos(np.pi * kh * self.xi[1])
                    * (self.w_k_j[0] - (-1) ** (kh) * self.w_k_j[n_bar - 1])
                    - 2 * np.sin((np.pi * kh) / (2 * (n_bar - 1))) * sum_sin
                )
            )
        return W_k_kh

    def get_W_R_k_kh(self, n_bar, N):
        W_R_k_kh = np.zeros((n_bar - 1, N))
        for kh in range(n_bar - 1):
            W_R_k_kh[kh] = (
                (n_bar - 1)
                * np.sqrt(2 / N)
                * self.lambda_kh[kh]
                * np.cos(np.pi * kh * (self.xi[n_bar] - 1))
                * self.w_k_j[n_bar - 1]
            )
        return W_R_k_kh
```

```python
def get_F_L_k_horizontal(arr, N, row, col):
    # w
    if col == 0:
#         w_block = np.zeros(N)
        w_block = arr[row, col * N : (col + 1) * N]
    else:
        w_block = arr[row, (col - 1) * N : col * N]

    return w_block
```

```python
def get_F_R_k_horizontal(arr, N, row, col):
    # e
    if col == arr.shape[1] // N - 1:
#         e_block = np.zeros(N)
        e_block = arr[row, col * N : (col + 1) * N]
    else:
        e_block = arr[row, (col + 1) * N : (col + 2) * N]

    return e_block
```

```python
def get_F_L_k_vertical(arr, N, row, col):
    # n
    if row == 0:
#         n_block = np.zeros(N)
        n_block = arr[row * N : (row + 1) * N, col]
    else:
        n_block = arr[(row - 1) * N : row * N, col]

    return n_block
```

```python
def get_F_R_k_vertical(arr, N, row, col):
    # s
    if row == arr.shape[0] // N - 1:
#         s_block = np.zeros(N)
        s_block = arr[row * N : (row + 1) * N, col]
    else:
        s_block = arr[(row + 1) * N : (row + 2) * N, col]

    return s_block
```

```python
# dmlct = DMLCT(n_bar, N)
```

```python
IMG = LENNA
# IMG = ImageLoader(MONO_DIR_PATH + "LENNA.bmp")
```

```python
Fk = np.zeros(IMG.img.shape)
```

# 順変換

## 縦方向

### DCT

```python
for row in range(IMG.img.shape[0] // N):
    for col in range(IMG.img.shape[1]):
        eight_points = IMG.img[N * row : N * (row + 1), col]
        c = scipy.fftpack.dct(eight_points, norm="ortho")
        Fk[N * row : N * (row + 1), col] = c
```

### 残差

```python
dmlct = DMLCT(n_bar, N)
for row in range(IMG.img.shape[0] // N):
    for col in range(IMG.img.shape[1]):
        # ビューなら直接いじっちゃう
        F = Fk[N * row : N * (row + 1), col]
        F_L = get_F_L_k_vertical(Fk, N, row, col)
        F_R = get_F_R_k_vertical(Fk, N, row, col)

        U_k_n_bar = np.zeros(N)
        for kh in range(n_bar - 2 + 1):
            U_k_n_bar += (
                F_L[kh] * dmlct.W_L_k_kh[kh]
                + F[kh] * dmlct.W_k_kh[kh]
                + F_R[kh] * dmlct.W_R_k_kh[kh]
            )

        # n_bar = 4 なら 0,1,2は残す 3,4,5,6,7を書き換える
        F[n_bar - 2 + 1 :] -= U_k_n_bar[n_bar - 2 + 1 :]
```

```python
# 0を残す
for k in reversed(range(1, n_bar - 2 + 1)):
    dmlct = DMLCT(k+1, N)
    for row in range(IMG.img.shape[0] // N):
        for col in range(IMG.img.shape[1]):
            # ビューなら直接いじっちゃう
            F = Fk[N * row : N * (row + 1), col]
            F_L = get_F_L_k_vertical(Fk, N, row, col)
            F_R = get_F_R_k_vertical(Fk, N, row, col)

            U_k_n_bar = np.zeros(N)

            for kh in range((k + 1) - 2 + 1):
                U_k_n_bar += (
                    F_L[kh] * dmlct.W_L_k_kh[kh]
                    + F[kh] * dmlct.W_k_kh[kh]
                    + F_R[kh] * dmlct.W_R_k_kh[kh]
                )

            F[k] -= U_k_n_bar[k]
```

## 横方向

### DCT

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // N):
        eight_points = Fk[row, N * col : N * (col + 1)]
        c = scipy.fftpack.dct(eight_points, norm="ortho")
        Fk[row, N * col : N * (col + 1)] = c
```

### 残差

```python
dmlct = DMLCT(n_bar, N)
for row in range(IMG.img.shape[0]):
    for col in range(IMG.img.shape[1] // N):
        F = Fk[row, N * col : N * (col + 1)]
        F_L = get_F_L_k_horizontal(Fk, N, row, col)
        F_R = get_F_R_k_horizontal(Fk, N, row, col)

        U_k_n_bar = np.zeros(N)
        for kh in range(n_bar - 2 + 1):
            U_k_n_bar += (
                F_L[kh] * dmlct.W_L_k_kh[kh]
                + F[kh] * dmlct.W_k_kh[kh]
                + F_R[kh] * dmlct.W_R_k_kh[kh]
            )

        # n_bar = 4 なら 0,1,2は残す 3,4,5,6,7を書き換える
        F[n_bar - 2 + 1 :] -= U_k_n_bar[n_bar - 2 + 1 :]
```

```python
# 0を残す
for k in reversed(range(1, n_bar - 2 + 1)):
    dmlct = DMLCT(k+1, N)
    for row in range(IMG.img.shape[0]):
        for col in range(IMG.img.shape[1] // N):
            F = Fk[row, N * col : N * (col + 1)]
            F_L = get_F_L_k_horizontal(Fk, N, row, col)
            F_R = get_F_R_k_horizontal(Fk, N, row, col)

            U_k_n_bar = np.zeros(N)

            for kh in range((k + 1) - 2 + 1):
                U_k_n_bar += (
                    F_L[kh] * dmlct.W_L_k_kh[kh]
                    + F[kh] * dmlct.W_k_kh[kh]
                    + F_R[kh] * dmlct.W_R_k_kh[kh]
                )

            F[k] -= U_k_n_bar[k]
```

```python
plt.imshow(Fk)
# pd.DataFrame(Fk).to_csv("DMLCT_lenna_hiroya_coef.csv",header=False,index=False)
```

# 係数の確保

```python
Fk_Ori = np.copy(Fk)
```

# 逆変換

```python
recover = np.zeros(IMG.img.shape)
```

## 横方向

### 残差

```python
for k in range(1, n_bar - 2 + 1):
    dmlct = DMLCT(k+1, N)
    for row in range(IMG.img.shape[0]):
        for col in range(IMG.img.shape[1] // N):
            F = Fk[row, N * col : N * (col + 1)]
            F_L = get_F_L_k_horizontal(Fk, N, row, col)
            F_R = get_F_R_k_horizontal(Fk, N, row, col)

            U_k_n_bar = np.zeros(N)
            for kh in range((k + 1) - 2 + 1):
                U_k_n_bar += (
                    F_L[kh] * dmlct.W_L_k_kh[kh]
                    + F[kh] * dmlct.W_k_kh[kh]
                    + F_R[kh] * dmlct.W_R_k_kh[kh]
                )

            F[k] += U_k_n_bar[k]
```

```python
dmlct = DMLCT(n_bar, N)
for row in range(IMG.img.shape[0]):
    for col in range(IMG.img.shape[1] // N):
        F = Fk[row, N * col : N * (col + 1)]
        F_L = get_F_L_k_horizontal(Fk, N, row, col)
        F_R = get_F_R_k_horizontal(Fk, N, row, col)

        U_k_n_bar = np.zeros(N)
        for kh in range(n_bar - 2 + 1):
            U_k_n_bar += (
                F_L[kh] * dmlct.W_L_k_kh[kh]
                + F[kh] * dmlct.W_k_kh[kh]
                + F_R[kh] * dmlct.W_R_k_kh[kh]
            )

        # n_bar = 4 なら 0,1,2は残す 3,4,5,6,7を書き換える
        F[n_bar - 2 + 1 :] += U_k_n_bar[n_bar - 2 + 1 :]
```

### IDCT

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // N):
        F = Fk[row, N * col : N * col + N]
        data = scipy.fftpack.idct(F, norm="ortho")
        # Fkに代入した後、縦方向に対して処理
        Fk[row, N * col : N * col + N] = data
```

## 縦方向

### 残差

```python
for k in range(1, n_bar - 2 + 1):
    dmlct = DMLCT(k+1, N)
    for row in range(IMG.img.shape[0] // N):
        for col in range(IMG.img.shape[1]):
            # ビューなら直接いじっちゃう
            F = Fk[N * row : N * (row + 1), col]
            F_L = get_F_L_k_vertical(Fk, N, row, col)
            F_R = get_F_R_k_vertical(Fk, N, row, col)

            U_k_n_bar = np.zeros(N)

            for kh in range((k + 1) - 2 + 1):
                U_k_n_bar += (
                    F_L[kh] * dmlct.W_L_k_kh[kh]
                    + F[kh] * dmlct.W_k_kh[kh]
                    + F_R[kh] * dmlct.W_R_k_kh[kh]
                )

            F[k] += U_k_n_bar[k]
```

```python
dmlct = DMLCT(n_bar, N)
for row in range(IMG.img.shape[0] // N):
    for col in range(IMG.img.shape[1]):
        # ビューなら直接いじっちゃう
        F = Fk[N * row : N * (row + 1), col]
        F_L = get_F_L_k_vertical(Fk, N, row, col)
        F_R = get_F_R_k_vertical(Fk, N, row, col)

        U_k_n_bar = np.zeros(N)
        for kh in range(n_bar - 2 + 1):
            U_k_n_bar += (
                F_L[kh] * dmlct.W_L_k_kh[kh]
                + F[kh] * dmlct.W_k_kh[kh]
                + F_R[kh] * dmlct.W_R_k_kh[kh]
            )

        # n_bar = 4 なら 0,1,2は残す 3,4,5,6,7を書き換える
        F[n_bar - 2 + 1 :] += U_k_n_bar[n_bar - 2 + 1 :]
```

### IDCT

```python
for row in range(Fk.shape[0] // N):
    for col in range(Fk.shape[1]):
        F = Fk[N * row : N * (row + 1), col]
        data = scipy.fftpack.idct(F, norm="ortho")
        # 復元画像
        recover[N * row : N * (row + 1), col] = data
```

```python
plt.imshow(recover.astype("u8"), cmap="gray")
```

もどった...！

# 量子化テーブル

```python
Q = 108
Q_Luminance = np.ones((32,32)) * Q
```

# 量子化

```python
Fk = np.copy(Fk_Ori)
Q_Fk = np.zeros(Fk.shape)
for row in range(IMG.img.shape[0] // N):
    for col in range(IMG.img.shape[1] // N):
        block = Fk[row * N : (row + 1) * N, col * N : (col + 1) * N]
        # 量子化
        block = np.round(block / Q_Luminance)
        # 逆量子化
        block = block * Q_Luminance
        Q_Fk[row * N : (row+1)*N, col * N : (col+1)*N] = block
```

```python
Fk = np.copy(Q_Fk)
Q_recover = np.zeros(Q_Fk.shape)
```

## 横方向

### 残差

```python
for k in range(1, n_bar - 2 + 1):
    dmlct = DMLCT(k+1, N)
    for row in range(IMG.img.shape[0]):
        for col in range(IMG.img.shape[1] // N):
            F = Fk[row, N * col : N * (col + 1)]
            F_L = get_F_L_k_horizontal(Fk, N, row, col)
            F_R = get_F_R_k_horizontal(Fk, N, row, col)

            U_k_n_bar = np.zeros(N)
            for kh in range((k + 1) - 2 + 1):
                U_k_n_bar += (
                    F_L[kh] * dmlct.W_L_k_kh[kh]
                    + F[kh] * dmlct.W_k_kh[kh]
                    + F_R[kh] * dmlct.W_R_k_kh[kh]
                )

            F[k] += U_k_n_bar[k]
```

```python
dmlct = DMLCT(n_bar, N)
for row in range(IMG.img.shape[0]):
    for col in range(IMG.img.shape[1] // N):
        F = Fk[row, N * col : N * (col + 1)]
        F_L = get_F_L_k_horizontal(Fk, N, row, col)
        F_R = get_F_R_k_horizontal(Fk, N, row, col)

        U_k_n_bar = np.zeros(N)
        for kh in range(n_bar - 2 + 1):
            U_k_n_bar += (
                F_L[kh] * dmlct.W_L_k_kh[kh]
                + F[kh] * dmlct.W_k_kh[kh]
                + F_R[kh] * dmlct.W_R_k_kh[kh]
            )

        # n_bar = 4 なら 0,1,2は残す 3,4,5,6,7を書き換える
        F[n_bar - 2 + 1 :] += U_k_n_bar[n_bar - 2 + 1 :]
```

### IDCT

```python
for row in range(Fk.shape[0]):
    for col in range(Fk.shape[1] // N):
        F = Fk[row, N * col : N * col + N]
        data = scipy.fftpack.idct(F, norm="ortho")
        # Fkに代入した後、縦方向に対して処理
        Fk[row, N * col : N * col + N] = data
```

## 縦方向

### 残差

```python
for k in range(1, n_bar - 2 + 1):
    dmlct = DMLCT(k+1, N)
    for row in range(IMG.img.shape[0] // N):
        for col in range(IMG.img.shape[1]):
            # ビューなら直接いじっちゃう
            F = Fk[N * row : N * (row + 1), col]
            F_L = get_F_L_k_vertical(Fk, N, row, col)
            F_R = get_F_R_k_vertical(Fk, N, row, col)

            U_k_n_bar = np.zeros(N)

            for kh in range((k + 1) - 2 + 1):
                U_k_n_bar += (
                    F_L[kh] * dmlct.W_L_k_kh[kh]
                    + F[kh] * dmlct.W_k_kh[kh]
                    + F_R[kh] * dmlct.W_R_k_kh[kh]
                )

            F[k] += U_k_n_bar[k]
```

```python
dmlct = DMLCT(n_bar, N)
for row in range(IMG.img.shape[0] // N):
    for col in range(IMG.img.shape[1]):
        # ビューなら直接いじっちゃう
        F = Fk[N * row : N * (row + 1), col]
        F_L = get_F_L_k_vertical(Fk, N, row, col)
        F_R = get_F_R_k_vertical(Fk, N, row, col)

        U_k_n_bar = np.zeros(N)
        for kh in range(n_bar - 2 + 1):
            U_k_n_bar += (
                F_L[kh] * dmlct.W_L_k_kh[kh]
                + F[kh] * dmlct.W_k_kh[kh]
                + F_R[kh] * dmlct.W_R_k_kh[kh]
            )

        # n_bar = 4 なら 0,1,2は残す 3,4,5,6,7を書き換える
        F[n_bar - 2 + 1 :] += U_k_n_bar[n_bar - 2 + 1 :]
```

### IDCT

```python
for row in range(Fk.shape[0] // N):
    for col in range(Fk.shape[1]):
        F = Fk[N * row : N * (row + 1), col]
        data = scipy.fftpack.idct(F, norm="ortho")
        # 復元画像
        Q_recover[N * row : N * (row + 1), col] = data
```

```python
Q_recover = np.round(Q_recover)
```

```python
plt.imshow(Q_recover, cmap="gray")
plt.imsave("DMLCT_16x16_n"+str(n_bar)+ "_LENNA.png",Q_recover,cmap="gray")
```

# 情報量

```python
qfk = pd.Series(Q_Fk.flatten())
pro = qfk.value_counts() / qfk.value_counts().sum()
pro.head()
```

     0.0      0.981461
     108.0    0.006153
    -108.0    0.005993
    -216.0    0.000957
     216.0    0.000805
    dtype: float64

```python
S = 0
for pi in pro:
    S -= pi * np.log2(pi)
S
```

    0.191999270242479

# PSNR

```python
MSE = np.sum(np.sum(np.power((IMG.img - Q_recover),2)))/(Q_recover.shape[0] * Q_recover.shape[1])
PSNR = 10 * np.log10(255 * 255 / MSE)
PSNR
```

    29.177596612468044

# MSSIM

```python
MSSIM = ssim(IMG.img,Q_recover.astype(IMG.img.dtype),gaussian_weights=True,sigma=1.5,K1=0.01,K2=0.03)
MSSIM
```

    0.7968425057431381

```python
dmlct = DMLCT(n_bar, N)
```

# MSDS

```python
MSDSt, MSDS1, MSDS2 = msds(N,Q_recover)
```

```python
MSDS1
```

    16495.56884765625

```python
MSDS2
```

    3861.55859375

```python

```


# Initial Analysis

```python
from src.surfaceAnalyzer import SurfaceAnalyzer
from common_python.ODEModel.LTIModel import LTIModel
from common_python.sympy import sympyUtil as su

import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
import numpy as np
import tellurium as te
import pandas as pd
import seaborn as sn
import sympy
import os
```

# Inferring a reaction network from a Jacobian

Let $A$ be an $N \times N$ matrix. We want to find a reaction network with linear kinetics such
that the Jacobian of the network is $A$.
The reaction network should be mass balanced with positive kinetics constants.

1. This problem is equivlent to finding a $N \times R$ matrix ${\bf S}$ and a $R \times N$ matrix ${\bf V}$ such
that ${ \bf A} = {\bf S} {\bf V}$. $S$ is the stoichiometry matrix and ${\bf V}$ is the Jacobian of the reaction rates ${\bf v}({\bf x}; {\bf k})$,
where ${\bf x} = (x_1 ,\cdots, x_N)$ are the floating species and ${\bf k}$ are kinetics contants.
1. What are the restrictions on ${\bf S}$?
   1. Each row has exactly one entry that is negative.
1. What are the restrictions on ${\bf V}$?
   1. All entries are $k_i$.
   1. There is only one non-zero value in each row.

One approach is to solve the matrix equation directly.
If ${\bf V}$ is specified, then ${\bf A} {\bf V}^T \left( {\bf V} {\bf V}^T \right)^{-1} = {\bf S}$. The same can be done if ${\bf S}$ is specified.
The key question becomes under what conditions does a pseudo inverse exist.
Need to specify ${\bf S}$ to ensure mass balance. Also, need to control ${\bf V}$ to ensure that ${\bf k} > 0$.

An issue here is that if $R > N$, then it can only be that ${\bf V}^T {\bf V}$ is invertible.
Similarly, only ${\bf S} {\bf S}^T$ is invertible.
However, we could still construct a network with a small number of reactions.

Another analysis of value is to show where contants appear in ${\bf A}$ as the stoichiometry is varied. Systems constructed in this way can then be studied to determine the convexity of fitting.

Since we cannot simultaneously ensure mass balance and the constraint on kinetic constants, we should drop the former since many models do not include mass balance since there are "implicit" species. This means that there will be fractional stoichiometries.

The procedure for finding the stoichiometry is:
* Given
   - $N$, the number of species
   - ${\bf k}$ the kinetics constant for each reaction
* Steps
   1. ${\bf V} = \{ v_{ij} \}$, where $v_{ij} = k_i$ and $j = \sigma(i)$, some assignment of the reactant species.
   1. ${\bf S} = {\bf A} {\bf V}^T \left( {\bf V} {\bf V}^T \right)^{-1}$.
   1. Construct the reaction network model from ${\bf S}, {\bf V}$.

To generate random networks with $N$ species and $R$ reactions:
1. for $r$ in range($R$)
    1. $n$ = randomly selected species
    1. $k$ = randomly selected kinetic constant
    1. $v_r$ = reaction row, with k in the column for the species
    1. $p_1 , \cdots, p_m$ = randomly selected products
    1. $s_r$ = column in  the stoichiometry matrix

# Analysis for Convexity of Linear Networks

Let
* ${\bf v}({\bf x}; {\bf k})$ be a reaction network with no more than 1st order kinetics.
* ${\bf N}$ be the stoichiometry matrix.
* ${\bf k}^{\star}$ be the true parameters for the network.
* ${\bf x}(t; {\bf k})$ be the solution with the parameters ${\bf k}$
* ${\bf x}^{\star} (t) = {\bf x;  }(t; {\bf k}^{\star})$
* ${\bf r}(t; {\bf k}) = {\bf x}^{\star} (t) - {\bf x} (t; {\bf k})$

Determine if the function $\int_0^{\infty} {\bf r}^T(t; {\bf k}) {\bf r}(t; {\bf k})$ is convex

## Linear Pathway

Consider a pathway of the form $S_0 \xrightarrow{v_0} S_1 \xrightarrow{v_1} \cdots \xrightarrow{v_N} S_N$ with mass action kinetics.
So, $v_i = k_i S_i$.
The Jacobian ${\bf A}$ for this system as $a_{ij}$ defined as:

* $j=i$: $-k_i$
* $j=i-1$: $k_i$
* otherwise: 0

\begin{equation*}
\begin{pmatrix}
a & b \\
c & d \\
\end{pmatrix}
\end{equation*}

```python
def mkSyms(i, j):
    aSymName = "a%d%d" % (i, j)
    kSymName = "k%d" % i
    su.addSymbols(aSymName + " " + kSymName)
    return eval(aSymName), eval(kSymName)
#
#su.addSymbols("a11 a12 a13 a14 a21 a22 a23 a24 a31 a32 a33 a34 a41 a42 a43 a44 k1 k2 k3 k4")
#mat = sympy.Matrix([ [a11, a12, a13, a14], [a21, a22, a23, a24], [a31, a32, a33, a34], [a41, a42, a43, a44]])
subs = {}
size = 6
matrix = []
for i in range(size):
    row = []
    matrix.append(row)
    for j in range(size):
        aSym, kSym = mkSyms(i, j)
        row.append(aSym)
        if j == i:
            value = -1 * kSym
        elif j == i - 1:
            value = "k%d" % (i - 1)
            value = eval(value)
        else:
            value = 0
        subs[aSym] = value
subSyms = ["k0"]
#subs = {globals()[k]: v for k, v in subs.items() if k in subSyms}
mat = sympy.Matrix(matrix)
#subs = {a11: -k1, a12: 0, a13: 0, a21: k1, a22: -k2, a23: 0, a31: 0, a32: k2, a33: -k3}
newMat = mat.subs(subs)
newMat
```

$\displaystyle \left[\begin{matrix}- k_{0} & 0 & 0 & 0 & 0 & 0\\k_{0} & - k_{1} & 0 & 0 & 0 & 0\\0 & k_{1} & - k_{2} & 0 & 0 & 0\\0 & 0 & k_{2} & - k_{3} & 0 & 0\\0 & 0 & 0 & k_{3} & - k_{4} & 0\\0 & 0 & 0 & 0 & k_{4} & - k_{5}\end{matrix}\right]$

```python
{k: v for k, v in subs.items() if v in subSyms}
```

    {}

```python
result = {}
for k,v in subs.items():
    if v == 0:
        continue
    if v.free_symbols in subSyms:
        result[k] = v

result
```

    {}

```python
v.free_symbols
```

    {k5}

```python
sympy.simplify(sympy.diff(sympy.expand(newMat ** 6 ), k2, k2))
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0 & 0\\2 k_{0} k_{1} \left(k_{0}^{2} + k_{0} k_{1} + 3 k_{0} k_{2} + k_{1}^{2} + 3 k_{1} k_{2} + 6 k_{2}^{2}\right) & - 2 k_{1} \left(k_{1}^{3} + 3 k_{1}^{2} k_{2} + 6 k_{1} k_{2}^{2} + 10 k_{2}^{3}\right) & 30 k_{2}^{4} & 0 & 0 & 0\\- 2 k_{0} k_{1} \left(k_{0}^{2} + k_{0} k_{1} + 3 k_{0} k_{2} + k_{0} k_{3} + k_{1}^{2} + 3 k_{1} k_{2} + k_{1} k_{3} + 6 k_{2}^{2} + 3 k_{2} k_{3} + k_{3}^{2}\right) & 2 k_{1} \left(k_{1}^{3} + 3 k_{1}^{2} k_{2} + k_{1}^{2} k_{3} + 6 k_{1} k_{2}^{2} + 3 k_{1} k_{2} k_{3} + k_{1} k_{3}^{2} + 10 k_{2}^{3} + 6 k_{2}^{2} k_{3} + 3 k_{2} k_{3}^{2} + k_{3}^{3}\right) & - 30 k_{2}^{4} - 20 k_{2}^{3} k_{3} - 12 k_{2}^{2} k_{3}^{2} - 6 k_{2} k_{3}^{3} - 2 k_{3}^{4} & 0 & 0 & 0\\2 k_{0} k_{1} k_{3} \left(k_{0} + k_{1} + 3 k_{2} + k_{3} + k_{4}\right) & - 2 k_{1} k_{3} \left(k_{1}^{2} + 3 k_{1} k_{2} + k_{1} k_{3} + k_{1} k_{4} + 6 k_{2}^{2} + 3 k_{2} k_{3} + 3 k_{2} k_{4} + k_{3}^{2} + k_{3} k_{4} + k_{4}^{2}\right) & 2 k_{3} \left(10 k_{2}^{3} + 6 k_{2}^{2} k_{3} + 6 k_{2}^{2} k_{4} + 3 k_{2} k_{3}^{2} + 3 k_{2} k_{3} k_{4} + 3 k_{2} k_{4}^{2} + k_{3}^{3} + k_{3}^{2} k_{4} + k_{3} k_{4}^{2} + k_{4}^{3}\right) & 0 & 0 & 0\\- 2 k_{0} k_{1} k_{3} k_{4} & 2 k_{1} k_{3} k_{4} \left(k_{1} + 3 k_{2} + k_{3} + k_{4} + k_{5}\right) & - 2 k_{3} k_{4} \left(6 k_{2}^{2} + 3 k_{2} k_{3} + 3 k_{2} k_{4} + 3 k_{2} k_{5} + k_{3}^{2} + k_{3} k_{4} + k_{3} k_{5} + k_{4}^{2} + k_{4} k_{5} + k_{5}^{2}\right) & 0 & 0 & 0\end{matrix}\right]$

Want to calculate the derivative w.r.t. $k_i$ of the solution.
Key is that all terms have the same sign.
Can I first calculate the second derivative of ${\bf A(k)}$?

Let ${\bf r}(t) = x^{\star} (t) - e^{{\bf A}(k) t} x(0)$.
Want $\frac{\partial^2 \left( {\bf r}(t) \right)^T {\bf r}(t)} {\partial^2 k}$
1. $\frac{\partial \left( {\bf r}(t) \right)^T {\bf r}(t)} {\partial k} =
\frac{\partial \left( {\bf r}(t) \right)^T }{\partial k} {\bf r}(t) +  \left( {\bf r}(t) \right)^T  \frac{\partial {\bf r}(t)}{\partial k}$
1. $\frac{\partial \left( {\bf r}(t) \right)^T }{\partial k} =  \left( \frac{\partial {\bf r}(t) }{\partial k} \right)^T$
1. $\frac{\partial {\bf r}(t) }{\partial k} = - \frac{\partial e^{{\bf A}(k) t}} {\partial k} x(0)$

```python
newMat ** 6
```

$\displaystyle \left[\begin{matrix}k_{0}^{6} & 0 & 0 & 0 & 0 & 0\\- k_{0}^{6} - k_{0}^{5} k_{1} - k_{0}^{4} k_{1}^{2} - k_{0}^{3} k_{1}^{3} - k_{0}^{2} k_{1}^{4} - k_{0} k_{1}^{5} & k_{1}^{6} & 0 & 0 & 0 & 0\\k_{0}^{5} k_{1} + k_{0}^{4} k_{1}^{2} + k_{0}^{4} k_{1} k_{2} + k_{0}^{3} k_{1}^{3} + k_{0}^{3} k_{1}^{2} k_{2} + k_{0}^{3} k_{1} k_{2}^{2} + k_{0}^{2} k_{1}^{4} + k_{0}^{2} k_{1}^{3} k_{2} + k_{0}^{2} k_{1}^{2} k_{2}^{2} + k_{0}^{2} k_{1} k_{2}^{3} + k_{0} k_{1}^{5} + k_{0} k_{1}^{4} k_{2} + k_{0} k_{1}^{3} k_{2}^{2} + k_{0} k_{1}^{2} k_{2}^{3} + k_{0} k_{1} k_{2}^{4} & - k_{1}^{6} - k_{1}^{5} k_{2} - k_{1}^{4} k_{2}^{2} - k_{1}^{3} k_{2}^{3} - k_{1}^{2} k_{2}^{4} - k_{1} k_{2}^{5} & k_{2}^{6} & 0 & 0 & 0\\- k_{0}^{4} k_{1} k_{2} - k_{0} k_{1} k_{2} k_{3}^{3} + \left(k_{0}^{3} + k_{0}^{2} k_{1} + k_{0} k_{1}^{2}\right) \left(- k_{1}^{2} k_{2} - k_{1} k_{2}^{2} - k_{1} k_{2} k_{3}\right) + \left(k_{2}^{3} + k_{2}^{2} k_{3} + k_{2} k_{3}^{2}\right) \left(- k_{0}^{2} k_{1} - k_{0} k_{1}^{2} - k_{0} k_{1} k_{2}\right) & k_{1}^{5} k_{2} + k_{1}^{4} k_{2}^{2} + k_{1}^{4} k_{2} k_{3} + k_{1}^{3} k_{2}^{3} + k_{1}^{3} k_{2}^{2} k_{3} + k_{1}^{3} k_{2} k_{3}^{2} + k_{1}^{2} k_{2}^{4} + k_{1}^{2} k_{2}^{3} k_{3} + k_{1}^{2} k_{2}^{2} k_{3}^{2} + k_{1}^{2} k_{2} k_{3}^{3} + k_{1} k_{2}^{5} + k_{1} k_{2}^{4} k_{3} + k_{1} k_{2}^{3} k_{3}^{2} + k_{1} k_{2}^{2} k_{3}^{3} + k_{1} k_{2} k_{3}^{4} & - k_{2}^{6} - k_{2}^{5} k_{3} - k_{2}^{4} k_{3}^{2} - k_{2}^{3} k_{3}^{3} - k_{2}^{2} k_{3}^{4} - k_{2} k_{3}^{5} & k_{3}^{6} & 0 & 0\\k_{0} k_{1} k_{2} \left(k_{3}^{3} + k_{3}^{2} k_{4} + k_{3} k_{4}^{2}\right) + k_{1} k_{2} k_{3} \left(k_{0}^{3} + k_{0}^{2} k_{1} + k_{0} k_{1}^{2}\right) + \left(- k_{0}^{2} k_{1} - k_{0} k_{1}^{2} - k_{0} k_{1} k_{2}\right) \left(- k_{2}^{2} k_{3} - k_{2} k_{3}^{2} - k_{2} k_{3} k_{4}\right) & - k_{1}^{4} k_{2} k_{3} - k_{1} k_{2} k_{3} k_{4}^{3} + \left(k_{1}^{3} + k_{1}^{2} k_{2} + k_{1} k_{2}^{2}\right) \left(- k_{2}^{2} k_{3} - k_{2} k_{3}^{2} - k_{2} k_{3} k_{4}\right) + \left(k_{3}^{3} + k_{3}^{2} k_{4} + k_{3} k_{4}^{2}\right) \left(- k_{1}^{2} k_{2} - k_{1} k_{2}^{2} - k_{1} k_{2} k_{3}\right) & k_{2}^{5} k_{3} + k_{2}^{4} k_{3}^{2} + k_{2}^{4} k_{3} k_{4} + k_{2}^{3} k_{3}^{3} + k_{2}^{3} k_{3}^{2} k_{4} + k_{2}^{3} k_{3} k_{4}^{2} + k_{2}^{2} k_{3}^{4} + k_{2}^{2} k_{3}^{3} k_{4} + k_{2}^{2} k_{3}^{2} k_{4}^{2} + k_{2}^{2} k_{3} k_{4}^{3} + k_{2} k_{3}^{5} + k_{2} k_{3}^{4} k_{4} + k_{2} k_{3}^{3} k_{4}^{2} + k_{2} k_{3}^{2} k_{4}^{3} + k_{2} k_{3} k_{4}^{4} & - k_{3}^{6} - k_{3}^{5} k_{4} - k_{3}^{4} k_{4}^{2} - k_{3}^{3} k_{4}^{3} - k_{3}^{2} k_{4}^{4} - k_{3} k_{4}^{5} & k_{4}^{6} & 0\\k_{0} k_{1} k_{2} \left(- k_{3}^{2} k_{4} - k_{3} k_{4}^{2} - k_{3} k_{4} k_{5}\right) + k_{2} k_{3} k_{4} \left(- k_{0}^{2} k_{1} - k_{0} k_{1}^{2} - k_{0} k_{1} k_{2}\right) & k_{1} k_{2} k_{3} \left(k_{4}^{3} + k_{4}^{2} k_{5} + k_{4} k_{5}^{2}\right) + k_{2} k_{3} k_{4} \left(k_{1}^{3} + k_{1}^{2} k_{2} + k_{1} k_{2}^{2}\right) + \left(- k_{1}^{2} k_{2} - k_{1} k_{2}^{2} - k_{1} k_{2} k_{3}\right) \left(- k_{3}^{2} k_{4} - k_{3} k_{4}^{2} - k_{3} k_{4} k_{5}\right) & - k_{2}^{4} k_{3} k_{4} - k_{2} k_{3} k_{4} k_{5}^{3} + \left(k_{2}^{3} + k_{2}^{2} k_{3} + k_{2} k_{3}^{2}\right) \left(- k_{3}^{2} k_{4} - k_{3} k_{4}^{2} - k_{3} k_{4} k_{5}\right) + \left(k_{4}^{3} + k_{4}^{2} k_{5} + k_{4} k_{5}^{2}\right) \left(- k_{2}^{2} k_{3} - k_{2} k_{3}^{2} - k_{2} k_{3} k_{4}\right) & k_{3}^{5} k_{4} + k_{3}^{4} k_{4}^{2} + k_{3}^{4} k_{4} k_{5} + k_{3}^{3} k_{4}^{3} + k_{3}^{3} k_{4}^{2} k_{5} + k_{3}^{3} k_{4} k_{5}^{2} + k_{3}^{2} k_{4}^{4} + k_{3}^{2} k_{4}^{3} k_{5} + k_{3}^{2} k_{4}^{2} k_{5}^{2} + k_{3}^{2} k_{4} k_{5}^{3} + k_{3} k_{4}^{5} + k_{3} k_{4}^{4} k_{5} + k_{3} k_{4}^{3} k_{5}^{2} + k_{3} k_{4}^{2} k_{5}^{3} + k_{3} k_{4} k_{5}^{4} & - k_{4}^{6} - k_{4}^{5} k_{5} - k_{4}^{4} k_{5}^{2} - k_{4}^{3} k_{5}^{3} - k_{4}^{2} k_{5}^{4} - k_{4} k_{5}^{5} & k_{5}^{6}\end{matrix}\right]$

```python
newMat ** 7
```

$\displaystyle \left[\begin{matrix}- k_{0}^{7} & 0 & 0 & 0 & 0 & 0\\k_{0}^{7} + k_{0}^{6} k_{1} + k_{0}^{5} k_{1}^{2} + k_{0}^{4} k_{1}^{3} + k_{0}^{3} k_{1}^{4} + k_{0}^{2} k_{1}^{5} + k_{0} k_{1}^{6} & - k_{1}^{7} & 0 & 0 & 0 & 0\\- k_{0}^{6} k_{1} - k_{0}^{5} k_{1}^{2} - k_{0}^{5} k_{1} k_{2} - k_{0}^{4} k_{1}^{3} - k_{0}^{4} k_{1}^{2} k_{2} - k_{0}^{4} k_{1} k_{2}^{2} - k_{0}^{3} k_{1}^{4} - k_{0}^{3} k_{1}^{3} k_{2} - k_{0}^{3} k_{1}^{2} k_{2}^{2} - k_{0}^{3} k_{1} k_{2}^{3} - k_{0}^{2} k_{1}^{5} - k_{0}^{2} k_{1}^{4} k_{2} - k_{0}^{2} k_{1}^{3} k_{2}^{2} - k_{0}^{2} k_{1}^{2} k_{2}^{3} - k_{0}^{2} k_{1} k_{2}^{4} - k_{0} k_{1}^{6} - k_{0} k_{1}^{5} k_{2} - k_{0} k_{1}^{4} k_{2}^{2} - k_{0} k_{1}^{3} k_{2}^{3} - k_{0} k_{1}^{2} k_{2}^{4} - k_{0} k_{1} k_{2}^{5} & k_{1}^{7} + k_{1}^{6} k_{2} + k_{1}^{5} k_{2}^{2} + k_{1}^{4} k_{2}^{3} + k_{1}^{3} k_{2}^{4} + k_{1}^{2} k_{2}^{5} + k_{1} k_{2}^{6} & - k_{2}^{7} & 0 & 0 & 0\\k_{2} \left(k_{0}^{5} k_{1} + k_{0}^{4} k_{1}^{2} + k_{0}^{4} k_{1} k_{2} + k_{0}^{3} k_{1}^{3} + k_{0}^{3} k_{1}^{2} k_{2} + k_{0}^{3} k_{1} k_{2}^{2} + k_{0}^{2} k_{1}^{4} + k_{0}^{2} k_{1}^{3} k_{2} + k_{0}^{2} k_{1}^{2} k_{2}^{2} + k_{0}^{2} k_{1} k_{2}^{3} + k_{0} k_{1}^{5} + k_{0} k_{1}^{4} k_{2} + k_{0} k_{1}^{3} k_{2}^{2} + k_{0} k_{1}^{2} k_{2}^{3} + k_{0} k_{1} k_{2}^{4}\right) - k_{3} \left(- k_{0}^{4} k_{1} k_{2} - k_{0} k_{1} k_{2} k_{3}^{3} + \left(k_{0}^{3} + k_{0}^{2} k_{1} + k_{0} k_{1}^{2}\right) \left(- k_{1}^{2} k_{2} - k_{1} k_{2}^{2} - k_{1} k_{2} k_{3}\right) + \left(k_{2}^{3} + k_{2}^{2} k_{3} + k_{2} k_{3}^{2}\right) \left(- k_{0}^{2} k_{1} - k_{0} k_{1}^{2} - k_{0} k_{1} k_{2}\right)\right) & - k_{1}^{6} k_{2} - k_{1}^{5} k_{2}^{2} - k_{1}^{5} k_{2} k_{3} - k_{1}^{4} k_{2}^{3} - k_{1}^{4} k_{2}^{2} k_{3} - k_{1}^{4} k_{2} k_{3}^{2} - k_{1}^{3} k_{2}^{4} - k_{1}^{3} k_{2}^{3} k_{3} - k_{1}^{3} k_{2}^{2} k_{3}^{2} - k_{1}^{3} k_{2} k_{3}^{3} - k_{1}^{2} k_{2}^{5} - k_{1}^{2} k_{2}^{4} k_{3} - k_{1}^{2} k_{2}^{3} k_{3}^{2} - k_{1}^{2} k_{2}^{2} k_{3}^{3} - k_{1}^{2} k_{2} k_{3}^{4} - k_{1} k_{2}^{6} - k_{1} k_{2}^{5} k_{3} - k_{1} k_{2}^{4} k_{3}^{2} - k_{1} k_{2}^{3} k_{3}^{3} - k_{1} k_{2}^{2} k_{3}^{4} - k_{1} k_{2} k_{3}^{5} & k_{2}^{7} + k_{2}^{6} k_{3} + k_{2}^{5} k_{3}^{2} + k_{2}^{4} k_{3}^{3} + k_{2}^{3} k_{3}^{4} + k_{2}^{2} k_{3}^{5} + k_{2} k_{3}^{6} & - k_{3}^{7} & 0 & 0\\k_{3} \left(- k_{0}^{4} k_{1} k_{2} - k_{0} k_{1} k_{2} k_{3}^{3} + \left(k_{0}^{3} + k_{0}^{2} k_{1} + k_{0} k_{1}^{2}\right) \left(- k_{1}^{2} k_{2} - k_{1} k_{2}^{2} - k_{1} k_{2} k_{3}\right) + \left(k_{2}^{3} + k_{2}^{2} k_{3} + k_{2} k_{3}^{2}\right) \left(- k_{0}^{2} k_{1} - k_{0} k_{1}^{2} - k_{0} k_{1} k_{2}\right)\right) - k_{4} \left(k_{0} k_{1} k_{2} \left(k_{3}^{3} + k_{3}^{2} k_{4} + k_{3} k_{4}^{2}\right) + k_{1} k_{2} k_{3} \left(k_{0}^{3} + k_{0}^{2} k_{1} + k_{0} k_{1}^{2}\right) + \left(- k_{0}^{2} k_{1} - k_{0} k_{1}^{2} - k_{0} k_{1} k_{2}\right) \left(- k_{2}^{2} k_{3} - k_{2} k_{3}^{2} - k_{2} k_{3} k_{4}\right)\right) & k_{3} \left(k_{1}^{5} k_{2} + k_{1}^{4} k_{2}^{2} + k_{1}^{4} k_{2} k_{3} + k_{1}^{3} k_{2}^{3} + k_{1}^{3} k_{2}^{2} k_{3} + k_{1}^{3} k_{2} k_{3}^{2} + k_{1}^{2} k_{2}^{4} + k_{1}^{2} k_{2}^{3} k_{3} + k_{1}^{2} k_{2}^{2} k_{3}^{2} + k_{1}^{2} k_{2} k_{3}^{3} + k_{1} k_{2}^{5} + k_{1} k_{2}^{4} k_{3} + k_{1} k_{2}^{3} k_{3}^{2} + k_{1} k_{2}^{2} k_{3}^{3} + k_{1} k_{2} k_{3}^{4}\right) - k_{4} \left(- k_{1}^{4} k_{2} k_{3} - k_{1} k_{2} k_{3} k_{4}^{3} + \left(k_{1}^{3} + k_{1}^{2} k_{2} + k_{1} k_{2}^{2}\right) \left(- k_{2}^{2} k_{3} - k_{2} k_{3}^{2} - k_{2} k_{3} k_{4}\right) + \left(k_{3}^{3} + k_{3}^{2} k_{4} + k_{3} k_{4}^{2}\right) \left(- k_{1}^{2} k_{2} - k_{1} k_{2}^{2} - k_{1} k_{2} k_{3}\right)\right) & - k_{2}^{6} k_{3} - k_{2}^{5} k_{3}^{2} - k_{2}^{5} k_{3} k_{4} - k_{2}^{4} k_{3}^{3} - k_{2}^{4} k_{3}^{2} k_{4} - k_{2}^{4} k_{3} k_{4}^{2} - k_{2}^{3} k_{3}^{4} - k_{2}^{3} k_{3}^{3} k_{4} - k_{2}^{3} k_{3}^{2} k_{4}^{2} - k_{2}^{3} k_{3} k_{4}^{3} - k_{2}^{2} k_{3}^{5} - k_{2}^{2} k_{3}^{4} k_{4} - k_{2}^{2} k_{3}^{3} k_{4}^{2} - k_{2}^{2} k_{3}^{2} k_{4}^{3} - k_{2}^{2} k_{3} k_{4}^{4} - k_{2} k_{3}^{6} - k_{2} k_{3}^{5} k_{4} - k_{2} k_{3}^{4} k_{4}^{2} - k_{2} k_{3}^{3} k_{4}^{3} - k_{2} k_{3}^{2} k_{4}^{4} - k_{2} k_{3} k_{4}^{5} & k_{3}^{7} + k_{3}^{6} k_{4} + k_{3}^{5} k_{4}^{2} + k_{3}^{4} k_{4}^{3} + k_{3}^{3} k_{4}^{4} + k_{3}^{2} k_{4}^{5} + k_{3} k_{4}^{6} & - k_{4}^{7} & 0\\k_{4} \left(k_{0} k_{1} k_{2} \left(k_{3}^{3} + k_{3}^{2} k_{4} + k_{3} k_{4}^{2}\right) + k_{1} k_{2} k_{3} \left(k_{0}^{3} + k_{0}^{2} k_{1} + k_{0} k_{1}^{2}\right) + \left(- k_{0}^{2} k_{1} - k_{0} k_{1}^{2} - k_{0} k_{1} k_{2}\right) \left(- k_{2}^{2} k_{3} - k_{2} k_{3}^{2} - k_{2} k_{3} k_{4}\right)\right) - k_{5} \left(k_{0} k_{1} k_{2} \left(- k_{3}^{2} k_{4} - k_{3} k_{4}^{2} - k_{3} k_{4} k_{5}\right) + k_{2} k_{3} k_{4} \left(- k_{0}^{2} k_{1} - k_{0} k_{1}^{2} - k_{0} k_{1} k_{2}\right)\right) & k_{4} \left(- k_{1}^{4} k_{2} k_{3} - k_{1} k_{2} k_{3} k_{4}^{3} + \left(k_{1}^{3} + k_{1}^{2} k_{2} + k_{1} k_{2}^{2}\right) \left(- k_{2}^{2} k_{3} - k_{2} k_{3}^{2} - k_{2} k_{3} k_{4}\right) + \left(k_{3}^{3} + k_{3}^{2} k_{4} + k_{3} k_{4}^{2}\right) \left(- k_{1}^{2} k_{2} - k_{1} k_{2}^{2} - k_{1} k_{2} k_{3}\right)\right) - k_{5} \left(k_{1} k_{2} k_{3} \left(k_{4}^{3} + k_{4}^{2} k_{5} + k_{4} k_{5}^{2}\right) + k_{2} k_{3} k_{4} \left(k_{1}^{3} + k_{1}^{2} k_{2} + k_{1} k_{2}^{2}\right) + \left(- k_{1}^{2} k_{2} - k_{1} k_{2}^{2} - k_{1} k_{2} k_{3}\right) \left(- k_{3}^{2} k_{4} - k_{3} k_{4}^{2} - k_{3} k_{4} k_{5}\right)\right) & k_{4} \left(k_{2}^{5} k_{3} + k_{2}^{4} k_{3}^{2} + k_{2}^{4} k_{3} k_{4} + k_{2}^{3} k_{3}^{3} + k_{2}^{3} k_{3}^{2} k_{4} + k_{2}^{3} k_{3} k_{4}^{2} + k_{2}^{2} k_{3}^{4} + k_{2}^{2} k_{3}^{3} k_{4} + k_{2}^{2} k_{3}^{2} k_{4}^{2} + k_{2}^{2} k_{3} k_{4}^{3} + k_{2} k_{3}^{5} + k_{2} k_{3}^{4} k_{4} + k_{2} k_{3}^{3} k_{4}^{2} + k_{2} k_{3}^{2} k_{4}^{3} + k_{2} k_{3} k_{4}^{4}\right) - k_{5} \left(- k_{2}^{4} k_{3} k_{4} - k_{2} k_{3} k_{4} k_{5}^{3} + \left(k_{2}^{3} + k_{2}^{2} k_{3} + k_{2} k_{3}^{2}\right) \left(- k_{3}^{2} k_{4} - k_{3} k_{4}^{2} - k_{3} k_{4} k_{5}\right) + \left(k_{4}^{3} + k_{4}^{2} k_{5} + k_{4} k_{5}^{2}\right) \left(- k_{2}^{2} k_{3} - k_{2} k_{3}^{2} - k_{2} k_{3} k_{4}\right)\right) & - k_{3}^{6} k_{4} - k_{3}^{5} k_{4}^{2} - k_{3}^{5} k_{4} k_{5} - k_{3}^{4} k_{4}^{3} - k_{3}^{4} k_{4}^{2} k_{5} - k_{3}^{4} k_{4} k_{5}^{2} - k_{3}^{3} k_{4}^{4} - k_{3}^{3} k_{4}^{3} k_{5} - k_{3}^{3} k_{4}^{2} k_{5}^{2} - k_{3}^{3} k_{4} k_{5}^{3} - k_{3}^{2} k_{4}^{5} - k_{3}^{2} k_{4}^{4} k_{5} - k_{3}^{2} k_{4}^{3} k_{5}^{2} - k_{3}^{2} k_{4}^{2} k_{5}^{3} - k_{3}^{2} k_{4} k_{5}^{4} - k_{3} k_{4}^{6} - k_{3} k_{4}^{5} k_{5} - k_{3} k_{4}^{4} k_{5}^{2} - k_{3} k_{4}^{3} k_{5}^{3} - k_{3} k_{4}^{2} k_{5}^{4} - k_{3} k_{4} k_{5}^{5} & k_{4}^{7} + k_{4}^{6} k_{5} + k_{4}^{5} k_{5}^{2} + k_{4}^{4} k_{5}^{3} + k_{4}^{3} k_{5}^{4} + k_{4}^{2} k_{5}^{5} + k_{4} k_{5}^{6} & - k_{5}^{7}\end{matrix}\right]$

**Directions**
1. Consider upper triangular ${\bf A}$ since ${\bf A}^n$ is lower triangular and the eigenvalues are the diagonal. (A lower triangular network is feedforward.)
    1. What about the eigenvectors for an upper triangular matrix?

# Preliminaries

```python
NRMSE = "nrmse"  # Normalized root of the mean square error (residuals)
```

```python
MODEL_DIR = os.path.join(os.getcwd(), "../models")
```

```python
path = os.path.join(MODEL_DIR, "Jana_WolfGlycolysis.antimony")
with open(path, "r") as fd:
    GMODEL = fd.readlines()
GMODEL = "\n".join(GMODEL)
```

```python
print(GMODEL)
```

    // Created by libAntimony v2.12.0.3

    model *Jana_WolfGlycolysis()

      // Compartments and Species:

      compartment compartment_;

      species Glucose in compartment_, fructose_1_6_bisphosphate in compartment_;

      species glyceraldehyde_3_phosphate in compartment_, glycerate_3_phosphate in compartment_;

      species pyruvate in compartment_, Acetyladehyde in compartment_, External_acetaldehyde in compartment_;

      species ATP in compartment_, ADP in compartment_, NAD in compartment_, NADH in compartment_;

      species $External_glucose in compartment_, $ethanol in compartment_, $Glycerol in compartment_;

      species $Sink in compartment_;

      // Reactions:

      J0: $External_glucose => Glucose; J0_inputFlux;

      J1: Glucose + 2 ATP => fructose_1_6_bisphosphate + 2 ADP; J1_k1*Glucose*ATP*(1/(1 + (ATP/J1_Ki)^J1_n));

      J2: fructose_1_6_bisphosphate => glyceraldehyde_3_phosphate + glyceraldehyde_3_phosphate; J2_k*fructose_1_6_bisphosphate;

      J3: glyceraldehyde_3_phosphate + NADH => NAD + $Glycerol; J3_k*glyceraldehyde_3_phosphate*NADH;

      J4: glyceraldehyde_3_phosphate + ADP + NAD => ATP + glycerate_3_phosphate + NADH; (J4_kg*J4_kp*glyceraldehyde_3_phosphate*NAD*ADP - J4_ka*J4_kk*glycerate_3_phosphate*ATP*NADH)/(J4_ka*NADH + J4_kp*ADP);

      J5: glycerate_3_phosphate + ADP => ATP + pyruvate; J5_k*glycerate_3_phosphate*ADP;

      J6: pyruvate => Acetyladehyde; J6_k*pyruvate;

      J7: Acetyladehyde + NADH => NAD + $ethanol; J7_k*Acetyladehyde*NADH;

      J8: Acetyladehyde => External_acetaldehyde; J8_k1*Acetyladehyde - J8_k2*External_acetaldehyde;

      J9: ATP => ADP; J9_k*ATP;

      J10: External_acetaldehyde => $Sink; J10_k*External_acetaldehyde;

      // Species initializations:

      Glucose = 0;

      fructose_1_6_bisphosphate = 0;

      glyceraldehyde_3_phosphate = 0;

      glycerate_3_phosphate = 0;

      pyruvate = 0;

      Acetyladehyde = 0;

      External_acetaldehyde = 0;

      ATP = 3;

      ADP = 1;

      NAD = 0.5;

      NADH = 0.5;

      External_glucose = 0;

      ethanol = 0;

      Glycerol = 0;

      Sink = 0;

      // Compartment initializations:

      compartment_ = 1;

      // Variable initializations:

      J0_inputFlux = 50;

      J1_k1 = 550;

      J1_Ki = 1;

      J1_n = 4;

      J2_k = 9.8;

      J3_k = 85.7;

      J4_kg = 323.8;

      J4_kp = 76411.1;

      J4_ka = 57823.1;

      J4_kk = 23.7;

      J5_k = 80;

      J6_k = 9.7;

      J7_k = 2000;

      J8_k1 = 375;

      J8_k2 = 375;

      J9_k = 28;

      J10_k = 80;

      // Other declarations:

      const compartment_, J0_inputFlux, J1_k1, J1_Ki, J1_n, J2_k, J3_k;

      const J4_kg, J4_kp, J4_ka, J4_kk, J5_k, J6_k, J7_k, J8_k1, J8_k2;

      const J9_k, J10_k;

    end

# One Species (First Order)

## Numerical Studies

Consider the pathway $X0 \xrightarrow{k_1} x \xrightarrow{k_2} X_1$, where $X_0$ and $X1$ have fixed concentrations and reactions kinetics
are mass action with kinetic constants $k_i$.
Then, $\frac{dx}{dt} = k_1 X_0 - k_2 x$.
We want to explore the fitting surface for $k_1, K_2$.

```python
MODEL = """

J1: $X0 -> x; k1*X0
J2: x -> $X1; k2*x

X0 = 1
x = 0
k1 = 1
k2 = 1
"""
```

```python
PARAMETER_DCT = {"k1": 1, "k2": 2}
```

```python
rr = te.loada(MODEL)
trueData = rr.simulate()
rr.plot(trueData)
```

```python
analyzer = SurfaceAnalyzer(MODEL, PARAMETER_DCT)
scales = [10, 2, 0.1, 0.03]
k2s = [0.5, 1.0, 2.0]
k2s = [1.0]
for scale in scales:
    for k2 in k2s:
        parameterDct = dict(PARAMETER_DCT)
        parameterDct["k2"] = k2
        analyzer = SurfaceAnalyzer(MODEL, parameterDct)
        analyzer.runExperiments(0.5, 50)
        title = "k2: %2.2f, scale: %2.4f" % (k2, scale)
        analyzer.plotSurface(scale=scale, title=title, xlim=[0.5, 1.5], ylim=[0.5, 1.5])
```

**observations**
1. Convex surface
1. Lower values on line with slope $k_1 / k_2$.
1. Steeper slope for minima for $k_2$ compared with $k_1$.

## Analysis

**Result**: Parameter fitting for a one species system is a convex optimization.
Given observational data $y(t)$ for a one species system and $t_{ss}$ such that
$y(t) = x_{ss}$ for $t \geq t_{ss}$.
Let $x(t; k_1, k_2)$ be the system with parameters $k_1, k_2$ and
$k_1^{\star}, k_2^{\star}$ such that $y(t) = x(t; k_1^{\star}, k_2^{\star})$.
We claim that $k_1^{\star}, k_2^{\star}$ are the solution to the following convex optimization:
minimize $\sum_t \left( y(t) - x(t; k_1, k_2) \right) ^2$ such that $\frac{k_1}{k_2} = x_{ss}$.

**Problem setup**
1. The one species system with a constant input $X_0$ and initial condition $x(0)=0$ has the solution.
$x(t) = X_0 \frac{k_1}{k_2}(1 - e^{-k_2 t})$. **[Generalize to non-zero initial conditions]**

1. The residuals are $r(t) =  y(t) - x(t; k_1, k_2) = x(t; k_1^{\star}, k_1^{\star}) - x(t; k_1, k_2)$

1. The sum of squares of the residuals is $R_{SSQ} = \int_0^{\infty} r(t)^2 dt$.

**Analysis for $\frac{k_1}{k_2} = \frac{k^{\star}_1}{k^{\star}_2}$**
1. $r(t) =  A( -e^{-k_2 t} + e^{-k^{\star}_2 t})$, where $A = X_0 \frac{k_1^{\star}}{k_2^{\star}}$.

1. $R_{SSQ} = A^2 \int_0^{\infty} \left( e^{-2k_2 t} -2 e^{-(k_2 + k^{\star}_2)t} + e^{-2k^{\star}_2t} \right)$

1. And so, $R_{SSQ} = A^2 \left( - \frac{1}{2k_2} e^{-2k_2 t} +  \frac{2}{k_2 + k^{\star}_2} e^{-(k_2 + k^{\star}_2)t} - \frac{1}{2k^{\star}_2} e^{-2k^{\star}_2t} \right) \biggr\vert_0^{\infty}$

1. $R_{SSQ} = A^2 \left( \frac{1}{2k_2} -  \frac{2}{k_2 + k^{\star}_2} + \frac{1} {2 k^{\star}_2}  \right)$

In the sequel, we simplify notation by using
$g(u; u^{\star}) = \frac{1}{2 u^{\star}} -  \frac{2}{u + u^{\star}} + \frac{1} {2 u}$, for $u, u^{\star} > 0$.

**Claim:** Given $g(u; u^{\star})$ as defined above:
1. $g(u; u^{\star})$ is positive semidefinite.
1. $g(u; u^{\star}) = 0$ iff $u = u^{\star}$
1. $u^{\star}$ is the solution to a convex optimization.

**Proof:**
The first two claims are established by the following:
$
g(u; u^{\prime})  \geq  0 \equiv u(u + u^{\star}) - 4 u u^{\star} + u^{\star} (u + u^{\star}) \geq 0
\equiv (u - u^{\star}) ^ 2 \geq 0$.
The third claim follow from the fact that minimizing $g(u; u^{\prime})$ is equivalent to solving a convex optimization.

**Claim**: $R_{SSQ}$ is convex in $k^{\star}_2$. **[This may be unnecessary.]**

1. Claim that $\frac{\partial T_{SE}}{\partial k^{\star}} \geq 0$.

1. Note that $\frac{\partial^2 R_{SSQ}}{\partial^2 k^{\star}_2}
= A^2 \left( \frac{1}{k^3_2} -  \frac{4 }{(k_2 + k^{\star}_2)^3} + \frac{1} {( k^{\star}_2)^3}  \right)$

1. Suffices to show that $ \left( \frac{1}{k^3_2} -  \frac{4 }{(k_2 + k^{\star}_2)^3} + \frac{1} {( k^{\star}_2)^3}  \right) \geq 0$

1. Equivalent to  $(k^{\star}_2)^3  (k^{\star}_2 + k_2)^3 -  4 k^3_2 (k^{\star}_2)^3  + k^3_2 (k^{\star}_2 + k_2)^3 \ge0$.
Let $x = k_2$ and $y = x^{\star}_2$. Then, we have
$x^3 (x + y)^3 -4 x^3 y^3 + y^3 (x + y)^3 \geq 0$, where $x, y > 0$.

$
\begin{align}
T & = & x^3 (x^3 + 3x^2y + 3xy^2 + y^3) - 4 x^3 y^3 + y^3 (x^3 + 3x^2y + 3xy^2 + y^3)  \\
& = & x^3 (x^3 + 3x^2y + 3xy^2) - 2 x^3 y^3 + y^3 ( 3x^2y + 3xy^2 + y^3)   \\
 & = & \left( x^3 - y^3 \right) ^2 + 6x^2y + 6 x y^2 \\
 & \ge 0
\end{align}
$

# Two Species (Second Order)

## Numerial Studies

## Analysis

Second order system: $y^{\prime \prime} + a y^{\prime} + b y = u(t)$

**Solution**

LaPlace Transform: $Y(s) = \frac{U(s)}{s^2 + a s + b}$

**Notes**
1. A key element in minimizing residuals over a longer time course is getting the correct steady state value. Need to constrain search for system parameters based on steady state.

# Convex Network

One direction is to establish a class of reaction networks for which fitting is a convex function.
My hypothesis is that uni-reactant, feedforward networks have a convex objective function under
the $l_2$ norm.
This is non-trival to prove.
So, a first step is to do a numerical evaluation.

# Linear System

## Convergence for Known A Matrix

Consider a network that is an LTI system. For example, this holds for the following class of reaction networks:
1. Mass action kinetics for at most one reactant.
1. A constant is used in combination with exactly one chemical species.

Let ${\bf A}^{\star}$ be the true matrix for the LTI system, and let ${\bf A} ({\bf k})$ be an estimate of the matrix in terms of the parameters
${\bf k} = (k_1, \cdots, k_N)$.
Our objective function to minimize is $g({\bf k}) = | {\bf A}^{\star} - {\bf A} ({\bf k}) |^2$.
It suffices to show that $\frac {\partial^2g (k_n)}{\partial ^2 k_n} \geq 0$ for each $k_n$.

$
g({k_n})  = | {\bf A}^{\star} - {\bf A} ({k_n}) |^2
$

And so
$\frac{\partial g(k_n)}{\partial k_n} =
2 | {\bf A}^{\star} - {\bf A} ({k_n}) |
\frac{\partial |A^{\star} - A (k_n)|}{\partial k_n}.
$

Finally,
$\frac{\partial^2 g(k_n)}{\partial^2 k_n} =
2
\left( \frac{\partial |A^{\star} - A (k_n)|}{\partial k_n} \right) ^2
- 2 | {\bf A}^{\star} - {\bf A} ({k_n}) | \frac{\partial^2 |A^{\star} - A (k_n)|}{\partial^2 k_n}.
$

We claim that
$
\frac{\partial^2 |A (k_n)|}{\partial^2 k_n} = 0.
$
This follows from the fact that $k_n$ occurs in only one column since it is associated with only one state variable.
As such, $k_n$ occurs in a product term of the determinant with a power of either 0 or 1.
The second derivative of such terms is 0; and the first derivative of terms without $k_n$ is 0.

**Issues**
1. What does this mean since $A^{\star}$ is unknown?

1. Let ${\bf x}^{\star}$ be the solution to $\dot{{\bf x}}^{\star} = {\bf A}^{\star} {\bf x}^{\star}$.
We have ${\bf x}^{\star}$.
So, we should consider
$h(k_n) = | {\bf x} (0) e^{ {\bf A}^{\star} t } - {\bf x}(0) e^{ {\bf A (k_n)} t } | \approx
| {\bf x}^{\star} (t)  - {\bf A} (k_n) {\bf x} (0) t |.$
Can make an argument for the $L1$ norm and that the differences are minimized if the matrices converge.

## Convergence of Residuals

Let $A({\bf k})$ be the parameterization of the a linear system with true parameters ${\bf k}^{\star}$.
Define ${\bf x}^{\star}$ as the true solution, and ${\bf r}(t, {\bf k}) =  {\bf x}^{\star} - e^{A({\bf k}) t} x(0)$.
Let
$g({\bf k}) = \int {\bf r}^T (t, {\bf k}) {\bf r} (t, {\bf k})dt$.
Show that $g({\bf k})$ is convex.

1. It suffices to show that $g$ is convex in each $k_n$ for each $t$.
1. $e^{A({\bf k} t} x(0) = \left( \sum_{n=0}^{\infty} \frac{ \left( A({\bf k}) \right)^n t^n}{n!} \right) {\bf x}(0)$.

1. ${\bf r}(t) = \left( \sum_{n=0}^{\infty} \frac{ \left( {\bf A}^{\star} \right)^n - \left( {\bf  A}({\bf k}) \right)^n t^n} {n!}
\right) {\bf x}(0)$.

1. Let $a_{ij}$ be the $i,j$ element of $\left( {\bf A} (k) \right) ^n$. Note that $a_{ij} = c_m k^m + \cdots + c_0 = P_{ij}^m(k)$,
where $m \leq 2^{n-1}$.
Define $p({\bf A} (k)) = \{ P_{ij}^m \}$.

1. Let $P^m(k)$ be an $m$-th order polynomial in $k$.
   1. $\frac{\partial P^m(k)}{\partial k} = P^{m-1}(k)$.
   1. $sign(c_n^m) = sign(c_{n-1}^{m-1})$ for $n > 0$; otherwise, $sign(c_n) = 0$.
   1. The order of the polynomial of $k$ in ${\bf A}(k)$ is the same as that for
   ${\bf A}(k) {\bf x}$, for a vector ${\bf x}$ that does not contain $k$.
   1. The order of the polynomial of $k$ in ${\bf A}(k)$ is the same as that for
   ${\bf A}^{\star} - {\bf A}(k)$, for a matrix ${\bf A}^{\star}$ that does not contain $k$.
   1. The order of the polynomial of $k$ in ${\bf A}_1(k) + {\bf A}_2(k)$ is
   the max of the order of the polynomials for ${\bf A}_1(k), {\bf A}_2(k)$.

1.
\begin{align*}
\frac{\partial^2 {\bf r}^T (t, k) {\bf r} (t, k)}{\partial^2 k}
& = &
   \frac {\partial^2 {\bf r}^T (t, k)} {\partial^2 k} {\bf r} (t, k) +
   2 \frac {\partial {\bf r}^T (t, k)} {\partial k}  \frac {\partial {\bf r} (t, k)} {\partial k} +
   {\bf r}^T (t, k)  \frac {\partial^2 {\bf r} (t, k)} {\partial^2 k} \\
\end{align*}

1. A first order Taylor series approximation of $k$ is convex because:
   1. ${\bf A (k)} = \{P^1_{ij} \}$
   1. The second derivatives of ${\bf r}(t, k) \}$ are 0 and the first derviative is squared, and so is positive.

1. If ${\bf A}$ is a decompled system, then ${\bf A}$ a diagnonal matrix. This could allow for some reduction in dimensionality for parameter estimation.
We still may have high order polynomials in $k$ for $a_{ii}$.

1. The key consideration is the sign of terms in the polynomials.
    1. If they are always positive, then we get convexity since
terms without $k$ disappear because of the derivative.
    1. If terms of the same sign are multiplied, then the result is positive.
    1. Terms on the diagnonal tend to be negative since they are degradation rates for a species.

1. For $A$, and $N \times N$ matrix with $k$ only in $a_{ij}$, find $m$ such that $A^{m+1}$ has the same cells with $k$ as $A^m$.

1. Suppose $X_i$ is the reactant in a reaction that produces $X_j$ with mass action kinetics and constant $k$.
Then, $k$ has a positive sign in $a_{ji}$ and a negative sign in $a_{ij}$.
Let $B = A^2$.
Then,
   * $b_{ii} = b_{jj}$ has the term $-k^2$
   * $b_{ik}$ has the term $-k$
   * $b_{kj}$ has the term $k$

1. Let $\{ a^m_{ij}\} = {\bf A}^m (k)$.
The sign of $a^m_{ij}$ depends on the sign of non-$k$ terms in the matrix.
In particular, the negative diagnonal.

1. The two main considerations in this analysis is the order of $P^m_{ij}(k)$ and the sign of the terms in the polynomial.

1. Another strategy is to construct an approximation to ${\bf x} (k)$ based on parts of different matrices in the
exact solution such that all terms with $k$ are positive.
This will yield a convex function.
The construction of this approximation depends on ${\bf A}$, where $-k$ occurs in $\left( {\bf A} (k) \right)^m$.
This seems feasible to construct if:

   * $ \{ a_{ij} \} $ is the some of terms of the same sign and
   * $k$ occurs only in only a few locations: $i,i$; $i,j$ and $j,i$; $i,j$, $j_1, i, \cdots, j_k, i$ (branching)

1. A second order approximation is convex since the only terms with $k$ that are second order are positive as long as their is no order 2 feedback (e.g., $X \rightarrow Y \rightarrow X$.
Can I generalize this to doing an approximation of order $m$ if there is no feedback of order $m \leq N$?

1. Assuming that only $x_1$ has a non-zero initial condition, then ${\bf x}(0)$ results in just the first column being selected. However, we may need more terms in the approximation for it to be of reasonable quality.

1. **Observation**. Let $r(t, {\bf k}) = {\bf x}^* (t) - \left( {\bf I} + {\bf A}({\bf k}) \right) {\bf x} (0) t$. Then, $min~ g({\bf k}) = min \int {\bf r}^T (t, {\bf k}) {\bf r} (t, {\bf k})$ is convex.

## Algorithm 1: Linear Approximations

Below outlines an algorithm for parameter estimation based on the observation that a first order approximation of a linear system has a convex optimization objective for $l^2$.

1. Given ${\bf x}^{\star} (t), {\bf v}({\bf k}, {\bf x})$, the vector of reaction rates.

1. Symbolically calculate ${\bf V}( {\bf k}, {\bf x}) = \frac{\partial v}{\partial x}$. ${\bf A}_i ({\bf k}) = {\bf V} ({\bf k}) |_{t = t_i}$. Note that ${\bf A}({\bf k})$ has unevaluated ${\bf k}$.

1. ${\bf d} (i+1, i) = {\bf \delta x}^{\star}_{t_{i+1}} - {\bf A}_i ({\bf k}) {\bf x}^{\star}_i (t_{i +1} - t_{i})$,
where $\delta {\bf x}^{\star}_i = {\bf x}^{\star}_{t_{i+1}} - {\bf x}^{\star}_{t_{i}}$.

1. $r_i ({\bf k}) = {\bf d} (i-1, i) + {\bf d}(i+1, i)$. This is a two sided difference. It could be redefined as a one sided difference.

1. $g({\bf k}) = \sum_i r^T_i ({\bf k}) r_i ({\bf k})$
1. Find ${\bf k}$ that minimizes $g({\bf k})$. Note that since each individual problem is convex, their sum is convex.

**Issues**
1. Accuracy of the linear approximation

## Algorithm 2: Decoupling

Assume that time courses are available for all floating species.

    for X in states
      fit the parameters for the state equation for X
    reconcile differences in parameter fits since the same parameter may be present in different state equations

**Appeal**
1. No approximations

**Issues**
1. Must have measurements of all state variables
1. Must reconcile differences in parameter fits (unlike S-systems, where parameters exist in only one state equation). An
example of this is the production of a species and its consumption are in different state equations.

## Analysis of the ${\bf A}$ Matrix

```python
Amat = sympy.symbols("Amat")
Amat = sympy.Matrix( [ [1,-1, 1], [1, 1, 1], [1, 1, 1]])
Amat**2
```

$\displaystyle \left[\begin{matrix}1 & -1 & 1\\3 & 1 & 3\\3 & 1 & 3\end{matrix}\right]$

# Numerical Analysis of Fitting Surface of Linear Pathway

# Arbitray Systems

## Glycolytic Oscillations

```python
parameterDct = {"J1_Ki": 1, "J1_k1": 550}
analyzer = SurfaceAnalyzer(GMODEL, parameterDct)
for scale in [0.0004]:
    analyzer.runExperiments(1.0, 50)
    analyzer.plotSurface(scale=scale)
```

```python
parameterDct = {"J1_Ki": 1, "J2_k": 9.8}
analyzer = SurfaceAnalyzer(GMODEL, parameterDct)
for scale in [0.03]:
    analyzer.runExperiments(0.2, 50)
    analyzer.plotSurface(scale=scale)
```

```python
parameterDct = {"J1_k1": 550, "J2_k": 9.8}
analyzer = SurfaceAnalyzer(GMODEL, parameterDct)
for scale in [0.0005]:
    analyzer.runExperiments(1.0, 40)
    analyzer.plotSurface(scale=scale)
```

# FFL Analysis

```python
A = sympy.eye(3)
b = sympy.Matrix([10, 0, 0])
system = A, b
su.addSymbols("x0 x1 x2")
x = sympy.Matrix([x0, x1, x2])
sympy.linsolve(system, *x)
```

$\displaystyle \left\{\left( 10, \  0, \  0\right)\right\}$

```python
su.addSymbols("k1 k2 k3")
Amat = sympy.Matrix( [[0, 0, 0, 0], [k1, -k2, 0, 0], [0, k2, -k3, 0], [0, 0, 0, 0]])
Amat.eigenvects()
```

    [(0,
      2,
      [Matrix([
       [k3/k1],
       [k3/k2],
       [    1],
       [    0]]),
       Matrix([
       [0],
       [0],
       [0],
       [1]])]),
     (-k2,
      1,
      [Matrix([
       [            0],
       [(-k2 + k3)/k2],
       [            1],
       [            0]])]),
     (-k3,
      1,
      [Matrix([
       [0],
       [0],
       [1],
       [0]])])]

```python
su.addSymbols("X Y Z k1 k2")
aMat = sympy.Matrix([ [0, 0, 0, 0], [1, -(1 + k2), 0, 0], [0, 1, -k1, 0], [0, k2, k1, -1]])
aMat
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0\\1 & - k_{2} - 1 & 0 & 0\\0 & 1 & - k_{1} & 0\\0 & k_{2} & k_{1} & -1\end{matrix}\right]$

```python
initialVec = sympy.Matrix([1, 0, 0, 0])
initialVec
```

$\displaystyle \left[\begin{matrix}1\\0\\0\\0\end{matrix}\right]$

```python
model = LTIModel(aMat, initialVec)
model.solve()
sympy.simplify(model.solutionVec)
```

$\displaystyle \left[\begin{matrix}1.0\\\frac{\left(1.0 e^{t \left(k_{2} + 1.0\right)} - 1.0\right) e^{- t \left(k_{2} + 1.0\right)}}{k_{2} + 1}\\\frac{1.0 \left(k_{1} e^{k_{1} t} - k_{1} e^{t \left(k_{1} + k_{2} + 1.0\right)} - k_{2} e^{t \left(k_{2} + 1.0\right)} + k_{2} e^{t \left(k_{1} + k_{2} + 1.0\right)} - e^{t \left(k_{2} + 1.0\right)} + e^{t \left(k_{1} + k_{2} + 1.0\right)}\right) e^{- t \left(k_{1} + k_{2} + 1.0\right)}}{k_{1} \left(- 1.0 k_{1} k_{2} - 1.0 k_{1} + 1.0 k_{2}^{2} + 2.0 k_{2} + 1.0\right)}\\- \frac{1.0 \left(1.0 k_{1} - 1.0 k_{2}\right) e^{- t \left(k_{2} + 1.0\right)}}{- 1.0 k_{1} k_{2} + 1.0 k_{2}^{2} + 1.0 k_{2}} + 1.0 - \frac{1.0 e^{- k_{1} t}}{1.0 k_{1}^{2} - 1.0 k_{1} k_{2} - 2.0 k_{1} + 1.0 k_{2} + 1.0} - \frac{1.0 \left(1.0 k_{1} k_{2} + 1.0 k_{1} - 1.0 k_{2}\right) e^{- 1.0 t}}{1.0 k_{1} k_{2} - 1.0 k_{2}}\end{matrix}\right]$

```python
sympy.expand(model.solutionVec)
```

$\displaystyle \left[\begin{matrix}1.0\\- \frac{1.0}{k_{2} e^{1.0 t} e^{k_{2} t} + e^{1.0 t} e^{k_{2} t}} + \frac{1.0}{k_{2} + 1}\\- \frac{1.0 k_{1}}{1.0 k_{1}^{2} k_{2} e^{1.0 t} e^{k_{2} t} + 1.0 k_{1}^{2} e^{1.0 t} e^{k_{2} t} - 1.0 k_{1} k_{2}^{2} e^{1.0 t} e^{k_{2} t} - 2.0 k_{1} k_{2} e^{1.0 t} e^{k_{2} t} - 1.0 k_{1} e^{1.0 t} e^{k_{2} t}} + \frac{1.0 k_{1}}{1.0 k_{1}^{2} k_{2} + 1.0 k_{1}^{2} - 1.0 k_{1} k_{2}^{2} - 2.0 k_{1} k_{2} - 1.0 k_{1}} + \frac{1.0 k_{2}}{1.0 k_{1}^{2} k_{2} e^{k_{1} t} + 1.0 k_{1}^{2} e^{k_{1} t} - 1.0 k_{1} k_{2}^{2} e^{k_{1} t} - 2.0 k_{1} k_{2} e^{k_{1} t} - 1.0 k_{1} e^{k_{1} t}} - \frac{1.0 k_{2}}{1.0 k_{1}^{2} k_{2} + 1.0 k_{1}^{2} - 1.0 k_{1} k_{2}^{2} - 2.0 k_{1} k_{2} - 1.0 k_{1}} + \frac{1.0}{1.0 k_{1}^{2} k_{2} e^{k_{1} t} + 1.0 k_{1}^{2} e^{k_{1} t} - 1.0 k_{1} k_{2}^{2} e^{k_{1} t} - 2.0 k_{1} k_{2} e^{k_{1} t} - 1.0 k_{1} e^{k_{1} t}} - \frac{1.0}{1.0 k_{1}^{2} k_{2} + 1.0 k_{1}^{2} - 1.0 k_{1} k_{2}^{2} - 2.0 k_{1} k_{2} - 1.0 k_{1}}\\- \frac{1.0 k_{1} k_{2}}{1.0 k_{1} k_{2} e^{1.0 t} - 1.0 k_{2} e^{1.0 t}} - \frac{1.0 k_{1}}{- 1.0 k_{1} k_{2} e^{1.0 t} e^{k_{2} t} + 1.0 k_{2}^{2} e^{1.0 t} e^{k_{2} t} + 1.0 k_{2} e^{1.0 t} e^{k_{2} t}} - \frac{1.0 k_{1}}{1.0 k_{1} k_{2} e^{1.0 t} - 1.0 k_{2} e^{1.0 t}} + \frac{1.0 k_{2}}{- 1.0 k_{1} k_{2} e^{1.0 t} e^{k_{2} t} + 1.0 k_{2}^{2} e^{1.0 t} e^{k_{2} t} + 1.0 k_{2} e^{1.0 t} e^{k_{2} t}} + \frac{1.0 k_{2}}{1.0 k_{1} k_{2} e^{1.0 t} - 1.0 k_{2} e^{1.0 t}} + 1.0 - \frac{1.0}{1.0 k_{1}^{2} e^{k_{1} t} - 1.0 k_{1} k_{2} e^{k_{1} t} - 2.0 k_{1} e^{k_{1} t} + 1.0 k_{2} e^{k_{1} t} + 1.0 e^{k_{1} t}}\end{matrix}\right]$

```python
su.addSymbols("k0 k1 k2")
SUBS = {k0: 1, k1: 1, k2: 3}
model.plot(0, 10, 100, subs=SUBS)
```

```python
import tellurium as te
MODEL = """
$X0 -> X; 1*X0
X -> Y; k0*X
Y -> Z; k1*Y
X -> Z; k2*X
Z -> ; 1*Z

$X0 = 1
k0 = 1
k1 = 2
k2 = 3
"""
rr = te.loada(MODEL)
rr.plot(rr.simulate(0, 10, 100))
```

```python
x = np.exp(-3+2j)
np.angle(x)
```

    2.0

# Solving Differential Equations with Complex Eigenvalues

```python
aMat = sympy.Matrix([ [1, 2], [-10, 1]])
aMat
```

$\displaystyle \left[\begin{matrix}1 & 2\\-10 & 1\end{matrix}\right]$

```python
eigenDct = {sympy.N(k): v for k, v in aMat.eigenvals().items()}
eigenDct
```

    {1.0 - 4.47213595499958*I: 1, 1.0 + 4.47213595499958*I: 1}

```python
result = aMat.eigenvects()
eigenVec1 = result[0][2][0]
eigenval1 = result[0][0]
eigenVec2 = result[1][2][0]
eigenval2 = result[1][0]
```

```python
aMat * eigenVec1
```

$\displaystyle \left[\begin{matrix}2 + \frac{\sqrt{5} i}{5}\\1 - 2 \sqrt{5} i\end{matrix}\right]$

```python
sympy.simplify(eigenVec1 * eigenval1)
```

$\displaystyle \left[\begin{matrix}2 + \frac{\sqrt{5} i}{5}\\1 - 2 \sqrt{5} i\end{matrix}\right]$

```python
eigenVec1, eigenVec2
```

    (Matrix([
     [sqrt(5)*I/5],
     [          1]]),
     Matrix([
     [-sqrt(5)*I/5],
     [           1]]))

# Notes

1. Objectives
   1. Research - relate that nature of the fitting surface to the chemical network. Show in which cases the fitting surface is convex. Provide insights into strategies for fitting.
   1. Teaching - provide insights into what is happening with the different fitting algorithms by animating their steps.

1. Research directions
   1. Fitting surfaces by network type: linear, branched, feedback.
   1. Preliminary insights
      1. Detecting easily fit networks
      1. When do starting positions matter. For which parameters (because of combinatorics). How narrow parameter ranges (to reduce combinatorics).

1. A strategy for a linear chain (and maybe more general networks) is to first optimize the ratio of species concentrations.
But which ratios?

1. One strategy is to synthesize first order systems and solve them separately or recursively. Consider a linear system.
   1. Remove the constant term, since this is steady state.
   1. Integrate the rest.
   1. If the result is 0, stop.
   1. Otherwise, go to (a)

   Solve the last system found, since this is the high order system. Use these constants, and repeat.

1. To do
    1. Show simulations for different points in the curve, relating the simulation results to $R^2$.
    1. Look at longer linear chains.
    1. Examine Wolf model