# Variational Autoencoder
The [Basic Concepts](./Basic%20Concepts.ipynb) notebook serves as an introduction to variational inference by means of toy examples. The purpose of this notebook is to demonstrate the application of variational inference in tackling a real life problem - that of generating natural images. The broad problem that is being addessed here is generative modeling. This notebook is based on the paper by ([Kingma et al. 2013](https://arxiv.org/pdf/1312.6114)).

## Generative Models
Given a dataset, $\mathcal{D} = \{x^{(1)}, x^{(2)}, ..., x^{(N)}\}$, generative models are probabilistic models that can be used to generate data "similar" to that in $\mathcal{D}$. The aim is to model $p(x)$ using $\mathcal{D}$ and then subsequently generate "similar" data by sampling from $p(x)$. For example $\mathcal{D}$ might contain images of handwritten digits and a generative model will find a $p(x)$ such that a sample drawn from $p(x)$ looks visually similar to one of the handwritten digit.

In this notebook we will focus on a generative model for images. In such a case, each element $x \in \mathcal{D}$ will be an image, thus $x$ can be viewed as a random vector. If we have a $28\times28$ single channel image then $x$ is a $784$ dimensional vector. Due to the high dimensionality of data modelling $p(x)$ directly is very challenging. Different components of $x$ interact with each other in complicated ways. For example adjacent pixels tend to have similar intensity values.

Due to these interactions one can not make the simplifying independence assumption among the components of $x$. One approach is to use latent variables to model complicated dependencies among input data while still keeping the model tractible.

**Figure 1: Latent Variable Model**

For example, in the graphical model shown in figure 1, $x \in \mathbb{R}^d$ are observed variables and $z \in \mathbb{R}^m$ are latent variables. Different components of $x$ are not independent. However given the values of all $z_i$'s the components of $x$ are conditionally independent of each other. Since $z_i$'s are not observed this allows the model to represent complicated dependencies among the observed data variables. The distribution $p(x_j|z)$ must be learned from the data for $j = 1, 2, ..., d$. This is easier to learn in most cases as compared to learning $p(x)$ directly. The assumption of conditional independence is more practical as compared to the assumption of mutual independence among components of $x$.

## Problem Setup
We are given a dataset $\mathcal{D} = \{x^{(1)}, x^{(2)}, ..., x^{(N)}\}$, such that each $x \in \mathcal{D}$ is an image. All the images in the dataset are assumed to have the same dimensions. We assume that the images in $\mathcal{D}$ were generated using the following generative process:

**Figure 2: Image Generation Process**

To generate an image $x$, first the corresponding latent variable $z$ is sampled from $p_\theta(z)$. Then conditioned on $z$, the image $x$ is sampled from $p_\theta(x|z)$. $\theta$ represents the parameters of the distribution. The actual generation process will have an optimal value of $\theta$. We will refer to this optimal value as $\theta^*$. In order to generate new samples we need to find $\theta^*$.

There are no labels attached to the images hence we operate in an unsupervised fashion.

## Optimization Problem
The objective is to find a setting of parameters which maximizes the log probability of observed data. Assuming that the points in $\mathcal{D}$ are iid, we wish to optimize the following function:
\begin{align}
    \sum_{i=1}^{N} \log p(x^{(i)}; \theta) &= \sum_{i=1}^{N} \log \frac{p(x^{(i)}, z; \theta)}{p(z|x^{(i)}; \theta)} \\
    &= \sum_{i=1}^{N} \log \frac{p(x^{(i)}|z; \theta) p(z; \theta)}{p(z|x^{(i)}; \theta)} \\
\end{align}

This equation holds for any value of latent variable $z$. The posterior distribution $p(z|x; \theta)$ is intractible and so by following the variational inference recepie outlined in [Basic Concepts](./Basic%20Concepts.ipynb) notebook we approximate it with a parameterised probability distribution $q(z | x ; \phi)$. The following equations hold:
\begin{align}
    \sum_{i=1}^{N} \log p(x^{(i)}; \theta) &= \sum_{i=1}^{N} \log \frac{p(x^{(i)}|z; \theta) p(z; \theta) q(z | x^{(i)} ; \phi)}{p(z|x^{(i)}; \theta) q(z | x^{(i)} ; \phi)} \\
    &= \sum_{i=1}^{N} \sum_{z} q(z | x^{(i)} ; \phi) \Big( \log \frac{p(x^{(i)}|z; \theta) p(z; \theta) q(z | x^{(i)} ; \phi)}{p(z|x^{(i)}; \theta) q(z | x^{(i)} ; \phi)} \Big) \\
    &= \sum_{i=1}^{N} \Big( \sum_{z} q(z | x^{(i)} ; \phi) \log \frac{p(x^{(i)}|z; \theta) p(z; \theta)}{q(z | x^{(i)} ; \phi)} + \sum_{z} q(z | x^{(i)} ; \phi) \log \frac{p(z|x^{(i)}; \theta)}{q(z | x^{(i)} ; \phi)} \Big) \\
    &= \sum_{i=1}^{N} \Big(E_{z \sim q(.;x^{(i)}, \phi)} [\log p(x^{(i)}|z; \theta) + \log p(z; \theta) - \log q(z | x^{(i)} ; \phi)] + KL(q, p | x^{(i)}, \theta, \phi) \Big) \\
\end{align}

Since KL divergence is always $\geq 0$, we get:
\begin{align}
    \sum_{i=1}^{N} \log p(x^{(i)}; \theta) &\geq \sum_{i=1}^{N} E_{z \sim q(.;x^{(i)}, \phi)} [\log p(x^{(i)}|z; \theta) + \log p(z; \theta) - \log q(z | x^{(i)} ; \phi)] = \mathcal{L}(\theta, \phi)\\
\end{align}

We will maximize $\mathcal{L}(\theta, \phi)$. The following equation can be obtained easily by rearranging the terms in the previous equation:

\begin{equation}
    \mathcal{L}(\theta, \phi) = \sum_{i=1}^{N} \Big(E_{z \sim q(z|x^{(i)}; \phi)} [\log p(x^{(i)}|z; \theta)] - KL[q(z | x^{(i)} ; \phi) || p(z; \theta)] \Big)
\end{equation}

The second term of this equation forces the distribution $q(z|x)$ to be close to the prior $p(z)$. The first term forces $\theta$ and $\phi$ to be adjusted in such a way that the probability of observed data given the sampled value of $z$ is high.

In order to solve this optimization problem we need to evaluate $\nabla_\phi \mathcal{L}(\theta, \phi)$. One can usually obtain an analytical expression for the KL divergence term and hence differentiating that part of $\mathcal{L}(\theta, \phi)$ is easy. The gradient of first term with respect to $\phi$ can be found using the Score Function (SF) estimator:
\begin{equation}
    \nabla_\phi E_{z \sim q(z|x^{(i)}; \phi)} [\log p(x^{(i)}|z; \theta)] = E_{z \sim q(z|x^{(i)}; \phi)} [\log p(x^{(i)}|z; \theta) \nabla_\phi \log q(z|x^{(i)}; \phi)]
\end{equation}

This estimator has a high variance and hence is not practical. One approach is to use the reparameterisation trick ([Kingma et al. 2013](https://arxiv.org/pdf/1312.6114)). The basic idea here is to represent $z$ as a deterministic function of $\phi$ and another random variable $\epsilon$, i.e. $z=g(\phi, \epsilon)$ where $\epsilon \sim p(\epsilon)$. Now the first term in the likelihood function can be written as: $E_{\epsilon \sim p(\epsilon)}[\log p(x^{(i)}|g(\phi, \epsilon); \theta)]$. Given this expression, one can employ the Pathwise Derivative (PD) estimator to calculate the gradient of first term with respect to $\phi$ as:
\begin{equation}
\nabla_\phi E_{\epsilon \sim p(\epsilon)}[\log p(x^{(i)}|g(\phi, \epsilon))] = E_{\epsilon \sim p(\epsilon)}[\nabla_\phi \log p(x^{(i)}|g(\phi, \epsilon))]
\end{equation}

This expectation can be evaluated using monte-carlo methods. In the extreme case a single sample can be used to approximate this expectation.

## Variational Autoencoder
An interesting case of the optimization problem posed above is when $\theta$ and $\phi$ are both neural networks. The _encoder_ network defined by $\phi$ determines the probability distribution $q(z|x;\phi)$ and the _decoder_ network defined by $\theta$ determines the probability distribution $p(x|z; \theta)$. We model $q(z|x^{(i)}; \phi)$ as follows:

\begin{equation}
    q(z|x^{(i)}; \phi) = \mathcal{N}(z| x^{(i)}, \mu^{(i)}, \Sigma^{(i)})
\end{equation}

Here $\mu^{(i)} \in \mathbb{R}^m$ is the output of the neural network which takes $x^{(i)}$ as input and $\Sigma^{(i)}$ is a diagonal matrix where the $j^{th}$ diagonal entry is equal to $e^{\sigma^{(i)}_j}$. $\sigma^{(i)} \in \mathbb{R}^m$ is the output of the neural network which again takes $x^{(i)}$ as input. A non-informative prior is imposed on $z$:

\begin{equation}
    p(z; \theta) = \mathcal{N}(z|0, I)
\end{equation}

The distribution $p(x^{(i)}|z; \theta)$ can be modelled as a multivariate Bernoulli distribution. In such a case we obtain the following expression:

\begin{equation}
    \log p(x^{(i)}|z; \theta) = \sum_{j=1}^{d} \Big(x^{(i)}_j \log y^{(i)}_j  + (1 - x^{(i)}_j) \log(1 - y^{(i)}_j) \Big)
\end{equation}

Here $y^{(i)}$ is the output of a neural network that takes in the $z$ corresponding to $x^{(i)}$ as input. Although $x$ is not strictly a Bernoulli random variable, we scale it to lie in the range $[0, 1]$ and then treat it as a Bernoulli random variable even though it can take values between $0$ and $1$ also. Note that there are other modelling choices for $p(x^{(i)}|z; \theta)$. One can also model this distribution as a normal distribution as was done in the case of $q(z|x^{(i)}; \phi)$. In this case we will have two neural networks, both of which take $z$ as input and produce mean and log-variance respectively as their output.

The final thing that we need to calculate is the KL divergence term $KL[q(z | x^{(i)} ; \phi) || p(z; \theta)]$. Since both $p$ and $q$ have been modelled as normal distributions, this KL divergence can be calculated analytically as:

\begin{align}
    KL[q(z | x^{(i)} ; \phi) || p(z; \theta)] = -\frac{1}{2} \sum_{j=1}^{m} \Big( 1 + \sigma^{(i)}_j - (\mu_j^{(i)})^2 - (e^{\sigma_j^{(i)}})^2 \Big)
\end{align}

Putting it all together the likelihood function that has to be maximized looks like this:

\begin{align}
    \mathcal{L}(\theta, \phi) &= \sum_{i=1}^{N} \Big(E_{z \sim q(z|x^{(i)}; \phi)} [\log p(x^{(i)}|z; \theta)] - KL[q(z | x^{(i)} ; \phi) || p(z; \theta)] \Big) \\
    &= \sum_{i=1}^{N} \sum_{j=1}^{d} \Big(x^{(i)}_j \log y^{(i)}_j  + (1 - x^{(i)}_j) \log(1 - y^{(i)}_j) \Big) + \frac{1}{2} \sum_{i=1}^{N} \sum_{j=1}^{m} \Big( 1 + \sigma^{(i)}_j - (\mu_j^{(i)})^2 - (e^{\sigma_j^{(i)}})^2 \Big)
\end{align}

## Experiments using MNIST
The file ``vae.py`` contains implementation of a general variational autoencoder. The architecture of the network is required as input and the implementation maxmimizes the likelihood function given above for the specified architecture. In the following example we use ``vae.py`` to train a variational autoencoder for MNIST dataset. The encoder networks for $\mu^{(i)}$ and $\sigma^{(i)}$ are both three layered fully connected MLPs where the first two layers are shared between both the networks. For the decoder part we again have a three layerd fully connected MLP.

```python
from vae import *
from tensorflow.examples.tutorials.mnist import input_data
import numpy as np
import matplotlib.pyplot as plt

# Define the identity function
def identity(x):
    return x

# Set up the network architecture
arch = dict()
arch['encoder'] = [\
    ('fc', identity, 784, -1), \
    ('fc', tf.nn.softplus, 500, 784), \
    ('fc', tf.nn.softplus, 500, 500), \
    ('fc', identity, 20, 500), \
    ('fc', identity, 20, 500) \
    ]
arch['decoder'] = [\
    ('fc', tf.nn.softplus, 500, 20), \
    ('fc', tf.nn.softplus, 500, 500), \
    ('fc', tf.nn.sigmoid, 784, 500) \
    ]
arch['input_shape'] = [784]
arch['first_roll_shape'] = (-1, -1, -1)

# Initialize the variational autoencoder
batch_size = 100
learning_rate = 1e-3
vae = VAE(arch, learning_rate, batch_size)

# Set up the data
mnist = input_data.read_data_sets('data/mnist/')

# Run the autoencoder
n_samples = mnist.train.num_examples
total_batch = int(n_samples/batch_size)
n_epochs = 100
display_step = 1

costs = []
for epoch in range(n_epochs):
    avg_cost = 0.0
    for batch in range(total_batch):
        x, _ = mnist.train.next_batch(batch_size)
        cost = vae.train(x)
        avg_cost += (cost - avg_cost) / (batch + 1)
    costs.append(avg_cost)

    if epoch % display_step == 0:
        print "\rEpoch:", '%04d' % (epoch+1), \
                "cost=", "{:.9f}".format(avg_cost),

# Generate sample images
z = np.random.normal(size=(batch_size, 20))
generated_images = vae.generate_image(z)

# Show generated images
big_image = np.zeros((280, 280))
for i in range(10):
    for j in range(10):
        n = i*10 + j
        big_image[28*i:28*(i+1), 28*j:28*(j+1)] = np.reshape(generated_images[n, :], (28, 28))

plt.imshow(big_image)
plt.show()

# Show cost vs epoch plot
plt.plot(costs)
plt.show()
```

The first figure above shows images that have been generated by the decoder network from random noise after training the variational autoencoder. It can be seen that the network generates almost real looking digits which in most cases can not be distinguished from handwritten digits using naked eye. The second plot shows the the negative of likelihood function with respect to the number of training epochs.

Convolutional neural networks usually perform better on vision related tasks as compared to standard fully connected MLPs by virtue of their architecture. In the next example we use a convolutional neural network both as an encoder and as a decoder (transposed convolutions are used in decoder). The dataset being used is still MNIST.

```python
from vae import *
from tensorflow.examples.tutorials.mnist import input_data
import numpy as np
import matplotlib.pyplot as plt

# Hyperparameters
batch_size = 100
learning_rate = 1e-4

# Define the identity function
def identity(x):
    return x

# Set up the network architecture
arch = dict()
arch['encoder'] = [\
    ('fc', identity, 784, -1), \
    ('conv', tf.nn.relu, 5, 1, 8), \
    ('conv', tf.nn.relu, 3, 8, 16), \
    ('pool', identity, 4), \
    ('fc', tf.nn.relu, 500, 784), \
    ('fc', identity, 20, 500), \
    ('fc', identity, 20, 500) \
    ]
arch['decoder'] = [\
    ('fc', tf.nn.relu, 500, 20), \
    ('fc', tf.nn.relu, 784, 500), \
    ('tconv', tf.nn.relu, 3, 16, 8, (batch_size, 14, 14, 8), (1, 2, 2, 1)), \
    ('tconv', tf.nn.sigmoid, 3, 8, 1, (batch_size, 28, 28, 1), (1, 2, 2, 1))
    ]
arch['input_shape'] = [28, 28, 1]
arch['first_roll_shape'] = (7, 7, 16)

# Initialize the variational autoencoder
vae = VAE(arch, learning_rate, batch_size)

# Set up the data
mnist = input_data.read_data_sets('data/mnist/')

# Run the autoencoder
n_samples = mnist.train.num_examples
total_batch = int(n_samples/batch_size)
n_epochs = 150
display_step = 1

costs = []
for epoch in range(n_epochs):
    avg_cost = 0.0
    for batch in range(total_batch):
        x, _ = mnist.train.next_batch(batch_size)
        x = np.reshape(x, (batch_size, 28, 28, 1))
        cost = vae.train(x)
        avg_cost += (cost - avg_cost) / (batch + 1)
    costs.append(avg_cost)

    if epoch % display_step == 0:
        print "\rEpoch:", '%04d' % (epoch+1), \
                "cost=", "{:.9f}".format(avg_cost),

# Generate sample images
z = np.random.normal(size=(batch_size, 20))
generated_images = vae.generate_image(z)

# Show generated images
big_image = np.zeros((280, 280))
for i in range(10):
    for j in range(10):
        n = i*10 + j
        big_image[28*i:28*(i+1), 28*j:28*(j+1)] = np.reshape(generated_images[n, :], (28, 28))

plt.imshow(big_image)
plt.show()

# Show cost vs epoch plot
plt.plot(costs)
plt.show()
```

## Experiment using CIFAR-10
We will next move on to another dataset called CIFAR-10. We will try to generate images which are like the images present in CIFAR-10 by training our variational autoencoder using this dataset. We will use a convolutional architecture for both encoder and decoder networks. ``read_cifar10.py`` is a helper file that reads the CIFAR-10 dataset. ``read_cifar10.py`` requires the location of CIFAR-10 data directory as input. The function ``get_data()`` returns a batch of images along with corresponding labels.

```python
from vae import *
from read_cifar10 import *
import numpy as np
import matplotlib.pyplot as plt

# Hyperparameters
batch_size = 100
learning_rate = 1e-4

# Define the identity function
def identity(x):
    return x

# Set up the network architecture
arch = dict()
arch['encoder'] = [\
    ('fc', identity, 3072, -1), \
    ('conv', tf.nn.relu, 5, 3, 8), \
    ('conv', tf.nn.relu, 4, 8, 16), \
    ('pool', identity, 4), \
    ('fc', tf.nn.relu, 500, 1024), \
    ('fc', identity, 80, 500), \
    ('fc', identity, 80, 500) \
    ]
arch['decoder'] = [\
    ('fc', tf.nn.relu, 500, 80), \
    ('fc', tf.nn.relu, 1024, 500), \
    ('tconv', tf.nn.relu, 3, 16, 8, (batch_size, 16, 16, 8), (1, 2, 2, 1)), \
    ('tconv', tf.nn.sigmoid, 3, 8, 3, (batch_size, 32, 32, 3), (1, 2, 2, 1))
    ]
arch['input_shape'] = [32, 32, 3]
arch['first_roll_shape'] = (8, 8, 16)

# Initialize the variational autoencoder
vae = VAE(arch, learning_rate, batch_size)

# Set up the data
print 'Reading data...'
cifar = CIFAR10('data/cifar-10/')
print 'Data read.'

# Run the autoencoder
n_samples = cifar.get_num_training_examples()
total_batch = int(n_samples/batch_size)
n_epochs = 150
display_step = 1

costs = []
print 'Starting Training...'
for epoch in range(n_epochs):
    avg_cost = 0.0
    for batch in range(total_batch):
        x, _ = cifar.get_data(batch_size)
        x = np.ndarray.astype(x, np.float32) / 255.0
        cost = vae.train(x)
        avg_cost += (cost - avg_cost) / (batch + 1)
    costs.append(avg_cost)

    if epoch % display_step == 0:
        print "\rEpoch:", '%04d' % (epoch+1), \
                "cost=", "{:.9f}".format(avg_cost),

# Generate sample images
z = np.random.normal(size=(batch_size, 80))
generated_images = vae.generate_image(z)

# Show generated images
big_image = np.zeros((320, 320, 3))
for i in range(10):
    for j in range(10):
        n = i*10 + j
        big_image[32*i:32*(i+1), 32*j:32*(j+1), :] = np.reshape(generated_images[n, :], (32, 32, 3))

plt.imshow(big_image)
plt.show()

# Show cost vs epoch plot
plt.plot(costs)
plt.show()
```

It can be seen that the images that are generated by the variational autoencoder in this case are blurry. Although fine tuning the architecture usually leads to better results, the vanilla variational autoencoder usually produces blurry images. There are many modifications done to the basic variational autoencoder. For example, the DRAW network [Gregor et al. 2015](https://arxiv.org/pdf/1502.04623) employs attention mechanism on top of a variational autoencoder. Such models are beyond the scope of this project.


```python
#import packages
import numpy as np
from numpy import loadtxt
import pylab as pl
from IPython import display
from RcTorch import *
from matplotlib import pyplot as plt
from scipy.integrate import odeint
import time

#this method will ensure that the notebook can use multiprocessing (train multiple
#RC's in parallel) on jupyterhub or any other linux based system.
try:
    mp.set_start_method("spawn")
except:
    pass
torch.set_default_tensor_type(torch.FloatTensor)
%matplotlib inline
start_time = time.time()
```

```python
lineW = 3
lineBoxW=2

font = {'family' : 'normal',
        'weight' : 'normal',#'bold',
        'size'   : 24}

plt.rc('font', **font)
plt.rcParams['text.usetex'] = True
```

## Rc and systems of ODEs: an overview

In this notebook we demonstrate that the RC can solve systems of differential equations. Any higher order ODE can be decomposed into a system of first order ODEs, hence solving systems of ODEs means that RC can solver higher order ODEs. This is a standard procedure followed by standard integrators . To apply the RC to systems, the RC architecture needs to be modified to return multiple outputs $N_j$, where $j$ indicates a different output. Specifically, the number of the outputs needs to be the same as the number of the equations in a system. Each output has a different set of weights $W_{out}^{(j)}$ while all the $N_j$ share the same hidden states.

We exploit the RC solver in solving the equations of motion for a nonlinear Hamiltonian system, the nonlinear oscillator.  The energy is conserved in this system and thus, we adopt hamiltonian energy regularization that drastically accelerates the training and improves the fidelity of the predicted solutions.

### Hamiltonian systems
Hamiltonian systems  obey the energy conservation law. In other words, these systems are characterized by a hamiltonian function that represents the total energy of the system which remains constant in time. The hamiltonian of a nonlinear oscillator with unity mass and frequency is given by:
\begin{align}
    \label{eq:NL_ham}
    \mathcal{H} = \frac{p^2}{2} + \frac{x^2}{2} + \frac{x^4}{4},
\end{align}
and the associated equations of motion reads:
\begin{align}
    \label{eq:NL_x}
    \dot x &= p \\
    \label{eq:NL_p}
    \dot p &= -x - x^3
\end{align}
where $p$ is the momentum and $x$ represents the position of the system. The loss function consists of three parts: $L_\text{ODE}$ for the ODEs of x and p; a hamiltonian penalty $L_{\mathcal{H}}$ that penalizes violations in the energy conservation and is defined by Eq

Subsequently, the total $L$  reads:
\begin{align}
    L &= L_\text{ODE}+ L_{\mathcal{H}} + L_\text{reg} \nonumber \\
     \label{eq:NL_loss}
     &= \sum_{n=0}^{K}
      \Big[ \left(\dot x_n-p_n\right)^2 + \left(\dot p_n + x_n + x_n^3  \right)^2 +\left(E - \mathcal{H}(x_n, p_n)\right)^2 \Big]   + \lambda \sum_{j=x,p} W_{out}^{(j)T} W_{out}^{(j)}.
\end{align}

```python
def plot_predictions(RC, results, integrator_model, ax = None):
    """plots a RC prediction and integrator model prediction for comparison
    Parameters
    ----------
    RC: RcTorchPrivate.esn
        the RcTorch echostate network to evaluate. This model should already have been fit.
    results: dictionary
        the dictionary of results returned by the RC after fitting
    integrator model: function
        the model to be passed to odeint which is a gold standard integrator numerical method
        for solving ODE's written in Fortran. You may find the documentation here:
        https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html
    ax: matplotlib.axes._subplots.AxesSubplot
        If provided, the function will plot on this subplot axes
    """
    X = RC.X.cpu()
    if not ax:
        fig, ax = plt.subplots(1,1, figsize = (6,6))
    for i, y in enumerate(results["ys"]):
        y = y.cpu()
        if not i:
            labels = ["RC", "Integrator Solution"]
        else:
            labels = [None, None]
        ax.plot(X, y, color = "dodgerblue", label = labels[0], linewidth = lineW + 1, alpha = 0.9)

        #calculate the integrator prediction:
        int_sol = odeint(integrator_model, y0s[i], np.array(X.cpu().squeeze()))
        int_sol = torch.tensor(int_sol)

        #plot the integrator prediction
        ax.plot(X, int_sol, '--', color = "red", alpha = 0.9, label = labels[1],  linewidth = lineW)

    plt.ylabel(r'$y(t)$');
    ax.legend();
    ax.tick_params(labelbottom=False)
    plt.tight_layout()

def covert_ode_coefs(t, ode_coefs):
    """ converts coefficients from the string 't**n' or 't^n' where n is any float
    Parameters
    ----------
    t: torch.tensor
        input time tensor
    ode_coefs: list
        list of associated floats. List items can either be (int/floats) or ('t**n'/'t^n')
    """
    type_t = type(t)
    for i, coef in enumerate(ode_coefs):
        if type(coef) == str:
            if coef[0] == "t" and (coef[1] == "*" or (coef[1] == "*" and coef[2] == "*")):
                pow_ = float(re.sub("[^0-9.-]+", "", coef))
                ode_coefs[i]  = t ** pow_
                print("alterning ode_coefs")
        elif type(coef) in [float, int, type_t]:
            pass
        else:
            assert False, "ode_coefs must be a list floats or strings of the form 't^pow', where pow is a real number."
    return ode_coefs

def plot_rmsr(RC, results, force, ax = None):
    """plots the residuals of a RC prediction directly from the loss function
    Parameters
    ----------
    RC: RcTorchPrivate.esn
        the RcTorch echostate network to evaluate. This model should already have been fit.
    results: dictionary
        the dictionary of results returned by the RC after fitting
    force: function
        the force function describing the force term in the population equation
    ax: matplotlib.axes._subplots.AxesSubplot
        If provided, the function will plot on this subplot axes
    """
    if not ax:
        fig, ax = plt.subplots(1,1, figsize = (10, 4))
    X = RC.X.cpu()
    ys, ydots = results["ys"], results["ydots"]

    residuals = []
    force_t = force(X)
    for i, y in enumerate(ys):
        ydot = ydots[i]
        y = y.cpu()
        ydot = ydot.cpu()

        ode_coefs = covert_ode_coefs(t = X, ode_coefs = RC.ode_coefs)

        resids = custom_loss(X, y, ydot, None,
                             force_t = force_t,
                             ode_coefs = RC.ode_coefs,
                             mean = False)
        if not i:
            resids_tensor = resids
            label = r'{Individual Trajectory RMSR}'
        else:
            resids_tensor = torch.cat((resids_tensor, resids), axis = 1)
            label = None
        resids_specific_rmsr = torch.sqrt(resids/1)

        ax.plot(X, resids_specific_rmsr, color = "orangered", alpha = 0.4, label = label, linewidth = lineW-1)
        residuals.append(resids)

    mean_resid = torch.mean(resids_tensor, axis =1)
    rmsr = torch.sqrt(mean_resid)
    ax.plot(X, rmsr,
               color = "blue",
               alpha = 0.9,
               label = r'{RMSR}',
               linewidth = lineW-0.5)

    ax.legend(prop={"size":16});

    ax.set_xlabel(r'$t$')
    ax.set_yscale("log")
    ax.set_ylabel(r'{RMSR}')
```

```python
def f(u, t ,lam=0,A=0,W=1):
            x,  px = u      # unpack current values of u
            derivs = [px, -x - lam*x**3 +A*np.sin(W*t)]     # you write the derivative here
            return derivs

# Scipy Solver
def NLosc_solution(t, x0,  px0, lam=0, A=0,W=1):
    u0 = [x0, px0]
    # Call the ODE solver
    solPend = odeint(f, u0, t.cpu(), args=(lam,A,W,))
    xP = solPend[:,0];        pxP = solPend[:,1];
    return xP, pxP

def plot_results(RC, results, integrator_model, y0s, ax = None):
    """plots a RC prediction and integrator model prediction for comparison
    Parameters
    ----------
    RC: RcTorchPrivate.esn
        the RcTorch echostate network to evaluate. This model should already have been fit.
    results: dictionary
        the dictionary of results returned by the RC after fitting
    integrator model: function
        the model to be passed to odeint which is a gold standard integrator numerical method
        for solving ODE's written in Fortran. You may find the documentation here:
        https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html
    ax: matplotlib.axes._subplots.AxesSubplot
        If provided, the function will plot on this subplot axes
    """
    X = RC.X.cpu().detach()
    #int_sols = []
    if not ax:
        fig, ax = plt.subplots(1,1, figsize = (6,6))

    for i, y in enumerate(results["ys"]):
        y = y.cpu().detach()
        if not i:
            labels = ["RC","integrator"]
        else:
            labels = [None, None, None, None]
        try:
            labels
        except:
            pass

        ax.plot(y[:,0], y[:,1], label = labels[0],
                linewidth =7, alpha = 0.9, color = "dodgerblue")

        #calculate the integrator prediction:
        y_truth, p_truth  = NLosc_solution(RC.X.squeeze().data,y0s[i],1,lam=1, A=0, W= 0)

        #p = y[:,1].cpu()# + v0
        #yy = y[:,0].cpu()# + y0

        #plot the integrator prediction
        ax.plot(y_truth, p_truth, color = "red", linewidth =3,
                alpha = 1.0, label = labels[1])
#         ax.plot(X, p, color = "red", alpha = 1.0, linewidth =3,
#                 label = labels[3])

    ax.set_xlabel("x")
    ax.set_ylabel("p")
    ax.set_ylim(-3,3)
    ax.set_xlim(-3,3)
    ax.legend();
    #return int_sols

def force(X, A = 0):
    return torch.zeros_like(X)

def plot_rmsr(RC, results, force, log = False, ax = None):
    """plots the residuals of a RC prediction directly from the loss function
    Parameters
    ----------
    RC: RcTorchPrivate.esn
        the RcTorch echostate network to evaluate. This model should already have been fit.
    results: dictionary
        the dictionary of results returned by the RC after fitting
    force: function
        the force function describing the force term in the population equation
    ax: matplotlib.axes._subplots.AxesSubplot
        If provided, the function will plot on this subplot axes
    """
    if not ax:
        fig, ax = plt.subplots(1,1, figsize = (10, 4))
    X = RC.X.cpu().detach()
    ys, ydots = results["ys"], results["ydots"]

    residuals = []
    for i, y in enumerate(ys):
        y = y.cpu().detach()
        ydot = ydots[i].cpu().detach()
        resids = custom_loss(X, y, ydot, None,
                             force = force,
                             ode_coefs = RC.ode_coefs,
                             mean = False,
                            init_conds = RC.init_conds,
                            ham = False)
        if not i:
            resids_tensor = resids
            label = r'{Individual Trajectory RMSR}'
        else:
            resids_tensor = torch.cat((resids_tensor, resids), axis = 1)
            label = None
        resids_specific_rmsr = torch.sqrt(resids/1)

        ax.plot(X, resids_specific_rmsr, color = "orangered", alpha = 0.4, label = label, linewidth = lineW-1)
        residuals.append(resids)

    mean_resid = torch.mean(resids_tensor, axis =1)
    rmsr = torch.sqrt(mean_resid)
    ax.plot(X, rmsr,
               color = "blue",
               alpha = 0.9,
               label = r'{RMSR}',
               linewidth = lineW-0.5)

    ax.legend(prop={"size":16});

    ax.set_xlabel(r'$t$')
    ax.set_yscale("log")
    ax.set_ylabel(r'{RMSR}')

def driven_force(X, A = 1):
    return A * torch.sin(X)

def no_force(X, A = 0):
    return A

#define a reparameterization function, empirically we find that g= 1-e^(-t) works well)
def reparam(t, order = 1):

    exp_t = torch.exp(-t)
    derivatives_of_g = []

    g = 1 - exp_t
    g_dot = 1 - g
    return g, g_dot

    #first derivative

    #example code for higher derivatives:
    #####################################

    #derivatives_of_g.append(g_dot)
    #derivatives_of_g.append(g)
#     for i in range(order):
#         if i %2 == 0:
#             #print("even")
#             derivatives_of_g.append(g_dot)
#         else:
#             #print("odd")
#             derivatives_of_g.append(-g_dot)
#    return derivatives_of_g

def custom_loss(X, y, ydot, out_weights, force = force,
                reg = False, ode_coefs = None, mean = True,
               enet_strength = None, enet_alpha = None, init_conds = None, lam = 1, ham = True):
    y, p = y[:,0].view(-1,1), y[:,1].view(-1,1)
    ydot, pdot = ydot[:,0].view(-1,1), ydot[:,1].view(-1,1)

    #with paramization
    L =  (ydot - p)**2 + (pdot + y + lam * y**3   - force(X))**2

    if mean:
        L = torch.mean(L)

    if reg:
        #assert False
        weight_size_sq = torch.mean(torch.square(out_weights))
        weight_size_L1 = torch.mean(torch.abs(out_weights))
        L_reg = enet_strength*(enet_alpha * weight_size_sq + (1- enet_alpha) * weight_size_L1)
        L = L + 0.1 * L_reg
    if ham:
        y0, p0 = init_conds
        ham = hamiltonian(y, p)
        ham0 = hamiltonian(y0, p0)
        L_H = (( ham - ham0).pow(2)).mean()
        assert L_H >0

        L = L +  0.1 * L_H

    #print("L1", hi, "L_elastic", L_reg, "L_H", L_H)
    return L

```

```python
def optimize_last_layer(esn,
                        SAVE_AFTER_EPOCHS = 1,
                        epochs = 45000,
                        custom_loss = custom_loss,
                        loss_threshold = 10**-10,#10 ** -8,
                        f = force,
                        lr = 0.05,
                        reg = None,
                        plott = True,
                        plot_every_n_epochs = 2000):#gamma 0.1, spikethreshold 0.07 works

    with torch.enable_grad():
        #define new_x
        new_X = esn.extended_states.detach()
        spikethreshold = esn.spikethreshold

        #force detach states_dot
        esn.states_dot = esn.states_dot.detach().requires_grad_(False)

        #define criterion
        criterion = torch.nn.MSELoss()

        #assert esn.LinOut.weight.requires_grad and esn.LinOut.bias.requires_grad
        #assert not new_X.requires_grad

        #define previous_loss (could be used to do a convergence stop)
        previous_loss = 0

        #define best score so that we can save the best weights
        best_score = 0

        #define the optimizer
        optimizer = optim.Adam(esn.parameters(), lr = lr)

        #optimizer = torch.optim.SGD(model.parameters(), lr=100)
        if esn.gamma_cyclic:
            cyclic_scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, 10**-6, 0.01,
                                              gamma = esn.gamma_cyclic,#0.9999,
                                              mode = "exp_range", cycle_momentum = False)

        scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=esn.gamma)
        lrs = []

        #define the loss history
        loss_history = []

        if plott:
            #use pl for live plotting
            fig, ax = pl.subplots(1,3, figsize = (16,4))

        t = esn.X#.view(*N.shape).detach()
        g, g_dot = esn.G
        y0  = esn.init_conds[0]

        floss_last = 0

        try:
            assert esn.LinOut.weight.requires_grad and esn.LinOut.bias.requires_grad
        except:
            esn.LinOut.weight.requires_grad_(True)
            esn.LinOut.bias.requires_grad_(True)

        #begin optimization loop
        for e in range(epochs):

            optimizer.zero_grad()

            N = esn.forward( esn.extended_states )
            N_dot = esn.calc_Ndot(esn.states_dot)

            y = g *N

            ydot = g_dot * N + g * N_dot

            y[:,0] = y[:,0] + esn.init_conds[0]
            y[:,1] = y[:,1] + esn.init_conds[1]

            assert N.shape == N_dot.shape, f'{N.shape} != {N_dot.shape}'

            loss = custom_loss(esn.X, y, ydot, esn.LinOut.weight, reg = reg, ode_coefs = esn.ode_coefs,
                              init_conds = esn.init_conds, enet_alpha= esn.enet_alpha, enet_strength = esn.enet_strength)
            loss.backward()
            optimizer.step()
            if esn.gamma_cyclic and e > 0 and e <5000:
                cyclic_scheduler.step()
            lrs.append(optimizer.param_groups[0]["lr"])

            floss = float(loss)

            if e > 0:
                loss_delta = float(np.log(floss_last) - np.log(floss))
                if loss_delta > esn.spikethreshold:# or loss_delta < -3:
                    lrs.append(optimizer.param_groups[0]["lr"])
                    scheduler.step()

            if not e and not best_score:
                best_bias, best_weight, best_fit = esn.LinOut.bias.detach(), esn.LinOut.weight.detach(), y.clone()

            if e > SAVE_AFTER_EPOCHS:
                if not best_score:
                    best_score = min(loss_history)
                best_bias, best_weight = esn.LinOut.bias.detach(), esn.LinOut.weight.detach()
                best_score = float(loss)
                best_fit = y.clone()
                best_pred = y.clone()
                best_ydot = ydot.clone()

            loss_history.append(floss)
            floss_last = floss
            if plott and e:

                if e % plot_every_n_epochs == 0:
                    for param_group in optimizer.param_groups:
                        print('lr', param_group['lr'])
                    ax[0].clear()
                    logloss_str = 'Log(L) ' + '%.2E' % Decimal((loss).item())
                    delta_loss  = ' delta Log(L) ' + '%.2E' % Decimal((loss-previous_loss).item())

                    print(logloss_str + ", " + delta_loss)
                    ax[0].plot(y.detach().cpu(), label = "exact")
                    ax[0].set_title(f"Epoch {e}" + ", " + logloss_str)
                    ax[0].set_xlabel("t")

                    ax[1].set_title(delta_loss)
                    ax[1].plot(N_dot.detach().cpu())
                    #ax[0].plot(y_dot.detach(), label = "dy_dx")
                    ax[2].clear()
                    #weight_size = str(weight_size_sq.detach().item())
                    #ax[2].set_title("loss history \n and "+ weight_size)

                    ax[2].loglog(loss_history)
                    ax[2].set_xlabel("t")

                    [ax[i].legend() for i in range(3)]
                    previous_loss = loss.item()

                    #clear the plot outputt and then re-plot
                    display.clear_output(wait=True)
                    display.display(pl.gcf())

        return {"weights": best_weight,
                "bias" : best_bias,
                "loss" : {"loss_history" : loss_history},
                "ydot" : best_ydot,
                "y" : best_pred,
                "best_score" : best_score}

```

```python
def force(X, A = 0):
    return torch.zeros_like(X)
lam =1
def hamiltonian(x, p, lam = lam):
    return (1/2)*(x**2 + p**2) + lam*x**4/4
```

```python
BURN_IN = 1000
x0,xf, nsteps = 0, 5, 1000
xtrain = torch.linspace(x0, xf, steps = nsteps, requires_grad=False)

#the length of xtrain won't matter above. Only dt , x0, and xf matter for ODEs.
#the reason for this is that the input time vector is reconstructed internally in rctorch
#in order to satisfy the specified dt.
xtrain = torch.linspace(x0, xf, steps = nsteps, requires_grad=False).view(-1,1)
```

```python
nl_oscillator_hp_set = {'dt': 0.001,
                        'regularization': 48.97788193684461,
                        'n_nodes': 500,
                        'connectivity': 0.017714821964432213,
                        'spectral_radius': 2.3660330772399902,
                        'leaking_rate': 0.0024312976747751236,
                        'bias': 0.37677669525146484,
                        'enet_alpha': 0.2082211971282959,
                        'enet_strength': 0.118459548397668,
                        'spikethreshold': 0.43705281615257263,
                        'gamma': 0.09469877928495407,
                        'gamma_cyclic': 0.999860422666841}
```

```python
base = 10*np.pi#10*np.pi
x0, xf= 0, base
nsteps = int(abs(xf - x0)/(nl_oscillator_hp_set["dt"]))
xtrain = torch.linspace(x0, xf, nsteps, requires_grad=False).view(-1,1)
float(xtrain[1]- xtrain[0])
```

```python
%%time
y0s = np.arange(0.7, 1.8, 0.2)
v0 = 1

RC = EchoStateNetwork(**nl_oscillator_hp_set,
                       random_state = 209,
                       id_ = 10,
                       activation_f = torch.sin,
                       act_f_prime = torch.cos,
                       dtype = torch.float32, n_outputs = 2)

train_args = {"burn_in" : int(BURN_IN),
              "ODE_order" : 1,
              "force" : force,
              "reparam_f" : reparam,
              "init_conditions" : [y0s, float(v0)],
              "ode_coefs" :       [1, 1],
              "X" :   xtrain.view(-1, 1),
              "eq_system" : True,
              #"out_weights" : out_weights
              }
#fit
results = RC.fit(**train_args,
                 SOLVE = True,
                 train_score = True,
                 backprop_f = optimize_last_layer,
                 epochs = 10000,
                 ODE_criterion = custom_loss)
```

```python
fig, ax = plt.subplots(1,2,figsize = (14,4))
plot_results(RC, results, NLosc_solution, y0s, ax = ax[0])
plot_data = plot_rmsr(RC,
                      results,
                      force = no_force,
                      log = True,
                      ax = ax[1])
```

```python
end_time = time.time()
print(f'Total notebook runtime: {end_time - start_time:.2f} seconds')
```

```python

```


## Sampling of Sinusoid

* continuos sinusoid $x(t)$를
* $T_s$로 sampling 할 경우
* **discrete sinusoid** $x[n]$을 얻음.

$$
\begin{align}
x[n] &= x(nT_s)\\
&= A \cos{(\omega n T_s)} \\
&= A \cos{(\omega T_s n )} \\
&= A \cos{ \Omega_0 n } \\
\end{align}
$$

where,
* $\Omega_0$ : 단위는 rad이며, 해당 discrete sinusoid $x[n]$의 angular freq.
* $A$ : Amplitude

Note that
* units of $\Omega_0$ is just **rad**, whereas units of $\omega_0$ is **rad/sec**.
* $\Omega_0$ don't have any temporal information,(units: **rad**).
* So, $x[n]$ is just a series(or list) of numbers.

```python
import numpy as np
import math
import matplotlib.pyplot as plt
import sys
from IPython.display import display, Markdown

t = np.arange(-1.*math.pi/4.,math.pi/2+0.01,0.001)
n = np.arange(-5,11)
A = 1

epsilon = sys.float_info.epsilon
def sinu(A,freq,n,Ts=None):
    if Ts is None:
        return A*np.cos(freq*t)
    return A*np.cos(Ts*freq*n)

freq = np.pi
Ts = 0.2
dsinu00 = sinu(A,freq,n,Ts)
sinu00 =  sinu(A,freq,t)

freq = 2.*np.pi
Ts = 0.1
dsinu01 = sinu(A,freq,n,Ts)
sinu01 =  sinu(A,freq,t)

freq = 4.*np.pi
Ts = 0.05
dsinu02 = sinu(A,freq,n,Ts)
sinu02 =  sinu(A,freq,t)

#plfigure()
plt.plot(t,sinu00,t,sinu01,t,sinu02)
plt.title='continuous'
plt.legend(('f=pi,ts=0.2','f=2pi,ts=0.1','f=4pi,ts=0.05'),loc='upper right')
plt.show()

#if np.all(dsinu00 == dsinu01) and np.all(dsinu01 == dsinu02) :
if np.all(np.isclose(dsinu00,dsinu01)) and np.all(np.isclose(dsinu00,dsinu02)) :
    display(Markdown('However, all discrete sinosoids are the same.'))
    display(Markdown('Becuase $\Omega_0$s are all the same!'))
```

* 즉, sampling에 의해 $\Omega_0$의 discrete sinusoid를 만들 수 있는 continuous sinusoid는 무수히 많음. ($\Omega_0 = T_s\omega_0$이기 때문.)
* 이는 discrete sinusoid에서 sampling 전의 continuous sinusoid를 복원할 때 문제가 됨.

### Even if the sampling interval, $T_s$ is known, **alias** problem remains.

```python
Ts = 0.8

freq = np.pi
dsinu00 = sinu(A,freq,n,Ts)

freq = 1.5*np.pi
dsinu01 = sinu(A,freq,n,Ts)

freq = 6.*np.pi
dsinu02 = sinu(A,freq,n,Ts)

#if np.all(abs(dsinu00-dsinu01) < 1e-10 ) and np.all(abs(dsinu00-dsinu02) < 1e-10):
if np.all(np.isclose(dsinu00,dsinu01)) and np.all(np.isclose(dsinu00,dsinu02)) :
    display(Markdown('Even if $T_s$ is known, $\Omega_0$ can be the same!'))
```

Even if $T_s$ is known, $\Omega_0$ can be the same!

When you use a sampling freq. $f_s$, following two sinusoids become the same discrete sinusoid!

* $\cos{(2\pi f_0)}$
* $\cos{(2\pi f_0 +k f_s)}$ where $k$ is any integer.

**Proof.**
$$
\begin{align}
\cos{(2\pi (f_0+k f_s) n T_s)} &= \cos{(2\pi f_0 n T_s + 2\pi k f_s n T_s)} \\
&= \cos{(2\pi f_0 n T_s + 2\pi kn f_s T_s)} \\
&= \cos{(2\pi f_0 n T_s + 2\pi kn )} \\
&= \cos{(2\pi f_0 n T_s + 2\pi)} \\
&= \cos{(2\pi f_0 n T_s)} \\
\end{align}
$$
 where,
 * All Sinusoids are $2\pi$ periodic. ($\cos{(\theta)} = \cos{(\theta +2\pi k)}$
 * $f_s = \frac{1}{T_s}$

### Freq. $f_0+k f_s$ is the **alias** of freq. $f_0$ with the sampling freq. $f_s$.

---

## The Properties of Discrete Sinusoid.

* When discrete sinusoid is periodic, the freq. $f_0$ or $\frac{\Omega_0}{2\pi}$ should be **rational number (유리수)**!
   * Discrete sinusoid is not always periodic!
   * $\frac{\Omega_0}{2\pi} = f_0 = \frac{k}{N}$ where $k$ and $n$ are integers.
* All (periodic)discrete sinusoid are $2\pi$ periodic
   * $e^{j2\pi k} = 1$
   * $e^{j (\Omega_0 +2\pi k) n} = e^{j\Omega_0n}e^{j2\pi kn} = e^{j\Omega_0n}$
   > 즉, $2\pi$ 정수배 떨어진 freq.를 갖는 discrete sinusoid는 구분할 수 없음.

```python

```


# GolemFlavor Tutorial

In this example, we will generate a fake measured flavor composition using a multivariate Gaussian distribution and sample from it using the [emcee](https://emcee.readthedocs.io/) MCMC algorithm.

```python
from __future__ import absolute_import, division, print_function

from functools import partial

import numpy as np
import matplotlib.pyplot as plt
```

## Astrophysical Neutrino Flavor Mixing

### Source Flavor Composition

The origins and accelerations mechanisms of astrophysically produced neutrinos is still an active puzzle, and is part of a bigger question on the origins of ultra-high-energy cosmic rays. The new but very active field of neutrino flavor physics can be used as a powerful probe to help idenfify these sources. The most common hypothesis of the neutrino flavor composition at the source is one produced by the decay of a pion, which results in the following source composition:

$$\pi\:\text{decay}\rightarrow\left(f_e:f_\mu:f_\tau\right)_\text{S}=\left(1:2:0\right)_\text{S}$$

where $f_\alpha$ is the flavor composition of a neutrino with flavor $\alpha\in\{e,\mu,\tau\}$ and the subscript S represents that this is the flavor composition at the source. In the code below we normalize this to 1 for later calculations.

```python
from golemflavor.fr import normalize_fr

source_composition = normalize_fr((1, 0, 0))
print('Source composition = ({:.2f} : {:.2f} : {:.2f})'.format(*source_composition))
```

    Source composition = (1.00 : 0.00 : 0.00)

### Neutrino Mixing

For the three massive neutrinos, the flavor eigenstates of the neutrino $|\nu_\alpha>$, $\alpha\in\{e,\mu,\tau\}$, are related to the mass eigenstates $|\nu_i>$, $i\in\{1,2,3\}$ via a unitary mixing matrix $U_{\alpha i}$ known as the PMNS matrix:

$$ |\nu_\alpha>=\sum^3_{i=1}U^*_{\alpha i}|\nu_i> $$

The determination of the values of this mixing matrix is currently a world-wide effort. We can import values of this mixing from GolemFlavor which are taken from a [global fit to world neutrino data](<https://doi.org/10.1007/JHEP01(2017)087>):

```python
from golemflavor.fr import NUFIT_U

print('Mixing Matrix =\n{}'.format(NUFIT_U))
```

    Mixing Matrix =
    [[ 0.82327921+0.j          0.54796108+0.j         -0.09913534+0.11010079j]
     [-0.30340559+0.06889398j  0.59033699+0.0458547j   0.74336952+0.j        ]
     [ 0.47090947+0.06045075j -0.58950774+0.04023502j  0.65226662+0.j        ]]

This mixing matrix says that neutrinos can oscillation from one flavor state $\alpha\in\{e,\mu,\tau\}$ to another $\beta\in\{e,\mu,\tau\}$ as a function of the propagation distance. The oscillation probability gives the probability that a neutrino produced in a flavor state $\alpha$ is then detected in a flavor state $\beta$ after a propagation distance $L$:

$$
\begin{align}
    P_{\nu_\alpha\rightarrow\nu_\beta}\left(L\right) &= \mid<\nu_\beta\left(L\right)|\nu_\alpha\left(0\right)>\mid^2\\
    &=\mid\sum_{i=1}^3U_{\beta i}U_{\alpha i}^*e^{-i\frac{m_i^2L}{2E}}\mid^2
\end{align}
$$

where $\Delta m_{ij}^2=m_i^2-m_j^2$ is the mass-squared differences and $E$ is the neutrino energy.

### Measured Flavor Composition

Once an astrophysical neutrino escapes the source it was produced from, they are free to propagate in the vacuum. Astrophysical neutrinos have $\mathcal{O}(\text{Mpc})$ or higher baselines, large enough that the mass eigenstates completely decouple ($L\rightarrow\infty$). This is useful for us, because the above oscillation probability simplifies so that:

$$
\begin{align}
    \phi_{i,\oplus}&=\sum_\alpha\phi_{\alpha,\text{S}}\mid{U_{\alpha i}}\mid^2\\
    \phi_{\alpha,\oplus}&=\sum_{i,\beta}
      \mid{U_{\alpha i}}\mid^2\mid{U_{\beta i}}\mid^2\phi_{\beta,\text{S}}
\end{align}
$$

This is nice because all the complicated interference terms drop out, and the oscillation depends only on the **square of the mixing matrix**. From this, the measured flavor composition on Earth is defined as $f_{\alpha,\oplus}=\phi_{\alpha,\oplus}/\sum_\alpha\phi_{\alpha,\oplus}$, where the $\oplus$ subscript denotes as measured on Earth. We can compute this using GolemFlavor:

```python
from golemflavor.fr import u_to_fr

measured_composition = u_to_fr(source_composition, NUFIT_U)
print('Measured composition = ({:.2f} : {:.2f} : {:.2f})'.format(*measured_composition))
```

    Measured composition = (0.55 : 0.18 : 0.27)

The effect of neutrino mixing has modified the flavor composition from $\left(1:0:0\right)_\text{S}\rightarrow\left(0.55:0.18:0.27\right)_\oplus$ at Earth!

Here is listed the expected measured compositions from some other source composition models:

$$
\begin{align}
    \left(0:1:0\right)_\text{S}&\rightarrow\left(0.18:0.44:0.38\right)_\oplus\\
    \left(1:2:0\right)_\text{S}&\rightarrow\left(0.31:0.35:0.34\right)_\oplus
\end{align}
$$

This can be more easily visualized using a [ternary plot](https://zenodo.org/badge/latestdoi/19505/marcharper/python-ternary), with axes being the fraction of each neutrino flavor:

```python
import golemflavor.plot as plot_utils
# getdist package requires `%matplotlib inline` to come after the import for inline notebook figures.
%matplotlib inline

nbins = 25
fontsize = 23

# Figure
fig = plt.figure(figsize=(12, 12))

# Axis
ax = fig.add_subplot(111)
tax = plot_utils.get_tax(ax, scale=nbins, rot_ax_labels=True)

# Plot source composition
tax.scatter([normalize_fr([1, 2, 0])*nbins], marker='o', s=350, facecolors='red',
            edgecolors='k', linewidth=2.3, label=r'$(1:2:0)_{\rm S}$', zorder=3)
tax.scatter([np.array([0, 1, 0])*nbins], marker='s', s=350, facecolors='green',
            edgecolors='k', linewidth=2.3, label=r'$(0:1:0)_{\rm S}$', zorder=3)
tax.scatter([np.array([1, 0, 0])*nbins], marker='^', s=350, facecolors='blue',
            edgecolors='k', linewidth=2.3, label=r'$(1:0:0)_{\rm S}$', zorder=3)

# Plot measured composition
tax.scatter([u_to_fr([1, 2, 0], NUFIT_U)*nbins], marker='o', s=350,
            edgecolors='k', facecolors='white', linewidth=2.3, zorder=3)
tax.scatter([u_to_fr([0, 1, 0], NUFIT_U)*nbins], marker='s', s=350,
            edgecolors='k', facecolors='white', linewidth=2.3, zorder=3)
tax.scatter([u_to_fr([1, 0, 0], NUFIT_U)*nbins], marker='^', s=350,
            edgecolors='k', facecolors='white', linewidth=2.3, zorder=3)

# Draw arrows
ax.annotate("", xy=np.array([0.415, 0.44])*nbins, xytext=np.array([0.499, 0.83])*nbins,
            arrowprops=dict(arrowstyle="-|>",facecolor='k',lw=2), zorder=3)
ax.annotate("", xy=np.array([0.505, 0.335])*nbins, xytext=np.array([0.64, 0.55])*nbins,
            arrowprops=dict(arrowstyle="-|>",facecolor='k',lw=2), zorder=3)
ax.annotate("", xy=np.array([0.67, 0.14])*nbins, xytext=np.array([0.975, 0.014])*nbins,
            arrowprops=dict(arrowstyle="-|>",facecolor='k',lw=2), zorder=3)

# Legend
l_size = fontsize
legend = plt.legend(loc=(0.7, 0.75), title=r'Source composition',
                    fontsize=l_size, prop={'size': fontsize})
plt.setp(legend.get_title(), fontsize=l_size)
ax.add_artist(legend)

plt.show()
```

The coloured circle, square and triangle show the source flavor compositions. The arrows show the effect of neutrino mixing on the flavor composition. The unfilled circle, square and triangle show the corresponding measured flavor composition. Neutrino mixing during propagation has the effect of averaging out the flavor contributions, which is why the arrows point towards the center of the triangle.

## Generate Fake Data

Using the things we have learned above, we can start to generate some data! Usually, this comes in the form of a likelihood fit comparing IceCube data to our models. GolemFlavor has built in hooks to the [GolemFit package](https://github.com/IceCubeOpenSource/GolemFit) for this, however `GolemFit` is only accessible to IceCube collaborators as it contains proprietary code/data. Instead, we can generate some fake data using a multivariate Gaussian likelihood. GolemFlavor has a convenient function to do such a task.

```python
from golemflavor.llh import multi_gaussian
help(multi_gaussian)
```

    Running without GolemFit
    Help on function multi_gaussian in module golemflavor.llh:

    multi_gaussian(fr, fr_bf, smearing, offset=-320)
        Multivariate Gaussian log likelihood.

        Parameters
        ----------
        fr : List[float], length 3
            The flavor composition to evaluate at.
        fr_bf : List[float], length 3
            The bestfit / injected flavor composition.
        smearing : float
            The amount of smearing.
        offset : float, optional
            An amount to offset the magnitude of the log likelihood.

        Returns
        ----------
        llh : float
            The log likelihood evaluated at `fr`.

### Smearing

In reality, a measurement does not have an arbitrary precision due to effects such as mis-reconstruction and model uncertainties. These effects are said to *smear* the data, and it can be described as in our Gaussian likelihood using the `smearing` keyword. Here we set the amount of smearing

```python
smearing = 0.02
```

### Anarchic Sampling

Now we have everything we need to do scan over our likelihood, from which we will be able to visualize the effect of this smearing. However, scanning directly in the space of the flavor composition would not be the correct way to do the scan. This particular parameterization has degeneracies, since the total flavor composition must add up to 1, $\sum_{\alpha}f_\alpha=1$, which introduces an unwanted prior dependence.

The effective number of parameters of the flavor composition can be reduced from three to two due to the requirement $\sum_\alpha f_\alpha=1$. Therefore, in order to make sure we have an unbiased prior, the parameters in which to sample in must be determined by the [*Haar measure*](https://doi.org/10.1016/j.physletb.2003.08.045) of the flavor composition volume element, $\text{d}f_{e,\oplus}\wedge\text{d} f_{\mu,\oplus}\wedge\text{d}f_{\tau,\oplus}$. The following *flavor angles* parameterization was created for this reason:

$$
\begin{align}
    f_{\alpha,\oplus}=
    \begin{pmatrix}
      f_{e,\oplus} \\ f_{\mu,\oplus} \\ f_{\tau,\oplus}
    \end{pmatrix}=
    \begin{pmatrix}
      \sin^2\phi_\oplus\,\cos^2\psi_\oplus \\
      \sin^2\phi_\oplus\,\sin^2\psi_\oplus \\
      \cos^2\phi_\oplus
    \end{pmatrix}
\end{align}
$$

$$
\begin{align}
    \text{d} f_{e,\oplus}\wedge\text{d} f_{\mu,\oplus}\wedge\text{d} f_{\tau,\oplus}=
    \text{d}\left(\sin^4\phi_\oplus\right)\wedge
    \text{d}\left(\cos\left(2\psi_\oplus\right)\right)
\end{align}
$$

This basically tells us that we should scan in the parameter space defined by $\sin^4\phi_\oplus$ and $\cos\left(2\psi_\oplus\right)$. GolemFlavor contains a convenient function `fr_to_angles` to convert from flavor compositions to flavor angles.

```python
from golemflavor.fr import fr_to_angles

measured_angles = fr_to_angles(measured_composition)
print('Measured composition = ({:.2f} : {:.2f} : {:.2f})'.format(*measured_composition))
print('Measured flavor angles = ({:.2f}, {:.2f})'.format(*measured_angles))
```

    Measured composition = (0.55 : 0.18 : 0.27)
    Measured flavor angles = (0.54, 0.50)

### Markov Chain Monte Carlo

Now we can define the wrappers to the [emcee package](https://emcee.readthedocs.io/en/stable/) which will sample over the flavor angles using an affine invariant MCMC algorithm. To do this, it is convenient to define our parameters using the GolemFlavor `ParamSet` class, as so:

```python
from golemflavor.enums import ParamTag
from golemflavor.param import Param, ParamSet

# Convert from flavor composition to flavor angles
measured_flavor_angles = fr_to_angles(measured_composition)

# Params can be tagged for later convenience
tag = ParamTag.BESTFIT

# Define the asimov `ParamSet`, with `Param` objects containing information such as name, value and ranges.
# Note: std defines the Prior standard deviation, however default behaviour is to use a flat prior.
# The assignment of `std=smearing` is just a placeholder for later. See `measurement.ipynb` example for further details.
asimov_paramset = [
    Param(name='measured_angle1', value=measured_flavor_angles[0], ranges=[ 0., 1.], std=smearing, tag=tag, tex=r'\sin^4\phi_\oplus'),
    Param(name='measured_angle2', value=measured_flavor_angles[1], ranges=[-1., 1.], std=smearing, tag=tag, tex=r'\cos(2\psi_\oplus)')
]
asimov_paramset = ParamSet(asimov_paramset)

# Define the llh `ParamSet`, with `Param` objects containing information such as name, value and ranges.
tag = ParamTag.BESTFIT
src_compositions = [
    Param(name='measured_angle1', value=0, ranges=[ 0., 1.], tag=tag, tex=r'\sin^4\phi_\oplus'),
    Param(name='measured_angle2', value=0, ranges=[-1., 1.], tag=tag, tex=r'\cos(2\psi_\oplus)')
]
llh_paramset = ParamSet(src_compositions)
```

Here we have 2 `ParamSet` objects:

- `asimov_paramset` contains the measured parameters
- `llh_paramset` contains the model parameter values

In this example, they contain the same parameters since we are doing a simple scan over the measured flavor angles to generate some fake data.

Now, we wrap our `multi_gaussian` likelihood into a function that accepts input parameters `theta` from the MCMC:

```python
from golemflavor.fr import angles_to_fr

def triangle_llh(theta, asimov_paramset, llh_paramset):
    """Log likelihood function for a given theta."""
    if len(theta) != len(llh_paramset):
        raise AssertionError(
            'Length of MCMC scan is not the same as the input '
            'params\ntheta={0}\nparamset]{1}'.format(theta, llh_paramset)
        )

    # Set llh_parameters values to the sampled parameters
    for idx, param in enumerate(llh_paramset):
        param.value = theta[idx]

    # Convert flavor angles to flavor compositions for the model parameters
    measured_angles = llh_paramset.from_tag(ParamTag.BESTFIT, values=True)
    measured_composition = angles_to_fr(measured_angles)

    # Convert flavor angles to flavor compositions for the injected parameters
    bestfit_measured_angles = asimov_paramset.from_tag(ParamTag.BESTFIT, values=True)
    bestfit_measured_comp = angles_to_fr(bestfit_measured_angles)

    # Get the value of `smearing`
    smearing = asimov_paramset['measured_angle1'].std

    # Calculate the log likelihood using `multi_gaussian`
    llh = multi_gaussian(measured_composition, bestfit_measured_comp, smearing)
    return llh
```

Last thing we need to setup is our prior distribution, which in this case is simply the bounds on the flavor angles. As we have defined this already in the `ParamSet` object using the `ranges` keyword, we can use the GolemFlavor function `lnprior` to do the work for us:

```python
from golemflavor.llh import lnprior

def ln_prob(theta, asimov_paramset, llh_paramset):
    """Posterior function for a given theta."""
    # Get the value of the log prior (in this case it will be either 0 or -inf)
    lp = lnprior(theta, paramset=llh_paramset)
    if not np.isfinite(lp):
        return -np.inf

    # Return the log prior + log likelihood
    return lp + triangle_llh(theta, asimov_paramset, llh_paramset)

# Evalaute the posterior using the defined `asimov_paramset` and `llh_paramset`
ln_prob_eval = partial(
    ln_prob,
    asimov_paramset=asimov_paramset,
    llh_paramset=llh_paramset
)
```

Now we simply define the number of walkers, burnin period and number of steps to run the MCMC and GolemFlavor takes care of the rest!

```python
import golemflavor.mcmc as mcmc_utils

# Reduce these values for a quicker runtime
nwalkers = 60
burnin = 1000
nsteps = 10000

# Generate initial seed using a flat distribution
p0 = mcmc_utils.flat_seed(
    llh_paramset, nwalkers=nwalkers
)

# Run the MCMC!
# Progress bar provided by tqdm (this took about 5mins on my laptop)
samples = mcmc_utils.mcmc(
    p0       = p0,
    ln_prob  = ln_prob_eval,
    ndim     = len(llh_paramset),
    nwalkers = nwalkers,
    burnin   = burnin,
    nsteps   = nsteps,
    threads  = 4
)
```

    Running burn-in

    HBox(children=(FloatProgress(value=0.0, max=1000.0), HTML(value='')))

    Finished burn-in
    Running

    HBox(children=(FloatProgress(value=0.0, max=10000.0), HTML(value='')))

    Finished
    acceptance fraction [0.7059 0.7071 0.7221 0.7087 0.7189 0.7099 0.7152 0.7069 0.7112 0.7217
     0.7203 0.7234 0.709  0.7185 0.7138 0.7104 0.7067 0.715  0.7182 0.7177
     0.7172 0.7156 0.7213 0.723  0.7152 0.7137 0.7165 0.7208 0.7159 0.714
     0.7172 0.7138 0.7222 0.7161 0.7132 0.718  0.7244 0.7142 0.7124 0.7143
     0.7097 0.7144 0.7167 0.7113 0.7074 0.7177 0.7107 0.7142 0.7047 0.7099
     0.7077 0.7167 0.7238 0.7165 0.7083 0.7205 0.7172 0.7265 0.7189 0.7131]
    sum of acceptance fraction 42.905499999999996
    np.unique(samples[:,0]).shape (429072,)
    autocorrelation [30.41597538 31.01818003]

## Visualize Fake Data

Now that we have generated the fake data, let's see if we can visualize it in a ternary plot. First we convert the data from flavor angles into flavor compositions:

```python
measured_angles = samples
measured_compositions = np.array(
    list(map(angles_to_fr, measured_angles))
)
```

Now we can display the 90%/99% credibility regions on the ternary plot to show how our fake data is distributed:

```python
nbins = 25
fontsize = 23

# Figure
fig = plt.figure(figsize=(12, 12))

# Axis
ax = fig.add_subplot(111)
tax = plot_utils.get_tax(ax, scale=nbins, rot_ax_labels=True)

# Plot source composition
tax.scatter([np.array([1, 0, 0])*nbins], marker='^', s=350, facecolors='blue',
            edgecolors='k', linewidth=2.3, label=r'$(1:0:0)_{\rm S}$', zorder=3)

# Plot measured composition posteriors
coverages = [(99, 'cornflowerblue'), (90, 'royalblue')]
for cov, color in coverages:
    plot_utils.flavor_contour(
        frs=measured_compositions,
        fill=True,
        ax=ax,
        nbins=nbins,
        coverage=cov,
        linewidth=2.5,
        color=color,
        alpha=0.7,
        oversample=8
    )

# Draw arrow
ax.annotate("", xy=np.array([0.67, 0.14])*nbins, xytext=np.array([0.975, 0.014])*nbins,
            arrowprops=dict(arrowstyle="-|>",facecolor='k',lw=2), zorder=3)

# Legend
l_size = fontsize
handles, labels = ax.get_legend_handles_labels()
legend = plt.legend(handles=[handles[-1]], labels=[labels[-1]], loc=(0.7, 0.85),
                    title=r'Source composition', fontsize=l_size, prop={'size': fontsize})
plt.setp(legend.get_title(), fontsize=l_size)
ax.add_artist(legend)

plt.show()
```

What a measurement! In truth, the accuracy shown here has not been reached thus far, however the field of neutrino flavor physics is developing quickly. If you are interested in comparing this to a real flavor contour then you can checkout [this paper](https://doi.org/10.1088/0004-637X/809/1/98) by the IceCube collaboration.

Thanks for reading! In the next example, `inference.ipynb`, we will see if we can make an inference of the source flavor composition using this fake data.


##  For preliminary  Research

```python
import numpy as np
import matplotlib.pyplot as plt

```

```python
import numpy as np
import matplotlib.pyplot as plt

x = [5,7,8,7,2,17,2,9,4,11,12,9,6]
y = [99,86,87,88,111,86,103,87,94,78,77,85,86]

plt.scatter(x, y)
plt.xlabel('x value', fontsize = 12)
plt.ylabel('Y value', fontsize = 12)
plt.title('Scatter Plot', fontsize = 15)
plt.savefig('Scatter.png')
plt.show()
#plt.save as 1.png
```

```python
import numpy
import matplotlib.pyplot as plt

x = numpy.random.normal(5.0, 1.0, 1000)
y = numpy.random.normal(10.0, 2.0, 1000)

plt.scatter(x, y)
plt.savefig('Sc.png')
plt.show()
```

```python
import numpy

speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]

x = numpy.mean(speed)

print(x)
```

    89.76923076923077

```python
import numpy

speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]

x = numpy.median(speed)

print(x)
```

    87.0

```python
from scipy import stats

speed = [99,86,87,88,111,86,103,87,94,78,77,85,86]

x = stats.mode(speed)

print(x)
```

    ModeResult(mode=array([86]), count=array([3]))

```python
from scipy import stats

x = [5,7,8,7,2,17,2,9,4,11,12,9,6]
y = [99,86,87,88,111,86,103,87,94,78,77,85,86]

slope, intercept, r, p, std_err = stats.linregress(x, y)

#r=R-Squared value is a quantifiable analysis of how well
#the line of best fit (linear regression model) fits your data.
#A value closer to 1 (100%) is usually good.

#The P value is the probability of finding the observed results when the null hypothesis of a statement is true.
# std_err= Standard error of data

#A P value of less than 0.001 is referred to as statistically highly significant
#(an extremely low chance of being wrong).

def myfunc(x):
  return slope * x + intercept

mymodel = list(map(myfunc, x))

plt.scatter(x, y)
plt.plot(x, mymodel)
plt.xlabel('x', fontsize = 12)
plt.ylabel('y', fontsize = 12)
plt.savefig('S.png')
plt.show()
print(r)
print(p)
print(std_err)
```

```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import scipy.stats as st
df = pd.read_csv('AS.csv')
#dataframe
```

```python
df
```

      Unnamed: 0
      Unnamed: 1
      Unnamed: 2
      RA
      Unnamed: 4
      Unnamed: 5
      Unnamed: 6
      DEC
      F60
      Unnamed: 9
      ...
      Unnamed: 11
      Unnamed: 12
      Unnamed: 13
      Unnamed: 14
      F100
      T
      B
      Unnamed: 18
      M
      Unnamed: 20

      0
      4
      40
      20.32
      70.084667
      45
      12
      36.8
      45.210222
      4.9004
      4
      ...
      20.32
      45
      12
      36.8
      23.223
      23.357923
      8.397450e-16
      8.397450
      2.624653e+28
      2.624653

      1
      4
      40
      19.95
      70.083125
      45
      11
      9.7
      45.186028
      4.8348
      4
      ...
      20.32
      45
      11
      9.7
      22.769
      23.393590
      8.476872e-16
      8.476872
      2.598671e+28
      2.598671

      2
      4
      40
      22.20
      70.092500
      45
      9
      42.5
      45.161806
      5.0315
      4
      ...
      22.20
      45
      9
      42.5
      22.565
      23.675585
      9.122802e-16
      9.122802
      2.413380e+28
      2.413380

      3
      4
      40
      21.08
      70.087833
      45
      8
      3.5
      45.134306
      5.8450
      4
      ...
      21.08
      45
      7
      59.6
      22.914
      24.487987
      1.116684e-15
      11.166844
      1.970421e+28
      1.970421

      4
      4
      40
      23.71
      70.098792
      45
      2
      15.1
      45.037528
      4.9721
      4
      ...
      23.71
      45
      2
      15.1
      22.856
      23.532299
      8.790589e-16
      8.790589
      2.497695e+28
      2.497695

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      195
      4
      37
      53.17
      69.471542
      45
      1
      52.4
      45.031222
      4.7870
      4
      ...
      53.17
      45
      1
      48.4
      23.138
      23.246238
      8.152016e-16
      8.152016
      2.692970e+28
      2.692970

      196
      4
      37
      53.24
      69.471833
      45
      0
      29.3
      45.008139
      4.7069
      4
      ...
      53.24
      45
      0
      21.3
      22.922
      23.204123
      8.060747e-16
      8.060747
      2.722065e+28
      2.722065

      197
      4
      37
      53.68
      69.473667
      44
      59
      2.2
      44.983944
      4.5735
      4
      ...
      53.67
      44
      59
      6.1
      22.614
      23.119057
      7.878520e-16
      7.878520
      2.783528e+28
      2.783528

      198
      4
      37
      56.37
      69.484875
      44
      57
      15.5
      44.954306
      4.6398
      4
      ...
      56.37
      44
      57
      11.5
      23.243
      23.046670
      7.725681e-16
      7.725681
      2.836725e+28
      2.836725

      199
      4
      37
      44.64
      69.436000
      45
      0
      36.4
      45.010111
      4.8512
      4
      ...
      45.02
      45
      0
      32.5
      23.385
      23.261467
      8.185192e-16
      8.185192
      2.680797e+28
      2.680797

200 rows × 21 columns

```python
df.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 200 entries, 0 to 199
    Data columns (total 21 columns):
     #   Column       Non-Null Count  Dtype
    ---  ------       --------------  -----
     0   Unnamed: 0   200 non-null    int64
     1   Unnamed: 1   200 non-null    int64
     2   Unnamed: 2   200 non-null    float64
     3   RA           200 non-null    float64
     4   Unnamed: 4   200 non-null    int64
     5   Unnamed: 5   200 non-null    int64
     6   Unnamed: 6   200 non-null    float64
     7   DEC          200 non-null    float64
     8   F60          200 non-null    float64
     9   Unnamed: 9   200 non-null    int64
     10  Unnamed: 10  200 non-null    int64
     11  Unnamed: 11  200 non-null    float64
     12  Unnamed: 12  200 non-null    int64
     13  Unnamed: 13  200 non-null    int64
     14  Unnamed: 14  200 non-null    float64
     15  F100         200 non-null    float64
     16  T            200 non-null    float64
     17  B            200 non-null    float64
     18  Unnamed: 18  200 non-null    float64
     19  M            200 non-null    float64
     20  Unnamed: 20  200 non-null    float64
    dtypes: float64(13), int64(8)
    memory usage: 32.9 KB

```python
df.describe()
```

      Unnamed: 0
      Unnamed: 1
      Unnamed: 2
      RA
      Unnamed: 4
      Unnamed: 5
      Unnamed: 6
      DEC
      F60
      Unnamed: 9
      ...
      Unnamed: 11
      Unnamed: 12
      Unnamed: 13
      Unnamed: 14
      F100
      T
      B
      Unnamed: 18
      M
      Unnamed: 20

      count
      200.0
      200.00000
      200.000000
      200.000000
      200.000000
      200.000000
      200.000000
      200.000000
      200.000000
      200.0
      ...
      200.00000
      200.000000
      200.000000
      200.000000
      200.000000
      200.000000
      2.000000e+02
      200.000000
      2.000000e+02
      200.000000

      mean
      4.0
      38.72000
      29.227200
      69.801780
      44.730000
      19.615000
      26.201000
      45.064195
      4.549389
      4.0
      ...
      29.32465
      44.720000
      20.175000
      27.592000
      22.153490
      23.200885
      8.061341e-16
      8.061341
      2.732409e+28
      2.732409

      std
      0.0
      0.75794
      17.693744
      0.169578
      0.445074
      23.201039
      17.228526
      0.088033
      0.249222
      0.0
      ...
      17.67544
      0.450126
      23.452815
      17.403775
      0.718893
      0.196075
      4.369863e-17
      0.436986
      1.338126e+27
      0.133813

      min
      4.0
      37.00000
      0.010000
      69.436000
      44.000000
      0.000000
      0.100000
      44.876889
      4.051400
      4.0
      ...
      0.02000
      44.000000
      0.000000
      0.300000
      20.901000
      22.838878
      7.298215e-16
      7.298215
      1.970421e+28
      1.970421

      25%
      4.0
      38.00000
      13.590000
      69.659458
      44.000000
      3.000000
      10.500000
      44.989583
      4.365625
      4.0
      ...
      13.59750
      44.000000
      3.000000
      12.750000
      21.509000
      23.081515
      7.798998e-16
      7.798998
      2.665628e+28
      2.665628

      50%
      4.0
      39.00000
      28.975000
      69.798937
      45.000000
      8.000000
      22.750000
      45.051667
      4.528500
      4.0
      ...
      29.07500
      45.000000
      8.000000
      25.400000
      22.166500
      23.204056
      8.060603e-16
      8.060603
      2.727378e+28
      2.727378

      75%
      4.0
      39.00000
      45.262500
      69.949406
      45.000000
      54.000000
      39.850000
      45.138542
      4.704800
      4.0
      ...
      45.51000
      45.000000
      55.000000
      42.575000
      22.780000
      23.292709
      8.253539e-16
      8.253539
      2.814424e+28
      2.814424

      max
      4.0
      40.00000
      59.980000
      70.103500
      45.000000
      59.000000
      59.700000
      45.256361
      5.845000
      4.0
      ...
      59.98000
      45.000000
      59.000000
      59.900000
      23.385000
      24.487987
      1.116684e-15
      11.166844
      3.005047e+28
      3.005047

8 rows × 21 columns

# Best fit Line

```python
from scipy import stats
F60 = df['F60']
F100= df['F100']
plt.figure(figsize=(8,8))
slope, intercept, r, p, std_err = stats.linregress(F100, F60)

def myfunc(F100):
  return slope * F100 + intercept

mymodel = list(map(myfunc, F100))

plt.scatter(F100,F60)
plt.plot(F100, mymodel, color = 'red', linewidth = 1)
plt.xlabel('F100(MJySr$^{-1}$)', fontsize = 12)
plt.ylabel('F60(MJySr$^{-1}$)', fontsize = 12)
plt.savefig('S.png')
```

```python
RA = df['RA']
DEC= df['DEC']
F100 = df['F100']
```

```python
plt.figure(figsize=(8,8))
plt.hexbin(RA, DEC, F100, gridsize=(20))
plt.colorbar()
plt.xlim(70.103500, 69.436000)
```

# 3D Contour plot

```python
#cmap: https://matplotlib.org/3.1.0/tutorials/colors/colormaps.html

plt.figure(figsize=(8,8))
plt.tricontourf(RA, DEC, F100, levels=8, cmap='coolwarm')
plt.colorbar()
#plt.clabel(r'F60($\mu$m)')
plt.xlim(70.103500, 69.436000)
plt.xlabel('RA(deg.)')
plt.ylabel('DEC(deg.)')
plt.savefig('F100.png')
```

```python
RA = df['RA']
DEC= df['DEC']
T = df['F60']
plt.figure(figsize=(8,8))
plt.hexbin(RA, DEC, F60, gridsize=(20))
plt.colorbar()
plt.xlim(70.103500, 69.436000,)

plt.figure(figsize=(8,8))
plt.tricontourf(RA, DEC, F60, levels=8, cmap='coolwarm')
plt.colorbar()
#plt.clabel(r'F60($\mu$m)')
plt.xlim(70.103500, 69.436000)
plt.xlabel('RA(deg.)')
plt.ylabel('DEC(deg.)')
plt.savefig('F60.png')
```

# Dust Color Temperature
There are many model for calculation of temperature with the help
of different data base. In this sort of dissertation work we use
data base from the IRAS 60 $\mu$m and 100 $\mu$m flux densities is
similar to that of Schnee $et$ $al.$ (2005) \cite{36}. By knowing
the flux densities at 60 $\mu$m and 100 $\mu$m, the temperature
contribution due to dust color can be calculated. The dust
temperature $T_{d}$ in each pixel of a FITS (Flexible Image
Transport System) image can be obtained by assuming that the dust
in a single beam is isothermal and that the observed ratio of 60
$\mu$m to 100 $\mu$m emission is due to black body radiation from
dust grains at $T_{d}$, modified by a power law of spectral
emissivity index. The flux density of emission at a wavelength
$\lambda_{i}$ is given by

\begin{equation}\label{3}
F_{i} = \bigg[\frac{2hc}{\lambda^{3}_{i}(e^{\frac{hc}{\lambda_{i}KT_{d}}}-1)}\bigg]N_{d}\alpha\lambda_{i}^{-\beta}\Omega_{i}
\end{equation}

where, $N_{d}$ is the column density of dust grains, $\alpha$ is a
constant which relates the flux with the optical depth of the
dust, $\beta$ is the spectral emissivity index, and $\Omega_{i}$
is the solid angle subtended at $\lambda_{i}$ by the detector.
Following Dupac $et$ $al.$ (2003)\cite{37}, we use the equation
\begin{equation}\label{4}
\beta = \frac{1}{(\delta+wT_{d})}
\end{equation}
to describe the observed inverse relationship between temperature
and emissivity spectral index. Here, $\delta$ and $\omega$ are fee
parameters found that the temperature dependence of the emissivity
index fits very well with the hyperbolic approximating
function \cite{37}.
\\Considering temperature as an independent variable, the best fit gives
$\delta$ = 0.40 $\pm$ 0.02 and $\omega$ = 0.0079 $\pm$ 0.0005 K$^{-1}$, with
the $\chi^{2}/$degree of freedom = 120/120. With the assumptions that the dust emission is optically thin at
60 $\mu$m and 100 $\mu$m and that $\Omega_{\omega}$$\simeq$
$\Omega_{100}$ (true for IRAS image), we can write the ratio, R,
of the flux densities at 60 $\mu$m and 100 $\mu$m as
\begin{equation}\label{5}
R =  0.6^{-(3+\beta)}\bigg[\frac{e^{\frac{144}{T_{d}}}-1}{e^{\frac{240}{T_{d}}}-1}\bigg]
\end{equation}
The value of $\beta$ depends on dust grain properties as
composition, size, and compactness. For reference, a pure
blackbody would have $\beta$ = 0, the amorphous layer-lattice
matter has $\beta \sim 1$, and the metals and
crystalline dielectrics have $\beta \sim 2$.\\
\\
For a smaller value of $T_{d}$, 1 can be dropped from both
numerator and denominator of equation (16) and it takes the form
\begin{equation}\label{6}
R =
0.6^{-(3+\beta)}\bigg[\frac{e^{\frac{144}{T_{d}}}}{e^{\frac{240}{T_{d}}}}\bigg]
\end{equation}
Taking natural logarithm on both sides of equation (2.50) we find
the expression for the temperature as
\begin{equation}\label{7}
T_{d} = \frac{-96}{ln\{R \times 0.6^{(3+\beta)}\}}
\end{equation}
where R is given by
\begin{equation}\label{8}
R = \frac{F(60 \mu m)}{F(100 \mu m)}
\end{equation}
F(60 $\mu$m) and F(100 $\mu$m) are the flux densities at 60 $\mu$m
and 100 $\mu$m, respectively. In this way we can use equation
(18) for the determination of the dust grain temperature.

```python
RA = df['RA']
DEC= df['DEC']
T = df['T']
plt.figure(figsize=(8,8))
plt.hexbin(RA, DEC, T, gridsize=(20))
plt.colorbar()
plt.xlim(70.103500, 69.436000,)

plt.figure(figsize=(8,8))
plt.tricontourf(RA, DEC, T, levels=8, cmap='coolwarm')
plt.colorbar()
#plt.clabel(r'F60($\mu$m)')
plt.xlim(70.103500, 69.436000)
plt.xlabel('RA(deg.)')
plt.ylabel('DEC(deg.)')
plt.savefig('Dev.jpg')
```

# Dust Mass Estimation
In order to estimate the dust masses from the infrared flux
densities at 100 $\mu$m, following the calculation of Young $et$
$al.$\ we need the background correction of flux and
convert the relative flux into absolute flux.
\begin{equation}\label{9}
M_{dust} = \frac{4}{{3}}\frac{a\rho}{{Q_{\nu}}}\bigg[\frac{S_{\nu}D^{2}}{{B(\nu, T)}}\bigg]
\end{equation}
where,
$a$ = weighted grain size

$\rho$ = grain density

$D$ = Distance to the star

$ Q_{\nu}$ = grain emissivity

$ S_{\nu}$ = f $\times$ MJy/Sr $\times$ 5.288 $\times$ 10$^{-9}$
\textbf{For 100 $\mu$m emitter}
\\
$a$ = 0.1 $\mu$m\\
$\rho$ = grain density =3000 kg m$^{3}$ ,
Q$_{\nu}$ = grain emissivity = 0.0010 (for 100 $\mu$m)\\
Q$_{\nu}$ = grain emissivity = 0.0046 (for 60 $\mu$m)
\cite{39}\\
$ S_{\nu}$ = f $\times$ MJy/Sr $\times$ 5.288 $\times$ 10$^{-9}$\\
where, 1 MJy/Sr = 1 $\times 10^{-20}$ kg s$^{-2}$ and f = relative
flux density measured from the Groningen IRAS image\cite{41}.\\
For 100 $\mu$m wavelength, the expression for the dust mass
reduces to,

\begin{equation}
M_{dust} = 0.40\bigg[\frac{S_{\nu}D^{2}}{{B(\nu, T)}}\bigg]
\end{equation}

The Planck's function is a well known function, given by this
equation,
\begin{equation}\label{10}
B(\nu, T) = \frac{2h\nu^{3}}{{c^{2}}}\bigg[\frac{1}{exp^(\frac{h\nu}{KT})-1}\bigg]
\end{equation}
Where,
\\
$h$ = Planck's constant\\
$c$ = velocity of light\\
${\nu}$ = frequency at which the emission is observed\\
$T$ = Temperature of each pixel

```python
RA = df['RA']
DEC= df['DEC']
M = df['M']
plt.figure(figsize=(8,8))
plt.hexbin(RA, DEC, M, gridsize=(20))
plt.colorbar()
plt.xlim(70.103500, 69.436000,)

plt.figure(figsize=(8,8))
plt.tricontourf(RA, DEC, M, levels=8, cmap='coolwarm')
plt.colorbar()
#plt.clabel(r'F60($\mu$m)')
plt.xlim(70.103500, 69.436000)
plt.xlabel('RA(deg.)')
plt.ylabel('DEC(deg.)')
plt.savefig('M.png')
```

```python

RA = df['RA']
DEC= df['DEC']
B = df['B']
plt.figure(figsize=(8,8))
plt.hexbin(RA, DEC, B, gridsize=(20))
plt.colorbar()
plt.xlim(70.103500, 69.436000,)

plt.figure(figsize=(8,8))
plt.tricontourf(RA, DEC, B, levels=8, cmap='coolwarm')
plt.colorbar()

#plt.clabel(r'F60($\mu$m)')
plt.xlim(70.103500, 69.436000)
plt.xlabel('RA(deg.)')
plt.ylabel('DEC(deg.)')
plt.savefig('B.png')

```

```python
from mpl_toolkits.mplot3d import axes3d
```

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(RA, DEC, F100)
```

\begin{thebibliography}{}
	\bibitem{4} Cox, D. P.(2005), The Three-Phase Interstellar Medium Revisited, Annu. Rev. Astron. Astrophys. 43, 337-385.
	\bibitem{5}  Tielens, A.G.G.M. (2005). \emph{ Physics and Chemistry of ISM}, 22, Cambridge University Press.
	\bibitem{akj} Jha, A. K.,   Aryal, B. and  Weinberger, R.(2018). A study of dust color temperature and dust mass distributions of
	four far infrared loops. Revista Mexicana de Astronomia y Astrofisica \textbf{53}, 3.
		\bibitem{B_1} Zijlstra, A. A. and    Weinberger, R. (2002). A WALL OF DUST AROUND A PROTO-MIRA?, The Astrophysical Journal, \textbf{572}, 1006–1011.
		\bibitem{B_2} Aryal, B. and  Weinberger R.(2006). A new large high latitude cone-like far-IR nebula, Astronomy \&
		Astrophysics, \textbf{446}, 213.
		\bibitem{B_4}  Aryal, B.,  Rajbahak, C. and  Weinberger, R.  (2010). A giant dusty bipolar structure around the planetary nebula NGC 1514,Mon. Not. R. Astron. Soc. \textbf{402}, 1307–1312.

	\bibitem{10} Weaver, R., McCray,  R., Castor,  J.,  Shapiro, P. and  Moore, R.(1977). Interstellar bubbles, II. structure and evolution, The Astronomical Journal,  \textbf{218}, 377-395.
	\bibitem{14} Jackson, J. D.(1962). \emph{Classical electrodynamics}, \textbf{62-8774.}, 183 John Wiley \& Sons, Inc., New York.

	\bibitem{G} Duffy, D.G. \emph{Green's  function with applications},  Chapman \&  Hall/CRC (2001).
	\bibitem{1}  Iben I. J. and  Renzini,  A.  (1983). Asymptotic giant branch evolution and beyond. A\&A \textbf{21}, 271.

	\bibitem{36} Schnee, S. L.,  Ridge, N. A.,  Goodman, A. A., and Jason, G. L. (2005). A Complete Look at The Use of IRAS
	Emission Maps to Estimate Extinction and Dust Temperature, Astrophysical Journal \textbf{634}  442- 450.
	\bibitem{37}  Dupac, X.,  Bernard, J. P., Boudet, N., Giard, M.,   Lamarre, J. M.,    Mény, C.,  Pajot, F.,  Ristorcelli, I.,  Serra, G.,  Stepnik, B.  and Torre, J. P. (2003). Inverse Temperature Dependence of the dust sub millimeter spectral index,
	Astronomy \& Astrophysics \textbf{404}, L11-L15.
	\bibitem{38} Hildebrand, R. H. (1983). The Determination of Cloud Masses and Dust Characteristics from Sub millimeter
	Thermal Emission, Journal of Royal Astronomical Society, \textbf{ 24}, 267-282.
	\bibitem{39} Young, K.,  Phillips, T. G., and  Knapp, G.R. (1993). Circumstellar Shells resolved in IRAS Survey data II-Analysis,
	Astrophysical Journal, \textbf{409}, 725.
	\bibitem{41} Beichman,  C. A., Neugebauer G.,  Habing,H. J.  Clegg,P. E. and  Chester, T. J.(1988). Infrared Astronomical Satellite (IRAS) Catalogues and
	Atlases \textbf{ I}: Explanatory Supplement. US Government
	Printing Office, Washington.
		\bibitem{13}http://www.spiegelteam.de/ngc1514.html (2018).
	\bibitem{34}http://skyview.gsfc.nasa.gov/current/cgi/query.pl (2018).
	\bibitem{35} http://simbad.u-strasbg.fr (2018).
	\bibitem{32} http://www.ipac.caltech.edu/system/projects/images/15/original/IRAS$\_$Sq.jpg?1291155530 (2018).
	\bibitem{33}http://coolcosmos.ipac.caltech.edu/image$\_$galleries/IRAS/iras2.html (2018).
	\bibitem{L} https://www.nasa.gov/sites/default/files/spitzer.jpg (2018).
	\bibitem{R}https://www.jpl.nasa.gov/spaceimages/images/largesize/PIA17444hires.jpg (2018).

\end{thebibliography}

```python

```


# 常微分方程式の数値解法

```python
#使用するモジュールのimport
import numpy as np
from matplotlib import pyplot as plt
```

## 常微分方程式

多くの自然現象や社会現象は微分方程式として定式化される。

身近な例としてはニュートンの運動方程式にはじまり、
金融や群集動態、感染症のモデルなどにも当てはまる。

このノートでは、微分方程式のうちとくに一変数関数の場合である
常微分方程式(ordinary differential equation, ODE)について
いくつかの代表的な数値解法を解説するとともに、
アルゴリズムを自分で実装してみよう。

例えば1階の常微分方程式は、一般に以下の形で書くことが出来る:
$dx(t)/dt = f(x,t)$ (Eq. 1)

これは、ある質点の時刻$t$での座標を$x(t)$と呼ぶことにすると、
質点の位置の変化が$f(x,t)$の関数として書けるような状況に相当する。

注) 別に$t$は時間である必要はないし、
$x(t)$は座標以外の量(金融商品の価格など)何でも構わない。

一般に常微分方程式が与えられた時、まずはその解が変数分離などを用いて
閉じた形で書き下せないかと考えるわけだが、
実際の多くの問題ではそうは上手く行かず、数値計算が必要となる。

以下では、主として$dx/dt = f(x,t)$を考えるが、
高階の微分を含む場合も、

$\left[
\begin{matrix}
x_1 \\
x_2 \\
\vdots \\
x_r
\end{matrix}
\right]
=
\left[
\begin{matrix}
x \\
dx/dt \\
\vdots \\
d^{r-1}x/dt^{r-1}
\end{matrix} \right]
$
と置き換えれば、連立1階常微分方程式:
$
\left[
\begin{matrix}
x_2 \\
x_3 \\
\vdots \\
x_{r}
\end{matrix}
\right]
=
\frac{d}{dx}
\left[
\begin{matrix}
x_1 \\
x_2 \\
\vdots \\
x_{r-1}
\end{matrix} \right]
$
とみなすことが出来る。

常微分方程式の解を求めるためには、
適切な条件(初期条件や境界条件)を設定しなければならない。

そのうち、変数$t$の最初の値(初期値,初期条件)$t_0$と
$x(t=t_0)$の値が与えられた問題を初期値問題と呼ぶ。
※$t_0$は問題によって0だったり、$-\infty$だったりする。

解の存在性や下記のリプシッツ条件など適切な条件の元での
解の一意性の証明等については教科書に譲る。

###リプシッツ条件

関数$f$が$U$(たとえば$R^n$の部分集合) 上でリプシッツ(Lipshitz)連続であるとは、
正実数$K$が存在し、任意の$x,y \in U$に対し、
$||f(x)-f(y)|| \leq K ||x −y||$が成立することである。
以下では$||\cdot||$はユークリッドノルムとし、$K$をリプシッツ定数と呼ぶ。

この$K$が存在しているとき、
前述の常微分方程式の初期値問題は一意の解を持つ。

## 代表定期な数値解法

### オイラー(Euler)法

1階の常微分方程式$dx(t)/dt=f(x,y)$を数値的に解く上で
最も基本的な考え方は時刻(便宜上そう呼ぶ)を
細かく分割した上で、$x$の任意時刻$t$での近似値を
初期値から初めて逐次的に求めていくことである。

以下では、その最も単純な例であるオイラー法を説明する。

微分の定義:
$dx(t)/dt = \lim_{h\to 0} \frac{x(t+h) - x(t)}{h}$から
微小変分$h$を十分小さく取れば、
$x_{i+1} = x_{i} + h f(x,t_i)$という近似式が得られる。

つまり、$i=0$の初期値$(x_0,t_0=0)$から順番に上の漸化式を用いて、  $i$番目の時刻$t_i$での座標$x_i$を推定していく。

※一般に微小変分$h$は各ステップ$i$に依存しても良いが、簡単のため共通とした。

例: $dx/dt = -2x^2t$, 初期条件$x(t=0)=1.0$のもとで
$0 < t \leq 1.0$での$x$の値を予想してみよう:

```python
# f(x,t)のトイモデルとして dx/dt = -2tx^2 というものを考える
def f(x,t):
    return -  2.0*  x * x * t
#時刻tを間隔hで細かく分割する
h = 1.e-4
tr = np.arange(0.0,1.0,h)
#初期条件
x0 = 1.0

#求解
x = x0
xs = [x0]
for t in tr:
    x += h * f(x,t)
    xs += [x]

#描画
fig = plt.figure(figsize=(10,3))
plt.xlabel("t");plt.ylabel("x")
plt.plot(tr,xs[:-1])
plt.show()
plt.close()
```

実は今の初期条件のもとでこの微分方程式は
閉じた形$x(t)=x_0/(x_0t^2+1)$で解が与えられるので、
真の解と数値解法による近似解を比較できる。

差分のlogを取ってみると...以下のようになる:

```python
def fexact(t):
    return x0/(x0* t*t + 1.0)
xe = np.array([ fexact(t) for t in tr])
fig = plt.figure(figsize=(10,3))
plt.xlabel("t"); plt.ylabel("abs. diff.")
plt.yscale("log")
plt.plot(tr,abs(xe-np.array(xs[:-1])))
plt.show()
plt.close()
```

Taylor展開するだけで簡単に調べられるようにEuler法は局所打ち切り誤差が$O(h^2)$、
大局打切り誤差が$\sim O(h^2)\times O(h^{-1})\sim O(h)$の手法となっている。
つまり1桁精度を上げたいなら$h$を1/10、つまり計算量を10倍にしなくてはならない。

上の$h$を変えてチェックしてみよう。

大局誤差が$h$の$p$乗に比例する手法を一般に$p$次公式と呼び
上のEuler法は1次公式となる。

Euler法に少し工夫を加えた修正Euler法は、2次公式であり
$h$を1/10すると、大局誤差は1/100にできる。

もちろん、計算時間が単純に1/100になる訳ではないが、
高次の公式が"簡単に"使えるのなら、
一般にその方が短い計算時間で同程度の精度を得られると期待できる。

### 4次のルンゲクッタ法 (RK4)

次に4次公式の代表例である4次のルンゲクッタ法を紹介しよう。
古典的ルンゲ=クッタ法などとも呼ばれるこの方法は、
幾つかの$(x,t)$に対して$f(x,t)$の値を計算し、
打切り誤差が互いにキャンセルしあうように
うまく重み付けをして次ステップの$x$の値を求めていく方法であり
様々なところで用いられている。

基本的には2変数関数のTaylor展開をやれば更新式を導出できるが
煩雑なので導出については省略することにして結果だけ示すと

$i$番目のステップでの$x,t$の値$x_i,t_i$が所与のとき、
以下の4点での$f(x,t)$の(近似)値を用意して
$ f_1 = f(x_i,t_i),
f_2 = f(x_i+\frac{h}{2} f_1,t_i+\frac{h}{2}),$
$f_3 = f(x_i+\frac{h}{2} f_2,t_i+\frac{h}{2}),
f_4 = f(x_i+h f_3,t_i+h)
$

$x_{i+1} = x_{i} + \frac{h}{6} (f_1+2f_2+2f_3 + f_4 )$
とすることで、4次公式が得られる。

また、前述のような"重み付け"の選択には自由度があり、
$ f_1 = f(x_i,t_i),
f_2 = f(x_i+\frac{h}{3} f_1,t_i+\frac{h}{3}),$
$
f_3 = f(x_i-\frac{h}{3}f_1 + h f_2,t_i+\frac{2h}{3}),
f_4 = f(x_i+hf_1-hf_2+h f_3,t_i+h)
$

$x_{i+1} = x_{i} + \frac{h}{8} (f_1+3f_2+3f_3 + f_4 )$

という公式も知られている。

前者を実装して、先程の例でEuler法と比べてみると...

```python
#求解(RK4)
xsRK4 = [x0]
x = x0
for t in tr:
    f1 = f(x,t)
    f2 = f(x+0.5*h*f1, t+0.5*h)
    f3 = f(x+0.5*h*f2, t+0.5*h)
    f4 = f(x+h*f3, t+h)
    x += h*(f1 + 2*f2 + 2*f3 + f4)/6.0
    xsRK4 += [x]
#描画
fig = plt.figure(figsize=(10,3))
plt.xlabel("t");plt.ylabel("abs. diff.")
plt.yscale("log")
plt.plot(tr,abs(xe-np.array(xs[:-1])),label="Euler")
plt.plot(tr,abs(xe-np.array(xsRK4[:-1])),label="RK4")
plt.legend()
plt.show()
plt.close()
```

となり、高精度で解が得られていることが分かる。

## 積分を用いた解法

常微分方程式 $\frac{dx}{dt} = f(x,t)$は形式的には
$x(t') = x(t_0) + \int^{t'}_{t_0} f(x(t),t,) dt$ と書くことが出来る。

右辺の積分を数値的に計算することで、
元の微分方程式の解を求めることを考えよう。

### Adams-Bashforth法

以下では特に、Adams-Bashforth法(AB法)と呼ばれる手法を考える。

まずは、2次のAB法を考えよう。

$t_i$と$t_{i+1}$の間の$f(x(t),t)$を、現在居る$i$番目のステップと
1個前の$i-1$番目のステップを用いて、1次のLagrange補間で近似してみる。

$
\begin{align}
x_{i+1} &= x_i + \int^{t_{i+1}}_{t_i} P_1(t) dt  \\
P_1(t) &= f_{i-1}L_{i-1}(t) + f_i L_i(t) \\
L_{i-1} &= \frac{t-t_i}{t_{i-1}-t_i}, L_{i} = \frac{t-t_{i-1}}{t_{i}-t_{i-1}}
\end{align}$

各ステップの刻み幅$t_{i}-t_{i-1}$ for $\forall i $を一定$h$としよう。
このもとで上の積分を評価すると、
$x_{i+1} = x_i + \frac{h}{2} (-f_{i-1}+3f_i)$
という更新式を得る。

この手法の局所打切り誤差は(またTaylor展開して)$O(h^3)$であり、
２次の公式となる。一般的な分類に倣えば、この手法は過去の2ステップの情報を利用する、
陽(explicit)解法かつ２段法(2-step method)となっている。

同様にして、一般に$m$ステップのAB法($n-1$次のLegendre補間を使用)は$m$次公式を与える。
たとえば4次公式は$x_{i+1}=x_i + \frac{h}{24}(-9f_{i-3} + 37f_{i-2} -59f_{i-1} +55f_{i})$

**Adams-Bashforth法とRunge-Kutta法の違い**

AB法では、過去に評価した$f$の値を用いるため、計算量の増加を防ぐことができる。
つまり、各$f$の評価は１回だけで良い。
一方で、RK4では各ステップで$f$を４回計算する必要がある。

これは$f$の評価が複雑な場合
(例えば解きたい微分方程式が行列(連立)になっていて、$f$に行列演算を含む場合など)
Runge-Kutta法よりも、高速に計算できるかもしれない。

一方で、$m$次のAB法では過去の情報を利用するため、
はじめの$m-1$回については、RK法などで予め求めておく必要がある。
また、Runge-Kutta法と比較すると、誤差が大きかったり、
数値的に不安定になりやすいことも知られている。

```python
def AB4(x0,m=4):
    xsAB = [x0]
    x = x0
    fs = [ 0.0 for i in range(m)]
    for i,t in enumerate(tr):
        if len(xsAB) < m : #RK4
            f1 = f(x,t)
            f2 = f(x+0.5*h*f1, t+0.5*h)
            f3 = f(x+0.5*h*f2, t+0.5*h)
            f4 = f(x+h*f3, t+h)
            x += h*(f1 + 2*f2+ 2*f3 + f4)/6.0
            fs[0] = f1; fs[1] = f2; fs[2] = f3; fs[3] = f4
        else:
            f1,f2,f3 = fs[1:]
            f4 = f(x,t)
            x += h *(-9*f1 +37*f2 -59*f3 +55*f4 )/24.0
            fs[0] = f1; fs[1] = f2; fs[2] = f3; fs[3] = f4
        xsAB += [x]
    return xsAB

xsAB4 = AB4(x0)
#描画
fig = plt.figure(figsize=(10,3))
plt.xlabel("t");plt.ylabel("abs. diff.")
plt.yscale("log")
plt.plot(tr,abs(xe-np.array(xs[:-1])),label="Euler")
plt.plot(tr,abs(xe-np.array(xsRK4[:-1])),label="RK4")
plt.plot(tr,abs(xe-np.array(xsAB4[:-1])),label="AB4")
plt.legend()
plt.show()
plt.close()
```

### Adams-Moulton法

AB法は、現在$i$と過去のステップの情報のみを使用したが、
$i+1$ステップの情報を使う(陰解法)ことで、より高精度の結果を得ることができる。

3次のAB法($i-2,i-1,i$の情報)に加えて$i+1$番目の情報を用いて
3次のLagrange補間多項式$P_3(t)=\sum^2_{j=-1}f_{i-j}L_{i-j}$のもと
$x_{i+1}=x_i + \sum^2_{j=-2} f_{i+j}  \int^{t_{i+1}}_{t_i} L_{i+j}(t) dt $
という
更新式を考える。愚直に上と同様に各積分を評価すると
$x_{i+1}=x_i + \frac{h}{24}(f_{i-2}-5f_{i-1}+19f_{i} +9f_{i+1}) $
という式が得られる。

これは、４次精度の公式になっており、3ステップ(4次)のAdams-Moulton法と呼ばれる。
右辺にある$f_{i+1}$の評価に関する計算コストの分だけ、3次のAB法に比べて増えることになる。

### Adams-Bashforth-Moulton法

AM法とAB法を組み合わせることで、使いやすい高精度な陽解法を作ろう
というのが予測子修正子法の発想である。

4ステップのAB法を予測子, 4ステップのAM法を修正子として採用すると
以下の更新式が得られる:

$\begin{align}
x^{P}_{i+1} &= x_i + \frac{h}{24}(-9f_{i-3}+37f_{i-2}-59f_{i-1}+55f_i) \\
x_{i+1} &= x_i + \frac{h}{24}(f_{i-2}-5f_{i-1}+19f_i + 9f(t_{i+1},x^P_{i+1}))
\end{align}
$

```python
def ABM(x0,m=4):
    xsABM = [x0]
    x = x0
    fs = [ 0.0 for i in range(m)]
    for i,t in enumerate(tr):
        if len(xsABM) < m : #RK4
            f1 = f(x,t)
            f2 = f(x+0.5*h*f1, t+0.5*h)
            f3 = f(x+0.5*h*f2, t+0.5*h)
            f4 = f(x+h*f3, t+h)
            x += h*(f1 + 2*f2+ 2*f3 + f4)/6.0
            fs[0] = f1; fs[1] = f2; fs[2] = f3; fs[3] = f4
        else:
            f1,f2,f3 = fs[1:]; f4 = f(x,t)
            ## 予測子
            xp = x + h *(-9*f1+37*f2-59*f3+55*f4) / 24.0
            f5 = f(xp,t+h)
            ## 修正子
            x += h *(f2 -5*f3 +19*f4 +9*f5 )/24.0
            fs[0] = f1; fs[1] = f2; fs[2] = f3; fs[3] = f4
        xsABM += [x]
    return xsABM

xsABM = ABM(x0)
fig = plt.figure(figsize=(10,3))
plt.xlabel("t");plt.ylabel("abs. diff.");plt.yscale("log")
plt.plot(tr,abs(xe-np.array(xs[:-1])),label="Euler")
plt.plot(tr,abs(xe-np.array(xsRK4[:-1])),label="RK4")
plt.plot(tr,abs(xe-np.array(xsAB4[:-1])),label="AB4")
plt.plot(tr,abs(xe-xsABM[:-1]),label="ABM4")
plt.legend();plt.show();plt.close()
```

1桁程度、AB法よりも精度が改善していることがわかる。