# Creating new model from scratch

The real power of `MultiModel` framework is in fast prototyping of heterogeneous models. To showcase this, in this notebook we create a brand new model in the framework (the famous Jansen-Rit model), and then create a thalamocortical mini network with 1 node representing a thalamus and 1 node Jansen-Rit model representing a cortical column.

## Jansen-Rit model
The Jansen-Rit model is a neural population model of a local cortical circuit. It contains three interconnected neural populations: one for the pyramidal projection neurons and two for excitatory and inhibitory interneurons forming feedback loops.

The equations for Jansen-Rit model reads:
\begin{align}
\ddot{x}_{0} & = Aa\cdot\mathrm{Sigm}\left(x_{1} - x_{2}\right) - 2a\dot{x}_{0} - a^{2}x_{0} \\
\ddot{x}_{1} & = Aa[p + C_{2}\mathrm{Sigm}\left(C_{1}x_{0}\right)] - 2a\dot{x}_{1} - a^2x_{1} \\
\ddot{x}_{2} & = BbC_{4}\mathrm{Sigm}\left( C_{3}x_{o} \right) - 2b\dot{x}_{2} - b^2x_{2} \\
\mathrm{Sigm}(x) & = \frac{v_{max}}{1 + \mathrm{e}^{r(v_{0} - x)}}
\end{align}
Of course, in order to implement the above equations numerically, the system of three second-order ODEs will be rewritten into a system of six first-order ODEs.

## `MultiModel` strategy
The actual implementation will be a bit more involved than simply writing down the above equations. The building block of any proper `MultiModel` is a `NeuralMass`. Jansen-Rit model actually summarises an activity of a cortical column consisting of three populations: a population of pyramidal cells interacting with two populations of interneurons - one excitatory and one inhibitory. Moreover, the $x$ represent the average membrane potential, but typically, neuronal models (at least in `neurolib`) are coupled via firing rate. For this reason, our main output variable would actually be a firing rate of a main, pyramidal population. The average membrane potential of a pyramidal population is $x = x_{1} - x_{2}$ and its firing rate is then $r = \mathrm{Sigm}(x) = \mathrm{Sigm}(x_{1} - x_{2})$. Similar strategy (sigmoidal transfer function for average membrane potential) is used for the thalamic model.

The coupling variable in `MultiModel` must be the same across all hierarchical levels. Individial populations in Jansen-Rit model are coupled via their average membrane potentials $x_{i}$, $i\in[0, 1, 2]$. However, the "global" coupling variable for the node would be the firing rate of pyramidal population $r$, introduced in the paragraph above. To reconcile this, two options exists in `MultiModel`:
* create a `NeuralMass` representing a pyramidal population with two coupling variables: $x_{0}$ and $r$; `NeuralMass` representing interneurons would have one coupling variable, $x_{1,2}$
    * advantages: cleaner (implementation), more modular (can create J-R `Node` with more masses than 3 very easily)
    * disadvantages: more code, might be harder to navigate for the beginner
* create one `NeuralMass` representing all three populations with one coupling variable $r$, since the others are not really coupling since the whole dynamics is contained in one object
    * advantages: less code, easier to grasp
    * disadvantages: less modular, cannot simply edit J-R model, since everything is "hardcoded" into one `NeuralMass` object

In order to build a basic understanding of building blocks, we will follow the second option here (less code, easier to grasp). For interested readers, the first option (modular one) will be implemented in `MultiModel`, so you can follow the model files.

Our strategy for this notebook then would be:
1. implement single `NeuralMass` object representing all three populations in the Jansen-Rit model, with single coupling variable $r$
2. implement a "dummy" `Node` with single `NeuralMass` (requirement, one cannot couple `Node` to a `NeuralMass` to build a network)
3. experiment with connecting this Jansen-Rit cortical model to the thalamic population model

Last note, all model's dynamics and parameters in `MultiModel` are defined in milliseconds, therefore we will scale the default parameters.

Let us start with the imports:

```python
import matplotlib.pyplot as plt
import numpy as np
import symengine as se
from IPython.display import display
from jitcdde import input as system_input
from neurolib.models.multimodel import MultiModel, ThalamicNode
from neurolib.models.multimodel.builder.base.constants import LAMBDA_SPEED
from neurolib.models.multimodel.builder.base.network import Network, Node
from neurolib.models.multimodel.builder.base.neural_mass import NeuralMass
from neurolib.utils.functions import getPowerSpectrum
from neurolib.utils.stimulus import Input, OrnsteinUhlenbeckProcess, StepInput
```

A quick detour before we dive into the model itself. Jansen-Rit model is typically driven with a uniformly distributed noise, as the authors wanted to model nonspecific input (they used the term *background spontaneous activity*). For this we quickly create our model input using the `ModelInput` class (the tutorial on how to use stimuli in `neurolib` is given elsewhere).

```python
class UniformlyDistributedNoise(Input):
    """
    Uniformly distributed noise process between two values.
    """

    def __init__(self, low, high, n=1, seed=None):
        # save arguments as attributes for later
        self.low = low
        self.high = high
        # init super
        super().__init__(n=n, seed=seed)

    def generate_input(self, duration, dt):
        # generate time vector
        self._get_times(duration=duration, dt=dt)
        # generate noise process itself with the correct shape
        # as (time steps x num. processes)
        return np.random.uniform(
            self.low, self.high, (self.n, self.times.shape[0])
        )
```

```python
# let us build a proper hierarchy, i.e. we firstly build a Jansen-Rit mass
class SingleJansenRitMass(NeuralMass):
    """
    Single Jansen-Rit mass implementing whole three population dynamics.

    Reference:
        Jansen, B. H., & Rit, V. G. (1995). Electroencephalogram and visual evoked potential
        generation in a mathematical model of coupled cortical columns. Biological cybernetics,
        73(4), 357-366.
    """

    # all these attributes are compulsory to fill in
    name = "Jansen-Rit mass"
    label = "JRmass"

    num_state_variables = 7  # 6 ODEs + firing rate coupling variable
    num_noise_variables = 1  # single external input
    # NOTE
    # external inputs (so-called noise_variables) are typically background noise drive in models,
    # however, this can be any type of stimulus - periodic stimulus, step stimulus, square pulse,
    # anything. Therefore you may want to add more stimuli, e.g. for Jansen-Rit model three to each
    # of its population. Here we do not stimulate our Jansen-Rit model, so only use actual noise
    # drive to excitatory interneuron population.
    # as dictionary {index of state var: it's name}
    coupling_variables = {6: "r_mean_EXC"}
    # as list
    state_variable_names = [
        "v_pyr",
        "dv_pyr",
        "v_exc",
        "dv_exc",
        "v_inh",
        "dv_inh",
        # to comply with other `MultiModel` nodes
        "r_mean_EXC",
    ]
    # as list
    # note on parameters C1 - C4 - all papers use one C and C1-C4 are
    # defined as various rations of C, typically: C1 = C, C2 = 0.8*C
    # C3 = C4 = 0.25*C, therefore we use only `C` and scale it in the
    # dynamics definition
    required_params = [
        "A",
        "a",
        "B",
        "b",
        "C",
        "v_max",
        "v0",
        "r",
        "lambda",
    ]
    # list of required couplings when part of a `Node` or `Network`
    # `network_exc_exc` is the default excitatory coupling between nodes
    required_couplings = ["network_exc_exc"]
    # here we define the default noise input to Jansen-Rit model (this can be changed later)
    # for a quick test, we follow the original Jansen and Rit paper and use uniformly distributed
    # noise between 120 - 320 Hz; but we do it in kHz, hence 0.12 - 0.32
    # fix seed for reproducibility
    _noise_input = [UniformlyDistributedNoise(low=0.12, high=0.32, seed=42)]

    def _sigmoid(self, x):
        """
        Sigmoidal transfer function which is the same for all populations.
        """
        # notes:
        # - all parameters are accessible as self.params - it is a dictionary
        # - mathematical definition (ODEs) is done in symbolic mathematics - all functions have to be
        #   imported from `symengine` module, hence se.exp which is a symbolic exponential function
        return self.params["v_max"] / (
            1.0 + se.exp(self.params["r"] * (self.params["v0"] - x))
        )

    def __init__(self, params=None, seed=None):
        # init this `NeuralMass` - use passed parameters or default ones
        # parameters are now accessible as self.params, seed as self.seed
        super().__init__(params=params or JR_DEFAULT_PARAMS, seed=seed)

    def _initialize_state_vector(self):
        """
        Initialize state vector.
        """
        np.random.seed(self.seed)
        # random in average potentials around zero
        self.initial_state = (
            np.random.normal(size=self.num_state_variables)
            # * np.array([10.0, 0.0, 10.0, 0.0, 10.0, 0.0, 0.0])
        ).tolist()

    def _derivatives(self, coupling_variables):
        """
        Here the magic happens: dynamics is defined here using symbolic maths package symengine.
        """
        # first, we need to unwrap state vector
        (
            v_pyr,
            dv_pyr,
            v_exc,
            dv_exc,
            v_inh,
            dv_inh,
            firing_rate,
        ) = self._unwrap_state_vector()  # this function does everything for us
        # now we need to write down our dynamics
        # PYR dynamics
        d_v_pyr = dv_pyr
        d_dv_pyr = (
            self.params["A"] * self.params["a"] * self._sigmoid(v_exc - v_inh)
            - 2 * self.params["a"] * dv_pyr
            - self.params["a"] ** 2 * v_pyr
        )
        # EXC dynamics: system input comes into play here
        d_v_exc = dv_exc
        d_dv_exc = (
            self.params["A"]
            * self.params["a"]
            * (
                # system input as function from jitcdde (also in symengine) with proper index:
                # in our case we have only one noise input (can be more), so index 0
                system_input(self.noise_input_idx[0])
                # C2 = 0.8*C, C1 = C
                + (0.8 * self.params["C"]) * self._sigmoid(self.params["C"] * v_pyr)
            )
            - 2 * self.params["a"] * dv_exc
            - self.params["a"] ** 2 * v_exc
        )
        # INH dynamics
        d_v_inh = dv_inh
        d_dv_inh = (
            self.params["B"] * self.params["b"]
            # C3 = C4 = 0.25 * C
            * (0.25 * self.params["C"])
            * self._sigmoid((0.25 * self.params["C"]) * v_pyr)
            - 2 * self.params["b"] * dv_inh
            - self.params["b"] ** 2 * v_inh
        )
        # firing rate computation
        # firing rate as dummy dynamical variable with infinitely fast
        # fixed point dynamics
        firing_rate_now = self._sigmoid(v_exc - v_inh)
        d_firing_rate = -self.params["lambda"] * (firing_rate - firing_rate_now)

        # now just return a list of derivatives in the correct order
        return [d_v_pyr, d_dv_pyr, d_v_exc, d_dv_exc, d_v_inh, d_dv_inh, d_firing_rate]
```

And we are done with the basics! Only thing we really need is to define attributes (such as how many variables we have, what couplings we have, what about noise, etc.) and the actual dynamics as symbolic expressions. Symbolic expressions are easy: are basic operators like `+`, `-`, `*`, `/`, or `**` are overloaded, which means you can simply use them and do not think about. Functions such as `sin`, `log`, or `exp` must be imported from `symengine` and used. Now we define a default set of parameters. Do not forget - `MultiModel` defines all in ms, therefore the parameters needs to be in ms, kHz, and similar.

```python
JR_DEFAULT_PARAMS = {
    "A": 3.25,  # mV
    "B": 22.0,  # mV
    # `a` and `b` are originally 100Hz and 50Hz
    "a": 0.1,  # kHz
    "b": 0.05,  # kHz
    "v0": 6.0,  # mV
    # v_max is originally 5Hz
    "v_max": 0.005,  # kHz
    "r": 0.56,  # m/V
    "C": 135.0,
    # parameter for dummy `r` dynamics
    "lambda": LAMBDA_SPEED,
}
```

The next step is to create a `Node`. `Node` is second level in the hierarchy and already it can be wrapped into `MultiModel` and treated as any other `neurolib` model. On our case, creating the `Node` is really simple: it has only one mass, no delays, and no connectivity.

```python
class JansenRitNode(Node):
    """
    Jansen-Rit node with 1 neural mass representing 3 population model.
    """

    name = "Jansen-Rit node"
    label = "JRnode"

    # if Node is integrated isolated, what network input we should use
    # zero by default = no network input for one-node model
    default_network_coupling = {"network_exc_exc": 0.0}

    # default output is the firing rate of pyramidal population
    default_output = "r_mean_EXC"

    # list of all variables that are accessible as outputs
    output_vars = ["r_mean_EXC", "v_pyr", "v_exc", "v_inh"]

    def __init__(self, params=None, seed=None):
        # in `Node` __init__, the list of masses is created and passed
        jr_mass = SingleJansenRitMass(params=params, seed=seed)
        # each mass has to have index, in this case it is simply 0
        jr_mass.index = 0
        # call super and properly initialize a Node
        super().__init__(neural_masses=[jr_mass])

        self.excitatory_masses = np.array([0])

    def _sync(self):
        # this function typically defines the coupling between masses
        # within one node, but in our case there is nothing to define
        return []
```

And we are done. At this point, we can integrate our Jansen-Rit model and see some results.

## Test and simulate Jansen-Rit model
In order to simulate our newly created model, we just need to wrap it with `MultiModel` as in the last example and see how it goes.

```python
# init model - fix seed for reproducibility (random init. conditions)
jr_model = MultiModel.init_node(JansenRitNode(seed=42))

# see parameters
print("Parameters:")
display(jr_model.params)
print("")

# see describe
print("Describe:")
display(jr_model.model_instance.describe())
```

    Parameters:

    {'JRnode_0.JRmass_0.A': 3.25,
     'JRnode_0.JRmass_0.B': 22.0,
     'JRnode_0.JRmass_0.a': 0.1,
     'JRnode_0.JRmass_0.b': 0.05,
     'JRnode_0.JRmass_0.v0': 6.0,
     'JRnode_0.JRmass_0.v_max': 0.005,
     'JRnode_0.JRmass_0.r': 0.56,
     'JRnode_0.JRmass_0.C': 135.0,
     'JRnode_0.JRmass_0.lambda': 10.0,
     'JRnode_0.JRmass_0.input_0.type': 'UniformlyDistributedNoise',
     'JRnode_0.JRmass_0.input_0.low': 0.12,
     'JRnode_0.JRmass_0.input_0.high': 0.32,
     'JRnode_0.JRmass_0.input_0.n': 1,
     'JRnode_0.JRmass_0.input_0.seed': 42,
     'duration': 2000,
     'dt': 0.1,
     'seed': None,
     'backend': 'jitcdde',
     'name': 'JRnode',
     'description': 'Jansen-Rit node',
     'N': 1,
     'Cmat': array([[0.]]),
     'sampling_dt': 0.1}

    Describe:

    {'index': 0,
     'name': 'Jansen-Rit node',
     'num_masses': 1,
     'num_num_state_variables': 7,
     'num_noise_variables': 1,
     'masses': [{'index': 0,
       'name': 'Jansen-Rit mass',
       'mass_type': None,
       'num_state_variables': 7,
       'num_noise_variables': 1,
       'state_variable_names': ['v_pyr',
        'dv_pyr',
        'v_exc',
        'dv_exc',
        'v_inh',
        'dv_inh',
        'r_mean_EXC'],
       'params': {'A': 3.25,
        'B': 22.0,
        'a': 0.1,
        'b': 0.05,
        'v0': 6.0,
        'v_max': 0.005,
        'r': 0.56,
        'C': 135.0,
        'lambda': 10.0,
        'input_0': {'type': 'UniformlyDistributedNoise',
         'low': 0.12,
         'high': 0.32,
         'n': 1,
         'seed': 42}}}]}

```python
# run model for 5 seconds - all in ms
jr_model.params["sampling_dt"] = 1.0
jr_model.params["duration"] = 5000
jr_model.params["backend"] = "jitcdde"
jr_model.run()
```

    /Users/nikola/.virtualenvs/neurolib/lib/python3.7/site-packages/jitcdde/_jitcdde.py:1491: UserWarning: Your input past does not begin at t=0 but at t=1.0. Values before the beginning of the past will be extrapolated. You very likely do not want this.
      warn(f"Your input past does not begin at t=0 but at t={input[0].time}. Values before the beginning of the past will be extrapolated. You very likely do not want this.")

    Using default integration parameters.

    100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5000/5000 [00:00<00:00, 26901.12it/s]

```python
_, axs = plt.subplots(
    nrows=2, ncols=2, figsize=(12, 6), gridspec_kw={"width_ratios": [2, 1]}
)

axs[0, 0].plot(jr_model.t, jr_model.v_exc.T - jr_model.v_inh.T, color="k")
axs[0, 0].set_ylabel("LFP")
axs[1, 0].plot(jr_model.t, jr_model.r_mean_EXC.T * 1000, color="navy")
axs[1, 0].set_ylabel("PYR r [Hz]")
axs[1, 0].set_xlabel("time [sec]")

fr, po = getPowerSpectrum(
    (jr_model.v_exc.T - jr_model.v_inh.T).squeeze(),
    dt=jr_model.params["sampling_dt"],
    maxfr=40.0,
)
axs[0, 1].semilogy(fr, po, color="indianred")
axs[0, 1].set_xlabel("frequency [Hz]")
axs[0, 1].set_ylabel("log power")

axs[1, 1].hist(jr_model.v_exc.T - jr_model.v_inh.T, bins=15, color="gray")
axs[1, 1].set_xlabel("LFP")
axs[1, 1].set_ylabel("#")
plt.suptitle("`jitcdde` backend")
```

The results looks good! With the same parameters as original Jansen and Rit paper, we got the $\alpha$ activity with spectral peak around 10 Hz. Just as a proof of concept - let us try the second `MultiModel` backend.

```python
# run model for 5 seconds - all in ms
jr_model.params["sampling_dt"] = 1.0
jr_model.params["dt"] = 0.1
jr_model.params["duration"] = 5000
jr_model.params["backend"] = "numba"
jr_model.run()
```

```python
_, axs = plt.subplots(
    nrows=2, ncols=2, figsize=(12, 6), gridspec_kw={"width_ratios": [2, 1]}
)

axs[0, 0].plot(jr_model.t, jr_model.v_exc.T - jr_model.v_inh.T, color="k")
axs[0, 0].set_ylabel("LFP")
axs[1, 0].plot(jr_model.t, jr_model.r_mean_EXC.T * 1000, color="navy")
axs[1, 0].set_ylabel("PYR r [Hz]")
axs[1, 0].set_xlabel("time [sec]")

fr, po = getPowerSpectrum(
    (jr_model.v_exc.T - jr_model.v_inh.T).squeeze(),
    dt=jr_model.params["sampling_dt"],
    maxfr=40.0,
)
axs[0, 1].semilogy(fr, po, color="indianred")
axs[0, 1].set_xlabel("frequency [Hz]")
axs[0, 1].set_ylabel("log power")

axs[1, 1].hist(jr_model.v_exc.T - jr_model.v_inh.T, bins=15, color="gray")
axs[1, 1].set_xlabel("LFP")
axs[1, 1].set_ylabel("#")
plt.suptitle("`numba` backend")
```

All works as it should, we have our Jansen-Rit model! In the next step, we will showcase how the new model can be connected and coupled to other models (similar as in the first example).

## Couple Jansen-Rit to thalamic model
Here we practically copy the previous example where we coupled ALN node with a thalamic node, but instead of ALN representing a cortical column, we use our brand new Jansen-Rit model.

```python
# let us start by subclassing the Network

class JansenRitThalamusMiniNetwork(Network):
    """
    Simple thalamocortical motif: 1 cortical node Jansen-Rit + 1 NMM thalamus.
    """

    # provide basic attributes as name and label
    name = "Jansen-Rit 1 node + Thalamus"
    label = "JRThlmNet"

    # define which variables are used to sync, i.e. what coupling variables our nodes need
    sync_variables = [
        # both nodes are connected via excitatory synapses
        "network_exc_exc",
        # and INH mass in thalamus also receives excitatory coupling
        "network_inh_exc",
    ]

    # lastly, we need to define what is default output of the network (this has to be the
    #     variable present in all nodes)
    # for us it is excitatory firing rates
    default_output = f"r_mean_EXC"
    # define all output vars of any interest to us - EXC and INH firing rates
    output_vars = [f"r_mean_EXC", f"r_mean_INH"]

    def __init__(self, connectivity_matrix, delay_matrix, seed=None):
        # self connections are resolved within nodes, so zeroes at the diagonal
        assert np.all(np.diag(connectivity_matrix) == 0.0)

        # init Jansen-Rit node with index 0
        jr_node = JansenRitNode(seed=seed)
        jr_node.index = 0
        # index where the state variables start - for first node it is always 0
        jr_node.idx_state_var = 0
        # set correct indices for noise input - in JR we have only one noise source
        jr_node[0].noise_input_idx = [0]

        # init thalamus node with index 1
        thalamus = ThalamicNode()
        thalamus.index = 1
        # thalamic state variables start where ALN state variables end - easy
        thalamus.idx_state_var = jr_node.num_state_variables
        # set correct indices of noise input - one per mass, after ALN noise
        # indices
        for mass in thalamus:
            mass.noise_input_idx = [jr_node.num_noise_variables + mass.index]

        # now super.__init__ network with these two nodes:
        super().__init__(
            nodes=[jr_node, thalamus],
            connectivity_matrix=connectivity_matrix,
            delay_matrix=delay_matrix,
        )

    # done! the only other thing we need to do, is to set the coupling variables
    # thalamus vs. Jansen-Rit are coupled via their firing rates and here we setup the
    # coupling matrices; the super class `Network` comes with some convenient
    # functions for this

    def _sync(self):
        """
        Set coupling variables - the ones we defined in `sync_variables`
        _sync returns a list of tuples where the first element in each tuple is the coupling "symbol"
        and the second is the actual mathematical expression
        for the ease of doing this, `Network` class contains convenience functions for this:
            - _additive_coupling
            - _diffusive_coupling
            - _no_coupling
        here we use additive coupling only
        """
        # get indices of coupling variables from all nodes
        exc_indices = [
            next(
                iter(
                    node.all_couplings(
                        mass_indices=node.excitatory_masses.tolist()
                    )
                )
            )
            for node in self
        ]
        assert len(exc_indices) == len(self)
        return (
            # basic EXC <-> EXC coupling
            # within_node_idx is a list of len 2 (because we have two nodes)
            # with indices of coupling variables within the respective state vectors
            self._additive_coupling(
                within_node_idx=exc_indices, symbol="network_exc_exc"
            )
            # EXC -> INH coupling (only to thalamus)
            + self._additive_coupling(
                within_node_idx=exc_indices,
                symbol="network_inh_exc",
                connectivity=self.connectivity,
            )
            + super()._sync()
        )
```

```python
# lets check what we have
# in the ALN-thalamus case the matrix was [0.0, 0.15], [1.2, 0.0] - JR produces firing rates around 5Hz (5 times lower than ALN)
SC = np.array([[0.0, 0.15], [6., 0.0]])
delays = np.array([[0.0, 13.0], [13.0, 0.0]]) # thalamocortical delay = 13ms
thalamocortical = MultiModel(JansenRitThalamusMiniNetwork(connectivity_matrix=SC, delay_matrix=delays, seed=42))
# original `MultiModel` instance is always accessible as `MultiModel.model_instance`
display(thalamocortical.model_instance.describe())
```

    {'name': 'Jansen-Rit 1 node + Thalamus',
     'num_nodes': 2,
     'num_state_variables': 24,
     'num_noise_variables': 3,
     'nodes': [{'index': 0,
       'name': 'Jansen-Rit node',
       'num_masses': 1,
       'num_num_state_variables': 7,
       'num_noise_variables': 1,
       'masses': [{'index': 0,
         'name': 'Jansen-Rit mass',
         'mass_type': None,
         'num_state_variables': 7,
         'num_noise_variables': 1,
         'state_variable_names': ['v_pyr',
          'dv_pyr',
          'v_exc',
          'dv_exc',
          'v_inh',
          'dv_inh',
          'r_mean_EXC'],
         'params': {'A': 3.25,
          'B': 22.0,
          'a': 0.1,
          'b': 0.05,
          'v0': 6.0,
          'v_max': 0.005,
          'r': 0.56,
          'C': 135.0,
          'lambda': 10.0,
          'input_0': {'type': 'UniformlyDistributedNoise',
           'low': 0.12,
           'high': 0.32,
           'n': 1,
           'seed': 42}}}]},
      {'index': 1,
       'name': 'Thalamic mass model node',
       'num_masses': 2,
       'num_num_state_variables': 17,
       'num_noise_variables': 2,
       'masses': [{'index': 0,
         'name': 'Thalamocortical relay mass',
         'mass_type': 'EXC',
         'num_state_variables': 10,
         'num_noise_variables': 1,
         'state_variable_names': ['V',
          'Ca',
          'h_T',
          'm_h1',
          'm_h2',
          's_e',
          's_i',
          'ds_e',
          'ds_i',
          'r_mean'],
         'params': {'tau': 20.0,
          'Q_max': 0.4,
          'theta': -58.5,
          'sigma': 6.0,
          'C1': 1.8137993642,
          'C_m': 1.0,
          'gamma_e': 0.07,
          'gamma_r': 0.1,
          'g_L': 1.0,
          'g_GABA': 1.0,
          'g_AMPA': 1.0,
          'g_LK': 0.018,
          'g_T': 3.0,
          'g_h': 0.062,
          'E_AMPA': 0.0,
          'E_GABA': -70.0,
          'E_L': -70.0,
          'E_K': -100.0,
          'E_Ca': 120.0,
          'E_h': -40.0,
          'alpha_Ca': -5.18e-05,
          'tau_Ca': 10.0,
          'Ca_0': 0.00024,
          'k1': 25000000.0,
          'k2': 0.0004,
          'k3': 0.1,
          'k4': 0.001,
          'n_P': 4.0,
          'g_inc': 2.0,
          'ext_current': 0.0,
          'lambda': 10.0,
          'input_0': {'type': 'OrnsteinUhlenbeckProcess',
           'mu': 0.0,
           'sigma': 0.0,
           'tau': 5.0,
           'n': 1,
           'seed': None}}},
        {'index': 1,
         'name': 'Thalamic reticular nuclei mass',
         'mass_type': 'INH',
         'num_state_variables': 7,
         'num_noise_variables': 1,
         'state_variable_names': ['V',
          'h_T',
          's_e',
          's_i',
          'ds_e',
          'ds_i',
          'r_mean'],
         'params': {'tau': 20.0,
          'Q_max': 0.4,
          'theta': -58.5,
          'sigma': 6.0,
          'C1': 1.8137993642,
          'C_m': 1.0,
          'gamma_e': 0.07,
          'gamma_r': 0.1,
          'g_L': 1.0,
          'g_GABA': 1.0,
          'g_AMPA': 1.0,
          'g_LK': 0.018,
          'g_T': 2.3,
          'E_AMPA': 0.0,
          'E_GABA': -70.0,
          'E_L': -70.0,
          'E_K': -100.0,
          'E_Ca': 120.0,
          'ext_current': 0.0,
          'lambda': 10.0,
          'input_0': {'type': 'ZeroInput', 'n': 1, 'seed': None}}}],
       'local_connectivity': array([[ 0.,  5.],
              [ 3., 25.]]),
       'local_delays': array([[0., 0.],
              [0., 0.]])}],
     'connectivity': array([[0.  , 0.15],
            [6.  , 0.  ]]),
     'delays': array([[ 0., 13.],
            [13.,  0.]])}

```python
# fix parameters for interesting regime
thalamocortical.params["*g_LK"] = 0.032 # K-leak conductance in thalamus
thalamocortical.params["*TCR*input*sigma"] = 0.005 # noise in thalamus
thalamocortical.params["*input*tau"] = 5.0 # timescale of OU process
thalamocortical.params["duration"] = 20000. # 20 seconds simulation
thalamocortical.params["sampling_dt"] = 1.0
thalamocortical.run()
```

    /Users/nikola/.virtualenvs/neurolib/lib/python3.7/site-packages/jitcdde/_jitcdde.py:1491: UserWarning: Your input past does not begin at t=0 but at t=1.0. Values before the beginning of the past will be extrapolated. You very likely do not want this.
      warn(f"Your input past does not begin at t=0 but at t={input[0].time}. Values before the beginning of the past will be extrapolated. You very likely do not want this.")

    Using default integration parameters.

    100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [00:01<00:00, 18832.12it/s]

```python
_, axs = plt.subplots(nrows=2, ncols=1, sharex=True, figsize=(12, 6))
axs[0].plot(thalamocortical.t, thalamocortical.r_mean_EXC[0, :].T)
axs[0].set_ylabel("Jansen-Rit firing rate [kHz]")
axs[1].plot(thalamocortical.t, thalamocortical.r_mean_EXC[1, :].T, color="C1")
axs[1].set_ylabel("thalamus firing rate [kHz]")
axs[1].set_xlabel("time [sec]")
```

So the model works, just like that! Of course, in this case we do not see anything interesting in the modelled dynamics, since we would need to proper investigation of various parameters. Consider this as a proof-of-concept, where we can very easily couple two very different population models.


#  CS-109B Introduction to Data Science
## Lab 3: Optimization in Artificial Neural Networks

**Harvard University**

**Spring 2019**

**Lab instructor** Eleni Kaxiras

**Instructors:** Pavlos Protopapas and Mark Glickman

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```

## Learning Goals
In this lab, we'll explore ways to optimize the loss function of a Multilayer Learning Perceptor (MLP) by tuning the model hyperparameters. We'll also explore the use of cross-validation as a technique for checking potential values for these hyperparameters.

By the end of this lab, you should:
- Be familiar with the use of `sklearn`'s `optimize` function.
- Be able to identify the hyperparameters that go into the training of a MLP.
- Be familiar with the implementation in `keras` of various optimization techniques.
- Apply cross-validation to check for multiple values of hyperparameters.

```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.optimize import minimize

%matplotlib inline
```

## Part 1:  Beale's function
### First let's look at function optimization in `scipy.optimize`, using Beale's function as an example

Optimizing a function $f: A\rightarrow R$, from some set A to the real numbers is finding an element $x_0\,\epsilon\, A$ such that $f(x_0)\leq f(x)$ for all $x\,\epsilon\, A$ (finding the minimum) or such that $f(x_0)\geq f(x)$ for all $x\,\epsilon\, A$ (finding the maximum).

To illustrate our point we will use a function of two parameters.  Our goal is to optimize over these 2 parameters. We can extend to higher dimensions by plotting pairs of parameters against each other.

The Wikipedia article on Test functions for optimization has a few functions that are useful for evaluating optimization algorithms. Here is Beale's function:

$f(x,y)$ = $(1.5−x+xy)^2+(2.25−x+xy^2)^2+(2.625−x+xy^3)^2$

\[source:[Wikipedia](https://en.wikipedia.org/wiki/Test_functions_for_optimization)\]

We already know that this function has a minimum at [3.0, 0.5]. Let's see if `scipy` will find it.

```python
# define Beale's function which we want to minimize
def objective(X):
    x = X[0]; y = X[1]
    return (1.5 - x + x*y)**2 + (2.25 - x + x*y**2)**2 + (2.625 - x + x*y**3)**2
```

```python
# function boundaries
xmin, xmax, xstep = -4.5, 4.5, .9
ymin, ymax, ystep = -4.5, 4.5, .9
```

```python
# Let's create some points
x1, y1 = np.meshgrid(np.arange(xmin, xmax + xstep, xstep), np.arange(ymin, ymax + ystep, ystep))
```

Let's make an initial guess

```python
# initial guess
x0 = [4., 4.]
f0 = objective(x0)
print (f0)
```

    68891.203125

```python
bnds = ((xmin, xmax), (ymin, ymax))
minimum = minimize(objective, x0, bounds=bnds)
```

```python
print(minimum)
```

          fun: 2.068025638865627e-12
     hess_inv: <2x2 LbfgsInvHessProduct with dtype=float64>
          jac: array([-1.55969780e-06,  9.89837957e-06])
      message: b'CONVERGENCE: NORM_OF_PROJECTED_GRADIENT_<=_PGTOL'
         nfev: 60
          nit: 14
       status: 0
      success: True
            x: array([3.00000257, 0.50000085])

```python
real_min = [3.0, 0.5]
print (f'The answer, {minimum.x}, is very close to the optimum as we know it, which is {real_min}')
print (f'The value of the objective for {real_min} is {objective(real_min)}')
```

    The answer, [3.00000257 0.50000085], is very close to the optimum as we know it, which is [3.0, 0.5]
    The value of the objective for [3.0, 0.5] is 0.0

## Part 2: Optimization in neural networks

In general: **Learning Representation --> Objective function --> Optimization algorithm**

A neural network can be defined as a framework that combines inputs and tries to guess the output. If we are lucky enough to have some results, called "the ground truth", to compare the outputs produced by the network, we can calculate the **error**. So the network guesses, calculates some error function, guesses again, trying to minimize this error, guesses again, until the error does not go down any more. This is optimization.

In neural networks the most common used optimization algorithms, are flavors of **GD (gradient descent)**. The *objective function* used in gradient descent is the *loss function* which we want to minimize .

### A `keras` Refresher

`Keras` is a Python library for deep learning that can run on top of both Theano or
TensorFlow, two powerful Python libraries for fast numerical computing created and released by Facebook and Google, respectevely.

Keras was developed to make developing deep learning models as fast and easy as
possible for research and practical applications. It runs on Python 2.7 or 3.5 and can seamlessly execute on GPUs and CPUs.

Keras is built on the idea of a model. At its core we have a sequence of layers called
the `Sequential` model which is a linear stack of layers. Keras also provides the `functional API`, a way to define complex models, such as multi-output models, directed acyclic graphs, or models with shared layers.

We can summarize the construction of deep learning models in Keras using the Sequential model as follows:
1. **Define your model**: create a `Sequential` model and add layers.
2. **Compile your model**: specify loss function and optimizers and call the `.compile()` function.
3. **Fit your model**: train the model on data by calling the `.fit()` function.
4. **Make predictions**: use the model to generate predictions on new data by calling functions such as `.evaluate()` or `.predict()`.

### Callbacks: taking a peek into our model while it's training

You can look at what is happening in various stages of your model by using `callbacks`. A callback is a set of functions to be applied at given stages of the training procedure. You can use callbacks to get a view on internal states and statistics of the model during training. You can pass a list of callbacks (as the keyword argument callbacks) to the `.fit()` method of the Sequential or Model classes. The relevant methods of the callbacks will then be called at each stage of the training.

- A callback function you are already familiar with is `keras.callbacks.History()`. This is automatically included in `.fit()`.

- Another very useful one is `keras.callbacks.ModelCheckpoint` which saves the model with its weights at a certain point in the training. This can prove useful if your model is running for a long time and a system failure happens. Not all is lost then. It's a good practice to save the model weights only when an improvement is observed as measured by the `acc`, for example.

- `keras.callbacks.EarlyStopping` stops the training when a monitored quantity has stopped improving.

- `keras.callbacks.LearningRateScheduler` will change the learning rate during training.

We will apply some callbacks later.

For full documentation on `callbacks` see https://keras.io/callbacks/

### What are the steps to optimizing our network?

```python
import tensorflow as tf
import keras
from keras import layers
from keras import models
from keras import utils
from keras.layers import Dense
from keras.models import Sequential
from keras.layers import Flatten
from keras.layers import Dropout
from keras.layers import Activation
from keras.regularizers import l2
from keras.optimizers import SGD
from keras.optimizers import RMSprop
from keras import datasets

from keras import losses
from sklearn.utils import shuffle

print(tf.VERSION)
print(tf.keras.__version__)

# fix random seed for reproducibility
np.random.seed(5)
```

    1.12.0
    2.1.6-tf

    Using TensorFlow backend.

### Step 1 -  Deciding on the network topology (not really considered optimization but is obviously very important)

We will use the MNIST dataset which consists of grayscale images of handwritten digits (0-9) whose dimension is 28x28 pixels. Each pixel is 8 bits so its value ranges from 0 to 255.

```python
#mnist = tf.keras.datasets.mnist
mnist = keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train.shape, y_train.shape
```

    ((60000, 28, 28), (60000,))

Each label is a number between 0 and 9

```python
print(y_train)
```

    [5 0 4 ... 5 6 8]

Let's look at some 10 of the images

```python
plt.figure(figsize=(10,10))
for i in range(10):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(x_train[i], cmap=plt.cm.binary)
    plt.xlabel(y_train[i])
```

```python
print(x_train[45].shape)
x_train[45, 15:20, 15:20]
```

    (28, 28)

    array([[ 11, 198, 231,  41,   0],
           [ 82, 252, 204,   0,   0],
           [253, 253, 141,   0,   0],
           [252, 220,  36,   0,   0],
           [252,  96,   0,   0,   0]], dtype=uint8)

```python
print(f'We have {x_train.shape[0]} train samples')
print(f'We have {x_test.shape[0]} test samples')
```

    We have 60000 train samples
    We have 10000 test samples

#### Preprocessing the data

To run our NN we need to pre-process the data

* First we need to make the 2D image arrays into 1D (flatten them). We can either perform this by using array reshaping with `numpy.reshape()` or the `keras`' method for this: a layer called `tf.keras.layers.Flatten` which transforms the format of the images from a 2d-array (of 28 by 28 pixels), to a 1D-array of 28 * 28 = 784 pixels.

* Then we need to normalize the pixel values (give them values between 0 and 1) using the following transformation:

\begin{align}
x := \dfrac{x - x_{min}}{x_{max} - x_{min}}
\textrm{}
\end{align}

In our case $x_{min} = 0$ and $x_{max} = 255$ so the formula becomes simply $x := {x}/255$

```python
# normalize the data
x_train, x_test = x_train / 255.0, x_test / 255.0
```

```python
# reshape the data into 1D vectors
x_train = x_train.reshape(60000, 784)
x_test = x_test.reshape(10000, 784)

num_classes = 10
```

```python
x_train.shape[1]
```

    784

Now let's prepare our class vector (y) to a binary class matrix, e.g. for use with categorical_crossentropy.

```python
# Convert class vectors to binary class matrices
y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
```

```python
y_train[0]
```

    array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)

Now we are ready to build the model!

### Step 2 - Adjusting the `learning rate`

One of the most common optimization algorithm is Stochastic Gradient Descent (SGD). The hyperparameters that can be optimized in SGD are `learning rate`, `momentum`, `decay` and `nesterov`.

`Learning rate` controls the weight at the end of each batch, and `momentum` controls how much to let the previous update influence the current weight update. `Decay` indicates the learning rate decay over each update, and `nesterov` takes the value True or False depending on if we want to apply Nesterov momentum. Typical values for those hyperparameters are lr=0.01, decay=1e-6, momentum=0.9, and nesterov=True.

The learning rate hyperparameter goes into the `optimizer` function which we will see below.

Let's implement a learning rate adaptation schedule in `Keras`. We'll start with SGD and a learning rate value of 0.1. We will then train the model for 40
epochs and set the decay argument to 0.002 (0.1/50). We also include a momentum value of 0.8 since that seems to work well when using an adaptive learning rate.

```python
epochs = 60
learning_rate = 0.1
decay_rate = learning_rate / epochs
momentum = 0.8

sgd = SGD(lr=learning_rate, momentum=momentum, decay=decay_rate, nesterov=False)
```

```python
# build the model
input_dim = x_train.shape[1]

lr_model = Sequential()
lr_model.add(Dense(64, activation=tf.nn.relu, kernel_initializer='uniform',
                input_dim = input_dim)) # fully-connected layer with 64 hidden units
lr_model.add(Dense(64, kernel_initializer='uniform', activation=tf.nn.relu))
lr_model.add(Dense(num_classes, kernel_initializer='uniform', activation=tf.nn.softmax))

# compile the model
lr_model.compile(loss='categorical_crossentropy',
              optimizer=sgd,
              metrics=['acc'])
```

```python
%%time
# Fit the model
batch_size = 28

lr_model_history = lr_model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/60
    60000/60000 [==============================] - 4s 73us/step - loss: 0.2859 - acc: 0.9069 - val_loss: 0.1374 - val_acc: 0.9579
    Epoch 2/60
    60000/60000 [==============================] - 4s 58us/step - loss: 0.1046 - acc: 0.9683 - val_loss: 0.1046 - val_acc: 0.9676
    Epoch 3/60
    60000/60000 [==============================] - 3s 58us/step - loss: 0.0780 - acc: 0.9767 - val_loss: 0.0928 - val_acc: 0.9723
    Epoch 4/60
    60000/60000 [==============================] - 3s 57us/step - loss: 0.0647 - acc: 0.9809 - val_loss: 0.0878 - val_acc: 0.9747
    Epoch 5/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0571 - acc: 0.9832 - val_loss: 0.0859 - val_acc: 0.9747
    Epoch 6/60
    60000/60000 [==============================] - 3s 53us/step - loss: 0.0515 - acc: 0.9849 - val_loss: 0.0840 - val_acc: 0.9756
    Epoch 7/60
    60000/60000 [==============================] - 3s 53us/step - loss: 0.0470 - acc: 0.9863 - val_loss: 0.0853 - val_acc: 0.9742
    Epoch 8/60
    60000/60000 [==============================] - 3s 57us/step - loss: 0.0437 - acc: 0.9873 - val_loss: 0.0858 - val_acc: 0.9740
    Epoch 9/60
    60000/60000 [==============================] - 3s 57us/step - loss: 0.0413 - acc: 0.9882 - val_loss: 0.0827 - val_acc: 0.9752
    Epoch 10/60
    60000/60000 [==============================] - 3s 58us/step - loss: 0.0389 - acc: 0.9888 - val_loss: 0.0833 - val_acc: 0.9750
    Epoch 11/60
    60000/60000 [==============================] - 3s 53us/step - loss: 0.0371 - acc: 0.9893 - val_loss: 0.0826 - val_acc: 0.9744
    Epoch 12/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0354 - acc: 0.9901 - val_loss: 0.0831 - val_acc: 0.9754
    Epoch 13/60
    60000/60000 [==============================] - 3s 53us/step - loss: 0.0341 - acc: 0.9904 - val_loss: 0.0824 - val_acc: 0.9755
    Epoch 14/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0328 - acc: 0.9909 - val_loss: 0.0822 - val_acc: 0.9762
    Epoch 15/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0318 - acc: 0.9915 - val_loss: 0.0821 - val_acc: 0.9761
    Epoch 16/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0307 - acc: 0.9918 - val_loss: 0.0823 - val_acc: 0.9756
    Epoch 17/60
    60000/60000 [==============================] - 3s 53us/step - loss: 0.0298 - acc: 0.9921 - val_loss: 0.0827 - val_acc: 0.9762
    Epoch 18/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0289 - acc: 0.9926 - val_loss: 0.0822 - val_acc: 0.9757
    Epoch 19/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0282 - acc: 0.9927 - val_loss: 0.0818 - val_acc: 0.9762
    Epoch 20/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0275 - acc: 0.9931 - val_loss: 0.0819 - val_acc: 0.9765
    Epoch 21/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0269 - acc: 0.9933 - val_loss: 0.0826 - val_acc: 0.9761
    Epoch 22/60
    60000/60000 [==============================] - 3s 53us/step - loss: 0.0263 - acc: 0.9933 - val_loss: 0.0825 - val_acc: 0.9764
    Epoch 23/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0257 - acc: 0.9935 - val_loss: 0.0826 - val_acc: 0.9764
    Epoch 24/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0252 - acc: 0.9937 - val_loss: 0.0826 - val_acc: 0.9758
    Epoch 25/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0247 - acc: 0.9939 - val_loss: 0.0828 - val_acc: 0.9765
    Epoch 26/60
    60000/60000 [==============================] - 3s 58us/step - loss: 0.0242 - acc: 0.9943 - val_loss: 0.0829 - val_acc: 0.9759
    Epoch 27/60
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0238 - acc: 0.9944 - val_loss: 0.0828 - val_acc: 0.9760
    Epoch 28/60
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0234 - acc: 0.9944 - val_loss: 0.0826 - val_acc: 0.9763
    Epoch 29/60
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0230 - acc: 0.9946 - val_loss: 0.0828 - val_acc: 0.9764
    Epoch 30/60
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0226 - acc: 0.9947 - val_loss: 0.0834 - val_acc: 0.9763
    Epoch 31/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0224 - acc: 0.9949 - val_loss: 0.0834 - val_acc: 0.9762
    Epoch 32/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0220 - acc: 0.9950 - val_loss: 0.0831 - val_acc: 0.9768
    Epoch 33/60
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0217 - acc: 0.9950 - val_loss: 0.0831 - val_acc: 0.9766
    Epoch 34/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0214 - acc: 0.9952 - val_loss: 0.0836 - val_acc: 0.9769
    Epoch 35/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0211 - acc: 0.9952 - val_loss: 0.0837 - val_acc: 0.9770
    Epoch 36/60
    60000/60000 [==============================] - 3s 58us/step - loss: 0.0209 - acc: 0.9956 - val_loss: 0.0837 - val_acc: 0.9766
    Epoch 37/60
    60000/60000 [==============================] - 3s 58us/step - loss: 0.0206 - acc: 0.9955 - val_loss: 0.0834 - val_acc: 0.9762
    Epoch 38/60
    60000/60000 [==============================] - 3s 57us/step - loss: 0.0203 - acc: 0.9955 - val_loss: 0.0835 - val_acc: 0.9768
    Epoch 39/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0201 - acc: 0.9957 - val_loss: 0.0838 - val_acc: 0.9769
    Epoch 40/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0199 - acc: 0.9957 - val_loss: 0.0838 - val_acc: 0.9766
    Epoch 41/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0197 - acc: 0.9956 - val_loss: 0.0837 - val_acc: 0.9768
    Epoch 42/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0194 - acc: 0.9959 - val_loss: 0.0841 - val_acc: 0.9768
    Epoch 43/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0192 - acc: 0.9960 - val_loss: 0.0842 - val_acc: 0.9767
    Epoch 44/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0190 - acc: 0.9961 - val_loss: 0.0839 - val_acc: 0.9771
    Epoch 45/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0189 - acc: 0.9961 - val_loss: 0.0839 - val_acc: 0.9765
    Epoch 46/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0186 - acc: 0.9961 - val_loss: 0.0843 - val_acc: 0.9764
    Epoch 47/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0185 - acc: 0.9961 - val_loss: 0.0842 - val_acc: 0.9771
    Epoch 48/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0183 - acc: 0.9963 - val_loss: 0.0842 - val_acc: 0.9772
    Epoch 49/60
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0181 - acc: 0.9963 - val_loss: 0.0844 - val_acc: 0.9770
    Epoch 50/60
    60000/60000 [==============================] - 3s 57us/step - loss: 0.0180 - acc: 0.9963 - val_loss: 0.0844 - val_acc: 0.9771
    Epoch 51/60
    60000/60000 [==============================] - 3s 57us/step - loss: 0.0178 - acc: 0.9966 - val_loss: 0.0844 - val_acc: 0.9768
    Epoch 52/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0177 - acc: 0.9965 - val_loss: 0.0845 - val_acc: 0.9766
    Epoch 53/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0175 - acc: 0.9965 - val_loss: 0.0847 - val_acc: 0.9769
    Epoch 54/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0173 - acc: 0.9966 - val_loss: 0.0848 - val_acc: 0.9772
    Epoch 55/60
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0172 - acc: 0.9966 - val_loss: 0.0850 - val_acc: 0.9771
    Epoch 56/60
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0171 - acc: 0.9967 - val_loss: 0.0853 - val_acc: 0.9770
    Epoch 57/60
    60000/60000 [==============================] - 3s 57us/step - loss: 0.0170 - acc: 0.9967 - val_loss: 0.0849 - val_acc: 0.9775
    Epoch 58/60
    60000/60000 [==============================] - 3s 57us/step - loss: 0.0168 - acc: 0.9967 - val_loss: 0.0851 - val_acc: 0.9774
    Epoch 59/60
    60000/60000 [==============================] - 3s 57us/step - loss: 0.0167 - acc: 0.9968 - val_loss: 0.0847 - val_acc: 0.9768
    Epoch 60/60
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0166 - acc: 0.9968 - val_loss: 0.0851 - val_acc: 0.9771
    CPU times: user 5min 48s, sys: 1min 23s, total: 7min 11s
    Wall time: 3min 20s

```python
fig, ax = plt.subplots(1, 1, figsize=(10,6))
ax.plot(np.sqrt(lr_model_history.history['loss']), 'r', label='train')
ax.plot(np.sqrt(lr_model_history.history['val_loss']), 'b' ,label='val')
ax.set_xlabel(r'Epoch', fontsize=20)
ax.set_ylabel(r'Loss', fontsize=20)
ax.legend()
ax.tick_params(labelsize=20)
```

```python
fig, ax = plt.subplots(1, 1, figsize=(10,6))
ax.plot(np.sqrt(lr_model_history.history['acc']), 'r', label='train')
ax.plot(np.sqrt(lr_model_history.history['val_acc']), 'b' ,label='val')
ax.set_xlabel(r'Epoch', fontsize=20)
ax.set_ylabel(r'Accuracy', fontsize=20)
ax.legend()
ax.tick_params(labelsize=20)
```

**Exercise:** Write a function that performs exponential learning rate decay using `LearningRateScheduler`

$lr = lr0*e^{(-kt)}$ t is the iteration number

### Step 3 - Choosing an `optimizer` and a `loss function`

When constructing a model and using it to make our predictions, for example to assign label scores to images ("cat", "plane", etc), we want to measure our success or failure by defining a "loss" function (or objective function). The goal of optimization is to efficiently calculate the parameters/weights that minimize this loss function. `keras` provides various types of [loss functions](https://github.com/keras-team/keras/blob/master/keras/losses.py).

Sometimes the "loss" function measures the "distance". We can define this "distance" between two data points in various ways suitable to the problem or dataset.

Distance

- Euclidean
- Manhattan
- others such as Hamming which measures distances between strings, for example. The Hamming distance of "carolin" and "cathrin" is 3.

Loss functions
- MSE (for regression)
- categorical cross-entropy (for classification)
- binary cross entropy (for classification)

```python
# build the model
input_dim = x_train.shape[1]

model = Sequential()
model.add(Dense(64, activation=tf.nn.relu, kernel_initializer='uniform',
                input_dim = input_dim)) # fully-connected layer with 64 hidden units
#model.add(Dropout(0.5))
model.add(Dense(64, kernel_initializer='uniform', activation=tf.nn.relu))
#model.add(Dropout(0.5))
model.add(Dense(num_classes, kernel_initializer='uniform', activation=tf.nn.softmax))
```

```python
# defining the parameters for RMSprop (I used the keras defaults here)
rms = RMSprop(lr=0.001, rho=0.9, epsilon=None, decay=0.0)

model.compile(loss='categorical_crossentropy',
              optimizer=rms,
              metrics=['acc'])
```

### Step 4 - Deciding on the `batch size` and `number of epochs`

```python
%%time
batch_size = 32
epochs = 20

model_history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/20
    60000/60000 [==============================] - 3s 58us/step - loss: 0.3924 - acc: 0.8847 - val_loss: 0.2036 - val_acc: 0.9383
    Epoch 2/20
    60000/60000 [==============================] - 3s 53us/step - loss: 0.1732 - acc: 0.9494 - val_loss: 0.1349 - val_acc: 0.9588
    Epoch 3/20
    60000/60000 [==============================] - 3s 53us/step - loss: 0.1246 - acc: 0.9637 - val_loss: 0.1142 - val_acc: 0.9647
    Epoch 4/20
    60000/60000 [==============================] - 3s 56us/step - loss: 0.1008 - acc: 0.9708 - val_loss: 0.1180 - val_acc: 0.9678
    Epoch 5/20
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0872 - acc: 0.9753 - val_loss: 0.1014 - val_acc: 0.9732
    Epoch 6/20
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0761 - acc: 0.9786 - val_loss: 0.1262 - val_acc: 0.9683
    Epoch 7/20
    60000/60000 [==============================] - 4s 58us/step - loss: 0.0685 - acc: 0.9806 - val_loss: 0.1107 - val_acc: 0.9713
    Epoch 8/20
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0627 - acc: 0.9829 - val_loss: 0.1198 - val_acc: 0.9715
    Epoch 9/20
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0577 - acc: 0.9837 - val_loss: 0.1448 - val_acc: 0.9674
    Epoch 10/20
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0530 - acc: 0.9858 - val_loss: 0.1179 - val_acc: 0.9739
    Epoch 11/20
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0496 - acc: 0.9867 - val_loss: 0.1215 - val_acc: 0.9743
    Epoch 12/20
    60000/60000 [==============================] - 3s 53us/step - loss: 0.0470 - acc: 0.9873 - val_loss: 0.1329 - val_acc: 0.9739
    Epoch 13/20
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0433 - acc: 0.9883 - val_loss: 0.1361 - val_acc: 0.9738
    Epoch 14/20
    60000/60000 [==============================] - 3s 56us/step - loss: 0.0403 - acc: 0.9891 - val_loss: 0.1227 - val_acc: 0.9761
    Epoch 15/20
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0366 - acc: 0.9905 - val_loss: 0.1414 - val_acc: 0.9731
    Epoch 16/20
    60000/60000 [==============================] - 3s 53us/step - loss: 0.0349 - acc: 0.9914 - val_loss: 0.1520 - val_acc: 0.9732
    Epoch 17/20
    60000/60000 [==============================] - 3s 55us/step - loss: 0.0314 - acc: 0.9913 - val_loss: 0.1444 - val_acc: 0.9760
    Epoch 18/20
    60000/60000 [==============================] - 3s 52us/step - loss: 0.0304 - acc: 0.9922 - val_loss: 0.1529 - val_acc: 0.9722
    Epoch 19/20
    60000/60000 [==============================] - 3s 53us/step - loss: 0.0281 - acc: 0.9927 - val_loss: 0.1463 - val_acc: 0.9749
    Epoch 20/20
    60000/60000 [==============================] - 3s 54us/step - loss: 0.0260 - acc: 0.9934 - val_loss: 0.1821 - val_acc: 0.9693
    CPU times: user 1min 58s, sys: 27 s, total: 2min 25s
    Wall time: 1min 5s

```python
from sklearn.metrics import r2_score as r2

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
print('Test R2:', r2(y_test, model.predict(x_test)))
```

    Test loss: 0.18207394694871187
    Test accuracy: 0.9693
    Test R2: 0.9419778253787543

```python
fig, ax = plt.subplots(1, 1, figsize=(10,6))
ax.plot(np.sqrt(model_history.history['acc']), 'r', label='train_acc')
ax.plot(np.sqrt(model_history.history['val_acc']), 'b' ,label='val_acc')
ax.set_xlabel(r'Epoch', fontsize=20)
ax.set_ylabel(r'Accuracy', fontsize=20)
ax.legend()
ax.tick_params(labelsize=20)
```

```python
fig, ax = plt.subplots(1, 1, figsize=(10,6))
ax.plot(np.sqrt(model_history.history['loss']), 'r', label='train')
ax.plot(np.sqrt(model_history.history['val_loss']), 'b' ,label='val')
ax.set_xlabel(r'Epoch', fontsize=20)
ax.set_ylabel(r'Loss', fontsize=20)
ax.legend()
ax.tick_params(labelsize=20)
```

```python
print(model_history.history.keys())
```

    dict_keys(['val_loss', 'val_acc', 'loss', 'acc'])

### Step 5 - Random restarts

This method does not seem to have an implementation in `keras`. We will leave it as a home exercise!

Hint: you can use `keras.callbacks.LearningRateScheduler`.

### Tuning the Hyperparameters using Cross Validation

Now instead of trying different values by hand, we will use GridSearchCV from Scikit-Learn to try out several values for our hyperparameters and compare the results.

To do cross-validation with `keras` we will use the wrappers for the Scikit-Learn API. They provide a way to use Sequential Keras models (single-input only) as part of your Scikit-Learn workflow.

There are two wrappers available:

`keras.wrappers.scikit_learn.KerasClassifier(build_fn=None, **sk_params)`, which implements the Scikit-Learn classifier interface,

`keras.wrappers.scikit_learn.KerasRegressor(build_fn=None, **sk_params)`, which implements the Scikit-Learn regressor interface.

```python
import numpy
from sklearn.model_selection import GridSearchCV
from keras.wrappers.scikit_learn import KerasClassifier
```

#### Trying different weight initializations

```python
# Function to create model, required for KerasClassifier
def create_model(init_mode='uniform'):
    # define model
    model = Sequential()
    model.add(Dense(64, kernel_initializer=init_mode, activation=tf.nn.relu, input_dim=784))
    model.add(Dropout(0.5))
    model.add(Dense(64, kernel_initializer=init_mode, activation=tf.nn.relu))
    model.add(Dropout(0.5))
    model.add(Dense(10, kernel_initializer=init_mode, activation=tf.nn.softmax))
    # compile model
    model.compile(loss='categorical_crossentropy',
              optimizer=RMSprop(),
              metrics=['accuracy'])
    return model
```

```python
%%time
seed = 7
numpy.random.seed(seed)
batch_size = 128
epochs = 10

model_CV = KerasClassifier(build_fn=create_model, epochs=epochs, batch_size=batch_size, verbose=0)
# define the grid search parameters
init_mode = ['uniform', 'lecun_uniform', 'normal', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform']

param_grid = dict(init_mode=init_mode)
grid = GridSearchCV(estimator=model_CV, param_grid=param_grid, n_jobs=-1, cv=3)
grid_result = grid.fit(x_train, y_train)

# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))
```

    Best: 0.949333 using {'init_mode': 'lecun_uniform'}
    0.945850 (0.002531) with: {'init_mode': 'uniform'}
    0.949333 (0.001327) with: {'init_mode': 'lecun_uniform'}
    0.947133 (0.001782) with: {'init_mode': 'normal'}
    0.112367 (0.002416) with: {'init_mode': 'zero'}
    0.947533 (0.001002) with: {'init_mode': 'glorot_normal'}
    0.949117 (0.001478) with: {'init_mode': 'glorot_uniform'}
    0.946383 (0.003040) with: {'init_mode': 'he_normal'}
    0.947383 (0.001852) with: {'init_mode': 'he_uniform'}
    CPU times: user 25.4 s, sys: 5.71 s, total: 31.1 s
    Wall time: 1min 51s

### Save Your Neural Network Model to JSON

The Hierarchical Data Format (HDF5) is a data storage format for storing large arrays of data including values for the weights in a neural network.
You can install HDF5 Python module: pip install h5py

Keras gives you the ability to describe and save any model using the JSON format.

```python
from keras.models import model_from_json

# serialize model to JSON
model_json = model.to_json()

with open("model.json", "w") as json_file:
    json_file.write(model_json)

# save weights to HDF5
model.save_weights("model.h5")
print("Model saved")

# when you want to retrieve the model: load json and create model
json_file = open('model.json', 'r')
saved_model = json_file.read()
# close the file as good practice
json_file.close()
model_from_json = model_from_json(saved_model)
# load weights into new model
model_from_json.load_weights("model.h5")
print("Model loaded")
```

    Model saved
    Model loaded

### Exercise 1: Perform a GridSearch for Learning rate and number of epochs combined

We will try various small standard learning rates, and momentum values from 0.1 to 0.9 in steps of 0.2.

We want to include the number of epochs in an optimization as there is a dependency between the learning rate, the batch size, and the number of epochs.

```python
# Function to create model, required for KerasClassifier
def create_model2(lr=0.01, momentum=0.9):
    # define model
    model = Sequential()
    model.add(Dense(64, activation=tf.nn.relu, input_dim=784))
    model.add(Dropout(0.5))
    model.add(Dense(64, activation=tf.nn.relu))
    model.add(Dropout(0.5))
    model.add(Dense(10, activation=tf.nn.softmax))
    # compile model
    model.compile(loss='categorical_crossentropy',
              optimizer=SGD(lr=lr, momentum=momentum),
              metrics=['accuracy'])
    return model
```

```python
%%time
seed = 12
numpy.random.seed(seed)
batch_size = 128
epochs = 10

model_CV = KerasClassifier(build_fn=create_model2, epochs=epochs, batch_size=batch_size, verbose=0)
# define the grid search parameters
lrs = [0.001, 0.01, 0.05, 0.1]
momentums = [0.1,0.3,0.5,0.7,0.9]

param_grid = dict(lr=lrs,momentum=momentums)
grid = GridSearchCV(estimator=model_CV, param_grid=param_grid, n_jobs=-1, cv=3)
grid_result = grid.fit(x_train, y_train)

# summarize results
print("Best: %f using %s" % (grid_result.best_score_, grid_result.best_params_))
means = grid_result.cv_results_['mean_test_score']
stds = grid_result.cv_results_['std_test_score']
params = grid_result.cv_results_['params']
for mean, stdev, param in zip(means, stds, params):
    print("%f (%f) with: %r" % (mean, stdev, param))
```

    Best: 0.947433 using {'lr': 0.1, 'momentum': 0.7}
    0.644717 (0.044976) with: {'lr': 0.001, 'momentum': 0.1}
    0.726167 (0.010186) with: {'lr': 0.001, 'momentum': 0.3}
    0.780017 (0.010354) with: {'lr': 0.001, 'momentum': 0.5}
    0.838217 (0.011302) with: {'lr': 0.001, 'momentum': 0.7}
    0.892733 (0.004777) with: {'lr': 0.001, 'momentum': 0.9}
    0.894867 (0.005373) with: {'lr': 0.01, 'momentum': 0.1}
    0.901750 (0.003902) with: {'lr': 0.01, 'momentum': 0.3}
    0.911750 (0.003221) with: {'lr': 0.01, 'momentum': 0.5}
    0.922217 (0.004031) with: {'lr': 0.01, 'momentum': 0.7}
    0.941050 (0.001995) with: {'lr': 0.01, 'momentum': 0.9}
    0.936417 (0.003528) with: {'lr': 0.05, 'momentum': 0.1}
    0.937250 (0.003182) with: {'lr': 0.05, 'momentum': 0.3}
    0.941383 (0.004048) with: {'lr': 0.05, 'momentum': 0.5}
    0.946967 (0.002925) with: {'lr': 0.05, 'momentum': 0.7}
    0.946200 (0.002777) with: {'lr': 0.05, 'momentum': 0.9}
    0.944200 (0.002947) with: {'lr': 0.1, 'momentum': 0.1}
    0.943217 (0.003100) with: {'lr': 0.1, 'momentum': 0.3}
    0.947000 (0.003581) with: {'lr': 0.1, 'momentum': 0.5}
    0.947433 (0.001782) with: {'lr': 0.1, 'momentum': 0.7}
    0.933600 (0.000982) with: {'lr': 0.1, 'momentum': 0.9}
    CPU times: user 24.2 s, sys: 5.01 s, total: 29.2 s
    Wall time: 3min 47s


# Differential rotation with `starry`

In this notebook, we'll explain how to express the radial velocity field of a star at an arbitrary orientation on the sky in terms of spherical harmonics. This allows us to use `starry` to model the Rossiter-McLaughlin waveform **analytically**!

First, let's import some stuff. We'll use `sympy` for the derivations.

```python
import matplotlib.pyplot as pl
%matplotlib notebook
import numpy as np
from IPython.display import display, Math
import sympy
from sympy import *
from sympy.functions.special.tensor_functions import KroneckerDelta
print("Using sympy version", sympy.__version__)

# Initialize the session
init_session(quiet=True)

# Define our symbols
x, y, z, mu, nu, l, m, j, k, p, q, n, A, B, C, alpha, omeq = symbols('x y z mu nu l m j k p q n A B C alpha \omega_{eq}')
```

    Using sympy version 1.3

## The radial velocity field of a star

The polynomial describing the radial component of the velocity field on the projected disk of a star is given by ([Short et al. 2018](https://arxiv.org/abs/1810.09565))

$f(\mathbf{x}, \mathbf{y}, \mathbf{z}) = \omega_{eq}(A\mathbf{x} + B\mathbf{y})(1 - \alpha(-B\mathbf{x} + A\mathbf{y} + C\mathbf{z})^2)$

where

$A = \sin(i)\cos(\lambda)$

$B = \sin(i)\sin(\lambda)$

$C = \cos(i)$

and $i$ and $\lambda$ are the stellar inclination and obliquity, respectively. The constant $\alpha$ is the shear due to differential rotation. We are assuming a simple linear shear according to the following equation:

$\omega = \omega_{eq}(1 - \alpha \sin^2\theta)$,

where $\omega$ is the angular rotational velocity at a point on the surface, $\omega_{eq}$ is the velocity at the equator, and $\theta$ is the polar angle (latitude).

## Let's expand this function

If we expand all the products in $f$, we can write it as a dot product of a polynomial coefficient vector and the polynomial basis defined in Luger et al. (2018):

```python
def poly_basis(n, x, y):
    """Return the n^th term in the polynomial basis."""
    l = Rational(floor(sqrt(n)))
    m = Rational(n - l * l - l)
    mu = Rational(l - m)
    nu = Rational(l + m)
    if (nu % 2 == 0):
        i = Rational(mu, 2)
        j = Rational(nu, 2)
        k = Rational(0)
    else:
        i = Rational(mu - 1, 2)
        j = Rational(nu - 1, 2)
        k = Rational(1)
    return x ** i * y ** j * sqrt(1 - x ** 2 - y ** 2) ** k

# Compute the polynomial basis
basis = Matrix([poly_basis(n, x, y) for n in range(16)]).T
```

Here's what the polynomial basis looks like (recall that $z = \sqrt{1 - x^2 - y^2}$):

```python
basis
```

And here's the function we wish to express:

```python
f = omeq * (A * x + B * y) * (1 - alpha * (-B * x + A * y + C * sqrt(1 - x ** 2 - y ** 2)) ** 2)
f
```

We can use `sympy` to figure out the (exact) representation of `f` in the polynomial basis:

```python
def Coefficient(expression, term):
    """Return the coefficient multiplying `term` in `expression`."""
    # Get the coefficient
    coeff = expression.coeff(term)
    # Set any non-constants in this coefficient to zero. If the coefficient
    # is not a constant, this is not the term we are interested in!
    coeff = coeff.subs(sqrt(1 - x ** 2 - y ** 2), 0).subs(x, 0).subs(y, 0)
    return coeff

vec = Matrix([Coefficient(expand(f), term) for term in basis])
vec
```

We can check that dotting this vector with the polynomial basis yields the original function $f$:

```python
simplify(factor(basis.dot(vec)) - f) == 0
```

    True

## Now let's represent it in terms of spherical harmonics

Now that we have the vector of polynomial coefficients `vec`, we want to apply a change-of-basis transformation to figure out their (exact) representation in terms of spherical harmonics.

As in Luger et al. (2018), let's compute the change of basis matrix from polynomials to spherical harmonic coefficients. This is the inverse of the $A_1$ matrix introduced in Luger et al. (2018). Note that it includes the normalization of $\frac{2}{\sqrt{\pi}}$ used internally by `starry`.

```python
def SA(l, m):
    """A spherical harmonic normalization constant."""
    return sqrt((2 - KroneckerDelta(m, 0)) * (2 * l + 1) * factorial(l - m) / (4 * pi * factorial(l + m)))

def SB(l, m, j, k):
    """Another spherical harmonic normalization constant."""
    try:
        ratio = factorial(Rational(l + m + k - 1, 2)) / factorial(Rational(-l + m + k - 1,  2))
    except ValueError:
        ratio = 0
    res = 2 ** l * Rational(factorial(m), (factorial(j) * factorial(k) * factorial(m - j) * factorial(l - m - k))) * ratio
    return simplify(res)

def SC(p, q, k):
    """Return the binomial theorem coefficient `C`."""
    res = factorial(Rational(k, 2)) / (factorial(Rational(q, 2)) * factorial(Rational(k - p, 2)) * factorial(Rational(p - q, 2)))
    return simplify(res)

def Y(l, m, x, y):
    """Return the spherical harmonic of degree `l` and order `m`."""
    res = 0
    z = sqrt(1 - x ** 2 - y ** 2)
    if (m >= 0):
        for j in range(0, m + 1, 2):
            for k in range(0, l - m + 1, 2):
                for p in range(0, k + 1, 2):
                    for q in range(0, p + 1, 2):
                        res += (-1) ** ((j + p) // 2) * SA(l, m) * SB(l, m, j, k) * SC(p, q, k) * x ** (m - j + p - q) * y ** (j + q)
            for k in range(1, l - m + 1, 2):
                for p in range(0, k, 2):
                    for q in range(0, p + 1, 2):
                        res += (-1) ** ((j + p) // 2) * SA(l, m) * SB(l, m, j, k) * SC(p, q, k - 1) * x ** (m - j + p - q) * y ** (j + q) * z
    else:
        for j in range(1, abs(m) + 1, 2):
            for k in range(0, l - abs(m) + 1, 2):
                for p in range(0, k + 1, 2):
                    for q in range(0, p + 1, 2):
                        res += (-1) ** ((j + p - 1) // 2) * SA(l, abs(m)) * SB(l, abs(m), j, k) * SC(p, q, k) * x ** (abs(m) - j + p - q) * y ** (j + q)
            for k in range(1, l - abs(m) + 1, 2):
                for p in range(0, k, 2):
                    for q in range(0, p + 1, 2):
                        res += (-1) ** ((j + p - 1) // 2) * SA(l, abs(m)) * SB(l, abs(m), j, k) * SC(p, q, k - 1) * x ** (abs(m) - j + p - q) * y ** (j + q) * z

    return res

def p_Y(l, m, lmax):
    """Return the polynomial basis representation of the spherical harmonic `Y_{lm}`."""
    ylm = Y(l, m, x, y)
    res = [ylm.subs(sqrt(1 - x ** 2 - y ** 2), 0).subs(x, 0).subs(y, 0)]
    for n in range(1, (lmax + 1) ** 2):
        res.append(Coefficient(ylm, poly_basis(n, x, y)))
    return res

def A1(lmax, norm = 2 / sqrt(pi)):
    """Return the change of basis matrix A1. The columns of this matrix are given by `p_Y`."""
    res = zeros((lmax + 1) ** 2, (lmax + 1) ** 2)
    n = 0
    for l in range(lmax + 1):
        for m in range(-l, l + 1):
            res[n] = p_Y(l, m, lmax)
            n += 1
    return res * norm
```

We can now evaluate the change of basis matrix from spherical harmonic coefficients to polynomials, $A_1$ for $l_\mathrm{max} = 3$. We then take the inverse to go from polynomial coeffiecients to $Y_{lm}$ coefficients:

```python
M = Matrix(A1(3)).inv()
M
```

## The end result

We can finally compute the spherical harmonic coefficients of the function $f$:

```python
ycoeffs = simplify(M * vec)
ycoeffs
```

Note that this is the **exact** spherical harmonic representation of the function `f`.

## Important note about current `starry` implementation

A few comments are in order regarding how exactly this is implemented in `starry`. This all happens behind the scenes, but it's useful to know if you're poking around in the code.

- When setting the spherical harmonic coefficients in `starry`, it is necessary to normalize the vector above by dividing it by $\pi$. This is because in `starry`, fluxes are normalized so that the integral of $Y_{0,0}$ over the disk is **unity** (instead of $\pi$).

- When limb darkening is present, `starry` requires the $Y_{0,0}$ coefficient of the map to be non-zero. But the spherical harmonic representation of the brightness-weighted velocity has $Y_{0,0} = 0$ (i.e., the star has zero net radial velocity). Overcoming this requires a bit of a *hack*. We set $Y_{0,0} = 1$ so we can limb-darken the map and compute the RM amplitude, but that means we have effectively computed $\int{(Iv + I)dS}$. We must therefore *subtract* $\int{IdS}$ to get the actual integrated brightness-weighted velocity. The RM effect amplitude is thus

$\ \ \ \ \ \ \ \ \ \ \Delta v = \frac{\int{(Iv + I)dS} - \int{IdS}}{\int{IdS}}$

We therefore compute the `starry` flux using two different `Map` instances: one to compute the $Iv + I$ term, and a uniform, limb-darkened map to compute the $I$ term.

## Interactive visualization

Below you can interactively see how different velocity profiles affect the Rossiter-McLaughlin waveform. We coded up a simple Jupyter widget to visualize the RM effect with `starry`. Try to find the parameters that best fit the measured RV data for the hot jupiter host HD 189733!

```python
from viz import visualize
visualize();
```

    interactive(children=(FloatSlider(value=0.1, continuous_update=False, description='$v_\\mathrm{eq}$ [km / s]:'…

## One possible solution

The following values should get you a pretty good fit. They are close to the means of the posterior distributions when we do a full MCMC fit of the dataset, but they are probably not the "true" values.

$v_{eq}$: 5.00

$\lambda$: -0.50

$i$: 80.0

$\alpha$: 0.65

$u_1$: 1.50

$u_2$: -0.54

$b$: -0.66

$r/R_\star$: 0.158


# Ejercicio 3 - Simulación de distribuciones condicionadas

### Julian Ferres - Nro.Padrón 101483

## Enunciado:

Sea $X$ \~ $N(0,1)$ truncada al intervalo $[-1,1]$

Imagine $m(x) = E[Y | X=x]$ como:

   \begin{equation}
     m(x) := \left\{
 \begin{array}{ll}
      \frac{(x + 2)^2}{2} & \mathrm{si\ } si -1\leq x<-0.5 \\
    \frac{x}{2}+0.875     & \mathrm{si\ } -0.5 \leq x \leq 0\\
    -5(x-0.2)^2 +1.075 & \mathrm{si\ } 0 < x \leq 0.5 \\
    x + 0.125 & \mathrm{si\ } 0.5 \leq x < 1
           \end{array}
         \right.
    \end{equation}

Dado un $x$, la distribución condicional de $Y - m(x)$ es $N(0, \sigma ^2(x))$,

con $\sigma(x)=0.2-0.1 * \cos(2x)$

- Se pide simular $200$ puntos $(X,Y)$, y graficarlos en un plano. Además, vamos a necesitar
Los $200$ pares ordenados en cuestión, para hacer análisis posteriores

- Reconstruir $m(x)$ con los $200$ puntos, para eso:

Realizar una partición de $[-1,1]$ en intervalos de longitud $h$ y en cada intervalo encontrar el polinomio $f$ de grado $M$ que minimice el error cuadratico medio $$ \frac{1}{n} \sum |f(X_i)-Y_i|^2$$

Usar:

1. $h = 0.5$ , $M=1$
2. $h = 0.1$ , $M=1$
3. $h = 0.25$ , $M=2$
4. $h = 0.5$ , $M=2$

## Solución:

#### Importo todas las librerias e inicializo funciones

```python
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
from math import cos, pi
from scipy.stats import truncnorm
```

```python
m1 = lambda x: (x+2)**2/2
m2 = lambda x: x/2 + 0.875
m3 = lambda x: -5*(x-0.2)**2 + 1.075
m4 = lambda x: x + 0.125
```

```python
def m(x):
    if -1 <= x < -0.5:
        return m1(x)
    if -0.5 <= x < 0:
        return m2(x)
    if 0 <= x < 0.5:
        return m3(x)
    if 0.5 <= x < 1:
        return m4(x)

m = np.vectorize(m)
```

```python
#Me genero 1000 valores entre -1 y 1 para graficar m(x) 'suave'
x_0 = np.linspace(-1,1,1000)
y_0 = m(x_0)
```

#### Normal truncada

```python
a , b = -1 , 1 #Limites de la normal truncada
```

```python
#Genero 200 cuantiles de la normal truncada
x1 = np.linspace(truncnorm.ppf(0.01, a, b),
                truncnorm.ppf(0.99, a, b), 200)
```

```python
plt.plot(x1, truncnorm.pdf(x1, a, b),
        'r-', lw=3, alpha=0.75, label='Normal truncada')
plt.title("Density Plot de X",fontsize='15')
plt.legend(loc='best', frameon= True)
plt.grid()
```

```python
x1 = truncnorm.rvs(a, b, size=200)
#Me genero la muestra de distribucion X
```

```python
sigma = np.vectorize(lambda x : 0.2 - 0.1 * cos(2*pi*x))
normal = np.vectorize(np.random.normal)

y1 = normal( m(x1),sigma(x1))
```

```python
fig, ax = plt.subplots(figsize=(11,7))
plt.plot(x_0, y_0, 'g-', linewidth = 5, label = 'Función m(x)=E[Y|X=x]')
plt.legend(loc='best', frameon= True)

plt.plot(x1, y1, 'ro' ,markersize= 5, alpha = 0.5 ,label = 'Dispersion (X,Y)')
plt.legend(loc='best', frameon= True)

plt.title("Scatter Plot de (X,Y) y Line plot de m(x)", fontsize='15')
plt.xlabel('X')
plt.ylabel('Y')

plt.show()
```

#### La muestra de los $200$ pares con distribución $(X,Y)$ se encuentra en la variable output

### Dejo los 200 puntos en el archivo simulacion.csv

```python
d = {'X': x1 , 'Y': y1 }
output = pd.DataFrame(data=d)
```

```python
output.to_csv("simulacion.csv" , index = False)
```

## Reconstruyo la regresión

#### Con h=0.5 y M=1

```python
partition = [[],[],[],[]]
for i in range(200):
    partition[int(2*(x1[i]+1))].append(i)
```

```python
polinomio_a_trozos = []

cuadrado_de_los_errores1 = 0

for i in range(4):
    x_aux , y_aux = [x1[j] for j in partition[i]],[y1[j] for j in partition[i]]
    z = np.polyfit(x_aux,y_aux,1)
    polinomio_a_trozos.append(np.poly1d(z))

    #sumo los errores para cada trozo del polinomio
    for j in range(len(x_aux)):
        cuadrado_de_los_errores1 += (polinomio_a_trozos[i](x_aux[j])-y_aux[j])**2
```

```python
xp=[]
xp.append(np.linspace(-1, -0.5, 200))
xp.append(np.linspace(-0.5,0, 200))
xp.append(np.linspace(0, 0.5, 200))
xp.append(np.linspace(0.5,1, 200))
```

```python
fig, ax = plt.subplots(figsize=(11,7))

plt.plot(x1, y1, 'ro', linewidth = 5, alpha = 0.5 ,label = 'Dispersion X,Y')
plt.legend(loc='best', frameon= True)

for i in range(4):
    plt.plot(xp[i], polinomio_a_trozos[i](xp[i]) ,'b-', linewidth = 5 )

plt.plot(x_0, y_0, 'g-', linewidth = 5, alpha = 0.75 ,label = 'Función m(x)=E[Y|X=x]')

plt.legend(loc='best', frameon= True)
plt.title("Estimación m(x) con h=0.5 y M=1", fontsize='15')
plt.xlabel('X')
plt.ylabel('Y')

plt.show()
```

La estimación parece ajustarse bien a la función de regresion, no obstante, el error cuadrático medio es alto ya que no esta Overfitteando
a la muestra.

#### Estimación del error cuadrático medio

```python
(cuadrado_de_los_errores1 / 200)**0.5
```

    0.21146615768136706

#### Con h=0.1 y M=1

```python
partition = [[] for i in range(20)]
for i in range(200):
    partition[int(10*(x1[i]+1))].append(i)
```

```python
polinomio_a_trozos = []
cuadrado_de_los_errores2 = 0

for i in range(20):
    x_aux , y_aux = [x1[j] for j in partition[i]],[y1[j] for j in partition[i]]
    z = np.polyfit(x_aux,y_aux,1)
    polinomio_a_trozos.append(np.poly1d(z))

    #sumo los errores para cada trozo del polinomio
    for j in range(len(x_aux)):
        cuadrado_de_los_errores2 += (polinomio_a_trozos[i](x_aux[j])-y_aux[j])**2
```

```python
xp=[]

for i in range(20):
    xp.append(np.linspace(-1+i*(1/10), -0.9+i*(1/10), 200))
```

```python
fig, ax = plt.subplots(figsize=(11,7))

plt.plot(x1, y1, 'ro', linewidth = 5, alpha = 0.5 ,label = 'Dispersion X,Y')
plt.legend(loc='best', frameon= True)

for i in range(20):
    plt.plot(xp[i], polinomio_a_trozos[i](xp[i]) ,'b-', linewidth = 5 )

plt.plot(x_0, y_0, 'g-', linewidth = 5, alpha = 0.75,label = 'Función m(x)=E[Y|X=x]')
plt.legend(loc='best', frameon= True)
plt.title("Estimación m(x) con h=0.1 y M=1", fontsize='15')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()
```

Se puede observar un claro caso de Overfitting, donde el error cuadrático medio es medianamente bajo, pero no estima correctamente la regresión.

#### Estimación del error cuadrático medio

```python
(cuadrado_de_los_errores2 / 200)**0.5
```

    0.1885956312812796

#### Con h=0.25 y M=2

```python
partition = [[] for i in range(8)]
for i in range(200):
    partition[int(4*(x1[i]+1))].append(i)
```

```python
polinomio_a_trozos = []
cuadrado_de_los_errores3 = 0

for i in range(8):
    x_aux , y_aux = [x1[j] for j in partition[i]],[y1[j] for j in partition[i]]
    z = np.polyfit(x_aux,y_aux,2)
    polinomio_a_trozos.append(np.poly1d(z))

    #sumo los errores para cada trozo del polinomio
    for j in range(len(x_aux)):
        cuadrado_de_los_errores3 += (polinomio_a_trozos[i](x_aux[j])-y_aux[j])**2
```

```python
xp=[]

for i in range(8):
    xp.append(np.linspace(-1+i*(1/4), -1+(i+1)*(1/4), 200))
```

```python
fig, ax = plt.subplots(figsize=(11,7))

plt.plot(x1, y1, 'ro', linewidth = 5,alpha = 0.5, label ='Dispersion X,Y')
plt.legend(loc='best', frameon= True)

for i in range(8):
    plt.plot(xp[i], polinomio_a_trozos[i](xp[i]) ,'b-', linewidth = 5 )

plt.plot(x_0, y_0, 'g-', linewidth = 5,alpha = 0.75 ,label = 'Función m(x)=E[Y|X=x]')
plt.legend(loc='best', frameon= True)
plt.title("Estimación m(x) con h=0.25 y M=2", fontsize='15')
plt.xlabel('X')
plt.ylabel('Y')
plt.show()
```

Se puede observar un claro caso de Overfitting, donde el error cuadrático medio es medianamente bajo, pero no estima correctamente la regresión.

#### Estimación del error cuadrático medio

```python
(cuadrado_de_los_errores3 / 200)**0.5
```

    0.20085090075741924

#### Con h=0.5 y M=2

```python
partition = [[] for i in range(4)]
for i in range(200):
    partition[int(2*(x1[i]+1))].append(i)
```

```python
polinomio_a_trozos = []
cuadrado_de_los_errores4 = 0

for i in range(4):
    x_aux , y_aux = [x1[j] for j in partition[i]],[y1[j] for j in partition[i]]
    z = np.polyfit(x_aux,y_aux,2)
    polinomio_a_trozos.append(np.poly1d(z))

    #sumo los errores para cada trozo del polinomio
    for j in range(len(x_aux)):
        cuadrado_de_los_errores4 += (polinomio_a_trozos[i](x_aux[j])-y_aux[j])**2
```

```python
xp=[]
for i in range(4):
    xp.append(np.linspace(-1+i*(1/2), -1+(i+1)*(1/2), 200))
```

```python
fig, ax = plt.subplots(figsize=(11,7))

plt.plot(x1, y1, 'ro', linewidth = 5,alpha = 0.5, label = 'Dispersion X,Y')
plt.legend(loc='best', frameon= True)

for i in range(4):
    plt.plot(xp[i], polinomio_a_trozos[i](xp[i]) ,'b-', linewidth = 5)

plt.plot(x_0, y_0, 'g-', linewidth = 5,alpha = 0.75 ,label = 'Función m(x)=E[Y|X=x]')
plt.legend(loc='best', frameon= True)
plt.title("Estimación m(x) con h=0.5 y M=2", fontsize='15')
plt.xlabel('X')
plt.ylabel('Y')

plt.show()
```

Se ve que el ECM es ligeramente superior a los casos con Overfitting, se ve que predice la regresión de forma bastante acertada.

#### Estimación del error cuadrático medio

```python
(cuadrado_de_los_errores4 / 200)**0.5
```

    0.20321524367619534

```python
(cuadrado_de_los_errores1 / 200)**0.5 ,\
(cuadrado_de_los_errores2 / 200)**0.5 ,\
(cuadrado_de_los_errores3 / 200)**0.5 ,\
(cuadrado_de_los_errores4 / 200)**0.5
```

    (0.21146615768136706,
     0.1885956312812796,
     0.20085090075741924,
     0.20321524367619534)

Link al Repo de GitHub: https://github.com/julianferres/Aprendizaje-Estadistico.git

Ejercicios/06-Reconstruir Regresion (Penultima Clase)/EstimacionRegresion.ipynb


# "유러피안 옵션의 가치 평가"

> "이항 모형으로 유러피안 콜옵션의 가치를 평가한다."

- toc: true
- badges: true
- author: 단호진
- categories: [finance]

현재 주가가 20달러이고, 3개월 마다 10% 움직인다. 무위험이자율이 8%일 때 잔여 만기 6개월, 행사 가격 20달러인 콜옵션의 현재적정가치를 구하여라. (연습문제 1-5)

참고:
1. 이재성, 금융수학개론, 2판, 청문각, 2018

```python
import plotly.graph_objects as go

p1 = 0.601
p2 = 1 - p1

fig = go.Figure(data=[go.Sankey(
    node = dict(
      pad = 15,
      thickness = 20,
      line = dict(color = "black", width = 0.5),
      label = [
          "S=20, c=1.46",
          "S=22, c=2.47=max(2.47, 2)",
          "S=18, c=0",
          "S=24.2, c=4.2",
          "S=19.8, c=0",
          "S=16.2, c=0"],
      color = "blue"
    ),
    link = dict(
      source = [0, 0, 1, 1, 2, 2], # indices correspond to labels, eg A1, A2, A1, B1, ...
      target = [1, 2, 3, 4, 4, 5],
      value = [p1, p2, p1 * p1, p1 * p2,  p2 * p1, p2 * p2]
  ))])

fig.update_layout(title_text="Stock price and call option value diagram", font_size=10)
fig.show()
```

계산의 최종 결과는 위 다이어 그램에 정리하였다. 주가의 상승 확률이 0.6으로 평가되며 현재 콜옵션의 가치는 1.46달러이다.

이 결과는 손익 행렬과 마팅게일 측도 식에서 옵션의 가치를 다음과 같이 평가하였다.

```python
import numpy as np
from sympy import *
```

```python
s0, sp, sm, c0, cp, cm, p, r, T = symbols('s0, sp, sm, c0, cp, cm, p, r, T')
```

```python
eq1 = sp * p + sm * (1 - p) - s0 * exp(r * T)
eq2 = cp * p + cm * (1 - p) - c0 * exp(r * T)
eq1, eq2
```

    (p*sp - s0*exp(T*r) + sm*(1 - p), -c0*exp(T*r) + cm*(1 - p) + cp*p)

```python
params = {r: 0.08, T: 1/4, s0: 22, sp: 24.2, sm: 19.8, cp: 4.2, cm: 0}
eq1s = eq1.subs(params)
eq2s = eq2.subs(params)
solve([eq1s, eq2s])
```

    {c0: 2.47424507450232, p: 0.601006700133780}

```python
params = {r: 0.08, T: 1/4, s0: 20, sp: 22, sm: 18, cp: 2.474, cm: 0}
eq1s = eq1.subs(params)
eq2s = eq2.subs(params)
solve([eq1s, eq2s])
```

    {c0: 1.45744817007588, p: 0.601006700133778}

```python

```


# Filtros de suavização

### Definindo novamente as funções de correlação e convolução

```python
import numpy as np
import matplotlib.pyplot as plt

def correlation(img, w):
    '''Calcula a correlação cruzada entre a imagem img e o filtro w.

       Parâmetros
       ----------
       img : numpy array.
           Imagem a ser filtrada.
       w : numpy array.
           Filtro a ser utilizada na correlação cruzada.

       Return
       -------
       img_filtered : numpy array, mesmo tamanho que img
           Imagem filtrada
    '''

    numRows, numCols = img.shape
    fNumRows, fNumCols = w.shape   # fNumRows=a+1 e fNumCols=b+1 (a e b da fórmula acima)

    halfRSize = fNumRows//2        # O operador // retorna a parte inteira da divisão
    halfCSize = fNumCols//2

    imgPadded = np.zeros((numRows+fNumRows-1, numCols+fNumCols-1), dtype=img.dtype)
    for row in range(numRows):
        for col in range(numCols):
            imgPadded[row+halfRSize, col+halfCSize] = img[row, col]
    # A estrutura for aninhada acima pode ser substituída por
    #imgPadded = np.pad(img, ((halfRSize,halfRSize),(halfCSize,halfCSize)), mode='constant')

    img_filtered = np.zeros((numRows, numCols))
    for row in range(numRows):
        for col in range(numCols):
            sum_region = 0
            for s in range(fNumRows):
                for t in range(fNumCols):
                    # O índice do array na soma abaixo é diferente do utilizado na fórmula da
                    # correlação porque estamos utilizando a imagem preenchida com zeros. O
                    # pixel (0,0)  na imagem preenchida corresponde ao pixel (x+s-a/2, y+t-b/2)
                    # da imagem original
                    sum_region += w[s, t]*imgPadded[row+s, col+t]
            img_filtered[row, col] = sum_region

            # O loop aninhado acima pode ser substituído por
            # patch = imgPadded[row:row+fNumRows, col:col+fNumCols]
            # img_filtered[row, col] = np.sum(w*patch)

    return img_filtered

def convolution(img, w):
    '''Calcula a convolução entre a imagem img e o filtro w.

       Parâmetros
       ----------
       img : numpy array.
           Imagem a ser filtrada.
       w : numpy array.
           Filtro a ser utilizada na correlação cruzada.

       Return
       -------
       img_filtered : numpy array, mesmo tamanho que img
           Imagem filtrada
    '''
    # Inverte o filtro
    w_inv = w[::-1, ::-1]

    img_filtered = correlation(img, w_inv)

    return img_filtered
```

```python
img = plt.imread('cameraman.tiff')
plt.figure(figsize=[7,7])
plt.imshow(img, 'gray')
```

# Filtro de média simples

Na imagme resultante, observe as diversas "linhas" que aparecem na imagem

```python
# Cria filtro de média simples com tamanho S
S = 5
w = np.full([S,S], 1./S**2)
print(w)
```

    [[0.04 0.04 0.04 0.04 0.04]
     [0.04 0.04 0.04 0.04 0.04]
     [0.04 0.04 0.04 0.04 0.04]
     [0.04 0.04 0.04 0.04 0.04]
     [0.04 0.04 0.04 0.04 0.04]]

```python
img_smoothed = convolution(img, w)
plt.imshow(img_smoothed, 'gray')
```

# Filtro de suavização gaussiana

Lembrando que a função gaussiana é definida pela equação
\begin{equation}
y(x) = \frac{1}{\sigma\sqrt{2\pi}}exp\left({-\frac{(x-\mu)^2}{2\sigma^2}}\right)
\end{equation}
onde $\mu$ é o ponto central da função. Consideraremos que $\mu=0$

## 1. Visualização da função gaussiana

```python
sigma = 3
x = np.linspace(-5*sigma, 5*sigma, 1000)
y = np.exp(-x**2/(2*sigma**2))
y = y/(sigma*np.sqrt(2*np.pi))
plt.plot(x, y)
```

## 2. Filtro gaussiano em uma dimensão

```python
def gaussian_filter_1d(filter_size):
    sigma = filter_size/6.
    x = np.linspace(-3*sigma, 3*sigma, filter_size)
    y = np.exp(-x**2/(2*sigma**2))

    # Filtros de suavização precisam ter soma igual a 1
    y = y/np.sum(y)

    return y

y = gaussian_filter_1d(5)
print(y)
```

    [0.00664603 0.19422555 0.59825683 0.19422555 0.00664603]

```python
# Sinal 1D simples, representado como uma matriz de uma única linha
signal = np.array([[0, 0, 0, 0, 1, 0, 0, 0, 0]])
# O filtro gaussiano definido acima é um array 1D, como nossa função de convolução
# espera um array 2D, redimensionamos o filtro para um array 2D de uma linha
w = y.reshape(1, len(y))
filtered_signal = convolution(signal, w)
print('Sinal filtrado: ')
print(filtered_signal)
```

    Sinal filtrado:
    [[0.         0.         0.00664603 0.19422555 0.59825683 0.19422555
      0.00664603 0.         0.        ]]

Note que o sinal filtrado possui exatamente os valores do filtro utilizado. Isso porque o sinal é um impulso (valor 1 cercado de zeros). Dessa forma, podemos interpretar a suavização da seguinte forma: o valor 1 do sinal é "espalhado" (difundido) ao longo da sua vizinhança.

```python
# Sinal 1D aleatório
S = 10
signal = np.random.randint(0, 10, size=(1, 50))
w = gaussian_filter_1d(S).reshape(1, S)
filtered_signal = convolution(signal, w)
plt.subplot(1, 2, 1)
plt.plot(signal[0])
plt.subplot(1, 2, 2)
plt.plot(filtered_signal[0])
```

```python
# Filtragem de uma função seno ruidosa
# Cria função seno com um pouco de ruído
x = np.linspace(0, 4*np.pi, 100)
corrupted_sin = np.sin(x) + 0.6*np.random.rand(len(x))
plt.plot(x, corrupted_sin)
plt.title('Sinal original')

# Filtra sinal
S = 21
w = gaussian_filter_1d(S).reshape(1, S)
corrupted_sin = corrupted_sin.reshape(1, len(corrupted_sin))
filtered_signal = convolution(corrupted_sin, w)
plt.figure()
plt.plot(x, filtered_signal[0])
plt.title('Sinal filtrado')
```

## 3. Filtro gaussiano bidimensional

Função gaussiana em duas dimensões:

\begin{equation}
z(x) = \frac{1}{2\pi\sigma^2}exp\left({-\frac{(x-\mu_x)^2 + (y-\mu_y)^2}{2\sigma^2}}\right)
\end{equation}
onde $\mu_x$ e $\mu_y$ indicam o ponto central da função em $x$ e $y$. Consideraremos que o centro da função é a origem ($\mu_x=0$ e $\mu_y=0$).

```python
def gaussian_filter_2d(filter_size):
    sigma = filter_size/6.
    x_vals = np.linspace(-3*sigma, 3*sigma, filter_size)
    y_vals = x_vals.copy()
    z = np.zeros((filter_size, filter_size))
    for row in range(filter_size):
        x = x_vals[row]
        for col in range(filter_size):
            y = y_vals[col]
            z[row, col] = np.exp(-(x**2+y**2)/(2*sigma**2))
    z = z/np.sum(z)

    return z
```

```python
w = gaussian_filter_2d(13)
plt.figure(figsize=[5,5])
plt.imshow(w, 'gray')
```

```python
img = plt.imread('cameraman.tiff')
img_filtered = convolution(img, w)
```

```python
plt.figure()
plt.imshow(img, 'gray')
plt.title('Original')
plt.figure()
plt.imshow(img_filtered, 'gray')
plt.title('Filtered')
```

```python
# Filtragem utilizando uma função do scipy
import scipy.signal
w = gaussian_filter_2d(25)
img_filtered_scipy = scipy.signal.convolve(img, w, mode='same')
plt.imshow(img_filtered_scipy, 'gray')
```

## Usando suavização para segmentar um objeto

#### Note que a imagem abaixo está muito ruidosa, e com isso não há como tornarmos a imagem binária utilizando uma transformação de limiarização.

```python
pacman = plt.imread('pacman.tiff')
plt.imshow(pacman, 'gray')
plt.figure()
_ = plt.hist(pacman.flatten(), 50)
```

```python
pacman_thresholded = pacman>110
plt.imshow(pacman_thresholded, 'gray')
```

#### Ao suavizarmos a imagem, o histograma (mostrado abaixo) passa a apresentar dois picos característicos. O pico à esquerda corresponde ao objeto, enquanto que o pico à direita corresponde ao fundo da imagem. Portanto, um limiar apropriado para a imagem é um valor entre os dois picos.

```python
import scipy.ndimage as ndi
w = gaussian_filter_2d(9)
pacman_filtered = scipy.signal.convolve(pacman, w, mode='same')
plt.imshow(pacman_filtered, 'gray')
plt.figure()
_ = plt.hist(pacman_filtered.flatten(), 50)
```

```python
pacman_filt_thresholded = pacman_filtered>110
plt.imshow(pacman_filt_thresholded, 'gray')
```

Lecture 02 - Intensity transforms and spatial filtering/Suavizacao.ipynb