# Symbolic Regression

This example combines neural differential equations with regularised evolution to discover the equations

$\frac{\mathrm{d} x}{\mathrm{d} t}(t) = \frac{y(t)}{1 + y(t)}$

$\frac{\mathrm{d} y}{\mathrm{d} t}(t) = \frac{-x(t)}{1 + x(t)}$

directly from data.

**References:**

This example appears as an example in:

```bibtex
@phdthesis{kidger2021on,
    title={{O}n {N}eural {D}ifferential {E}quations},
    author={Patrick Kidger},
    year={2021},
    school={University of Oxford},
}
```

Whilst drawing heavy inspiration from:

```bibtex
@inproceedings{cranmer2020discovering,
    title={{D}iscovering {S}ymbolic {M}odels from {D}eep {L}earning with {I}nductive
           {B}iases},
    author={Cranmer, Miles and Sanchez Gonzalez, Alvaro and Battaglia, Peter and
            Xu, Rui and Cranmer, Kyle and Spergel, David and Ho, Shirley},
    booktitle={Advances in Neural Information Processing Systems},
    publisher={Curran Associates, Inc.},
    year={2020},
}

@software{cranmer2020pysr,
    title={PySR: Fast \& Parallelized Symbolic Regression in Python/Julia},
    author={Miles Cranmer},
    publisher={Zenodo},
    url={http://doi.org/10.5281/zenodo.4041459},
    year={2020},
}
```

This example is available as a Jupyter notebook [here](https://github.com/patrick-kidger/diffrax/blob/main/examples/symbolic_regression.ipynb).

```python
import tempfile
from typing import List

import equinox as eqx  # https://github.com/patrick-kidger/equinox
import jax
import jax.numpy as jnp
import optax  # https://github.com/deepmind/optax
import pysr  # https://github.com/MilesCranmer/PySR
import sympy

# Note that PySR, which we use for SymbolicRegression, uses Julia as a backend.
# You'll need to install a recent version of Julia if you don't have one.
# (And can get funny errors if you have a too-old version of Julia already.)
# You may also need to restart Python after running `pysr.install()` the first time.
pysr.silence_julia_warning()
pysr.install(quiet=True)
```

Now for a bunch of helpers. We'll use these in a moment; skip over them for now.

```python
def quantise(expr, quantise_to):
    if isinstance(expr, sympy.Float):
        return expr.func(round(float(expr) / quantise_to) * quantise_to)
    elif isinstance(expr, sympy.Symbol):
        return expr
    else:
        return expr.func(*[quantise(arg, quantise_to) for arg in expr.args])

class SymbolicFn(eqx.Module):
    fn: callable
    parameters: jnp.ndarray

    def __call__(self, x):
        # Dummy batch/unbatching. PySR assumes its JAX'd symbolic functions act on
        # tensors with a single batch dimension.
        return jnp.squeeze(self.fn(x[None], self.parameters))

class Stack(eqx.Module):
    modules: List[eqx.Module]

    def __call__(self, x):
        return jnp.stack([module(x) for module in self.modules], axis=-1)

def expr_size(expr):
    return sum(expr_size(v) for v in expr.args) + 1

def _replace_parameters(expr, parameters, i_ref):
    if isinstance(expr, sympy.Float):
        i_ref[0] += 1
        return expr.func(parameters[i_ref[0]])
    elif isinstance(expr, sympy.Symbol):
        return expr
    else:
        return expr.func(
            *[_replace_parameters(arg, parameters, i_ref) for arg in expr.args]
        )

def replace_parameters(expr, parameters):
    i_ref = [-1]  # Distinctly sketchy approach to making this conversion.
    return _replace_parameters(expr, parameters, i_ref)
```

Now for the main program, which we can run with `main()`. We discuss what's happening at each step in the comments -- read on:

```python
def main(
    symbolic_dataset_size=2000,
    symbolic_num_populations=100,
    symbolic_population_size=20,
    symbolic_migration_steps=4,
    symbolic_mutation_steps=30,
    symbolic_descent_steps=50,
    pareto_coefficient=2,
    fine_tuning_steps=500,
    fine_tuning_lr=3e-3,
    quantise_to=0.01,
):
    #
    # First obtain a neural approximation to the dynamics.
    # We begin by running the previous example.
    #

    # Runs the Neural ODE example.
    # This defines the variables `ts`, `ys`, `model`.
    print("Training neural differential equation.")
    %run neural_ode.ipynb

    #
    # Now symbolically regress across the learnt vector field, to obtain a Pareto
    # frontier of symbolic equations, that trades loss against complexity of the
    # equation. Select the "best" from this frontier.
    #

    print("Symbolically regressing across the vector field.")
    vector_field = model.func.mlp  # noqa: F821
    dataset_size, length_size, data_size = ys.shape  # noqa: F821
    in_ = ys.reshape(dataset_size * length_size, data_size)  # noqa: F821
    in_ = in_[:symbolic_dataset_size]
    out = jax.vmap(vector_field)(in_)
    with tempfile.TemporaryDirectory() as tempdir:
        symbolic_regressor = pysr.PySRRegressor(
            niterations=symbolic_migration_steps,
            ncyclesperiteration=symbolic_mutation_steps,
            populations=symbolic_num_populations,
            npop=symbolic_population_size,
            optimizer_iterations=symbolic_descent_steps,
            optimizer_nrestarts=1,
            procs=1,
            verbosity=0,
            tempdir=tempdir,
            temp_equation_file=True,
            output_jax_format=True,
        )
        symbolic_regressor.fit(in_, out)
        best_equations = symbolic_regressor.get_best()
        expressions = [b.sympy_format for b in best_equations]
        symbolic_fns = [
            SymbolicFn(b.jax_format["callable"], b.jax_format["parameters"])
            for b in best_equations
        ]

    #
    # Now the constants in this expression have been optimised for regressing across
    # the neural vector field. This was good enough to obtain the symbolic expression,
    # but won't quite be perfect -- some of the constants will be slightly off.
    #
    # To fix this we now plug our symbolic function back into the original dataset
    # and apply gradient descent.
    #

    print("Optimising symbolic expression.")

    symbolic_fn = Stack(symbolic_fns)
    flat, treedef = jax.tree_flatten(
        model, is_leaf=lambda x: x is model.func.mlp  # noqa: F821
    )
    flat = [symbolic_fn if f is model.func.mlp else f for f in flat]  # noqa: F821
    symbolic_model = jax.tree_unflatten(treedef, flat)

    @eqx.filter_grad
    def grad_loss(symbolic_model):
        vmap_model = jax.vmap(symbolic_model, in_axes=(None, 0))
        pred_ys = vmap_model(ts, ys[:, 0])  # noqa: F821
        return jnp.mean((ys - pred_ys) ** 2)  # noqa: F821

    optim = optax.adam(fine_tuning_lr)
    opt_state = optim.init(eqx.filter(symbolic_model, eqx.is_inexact_array))

    @eqx.filter_jit
    def make_step(symbolic_model, opt_state):
        grads = grad_loss(symbolic_model)
        updates, opt_state = optim.update(grads, opt_state)
        symbolic_model = eqx.apply_updates(symbolic_model, updates)
        return symbolic_model, opt_state

    for _ in range(fine_tuning_steps):
        symbolic_model, opt_state = make_step(symbolic_model, opt_state)

    #
    # Finally we round each constant to the nearest multiple of `quantise_to`.
    #

    trained_expressions = []
    for module, expression in zip(symbolic_model.func.mlp.modules, expressions):
        expression = replace_parameters(expression, module.parameters.tolist())
        expression = quantise(expression, quantise_to)
        trained_expressions.append(expression)

    print(f"Expressions found: {trained_expressions}")
```

```python
main()
```


# Tutorial on Active Inference with `pymdp`

This set of 3 tutorial notebooks aims to be an accessible introduction to discrete-state-space active inference modelling with the `pymdp` package. We assume no prerequisites other than a good grasp of Python and some basic mathematical knowledge (specifically some familiarity with probability and linear algebra). We assume no prior knowledge of active inference. Hopefully, by the end of this series of notebooks, you will understand active inference well enough to understand the recent literature, as well as implement your own agents!

These tutorials will walk you through the specification, and construction of an active inference agent which can solve a simple navigation task in a 2-dimensional grid-world environment. The goal here is to implement the agent 'from scratch'. Specifically, instead of just using magical library functions, we will show you an example of how these functions could be implemented from pure Python and numpy code. The goal at the end of these tutorials is that you understand at a fairly detailed level how active inference works in discrete state spaces and how to apply it, as well as how you could implement a simple agent without using the `pymdp` package. Once you understand what the `pymdp` package aims to abstract, we will go through the structure and functionality offered by the package, and show how you can construct complex agents in a simple and straightforward way.

# What is Active Inference?

Fundamentally, the core contention of active inference is that the brain (and agents in general) can be thought of as fundamentally performing (Bayesian) inference about the world. Specifically, an agent performs two functions.

1.) **Perception**. An agent does not necessarily know the true state of the world, but instead must infer it from a limited set of (potentially ambiguous) observations.

2.) **Action**. Typically, the agent can also perform the actions which change the state of the world. The agent can use these actions to drive the world towards a set of states that it desires.

The theory of Active Inference argues that **both** perception and action can be represented and solved as Bayesian inference problems.

# What is Bayesian Inference?

Bayesian inference provides a recipe for performing *optimal* inference. That is, if you have some set of Hypotheses $H$ and some set of data $D$, then Bayesian inference allows you to compute the *best possible* update to your hypotheses given the data that you have. In other words, the best explanation $H$ that accounts for your actual data $D$.

For instance, suppose you are a trader and you want to know whether some stock will go up tomorrow. And you have a set of information about that stock (for instance earnings reports, sales data, rumours a friend of a friend told you etc). Then Bayesian inference provides the *optimal way* to estimate the probability that the stock will go up. In this scenario, our "hypotheses" $H$ is that the stock will go up, or it will go down and our "data" $D$ is the various pieces of information we hold.

The fundamental equation in Bayesian inference is **Bayes Rule**. Which is
$$
\begin{align}
p(H | D) = \frac{p(D | H)p(H)}{p(D)}
\end{align}
$$
Here $p(H)$ etc are *probability distributions*. All a probability distribution is is a function that assigns a probability value (between 0 and 1) to a specific outcome. A probability distribution then represents the probability of that outcome for every possible outcome. For instance, take p(H). This is the probability distribution over our *hypothesis space*, which you can think of as our baseline assumptions about whether the stock tends to go up or down i.e. $p(H) = [p(stock\_goes\_up), p(stock\_goes\_down)]$, before we've encountered any data that provides evidence for/against either hypothesis.

If we assume we have no idea whether the stock will go up or down, we can say that the probability in each case is 0.5 so that $p(H) = [0.5, 0.5]$. When there is a discrete set of possible outcomes or states, probability distributions can be simply represented as vectors with one element for each outcome - where the element itself is simply the probability of seeing that outcome. The sum of all the elements of a probability distribution must equal 1. The vector's elements encode the probabilities of *all possible events*, so one of them *must* occur -- i.e. the probability of seeing *some* outcome is 100%.

There are three fundamental quantities in Bayesian inference: the **posterior**, the **likelihood** and the **prior**. The posterior is ultimately the goal of inference, it is $p(H | D)$. What this represents is the probability of each hypothesis in the hypothesis space *given the data $D$*. You can think of as you best guesses at the truth of each hypothesis after optimally integrating the data. Next is the **prior** $p(H)$ which represents your assumptions about how likely each hypothesis is *prior to seeing the data*. Finally, there is the likelihood $p(D | H)$ which quantifies how likely each outcome you see is, given the different hypotheses. The likelihood distribution can also be thought of as a *model* for how the data relates to the hypotheses.

The key insight of Bayes rule is simply that the posterior probability -- how likely the hypothesis is, given the data -- is simply the likelihood times the prior. This multiplication can be expressed as computing the following: how likely is the data you actually saw, given the different hypotheses ($P(D = d | H)$), multiplied by the prior probability you assign to each hypothesis ($P(H)$). So the full posterior is a distribution over the different hypotheses - in the case of discrete / Categorical distributions, your posterior $p(H |D)$ will also be a vector of probabilities, e.g. $p(H | D) = [0.75, 0.25]$.

Effectively, hypotheses are more likely if they can predict the data well, but are also weighted by their a-priori probability. The marginal likelihood $p(D)$ is basically just there to normalize the posterior (i.e. make sure it equals 1).

# Generative Models

In Active Inference we typically talk about *generative models* as a core component. But what are generative models? Technically a generative model is simply the product of a likelihood and a prior, for all the possible data points $D$ and hypotheses $H$. This is also known as a *joint distribution* $p(H,D) = p(D | H)p(H)$. The generative model, then, is simply the numerator of Bayes rule, and if we normalize it we can compute posterior probabilities. It is called a generative model because it allows you to *generate* samples of the data. To do so, we follow the following steps:

1.) Sample a hypothesis $h_i$ from the prior -- i.e. $p(H)$

2.) Sample a datum $d_i$ from the likelihood distribution, given the particular hypothesis $h_i$ that you sampled i.e. sample $d_i$ from $p(D | H = h_i)$.

Another way to think about a generative model is that it is simply a model, or set of beliefs/assumptions, of how observed data are generated. This is often a very helpful way to think about inference problems, since it aligns with our notion of causality -- i.e. there are unknown processes in the world which generate data. If we can imagine a process and then imagine the data generated by this process, then we are imagining a generative model. Inference of the posterior, on the other hand, is more difficult because it goes in the *reverse* direction -- i.e. you have some set of observations and want to reconstruct the process that gave rise to them. Fundamentally, all of Bayesian statistics can be broken down into two steps:

1.) Make a mathematical model of the data-generating process - the sort of environmental / world structures you think could give rise to the sort of data you have (i.e. come up with a generative model). Generative models are classically written down using a set of unknown parameters (e.g. parameters that describe probability distributions, like sufficient statistics). You want to "fit" (read: infer the values of) these parameters, given some data.

2.) Given your generative model and some data, compute the posterior distribution over the unknown parameters using Bayes rule, or some approximation of Bayes rule.

3.) Be happy!

All the methods in Bayesian statistics essentially fall into two classes. Coming up with more expressive and powerful generative models and then figuring out algorithms to perform inference on them.

# Why is Bayesian Inference hard?

At this point, you may be wondering: Bayesian inference seems pretty easy. We known Bayes rule. We can invent generative models easily enough. Computing the posterior is just taking the generative model and dividing it by $p(D)$. Why all this fuss? Why do we need to make a whole academic field out of this anyway? Luckily this has a straightforward answer. Bayesian inference is hard for essentially just one reason: that computing $p(D)$, the normalizing constant, is hard.

Let's think about why. $p(D)$, which is known as the **marginal likelihood**, is fundamentally just the probability of the data. What does that mean? How is there just some free-floating probability of the data? Fundamentally there isn't. $p(D)$ is the probability of the data *averaged over all possible hypotheses*. We can write this as,
$$
\begin{align}
p(D) = \sum_h p(D | H)p(H)
\end{align}
$$

Effectively, $p(D)$ is the sum over all possible hypotheses of the probability of the data given that hypothesis, weighted by the prior probability of that hypothesis. This is challenging to compute in practice because you are often using really large (or indeed often infinite) hypothesis spaces. For instance, suppose your trader doesn't just want to know whether the stock will go up or down but *how much* it will go up or down. Now, with this simple change, you have an *infinite* amount of hypotheses: $p(H) = [p(stock\_goes\_up\_0.00001), p(stock\_goes\_up\_0.00002), p(stock\_goes\_up\_0.00003) ...]$. Then, if we want to compute the posterior in this case, we need to sum over every one of this infinite amount of hypotheses. You can see why this ends up being quite challenging in practice.

Because of this intrinsic difficulty, there are a large number of special case algorithms which can solve (or usually approximate) the Bayesian posterior in various cases, and a large amount of work in statistics goes into inventing new ones or improving existing methods. Active Inference agents use one special class of approximate Bayesian inference methods called *variational methods* or *variational inference*. This will be discussed in much more detail in notebook 2.

Beyond merely the difficulty of performing inference, another reason why Bayesian statistics is hard is that you often *don't know* the generative model. Or at least you are uncertain about some aspects of it. There is also a wide class of methods which let you perform inference and *learning* simultaneously, including for active inference, although they won't be covered in this tutorial.

# A Generative Model for an Agent

In Active Inference, we typically go a step beyond the simple cases of Bayesian inference described above, where we have a static set of hypotheses and some static data. We are instead interested in the case of an *agent* interacting with a *dynamic environment*. The key thing we need to add to the formalism to accomodate this is a notion of *time*. We consider an environment consisting of a set of *states* $[x_1, x_2, \dots x_t]$ evolving over time. Moreover, there is an agent which is in the environment over time. This agent receives a set of *observations* $[o_1, o_2 \dots o_t]$, which are a function of the state of the environment, and it can emit *actions* $[a_1, a_2 \dots a_t]$ which can change the state of the environment. The key step to get a handle on this situation mathematically is to define a *generative model* of it.

To start, we must make some assumptions to simplify the problem. Specifically, we assume that the *state* of the environment $x_t$ only depends on the state at the previous timestep $x_{t-1}$ and the action emitted by the agent at the previous timestep $a_{t-1}$. Then, we assume that the observation $o_t$ at a given time-step is only a function of the environmental state at the current timestep $x_t$. Together, these assumptions are often called the **Markov Assumptions** and if the environment adheres to them it is often called a **Markov Decision Process**.

The general computational "flow" of a Markov decision process can be thought of as following a sequence of steps.

1.) The state of the environment is $x_t$.

2.) The environment state $x_t$ generates an observation $o_t$.

3.) The agent receives an observation $o_t$, and based on it (or inferences derived thereof) decides to take some action $a_t$, which it executes in the environment.

4.) Given the current state $x_t$ and the agent's action $a_t$, the environment updates its own state to produce $x_{t+1}$

5.) Go back to step 1.

Now that we have this series of steps, we can try to define what it means mathematically. Specifically, we need to define two quantities.

a.) We need to know how the state of the environment $x_t$ is reflected in the observation sent to the agent $o_t$. In Bayesian terms from the agent's perspective, the true state of the environment is unknown, and its various possible states can be thought of as "hypotheses", while the observations it receives are "data". The agent's generative model encodes some prior assumptions about each possible state $x_t$ (i.e. each hypothesis) relates to the probability of seeing each possible outcome $o_t$. This relationship (from the agent's perspective) is a function known as the **likelihood distribution** $p(D | H)$ or, in our new notation $p(o_t | x_t)$.

b.) We need to know how the environment updates itself to form its new state $x_{t+1}$, given the old one $x_t$ and the action $a_t$. This can be thought of (from the perspective of the agent who does not observe it) as the **prior** since it can be thought of as the default expectation the agent can have about the state of the environment prior to receiving any observations. It can be written as $p(x_t | x_{t-1}, a_{t-1})$. This distribution is also known as the **transition distribution** since it specifies (the agent's assumptions about) how the environment transitions from one state to another.

These two distributions $p(o_t | x_t)$ and $p(x_t | x_{t-1}, a_{t-1})$ are all that is needed to specify the evolution of the environment. At this point, it is necessary to make a distinction between the *actual* evolution of the environment -- known as the **generative process** -- and the *agent's model* of the evolution of the environment, which is known as the **generative model**. These are not necessarily the same, although in some sense the goal of the agent is to figure out a generative model that is as close to the true generative process as possible. In this example, we will consider a scenario in which the agent knows the true model, so the generative model and the generative process are the same, although this is not always the case.

To make this concrete, all that is necessary to do is to specify precisely what the **likelihood** and **transition** distributions actually are.

In the case of discrete state spaces, where it is possible to explicitly enumerate all states, a very generic way of representing these distributions is as *matrices*. Specifically, we can represent the likelihood distribution as a matrix denoted $\textbf{A}$, which is of shape $dimension\_of\_observation \times dimension\_of\_state$. The element $A_{i,j}$ of $\textbf{A}$ represents the probability that observation $i$ is generated by state $j$.

Secondly, we can represent the transition distribution by a matrix $\textbf{B}$ of shape $state\_dim \times state\_dim \times action\_dim$ where element $\textbf{B}_{i,j,k}$ represents the probability of the environment moving to state $i$ given that it was previously in state $j$ and that action $k$ was taken.

In the rest of this notebook, we will explicitly write down the generative process / generative model for a simple grid-world environment in code, to get a better handle on how the environment and the agent's model is specified. In the next notebook, we will turn to inference and action selection and discuss how active inference solves these two tricky problems.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
```

## Constructing a Generative Model

For the rest of this notebook, we will construct a simple generative model for an active inference agent navigating a 3x3 grid world environment. The agent can perform one of 5 movement actions at each time step: `LEFT, RIGHT, UP, DOWN, STAY`. The goal of the agent is to navigate to its preferred position.

We will create matrices for both the environment as well as the agent itself. As we go up levels of abstraction, these environment and generative model matrices will be imported from classes - but this notebook is the lowest level representation of construction, to show how everything is built from the ground up.

## Understanding the state space

The first thing to note is that we are in a 3x3 grid world which means we have 9 states in total. We can define the following mapping to better understand the space.

```python
state_mapping = {0: (0,0), 1: (1,0), 2: (2,0), 3: (0,1), 4: (1,1), 5:(2,1), 6: (0,2), 7:(1,2), 8:(2,2)}
```

All we're doing with this mapping dictionary is assigning a particular index (`0`, `1`, `2`, ..., `8`) to each grid position, which is defined as a pair of `(x, y)` coordinates ( `(0, 0), (1, 0)`, ..., `(2, 2)`). We will use the linear indices to refer to the grid positions in our probability distributions (e.g. `P(o_t | x_t = 5)`), so this `state_mapping` will allow us to easily move between these linear indices and the grid world indices. These kinds of mappings are very handy for intuition and visualization.

And the following heatmap just represents how the coordinates map to the real grid space

```python
grid = np.zeros((3,3))
for linear_index, xy_coordinates in state_mapping.items():
    x, y = xy_coordinates
    grid[y,x] = linear_index # rows are the y-coordinate, columns are the x-coordinate -- so we index into the grid we'll be visualizing using '[y, x]'
fig = plt.figure(figsize = (3,3))
sns.set(font_scale=1.5)
sns.heatmap(grid, annot=True,  cbar = False, fmt='.0f', cmap='crest')
```

## Likelihood Matrix: A

The likelihood matrix represents $P(o_t | x_t)$ , the probability of an observation given a state. In a grid world environment, the likelihood matrix of the agent is identical to that of the environment. It is simply the identity matrix over all states (in this case 9 states, for a 3x3 grid world) which represents the fact that the agent has probability 1 of observing that it is occupying any state x, given that it is in state x. This just means that the agent has full transparency over its own location in the grid.

```python
A = np.eye(9)
```

```python
A
```

    array([[1., 0., 0., 0., 0., 0., 0., 0., 0.],
           [0., 1., 0., 0., 0., 0., 0., 0., 0.],
           [0., 0., 1., 0., 0., 0., 0., 0., 0.],
           [0., 0., 0., 1., 0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 1., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0., 1., 0., 0., 0.],
           [0., 0., 0., 0., 0., 0., 1., 0., 0.],
           [0., 0., 0., 0., 0., 0., 0., 1., 0.],
           [0., 0., 0., 0., 0., 0., 0., 0., 1.]])

We can also plot the likelihood matrix as follows:

```python
labels = [state_mapping[i] for i in range(A.shape[1])]
def plot_likelihood(A):
    fig = plt.figure(figsize = (6,6))
    ax = sns.heatmap(A, xticklabels = labels, yticklabels = labels, cbar = False)
    plt.title("Likelihood distribution (A)")
    plt.show()

```

```python
plot_likelihood(A)
```

## Transition matrix: B

The transition matrix determines how the agent can move around the gridworld given each of the 5 available actions (UP, DOWN, LEFT, RIGHT, STAY).
So the transition matrix will be a 9x9x5 matrix, where each entry corresponds to an end state, a starting state, and the action that defines that specific transition.

To construct this matrix, we have to understand that when the agent is at the edges of the grid, it cannot move outward, so trying to move right at the right wall will cause the agent to stay still.

We will start by constructing a dictionary which we call P, which maps each state to its next state given an action

```python
state_mapping
```

    {0: (0, 0),
     1: (1, 0),
     2: (2, 0),
     3: (0, 1),
     4: (1, 1),
     5: (2, 1),
     6: (0, 2),
     7: (1, 2),
     8: (2, 2)}

```python
P = {}
dim = 3
actions = {'UP':0, 'RIGHT':1, 'DOWN':2, 'LEFT':3, 'STAY':4}

for state_index, xy_coordinates in state_mapping.items():
    P[state_index] = {a : [] for a in range(len(actions))}
    x, y = xy_coordinates

    '''if your y-coordinate is all the way at the top (i.e. y == 0), you stay in the same place -- otherwise you move one upwards (achieved by subtracting 3 from your linear state index'''
    P[state_index][actions['UP']] = state_index if y == 0 else state_index - dim

    '''f your x-coordinate is all the way to the right (i.e. x == 2), you stay in the same place -- otherwise you move one to the right (achieved by adding 1 to your linear state index)'''
    P[state_index][actions["RIGHT"]] = state_index if x == (dim -1) else state_index+1

    '''if your y-coordinate is all the way at the bottom (i.e. y == 2), you stay in the same place -- otherwise you move one down (achieved by adding 3 to your linear state index)'''
    P[state_index][actions['DOWN']] = state_index if y == (dim -1) else state_index + dim

    ''' if your x-coordinate is all the way at the left (i.e. x == 0), you stay at the same place -- otherwise, you move one to the left (achieved by subtracting 1 from your linear state index)'''
    P[state_index][actions['LEFT']] = state_index if x == 0 else state_index -1

    ''' Stay in the same place (self explanatory) '''
    P[state_index][actions['STAY']] = state_index
```

```python
P
```

    {0: {0: 0, 1: 1, 2: 3, 3: 0, 4: 0},
     1: {0: 1, 1: 2, 2: 4, 3: 0, 4: 1},
     2: {0: 2, 1: 2, 2: 5, 3: 1, 4: 2},
     3: {0: 0, 1: 4, 2: 6, 3: 3, 4: 3},
     4: {0: 1, 1: 5, 2: 7, 3: 3, 4: 4},
     5: {0: 2, 1: 5, 2: 8, 3: 4, 4: 5},
     6: {0: 3, 1: 7, 2: 6, 3: 6, 4: 6},
     7: {0: 4, 1: 8, 2: 7, 3: 6, 4: 7},
     8: {0: 5, 1: 8, 2: 8, 3: 7, 4: 8}}

From here, we can easily construct the transition matrix

```python
num_states = 9
B = np.zeros([num_states, num_states, len(actions)])
for s in range(num_states):
    for a in range(len(actions)):
        ns = int(P[s][a])
        B[ns, s, a] = 1

```

B is a very large matrix, we can see its shape below, which is as expected:

```python
B.shape
```

    (9, 9, 5)

We can also visualize B on the plots below. The x axis is the starting state, and the y axis is the ending state, and each plot corresponds to an action given by the title.

```python
fig, axes = plt.subplots(2,3, figsize = (15,8))
a = list(actions.keys())
count = 0
for i in range(dim-1):
    for j in range(dim):
        if count >= 5:
            break
        g = sns.heatmap(B[:,:,count], cmap = "OrRd", linewidth = 2.5, cbar = False, ax = axes[i,j], xticklabels=labels, yticklabels=labels)
        g.set_title(a[count])
        count +=1
fig.delaxes(axes.flatten()[5])
plt.tight_layout()
plt.show()

```

Now our generative model and environment are set up, and we can move on to Notebook 2, where we go through the core mechanics of how to perform inference and planning on this environment with this generative model.


# Calculando o $\mathcal{R}_0$ para o Modelo SEIAHR
Neste notebook vamos aprender a como utilizar o méto da matriz de próxima geração para calcular o $\mathcal{R}_0$.
## SEIAR Model for COVID-19

Neste notebook está implementado um modelo SEIAR com quarentena implícita. Que inclui também assintomáticos e hospitalizações. Veja o Suplemento 2 para maiores detalhes sobre este modelo.

Seja $\lambda=\beta(I+A)$ a força de infeção da doença.

\begin{align}
\frac{dS}{dt}&=-\lambda (1-\chi) S\\
\frac{dE}{dt}&= \lambda (1-\chi) S-\alpha E\\
\frac{dI}{dt}&= (1-p)\alpha E - \delta I\\
\frac{dA}{dt}&= p\alpha E - \delta A\\
\frac{dH}{dt}&= \phi \delta I -(\rho+\mu) H\\
\frac{dR}{dt}&= (1-\phi)\delta I + \rho H+\delta A
\end{align}

onde $\chi$ é a fração dos suscetíveis em quarentena (distanciamento social)

```python
import sympy
```

```python
#%typeset_mode True
%display typeset
```

```python
var('S E I A H Lambda beta chi alpha p phi delta rho mu')
```

Para esta análise podemos ignorar o compartimento dos Recuperados pois é desacoplado da dinâmica.

```python
Lambda = beta*(I+A)
dsdt = -Lambda*(1-chi)*S
dedt = Lambda*(1-chi)*S - alpha*E
didt = (1-p)*alpha*E -delta*I
dadt = p*alpha*E - delta*A
dhdt = phi*delta*I-(rho+mu)*H
```

```python

```

```python

```

## Equilibrios
Naturalmente para este sistema só conseguimos achar o equilíbrio livre-de-doença.

```python
solve([dsdt,dedt,didt,dadt,dhdt],[S,E,I,A,H])
```

## Matriz Jacobiana

```python
jack=jacobian([dsdt,dedt,didt,dadt,dhdt],[S,E,I,A,H])
jack
```

```python
cp = jack.characteristic_polynomial()
cp
```

```python

```

```python
jack.eigenvalues()
```

## Calculo do $\mathcal{R}_0$
O método que iremos utilizar foi proposto por P. van den Driessche no seguinte artigo:

[Reproduction numbers and sub-threshold endemic equilibria for compartmental models of disease transmission](https://pdfs.semanticscholar.org/3cf7/1968a86800215b4e129ec3eda67520832cf0.pdf)

 Seja $X=(x_1,\ldots, x_n)^t$, tal que $x_i\geq 0$, o número de indivíduos em cada compartimento.

Vamos definir $X_s$ como o conjunto de todos os estados livres de doença.

$$X_s=\{x \geq 0|x_i=0, i=1\ldots,m\}$$

Supõe-se que cada função é continuamente diferenciável pelo menos duas vezes ($C^2$) em cada variável. As equações são reordenadas para que as $m$ primeiras equação sejam aquelas que contém infectados. Seja ${\cal F}_i(x)$ a taxa de aparecimento de novas infecções no compartimento $i$, ${\cal V}_i^+(x)$ a taxa de entrada de indivíduos no compartimento $i$ por outros meios e ${\cal V}_i^-(x)$ a taxa de saída de indivíduos do compartimento $i$. O modelo de transmissão da doença consiste em condições iniciais não negativas juntamente com o seguinte sistema de equações:

$\dot{x}=f_i(x)={\cal F}_i(x)-{\cal V}_i(x), i=1\ldots, n$

onde, ${\cal V}_i (x) = {\cal V}_i(x)^{-} - {\cal V}_i(x)^+$ e as funções satisfazem os pressupostos  (A1) - (A5) descritos abaixo. Desde que cada função representa uma transferência dirigida de indivíduos, todos elas são não-negativos.

(A1) Se $x \geq 0 $, então ${\cal F}_i, {\cal V}_i^+, {\cal V}_i ^- \geq 0$ para $i=1, \ldots, n$

ou seja, se um compartimento estiver vazio, não pode haver saída de indivíduos deste, por morte, infecção ou qualquer outro meio.

(A2) Se $x_i=0$ então ${\cal V}_i^-(x)=0$. Em particular, se $x \in X_s$, então ${\cal V}_i^-(x)=0$ para $i=1,\ldots, m$

(A3) ${\cal F}_i=0$ se $i>m$

(A4) Se $x \in X_S$, então ${\cal F}_i(x) = 0$ e ${\cal V}_i^+(x)=0$ para $i=1,\ldots, m$

(A5) Se ${\cal F}(x)$ é um vetor nulo, então todos os autovalores de $Df(x_0)$ tem parte real negativa.

 ---

Para calcular o $R_0$ é importante distinguir as novas infecções de todas as outras mudanças na população. No modelo proposto, os compartimentos que correspondem aos indivíduos infectados são $E$, $I$ E $A$, portanto, ${\bf m=3}$. A fim de clareza, vamos ordenar os $n=5$ compartimentos da seguinte forma: $[E, I, A, H, S]$, separando os $m$ primeiros compartimentos do restante. Vale ressaltar que as transferências dos compartimentos expostos para os infectados e Assintomáticos e de infectados para Hospitalizados não são consideradas novas infecções, mas sim a progressão de um indivíduo infectado através dos vários compartimentos. Portanto,

$$ {\cal F}_i(x): \text{ taxa de surgimento de novos infectados no compartimento } i $$

$$ {\cal F} =\begin{bmatrix}
\Lambda(1-\chi) S\\
0\\
0\\
\end{bmatrix} $$

Onde, ${\color{red}\Lambda=\beta(I+A)}$.

```python
F_cal = matrix([[Lambda*(1-chi)*S],[0],[0]])
F_cal
```

Além disso, temos

$$ {\cal V}_i(x)^-: \text{ taxa de saída do compartimento } i $$

$$ {\cal V}_i(x)^+: \text{ taxa de entrada do compartimento } i $$

Logo,

$$
\begin{equation}
{\cal V^-} = \begin{bmatrix}
\alpha E\\
\delta I\\
\delta A
\end{bmatrix}
\end{equation}
$$
$$
\begin{equation}
\qquad {\cal V^+} = \begin{bmatrix}
0\\
(1-p) \alpha E\\
p \alpha E\\
\end{bmatrix}
\end{equation}
$$

```python
V_cal_neg = matrix([[alpha*E],[delta*I],[delta*A]])
V_cal_neg
```

```python
V_cal_pos = matrix([[0],[(1-p)*alpha*E],[p*alpha*E]])
V_cal_pos
```

$${\cal V}_i (x) = {\cal V}_i(x)^{-} - {\cal V}_i(x)^+$$

Então,
\begin{equation}
{\cal V} =
\begin{bmatrix}
\alpha E\\
(p-1)\alpha E+\delta I\\
-p\alpha E+ \delta A\\
\end{bmatrix}
\end{equation}

```python
V_cal = V_cal_neg-V_cal_pos
V_cal
```

Definimos também $F=\left[\frac{\partial {\cal F}_i (x_0)}{\partial x_j}\right]$ e $V=\left[\frac{\partial {\cal V}_i (x_0) }{\partial x_j}\right]$, onde $x_0$ é um DFE e $1\leq i,j \leq m$.

Isto equivale à jacobiana  destas duas matrizes, após substituir $x_0$ ou seja, $S=1$.

```python
F = jacobian(F_cal(S=1),[E,I,A])
F
```

```python
V = jacobian(V_cal(S=1),[E,I,A])
V
```

${\cal R}_0 = \rho (FV^{-1})$

```python
M= F*V.inverse()
```

```python
M=M.simplify_full()
M
```

Since only the first row of $M$ is non-zero, all but one of the eigenvalues is $0$ and the dominant eigenvalue or spectral radius is $M_{11}$, thus

```python
R0=M[0,0].simplify_full()
R0
```

```python
R0.variables()
```

```python
show('$R_t=$')
Rt = M[0,0]
Rt
```

```python
print(Rt)
```

```python
plot(R0(chi=0.3,phi=.1,rho=.6,delta=.1,alpha=.3, p=.75),(beta,0.01,0.32))
```

```python
R0(chi=0.1,phi=.1,rho=.6,delta=.1,alpha=10, p=.75,beta=.5)
```

```python
plot(Rt(chi=0.3,phi=.1,rho=.6,delta=.1,alpha=10,beta=.2, p=.75),(S,0,1))
```

```python

```

```python

```

```python

```

Planilhas Sage/Aula 9 - Calculo do R0.ipynb


# [HW12] Logistic Regression

우리는 지금까지 주어진 데이터와 가장 잘 맞는 직선을 찾는 linear regression을 진행하였습니다. 하지만 예측 값이 연속적인 값을 갖지 않는다면 어떻게 할까요?

가장 대표적인 것은 binary classification 입니다. 이는 종류가 2개로 나뉘어진 데이터가 있고 이를 분류하는 문제입니다. 수많은 메일들 중에서 스팸인 메일과 스팸이 아닌 메일을 분류하는 것이 하나의 예시입니다. 이 문제는 예측 값이 연속적인 값이 아닌 0 또는 1입니다.

이때는 우리의 예측 값을 확률 값으로 만든 다음에 확률 값이 우리의 기준보다 높으면 1, 아니면 0으로 분류합니다.

이러한 문제를 푸는 방법을 logistic regression이라고 합니다.

## 1.1 Logistic function

Logistic regression을 진행하기 위해서는 출력 값을 0과 1의 값으로 맞춰주어야 합니다. 이를 위해서 우리는 **logistic function** 을 사용합니다. Logistic function은 다음과 같습니다.

$$\sigma(z) = \frac{1}{1 + e^{-z}}$$

Logistic regression을 진행할 때 입력 데이터를 $x$, 실제 class 값을 $y$, 예측된 출력 값을 $\hat{y}$라고 하면 $x$는 두가지 변환을 거쳐서 $\hat{y}$가 됩니다.
$$z = wx + b$$
$$\hat{y} = \sigma(z)$$
우리의 목표는 $\hat{y}$가 실제 $y$와 가장 가깝게 되도록 하는 $w$와 $b$를 찾는 것 입니다.

Logistic function을 코드를 통해서 알아보도록 하겠습니다.

```python
import sympy
import numpy

from matplotlib import pyplot
%matplotlib inline
```

```python
z = sympy.Symbol('z', real=True)

logistic = 1/(1+ sympy.exp(-z))
logistic
```

$\displaystyle \frac{1}{1 + e^{- z}}$

```python
sympy.plotting.plot(logistic);
```

위 그래프를 보면 $z=0$ 일 때 출력 값이 0.5가 됩니다. 그리고 양수 일 때는 1에 가까워지고 음수일 때는 0으로 가까워지게 됩니다. 이렇게 $z$값을 0과 1 사이로 표현할 수 있게 되었습니다.

이제 데이터를 직접 만들어서 진행해 보도록 하겠습니다.

```python
# synthetic data
x_data = numpy.linspace(-5, 5, 100)
w = 2
b = 1
numpy.random.seed(0)
z_data = w * x_data + b + numpy.random.normal(size=len(x_data))
y_data = 1 / (1+ numpy.exp(-z_data))

pyplot.scatter(x_data, y_data, alpha=0.4);
```

이번에도 [HW10]에서 진행한 것과 같이 조금의 noise를 추가해서 데이터를 생성하였습니다.

이제 실제 class 값을 정해주어야 합니다. Numpy 패키지 안의 where 함수로 0.5 보다 큰 값을 1, 작은 값을 0으로 class를 부여해 주겠습니다.

```python
y_data = numpy.where(y_data >= 0.5, 1, 0)
pyplot.scatter(x_data, y_data, alpha=0.4);
```

## 1.2 Logistic loss function

$\hat{y}$가 실제 $y$와 가장 가깝게 되도록 하는 $w$와 $b$를 찾으려면 우리는 cost function을 정의해야 합니다.

Linear regression 문제를 해결할 때는 mean square error를 사용했습니다.

하지만 logistic regression에 적용하면 문제가 생깁니다.

기존의 linear regression에서의 mean square error 에서는

$$\frac{1}{n} \sum_{i=1}^n (y_i - (wx_i + b))^2$$

의 형태를 이루고 있어서 convex 한 형태를 이루고 있었습니다.

그런데 logistic function을 포함한 logistic regression에서는

$$\frac{1}{n} \sum_{i=1}^n (y_i - \sigma(wx_i + b))^2$$

$\sigma$인 logistic function 때문에 더이상 convex 한 형태가 아닙니다. 예시를 통해서 왜 convex가 아닌지 알아보겠습니다.

간단한 예시를 위해 $w = 1, b=0$일 때 3개의 데이터를 통해서 알아보겠습니다.

$(x, y) : (-1, 2), (-20, -1), (-5, 5)$ 일 때 cost function을 그래프로 나타내면 다음과 같습니다.

```python
badloss = (2 - 1/(1+ sympy.exp(-z)))**2 + \
          (-1 - 1/(1+ sympy.exp(-20*z)))**2  + \
          (5 - 1/(1+ sympy.exp(-5*z)))**2
badloss
```

$\displaystyle \left(-1 - \frac{1}{1 + e^{- 20 z}}\right)^{2} + \left(2 - \frac{1}{1 + e^{- z}}\right)^{2} + \left(5 - \frac{1}{1 + e^{- 5 z}}\right)^{2}$

```python
sympy.plotting.plot(badloss, xlim=(-1,1));
```

Gradient descent 방식으로 위 cost function의 최솟값을 구하게 되면 중간에 기울기가 0인 지점에서 멈추게 되고, 우리는 원하는 가장 작은 값에 도달하지 못하고 local minimum에 도달하게 됩니다.

그래서 mean square error말고 다른 방법을 찾기 위해 cost function의 의미를 다시 한번 생각해 보겠습니다.

우리가 어떤 값을 예측할 때 많이 틀렸다면, 예측하는데 쓰인 변수들을 많이 바꾸어야 합니다. 그에 비해 조금 틀렸다면, 이미 잘 예측하고 있기 때문에 변수들을 조금 바꾸어야 합니다. 많이 바꾸고, 조금 바꾸는 것은 기울기의 크기가 정합니다. 이러한 원리를 사용해서 linear regression에서는 square error를 쓰는 것입니다.

이 원리를 logistic regression에도 적용해 보겠습니다.

$z = wx + b$ 일 때 cost function $L$을 b에 대해서 미분을 해보겠습니다. Chain rule을 사용하면 다음과 같게 됩니다.

$$\frac{\partial{L}}{\partial{b}} = \frac{\partial{L}}{\partial{\sigma(z)}} \frac{\partial{\sigma(z)}}{\partial{z}}\frac{\partial{z}}{\partial{b}}$$

이 때 $\frac{\partial{z}}{\partial{b}}$는 1이 돼서 사라집니다.

이제 $\frac{\partial{\sigma(z)}}{\partial{z}}$에 대해서 알아보겠습니다.

```python
lprime = logistic.diff(z)
lprime
```

$\displaystyle \frac{e^{- z}}{\left(1 + e^{- z}\right)^{2}}$

위에서 나온 $\sigma'(z)$를 정리를 해보겠습니다.

$$\frac{\partial{\sigma(z)}}{\partial{z}} = \frac{e^{-z}}{(1+e^{-z})^2} = \frac{1}{1+e^{-z}} \times \frac{e^{-z}}{1+e^{-z}} = \sigma(z)\times \frac{e^{-z}}{1+e^{-z}} $$
가 되고, 여기서 $\frac{e^{-z}}{1+e^{-z}} $를 다시 정리해보면 다음과 같습니다.

$$\frac{e^{-z}}{1+e^{-z}} = \frac{1 + e^{-z} -1}{1 + e^{-z}} = 1 - \frac{1}{1+e^{-z}} = 1-\sigma(z)$$

결론적으로,
$$\sigma'(z) = \sigma(z) ( 1 - \sigma(z))$$
가 됩니다.

다시한번 위 식을 정리해보면 다음과 같습니다.

$$\frac{\partial{L}}{\partial{b}} = \frac{\partial{L}}{\partial{\sigma(z)}} \frac{\partial{\sigma(z)}}{\partial{z}} = \frac{\partial{L}}{\partial{\sigma(z)}} \sigma(z) (1-\sigma(z))$$

여기서 우리는 $\frac{\partial{L}}{\partial{b}}$의 값이 예측 값과 실제 값의 차이가 클수록 크고, 작을수록 작게 하고 싶기 때문에 $\frac{\partial{L}}{\partial{b}} = (y - \sigma(z))$로 정의하겠습니다. 정리하면 다음과 같습니다.

$$\frac{\partial{L}}{\partial{\sigma(z)}} = \frac{(y-\sigma(z))}{\sigma(z)(1-\sigma(z))}$$

이제 위 식을 $L$에 대해서 코드를 통해서 정리해보겠습니다. 코드를 간단히 하기 위해서 $\sigma(z) = a$라고 치환해서 생각하겠습니다.

```python
a, y = sympy.symbols('a y', real=True)
```

```python
dLda = (y-a)/a/(1-a)
dLda
```

$\displaystyle \frac{- a + y}{a \left(1 - a\right)}$

```python
L = sympy.integrate(dLda, a)
L
```

$\displaystyle y \log{\left(a \right)} + \left(1 - y\right) \log{\left(a + \frac{1 - 2 y}{2 y - 1} \right)}$

```python
sympy.simplify(L)
```

$\displaystyle y \log{\left(a \right)} - \left(y - 1\right) \log{\left(a - 1 \right)}$

여기서 $a = \sigma(z)$이기 때문에 $ a < 1 $이 됩니다. 그래서 $\log$ 안의 값이 음수가 되면 안되기 때문에 식을 변환해 줍니다.

```python
L = -y*sympy.log(a) + (y-1)*sympy.log(1-a)
L
```

$\displaystyle - y \log{\left(a \right)} + \left(y - 1\right) \log{\left(1 - a \right)}$

우리가 구한 cost function $L$은
$$L = -y \log(a) + (y-1)\log(1-a)$$이 됩니다.

이제 실제로 차이가 클 때 $L$값이 커지는지 알아보도록 하겠습니다.

먼저 만약 $y=1$이라면 $L = -\log(a)$만 남게 됩니다. 그래프로 표현하면 다음과 같습니다.

```python
sympy.plotting.plot(-sympy.log(a), xlim=(0,1));
```

실제 class 가 1일 때 예측 값이 0에 가까워지면 cost function값이 커지고, 1에 가까워지면 cost function이 작아지는 것을 알 수 있습니다. 이는 우리가 원래 목표했던 것과 동일합니다.

이제 $y=0$이라면 $L = \log(1-a)$ 만 남게 됩니다. 이를 또한 그래프로 표현하면 다음과 같습니다.

```python
sympy.plotting.plot(-sympy.log(1-a), xlim=(0,1));
```

이번에도 예측값이 실제 값이랑 가까워지면 cost function값이 작아지고 멀어지면 커지게 됨을 알 수 있습니다.

## 1.3 Find the parameters using autograd function

이제 logistic regression의 전체적인 과정을 코드를 통해서 알아보도록 하겠습니다.

```python
logistic
```

$\displaystyle \frac{1}{1 + e^{- z}}$

```python
w, b, x, y = sympy.symbols('w b x y')
logistic = 1/(1+ sympy.exp(-w*x-b)) # redefined with the composition

Loss = -y*sympy.log(logistic) - (1-y)*sympy.log(1-logistic)
Loss
```

$\displaystyle - y \log{\left(\frac{1}{e^{- b - w x} + 1} \right)} - \left(1 - y\right) \log{\left(1 - \frac{1}{e^{- b - w x} + 1} \right)}$

지금까지 diff 를 통해서 기울기를 구했습니다. 그런데 식이 복잡해질수록 속도가 느려지기 때문에 autograd를 통해서 구해보겠습니다.

```python
# import the autograd-wrapped version of numpy
from autograd import numpy
```

```python
# import the gradient calculator
from autograd import grad
```

```python
# note: the namespace numpy is the autograd wrapper to NumPy

def logistic(z):
    '''The logistic function'''
    return 1 / (1 + numpy.exp(-z))

def logistic_model(params, x):
    '''A prediction model based on the logistic function composed with wx+b
    Arguments:
       params: array(w,b) of model parameters
       x :  array of x data'''
    w = params[0]
    b = params[1]
    z = w * x + b
    y = logistic(z)
    return y

def log_loss(params, model, x, y):
    '''The logistic loss function
    Arguments:
       params: array(w,b) of model parameters
       model:  the Python function for the logistic model
       x, y:   arrays of input data to the model'''
    y_pred = model(params, x)
    return -numpy.mean(y * numpy.log(y_pred) + (1-y) * numpy.log(1 - y_pred))
```

```python
# get a function to compute the gradient of the logistic loss
gradient = grad(log_loss)
```

```python
type(gradient)
```

    function

이 떄 grad 함수는 변수 개수만큼 output을 만들게 됩니다. 우리는 $w, b$ 2개의 변수가 있습니다. 먼저 예시로 랜덤으로 초기화한 변수를 넣어서 기울기값을 구해보겠습니다.

```python
numpy.random.seed(0)
params = numpy.random.rand(2)
print(params)
```

    [0.5488135  0.71518937]

```python
gradient(params, logistic_model, x_data, y_data)
```

    array([-0.42734877,  0.08274066])

이렇게 2개의 변수에 대해서 각각 기울기 값을 반환해줍니다.

이번에 gradient descent 를 진행할 때는 새로운 조건을 추가해서 진행하겠습니다. 우리가 정한 반복 수 외의 기울기 값이 0에 가까워지면 더이상 반복을 하지 않는 조건을 추가하겠습니다. 0에 가까운 값을 설정한 뒤 그것보다 작아지면 while 문이 멈추도록 설정하여서 gradient descent 를 진행하겠습니다.

```python
max_iter = 5000
i = 0
descent = numpy.ones(len(params))

while numpy.linalg.norm(descent) > 0.001 and i < max_iter:

    descent = gradient(params, logistic_model, x_data, y_data)
    params = params - descent * 0.01
    i += 1

print('Optimized value of w is {} vs. true value: 2'.format(params[0]))
print('Optimized value of b is {} vs. true value: 1'.format(params[1]))
print('Exited after {} iterations'.format(i))

pyplot.scatter(x_data, y_data, alpha=0.4)
pyplot.plot(x_data, logistic_model(params, x_data), '-r');
```

빨간색 곡선이 우리의 모델입니다.

이제 기준값을 정하고 그것보다 크면 1, 작으면 0으로 분류를 하면 됩니다.

이번에는 0.5로 설정해서 진행하겠습니다.

```python
def decision_boundary(y):
    return 1 if y >= .5 else 0
```

모든 점을 함수에 넣어야 하는데 하나씩 넣으면 반복문을 돌아야해서 오래걸리기 때문에 numpy의 vectorize 함수를 사용하겠습니다.

```python
decision_boundary = numpy.vectorize(decision_boundary)
```

```python
def classify(predictions):
    '''
    Argument:
    predictions, an array of values between 0 and 1

    Returns:
    classified, an array of 0 and 1 values'''

    return decision_boundary(predictions).flatten()

pyplot.scatter(x_data, y_data, alpha=0.4,
               label='true value')
pyplot.scatter(x_data, classify(logistic_model(params, x_data)), alpha=0.4,
               label='prediciton')

pyplot.legend();
```

거의 모든 데이터들을 정확하게 예측한 것을 알 수 있습니다.

이번 시간에는 직접 데이터를 만들어서 간단한 logistic regression 문제를 해결해 보았습니다.

질문 있으면 편하게 해주세요~

```python

```