# Survial Models

## Time to event data

Often, we are interested in determining how some variable or intervention affects the time to occurence of a certain outcome. For example, a researcher may be interested in how a particular drug affects how long a person will survive with a particular disease. We have already seen how the Bayesian paradigm is able to model binary events. Now, we wish to go a step further and examine the waiting time for the occurence of such events.

Recall that the Bayesian paradigm consists of specifying a probability model for the observed data $D$, given an unknown vector of parameters leading to a likelihood function $L(\theta|D)$. We then assume that the parameters are random and follow some prior distribution $\pi(\theta)$. Inference for $\theta$ is carried out by way of the posterior distribution obtained by Bayes' theorem:

$$
\pi(\theta|D)\propto L(\theta|D)\pi(\theta).
$$

In survival analysis, the likelihood is somewhat more involved in its formulation. To be able to properly discuss the likelihood specification in survival models, however, we must first discuss several key quantities in survival analysis. As discussed our main outcome of interest is time, which has support $0\leq t< \infty$. We denote our time as with a random variable $T$.

## Important quantities in Survival Analysis

If the random variable $T$ is the time to the event we are studying, we can make probability statements about the lifetime of individuals. For example, we can calculate the probability of surviving past a particular time $t$ (i.e. $P(T>t)$). This probability, known as the survival function is the primary object of interest in survival analysis. The survival function is denoted by

$$S(t) = P(T > t) = 1 - F(t),$$

where $F$ is the [CDF](https://en.wikipedia.org/wiki/Cumulative_distribution_function) of $T$.  It is mathematically convenient to express the survival function in terms of the [hazard rate](https://en.wikipedia.org/wiki/Survival_analysis#Hazard_function_and_cumulative_hazard_function), $h(t)$.  The hazard rate is the instantaneous probability that the event occurs at time $t$ given that it has not yet occured.  That is,

$$\begin{align*}
h(t)
    & = \lim_{\Delta t \to 0} \frac{P(t < T < t + \Delta t\ |\ T > t)}{\Delta t} \\
    & = \frac{f(t)}{S(t)}
\end{align*}$$

Furthermore, we have that

$$S(t) = \exp\left(-\int_0^s h(s)\ ds\right).$$

This representation of the survival function shows that the cumulative hazard function

$$H(t) = \int_0^t h(s)\ ds$$

is an important quantity in survival analysis, since we may consicesly write $S(t) = \exp(-H(t)).$

An important, but subtle, point in survival analysis is [censoring](https://en.wikipedia.org/wiki/Survival_analysis#Censoring). In many survival studies, many data points are `right censored` or rather survival times are usually only known for a portion for those under study where as the remainder are only known to exceed certain values. Specifically, an observation is said to be right censored at $c$ if the exact value of the observation is not known but only that it is greater than or equal to $c$.

### Data notation

Suppose that there are $n$ subjects under study and that associated with the $i^{th}$ individual is a survival time $t_i*$ and a fixed censoring time $c_i$. The exact survival time $t_i$ will be observed only if $t_i* \leq c_i$. Hence our data is of the form $(y_i,\delta_i)$ where
$$
\begin{equation}
t_i = min(t_i*, c_i)
\end{equation}
$$
and
$$
\begin{equation}
\delta_i= I(t_i*\leq c_i)
\end{equation}
$$

In our dataset, $\delta_i$ is given by `df.event` and

```python
%matplotlib inline
import numpy as np
import pandas as pd
import pymc3 as pm
import arviz as az
from pymc3 import Gamma, Poisson, Normal, Model, sample, forestplot, NUTS, Metropolis, find_MAP, starting, traceplot
import seaborn as sns
import matplotlib.pyplot as plt
from statsmodels import datasets
from theano import tensor as tt
from io import StringIO
import warnings
warnings.simplefilter('ignore')

sns.set_style("ticks")

SEEDS = 2234323, 172354266
np.random.seed(SEEDS)
```

### Mastectomy Data

In this tutorial, we will use an oncological dataset.
In this dataset, each row represents observations from a woman diagnosed with breast cancer that underwent a mastectomy. The column `time` ($t_i$) represents the time (in months) post-surgery that the woman was observed. The column `event` indicates whether or not the woman died during the observation period (i.e. $\delta_i$). The column `metastized` represents whether the cancer had metastized prior to surgery.

We will analyze the relationship between survival time post-mastectomy and whether or not the cancer had metastized.

```python
df = datasets.get_rdataset('mastectomy', 'HSAUR', cache=True).data
df.event = df.event.astype(np.int64)
df.metastized = (df.metastized == 'yes').astype(np.int64)
n_patients = df.shape[0]
patients = np.arange(n_patients)
```

```python
fig, ax = plt.subplots(figsize=(14, 10))

blue, _, green = sns.color_palette()[:3]

ax.hlines(patients[df.metastized.values == 0], 0, df[df.metastized.values == 0].time,
          color=blue, label='Not metastized')

ax.hlines(patients[df.metastized.values == 1], 0, df[df.metastized.values == 1].time,
          color=green, label='Metastized')

ax.scatter(df[df.event.values == 1].time, patients[df.event.values == 1],
           color='k', zorder=10, label='Death')

ax.set_xlim(left=0)
ax.set_xlabel('Months since mastectomy')
ax.set_yticks([])
ax.set_ylabel('Subject')

ax.set_ylim(-0.25, n_patients + 0.25)

ax.legend(loc='center right');
```

As an initial look into survival, we plot the non-parametric Kaplan Meier estimates of the survival via the package `lifelines`

```python
from lifelines import KaplanMeierFitter
kmf = KaplanMeierFitter()
```

```python
kmf.fit(df.time, event_observed=df.event)
```

```python
time = df.time
event = df.event
ix0 = df.metastized==0
ix1 = df.metastized==1

kmf.fit(time[ix0], event[ix0], label='no Metastasis')
ax = kmf.plot(color=green)

kmf.fit(time[ix1], event[ix1], label='Metasasis')
kmf.plot(ax=ax, ls='dashed', color=blue, figsize=(14,10))
ax.set_xlabel('Time (months)')
ax.set_ylabel('Survival');
```

# Model Specification

We wish to relate a treatment effect in some way to the time to event outcome. There are several ways to do this, but two main paradigms are pervasive in the literature:

1. Proportional Hazards
2. Accelerated Failure Time

In this lecture we will focus on the first of these models.

There are a number of probability models one could use for lifetime data but the Weibull distribution is a very flexible model that is popular with many researchers. Its flexibility comes from the fact that it has a hazard rate which is either monotone increasing, decreasing, or constant. Additionally, it is the only parametric regression model which has both a proportional hazards representation and an accelerated failure-time representation. Again, however, we will only be focusing on the Proportional hazards specification.

## Proportional Hazards Model

The main objective at this point is to specify a model that relates the group effect to the time outcome. Recall above that the hazard quantifies the instantaneous risk of disease at a given time point.

It turns out that the easiest survival parameter to model is the hazard rate: which tells us how quickly individuals of a certain age are experiencing the event of interest. The major approach to modeling the effects of covariates on survival is to model the conditional hazard rate as a function of the covariates. Two general classes of models have been used to relate covariate effects to survival, the family of multiplicative hazard models and the family of additive hazard rate models. We will focus on the former of these here.

For the family of multiplicative hazard rate models the conditional hazard rate of an individual with covariate vector $X$ is a product of a baseline hazard rate $h_0(t)$ and a non-negative function of the covariates,
$c(\beta^T X)$, that is, $h (t | z) =h_0(t)c(\beta^T X)$.

   -$h_0(x)$ may have a specified parametric form or it may be left as an arbitrary nonnegative function.

   - Any nonnegative function can be used for the link function $c()$. Most applications use the Cox (1972) model with $c(\beta^T X) = \exp(\beta^T X)$ which is chosen for its simplicity
   - Key Feature: when all the covariates are fixed at time 0, the hazard rates of two individuals with distinct
   values of X are proportional.

Using the exponential link, we have
$$
{\displaystyle h (t|X_{i})=h _{0}(t)\exp(\beta^T X_{i} )}
$$

That is to say, the model is formulated in such a fashion that the group effect has a multiplicative effect on the hazard. Typically, semi-parametric models which don't have explicit representations of $h_0(t)$ are preferred, however for the purposes of this tutorial, we will assume a particular form for this baseline hazard.

The Weibull proportional hazards model models the hazard as

$$
{\displaystyle h (t|X_{i})=\alpha\lambda t^{\alpha-1}\exp(\beta^TX_{i})}
$$

## Likelihood Specification

Due to right censoring, we need to adjust our likelihood to account for observations where we know that an individual survived up to a particular point. If a person is known to have survived at least to time $t$, then their likelihood contribution is simply the survival function evaluated at the censoring time. We typically express this with an indicator variable $(\delta_i)$ for each observation denoting whether or not the individual was right censored or observed.

First, recall that the Weibull proportional hazards model models the hazard as a function of the covariates. The probability density function is related to the hazard by:

$$
\begin{equation}
f(t) = h(t)S(t)
\end{equation}
$$

The likelihood specification for an individual is therefore given by:
$$
\begin{eqnarray}
L_i &=& [f(t|X)]^{\delta_i}[S(t|X)]^{1-\delta_i}\\
&=& [h(t|X)S(t|X)]^{\delta_i}[S(t|X)]^{1-\delta_i}\\
&=& [\alpha\lambda t^{\alpha-1}\exp(\beta^T X)exp(-\lambda t^{\alpha}\exp(\beta^T X))]^{\delta_i}[exp(-\lambda t^{\alpha}\exp(\beta^T X))]^{1-\delta_i}\\
&=&[\alpha\lambda t^{\alpha-1}\exp(\beta^T X-\lambda t^{\alpha}\exp(\beta^T X))]^{\delta_i}[exp(-\lambda t^{\alpha}\exp(\beta^T X))]^{1-\delta_i}\\
\end{eqnarray}
$$

### Specification in PyMC3

We've formulated the mathematical form of our likelihood - we now need to implement it. PyMC3 makes this really easy to do with the `DenistyDist` class. All that is needed to specify a likelihood is a python function that passes the log of the likelihood.

Continuing from before:

$$
\begin{eqnarray}
log L_i &=& \log[[\alpha\lambda t^{\alpha-1}\exp(\beta^T X-\lambda t^{\alpha}\exp(\beta^T X))]^{\delta_i}[exp(-\lambda t^{\alpha}\exp(\beta^T X))]^{1-\delta_i}]\\
&=& \delta_i \log[\alpha\lambda t^{\alpha-1}\exp(\beta^T X-\lambda t^{\alpha}\exp(\beta^T X))]+(1-\delta_i)(-\lambda t^{\alpha}\exp(\beta^T X))\\
&=&\delta_i(\log\alpha+\log\lambda+(\alpha-1)\log t + \beta^T X) - \lambda t^{\alpha}\exp(\beta^T X)
\end{eqnarray}
$$

Implemented in PyMC3, this is:

```python
def logp(event):
    return (event*(tt.log(α) + tt.log(λ)
                   + tt.log(tt.exp(β*np.array(metast)))
                   + (α-1)*tt.log(np.array(time)))
            - (λ*tt.exp(β*np.array(metast))
                   * np.array(time)**α))

survival = pm.DensityDist('survival', logp, observed={'event':event})
```

### Prior Distribution specification:
Recall that in a Bayesian framework, estimated parameters are considered random variables - so we need to place a prior distribution on them. Looking at the parameters to estimate, we have

1. Weibull baseline hazard parameters: $\alpha$,$\lambda>0$
    - A natural prior for these parameters would be the Gamma distribution.

```python
from pymc3 import Gamma

params = (5, 1), (1, 3), (5, 5), (0.5, 0.5), (10, 1)

fig, axes = plt.subplots(1, len(params), figsize=(14, 4), sharey=True)
for ax, (alpha, beta) in zip(axes, params):
    sns.distplot(Gamma.dist(alpha, beta).random(size=1000), ax=ax, kde=False)
    ax.set_xlim(0, 20)
    ax.set_title(r'$\alpha={0}, \beta={1}$'.format(alpha, beta));
```

 - Alternatively, one might specify a Half Cauchy distribution:

```python
sns.distplot(pm.HalfCauchy.dist(1).random(size=1000), kde=False)
```

2. Group effect: $-\infty<\beta<\infty$

For the effect of metastasis, the most natural prior is the Normal distribution.

```python
sns.distplot(pm.Normal.dist(0,100).random(size=3000), kde=False)
```

### Implementation in PyMC3

Putting all the pieces together, we can fit the model using the pymc framework as so:

```python
metast = df.metastized.values
```

```python
n_iterations = 1000
n_tune = 1000
```

```python
with Model() as weibull_model:

    # Weibull shape parameter
    α = pm.HalfCauchy("α", 1)

    # Effects of metastisis
    λ = pm.HalfCauchy("λ", 1)

    # Weibull scales
    β = pm.Normal('β', 0, 100)

    linear = tt.exp(β*metast)

    # Weibull survival likelihood, accounting for censoring
    def logp(event):
        return event*(tt.log(α) + tt.log(λ) + tt.log(linear)
                      + (α-1)*tt.log(np.array(time))) - (λ*linear * np.array(time)**α)

    survival = pm.DensityDist('survival', logp, observed={'event':event})

    weibull_trace = pm.sample(n_iterations, tune=n_tune, cores=2)
```

We can view the posterior samples and assess convergence of the Markov Chain:

```python
az.plot_trace(weibull_trace, var_names=['α', 'λ', 'β']);
```

Since we have fit a proportional hazards model, the interpretation of the group effect is straightforward (at least with respect to the hazard). We need only to exponentiate the $\beta$ posterior samples for the effect size and corresponding credible intervals.

```python
hr_samps = np.exp(weibull_trace['β'])
print('Hazard ratio (metastisized vs not metastisized) at 5 years: {0:.2f} {1}'.format(hr_samps.mean(),
                                                            np.percentile(hr_samps, [2.5, 97.5]).round(2)))
```

> Interpretation: This means that the hazard for the group with metastasis is bigger
than the one for the group with no metastasis. This indicates metastasis has a negative effect on survival.

### Posterior Survival quantites

Given that we have posterior samples for each of the parameters, it is straightforward to obtain posterior survival curves, hazard rates, etc, with corresponding credible intervals.

```python
## Plot survival curves
n_intervals = 120
interval_bounds = np.arange(0, n_intervals + 1, 1)
```

```python
#baseline_lam = np.exp(weibull_trace['β'][n_burn:,0])
baseline_lam = np.repeat(1., n_iterations*2)
metast_lam = np.exp(weibull_trace['β'])
λ_trace = weibull_trace['λ']
α_trace = weibull_trace['α']
```

```python
def plot_with_hpd(x, lam, f, ax, linestyle='-', color=None, label=None, alpha=0.05, debug=False, **kwargs):

    m = np.median(f(lam, **kwargs), axis=0)

    percentiles = 100 * np.array([alpha / 2., 1. - alpha / 2.])
    hpd = np.percentile(f(lam, **kwargs), percentiles, axis=0)

    if debug:
        import pdb;pdb.set_trace()
    ax.fill_between(x, hpd[0], hpd[1], color=color, alpha=0.15)
    ax.step(x, m, color=color, linestyle=linestyle, label=label);
```

```python
def cum_weibull_hazard(lam, α=α_trace ,λ=λ_trace, t=n_intervals):
    return (λ_trace.reshape(-1,1)*lam.reshape(-1,1) * np.arange(t)) ** α.reshape(-1,1)

def weibull_survival(lam,α=α_trace, λ=λ_trace, t=n_intervals):
    return np.exp(-cum_weibull_hazard(lam, α, λ, t))

def cum_hazard(hazard, n=n_intervals):
    return (np.arange(n) * hazard.reshape(-1,1)).cumsum(axis=-1)

def survival(hazard):
    return np.exp(-cum_hazard(hazard))
```

```python
fig, (hazard_ax, surv_ax) = plt.subplots(ncols=2, sharex=True, sharey=False, figsize=(16, 6))

plot_with_hpd(interval_bounds[:-1], baseline_lam, cum_weibull_hazard,
              hazard_ax, color=green, linestyle=':', label='No Metastasis')
plot_with_hpd(interval_bounds[:-1], metast_lam, cum_weibull_hazard,
              hazard_ax, color=blue, linestyle='--', label='Metastisized')

hazard_ax.set_xlim(0, n_intervals);
hazard_ax.set_xlabel('Followup time');

hazard_ax.set_ylabel(r'Cumulative hazard $\Lambda(t)$');

hazard_ax.legend(loc=2);

plot_with_hpd(interval_bounds[:-1], baseline_lam, weibull_survival,
              surv_ax, color=green, linestyle=':')
plot_with_hpd(interval_bounds[:-1], metast_lam, weibull_survival,
              surv_ax, color=blue, linestyle='--')

surv_ax.set_xlim(0, n_intervals);
surv_ax.set_xlabel('Followup time');

surv_ax.set_ylabel('Survival function $S(t)$');

fig.suptitle('Bayesian survival model');
```

Looking at our KM estimate from earlier, we see that the posterior survival curves follow closely to the non-parametric K-M estimates.

```python
## Hazard ratio comparing 2 groups at a given time point
def weibull_hazard(lam, t,α=α_trace, λ=λ_trace):
    return (α.reshape(-1,1) * λ.reshape(-1,1) * lam.reshape(-1,1) * t**(α.reshape(-1,1) - 1))
```

We can of course, obtain our hazard ratio from the posterior samples:

```python
hazard_ratio = weibull_hazard(metast_lam, 60)/weibull_hazard(baseline_lam, 60)
print('Hazard ratio (metastisized vs notmetastisized) at 5 years: {0} {1}'.format(hazard_ratio.mean(),
                                                                     np.percentile(hazard_ratio, [2.5, 97.5])))
```

```python
### Other quantities of interest:

baseline_survival = weibull_survival(baseline_lam)
metast_survival = weibull_survival(metast_lam)
```

```python
#Time where surivial is .5
np.where(baseline_survival.mean(axis=0)<=0.5)[0][0]
np.where(metast_survival.mean(axis=0)<=0.5)[0][0]
```

```python
baseline_survival_5 = baseline_survival[:, 60]
metast_survival_5 = metast_survival[:, 60]
print('Baseline 5-year median survival: {0:.2f} {1}'.format(np.median(baseline_survival),
                                                        np.percentile(baseline_survival, [2.5, 97.5]).round(2)))

print('Metastasis group median 5-year survival: {0:.2f} {1}'.format(np.median(metast_survival_5),
                                                        np.percentile(metast_survival_5, [2.5, 97.5]).round(2)))
```

## Exercise: Heart transplant survival

Try your hand at building a survival model using PyMC3. Here is a small dataset based on 69 patients receiving heart transplants. Taken from "The Statistical Analysis of Failure Time Data" by Kalbfleisch and Prentice, Appendix I, pages 230-232 from stalib data depository. http://lib.stat.cmu.edu/datasets/stanford

Data columns:
- Age at transplant in years
- Survival Status 1=dead 0=alive
- Survival Time after transplant in days

Estimate the relationship between age at transplant and survival probability.

```python
data_str = """
41 1 5
40 1 16
54 1 16
29 1 17
55 1 28
52 1 30
40 1 39
35 0 39
56 1 43
36 1 45
42 1 51
50 1 53
42 1 58
52 1 61
61 1 66
45 1 68
49 1 68
53 1 72
47 1 72
64 1 77
51 1 78
53 1 80
54 1 81
56 1 90
53 1 96
48 1 100
28 0 109
46 1 110
23 0 131
47 1 153
43 1 165
26 0 180
52 1 186
47 1 188
51 1 207
51 1 219
47 0 265
48 1 285
19 1 285
49 1 308
42 1 334
44 0 340
47 1 342
54 0 370
48 0 397
52 0 445
46 0 482
48 0 515
52 0 545
48 1 583
26 0 596
47 0 630
47 0 670
50 1 675
58 1 733
32 0 841
44 1 852
41 0 915
38 0 941
45 1 979
48 1 995
43 1 1032
36 0 1141
45 0 1321
53 1 1386
48 0 1407
40 0 1571
48 0 1586
33 0 1799
"""

transplant_data = pd.read_table(StringIO(data_str), sep='\s+', names=['transplant_age', 'death', 'time'])
transplant_data.head()
```

```python
# Write your answer here
```

---
# References

Ibrahim, Joseph G., Ming‐Hui Chen, and Debajyoti Sinha. Bayesian survival analysis. John Wiley & Sons, Ltd, 2005.

Klein, John P., Moeschberger, Melvin L. Survival Analysis: Techniques for Censored and Truncated Data Second Edition. Springer, 2003.

[Survival Distributions, Hazard Functions, Cumulative Hazards](https://web.stanford.edu/~lutian/coursepdf/unit1.pdf)


# Assignment 2 - Semi-gradient TD with a Neural Network

Welcome to Course 3 Programming Assignment 2. In the previous assignment, you implemented semi-gradient TD with State Aggregation for solving a **policy evaluation task**. In this assignment, you will implement **semi-gradient TD with a simple Neural Network** and use it for the same policy evaluation problem.

You will implement an agent to evaluate a fixed policy on the 500-State Randomwalk. As you may remember from the previous assignment, the 500-state Randomwalk includes 500 states. Each episode begins with the agent at the center and terminates when the agent goes far left beyond state 1 or far right beyond state 500. At each time step, the agent selects to move either left or right with equal probability. The environment determines how much the agent moves in the selected direction.

**In this assignment, you will:**
- Implement stochastic gradient descent method for state-value prediction.
- Implement semi-gradient TD with a neural network as the function approximator and Adam algorithm.
- Compare performance of semi-gradient TD with a neural network and semi-gradient TD with tile-coding.

## Packages
We import the following libraries that are required for this assignment:

- [numpy](www.numpy.org) : Fundamental package for scientific computing with Python.
- [matplotlib](http://matplotlib.org) : Library for plotting graphs in Python.
- [RL-Glue](http://www.jmlr.org/papers/v10/tanner09a.html) : Library for reinforcement learning experiments.
- [tqdm](https://tqdm.github.io/) : A package to display progress bar when running experiments.
- BaseOptimizer : An abstract class that specifies the optimizer API for Agent.
- plot_script : Custom script to plot results.
- RandomWalkEnvironment : The Randomwalk environment script from Course 3 Assignment 1.

```python
# Do not modify this cell!

# Import necessary libraries
# DO NOT IMPORT OTHER LIBRARIES - This will break the autograder.
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

import os, shutil
from tqdm import tqdm

from rl_glue import RLGlue
from environment import BaseEnvironment
from agent import BaseAgent
from optimizer import BaseOptimizer
import plot_script
from randomwalk_environment import RandomWalkEnvironment
```

## Section 1: Create semi-gradient TD with a Neural Network
In this section, you will implement an Agent that learns with semi-gradient TD with a neural network. You will use a neural network with one hidden layer. The input of the neural network is the one-hot encoding of the state number. We use the one-hot encoding of the state number instead of the state number itself because we do not want to build the prior knowledge that integer number inputs close to each other have similar values. The hidden layer contains 100 rectifier linear units (ReLUs) which pass their input if it is bigger than one and return 0 otherwise. ReLU gates are commonly used in neural networks due to their nice properties such as the sparsity of the activation and having non-vanishing gradients. The output of the neural network is the estimated state value. It is a linear function of the hidden units as is commonly the case when estimating the value of a continuous target using neural networks.

The neural network looks like this:

For a given input, $s$, value of $s$ is computed by:
$$
\begin{align}
\psi &= sW^{[0]} + b^{[0]} \\
x &= \textit{max}(0, \psi) \\
v &= xW^{[1]} + b^{[1]}
\end{align}
$$

where $W^{[0]}$, $b^{[0]}$, $W^{[1]}$, $b^{[1]}$  are the parameters of the network and will be learned when training the agent.

## 1-1: Implement helper methods

Before implementing the agent, you first implement some helper functions which you will later use in agent's main methods.

### Implement `get_value()`
First, you will implement get_value() method which feeds an input $s$ into the neural network and returns the output of the network $v$ according to the equations above. To implement get_value(), take into account the following notes:

- `get_value()` gets the one-hot encoded state number denoted by s as an input.
- `get_value()` receives the weights of the neural network as input, denoted by weights and structured as an array of dictionaries. Each dictionary corresponds to weights from one layer of the neural network to the next. Each dictionary includes $W$ and $b$. The shape of the elements in weights are as follows:
    - weights[0]["W"]: num_states $\times$ num_hidden_units
    - weights[0]["b"]: 1 $\times$ num_hidden_units
    - weights[1]["W"]: num_hidden_units $\times$ 1
    - weights[1]["b"]: 1 $\times$ 1

- The input of the neural network is a sparse vector. To make computation faster, we take advantage of input sparsity. To do so, we provided a helper method `my_matmul()`. **Make sure that you use `my_matmul()` for all matrix multiplications except for element-wise multiplications in this notebook.**
- The max operator used for computing $x$ is element-wise.

```python
def my_matmul(x1, x2):
    """
    Given matrices x1 and x2, return the multiplication of them
    """

    result = np.zeros((x1.shape[0], x2.shape[1]))
    x1_non_zero_indices = x1.nonzero()
    if x1.shape[0] == 1 and len(x1_non_zero_indices[1]) == 1:
        result = x2[x1_non_zero_indices[1], :]
    elif x1.shape[1] == 1 and len(x1_non_zero_indices[0]) == 1:
        result[x1_non_zero_indices[0], :] = x2 * x1[x1_non_zero_indices[0], 0]
    else:
        result = np.matmul(x1, x2)
    return result
```

```python
# -----------
# Graded Cell
# -----------

def get_value(s, weights):
    """
    Compute value of input s given the weights of a neural network
    """
    ### Compute the ouput of the neural network, v, for input s

    # ----------------
    W_0 = weights[0]['W']
    b_0 = weights[0]['b']
    W_1 = weights[1]['W']
    b_1 = weights[1]['b']

    psi = my_matmul(s, W_0) + b_0
    relu = np.maximum(psi,0)
    v = my_matmul(relu, W_1) + b_1

    # ----------------
    return v
```

Run the following code to test your implementation of the `get_value()` function:

```python
# -----------
# Tested Cell
# -----------
# The contents of the cell will be tested by the autograder.
# If they do not pass here, they will not pass there.

# Suppose num_states = 5, num_hidden_layer = 1, and num_hidden_units = 10
num_hidden_layer = 1
s = np.array([[0, 0, 0, 1, 0]])

weights_data = np.load("asserts/get_value_weights.npz")
weights = [dict() for i in range(num_hidden_layer+1)]
weights[0]["W"] = weights_data["W0"]
weights[0]["b"] = weights_data["b0"]
weights[1]["W"] = weights_data["W1"]
weights[1]["b"] = weights_data["b1"]

estimated_value = get_value(s, weights)
print ("Estimated value: {}".format(estimated_value))

assert(np.allclose(estimated_value, [[-0.21915705]]))
```

    Estimated value: [[-0.21915705]]

**Expected output**:

    Estimated value: [[-0.21915705]]

### Implement `get_gradient()`
You will also implement `get_gradient()` method which computes the gradient of the value function for a given input, using backpropagation. You will later use this function to update the value function.

As you know, we compute the value of a state $s$ according to:
$$
\begin{align}
\psi &= sW^{[0]} + b^{[0]} \\
x &= \textit{max}(0, \psi) \\
v &= xW^{[1]} + b^{[1]}
\end{align}
$$

To update the weights of the neural network ($W^{[0]}$, $b^{[0]}$, $W^{[1]}$, $b^{[1]}$), we compute the gradient of $v$ with respect to the weights according to:

$$
\begin{align}
\frac{\partial v}{\partial W^{[0]}} &= s^T(W^{[1]T} \odot I_{x>0}) \\
\frac{\partial v}{\partial b^{[0]}} &= W^{[1]T} \odot I_{x>0} \\
\frac{\partial v}{\partial W^{[1]}} &= x^T \\
\frac{\partial v}{\partial b^{[1]}} &= 1
\end{align}
$$
where $\odot$ denotes element-wise matrix multiplication and $I_{x>0}$ is the gradient of the ReLU activation function which is an indicator whose $i$th element is 1 if $x[i]>0$ and 0 otherwise.

```python
# -----------
# Graded Cell
# -----------

def get_gradient(s, weights):
    """
    Given inputs s and weights, return the gradient of v with respect to the weights
    """

    ### Compute the gradient of the value function with respect to W0, b0, W1, b1 for input s
    # grads[0]["W"] = ?
    # grads[0]["b"] = ?
    # grads[1]["W"] = ?
    # grads[1]["b"] = ?
    # Note that grads[0]["W"], grads[0]["b"], grads[1]["W"], and grads[1]["b"] should have the same shape as
    # weights[0]["W"], weights[0]["b"], weights[1]["W"], and weights[1]["b"] respectively
    # Note that to compute the gradients, you need to compute the activation of the hidden layer (x)

    grads = [dict() for i in range(len(weights))]

    W_0 = weights[0]['W']
    b_0 = weights[0]['b']
    W_1 = weights[1]['W']
    b_1 = weights[1]['b']

    x = np.maximum(my_matmul(s, W_0) + b_0, 0)
    # ----------------
    grads[0]["W"] = s.T *(W_1.T * (x>0))
    grads[0]["b"] = W_1.T * (x>0)
    grads[1]["W"] = x.T
    grads[1]["b"] = 1
    # ----------------

    return grads

```

Run the following code to test your implementation of the `get_gradient()` function:

```python
# -----------
# Tested Cell
# -----------
# The contents of the cell will be tested by the autograder.
# If they do not pass here, they will not pass there.

# Suppose num_states = 5, num_hidden_layer = 1, and num_hidden_units = 2
num_hidden_layer = 1
s = np.array([[0, 0, 0, 1, 0]])

weights_data = np.load("asserts/get_gradient_weights.npz")
weights = [dict() for i in range(num_hidden_layer+1)]
weights[0]["W"] = weights_data["W0"]
weights[0]["b"] = weights_data["b0"]
weights[1]["W"] = weights_data["W1"]
weights[1]["b"] = weights_data["b1"]

grads = get_gradient(s, weights)

grads_answer = np.load("asserts/get_gradient_grads.npz")

assert(np.allclose(grads[0]["W"], grads_answer["W0"]))
assert(np.allclose(grads[0]["b"], grads_answer["b0"]))
assert(np.allclose(grads[1]["W"], grads_answer["W1"]))
assert(np.allclose(grads[1]["b"], grads_answer["b1"]))
```

**Expected output**:

    grads[0]["W"]
     [[0.         0.        ]
     [0.         0.        ]
     [0.         0.        ]
     [0.76103773 0.12167502]
     [0.         0.        ]]

    grads[0]["b"]
     [[0.76103773 0.12167502]]

    grads[1]["W"]
     [[0.69198983]
     [0.82403662]]

    grads[1]["b"]
     [[1.]]

### Implement stochastic gradient descent method for state-value prediction
In this section, you will implement stochastic gradient descent (SGD) method for state_value prediction. Here is the basic SGD update for state-value prediction with TD:

$$\mathbf{w_{t+1}} = \mathbf{w_{t}} + \alpha \delta_t \nabla \hat{v}(S_t,\mathbf{w_{t}})$$

At each time step, we update the weights in the direction  $g_t = \delta_t \nabla \hat{v}(S_t,\mathbf{w_t})$ using a fixed step-size $\alpha$. $\delta_t = R_{t+1} + \gamma \hat{v}(S_{t+1},\mathbf{w_{t}}) - \hat{v}(S_t,\mathbf{w_t})$ is the TD-error. $\nabla \hat{v}(S_t,\mathbf{w_{t}})$ is the gradient of the value function with respect to the weights.

The following cell includes the SGD class. You will complete the `update_weight()` method of SGD assuming that the weights and update g are provided.

**As you know, in this assignment, we structured the weights as an array of dictionaries. Note that the updates $g_t$, in the case of TD, is $\delta_t \nabla \hat{v}(S_t,\mathbf{w_t})$. As a result, $g_t$ has the same structure as $\nabla \hat{v}(S_t,\mathbf{w_t})$ which is also an array of dictionaries.**

```python
# -----------
# Graded Cell
# -----------

class SGD(BaseOptimizer):
    def __init__(self):
        pass

    def optimizer_init(self, optimizer_info):
        """Setup for the optimizer.

        Set parameters needed to setup the stochastic gradient descent method.

        Assume optimizer_info dict contains:
        {
            step_size: float
        }
        """
        self.step_size = optimizer_info.get("step_size")

    def update_weights(self, weights, g):
        """
        Given weights and update g, return updated weights
        """
        for i in range(len(weights)):
            for param in weights[i].keys():

                ### update weights
                weights[i][param] += self.step_size * g[i][param]

        return weights
```

Run the following code to test your implementation of the `update_weights()` function:

```python
# -----------
# Tested Cell
# -----------
# The contents of the cell will be tested by the autograder.
# If they do not pass here, they will not pass there.

# Suppose num_states = 5, num_hidden_layer = 1, and num_hidden_units = 2
num_hidden_layer = 1

weights_data = np.load("asserts/update_weights_weights.npz")
weights = [dict() for i in range(num_hidden_layer+1)]
weights[0]["W"] = weights_data["W0"]
weights[0]["b"] = weights_data["b0"]
weights[1]["W"] = weights_data["W1"]
weights[1]["b"] = weights_data["b1"]

g_data = np.load("asserts/update_weights_g.npz")
g = [dict() for i in range(num_hidden_layer+1)]
g[0]["W"] = g_data["W0"]
g[0]["b"] = g_data["b0"]
g[1]["W"] = g_data["W1"]
g[1]["b"] = g_data["b1"]

test_sgd = SGD()
optimizer_info = {"step_size": 0.3}
test_sgd.optimizer_init(optimizer_info)
updated_weights = test_sgd.update_weights(weights, g)

# updated weights asserts
updated_weights_answer = np.load("asserts/update_weights_updated_weights.npz")

assert(np.allclose(updated_weights[0]["W"], updated_weights_answer["W0"]))
assert(np.allclose(updated_weights[0]["b"], updated_weights_answer["b0"]))
assert(np.allclose(updated_weights[1]["W"], updated_weights_answer["W1"]))
assert(np.allclose(updated_weights[1]["b"], updated_weights_answer["b1"]))
```

**Expected output**:

    updated_weights[0]["W"]
     [[ 1.17899492  0.53656321]
     [ 0.58008221  1.47666572]
     [ 1.01909411 -1.10248056]
     [ 0.72490408  0.06828853]
     [-0.20609725  0.69034095]]

    updated_weights[0]["b"]
     [[-0.18484533  0.92844539]]

    updated_weights[1]["W"]
     [[0.70488257]
     [0.58150878]]

    updated_weights[1]["b"]
     [[0.88467086]]

### Adam Algorithm
In this assignment, instead of using SGD for updating the weights, we use a more advanced algorithm called Adam. The Adam algorithm improves the SGD update with two concepts: adaptive vector step-sizes and momentum. It keeps estimates of the mean and second moment of the updates, denoted by $\mathbf{m}$ and $\mathbf{v}$ respectively:
$$\mathbf{m_t} = \beta_m \mathbf{m_{t-1}} + (1 - \beta_m)g_t \\
\mathbf{v_t} = \beta_v \mathbf{v_{t-1}} + (1 - \beta_v)g^2_t
$$

Given that $\mathbf{m}$ and $\mathbf{v}$ are initialized to zero, they are biased toward zero. To get unbiased estimates of the mean and second moment, Adam defines $\mathbf{\hat{m}}$ and $\mathbf{\hat{v}}$ as:
$$ \mathbf{\hat{m_t}} = \frac{\mathbf{m_t}}{1 - \beta_m^t} \\
\mathbf{\hat{v_t}} = \frac{\mathbf{v_t}}{1 - \beta_v^t}
$$

The weights are then updated as follows:
$$ \mathbf{w_t} = \mathbf{w_{t-1}} + \frac{\alpha}{\sqrt{\mathbf{\hat{v_t}}}+\epsilon} \mathbf{\hat{m_t}}
$$

When implementing the agent you will use the Adam algorithm instead of SGD because it is more efficient. We have already provided you the implementation of the Adam algorithm in the cell below. You will use it when implementing your agent.

```python
# ---------------
# Discussion Cell
# ---------------
class Adam(BaseOptimizer):
    def __init__(self):
        pass

    def optimizer_init(self, optimizer_info):
        """Setup for the optimizer.

        Set parameters needed to setup the Adam algorithm.

        Assume optimizer_info dict contains:
        {
            num_states: integer,
            num_hidden_layer: integer,
            num_hidden_units: integer,
            step_size: float,
            self.beta_m: float
            self.beta_v: float
            self.epsilon: float
        }
        """

        self.num_states = optimizer_info.get("num_states")
        self.num_hidden_layer = optimizer_info.get("num_hidden_layer")
        self.num_hidden_units = optimizer_info.get("num_hidden_units")

        # Specify Adam algorithm's hyper parameters
        self.step_size = optimizer_info.get("step_size")
        self.beta_m = optimizer_info.get("beta_m")
        self.beta_v = optimizer_info.get("beta_v")
        self.epsilon = optimizer_info.get("epsilon")

        self.layer_size = np.array([self.num_states, self.num_hidden_units, 1])

        # Initialize Adam algorithm's m and v
        self.m = [dict() for i in range(self.num_hidden_layer+1)]
        self.v = [dict() for i in range(self.num_hidden_layer+1)]

        for i in range(self.num_hidden_layer+1):

            # Initialize self.m[i]["W"], self.m[i]["b"], self.v[i]["W"], self.v[i]["b"] to zero
            self.m[i]["W"] = np.zeros((self.layer_size[i], self.layer_size[i+1]))
            self.m[i]["b"] = np.zeros((1, self.layer_size[i+1]))
            self.v[i]["W"] = np.zeros((self.layer_size[i], self.layer_size[i+1]))
            self.v[i]["b"] = np.zeros((1, self.layer_size[i+1]))

        # Initialize beta_m_product and beta_v_product to be later used for computing m_hat and v_hat
        self.beta_m_product = self.beta_m
        self.beta_v_product = self.beta_v

    def update_weights(self, weights, g):
        """
        Given weights and update g, return updated weights
        """

        for i in range(len(weights)):
            for param in weights[i].keys():

                ### update self.m and self.v
                self.m[i][param] = self.beta_m * self.m[i][param] + (1 - self.beta_m) * g[i][param]
                self.v[i][param] = self.beta_v * self.v[i][param] + (1 - self.beta_v) * (g[i][param] * g[i][param])

                ### compute m_hat and v_hat
                m_hat = self.m[i][param] / (1 - self.beta_m_product)
                v_hat = self.v[i][param] / (1 - self.beta_v_product)

                ### update weights
                weights[i][param] += self.step_size * m_hat / (np.sqrt(v_hat) + self.epsilon)

        ### update self.beta_m_product and self.beta_v_product
        self.beta_m_product *= self.beta_m
        self.beta_v_product *= self.beta_v

        return weights

```

## 1-2: Implement Agent Methods
In this section, you will implement `agent_init()`, `agent_start()`, `agent_step()`, and `agent_end()`.

In `agent_init()`, you will:

   - specify the neural network structure by filling self.layer_size with the size of the input layer, hidden layer, and output layer.
   - initialize the network's parameters. We show the parameters as an array of dictionaries, self.weights, where each dictionary corresponds to weights from one layer to the next. Each dictionary includes $W$ and $b$. To initialize the parameters, you will use a normal distribution with mean 0 and standard deviation $\sqrt{\frac{2}{\text{# input of each node}}}$. This initialization heuristic is commonly used when using ReLU gates and helps keep the output of a neuron from getting too big or too small. To initialize the network's parameters, use **self.rand_generator.normal()** which draws random samples from a normal distribution. The parameters of self.rand_generator.normal are mean of the distribution, standard deviation of the distribution, and output shape in the form of tuple of integers.

In `agent_start()`, you will:
   - specify self.last_state and self.last_action.

In `agent_step()` and `agent_end()`, you will:
   - compute the TD error using $v(S_t)$ and $v(S_{t+1})$. To compute the value function for $S_t$ and $S_{t+1}$, you will get their one-hot encoding using `one_hot()` method that we provided below. You feed the one-hot encoded state number to the neural networks using `get_value()` method that you implemented above. Note that `one_hot()` method returns the one-hot encoding of a state as a numpy array of shape (1, num_states).
   - retrieve the gradients using `get_gradient()` function that you implemented.
   - use Adam_algorithm that we provided to update the neural network's parameters, self.weights.
   - use `agent_policy()` method to select actions with. (only in `agent_step()`)

```python
# ---------------
# Discussion Cell
# ---------------
def one_hot(state, num_states):
    """
    Given num_state and a state, return the one-hot encoding of the state
    """
    # Create the one-hot encoding of state
    # one_hot_vector is a numpy array of shape (1, num_states)

    one_hot_vector = np.zeros((1, num_states))
    one_hot_vector[0, int((state - 1))] = 1

    return one_hot_vector
```

```python
# -----------
# Graded Cell
# -----------

class TDAgent(BaseAgent):
    def __init__(self):
        self.name = "td_agent"
        pass

    def agent_init(self, agent_info={}):
        """Setup for the agent called when the experiment first starts.

        Set parameters needed to setup the semi-gradient TD with a Neural Network.

        Assume agent_info dict contains:
        {
            num_states: integer,
            num_hidden_layer: integer,
            num_hidden_units: integer,
            step_size: float,
            discount_factor: float,
            self.beta_m: float
            self.beta_v: float
            self.epsilon: float
            seed: int
        }
        """

        # Set random seed for weights initialization for each run
        self.rand_generator = np.random.RandomState(agent_info.get("seed"))

        # Set random seed for policy for each run
        self.policy_rand_generator = np.random.RandomState(agent_info.get("seed"))

        # Set attributes according to agent_info
        self.num_states = agent_info.get("num_states")
        self.num_hidden_layer = agent_info.get("num_hidden_layer")
        self.num_hidden_units = agent_info.get("num_hidden_units")
        self.discount_factor = agent_info.get("discount_factor")

        ### Define the neural network's structure
        # Specify self.layer_size which shows the number of nodes in each layer
        # self.layer_size = np.array([None, None, None])
        # Hint: Checkout the NN diagram at the beginning of the notebook

        # ----------------
        self.layer_size = np.array([self.num_states, self.num_hidden_units, 1])

        # ----------------

        # Initialize the neural network's parameter
        self.weights = [dict() for i in range(self.num_hidden_layer+1)]
        for i in range(self.num_hidden_layer+1):

            ### Initialize self.weights[i]["W"] and self.weights[i]["b"] using self.rand_generator.normal()
            # Note that The parameters of self.rand_generator.normal are mean of the distribution,
            # standard deviation of the distribution, and output shape in the form of tuple of integers.
            # To specify output shape, use self.layer_size.

            # ----------------
            ins, outs = self.layer_size[i], self.layer_size[i+1]
            self.weights[i]['W'] = self.rand_generator.normal(0, np.sqrt(2/ins), (ins, outs))
            self.weights[i]['b'] = self.rand_generator.normal(0, np.sqrt(2/ins), (1, outs))
            # ----------------

        # Specify the optimizer
        self.optimizer = Adam()
        self.optimizer.optimizer_init({
            "num_states": agent_info["num_states"],
            "num_hidden_layer": agent_info["num_hidden_layer"],
            "num_hidden_units": agent_info["num_hidden_units"],
            "step_size": agent_info["step_size"],
            "beta_m": agent_info["beta_m"],
            "beta_v": agent_info["beta_v"],
            "epsilon": agent_info["epsilon"],
        })

        self.last_state = None
        self.last_action = None

    def agent_policy(self, state):

        ### Set chosen_action as 0 or 1 with equal probability.
        chosen_action = self.policy_rand_generator.choice([0,1])
        return chosen_action

    def agent_start(self, state):
        """The first method called when the experiment starts, called after
        the environment starts.
        Args:
            state (Numpy array): the state from the
                environment's evn_start function.
        Returns:
            The first action the agent takes.
        """
        ### select action given state (using self.agent_policy()), and save current state and action
        self.last_state = state
        self.last_action = self.agent_policy(state)

        return self.last_action

    def agent_step(self, reward, state):
        """A step taken by the agent.
        Args:
            reward (float): the reward received for taking the last action taken
            state (Numpy array): the state from the
                environment's step based, where the agent ended up after the
                last step
        Returns:
            The action the agent is taking.
        """

        ### Compute TD error
        last_state_vec = one_hot(self.last_state, self.num_states)
        last_value = get_value(last_state_vec, self.weights)

        state_vec = one_hot(state, self.num_states)
        value = get_value(state_vec, self.weights)

        delta = reward + self.discount_factor * value - last_value

        ### Retrieve gradients
        grads = get_gradient(last_state_vec, self.weights)

        ### Compute g (1 line)
        g = [dict() for i in range(self.num_hidden_layer+1)]
        for i in range(self.num_hidden_layer+1):
            for param in self.weights[i].keys():

                # g[i][param] = None
                # ----------------
                g[i][param] = delta * grads[i][param]
                # ----------------

        ### update the weights using self.optimizer
        # self.weights = None

        # ----------------
        self.weights = self.optimizer.update_weights(self.weights, g)

        # ----------------

        ### update self.last_state and self.last_action

        # ----------------
        self.last_state = state
        self.last_action = self.agent_policy(state)

        # ----------------

        return self.last_action

    def agent_end(self, reward):
        """Run when the agent terminates.
        Args:
            reward (float): the reward the agent received for entering the
                terminal state.
        """

        ### compute TD error
        # delta = None

        # ----------------
        last_state_vec = one_hot(self.last_state, self.num_states)
        last_value = get_value(last_state_vec, self.weights)
        delta = reward - last_value
        # ----------------

        ### Retrieve gradients
        # grads = None

        # ----------------
        grads = get_gradient(last_state_vec, self.weights)

        # ----------------

        ### Compute g
        g = [dict() for i in range(self.num_hidden_layer+1)]
        for i in range(self.num_hidden_layer+1):
            for param in self.weights[i].keys():

                # g[i][param] = None
                # ----------------
                g[i][param] = delta * grads[i][param]

                # ----------------

        ### update the weights using self.optimizer
        # self.weights = None

        # ----------------
        self.weights = self.optimizer.update_weights(self.weights, g)

        # ----------------

    def agent_message(self, message):
        if message == 'get state value':
            state_value = np.zeros(self.num_states)
            for state in range(1, self.num_states + 1):
                s = one_hot(state, self.num_states)
                state_value[state - 1] = get_value(s, self.weights)
            return state_value
```

Run the following code to test your implementation of the `agent_init()` function:

```python
# -----------
# Tested Cell
# -----------
# The contents of the cell will be tested by the autograder.
# If they do not pass here, they will not pass there.

agent_info = {
    "num_states": 5,
    "num_hidden_layer": 1,
    "num_hidden_units": 2,
    "step_size": 0.25,
    "discount_factor": 0.9,
    "beta_m": 0.9,
    "beta_v": 0.99,
    "epsilon": 0.0001,
    "seed": 0,
}

test_agent = TDAgent()
test_agent.agent_init(agent_info)

print("layer_size: {}".format(test_agent.layer_size))
assert(np.allclose(test_agent.layer_size, np.array([agent_info["num_states"],
                                                    agent_info["num_hidden_units"],
                                                    1])))

assert(test_agent.weights[0]["W"].shape == (agent_info["num_states"], agent_info["num_hidden_units"]))
assert(test_agent.weights[0]["b"].shape == (1, agent_info["num_hidden_units"]))
assert(test_agent.weights[1]["W"].shape == (agent_info["num_hidden_units"], 1))
assert(test_agent.weights[1]["b"].shape == (1, 1))

agent_weight_answer = np.load("asserts/agent_init_weights_1.npz")
assert(np.allclose(test_agent.weights[0]["W"], agent_weight_answer["W0"]))
assert(np.allclose(test_agent.weights[0]["b"], agent_weight_answer["b0"]))
assert(np.allclose(test_agent.weights[1]["W"], agent_weight_answer["W1"]))
assert(np.allclose(test_agent.weights[1]["b"], agent_weight_answer["b1"]))
```

    layer_size: [5 2 1]

**Expected output**:

    layer_size: [5 2 1]
    weights[0]["W"] shape: (5, 2)
    weights[0]["b"] shape: (1, 2)
    weights[1]["W"] shape: (2, 1)
    weights[1]["b"] shape: (1, 1)

    weights[0]["W"]
     [[ 1.11568467  0.25308164]
     [ 0.61900825  1.4172653 ]
     [ 1.18114738 -0.6180848 ]
     [ 0.60088868 -0.0957267 ]
     [-0.06528133  0.25968529]]

    weights[0]["b"]
     [[0.09110115 0.91976332]]

    weights[1]["W"]
     [[0.76103773]
     [0.12167502]]

    weights[1]["b"]
     [[0.44386323]]

Run the following code to test your implementation of the `agent_start()` function:

```python
# -----------
# Tested Cell
# -----------
# The contents of the cell will be tested by the autograder.
# If they do not pass here, they will not pass there.

agent_info = {
    "num_states": 500,
    "num_hidden_layer": 1,
    "num_hidden_units": 100,
    "step_size": 0.1,
    "discount_factor": 1.0,
    "beta_m": 0.9,
    "beta_v": 0.99,
    "epsilon": 0.0001,
    "seed": 10,
}

# Suppose state = 250
state = 250

test_agent = TDAgent()
test_agent.agent_init(agent_info)
test_agent.agent_start(state)

assert(test_agent.last_state == 250)
assert(test_agent.last_action == 1)
```

**Expected output**:

    Agent state: 250
    Agent selected action: 1

Run the following code to test your implementation of the `agent_step()` function:

```python
# -----------
# Tested Cell
# -----------
# The contents of the cell will be tested by the autograder.
# If they do not pass here, they will not pass there.

agent_info = {
    "num_states": 5,
    "num_hidden_layer": 1,
    "num_hidden_units": 2,
    "step_size": 0.1,
    "discount_factor": 1.0,
    "beta_m": 0.9,
    "beta_v": 0.99,
    "epsilon": 0.0001,
    "seed": 0,
}

test_agent = TDAgent()
test_agent.agent_init(agent_info)

# load initial weights
agent_initial_weight = np.load("asserts/agent_step_initial_weights.npz")
test_agent.weights[0]["W"] = agent_initial_weight["W0"]
test_agent.weights[0]["b"] = agent_initial_weight["b0"]
test_agent.weights[1]["W"] = agent_initial_weight["W1"]
test_agent.weights[1]["b"] = agent_initial_weight["b1"]

# load m and v for the optimizer
m_data = np.load("asserts/agent_step_initial_m.npz")
test_agent.optimizer.m[0]["W"] = m_data["W0"]
test_agent.optimizer.m[0]["b"] = m_data["b0"]
test_agent.optimizer.m[1]["W"] = m_data["W1"]
test_agent.optimizer.m[1]["b"] = m_data["b1"]

v_data = np.load("asserts/agent_step_initial_v.npz")
test_agent.optimizer.v[0]["W"] = v_data["W0"]
test_agent.optimizer.v[0]["b"] = v_data["b0"]
test_agent.optimizer.v[1]["W"] = v_data["W1"]
test_agent.optimizer.v[1]["b"] = v_data["b1"]

# Assume the agent started at State 3
start_state = 3
test_agent.agent_start(start_state)

# Assume the reward was 10.0 and the next state observed was State 1
reward = 10.0
next_state = 1
test_agent.agent_step(reward, next_state)

agent_updated_weight_answer = np.load("asserts/agent_step_updated_weights.npz")
assert(np.allclose(test_agent.weights[0]["W"], agent_updated_weight_answer["W0"]))
assert(np.allclose(test_agent.weights[0]["b"], agent_updated_weight_answer["b0"]))
assert(np.allclose(test_agent.weights[1]["W"], agent_updated_weight_answer["W1"]))
assert(np.allclose(test_agent.weights[1]["b"], agent_updated_weight_answer["b1"]))

assert(test_agent.last_state == 1)
assert(test_agent.last_action == 1)
```

**Expected output**:

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.23397791 -0.48152743]
     [ 0.72792093 -0.15829832]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.29798822 0.96254535]]

    updated_weights[1]["W"]
     [[0.76628754]
     [0.11486511]]

    updated_weights[1]["b"]
     [[0.58530057]]

    Agent last state: 1
    Agent last action: 1

Run the following code to test your implementation of the `agent_end()` function:

```python
# -----------
# Tested Cell
# -----------
# The contents of the cell will be tested by the autograder.
# If they do not pass here, they will not pass there.

agent_info = {
    "num_states": 5,
    "num_hidden_layer": 1,
    "num_hidden_units": 2,
    "step_size": 0.1,
    "discount_factor": 1.0,
    "beta_m": 0.9,
    "beta_v": 0.99,
    "epsilon": 0.0001,
    "seed": 0,
}

test_agent = TDAgent()
test_agent.agent_init(agent_info)

# load initial weights
agent_initial_weight = np.load("asserts/agent_end_initial_weights.npz")
test_agent.weights[0]["W"] = agent_initial_weight["W0"]
test_agent.weights[0]["b"] = agent_initial_weight["b0"]
test_agent.weights[1]["W"] = agent_initial_weight["W1"]
test_agent.weights[1]["b"] = agent_initial_weight["b1"]

# load m and v for the optimizer
m_data = np.load("asserts/agent_step_initial_m.npz")
test_agent.optimizer.m[0]["W"] = m_data["W0"]
test_agent.optimizer.m[0]["b"] = m_data["b0"]
test_agent.optimizer.m[1]["W"] = m_data["W1"]
test_agent.optimizer.m[1]["b"] = m_data["b1"]

v_data = np.load("asserts/agent_step_initial_v.npz")
test_agent.optimizer.v[0]["W"] = v_data["W0"]
test_agent.optimizer.v[0]["b"] = v_data["b0"]
test_agent.optimizer.v[1]["W"] = v_data["W1"]
test_agent.optimizer.v[1]["b"] = v_data["b1"]

# Assume the agent started at State 4
start_state = 4
test_agent.agent_start(start_state)

# Assume the reward was 10.0 and reached the terminal state
reward = 10.0
test_agent.agent_end(reward)

# updated weights asserts
agent_updated_weight_answer = np.load("asserts/agent_end_updated_weights.npz")
assert(np.allclose(test_agent.weights[0]["W"], agent_updated_weight_answer["W0"]))
assert(np.allclose(test_agent.weights[0]["b"], agent_updated_weight_answer["b0"]))
assert(np.allclose(test_agent.weights[1]["W"], agent_updated_weight_answer["W1"]))
assert(np.allclose(test_agent.weights[1]["b"], agent_updated_weight_answer["b1"]))
```

**Expected output:**

    updated_weights[0]["W"]
     [[ 1.10893459  0.30763738]
     [ 0.63690565  1.14778865]
     [ 1.17531054 -0.51043162]
     [ 0.75062903 -0.13736817]
     [ 0.15021996  0.39822163]]

    updated_weights[0]["b"]
     [[0.30846523 0.95937346]]

    updated_weights[1]["W"]
     [[0.68861703]
     [0.15986364]]

    updated_weights[1]["b"]
     [[0.586074]]

## Section 2 - Run Experiment

Now that you implemented the agent, we can run the experiment. Similar to Course 3 Programming Assignment 1, we will plot the learned state value function and the learning curve of the TD agent. To plot the learning curve, we use Root Mean Squared Value Error (RMSVE).

## 2-1: Run Experiment for Semi-gradient TD with a Neural Network

We have already provided you the experiment/plot code, so you can go ahead and run the two cells below.

Note that running the cell below will take **approximately 12 minutes**.

```python
# ---------------
# Discussion Cell
# ---------------

true_state_val = np.load('data/true_V.npy')
state_distribution = np.load('data/state_distribution.npy')

def calc_RMSVE(learned_state_val):
    assert(len(true_state_val) == len(learned_state_val) == len(state_distribution))
    MSVE = np.sum(np.multiply(state_distribution, np.square(true_state_val - learned_state_val)))
    RMSVE = np.sqrt(MSVE)
    return RMSVE

# Define function to run experiment
def run_experiment(environment, agent, environment_parameters, agent_parameters, experiment_parameters):

    rl_glue = RLGlue(environment, agent)

    # save rmsve at the end of each episode
    agent_rmsve = np.zeros((experiment_parameters["num_runs"],
                            int(experiment_parameters["num_episodes"]/experiment_parameters["episode_eval_frequency"]) + 1))

    # save learned state value at the end of each run
    agent_state_val = np.zeros((experiment_parameters["num_runs"],
                                environment_parameters["num_states"]))

    env_info = {"num_states": environment_parameters["num_states"],
                "start_state": environment_parameters["start_state"],
                "left_terminal_state": environment_parameters["left_terminal_state"],
                "right_terminal_state": environment_parameters["right_terminal_state"]}

    agent_info = {"num_states": environment_parameters["num_states"],
                  "num_hidden_layer": agent_parameters["num_hidden_layer"],
                  "num_hidden_units": agent_parameters["num_hidden_units"],
                  "step_size": agent_parameters["step_size"],
                  "discount_factor": environment_parameters["discount_factor"],
                  "beta_m": agent_parameters["beta_m"],
                  "beta_v": agent_parameters["beta_v"],
                  "epsilon": agent_parameters["epsilon"]
                 }

    print('Setting - Neural Network with 100 hidden units')
    os.system('sleep 1')

    # one agent setting
    for run in tqdm(range(1, experiment_parameters["num_runs"]+1)):
        env_info["seed"] = run
        agent_info["seed"] = run
        rl_glue.rl_init(agent_info, env_info)

        # Compute initial RMSVE before training
        current_V = rl_glue.rl_agent_message("get state value")
        agent_rmsve[run-1, 0] = calc_RMSVE(current_V)

        for episode in range(1, experiment_parameters["num_episodes"]+1):
            # run episode
            rl_glue.rl_episode(0) # no step limit

            if episode % experiment_parameters["episode_eval_frequency"] == 0:
                current_V = rl_glue.rl_agent_message("get state value")
                agent_rmsve[run-1, int(episode/experiment_parameters["episode_eval_frequency"])] = calc_RMSVE(current_V)
            elif episode == experiment_parameters["num_episodes"]: # if last episode
                current_V = rl_glue.rl_agent_message("get state value")

        agent_state_val[run-1, :] = current_V

    save_name = "{}".format(rl_glue.agent.name).replace('.','')

    if not os.path.exists('results'):
                os.makedirs('results')

    # save avg. state value
    np.save("results/V_{}".format(save_name), agent_state_val)

    # save avg. rmsve
    np.savez("results/RMSVE_{}".format(save_name), rmsve = agent_rmsve,
                                                   eval_freq = experiment_parameters["episode_eval_frequency"],
                                                   num_episodes = experiment_parameters["num_episodes"])

# Run Experiment

# Experiment parameters
experiment_parameters = {
    "num_runs" : 20,
    "num_episodes" : 1000,
    "episode_eval_frequency" : 10 # evaluate every 10 episode
}

# Environment parameters
environment_parameters = {
    "num_states" : 500,
    "start_state" : 250,
    "left_terminal_state" : 0,
    "right_terminal_state" : 501,
    "discount_factor" : 1.0
}

# Agent parameters
agent_parameters = {
    "num_hidden_layer": 1,
    "num_hidden_units": 100,
    "step_size": 0.001,
    "beta_m": 0.9,
    "beta_v": 0.999,
    "epsilon": 0.0001,
}

current_env = RandomWalkEnvironment
current_agent = TDAgent

# run experiment
run_experiment(current_env, current_agent, environment_parameters, agent_parameters, experiment_parameters)

# plot result
plot_script.plot_result(["td_agent"])

shutil.make_archive('results', 'zip', 'results')
```

You plotted the learning curve for 1000 episodes. As you can see the RMSVE is still decreasing. Here we provide the pre-computed result for 5000 episodes and 20 runs so that you can see the performance of semi-gradient TD with a neural network after being trained for a long time.

Does semi-gradient TD with a neural network find a good approximation within 5000 episodes?

As you may remember from the previous assignment, semi-gradient TD with 10-state aggregation converged within 100 episodes. Why is TD with a neural network slower?

Would it be faster if we decrease the number of hidden units? Or what about if we increase the number of hidden units?

## 2-2: Compare Performance of Semi-gradient TD with a Neural Network and Semi-gradient TD with Tile-coding

In this section, we compare the performance of semi-gradient TD with a Neural Network and semi-gradient TD with tile-coding. Tile-coding is a kind of coarse coding that uses multiple overlapping partitions of the state space to produce features. For tile-coding, we used 50 tilings each with 6 tiles. We set the step-size for semi-gradient TD with tile-coding to $\frac{0.1}{\text{# tilings}}$. See the figure below for the comparison between semi-gradient TD with tile-coding and semi-gradient TD with a neural network and Adam algorithm. This result is for 5000 episodes and 20 runs:

How are the results?

Semi-gradient TD with tile-coding is much faster than semi-gradient TD with a neural network. Why?

Which method has a lower RMSVE at the end of 5000 episodes?

### Wrapping up!

You have successfully implemented Course 3 Programming Assignment 2.

You have implemented **semi-gradient TD with a Neural Network and Adam algorithm** in 500-state Random Walk.

You also compared semi-gradient TD with a neural network and semi-gradient TD with tile-coding.

From the experiments and lectures, you should be more familiar with some of the strengths and weaknesses of using neural networks as the function approximator for an RL agent. On one hand, neural networks are powerful function approximators capable of representing a wide class of functions. They are also capable of producing features without exclusively relying on hand-crafted mechanisms. On the other hand, compared to a linear function approximator with tile-coding, neural networks can be less sample efficient. When implementing your own Reinforcement Learning agents, you may consider these strengths and weaknesses to choose the proper function approximator for your problems.


```python
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
```

```python
nx = 81 # tente mudar este número de 41 para 81 e rodar tudo novamente ... o que acontece?
dx = 2/(nx-1)
nt = 25
dt = .02
c = 1 # suponha velocidade da onda c = 1
x = np.linspace(0,2,nx)
u = np.ones(nx)      # numpy function ones()
u[int(.5 / dx):int(1 / dx + 1)] = 2  # setting u = 2 between 0.5 and 1 as per our I.C.s
print(x)
print(u)
plt.plot(x,u,'o-')
plt.grid()
```

```python
un = np.ones(nx) #initialize a temporary array

for n in range(nt):  #loop for values of n from 0 to nt, so it will run nt times
    un = u.copy() ##copy the existing values of u into un
    for i in range(1, nx): ## you can try commenting this line and...
    #for i in range(nx): ## ... uncommenting this line and see what happens!
        u[i] = un[i] - c * dt / dx * (un[i] - un[i-1])
```

```python
plt.plot(x, u, 'o-')
plt.grid()
```

## Implementação com Devito

```python
#NBVAL_IGNORE_OUTPUT
from examples.cfd import plot_field, init_hat
from devito import Grid, TimeFunction

grid = Grid(shape=(nx,), extent=(2.,))
u = TimeFunction(name='u', grid=grid)

u.data[0][:] = 1.
u.data[0][int(.5 / dx):int(1 / dx + 1)] = 2.
# Note que u.data[0] == u.data[0,:]
print(u.data[0],u.data.shape,u.data.size)
plt.plot(x,u.data[0],'o-')
plt.grid()
```

```python
from devito import Eq

# Specify the `interior` flag so that the stencil is only
# applied to the interior of the domain.
eq = Eq(u.dt + c*u.dxl, subdomain=grid.interior)

print(eq)
eq
```

```python
u.forward
```

```python
from devito import solve
from sympy import nsimplify, pprint

stencil = solve(eq, u.forward)

pprint(stencil)
```

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

# Reset our initial condition in both buffers.
# This is required to avoid 0s propagating into
# our solution, which has a background value of 1.
for i in range(u.data.shape[0]):
    u.data[i][:] = 1.
    u.data[i][int(.5 / dx):int(1 / dx + 1)] = 2.

# Create an operator that updates the forward stencil point
op = Operator(Eq(u.forward, stencil, subdomain=grid.interior))

# Apply the operator for a number of timesteps
op.apply(time=nt, dt=dt)
```

```python
plt.plot(np.linspace(0, 2, nx), u.data[0], 'o-')
plt.grid()
```

```python
print(op.ccode)
```

### Outra condição inicial

```python
nx = 41
L = 1.0
dx = L/(nx-1)
print(dx)
x = np.linspace(0,L,nx)
u = np.sin(2.0*np.pi*x)
plt.plot(x,u,'-o')
plt.grid()
```

```python
c = 1.0 # m/s - wave speed
dt = dx/c  # run just at the CFL condition
cfl = c*dt/dx
print(dt, int(1./dt))

un = u.copy() ##copy the existing values of u into un
u[1:-1] = un[1:-1] - cfl * (un[1:-1] - un[:-2])
u[-1] = un[-1] - cfl*(un[-1] - un[-2]) # compute last point on the right using periodicity
u[0] = u[-1] # set periodic boundary on the left
print(u.size,u.shape)
plt.plot(x,un,'-o',x,u,'--o')
plt.grid()
```

```python
grid = Grid(shape=(nx,), extent=(1.,))
u = TimeFunction(name='u', grid=grid)

u.data[0][:] = np.sin(2.0*np.pi*x)
# Note que u.data[0] == u.data[0,:]
print(u.data[0],u.data.shape,u.data.size)
plt.plot(x,u.data[0],'o-')
plt.grid()
```

```python
eq = Eq(u.dt + c*u.dxl, subdomain=grid.interior)

stencil = solve(eq, u.forward)
update = Eq(u.forward, stencil)#, subdomain=grid.interior)
update
```

```python
x = grid.dimensions
t = grid.stepping_dim
bc_left = Eq(u[t + 1, 0], u[t + 1, nx-1])
```

```python
for i in range(u.data.shape[0]):
    u.data[i][:] = np.sin(2.0*np.pi*np.linspace(0,L,nx))

#op2 = Operator([update], opt='noop')
op2 = Operator([update]+[bc_left], opt='noop')
op2(time=0, dt=dt)
```

```python
plt.plot(np.linspace(0,L,nx),u.data[0],'o-',np.linspace(0,L,nx),u.data[1],'--o')
plt.grid()
```

```python
print(op2.ccode)
```

```python
for i in range(u.data.shape[0]):
    u.data[i][:] = np.sin(2.0*np.pi*np.linspace(0,L,nx))

op2(time=nt, dt=dt)
```

```python
plt.plot(np.linspace(0,L,nx),u.data[1],'o-')
plt.grid()
```

## Modelagem sísmica

```python
import numpy as np
%matplotlib inline
```

```python
from examples.seismic import Model, plot_velocity

# Devine a physical size
shape = (101, 101) # Number of grid point (nz, nx)
spacing = (10., 10.) # Grid spacing in m. This domain size is now 1km by 1km
origin = (0., 0.) # What is the location of the top left corner. This is necessary to define

# Define a velocity profile. The velocity is in km/s
v = np.empty(shape, dtype=np.float32)
v[:, :51] = 1.5
v[:, 51:] = 2.5

# With the velocity and model size defined, we can create a seismic model that encapsulates this properties.
# we also define the size of the absorbing layer as 10 grid points.
model = Model(vp=v, origin=origin, shape=shape, spacing=spacing,
              space_order=2, nbl=30, bcs="damp")

plot_velocity(model)
```

### Acquisition geometry

```python
from examples.seismic import TimeAxis
t0 = 0. # Simulation starts a t=0
tn = 1000. # Simulation last 1 second (1000 ms)
dt = model.critical_dt # Time

time_range = TimeAxis(start=t0, stop=tn, step=dt)
```

```python
from examples.seismic import RickerSource

f0 = 0.010 # Source peak frequency is 10Hz (0.010 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   npoint=1, time_range=time_range)

src.coordinates.data[0,:] = np.array(model.domain_size) * 0.5
src.coordinates.data[0,-1] = 20. # Depth is 20m

src.show()
```

```python
from examples.seismic import Receiver

# Create symbol for 101 receivers
rec = Receiver(name='rec', grid=model.grid, npoint=101, time_range=time_range)

# Prescribe even spacing for receivers along the x-axis
rec.coordinates.data[:, 0] = np.linspace(0, model.domain_size[0], num=101)

# We can now show the source and receivers within our domain:
# Red dot: Source location
# Green dots: Receiver locations (every 4th point)
plot_velocity(model, source=src.coordinates.data,
              receiver=rec.coordinates.data[::4, :])
```

### Finite-difference discretization

```python
from devito import TimeFunction

# Define the wavefield with the size of the model and the time dimension
u = TimeFunction(name="u", grid=model.grid, time_order=2, space_order=2)

# We can now write the PDE
pde = model.m * u.dt2 - u.laplace + model.damp * u.dt

# The PDE representation is as on paper
pde
```

```python
# This discrete PDE can be solved in a time-marching way updating u(t+dt) from the previous time step
# Devito as a shotcut for u(t+dt) which is u.forward. We can then rewrite the PDE as a time marching
# updating equation known as a stencil using customized SymPy functions
from devito import Eq, solve

stencil = Eq(u.forward, solve(pde, u.forward))
stencil
```

### Source injection and receiver interpolation

```python
# Finally we define the source injection and receiver read function to generate the corresponding code
src_term = src.inject(field=u.forward, expr=src * dt**2 / model.m)

# Create interpolation expression for receivers
rec_term = rec.interpolate(expr=u.forward)
```

### Devito operator and solve

```python
from devito import Operator

op = Operator([stencil] + src_term + rec_term, subs=model.spacing_map)
```

```python
op(time=time_range.num-1, dt=model.critical_dt)
```

```python
from examples.seismic import plot_shotrecord

plot_shotrecord(rec.data, model, t0, tn)
```


## Two-dimensional advection equation

We want to solve the following PDE:

\begin{equation}
    \frac{\partial \phi}{\partial t} + u \frac{\partial \phi}{\partial x} + v \frac{\partial \phi}{\partial y} = 0
\end{equation}

The independen variables (i.e, $x$, $y$ and $t$) are used as input values for the NN, and the solution (i.e. $\phi(x,y,t)$) is the output. In order to find the solution, at each step the NN outputs are derived w.r.t the inputs. Then, a loss function that matches the PDE is built and the weights are updated accordingly. If the loss function goes to zero, we can assume that our NN is indeed the solution to our PDE. We will try to find a general solution for different values of $u$, so it will be set also as an input.

```python
# imports
import numpy as np
import matplotlib.pyplot as plt
import nangs
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

nangs.__version__, torch.__version__
```

    Bad key "text.kerning_factor" on line 4 in
    C:\Users\sensio\miniconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test_patch.mplstyle.
    You probably need to get an updated matplotlibrc file from
    https://github.com/matplotlib/matplotlib/blob/v3.1.3/matplotlibrc.template
    or from the matplotlib source distribution

    ('0.1.2', '1.5.0')

```python
from nangs import PDE

U, V = -0.5, 1

class Adv2d(PDE):
    def computePDELoss(self, inputs, outputs):

        # compute gradients
        grads = self.computeGrads(outputs, inputs)

        # compute loss
        dpdx, dpdy, dpdt = grads[:, 0], grads[:, 1], grads[:, 2]
        return {'pde': dpdt + U*dpdx + V*dpdy}

pde = Adv2d(inputs=('x', 'y', 't'), outputs='p')
```

```python
# define the mesh

from nangs import Mesh

x = np.linspace(0,1,20)
y = np.linspace(0,1,20)
t = np.linspace(0,1,20)

mesh = Mesh({'x': x, 'y': y, 't': t}, device=device)
pde.set_mesh(mesh)
```

```python
# initial condition (t = 0)

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from matplotlib import cm

t0 = np.array([0])
_x, _y = np.meshgrid(x, y)
p0 = np.sin(2*np.pi*_x)*np.sin(2*np.pi*_y)

fig = plt.figure()
ax = fig.gca(projection='3d')
surf = ax.plot_surface(_x, _y, p0.reshape((len(_y),len(_x))), cmap=cm.coolwarm, linewidth=0, antialiased=False)
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()
```

```python
from nangs import Dirichlet, Periodic

# initial condition
initial_condition = Dirichlet({'x': x, 'y': y, 't': t0}, {'p': p0.reshape(-1)}, device=device, name="initial")
pde.add_boco(initial_condition)

# periodic bocos
x1 = np.array([0.])
x2 = np.array([1.])
periodic1 = Periodic({'x': x1, 'y': y, 't': t}, {'x': x2, 'y': y, 't': t}, name="boco_x", device=device)
periodic2 = Periodic({'x': x, 'y': x1, 't': t}, {'x': x, 'y': x2, 't': t}, name="boco_y", device=device)
pde.add_boco(periodic1)
pde.add_boco(periodic2)
```

```python
from nangs import MLP

BATCH_SIZE = 256
LR = 1e-2
EPOCHS = 30
NUM_LAYERS = 3
NUM_HIDDEN = 128

mlp = MLP(len(pde.inputs), len(pde.outputs), NUM_LAYERS, NUM_HIDDEN).to(device)
optimizer = torch.optim.Adam(mlp.parameters())
scheduler = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr=LR, pct_start=0.1, total_steps=EPOCHS)

pde.compile(mlp, optimizer, scheduler)
%time hist = pde.solve(EPOCHS, BATCH_SIZE)
```

```python
t = 0.912312
x = np.linspace(0,1,100)
y = np.linspace(0,1,100)

_x, _y = np.meshgrid(x, y)
p0 = np.sin(2*np.pi*(_x - U*t))*np.sin(2*np.pi*(_y - V*t))

eval_mesh = Mesh({'x': x, 'y': y, 't': t}, device=device)
p = pde.eval(eval_mesh)
p = p.cpu().numpy()

fig = plt.figure(figsize=(15,5))
ax1 = fig.add_subplot(1, 2, 1, projection='3d')
ax2 = fig.add_subplot(1, 2, 2, projection='3d')
ax1.plot_surface(_x, _y, p0.reshape((len(_y),len(_x))), cmap=cm.coolwarm, linewidth=0, antialiased=False)
ax2.plot_surface(_x, _y, p.reshape((len(_y),len(_x))), cmap=cm.coolwarm, linewidth=0, antialiased=False)
plt.show()
print("L2 error: ", np.sqrt(sum((p0.ravel()-p.ravel())**2)))
```


```python
import numpy as np
import networkx as nx
import sympy as sym
import scipy as sci
import matplotlib.pyplot as plt

```

```python
from hypothesis import (
    given,
    settings,
)
from hypothesis.strategies import (
    floats,
    integers,
    booleans,
)
from hypothesis.extra.numpy import arrays

```

```python
from ambulance_game.markov.markov import (
    build_states,
    visualise_ambulance_markov_chain,
    get_transition_matrix_entry,
    get_symbolic_transition_matrix,
    get_transition_matrix,
    convert_symbolic_transition_matrix,
    is_steady_state,
    get_steady_state_numerically,
    augment_Q,
    get_steady_state_algebraically,
    get_mean_number_of_patients_in_system,
    get_mean_number_of_patients_in_hospital,
    get_mean_ambulances_blocked,
)

```

# build_states

```python
@given(
    threshold=integers(min_value=0, max_value=1000),
    system_capacity=integers(min_value=0, max_value=1000),
    buffer_capacity=integers(min_value=0, max_value=1000),
)
def test_build_states(threshold, system_capacity, buffer_capacity):

    states = build_states(
        threshold=threshold,
        system_capacity=system_capacity,
        buffer_capacity=buffer_capacity,
    )

    states_after_threshold = system_capacity - threshold + 1
    S_2_size = states_after_threshold if states_after_threshold >= 0 else 0
    all_states_size = S_2_size * (buffer_capacity + 1) + threshold

    assert len(states) == all_states_size

```

```python
test_build_states()

```

# visualise_ambulance_markov_chain

```python
@given(
    num_of_servers=integers(min_value=2, max_value=10),
    threshold=integers(min_value=2, max_value=10),
    buffer_capacity=integers(min_value=2, max_value=10),
    system_capacity=integers(min_value=2, max_value=10),
)
@settings(deadline=None)
def test_visualise_ambulance_markov_chain(
    num_of_servers, threshold, system_capacity, buffer_capacity
):

    all_states = build_states(
        threshold=threshold,
        system_capacity=system_capacity,
        buffer_capacity=buffer_capacity,
    )
    set_of_all_states = set(all_states)

    markov_chain_plot = visualise_ambulance_markov_chain(
        num_of_servers=num_of_servers,
        threshold=threshold,
        system_capacity=system_capacity,
        buffer_capacity=buffer_capacity,
    )
    set_of_nodes = set(markov_chain_plot.nodes)

    assert type(markov_chain_plot) == nx.classes.multidigraph.MultiDiGraph
    assert set_of_all_states == set_of_nodes
    plt.close()

```

```python
test_visualise_ambulance_markov_chain()

```

# get_transition_matrix_entry

```python
@given(
    ambulance_state=integers(min_value=0),
    hospital_state=integers(min_value=0),
    lambda_2=floats(min_value=0, allow_nan=False, allow_infinity=False),
    lambda_1=floats(min_value=0, allow_nan=False, allow_infinity=False),
    mu=floats(min_value=0, allow_nan=False, allow_infinity=False),
    num_of_servers=integers(min_value=1),
    threshold=integers(min_value=0),
    symbolic=booleans(),
)
def test_get_transition_matrix_entry(
    ambulance_state,
    hospital_state,
    lambda_2,
    lambda_1,
    mu,
    num_of_servers,
    threshold,
    symbolic,
):

    Lambda = lambda_2 + lambda_1

    if symbolic:
        Lambda = sym.symbols("Lambda")
        lambda_1 = sym.symbols("lambda") ** sym.symbols("o")
        lambda_2 = sym.symbols("lambda") ** sym.symbols("A")
        mu = sym.symbols("mu")

    origin_state = (ambulance_state, hospital_state)
    destination_state_1 = (ambulance_state, hospital_state + 1)
    destination_state_2 = (ambulance_state + 1, hospital_state)
    destination_state_3 = (ambulance_state, hospital_state - 1)
    destination_state_4 = (ambulance_state - 1, hospital_state)

    entry_1 = get_transition_matrix_entry(
        origin_state,
        destination_state_1,
        threshold=threshold,
        lambda_2=lambda_2,
        lambda_1=lambda_1,
        Lambda=Lambda,
        mu=mu,
        num_of_servers=num_of_servers,
    )
    entry_2 = get_transition_matrix_entry(
        origin_state,
        destination_state_2,
        threshold=threshold,
        lambda_2=lambda_2,
        lambda_1=lambda_1,
        Lambda=Lambda,
        mu=mu,
        num_of_servers=num_of_servers,
    )
    entry_3 = get_transition_matrix_entry(
        origin_state,
        destination_state_3,
        threshold=threshold,
        lambda_2=lambda_2,
        lambda_1=lambda_1,
        Lambda=Lambda,
        mu=mu,
        num_of_servers=num_of_servers,
    )
    entry_4 = get_transition_matrix_entry(
        origin_state,
        destination_state_4,
        threshold=threshold,
        lambda_2=lambda_2,
        lambda_1=lambda_1,
        Lambda=Lambda,
        mu=mu,
        num_of_servers=num_of_servers,
    )

    assert entry_1 == (Lambda if hospital_state < threshold else lambda_1)
    assert entry_2 == lambda_2
    assert entry_3 == (
        mu * hospital_state if hospital_state <= num_of_servers else mu * num_of_servers
    )
    assert entry_4 == (threshold * mu if hospital_state == threshold else 0)

```

```python
test_get_transition_matrix_entry()

```

# get_symbolic_transition_matrix

```python
@given(
    num_of_servers=integers(min_value=1, max_value=5),
    threshold=integers(min_value=0, max_value=5),
    system_capacity=integers(min_value=5, max_value=10),
    buffer_capacity=integers(min_value=1, max_value=5),
)
def test_get_symbolic_transition_matrix(
    num_of_servers, threshold, system_capacity, buffer_capacity
):

    states_after_threshold = system_capacity - threshold + 1
    S_2_size = states_after_threshold if states_after_threshold >= 0 else 0
    matrix_size = S_2_size * (buffer_capacity + 1) + threshold
    result = get_symbolic_transition_matrix(
        num_of_servers=num_of_servers,
        threshold=threshold,
        system_capacity=system_capacity,
        buffer_capacity=buffer_capacity,
    )

    assert matrix_size == result.shape[0]

```

```python
test_get_symbolic_transition_matrix()

```

# get_transition_matrix

```python
@given(
    system_capacity=integers(min_value=10, max_value=20),
    buffer_capacity=integers(min_value=1, max_value=20),
    lambda_2=floats(
        min_value=0.05, max_value=100, allow_nan=False, allow_infinity=False
    ),
    lambda_1=floats(
        min_value=0.05, max_value=100, allow_nan=False, allow_infinity=False
    ),
    mu=floats(min_value=0.05, max_value=5, allow_nan=False, allow_infinity=False),
)
def test_get_transition_matrix(
    system_capacity, buffer_capacity, lambda_2, lambda_1, mu
):

    num_of_servers = 10
    threshold = 8

    states_after_threshold = system_capacity - threshold + 1
    S_2_size = states_after_threshold if states_after_threshold >= 0 else 0
    matrix_size = S_2_size * (buffer_capacity + 1) + threshold

    transition_matrix = get_transition_matrix(
        lambda_2=lambda_2,
        lambda_1=lambda_1,
        mu=mu,
        num_of_servers=num_of_servers,
        threshold=threshold,
        system_capacity=system_capacity,
        buffer_capacity=buffer_capacity,
    )

    assert matrix_size == np.shape(transition_matrix)[0]
    mid = int(matrix_size / 2)
    assert transition_matrix[0][0] == -sum(transition_matrix[0][1:])
    assert transition_matrix[-1][-1] == -sum(transition_matrix[-1][:-1])

    mid_row_sum = sum(transition_matrix[mid][:mid]) + sum(
        transition_matrix[mid][mid + 1 :]
    )
    assert np.isclose(transition_matrix[mid][mid], -mid_row_sum)

```

```python
test_get_transition_matrix()

```

# convert_symbolic_transition_matrix

```python
@given(threshold=integers(min_value=0, max_value=10))
def test_convert_symbolic_transition_matrix(threshold):

    lambda_2 = 0.3
    lambda_1 = 0.2
    mu = 0.05
    num_of_servers = 10
    system_capacity = 8
    buffer_capacity = 2

    transition_matrix = get_transition_matrix(
        lambda_2=lambda_2,
        lambda_1=lambda_1,
        mu=mu,
        num_of_servers=num_of_servers,
        threshold=threshold,
        system_capacity=system_capacity,
        buffer_capacity=buffer_capacity,
    )

    sym_transition_matrix = get_symbolic_transition_matrix(
        num_of_servers=num_of_servers,
        threshold=threshold,
        system_capacity=system_capacity,
        buffer_capacity=buffer_capacity,
    )
    converted_matrix = convert_symbolic_transition_matrix(
        sym_transition_matrix, lambda_2, lambda_1, mu
    )

    assert np.allclose(converted_matrix, transition_matrix)

```

```python
test_convert_symbolic_transition_matrix()

```

# is_steady_state

```python
def test_is_steady_state():
    steady_1 = [6 / 17, 6 / 17, 5 / 17]
    generator_matrix_1 = np.array(
        [[-2 / 3, 1 / 3, 1 / 3], [1 / 2, -1 / 2, 0], [1 / 5, 1 / 5, -2 / 5]]
    )

    steady_2 = np.array([0.0877193, 0.38596491, 0.52631579])
    generator_matrix_2 = np.array([[-0.6, 0.4, 0.2], [0, -0.5, 0.5], [0.1, 0.3, -0.4]])

    steady_3 = np.array([1, 2, 3])
    generator_matrix_3 = np.array([[-4, 2, 2], [0, -2, 2], [1, 5, -6]])

    assert is_steady_state(steady_1, generator_matrix_1)
    assert is_steady_state(steady_2, generator_matrix_2)
    assert not is_steady_state(steady_3, generator_matrix_3)

```

```python
test_is_steady_state()

```

# get_steady_state_numerically

```python
@given(
    a=floats(min_value=0, max_value=10),
    b=floats(min_value=0, max_value=10),
    c=floats(min_value=0, max_value=10),
    d=floats(min_value=0, max_value=10),
    e=floats(min_value=0, max_value=10),
    f=floats(min_value=0, max_value=10),
)
def test_get_steady_state_numerically_odeint(a, b, c, d, e, f):
    Q = np.array([[-a - b, a, b], [c, -c - d, d], [e, f, -e - f]])
    steady = get_steady_state_numerically(Q, integration_function=sci.integrate.odeint)
    assert is_steady_state(steady, Q)

```

```python
test_get_steady_state_numerically_odeint()

```

```python
@given(
    a=floats(min_value=0, max_value=10),
    b=floats(min_value=0, max_value=10),
    c=floats(min_value=0, max_value=10),
    d=floats(min_value=0, max_value=10),
    e=floats(min_value=0, max_value=10),
    f=floats(min_value=0, max_value=10),
)
def test_get_steady_state_numerically_solve_ivp(a, b, c, d, e, f):
    Q = np.array([[-a - b, a, b], [c, -c - d, d], [e, f, -e - f]])
    steady = get_steady_state_numerically(
        Q, integration_function=sci.integrate.solve_ivp
    )
    assert is_steady_state(steady, Q)

```

```python
test_get_steady_state_numerically_solve_ivp()

```

# augment_Q

```python
@given(Q=arrays(np.int8, (10, 10)))
def test_augment_Q(Q):
    M, b = augment_Q(Q)
    assert M.shape == (10, 10)
    assert b.shape == (10, 1)
    assert all(b[0:-1]) == 0
    assert b[-1] == 1

```

```python
test_augment_Q()

```

# get_steady_state_algebraically

```python
@given(
    a=floats(min_value=1, max_value=10000),
    b=floats(min_value=1, max_value=10000),
    c=floats(min_value=1, max_value=10000),
    d=floats(min_value=1, max_value=10000),
    e=floats(min_value=1, max_value=10000),
    f=floats(min_value=1, max_value=10000),
)
def test_get_steady_state_algebraically_solve(a, b, c, d, e, f):
    Q = np.array([[-a - b, a, b], [c, -c - d, d], [e, f, -e - f]])
    steady = get_steady_state_algebraically(Q, algebraic_function=np.linalg.solve)
    assert is_steady_state(steady, Q)

```

```python
test_get_steady_state_algebraically_solve()

```

```python
@given(
    a=floats(min_value=1, max_value=10000),
    b=floats(min_value=1, max_value=10000),
    c=floats(min_value=1, max_value=10000),
    d=floats(min_value=1, max_value=10000),
    e=floats(min_value=1, max_value=10000),
    f=floats(min_value=1, max_value=10000),
)
def test_get_steady_state_algebraically_lstsq(a, b, c, d, e, f):
    Q = np.array([[-a - b, a, b], [c, -c - d, d], [e, f, -e - f]])
    steady = get_steady_state_algebraically(Q, algebraic_function=np.linalg.lstsq)
    assert is_steady_state(steady, Q)

```

```python
test_get_steady_state_algebraically_lstsq()

```

```python

```

```python

```

```python

```

```python
lambda_2 = 0.3
lambda_1 = 0.2
mu = 0.05
num_of_servers = 10
system_capacity = 8
buffer_capacity = 2
threshold = 0

```


# When do the hands of the clock face each other?
At which times are the hour and the minute hand of the clock located exaclty opposite of each other?

```python
# Imports
%matplotlib inline
from pylab import *
from scipy.optimize import fmin
rcParams['figure.figsize'] = [9.0, 6.0]
```

## Positionen der Zeiger
### Stundenzeiger
Der Winkel $\varphi$ beschreibt die Position des Stundenzeigers.
### Minutenzeiger
Die Position des Minutenzeigers $\varphi_m$ ist abhängig von $\varphi$: Bei einer Drehung des Stundenzeigers, dreht er sich 12 mal im Kreis:

$ \phi_m = \phi * 12 $.

Durch Modulorechnung mit $2 \pi$ wird ein Winkel $< 2 \pi$ bestimmt:

```python
phim = lambda phi: np.mod(phi*12.0, 2*np.pi)
```

### Gegenüberliegende Richtung
Die einem Zeiger gegenüberliegende Winkelposition wird durch Verdrehen um $\pi$ und eine erneute Modulorechnung bestimmt:

```python
gegenueber = lambda p: np.mod(p+np.pi, 2*np.pi)
```

## Grob bestimmte Zeitpunkte

Zur Bestimmung der ungefähren Zeitpunkte werden die Zeigerpositionen für 500 Zeitpunkte ausgewertet:

```python
phi = np.linspace(-0.1, 2*np.pi+0.1, 500)
```

Die interessierenden Zeitpunkte sind die, bei denen die Stundenzeigerposition gleich der dem
Minutenzeiger gegeüberliegenden ist:

$\varphi$ == gegenueber($\varphi_m$).

Es weden die Indices des Vektors *phi* gesucht, bei denen sich die beiden Funktionswerte schneiden, d.h. die Indices $i$ bei denen

$phi_i >= gegenueber(phim_i)$

ist und

$phi_{i+1} < gegenueber(phim_{i+1})$.

```python
idc = np.where((phi[:-1]>=gegenueber(phim(phi[:-1])))*
               (phi[1:]<gegenueber(phim(phi[1:]))))[0]
print('Stundenzeigerpositionen: ', phi[idc])
```

    Stundenzeigerpositionen:  [ 0.2767783   0.84844194  1.42010557  1.99176921  2.56343284  3.13509648
      3.70676011  4.27842374  4.85008738  5.42175101  5.99341465]

```python
# Dartstellung der Winkelverläufe und der grob bestimmten Schnittpunkte:
plot(phi, phi, label='$phi$')
plot(phi, phim(phi), ':', label='$phi_m$')
plot(phi, gegenueber(phim(phi)), label='$phi_m^g$')
plot(phi[idc], phi[idc], '*', label='phi_X')
legend()
grid(True)
```

## Numerische Optimierung: Bestimmung der Schnittpunkte

```python
fun = lambda p: (p-gegenueber(phim(p)))**2

phi_X = [fmin(fun, np.array([phi[i]]), disp=False, ftol=1e-12)[0]
         for i in idc]

print(phi[idc]-phi_X)
```

    [-0.008821   -0.00835612 -0.00789107 -0.00742609 -0.0069611  -0.00649615
     -0.00603114 -0.00556542 -0.00510124 -0.00463633 -0.00417129]

```python
plot(phi, phi, label='$phi$')
plot(phi, gegenueber(phim(phi)), label='$phi_m^g$')
plot(phi[idc], phi[idc], 'o', label='phi_X')
plot(phi[idc], phi_X, '*', label='phi_Xopt')
legend()

```

```python
for p in phi_X:
    hf = p/2/np.pi*12
    h = int(hf)
    m = int((hf-h)*60)
    s = (hf-h-m/60)*60*60
    print('{h:02d}:{m:02d}\'{s:2.4f}'.format(h=h, m=m, s=s))
```

    00:32'43.6362
    01:38'10.9095
    02:43'38.1817
    03:49'5.4544
    04:54'32.7270
    05:59'59.9998
    07:05'27.2722
    08:10'54.5399
    09:16'21.8180
    10:21'49.0912
    11:27'16.3634

```python
# Gütefunktion:
plot(phi, fun(phi))
```

```python
plot(phi, phi*12+np.pi, label='$phi_{gegenüber}$')
for offs in range(12):
    plot(phi, phi+offs*2*np.pi, 'g', label='$phi$')
plot(phi[idc], phi[idc]*12+np.pi, 'o', label='phi_Xopt')
legend(('gegenüber Minutenzeiger', 'Stundenzeiger + $n\pi$'))
xlabel('Winkel des Stundenzeigers')
ylabel('Winkel')
grid(True)
```

# Symbolisch: Einfacher?

```python
from sympy import *
init_printing()
phi, phim, n, h = symbols('varphi, varphi_m, n, h')
```

$ h = phi/{2\pi} *12$

$\varphi == gegenueber(\varphi_m) - n * 2\pi$

$\varphi == 12 \varphi + \pi - n 2\pi$

$\varphi - 12 \varphi - \pi + n  2\pi == 0$

```python
phi = h/12 * 2 * pi
phim = 12 * phi
eq = phi + n*2*pi-phim+pi
pprint(eq)
h_eq = solve(eq, h)[0]
h_eq
```

```python
for val in range(1, 12):
    hS = (h_eq).subs(n, val).n(20)
    print('{org:015.12f}: {h:02.0f}:{m:02.0f}\'{s:08.5f}"'.format(
        org=float(hS),
        h=float(np.floor(hS)),
        m=float(np.floor(np.mod(hS, 1.0)*60)),
        s=float(np.mod((np.mod(hS, 1.0)*60), 1.0)*60)))

```

    01.636363636364: 01:38'10.90909"
    02.727272727273: 02:43'38.18182"
    03.818181818182: 03:49'05.45455"
    04.909090909091: 04:54'32.72727"
    06.000000000000: 06:00'00.00000"
    07.090909090909: 07:05'27.27273"
    08.181818181818: 08:10'54.54545"
    09.272727272727: 09:16'21.81818"
    10.363636363636: 10:21'49.09091"
    11.454545454545: 11:27'16.36364"
    12.545454545455: 12:32'43.63636"

## Zeiger in die gleiche Richtung

```python
phi = h/12 * 2 * pi
phim = 12 * phi
eq = phi + n*2*pi - phim
pprint(eq)
h_eq = solve(eq, h)[0]
h_eq
```

```python
for val in range(1, 12):
    hS = (h_eq).subs(n, val).n(20)
    print('{org:015.12f}: {h:02.0f}:{m:02.0f}\'{s:08.5f}"'.format(
        org=float(hS),
        h=float(np.floor(hS)),
        m=float(np.floor(np.mod(hS, 1.0)*60)),
        s=float(np.mod((np.mod(hS, 1.0)*60), 1.0)*60)))

```

    01.090909090909: 01:05'27.27273"
    02.181818181818: 02:10'54.54545"
    03.272727272727: 03:16'21.81818"
    04.363636363636: 04:21'49.09091"
    05.454545454545: 05:27'16.36364"
    06.545454545455: 06:32'43.63636"
    07.636363636364: 07:38'10.90909"
    08.727272727273: 08:43'38.18182"
    09.818181818182: 09:49'05.45455"
    10.909090909091: 10:54'32.72727"
    12.000000000000: 12:00'00.00000"

```python

```


```python
from scipy.interpolate import lagrange
def f(x):
    return 1/(1+x*x)
x=[-1,-0.8,-0.6,-0.4,-0.2,0,0.2,0.4,0.6,0.8,1]
y=[0.047058823529411764,0.058823529411764705,0.07547169811320754,0.1,0.13793103448275862,0.2,0.3076923076923077,0.5,0.8,1.0,0.8,0.5,0.3076923076923077,0.2,0.13793103448275862,0.1,0.07547169811320754,0.058823529411764705,0.047058823529411764]
a=lagrange(x,y)
print(a)
a(0.75)
```

```python
from sympy import *
import math
from decimal import *
```

```python
def h(x,y,a):
    ans=0.0
    for i in range(len(y)):
        t=y[i]
        for j in range(len(y)):
            if i !=j:
                t*=(a-x[j])/(x[i]-x[j])
        ans +=t
    return ans

x=[-5.0,-4.5,-4,-3.5,3.0,-2.5,-2,-1.5,-1,-0.5,0,0.5,1,1.5,2,2.5,3,3.5,4,4.5,5]
y=[0.038461538461538464,0.047058823529411764,0.058823529411764705,0.07547169811320754,0.1,0.13793103448275862,0.2,0.3076923076923077,0.5,0.8,1.0,0.8,0.5,0.3076923076923077,0.2,0.13793103448275862,0.1,0.07547169811320754,0.058823529411764705,0.047058823529411764,0.038461538461538464
print(h(x,y,0.75))
```

```python
def f(x):
    return 1/(1+x*x)
x=-0.05
f(x)
```

    0.9975062344139651

```python
from sympy import *
import math
from decimal import *
def f(x):
    return exp(x)
x=4.75
f(x)
```

    115.584284527188

```python
from sympy import *
import math
from decimal import *
def f(k):
    return cos(((2*k+1)*pi)/(2*(10+1)))
print('%f'%f(6))
```

    -0.281733

```python
from sympy import *
import math
from decimal import *
from scipy.interpolate import lagrange
def f(x):
    return exp(x)
x=[0.989821,0.909632,0.755750,0.540641,0.281733,0,-0.281733,-0.540641,-0.755750,-0.909632,-0.989821]
y=[f(0.989821),f(0.909632),f(0.755750),f(0.540641),f(0.281733),f(0),f(-0.281733),f(-0.540641),f(-0.755750),f(-0.909632),f(-0.989821)]
a=lagrange(x,y)
print(a)
a(0.95)
```

               10             9            8             7            6
    2.814e-07 x  + 2.825e-06 x + 2.48e-05 x + 0.0001983 x + 0.001389 x
                 5           4          3       2
     + 0.008333 x + 0.04167 x + 0.1667 x + 0.5 x + 1 x + 1

    2.58570965934257

```python
from sympy import *
import math
from decimal import *
from scipy.interpolate import lagrange
def f(x):
    return exp(x)
x=[0.965926,0.707107,0.258819,-0.258819,-0.707107,-0.965926]
y=[f(0.965926),f(0.707107),f(0.258819),f(-0.258819),f(-0.707107),f(-0.965926)]
a=lagrange(x,y)
print(a)
a(0.95)
```

              5           4          3          2
    0.008636 x + 0.04379 x + 0.1666 x + 0.4992 x + 1 x + 1

    2.58572653329289

```python
import math
from scipy.interpolate import lagrange
def f(x):
    return exp(x)
x=[-1,-0.8,-0.6,-0.4,-0.2,0,0.2,0.4,0.6,0.8,1]
y=[f(-1),f(-0.8),f(-0.6),f(-0.4),f(-0.2),f(0),f(0.2),f(0.4),f(0.6),f(0.8),f(1)]
a=lagrange(x,y)
print(a)
print('%f'%a(0.05))
```

               10             9            8             7            6
    2.802e-07 x  + 2.811e-06 x + 2.48e-05 x + 0.0001984 x + 0.001389 x
                 5           4          3       2
     + 0.008333 x + 0.04167 x + 0.1667 x + 0.5 x + 1 x + 1
    1.051271

```python
def f(x):
    return -2.262e-05*x**10  - 3.611e-20*x**9 + 0.001267*x**8 + 7.619e-18*x**7 - 0.02441*x**6+1.046e-16*x**5 + 0.1974*x**4 - 1.3e-16*x**3 - 0.6742*x**2 - 1.546e-16*x + 1
x=0.95
f(x)
```

    0.5352014509567211

```python
f(0.95)
```

    2.58570965931585

```python
from scipy.interpolate import lagrange
def f(x):
    return 1/(1+x*x)
x=[-1,-0.8,-0.6,-0.4,-0.2,0,0.2,0.4,0.6,0.8,1]
y=[f(-1),f(-0.8),f(-0.6),f(-0.4),f(-0.2),f(0),f(0.2),f(0.4),f(0.6),f(0.8),f(1)]
a=lagrange(x,y)
print(a)
a(0.95)
```

             10             9          8             7          6
    -0.1858 x  + 1.393e-12 x + 0.5946 x + 2.734e-12 x - 0.8988 x
                  5          4             3          2
     + 9.216e-13 x + 0.9897 x + 1.027e-13 x - 0.9997 x + 3.305e-16 x + 1

    0.526407983103465

```python
from scipy.interpolate import lagrange
def f(x):
    return 1/(1+x*x)
x=[0.965926,-0.6,-0.2,0.2,0.6,1]
y=[f(-1),f(-0.6),f(-0.2),f(0.2),f(0.6),f(1)]
a=lagrange(x,y)
print(a)
a(0.05)
```

               5          4             3          2
    6.661e-16 x + 0.3535 x - 7.244e-15 x - 0.8484 x + 2.724e-16 x + 0.9949

    0.9927906709558821

```python
from sympy import *
import math
from decimal import *
from scipy.interpolate import lagrange
def f(x):
    return exp(x)
x=[-5,-4,-3,-2,-1,0,1,2,3,4,5]
y=[f(-5),f(-4),f(-3),f(-2),f(-1),f(0),f(1),f(2),f(3),f(4),f(5)]
a=lagrange(x,y)
print(a)
print('%f'%a(4.75))
```

               10             9             8             7           6
    4.166e-07 x  + 4.507e-06 x + 2.202e-05 x + 0.0001636 x + 0.00141 x
                 5           4          3       2
     + 0.008601 x + 0.04161 x + 0.1659 x + 0.5 x + 1.001 x + 1
    115.607360

```python
from sympy import *
import math
from decimal import *
from scipy.interpolate import lagrange
def f(x):
    return sqrt(x)
x=[169,196,225]
y=[f(169),f(196),f(225)]
a=lagrange(x,y)
print(a)
a(185)
```

                2
    -4.561e-05 x + 0.05369 x + 5.23

    13.6006203247583

```python
print('%f'%sqrt(185))
```

    13.601471

```python

```