# Bayesian Model Example - The Deregulation Act

Basic idea: have taxi- and non-taxi-related crime (i.e. what Jacob is doing) diverged after the deregulation act?

Method:
 - Calculate the difference in proportions between the two datasets.
 - Model the change in proportion following the text messaging example from Probabilistic Programming book. I.e.:
   - Model the difference using a Poisson distribution (defined by $\lambda$ parameter)
   - Assume that two distributions exist: one before the Act, and one after.
   - Define $\tau$ as the point at which they switch (i.e. when the Act came into force).
   - Create two $\lambda$ parameters - one before $\tau$ and one after.
   - See what the posterior distributions of $\lambda_1$, $\lambda_2$, and $\tau$ are. Do they correspond with the introduction of the Act?

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import scipy.stats as stats
from IPython.core.pylabtools import figsize

import pymc3 as pm
import theano.tensor as tt
```

    /Users/nick/anaconda3/envs/py36/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

## Data

Prepare the data

```python
data = pd.read_csv("./data/taxi_crime.csv")

date = np.array(data.loc[:,"Month"])
taxi_crime = np.array(data.loc[:,"Taxi Crime"])
all_crime = np.array(data.loc[:,"All Crime"])
```

Plot them

```python
figsize(12.5, 4)
plt.plot(np.arange(len(date)), taxi_crime*100, color="#348ABD", label="Taxi crime (*100)")
plt.plot(np.arange(len(date)), all_crime, color="#00AA00", label="All crime")
plt.xlabel("Time (Months)")
plt.ylabel("Count")
plt.title("Crime volumes")
plt.legend()
```

As _all crime_ has much larger volumes than _taxi crime_, index to the first point and then calculate the difference. This will give them comparable sizes.

```python
taxi_i = np.array( [ x /  taxi_crime[0]  for x in taxi_crime ] )
all_i  = np.array( [ x /  all_crime[0]  for x in all_crime ] )
diff = all_i - taxi_i
```

```python
figsize(12.5, 4)
plt.plot(np.arange(len(date)), taxi_i, color="#00AA00", label="Taxi indexed")
plt.plot(np.arange(len(date)), all_i, color="#348ABD", label="All indexed")
plt.plot(np.arange(len(date)), diff, color="#A60628", label="Difference")
plt.xlabel("Time (Months)")
plt.ylabel("Count")
plt.legend()
```

It doesn't work very well using these noisy data. So try calculating the difference between the two proportions as a 12-month rolling average

```python
avg_diff = []
avg_taxi = []
avg_all = []
for i in range(12, len(taxi_i)):
    taxi_mean =  np.mean(taxi_i[i-12:i])
    all_mean =  np.mean(all_i[i-12:i])
    avg_taxi.append(taxi_mean)
    avg_all.append(all_mean)
    avg_diff.append(all_mean - taxi_mean)

avg_diff = np.array(avg_diff)
avg_taxi = np.array(avg_taxi)
avg_all  = np.array(avg_all)
```

```python
figsize(12.5, 4)
plt.plot(np.arange(len(avg_diff)), avg_taxi, color="#00AA00", label="Taxi indexed 12-month average")
plt.plot(np.arange(len(avg_diff)), avg_all, color="#348ABD", label="All indexed 12-month average")
plt.plot(np.arange(len(avg_diff)), avg_diff, color="#A60628", label="Difference 12-month average")
plt.xlabel("Time (Months)")
plt.ylabel("Count")
plt.title("12-month rolling average differences")
plt.legend()
```

## Model Conception

We want to model the difference between the two indexed crime volumes. The difference is continuous and could be positive or negative. Therefore lets assume that the difference can be modelled using a normal distribution:

$$X \sim N(\mu, 1/\tau)$$

The values of $\mu$ and $\tau$ determine the shape of the distribution. The $\tau$ will give us information about the spread of the difference. $\mu$ is especially useful: a _larger_ value for $\mu$ will higher probability of _larger_ variable values being chosen. Therefore if a model suggests that $\mu$ increases after some point (i.e. the introduction of the Deregulation Act) then it would appear that the differnce has increased.

```python

nor = stats.norm
x = np.linspace(-8, 7, 150)
mu = (-2, 0, 3)
tau = (.7, 1, 2.8)
colors = ["#348ABD", "#A60628", "#7A68A6"]
parameters = zip(mu, tau, colors)

for _mu, _tau, _color in parameters:
    plt.plot(x, nor.pdf(x, _mu, scale=1./_tau),
             label="$\mu = %d,\;\\tau = %.1f$" % (_mu, _tau), color=_color)
    plt.fill_between(x, nor.pdf(x, _mu, scale=1./_tau), color=_color,
                     alpha=.33)

plt.legend(loc="upper right")
plt.xlabel("$x$")
plt.ylabel("density function at $x$")
plt.title("Probability distribution of three different Normal random \
variables");
```

We want to test whether at some point (e.g. at the introduction of the Deregulation Act) the difference in crime volumes changed. Call this point $z$. After $z$, the difference suddenly becomes larger; $\mu$ increases. Therefore there are two $\mu$ parameters, one before $z$ and one after (and each will have its own $\tau$ parameter):

$$
X \sim N
\begin{cases}
    (\mu_1, 1/\tau_1 ) & \text{if } t \lt z \cr
    (\mu_2, 1/\tau_2 ) & \text{if } t \ge z
\end{cases}
$$

If, in reality, no sudden change occurred and indeed $\mu_1 = \mu_2$, then the $\mu$s posterior distributions should look about equal.

We are interested in inferring the unknown $\mu$s and $\tau$s. To use Bayesian inference, we need to assign prior probabilities to their different possible values. $\mu$ is continuous and might be negative (this would happen if taxi crime were more abundant than all other crime), so use a normal distrubtion again. $\tau$ is also continuous, but cannot be negative, so use an exponential distribution.  Each of those distributions has their own parameters, so we now have six _hyper-parameters_ (two for each of the $\mu$s and one for each $\tau$). Name them $\alpha$, $\beta$ and $\gamma$:

$$
\begin{align}
& \mu_1 \sim \text{N} (\alpha_1, 1/\beta_1) \text{ , } \tau_1 \sim \text{Exp} (\gamma_1) \\\
& \mu_2 \sim \text{N} (\alpha_2, 1/\beta_2) \text{ , } \tau_2 \sim \text{Exp} (\gamma_2)
\end{align}
$$

The $\alpha$, $\beta$ and $\gamma$ parameters influence other parameters, so our initial guesses do not influence the model too strongly, so we have some flexibility in our choice.

We also need a distribution for the point at which the difference changes ($z$). Assign a *uniform prior belief* to every possible time point. This will reduce any personal bias.

\begin{align}
& z \sim \text{DiscreteUniform(1,N) }\\\\
& \Rightarrow P( \tau = k ) = \frac{1}{N}
\end{align}

where $N$ is the number of time points in the data.

## Model Definition

Now define the model using PYMC. (_Note that we use the same values for the hyperparameters, although above I talked about them as separate variables for each of the two Normal distributions_).

```python

with pm.Model() as model:
    # The hyperparameters
    alpha = 1.0
    beta = 1.0
    gamma = 1.0/avg_diff.mean()

    # The two mean and sd variables parameters.
    # (We want to see if their posterior distributions change)
    mu_1 = pm.Normal("mu_1", mu = alpha, tau = beta)
    mu_2 = pm.Normal("mu_2", mu = alpha, tau = beta)

    tau_1 = pm.Exponential("tau_1", gamma)
    tau_2 = pm.Exponential("tau_2", gamma)
    #taus = pm.Exponential("taus", gamma, shape=2)

    # Z is the point at which the texting behaviour changed.
    # Don't know anything about this so assign a uniform prior belief
    z = pm.DiscreteUniform("z", lower=0, upper=len(avg_diff) - 1)

    # Creates new functions using switch() to assign the correct mu or tau variables
    # These can be thought of as the random variables mu and tau definied initially.
    idx = np.arange(len(avg_diff)) # Index
    mu_ =  pm.math.switch(z > idx, mu_1, mu_2)
    tau_ = pm.math.switch(z > idx, tau_1, tau_2)

    # Combine the observation data (we assume that it has been generated by
    # a normal distribution with our mu and tau parameters).
    obs = pm.Normal("obs", mu=mu_, tau=tau_, observed=avg_diff)
```

The model has been defined, now we can perform MCMC to sample from the posterior distribution

```python
N = 5000 # Number of samples
with model:
    step = pm.Metropolis()
    start = pm.find_MAP() # Help it to start from a good place
    trace = pm.sample(N, tune=int(N*2), step=step, start=start, njobs=4)
```

    logp = -47.479, ||grad|| = 0.19387: 100%|██████████| 17/17 [00:00<00:00, 1129.13it/s]
    100%|██████████| 15000/15000 [00:13<00:00, 1083.95it/s]

The model has finished. Now example our samples

```python
mu_1_samples = trace['mu_1']
mu_2_samples = trace['mu_1']
tau_1_samples = trace['tau_1']
tau_2_samples = trace['tau_2']
z_samples = trace['z']
```

```python
figsize(12.5, 10)
#histogram of the samples:

# MUs

ax = plt.subplot(321) # rows, columns, index
ax.set_autoscaley_on(False)
plt.hist(mu_1_samples, histtype='stepfilled', bins=100, alpha=0.85,
         label="posterior of $\mu_1$", color="#A60628", normed=True)
plt.legend(loc="upper left")
plt.title(r"""Posterior distributions of the variables""")
#plt.xlim([15, 30])
#plt.xlabel("$\lambda_1$ value")

ax = plt.subplot(322) # rows, columns, index
ax.set_autoscaley_on(False)
plt.hist(mu_2_samples, histtype='stepfilled', bins=100, alpha=0.85,
         label="posterior of $\mu_2$", color="#A60628", normed=True)
plt.legend(loc="upper left")
#plt.xlim([15, 30])
#plt.xlabel("$\lambda_1$ value")

# TAUs

ax = plt.subplot(323)
ax.set_autoscaley_on(False)
plt.hist(tau_1_samples, histtype='stepfilled', bins=100, alpha=0.85,
         label="posterior of $\tau_1$", color="#7A68A6", normed=True)
plt.legend(loc="upper left")
#plt.xlim([15, 30])
plt.xlabel("$\tau_1$ value")

ax = plt.subplot(324)
ax.set_autoscaley_on(False)
plt.hist(tau_2_samples, histtype='stepfilled', bins=100, alpha=0.85,
         label="posterior of $\tau_2$", color="#7A68A6", normed=True)
plt.legend(loc="upper left")
#plt.xlim([15, 30])
plt.xlabel("$\tau_2$ value")

plt.subplot(325)
w = 1.0 / z_samples.shape[0] * np.ones_like(z_samples)
plt.hist(z_samples, bins=len(diff), alpha=1,
         label=r"posterior of $z$",
         color="#467821", weights=w, rwidth=2.)
plt.xticks(np.arange(len(diff)))

plt.legend(loc="upper left")
#plt.ylim([0, .75])
#plt.xlim([35, len(dif)-20])
plt.xlabel(r"$z$ (in months)")
plt.ylabel("probability");
```

Some more diagnostic visualisations

```python
pm.plots.traceplot(trace=trace, varnames=["mu_1", "mu_2", "tau_1", "tau_2"])
```

```python
pm.plots.plot_posterior(trace=trace, varnames=["mu_1", "mu_2", "tau_1", "tau_2", "z"])
#pm.plots.plot_posterior(trace=trace["centers"][:,1])
#pm.plots.autocorrplot(trace=trace, varnames=["centers"]);
```

```python

```

```python

```

## Next steps:

Think about How to detect structural change in a timeseries.

E.g.: [https://stats.stackexchange.com/questions/16953/how-to-detect-structural-change-in-a-timeseries]

    Can involve Jose too.

```python

```


# PRACTICE DESIGNING CLOSED LOOP SYSTEMS

This notebook describes the proportional, integral, and differential controllers.

# Preliminaries

```python
!pip install -q control
!pip install -q tellurium
!pip install -q controlSBML

import control
import controlSBML as ctl
from IPython.display import HTML, Math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import sympy
import tellurium as te
```

```python
TIMES = ctl.makeSimulationTimes(0, 5, 500)
```

```python
# Constants
CONSTANT_DCT = {"k1": 1, "k2": 2, "k3": 3, "k4": 4}
s = sympy.Symbol("s")
REF = 10
```

```python
MODEL1 = """
$S1 -> S2; k1*$S1
J1: S2 -> S3; k2*S2
J2: S3 -> S2; k3*S3
J3: S2 -> ; k4*S2

k1 = 1
k2 = 2
k3 = 3
k4 = 4
$S1 = 10
S2 = 0
S3 = 0
S4 = 0
"""
RR = te.loada(MODEL1)
RR.simulate()
RR.plot()
```

# Helpers

```python
def plotTFs(Gs, kp=0, ki=0, kd=0, times=TIMES, ylim=None, title=None, is_plot=True):
    """
    Constructs the transfer functions for the proportional controller, and filter.
    Calculates the transfer functions HRYs, HREs, HNYs, HDYs and plots them.

    Parameters
    ----------
    Gs: control.TransferFunction
    kp: float
    ki: float
    kd: float
    times: list-float
    ylim: (float, float)
        limits of y-values
    title: str

    Returns
    -------
    dct
        key: name of transfer function
        value: control.TransferFunction
    """
    Cs = control.TransferFunction([kp], [1]) + control.TransferFunction([ki], [1, 0])  \
        + control.TransferFunction([kd, 0], [1])
    Fs = 1
    denom = 1 + Cs*Gs*Fs
    # Construct the transfer functions
    tf_dct = {
        "HRYs": Cs*Gs/denom,
        "HREs": 1/denom,
        "HNYs": -Fs/denom,
        "HDYs": Cs/denom,
    }
    # Construct the plots
    _, ax = plt.subplots(1)
    for tf in tf_dct.values():
        result = control.forced_response(tf, T=times, U=1)
        plt.plot(result.t.flatten(), result.y.flatten())
    # Refine plots
    plt.legend(list(tf_dct.keys()))
    xmax = max(result.t.flatten())
    plt.plot([0, xmax], [0, 0], linestyle="--", color="black")
    plt.plot([0, xmax], [1, 1], linestyle="--", color="grey")
    plt.ylim([-5, 5])
    title = "kp=%2.2f ki=%2.2f kd=%2.2f" % (kp, ki, kd)
    plt.title(title)
    if not is_plot:
        plt.close()
    return tf_dct

# Tests
Gs = control.TransferFunction([2], [1, 3])
dct = plotTFs(Gs, kp=10, ylim=[0, 3], title="Example", is_plot=False)
assert(len(dct) == 4)
assert("TransferFunction" in str(type(dct["HRYs"])))
print("OK!")
```

    OK!

```python
def makeHRY(model, input_name, output_name, time=0, kp=0, ki=0, kd=0):
    """
    Calculates the transfer function from the reference input to the output.

    Parameters
    ----------
    model: str
    input_name: str
    output_name: str
    time: float
    kp: float
    ki: float
    kd: float

    Returns
    -------
    control.TransferFunction
    """
    ctlsb = ctl.ControlSBML(model, input_names=[input_name], output_names=[output_name])
    Gs = ctlsb.makeTransferFunction(time=time)
    dct = plotTFs(Gs, kp=kp, ki=ki, kd=kd, is_plot=False)
    return dct["HRYs"]

# TESTS
tf = makeHRY(MODEL1, "S2", "S3", kp=0)
assert(tf.dcgain() == 0)
print("OK!")
```

    OK!

# More On Filters

How do filters help with sinusoidal noise and disturbances?

We consider filters with a DC gain of one.
That is, given the negative $a$, the filter is $-a e^{at}$.
This has the transfer function $\frac{a}{a + s}$.

```python
# Create a testsbed
factory = ctl.IOSystemFactory()
fltr = factory.makeFilter("fltr", -1)
noise = factory.makeSinusoid("noise", 1, 20)
```

```python
# Create the testbed
test_bed = control.interconnect(
  [noise, fltr ],
  connections=[
    ['fltr.in', 'noise.out'],    # e(t)

  ],
  outlist=["fltr.out"],
)
```

```python
# Simulate it
times = ctl.makeSimulationTimes(0, 20, 500)
result = control.input_output_response(test_bed, T=times)
plt.plot(result.t, result.outputs.flatten())
plt.ylim([-1, 1])
```

# PID Controllers

The controllers considered here are systems that input the control error and produce a control signal used
to regulate the system under control.
The input signal is $e(t)$ and the output signal is $u(t)$.

A **proportional controller** has the parameter $k_P$.
This controller outputs a signal that is proportional to the control error.
That is, $u(t) = k_P e(t)$.
The transfer function for this controller is $H_{P} (s) = k_p$.

An **integral controller** has the parameter $k_I$.
This controller outputs a signal that is proportional to the *integral* of the control error.
That is, $u(t) = k_I \int_0^{t} e(\tau) d \tau$.
The transfer function for this controller is $H_{I} (s) = \frac{k_I}{s}$.

An **differential controller** has the parameter $k_D$.
This controller outputs a signal that is proportional to the derivative of the control error.
That is, $u(t) = k_D \frac{de(t)}{d t}$.
The transfer function for this controller is $H_{D} (s) = s k_D$.

These controllers can be used in combination.
For example, a PI controller produces
$u(t) = k_P e(t) +  k_I \int_0^{t} e(\tau) d \tau + k_D \frac{d e(t)}{d t}$,
and its transfer function is $H_{CPI}(s) = k_P + \frac{k_I}{s} + s k_D$.

# Model of Glycoytic Oscillations (BioModels 206)

```python
MODEL = "https://www.ebi.ac.uk/biomodels/model/download/BIOMD0000000206.2?filename=BIOMD0000000206_url.xml"
```

**Reactions in Glycolytic Oscillations**

``at`` is ``ATP``; ``s5`` is ``pyruvate``.

    v1: s1 + 2 at_ -> s2; compartment_*k1*s1*at_/(1 + (at_/ki)^n);
    v2: s2 -> 2 s3; compartment_*k2*s2;
    v3: s3 + na -> s4 + at_;compartment_*((k31*k32*s3*na*(atot - at_)
        - k33*k34*s4*at_*(ntot - na))/(k33*(ntot - na) + k32*(atot - at_)));
    v4: s4 -> s5 + at_; compartment_*k4*s4*(atot - at_);
    v5: s5 -> s6; compartment_*k5*s5;
    v7: at_ -> ; compartment_*k7*at_;
    v8: s3 -> na; compartment_*k8*s3*(ntot - na);
    v9: s6o -> ; compartment_*k9*s6o;
    v10: s6 -> 0.1 s6o; compartment_*k10*(s6 - s6o);
    v6: s6 -> na; compartment_*k6*s6*(ntot - na);
    v0:  -> s1; compartment_*k0;

```python
ctlsb = ctl.ControlSBML(MODEL)
ctlsb.plotTrueModel(figsize=(5,5))
```

# Transfer Function Analysis

```python
makeHRY(MODEL, "at", "s5", time=0, kp=1, ki=0, kd=0)
```

$$\frac{-56 s^{15} - 2.351 \times 10^{5} s^{14} - 3.829 \times 10^{8} s^{13} - 3.169 \times 10^{11} s^{12} - 1.514 \times 10^{14} s^{11} - 4.501 \times 10^{16} s^{10} - 8.753 \times 10^{18} s^9 - 1.151 \times 10^{21} s^8 - 1.044 \times 10^{23} s^7 - 6.595 \times 10^{24} s^6 - 2.874 \times 10^{26} s^5 - 8.299 \times 10^{27} s^4 - 1.434 \times 10^{29} s^3 - 1.187 \times 10^{30} s^2 - 3.615 \times 10^{30} s - 7.283 \times 10^{29}}{s^{16} + 4282 s^{15} + 7.194 \times 10^{6} s^{14} + 6.255 \times 10^{9} s^{13} + 3.212 \times 10^{12} s^{12} + 1.052 \times 10^{15} s^{11} + 2.304 \times 10^{17} s^{10} + 3.47 \times 10^{19} s^9 + 3.636 \times 10^{21} s^8 + 2.634 \times 10^{23} s^7 + 1.286 \times 10^{25} s^6 + 4.012 \times 10^{26} s^5 + 7.294 \times 10^{27} s^4 + 6.957 \times 10^{28} s^3 + 4.803 \times 10^{29} s^2 + 3.238 \times 10^{30} s + 1.069 \times 10^{31}}$$

# Testbed

## Piece Parts of Testbed for Closed Loop System

```python
def callback_print(call_name, time, x_vec, u_vec, dct, outputs):
    """
    Callback function used by IOSystemFactory that prints the inputs.

    Parameters
    ----------
    name: name of the component
    time: float
    x_vec: list-float
    u_vec: list-float
    dct; dict
        additional constant parameters
    outputs: list-float
    is_output: bool

    Returns
    -------
    str
    """
    line = call_name
    line += "  %2.6f" % time
    line += "  %s" % str(x_vec)
    line += "  %s" % str(u_vec)
    line += "  %s" % str(outputs)
    line += "  %s" % str(dct)
    print(line)

```

```python
# Elements of the system

factory = ctl.IOSystemFactory()
kp = 0.01
ki = 0
kd = 0
input_name = "at"
output_name = "s5"
# Create the elements of the feedback loop
ctlsb = ctl.ControlSBML(MODEL, input_names=[input_name], output_names=[output_name])
system = ctlsb.makeNonlinearIOSystem("system")
controller = factory.makePIDController("controller", kp=kp, ki=ki, kd=kd)
sum_R_S = factory.makeAdder("sum_R_S")
```

```python
# Create the closed loop system
system_in = "system.%s" % input_name
system_out = "system.%s" % output_name
closed_loop = control.interconnect(
  [sum_R_S, system, controller ],
  connections=[
    ['controller.in', 'sum_R_S.out'],    # e(t)
    [system_in,   'controller.out'],   # u(t)
    ['sum_R_S.in1', "-" + system_out],
  ],
  inplist=["sum_R_S.in2"],
  outlist=["sum_R_S.in2", system_in, system_out],
)
```

```python
X0 = ctl.makeStateVector(closed_loop)
```

```python
times = ctl.makeSimulationTimes(0, 10, 10)
# Need the initial state
result = control.input_output_response(closed_loop, T=times, U=6, X0=X0)
plt.plot(result.t, result.outputs[0].flatten())
plt.plot(result.t, result.outputs[1].flatten())
plt.plot(result.t, result.outputs[2].flatten())
#plt.plot(result.t, result.outputs[3].flatten())
plt.ylim([0, 10])
legends = ["input", system_in, "output"]
plt.legend(legends)
```

## Create Evaluation Environment

# Evaluation Environment

We want to repeatedly evaluate different controller parameters.
This is cumbersome to do if we have to rerun multiple cells and reset various parameters.
It's much more efficient to create an evaluation function.

```python
def runTestbed(model=MODEL, input_name="S2", output_name="S3", kp=0, ki=0, kd=0, noise_amp=0, disturbance_amp=0):
    """
    Run the testbed and plot the results.

    Parameters
    ----------
    model: str
        System under control
    input_name: str
    output_name: str
    kp: float
    ki: float
    kd: float
    noise_amp: float
    disturbance_amp: float

    Results
    -------
    control.InterconnectedSystem
    """
```


# Design of Digital Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Design of Non-Recursive Filters using the Frequency Sampling Method

For some applications, the desired frequency response is not given at all frequencies but rather at a number of discrete frequencies. For this case, the frequency sampling method provides a solution for the design of non-recursive filters.

### The Frequency Sampling Method

Let's assume that the desired transfer function $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is specified at a set of $N$ equally spaced frequencies $\Omega_\mu = \frac{2 \pi}{N} \mu$

\begin{equation}
H_\text{d}[\mu] = H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\frac{2 \pi}{N} \mu})
\end{equation}

for $\mu = 0, 1, \dots, N-1$. The coefficients of a non-recursive filter with finite impulse response (FIR) can then be computed by inverse discrete Fourier transformation (DFT) of $H_\text{d}[\mu]$

\begin{equation}
h[k] = \text{DFT}_N^{-1} \{ H_\text{d}[\mu] \} = \frac{1}{N} \sum_{\mu = 0}^{N-1} H_\text{d}[\mu] \; \mathrm{e}^{\,\mathrm{j}\,\frac{2 \pi}{N} \mu\,k}
\end{equation}

for $k = 0,1, \dots, N-1$.

In order to investigate the properties of the designed filter, its transfer function $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is computed. It is given by discrete-time Fourier transformation (DTFT) of its impulse response $h[k]$

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k = 0}^{N-1} h[k] \; \mathrm{e}^{\,-\mathrm{j}\,\Omega\,k} =
\sum_{\mu = 0}^{N-1} H_\text{d}[\mu] \cdot \frac{1}{N} \sum_{k = 0}^{N-1} \mathrm{e}^{\,-\mathrm{j}\,k\,(\Omega - \frac{2 \pi}{N}\,\mu)}
\end{equation}

When comparing this result with the [interpolation of a DFT](../spectral_analysis_deterministic_signals/zero_padding.ipynb#Interpolation-of-the-Discrete-Fourier-Transformation), it can be concluded that $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is yielded by interpolation of the desired transfer function $H_\text{d}[\mu]$

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{\mu=0}^{N-1} H_\text{d}[\mu] \cdot \mathrm{e}^{-\,\mathrm{j}\, \frac{( \Omega - \frac{2 \pi}{N} \mu ) (N-1)}{2}} \cdot \text{psinc}_N ( \Omega - \frac{2 \pi}{N} \mu)
\end{equation}

where $\text{psinc}_N(\cdot)$ denotes the $N$-th order periodic sinc function.

Both the transfer function of the filter $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the desired transfer function $H_\text{d}[\mu]$ are equal at the specified frequencies $\Omega_\mu = \frac{2 \pi}{N} \mu$. Values in between adjacent $\Omega_\mu$ are interpolated by the periodic sinc function. This is illustrated in the following.

#### Example: Approximation of an ideal low-pass

The design of an ideal low-pass filter using the frequency sampling method is considered. For $|\Omega| < \pi$ the transfer function of the ideal low-pass is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \begin{cases}
1 & \text{for } |\Omega| \leq \Omega_\text{c} \\
0 & \text{otherwise}
\end{cases}
\end{equation}

where $\Omega_\text{c}$ denotes its corner frequency. The desired transfer function $H_\text{d}[\mu]$ for the frequency sampling method is derived by sampling $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. Note that for sampling on the unit circle with $0 \leq \Omega < 2 \pi$, the periodicity $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = H_\text{d}(\mathrm{e}^{\,\mathrm{j}(\Omega + n 2 \pi)})$ for $n \in \mathbb{Z}$ has to be considered.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 32  # length of filter
Omc = np.pi/3  # corner frequency of low-pass

# specify desired frequency response
Ommu = 2*np.pi/N*np.arange(N)
Hd = np.zeros(N)
Hd[Ommu <= Omc] = 1
Hd[Ommu >= (2*np.pi-Omc)] = 1

# compute impulse response of filter
h = np.fft.ifft(Hd)
h = np.real(h)  # due to round-off errors
# compute frequency response of filter
Om, H = sig.freqz(h, worN=8192)

# plot impulse response
plt.figure(figsize = (10,3))
plt.stem(h)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot transfer functions
plt.figure(figsize = (10,3))
plt.plot(Om, np.abs(H), 'b-', label=r'designed $|H(e^{j \Omega})|$')
plt.stem(Ommu, np.abs(Hd), 'g', label=r'desired $|H_d[\mu]|$')
plt.plot([0, Omc, Omc], [1, 1, 0], 'r--')
plt.title('Magnitude response of desired/designed filter')
plt.xlabel(r'$\Omega$')
plt.legend()
plt.axis([0, np.pi, -0.05, 1.5])
# plot phase
plt.figure(figsize = (10,3))
plt.plot(Om, np.unwrap(np.angle(H)))
plt.title('Phase of designed filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi(\Omega)$')
plt.grid()
```

**Exercises**

* Does the resulting filter approximate the desired magnitude and phase response well?
* Increase the length `N` of the filter. Does the attenuation in the stop-band improve?

Solution: The desired magnitude and phase response is an ideal low-pass with zero-phase. The magnitude response of the designed filter approximates this only very coarsely and shows major ringing artifacts. The phase of the designed filter is also not zero for all frequencies. Increasing the length `N` of the filter does not improve the attenuation in the stop-band, it only changes the frequencies of the local minima of the ripples. The reason for the poor performance of the designed filter is the zero-phase of the desired transfer function which cannot be realized by a causal non-recursive system. This was [already discussed for the window method](../filter_design/window_method.ipynb#Zero-Phase-Filters). The frequency sampling method suffers additionally from time-domain aliasing due to the periodicity of the DFT. Again a linear-phase design is better in such situations.

### Design of Linear-Phase Filters

The design of non-recursive FIR filters with a linear phase is often desired due to their constant group delay. As for the [window method](../filter_design/window_method.ipynb), the design of a digital filter with a generalized linear phase is considered in the following. For $|\Omega| < \pi$ its transfer function is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot \mathrm{e}^{\,-\mathrm{j} \alpha \Omega + \mathrm{j} \beta}
\end{equation}

where $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \in \mathbb{R}$ denotes the amplitude of the filter, $-\alpha\,\Omega$ its linear phase and $\beta$ a constant phase offset. The impulse response $h[k]$ of a linear-phase filter shows specific symmetries which have already been discussed for the [design of linear-phase filters using the window method](../filter_design/window_method.ipynb#Causal-Linear-Phase-Filters). For the resulting four types of linear-phase FIR filters, the properties of $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the values of $\alpha$ and $\beta$ have to be chosen accordingly for the formulation of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $H_d[\mu]$, respectively. This is illustrated in the following for the design of a low-pass filter.

#### Example: Linear-phase approximation of an ideal low-pass

We aim at the approximation of an ideal low-pass as a linear-phase non-recursive FIR filter. For the sake of comparison with a similar [example for the window method](../filter_design/window_method.ipynb#Example:-Causal-linear-phase-approximation-of-ideal-low-pass), we choose a type 1 filter with odd filter length $N$, $\alpha = \frac{N-1}{2}$ and $\beta = 0$. The desired frequency response $H_\text{d}[\mu]$ is given by sampling

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathrm{e}^{\,-\mathrm{j} \frac{N-1}{2} \Omega} \cdot \begin{cases}
1 & \text{for } |\Omega| \leq \Omega_\text{c} \\
0 & \text{otherwise}
\end{cases}
\end{equation}

which is defined for $|\Omega| < \pi$. Note that for sampling on the unit circle with $0 \leq \Omega < 2 \pi$, the periodicity $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = H_\text{d}(\mathrm{e}^{\,\mathrm{j}(\Omega+ n 2 \pi)})$ for $n \in \mathbb{Z}$ has to be considered.

```python
N = 33  # length of filter
Omc = np.pi/3  # corner frequency of low-pass

# specify desired frequency response
Ommu = 2*np.pi/N*np.arange(N)
Hd = np.zeros(N)
Hd[Ommu <= Omc] = 1
Hd[Ommu >= (2*np.pi-Omc)] = 1
Hd = Hd * np.exp(-1j*Ommu*(N-1)/2)

# compute impulse response of filter
h = np.fft.ifft(Hd)
h = np.real(h)  # due to round-off errors
# compute frequency response of filter
Om, H = sig.freqz(h, worN=8192)

# plot impulse response
plt.figure(figsize = (10,3))
plt.stem(h)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot frequency response
plt.figure(figsize = (10,3))
plt.plot(Om, np.abs(H), 'b-', label=r'designed $|H(e^{j \Omega})|$')
plt.stem(Ommu, np.abs(Hd), 'g', label=r'desired $|H_d[\mu]|$')
plt.plot([0, Omc, Omc], [1, 1, 0], 'r--')
plt.title('Magnitude response of desired/designed filter')
plt.xlabel(r'$\Omega$')
plt.legend()
plt.axis([0, np.pi, -0.05, 1.5])
# plot phase
plt.figure(figsize = (10,3))
plt.plot(Om, np.unwrap(np.angle(H)))
plt.title('Phase of designed filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi(\Omega)$')
plt.grid()
```

**Exercises**

* Does the designed filter have the desired linear phase?
* Increase the length `N` of the filter. What is different to the previous example?
* How could the method be modified to change the properties of the frequency response?

Solution: The designed filter has the desired linear phase in the pass-band, here below $\Omega = \frac{\pi}{3}$. Increasing the length `N` of the filter results in a better approximation of the desired ideal low-pass, especially in a higher attenuation in the stop-band.

### Comparison to Window Method

For a comparison of the frequency sampling to the [window method](../filter_design/window_method.ipynb) it is assumed that the desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is given. For both methods, the coefficients $h[k]$ of an FIR approximation are computed as follows

1. Window Method
\begin{align}
h_\text{d}[k] &= \mathcal{F}_{*}^{-1} \{ H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \} \\
h[k] &= h_\text{d}[k] \cdot w[k]
\end{align}

2. Frequency Sampling Method
\begin{align}
H_\text{d}[\mu] &= H_\text{d}(\mathrm{e}^{\,\mathrm{j} \frac{2 \pi}{N} \mu}) \\
h[k] &= \text{DFT}_N^{-1} \{ H_\text{d}[\mu] \}
\end{align}

For finite lengths $N$, the difference between both methods is related to the periodicity of the DFT. For a desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ which does not result in a FIR $h_\text{d}[k]$ of length $N$, the inverse DFT in the frequency sampling method will suffer from time-domain aliasing. In the general case, filter coefficients computed by the window and frequency sampling method will hence differ.

However, for a rectangular window $w[k]$ and $N \to \infty$ both methods will become equivalent. This reasoning motivates an oversampled frequency sampling method, where $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is sampled at $M \gg N$ points in order to derive an approximation of $h_\text{d}[k]$ which is then windowed to the target length $N$. The method is beneficial in cases where a closed-form inverse DTFT of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, as required for the window method, cannot be found.

#### Example: Oversampled frequency sampling method

We consider the design of a linear-phase approximation of an ideal low-pass filter using the oversampled frequency sampling method. For the sake of comparison, the parameters have been chosen in accordance to a [similar example using the window method](../filter_design/window_method.ipynb#Example:-Causal-linear-phase-approximation-of-ideal-low-pass). Using $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ from the previous example in this section, the filter is computed by

1. (Over)-Sampling the desired response at $M$ frequencies
\begin{equation}
H_\text{d}[\mu] = H_\text{d}(\mathrm{e}^{\,\mathrm{j} \frac{2 \pi}{M} \mu})
\end{equation}

2. Inverse DFT of length $M$
\begin{equation}
h[k] = \text{DFT}_M^{-1} \{ H_\text{d}[\mu] \}
\end{equation}

3. Windowing to desired length $N$
\begin{equation}
h[k] = h_\text{d}[k] \cdot w[k]
\end{equation}

```python
N = 33  # length of filter
M = 8192  # number of frequency samples
Omc = np.pi/2  # corner frequency of low-pass

# specify desired frequency response
Ommu = 2*np.pi/M*np.arange(M)
Hd = np.zeros(M)
Hd[Ommu <= Omc] = 1
Hd[Ommu >= (2*np.pi-Omc)] = 1
Hd = Hd * np.exp(-1j*Ommu*(N-1)/2)

# compute impulse response of filter
h = np.fft.ifft(Hd)
h = np.real(h)  # due to round-off errors
h = h[0:N]  # rectangular window
# compute frequency response of filter
Om, H = sig.freqz(h, worN=8192)

# plot impulse response
plt.figure(figsize = (10,3))
plt.stem(h)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot frequency response
plt.figure(figsize = (10,3))
plt.plot(Om, 20 * np.log10(abs(H)), label='rectangular window')
plt.plot([0, Omc, Omc], [0, 0, -100], 'r--')
plt.title('Magnitude response of designed filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H(e^{j \Omega})|$ in dB')
plt.axis([0, np.pi, -100, 3])
plt.legend()
plt.grid()
# plot phase
plt.figure(figsize = (10,3))
plt.plot(Om, np.unwrap(np.angle(H)))
plt.title('Phase of designed filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi(\Omega)$')
plt.grid()
```

**Exercises**

* Compare the designed filter and its properties to the [same design using the window method](../filter_design/window_method.ipynb#Example:-Causal-linear-phase-approximation-of-ideal-low-pass)
* Change the number of samples `M` used for sampling the desired response. What changes if you increase/decrease `M`?

Solution: Comparison of above results to the window method (with a rectangular window) reveals that both give a similar performance in terms of achievable stop-band attenuation. Increasing `M` in above example does not result in obvious changes since the initial choice constitutes already a high amount of oversampling. However, decreasing `M` till it is the range of the filter length `N` results in a smooth degradation of the performance towards the original frequency sampling method.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


# sympy Laplace Transforms for solving linear ODEs
> I have found that the Laplace transform  utility in sympy doesn't do what we need for solving linear ODEs.  We've made some needed improvements and included the code in the MATH280 package

- toc: true

In our workflow, we assume sympy as the default, and load the custom-made content from [MATH280 Package](https://github.com/ejbarth/MATH280).

```python
from sympy import *
import MATH280

x = Function("x")
t,s = symbols("t s",real=True,  nonnegative=true )
```

    MATH280

A famously intense pen-and-paper technique from elementary ODEs class:  Laplace Transforms.
Sympy has Laplace capabilities built-in, but with maddening shortcomings.

Let's work with a 2nd order, linear, constant-coefficient, nonhomogeous IVP:
$$ \ddot{x}+3\dot{x}+4x = \sin(2t), \;\;\;\;\;\; x(0)=1, \dot{x}(0)=0$$

```python
ode = Eq(x(t).diff(t,2)+3*x(t).diff(t)+4*x(t) , sin(2*t) )
ode
```

$\displaystyle 4 x{\left(t \right)} + 3 \frac{d}{d t} x{\left(t \right)} + \frac{d^{2}}{d t^{2}} x{\left(t \right)} = \sin{\left(2 t \right)}$

## laplace_transform() in sympy

First let's see how the built-in sympy `laplace_transform()` handles that equation:

```python
laplace_transform(ode,t,s)
```

Look at that!  **'Equality' object has no attribute**.  `laplace_transform()` would prefer that we assume the expression we enter is equal to 0:

```python
Lx=laplace_transform(x(t).diff(t,2)+3*x(t).diff(t)+4*x(t) - sin(2*t),t,s)
Lx
```

    (4*LaplaceTransform(x(t), t, s) + 3*LaplaceTransform(Derivative(x(t), t), t, s) + LaplaceTransform(Derivative(x(t), (t, 2)), t, s) - 2/(s**2 + 4),
     0,
     True)

We see the output above is a tuple that includes some conditional statements at the end.  To hide that and see just the transform itself, use the option `noconds=True`

```python
Lx=laplace_transform(x(t).diff(t,2)+3*x(t).diff(t)+4*x(t) - sin(2*t),t,s,noconds=True)
Lx
```

$\displaystyle 4 \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) + 3 \mathcal{L}_{t}\left[\frac{d}{d t} x{\left(t \right)}\right]\left(s\right) + \mathcal{L}_{t}\left[\frac{d^{2}}{d t^{2}} x{\left(t \right)}\right]\left(s\right) - \frac{2}{s^{2} + 4}$

### Laplace Transform and Derivatives

Notice in that output another hassle:  `laplace_transform()` ignores the single most useful property of the Laplace Transform:
$$ {\cal L}\{x'(t)\} = s{\cal L}\{x(t)\} - x(0), \mbox{ and } {\cal L}\{x''(t)\} = s^2{\cal L}\{x(t)\} - sx(0)- x'(0).$$

## laplace() in MATH280

We've reworked that and included an alternative `laplace{}` in the [MATH280 Module](https://github.com/ejbarth/MATH280).

### Solving an equation with Laplace Transforms in four steps:

#### 1. take the transform of everything

```python
L = MATH280.laplace(ode,t,s)
L
```

$\displaystyle s^{2} \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) + 3 s \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) - s x{\left(0 \right)} + 4 \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) - 3 x{\left(0 \right)} - \left. \frac{d}{d t} x{\left(t \right)} \right|_{\substack{ t=0 }} = \frac{2}{s^{2} + 4}$

#### 2. plug in the initial conditions

The second step in solving the equation is to plug in the initial conditions:

```python
L0=L.subs(x(0),1).subs(Subs(Derivative(x(t), t), t, 0),0)
L0
```

$\displaystyle s^{2} \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) + 3 s \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) - s + 4 \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) - 3 = \frac{2}{s^{2} + 4}$

#### 3. solve for the lapace transform of the solution function

The third step is to solve the resulting equation for the symbol ${\cal L}\{x(t)\}$

```python
Lx=solve(L0,LaplaceTransform(x(t),t,s))
Lx
```

    [(s**3 + 3*s**2 + 4*s + 14)/(s**4 + 3*s**3 + 8*s**2 + 12*s + 16)]

#### 4. look up the laplace transform to determine the solution

The fourth and final step is to "look up" that complicated expression in the variable $s$ to determine the function of $t$ with that Laplace transform. The built-in sympy funtion `inverse_laplace_transform()` works fine for that, with the slight annoyance that it doesn't align perfectly with the list format of output from `solve()`.  So I've included in  MATH280 a little wrapper called `laplaceInv()` that extracts the zeroth entry from that list. **I've noticed that this step can run really slowly.**

```python
sol = MATH280.laplaceInv(Lx,s,t)
sol
```

$\displaystyle - \frac{\cos{\left(2 t \right)} \theta\left(t\right)}{6} + \frac{\sqrt{7} e^{- \frac{3 t}{2}} \sin{\left(\frac{\sqrt{7} t}{2} \right)} \theta\left(t\right)}{2} + \frac{7 e^{- \frac{3 t}{2}} \cos{\left(\frac{\sqrt{7} t}{2} \right)} \theta\left(t\right)}{6}$

Something to notice is that every term as a $\theta(t)$.  That's the unit step function:  $\theta(t)=1$ if $t\gt 0$ and otherwise zero.  Seems to be enforcing the assumption that our solution only makes sense for nonnegative time $t$.

We can plot the solution with sympy plot:

```python
plot(sol,(t,0,20))
```

And just in case there was any doubt, we notice that `dsolve()` produces the same solution.

```python
dsolve(ode,x(t),ics={x(0):1, x(t).diff(t).subs(t,0):0})
```

$\displaystyle x{\left(t \right)} = \left(\frac{\sqrt{7} \sin{\left(\frac{\sqrt{7} t}{2} \right)}}{2} + \frac{7 \cos{\left(\frac{\sqrt{7} t}{2} \right)}}{6}\right) e^{- \frac{3 t}{2}} - \frac{\cos{\left(2 t \right)}}{6}$

## Discontinuous Forcing Functions

We haven't yet explored what I think is *the* reason to consider Laplace Transforms in the first place:  discontinuous forcing functions.

### A first-order model with a square wave switch function

Suppose we model a capacitor with an external voltage source that switches on at $t=2$ and off at $t=5$.
The equation, in some idealized units, could be
$$\dot{x} - x = \theta(t-2) - \theta(t-5), \;\;\;\;\;  x(0)=0 $$
where $\theta(t-a)$ is the Heaviside unit step function that turns on at $t=a$.

```python
capmodel = Eq(x(t).diff(t)+x(t), Heaviside(t-2)-Heaviside(t-5))
capmodel
```

$\displaystyle x{\left(t \right)} + \frac{d}{d t} x{\left(t \right)} = - \theta\left(t - 5\right) + \theta\left(t - 2\right)$

We'll complete the four steps as in the previous example:

```python
L=MATH280.laplace(capmodel,t,s)
L
```

$\displaystyle s \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) + \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) - x{\left(0 \right)} = \frac{e^{- 2 s}}{s} - \frac{e^{- 5 s}}{s}$

```python
L0=L.subs(x(0),0)
L0
```

$\displaystyle s \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) + \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) = \frac{e^{- 2 s}}{s} - \frac{e^{- 5 s}}{s}$

```python
Ls=solve(L0,LaplaceTransform(x(t),t,s))
Ls
```

    [(exp(3*s) - 1)*exp(-5*s)/(s*(s + 1))]

```python
capsol=MATH280.laplaceInv(Ls,s,t)
capsol
```

$\displaystyle - \theta\left(t - 5\right) + \theta\left(t - 2\right) + e^{5} e^{- t} \theta\left(t - 5\right) - e^{2} e^{- t} \theta\left(t - 2\right)$

The functional form of the solution looks a little opaque, so lets make a plot of the solution together with the right hand side function:

```python
plot(capsol,
     Heaviside(t-2)-Heaviside(t-5),
    (t,0,15),)
```

### A second-order example with a delta function

Here's an example from the undergraduate ODE textbook by Blanchard, Devaney and Hall:
$$\ddot{x} +2\dot{x}+3x= \delta(t-1)+3\delta(t-4), \;\;\;\;\; x(0)=0, \dot{x}(0)=0$$

```python
ode2 = Eq(x(t).diff(t,2)+2*x(t).diff(t)+3*x(t),DiracDelta(t-1)+3*DiracDelta(t-4))
ode2
```

$\displaystyle 3 x{\left(t \right)} + 2 \frac{d}{d t} x{\left(t \right)} + \frac{d^{2}}{d t^{2}} x{\left(t \right)} = 3 \delta\left(t - 4\right) + \delta\left(t - 1\right)$

```python
L=MATH280.laplace(ode2,t,s)
L
```

$\displaystyle s^{2} \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) + 2 s \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) - s x{\left(0 \right)} + 3 \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) - 2 x{\left(0 \right)} - \left. \frac{d}{d t} x{\left(t \right)} \right|_{\substack{ t=0 }} = e^{- s} + 3 e^{- 4 s}$

```python
Ls=L.subs(x(0),0).subs(Subs(Derivative(x(t), t), t, 0),0)
Ls
```

$\displaystyle s^{2} \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) + 2 s \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) + 3 \mathcal{L}_{t}\left[x{\left(t \right)}\right]\left(s\right) = e^{- s} + 3 e^{- 4 s}$

```python
Lx=solve(Ls,LaplaceTransform(x(t),t,s))
Lx
```

    [(exp(3*s) + 3)*exp(-4*s)/(s**2 + 2*s + 3)]

```python
sol2 = MATH280.laplaceInv(Lx,s,t)
sol2
```

$\displaystyle \frac{3 \sqrt{2} e^{4} e^{- t} \sin{\left(\sqrt{2} t \right)} \cos{\left(4 \sqrt{2} \right)} \theta\left(t - 4\right)}{2} + \frac{\sqrt{2} e e^{- t} \sin{\left(\sqrt{2} t - \sqrt{2} \right)} \theta\left(t - 1\right)}{2} - \frac{3 \sqrt{2} e^{4} e^{- t} \sin{\left(4 \sqrt{2} \right)} \cos{\left(\sqrt{2} t \right)} \theta\left(t - 4\right)}{2}$

```python
plot(sol2,(t,0,20))
```

We can see in the plot above how the impulse at $t=1$ and $t=4$ re-energizes the system.

```python

```


# The Implicit Crank-Nicolson Difference Equation for the Heat Equation

#### John S Butler john.s.butler@tudublin.ie   [Course Notes](https://johnsbutler.netlify.com/files/Teaching/Numerical_Analysis_for_Differential_Equations.pdf)    [Github](https://github.com/john-s-butler-dit/Numerical-Analysis-Python)
## Overview
This notebook will illustrate the Crank-Nicolson Difference method for the Heat Equation.
## The Heat Equation
The Heat Equation is the first order in time ($t$) and second order in space ($x$) Partial Differential Equation:
$$  \frac{\partial u}{\partial t} = \frac{\partial^2 u}{\partial x^2},$$
The equation describes heat transfer on a domain
$$ \Omega = \{ t \geq, 0\leq x \leq 1\}. $$
with an initial condition at time $t=0$ for all $x$ and boundary condition on the left ($x=0$) and right side

This notebook will illustrate the Crank-Nicolson Difference method for the Heat Equation with the __initial conditions__
$$ u(x,0)=2x, \ \ 0 \leq x \leq \frac{1}{2}, $$
$$ u(x,0)=2(1-x), \ \ \frac{1}{2}  \leq x \leq 1, $$
and __boundary condition__
$$ u(0,t)=0,  u(1,t)=0. $$
## Crank-Nicolson Difference method

The implicit Crank-Nicolson difference equation of the Heat Equation
is derived by discretising the
$$  \frac{\partial u_{ij+\frac{1}{2}}}{\partial t} = \frac{\partial^2 u_{ij+\frac{1}{2}}}{\partial x^2},$$
around $(x_i,t_{j+\frac{1}{2}})$ giving the difference equation
$$
\frac{w_{ij+1}-w_{ij}}{k}=\frac{1}{2}\left(\frac{w_{i+1j+1}-2w_{ij+1}+w_{i-1j+1}}{h^2}+\frac{w_{i+1j}-2w_{ij}+w_{i-1j}}{h^2}\right).
$$
Rearranging give the difference equation
$$
-rw_{i-1j+1}+(2+2r)w_{ij+1}-rw_{i+1j+1}=rw_{i-1j}+(2-2r)w_{ij}+rw_{i+1j}
$$
for $i=1,...9$ where $r=\frac{k}{h^2}$.

```python
# LIBRARY
# vector manipulation
import numpy as np
# math functions
import math

# THIS IS FOR PLOTTING
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import warnings
warnings.filterwarnings("ignore")

```

## Discete Grid
The region $\Omega$ is discretised into a uniform mesh $\Omega_h$. In the space $x$ direction into $N$ steps giving a stepsize of
$$ h=\frac{1-0}{N},$$
resulting in
$$x[i]=0+ih, \ \ \  i=0,1,...,N,$$
and into $N_t$ steps in the time $t$ direction giving a stepsize of
$$ k=\frac{1-0}{N_t}$$
resulting in
$$t[i]=0+ik, \ \ \ k=0,...,K.$$
The Figure below shows the discrete grid points for $N=10$ and $N_t=100$ , the red dots are the unknown values, the green dots are the known boundary conditions and the blue dots are the known initial conditions of the Heat Equation.

```python
N=10
Nt=100
h=1/N
k=1/Nt
r=k/(h*h)
time_steps=15
time=np.arange(0,(time_steps+.5)*k,k)
x=np.arange(0,1.0001,h)
X, Y = np.meshgrid(x, time)
fig = plt.figure()
plt.plot(X,Y,'ro');
plt.plot(x,0*x,'bo',label='Initial Condition');
plt.plot(np.ones(time_steps+1),time,'go',label='Boundary Condition');
plt.plot(x,0*x,'bo');
plt.plot(0*time,time,'go');
plt.xlim((-0.02,1.02))
plt.xlabel('x')
plt.ylabel('time (ms)')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.title(r'Discrete Grid $\Omega_h,$ h= %s, k=%s'%(h,k),fontsize=24,y=1.08)
plt.show();
```

## Discrete Initial and Boundary Conditions

The discrete initial conditions are
$$ w[i,0]=2x[i], \ \ 0 \leq x[i] \leq \frac{1}{2} $$
$$ w[i,0]=2(1-x[i]), \ \ \frac{1}{2}  \leq x[i] \leq 1 $$
and the discete boundary conditions are
$$ w[0,j]=0,  w[10,j]=0, $$
where $w[i,j]$ is the numerical approximation of $U(x[i],t[j])$.

The Figure below plots values of $w[i,0]$ for the inital (blue) and boundary (red) conditions for $t[0]=0.$

```python
w=np.zeros((N+1,time_steps+1))
b=np.zeros(N-1)
# Initial Condition
for i in range (1,N):
    w[i,0]=2*x[i]
    if x[i]>0.5:
        w[i,0]=2*(1-x[i])

# Boundary Condition
for k in range (0,time_steps):
    w[0,k]=0
    w[N,k]=0

fig = plt.figure(figsize=(8,4))
plt.plot(x,w[:,0],'o:',label='Initial Condition')
plt.plot(x[[0,N]],w[[0,N],0],'go',label='Boundary Condition t[0]=0')
#plt.plot(x[N],w[N,0],'go')
plt.xlim([-0.1,1.1])
plt.ylim([-0.1,1.1])
plt.title('Intitial and Boundary Condition',fontsize=24)
plt.xlabel('x')
plt.ylabel('w')
plt.legend(loc='best')
plt.show()
```

## The Implicit Crank-Nicolson Difference Equation
The implicit Crank-Nicolson difference equation of the Heat Equation
is derived by discretising the
$$  \frac{\partial u_{ij+\frac{1}{2}}}{\partial t} = \frac{\partial^2 u_{ij+\frac{1}{2}}}{\partial x^2},$$
around $(x_i,t_{j+\frac{1}{2}})$ giving the difference equation
$$
\frac{w_{ij+1}-w_{ij}}{k}=\frac{1}{2}\left(\frac{w_{i+1j+1}-2w_{ij+1}+w_{i-1j+1}}{h^2}+\frac{w_{i+1j}-2w_{ij}+w_{i-1j}}{h^2}\right)
$$
Rearranging the equation we get
$$
-rw_{i-1j+1}+(2+2r)w_{ij+1}-rw_{i+1j+1}=rw_{i-1j}+(2-2r)w_{ij}+rw_{i+1j}
$$
for $i=1,...9$ where $r=\frac{k}{h^2}$.

This gives the formula for the unknown term $w_{ij+1}$ at the $(ij+1)$ mesh points
in terms of $x[i]$ along the jth time row.

Hence we can calculate the unknown pivotal values of $w$ along the first row of $j=1$ in terms of the known boundary conditions.

This can be written in matrix form
$$ A\mathbf{w}_{j+1}=B\mathbf{w}_{j} +\mathbf{b}_{j}+\mathbf{b}_{j+1} $$
for which $A$ is a $9\times9$ matrix:
$$
\left(\begin{array}{cccc cccc}
2+2r&-r& 0&0&0 &0&0&0\\
-r&2+2r&-r&0&0&0 &0&0&0\\
0&-r&2+2r &-r&0&0& 0&0&0\\
0&0&-r&2+2r &-r&0&0& 0&0\\
0&0&0&-r&2+2r &-r&0&0& 0\\
0&0&0&0&-r&2+2r &-r&0&0\\
0&0&0&0&0&-r&2+2r &-r&0\\
0&0&0&0&0&0&-r&2+2r&-r\\
0&0&0&0&0&0&0&-r&2+2r\\
\end{array}\right)
\left(\begin{array}{c}
w_{1j+1}\\
w_{2j+1}\\
w_{3j+1}\\
w_{4j+1}\\
w_{5j+1}\\
w_{6j+1}\\
w_{7j+1}\\
w_{8j+1}\\
w_{9j+1}\\
\end{array}\right)=
\left(\begin{array}{cccc cccc}
2-2r&r& 0&0&0 &0&0&0\\
r&2-2r&r&0&0&0 &0&0&0\\
0&r&2-2r &r&0&0& 0&0&0\\
0&0&r&2-2r &r&0&0& 0&0\\
0&0&0&r&2-2r &r&0&0& 0\\
0&0&0&0&r&2-2r &r&0&0\\
0&0&0&0&0&r&2-2r &r&0\\
0&0&0&0&0&0&r&2-2r&r\\
0&0&0&0&0&0&0&r&2-2r\\
\end{array}\right)
\left(\begin{array}{c}
w_{1j}\\
w_{2j}\\
w_{3j}\\
w_{4j}\\
w_{5j}\\
w_{6j}\\
w_{7j}\\
w_{8j}\\
w_{9j}\\
\end{array}\right)+
\left(\begin{array}{c}
rw_{0j}\\
0\\
0\\
0\\
0\\
0\\
0\\
0\\
rw_{10j}\\
\end{array}\right)+
\left(\begin{array}{c}
rw_{0j+1}\\
0\\
0\\
0\\
0\\
0\\
0\\
0\\
rw_{10j+1}\\
\end{array}\right).
$$
It is assumed that the boundary values $w_{0j}$ and $w_{10j}$ are known for $j=1,2,...$, and $w_{i0}$ for $i=0,...,10$ is the initial condition.

The Figure below shows the values of the $9\times 9$  matrix in colour plot form for $r=\frac{k}{h^2}$.

```python
A=np.zeros((N-1,N-1))
B=np.zeros((N-1,N-1))
for i in range (0,N-1):
    A[i,i]=2+2*r
    B[i,i]=2-2*r

for i in range (0,N-2):
    A[i+1,i]=-r
    A[i,i+1]=-r
    B[i+1,i]=r
    B[i,i+1]=r

Ainv=np.linalg.inv(A)
fig = plt.figure(figsize=(12,4));
plt.subplot(121)
plt.imshow(A,interpolation='none');
plt.xticks(np.arange(N-1), np.arange(1,N-0.9,1));
plt.yticks(np.arange(N-1), np.arange(1,N-0.9,1));
clb=plt.colorbar();
clb.set_label('Matrix elements values');
clb.set_clim((-1,1));
plt.title('Matrix A r=%s'%(np.round(r,3)),fontsize=24)

plt.subplot(122)
plt.imshow(B,interpolation='none');
plt.xticks(np.arange(N-1), np.arange(1,N-0.9,1));
plt.yticks(np.arange(N-1), np.arange(1,N-0.9,1));
clb=plt.colorbar();
clb.set_label('Matrix elements values');
clb.set_clim((-1,1));
plt.title(r'Matrix $B$ r=%s'%(np.round(r,3)),fontsize=24)
fig.tight_layout()
plt.show();
```

## Results
To numerically approximate the solution at $t[1]$ the matrix equation becomes
$$ \mathbf{w}_{1}=A^{-1}(B\mathbf{w}_{0} +\mathbf{b}_{1}+\mathbf{b}_{0}) $$
where all the right hand side is known.
To approximate solution at time $t[2]$ we use the matrix equation
$$ \mathbf{w}_{2}=A^{-1}(B\mathbf{w}_{1}+\mathbf{b}_{2} +\mathbf{b}_{1}). $$
Each set of numerical solutions $w[i,j]$ for all $i$ at the previous time step is used to approximate the solution $w[i,j+1]$.
The left and right plot below show the numerical approximation $w[i,j]$ of the Heat Equation using the Crank-Nicolson method for $x[i]$ for $i=0,...,10$ and time steps $t[j]$ for $j=1,...,15$.

```python
fig = plt.figure(figsize=(12,6))
plt.subplot(121)
for j in range (0,time_steps+1):
    b[0]=r*w[0,j-1]+r*w[0,j]
    b[N-2]=r*w[N,j-1]+r*w[N,j]
    v=np.dot(B,w[1:(N),j-1])
    w[1:(N),j]=np.dot(Ainv,v+b)
    plt.plot(x,w[:,j],'o:',label='t[%s]=%s'%(j,time[j]))
plt.xlabel('x')
plt.ylabel('w')
#plt.legend(loc='bottom', bbox_to_anchor=(0.5, -0.1))
plt.legend(bbox_to_anchor=(-.4, 1), loc=2, borderaxespad=0.)

plt.subplot(122)
plt.imshow(w.transpose())
plt.xticks(np.arange(len(x)), x)
plt.yticks(np.arange(len(time)), time)
plt.xlabel('x')
plt.ylabel('time')
clb=plt.colorbar()
clb.set_label('Temperature (w)')
plt.suptitle('Numerical Solution of the  Heat Equation r=%s'%(np.round(r,3)),fontsize=24,y=1.08)
fig.tight_layout()
plt.show()
```

## Local Trunction Error
The local truncation error of the classical explicit difference approach to
\begin{equation}
\frac{\partial U}{\partial t} - \frac{\partial^2 U}{\partial x^2}=0,
\end{equation}
with

\begin{equation}
F_{ij+\frac{1}{2}}(w)=\frac{w_{ij+1}-w_{ij}}{k}-\frac{1}{2}\left(\frac{w_{i+1j}-2w_{ij}+w_{i-1j}}{h^2}
+\frac{w_{i+1j+1}-2w_{ij+1}+w_{i-1j+1}}{h^2}
\right)=0,\end{equation}
is
\begin{equation}
T_{ij+\frac{1}{2}}=F_{ij+\frac{1}{2}}(U)=\frac{U_{ij+1}-U_{ij}}{k}-\frac{1}{2}\left(\frac{U_{i+1j}-2U_{ij}+U_{i-1j}}{h^2}+\frac{U_{i+1j+1}-2U_{ij+1}+U_{i-1j+1}}{h^2}\right),
\end{equation}
simplify
\begin{equation}
U_{ij+\frac{1}{2}}=\frac{U_{ij+1}+U_{ij}}{2},
\end{equation}
substitute into equation
\begin{equation}
T_{ij+\frac{1}{2}}=F_{ij+\frac{1}{2}}(U)=\frac{U_{ij+1}-U_{ij}}{k}-\frac{U_{i+1j+\frac{1}{2}}-2U_{ij+\frac{1}{2}}+U_{i-1j+\frac{1}{2}}}{h^2},
\end{equation}
By Taylors expansions we have
\begin{eqnarray*}
U_{i+1j+\frac{1}{2}}&=&U((i+1)h,(j+\frac{1}{2})k)=U(x_i+h,t_j+\frac{1}{2}k)\\
&=&U_{ij+\frac{1}{2}}+h\left(\frac{\partial U}{\partial x} \right)_{ij+\frac{1}{2}}+\frac{h^2}{2}\left(\frac{\partial^2 U}{\partial x^2} \right)_{ij+\frac{1}{2}}+\frac{h^3}{6}\left(\frac{\partial^3 U}{\partial x^3} \right)_{ij+\frac{1}{2}} +...\\
U_{i-1j+\frac{1}{2}}&=&U((i-1)h,(j+\frac{1}{2})k)=U(x_i-h,t_j+\frac{1}{2}k)\\
&=&U_{ij+\frac{1}{2}}-h\left(\frac{\partial U}{\partial x} \right)_{ij+\frac{1}{2}}+\frac{h^2}{2}\left(\frac{\partial^2 U}{\partial x^2} \right)_{ij+\frac{1}{2}}-\frac{h^3}{6}\left(\frac{\partial^3 U}{\partial x^3} \right)_{ij+\frac{1}{2}} +...\\
U_{ij+1}&=&U(ih,(j+1)k)=U(x_i,t_j+k)\\
&=&U_{ij+\frac{1}{2}}+\frac{k}{2}\left(\frac{\partial U}{\partial t} \right)_{ij+\frac{1}{2}}+\frac{(\frac{k}{2})^2}{2}\left(\frac{\partial^2 U}{\partial t^2} \right)_{ij+\frac{1}{2}}+\frac{(\frac{k}{2})^3}{6}\left(\frac{\partial^3 U}{\partial t^3} \right)_{ij+\frac{1}{2}} +...\\
U_{ij-1}&=&U(ih,(j-1)k)=U(x_i,t_j-k)\\
&=&U_{ij+\frac{1}{2}}-\frac{k}{2}\left(\frac{\partial U}{\partial t} \right)_{ij+\frac{1}{2}}+\frac{(\frac{k}{2})^2}{2}\left(\frac{\partial^2 U}{\partial t^2} \right)_{ij+\frac{1}{2}}-\frac{(\frac{k}{2})^3}{6}\left(\frac{\partial^3 U}{\partial t^3} \right)_{ij+\frac{1}{2}} +...
\end{eqnarray*}

substitution into the expression for $T_{ij+\frac{1}{2}}$ then gives

\begin{eqnarray*}
T_{ij+\frac{1}{2}}&=&\left(\frac{\partial U}{\partial t} - \frac{\partial^2 U}{\partial x^2} \right)_{ij+\frac{1}{2}}
-\frac{h^2}{12}\left(\frac{\partial^4 U}{\partial x^4} \right)_{ij+\frac{1}{2}}\\
& &	+\frac{k^2}{24}\left(\frac{\partial^3 U}{\partial t^3} \right)_{ij+\frac{1}{2}}
-\frac{h^4}{360}\left(\frac{\partial^6 U}{\partial x^6} \right)_{ij+\frac{1}{2}}+ ...
\end{eqnarray*}
But $U$ is the solution to the differential equation so
\begin{equation} \left(\frac{\partial U}{\partial t} - \frac{\partial^2 U}{\partial x^2} \right)_{ij+\frac{1}{2}}=0,\end{equation}

the principal part of the local truncation error is

\begin{equation}
\frac{k^2}{24}\left(\frac{\partial^3 U}{\partial t^3} \right)_{ij+\frac{1}{2}}-\frac{h^2}{12}\left(\frac{\partial^4 U}{\partial x^4} \right)_{ij+\frac{1}{2}}.
\end{equation}

Hence the truncation error is
\begin{equation}
T_{ij}=O(k^2)+O(h^2).
\end{equation}

## Stability Analyis

To investigating the stability of the fully implicit CN difference method of the Heat Equation, we will use the von Neumann method.
The FTCS difference equation is:
$$\frac{1}{k}(w_{pq+1}-w_{pq})=\frac{1}{2}\left(\frac{1}{h_x^2}(w_{p-1q+1}-2w_{pq+1}+w_{p+1q+1})+\frac{1}{h_x^2}(w_{p-1q}-2w_{pq}+w_{p+1q})\right),$$
approximating
$$\frac{\partial U}{\partial t}=\frac{\partial^2 U}{\partial x^2}$$

at $(ph,(q+\frac{1}{2})k)$. Substituting $w_{pq}=e^{i\beta x}\xi^{q}$ into the difference equation gives:
$$e^{i\beta ph}\xi^{q+1}-e^{i\beta ph}\xi^{q}=\frac{1}{2}\left(r\{e^{i\beta (p-1)h}\xi^{q+1}-2e^{i\beta ph}\xi^{q+1}+e^{i\beta (p+1)h}\xi^{q+1}+e^{i\beta (p-1)h}\xi^{q}-2e^{i\beta ph}\xi^{q}+e^{i\beta (p+1)h}\xi^{q} \right) \}
$$

where $r=\frac{k}{h_x^2}$. Divide across by $e^{i\beta (p)h}\xi^{q}$ leads to

$$ \xi-1=r \xi (e^{i\beta (-1)h} -2+e^{i\beta h})+r (e^{i\beta (-1)h} -2+e^{i\beta h}), $$

$$\xi-\xi r (2\cos(\beta h)-2)= 1+r (2\cos(\beta h)-2),$$

$$\xi(1+4r\sin^2(\beta\frac{h}{2})) =1-4r\sin^2(\beta\frac{h}{2})$$
Hence
$$\xi=\frac{1-4r\sin^2(\beta\frac{h}{2})}{(1+4r\sin^2(\beta\frac{h}{2}))} \leq 1$$
therefore the equation is unconditionally stable as $0 < \xi \leq 1$ for all $r$ and all $\beta$ .

## References
[1] G D Smith Numerical Solution of Partial Differential Equations: Finite Difference Method Oxford 1992

[2] Butler, J. (2019). John S Butler Numerical Methods for Differential Equations. [online] Maths.dit.ie. Available at: http://www.maths.dit.ie/~johnbutler/Teaching_NumericalMethods.html [Accessed 14 Mar. 2019].

[3] Wikipedia contributors. (2019, February 22). Heat equation. In Wikipedia, The Free Encyclopedia. Available at: https://en.wikipedia.org/w/index.php?title=Heat_equation&oldid=884580138 [Accessed 14 Mar. 2019].

Chapter 08 - Heat Equations/.ipynb_checkpoints/803_Heat Equation-Crank_Nicolson-checkpoint.ipynb


## 1 Logistic regression

在这部分的练习中，你将建立一个逻辑回归模型来预测一个学生是否能进入大学。假设你是一所大学的行政管理人员，你想根据两门考试的结果，来决定每个申请人是否被录取。你有以前申请人的历史数据，可以将其用作逻辑回归训练集。对于每一个训练样本，你有申请人两次测评的分数以及录取的结果。为了完成这个预测任务，我们准备构建一个可以基于两次测试评分来评估录取可能性的分类模型。

### 1.1 Visualizing the data
在开始实现任何学习算法之前，如果可能的话，最好将数据可视化。

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
```

```python
data = pd.read_csv('ex2data1.txt', names=['exam1', 'exam2', 'admitted'])
data.head()
```

      exam1
      exam2
      admitted

      0
      34.623660
      78.024693
      0

      1
      30.286711
      43.894998
      0

      2
      35.847409
      72.902198
      0

      3
      60.182599
      86.308552
      1

      4
      79.032736
      75.344376
      1

```python
data.describe()
```

      exam1
      exam2
      admitted

      count
      100.000000
      100.000000
      100.000000

      mean
      65.644274
      66.221998
      0.600000

      std
      19.458222
      18.582783
      0.492366

      min
      30.058822
      30.603263
      0.000000

      25%
      50.919511
      48.179205
      0.000000

      50%
      67.032988
      67.682381
      1.000000

      75%
      80.212529
      79.360605
      1.000000

      max
      99.827858
      98.869436
      1.000000

让我们创建两个分数的散点图，并使用颜色编码来可视化，如果样本是正的（被接纳）或负的（未被接纳）。

```python
positive = data[data.admitted.isin(['1'])]  # 1
negetive = data[data.admitted.isin(['0'])]  # 0

fig, ax = plt.subplots(figsize=(6,5))
ax.scatter(positive['exam1'], positive['exam2'], c='b', label='Admitted')
ax.scatter(negetive['exam1'], negetive['exam2'], s=50, c='r', marker='x', label='Not Admitted')
# 设置图例显示在图的上方
box = ax.get_position()
ax.set_position([box.x0, box.y0, box.width , box.height* 0.8])
ax.legend(loc='center left', bbox_to_anchor=(0.2, 1.12),ncol=3)
# 设置横纵坐标名
ax.set_xlabel('Exam 1 Score')
ax.set_ylabel('Exam 2 Score')
plt.show()
```

看起来在两类间，有一个清晰的决策边界。现在我们需要实现逻辑回归，那样就可以训练一个模型来预测结果。

### 1.2 Sigmoid function

首先来回顾下 logistic回归的假设函数：
$${{h}_{\theta }}\left( x \right)=g(\theta^{T}x)=\frac{1}{1+{{e}^{-{{\theta }^{T}}X}}}$$
其中的 g代表一个常用的logistic function为S形函数（Sigmoid function）：
$$g\left( z \right)=\frac{1}{1+e^{-z}}$$

```python
def sigmoid(z):
    return 1 / (1 + np.exp(- z))
```

让我们做一个快速的检查，来确保它可以工作。

```python
x1 = np.arange(-10, 10, 0.1)
plt.plot(x1, sigmoid(x1), c='r')
plt.show()
```

感觉很不错~~~

### 1.3 Cost function

$$J\left( \theta  \right)=\frac{1}{m}\sum\limits_{i=1}^{m}{[-{{y}^{(i)}}\log \left( {{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)-\left( 1-{{y}^{(i)}} \right)\log \left( 1-{{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)]}$$
$${{h}_{\theta }}\left( x \right)=g(\theta^{T}x)$$

```python
def cost(theta, X, y):
    first = (-y) * np.log(sigmoid(X @ theta))
    second = (1 - y)*np.log(1 - sigmoid(X @ theta))
    return np.mean(first - second)  # Returns the average of the array elements.
```

现在，我们要做一些设置，获取我们的训练集数据。

```python
# add a ones column - this makes the matrix multiplication work out easier
if 'Ones' not in data.columns:
    data.insert(0, 'Ones', 1)

# set X (training data) and y (target variable)
X = data.iloc[:, :-1].values  # Convert the frame to its Numpy-array representation.
y = data.iloc[:, -1].values # Return is NOT a Numpy-matrix, rather, a Numpy-array.

theta = np.zeros(X.shape[1])
```

让我们来检查矩阵的维度，确保一切良好。

```python
X.shape, theta.shape, y.shape
```

    ((100, 3), (3,), (100,))

```python
cost(theta, X, y)
```

    0.6931471805599453

看起来不错，接下来，我们需要一个函数来计算我们的训练数据、标签和一些参数thate的梯度。

### 1.4 Gradient

* 这是批量梯度下降（batch gradient descent）
* 转化为向量化计算： $\frac{1}{m} X^T( Sigmoid(X\theta) - y )$
$$\frac{\partial J\left( \theta  \right)}{\partial {{\theta }_{j}}}=\frac{1}{m}\sum\limits_{i=1}^{m}{({{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}})x_{_{j}}^{(i)}}$$

```python
def gradient(theta, X, y):
    return (X.T @ (sigmoid(X @ theta) - y))/len(X)
# the gradient of the cost is a vector of the same length as θ where the jth element (for j = 0, 1, . . . , n)
```

```python
gradient(theta, X, y)
```

    array([ -0.1       , -12.00921659, -11.26284221])

### 1.5 Learning θ parameters

注意，我们实际上没有在这个函数中执行梯度下降，我们仅仅在计算梯度。在练习中，一个称为“fminunc”的Octave函数是用来优化函数来计算成本和梯度参数。由于我们使用Python，我们可以用SciPy的“optimize”命名空间来做同样的事情。

这里我们使用的是高级优化算法，运行速度通常远远超过梯度下降。方便快捷。
只需传入cost函数，已经所求的变量theta，和梯度。cost函数定义变量时变量tehta要放在第一个，若cost函数只返回cost，则设置fprime=gradient。

```python
import scipy.optimize as opt
```

这里使用fimin_tnc或者minimize方法来拟合，minimize中method可以选择不同的算法来计算，其中包括TNC

```python
result = opt.fmin_tnc(func=cost, x0=theta, fprime=gradient, args=(X, y))
result
# help(opt.fmin_tnc)
```

    (array([-25.16131855,   0.20623159,   0.20147149]), 36, 0)

最常使用的参数：

func：优化的目标函数

x0：初值

fprime：提供优化函数func的梯度函数，不然优化函数func必须返回函数值和梯度，或者设置approx_grad=True

approx_grad :如果设置为True，会给出近似梯度

args：元组，是传递给优化函数的参数

返回：

x ： 数组，返回的优化问题目标值

nfeval ： 整数，function evaluations的数目

在进行优化的时候，每当目标优化函数被调用一次，就算一个function evaluation。在一次迭代过程中会有多次function evaluation。这个参数不等同于迭代次数，而往往大于迭代次数。

rc ： int,Return code, see below
Return codes are defined as follows:

-1 : Infeasible (lower bound > upper bound)

 0 : Local minimum reached (|pg| ~= 0)

 1 : Converged (|f_n-f_(n-1)| ~= 0)

 2 : Converged (|x_n-x_(n-1)| ~= 0)

 3 : Max. number of function evaluations reached

 4 : Linear search failed

 5 : All lower bounds are equal to the upper bounds

 6 : Unable to progress

 7 : User requested end of minimization

下面是第二种方法，结果是一样的

```python
res = opt.minimize(fun=cost, x0=theta, args=(X, y), method='TNC', jac=gradient)
res
# help(opt.minimize)
# res.x  # final_theta
```

         fun: 0.20349770158947492
         jac: array([9.05849127e-09, 9.04782678e-08, 4.78076698e-07])
     message: 'Local minimum reached (|pg| ~= 0)'
        nfev: 36
         nit: 17
      status: 0
     success: True
           x: array([-25.16131855,   0.20623159,   0.20147149])

调用：

scipy.optimize.minimize(fun, x0, args=(), method=None, jac=None, hess=None, hessp=None, bounds=None, constraints=(), tol=None, callback=None, options=None)

参数：

fun ：优化的目标函数

x0 ：初值，一维数组，shape (n,)(theta)

args ： 元组，可选，额外传递给优化函数的参数(func的除theta外的)

method：求解的算法，选择TNC则和fmin_tnc()类似

jac：返回梯度向量的函数

返回：

返回优化结果对象，x：优化问题的目标数组。success: True表示成功与否，不成功会给出失败信息。

```python
cost(result[0], X, y)
```

    0.20349770158947492

### 1.6 Evaluating logistic regression

学习好了参数θ后，我们来用这个模型预测某个学生是否能被录取。

接下来，我们需要编写一个函数，用我们所学的参数theta来为数据集X输出预测。然后，我们可以使用这个函数来给我们的分类器的训练精度打分。

逻辑回归模型的假设函数：
	\\[{{h}_{\theta }}\left( x \right)=\frac{1}{1+{{e}^{-{{\theta }^{T}}X}}}\\]
当${{h}_{\theta }}$大于等于0.5时，预测 y=1

当${{h}_{\theta }}$小于0.5时，预测 y=0 。

```python
def predict(theta, X):
    probability = sigmoid(X@theta)
    return [1 if x >= 0.5 else 0 for x in probability]  # return a list
```

```python
final_theta = result[0]
predictions = predict(final_theta, X)
correct = [1 if a==b else 0 for (a, b) in zip(predictions, y)]  # python is amazon
accuracy = sum(correct) / len(X)
accuracy
```

    0.89

可以看到我们预测精度达到了89%，not bad.

也可以用skearn中的方法来检验。

```python
from sklearn.metrics import classification_report
print(classification_report(predictions, y))
```

### 1.6 Decision boundary（决策边界）

$X \times \theta = 0$  (this is the line)

$\theta_0 + x_1\theta_1 + x_2\theta_2=0$

```python
x1 = np.arange(20, 110, step=0.1)
x2 = -(final_theta[0] + x1*final_theta[1]) / final_theta[2]
```

```python
fig, ax = plt.subplots(figsize=(8,5))
ax.scatter(positive['exam1'], positive['exam2'], c='b', label='Admitted')
ax.scatter(negetive['exam1'], negetive['exam2'], s=50, c='r', marker='x', label='Not Admitted')
ax.plot(x1, x2)
ax.set_xlim(20, 110)
ax.set_ylim(20, 110)
ax.set_xlabel('x1')
ax.set_ylabel('x2')
ax.set_title('Decision Boundary')
plt.show()
```

## 2 Regularized logistic regression

在训练的第二部分，我们将要通过加入正则项提升逻辑回归算法。简而言之，正则化是成本函数中的一个术语，它使算法更倾向于“更简单”的模型（在这种情况下，模型将更小的系数）。这个理论助于减少过拟合，提高模型的泛化能力。这样，我们开始吧。

设想你是工厂的生产主管，你有一些芯片在两次测试中的测试结果。对于这两次测试，你想决定是否芯片要被接受或抛弃。为了帮助你做出艰难的决定，你拥有过去芯片的测试数据集，从其中你可以构建一个逻辑回归模型。

### 2.1 Visualizing the data

```python
data2 = pd.read_csv('ex2data2.txt', names=['Test 1', 'Test 2', 'Accepted'])
data2.head()
```

      Test 1
      Test 2
      Accepted

      0
      0.051267
      0.69956
      1

      1
      -0.092742
      0.68494
      1

      2
      -0.213710
      0.69225
      1

      3
      -0.375000
      0.50219
      1

      4
      -0.513250
      0.46564
      1

```python
data2.describe()
```

      Test 1
      Test 2
      Accepted

      count
      118.000000
      118.000000
      118.000000

      mean
      0.054779
      0.183102
      0.491525

      std
      0.496654
      0.519743
      0.502060

      min
      -0.830070
      -0.769740
      0.000000

      25%
      -0.372120
      -0.254385
      0.000000

      50%
      -0.006336
      0.213455
      0.000000

      75%
      0.478970
      0.646562
      1.000000

      max
      1.070900
      1.108900
      1.000000

```python
def plot_data():
    positive = data2[data2['Accepted'].isin([1])]
    negative = data2[data2['Accepted'].isin([0])]

    fig, ax = plt.subplots(figsize=(8,5))
    ax.scatter(positive['Test 1'], positive['Test 2'], s=50, c='b', marker='o', label='Accepted')
    ax.scatter(negative['Test 1'], negative['Test 2'], s=50, c='r', marker='x', label='Rejected')
    ax.legend()
    ax.set_xlabel('Test 1 Score')
    ax.set_ylabel('Test 2 Score')

plot_data()
```

注意到其中的正负两类数据并没有线性的决策界限。因此直接用logistic回归在这个数据集上并不能表现良好，因为它只能用来寻找一个线性的决策边界。

所以接下会提到一个新的方法。

### 2.2 Feature mapping

一个拟合数据的更好的方法是从每个数据点创建更多的特征。

我们将把这些特征映射到所有的x1和x2的多项式项上，直到第六次幂。

https://www.zhihu.com/question/65020904
```
for i in 0..power
      for p in 0..i:
        output x1^(i-p) * x2^p```

$${\rm{mapFeature}}(x)=\begin{bmatrix}
{1}\\
{x_1}\\
{x_2}\\
{x_1^2}\\
{x1x2}\\
{x_2^2}\\
{x_1^3}\\
\vdots\\
{x_1x_2^5}\\
{x_2^6}
\end{bmatrix}$$

```python
def feature_mapping(x1, x2, power):
    data = {}
    for i in np.arange(power + 1):
        for p in np.arange(i + 1):
            data["f{}{}".format(i - p, p)] = np.power(x1, i - p) * np.power(x2, p)

#     data = {"f{}{}".format(i - p, p): np.power(x1, i - p) * np.power(x2, p)
#                 for i in np.arange(power + 1)
#                 for p in np.arange(i + 1)
#             }
    return pd.DataFrame(data)
```

```python
x1 = data2['Test 1'].values
x2 = data2['Test 2'].values
```

```python
_data2 = feature_mapping(x1, x2, power=6)
_data2.head()
```

      f00
      f10
      f01
      f20
      f11
      f02
      f30
      f21
      f12
      f03
      ...
      f23
      f14
      f05
      f60
      f51
      f42
      f33
      f24
      f15
      f06

      0
      1.0
      0.051267
      0.69956
      0.002628
      0.035864
      0.489384
      0.000135
      0.001839
      0.025089
      0.342354
      ...
      0.000900
      0.012278
      0.167542
      1.815630e-08
      2.477505e-07
      0.000003
      0.000046
      0.000629
      0.008589
      0.117206

      1
      1.0
      -0.092742
      0.68494
      0.008601
      -0.063523
      0.469143
      -0.000798
      0.005891
      -0.043509
      0.321335
      ...
      0.002764
      -0.020412
      0.150752
      6.362953e-07
      -4.699318e-06
      0.000035
      -0.000256
      0.001893
      -0.013981
      0.103256

      2
      1.0
      -0.213710
      0.69225
      0.045672
      -0.147941
      0.479210
      -0.009761
      0.031616
      -0.102412
      0.331733
      ...
      0.015151
      -0.049077
      0.158970
      9.526844e-05
      -3.085938e-04
      0.001000
      -0.003238
      0.010488
      -0.033973
      0.110047

      3
      1.0
      -0.375000
      0.50219
      0.140625
      -0.188321
      0.252195
      -0.052734
      0.070620
      -0.094573
      0.126650
      ...
      0.017810
      -0.023851
      0.031940
      2.780914e-03
      -3.724126e-03
      0.004987
      -0.006679
      0.008944
      -0.011978
      0.016040

      4
      1.0
      -0.513250
      0.46564
      0.263426
      -0.238990
      0.216821
      -0.135203
      0.122661
      -0.111283
      0.100960
      ...
      0.026596
      -0.024128
      0.021890
      1.827990e-02
      -1.658422e-02
      0.015046
      -0.013650
      0.012384
      -0.011235
      0.010193

5 rows × 28 columns

经过映射，我们将有两个特征的向量转化成了一个28维的向量。

在这个高维特征向量上训练的logistic回归分类器将会有一个更复杂的决策边界，当我们在二维图中绘制时，会出现非线性。

虽然特征映射允许我们构建一个更有表现力的分类器，但它也更容易过拟合。在接下来的练习中，我们将实现正则化的logistic回归来拟合数据，并且可以看到正则化如何帮助解决过拟合的问题。

也可利用 PolynomialFeatures 函数映射多项式特征

```python
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(6)
poly.fit_transform(data2.iloc[:, 0:2].values)
```

### 2.3 Regularized Cost function

$$J\left( \theta  \right)=\frac{1}{m}\sum\limits_{i=1}^{m}{[-{{y}^{(i)}}\log \left( {{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)-\left( 1-{{y}^{(i)}} \right)\log \left( 1-{{h}_{\theta }}\left( {{x}^{(i)}} \right) \right)]}+\frac{\lambda }{2m}\sum\limits_{j=1}^{n}{\theta _{j}^{2}}$$

注意：不惩罚第一项$\theta_0$

先获取特征，标签以及参数theta，确保维度良好。

```python
X = _data2.values
y = data2['Accepted'].values
theta = np.zeros(X.shape[1])
X.shape, y.shape, theta.shape
```

    ((118, 28), (118,), (28,))

```python
def costReg(theta, X, y, l=1):
    # 不惩罚第一项
    _theta = theta[1: ]
    reg = (l / (2 * len(X))) *(_theta @ _theta)  # _theta@_theta == inner product

    return cost(theta, X, y) + reg
```

```python
costReg(theta, X, y, l=1)
```

    0.6931471805599454

### 2.4 Regularized gradient

因为我们未对${{\theta }_{0}}$ 进行正则化，所以梯度下降算法将分两种情形：
\begin{align}
  & Repeat\text{ }until\text{ }convergence\text{ }\!\!\{\!\!\text{ } \\
 & \text{     }{{\theta }_{0}}:={{\theta }_{0}}-a\frac{1}{m}\sum\limits_{i=1}^{m}{[{{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}}]x_{_{0}}^{(i)}} \\
 & \text{     }{{\theta }_{j}}:={{\theta }_{j}}-a\frac{1}{m}\sum\limits_{i=1}^{m}{[{{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}}]x_{j}^{(i)}}+\frac{\lambda }{m}{{\theta }_{j}} \\
 & \text{          }\!\!\}\!\!\text{ } \\
 & Repeat \\
\end{align}

对上面的算法中 j=1,2,...,n 时的更新式子进行调整可得：
${{\theta }_{j}}:={{\theta }_{j}}(1-a\frac{\lambda }{m})-a\frac{1}{m}\sum\limits_{i=1}^{m}{({{h}_{\theta }}\left( {{x}^{(i)}} \right)-{{y}^{(i)}})x_{j}^{(i)}}$

同样不惩罚第一个θ

```python
def gradientReg(theta, X, y, l=1):
    reg = (1 / len(X)) * theta
    reg[0] = 0
    return gradient(theta, X, y) + reg
```

```python
gradientReg(theta, X, y, 1)
```

    array([8.47457627e-03, 1.87880932e-02, 7.77711864e-05, 5.03446395e-02,
           1.15013308e-02, 3.76648474e-02, 1.83559872e-02, 7.32393391e-03,
           8.19244468e-03, 2.34764889e-02, 3.93486234e-02, 2.23923907e-03,
           1.28600503e-02, 3.09593720e-03, 3.93028171e-02, 1.99707467e-02,
           4.32983232e-03, 3.38643902e-03, 5.83822078e-03, 4.47629067e-03,
           3.10079849e-02, 3.10312442e-02, 1.09740238e-03, 6.31570797e-03,
           4.08503006e-04, 7.26504316e-03, 1.37646175e-03, 3.87936363e-02])

### 2.5 Learning parameters

```python
result2 = opt.fmin_tnc(func=costReg, x0=theta, fprime=gradientReg, args=(X, y, 1))
result2
```

    (array([ 1.27271026,  0.62529964,  1.18111686, -2.019874  , -0.91743188,
            -1.43166928,  0.12393227, -0.36553118, -0.35725405, -0.17516292,
            -1.45817009, -0.05098418, -0.61558558, -0.27469166, -1.19271299,
            -0.24217841, -0.20603303, -0.04466177, -0.27778948, -0.29539514,
            -0.45645982, -1.04319154,  0.02779373, -0.29244865,  0.01555759,
            -0.32742404, -0.14389149, -0.92467488]),
     32,
     1)

我们还可以使用高级Python库scikit-learn来解决这个问题。

```python
from sklearn import linear_model#调用sklearn的线性回归包
model = linear_model.LogisticRegression(penalty='l2', C=1.0)
model.fit(X, y.ravel())
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
              penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
              verbose=0, warm_start=False)

```python
model.score(X, y)
```

    0.8305084745762712

### 2.6 Evaluating logistic regression

```python
final_theta = result2[0]
predictions = predict(final_theta, X)
correct = [1 if a==b else 0 for (a, b) in zip(predictions, y)]
accuracy = sum(correct) / len(correct)
accuracy
```

    0.8305084745762712

或者用skearn中的方法来评估结果。

```python
print(classification_report(y, predictions))
```

                 precision    recall  f1-score   support

              0       0.85      0.75      0.80        60
              1       0.77      0.86      0.81        58

    avg / total       0.81      0.81      0.80       118

可以看到和skearn中的模型精确度差不多，这很不错。

### 2.7 Decision boundary（决策边界）

$X \times \theta = 0$  (this is the line)

```python
x = np.linspace(-1, 1.5, 250)
xx, yy = np.meshgrid(x, x)

z = feature_mapping(xx.ravel(), yy.ravel(), 6).values
z = z @ final_theta
z = z.reshape(xx.shape)

plot_data()
#plt.contour(xx, yy, z, 0)
CS = plt.contour(xx, yy, z, 0, colors='green')
plt.clabel(CS, fontsize=10)
plt.xlim(-1, 1.3)
plt.ylim(-1, 1.3)
```


# 07 Solution: Sine Series

* *Computational Physics*:  2.5

## Problem: Summing Series: sin(x)

Evaluate the $\sin$ function from its series representation
$$
\sin x = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \dots
$$

A naive algorithm is to sum the series up to the $N$th term:
$$
\sin x \approx \sum_{n=1}^N \frac{(-1)^{n-1} x^{2n-1}}{(2n - 1)!}
$$

Problems:

- How to decide when to stop summing?
- Division of large terms (overflows!)
- Powers and factorials are very expensive to compute.

Better approach: Build up series terms $a_n$ using previous term $a_{n-1}$ through a recursion:

\begin{align}
a_n &= a_{n-1} \times q_n\\
a_n &= \frac{(-1)^{n-1} x^{2n-1}}{(2n - 1)!} = \frac{(-1)^{n-2} x^{2n-3}}{(2n - 3)!} \frac{-x^2}{(2n - 1)(2n - 2)}\\
a_n & = a_{n-1} \frac{-x^2}{(2n - 1)(2n - 2)}
\end{align}

Accuracy of this approach? Not clear in absolute terms but we can make the assumption that the error is approximately the last term in the sum, $a_N$. Hence we can strive to make the relative error smaller than the machine precision
$$
\left| \frac{a_N}{\sum_{n=1}^N a_n} \right| < \epsilon_m
$$

```python
import numpy as np
```

```python
def sin_series(x, eps=1e-16):
    """Calculate sin(x) to precision eps with recursion.

    Arguments
    ---------
    x : float
        argument of sin(x)
    eps : float, optional
        desired precision

    Returns
    -------
    func_value, N

    where func_value = sin(x) and N is the number
    of terms included in the approximation.
    """
    if x == 0:
        return 0., 1
    a_n = sumN = x   # a_1 is the sum
    n = 2            # start with calculating a_2
    while abs(a_n/sumN) > eps:
        a_n *= -x*x/((2*n - 1)*(2*n - 2))
        sumN += a_n
        n += 1
    return sumN, n-1
```

Bonus problem: the really bad implementation:

```python
import math
def sin_series_bad(x, eps=1e-16):
    """Calculate sin(x) to precision eps... BADLY"""
    if x == 0:
        return 0., 1
    a_n = x
    n = 1
    sumN = 0.
    while True:
        a_n = (-1)**(n-1) * x**(2*n-1) / math.factorial(2*n-1)
        sumN += a_n
        if abs(a_n/sumN) <=  eps:
            break
        n += 1
    return sumN, n
```

```python
sin_series_bad(10*np.pi)
```

    (0.0006695609195825486, 61)

```python
sin_series(10*np.pi, eps=1e-16)
```

    (-0.0003048974510408005, 61)

Test the implementation against the "exact" numpy function `np.sin()`.

Report
1. `x`
2. maximum `n`
3. `sin_series(x)`
4. relative error `abs(sin_series(x) - sin(x))/abs(sin(x))`

Plot against `x` the quantities above and also `sin(x)`.
* `x` - `sin_series(x)`
* `x` - `sin(x)`
* `x` - max `n`
* `x` - relative error (semilogy plot!)

For a range of numbers look at `Xsmall` and `Xlarge`:

```python
Xsmall = np.pi*np.arange(-2, 2.05, 0.05)
Xlarge = np.pi*np.arange(-50, 50.05, 0.1)
```

Implementation (modularized compared to the problem so that I can use it for `sin_series()` and `sin_series_bad()`):

```python
import numpy as np
def test_sin(x):
    y, nmax = sin_series(x)
    return compare_to_sin(x, y, nmax)

def test_sin_bad(x):
    y, nmax = sin_series_bad(x)
    return compare_to_sin(x, y, nmax)

def compare_to_sin(x, y, nmax):
    y0 = np.sin(x)
    delta = y - y0
    if y0 != 0:
        relative_error = delta/y0
    else:
        relative_error = 0
    # print(x, y, y0, delta, relative_error)
    #return x, y, y0, delta, relative_error
    return x, nmax, y, relative_error, delta
```

The functions do not work as numpy ufuncs on whole arrays; therefore, loop over all input values $x$ like the following code:

```python
results = np.array([test_sin(x) for x in Xsmall])
```

Plot the approximation and the exact solution:

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
def test_plot_sine(X, filename="sine_error.pdf"):
    results = np.array([test_sin(x) for x in X])
    return plot_results(results, filename)

def test_plot_bad_sine(X, filename="sine_bad_error.pdf"):
    results = np.array([test_sin_bad(x) for x in X])
    return plot_results(results, filename)

def plot_results(results, filename):
    fig = plt.figure(figsize=(6, 8))

    ax1 = fig.add_subplot(4,1,1)
    ax1.plot(results[:, 0], results[:, 2], 'k-', lw=1, label="series")
    ax1.plot(results[:, 0], np.sin(results[:, 0]), 'g--', lw=2, label="sin x")
    ax1.legend(loc="best")

    ax2 = fig.add_subplot(4,1,2)
    ax2.plot(results[:, 0], results[:, 1], label="max n")
    ax2.legend(loc="best")

    ax3 = fig.add_subplot(4,1,3)
    ax3.semilogy(results[:, 0], results[:, 3], label="rel.error")
    ax3.legend(loc="best")

    ax4 = fig.add_subplot(4, 1, 4)
    ax4.semilogy(results[:, 0], results[:, 4], label="abs.error")
    ax4.legend(loc="best")

    fig.suptitle("sine series approximation")

    fig.savefig(filename)
    print("saved to file {0}".format(filename))
```

### Recursive implementation

Now test the two ranges of numbers and write to different files:

For a few oscillations near $x=0$ the approximation appears to hold pretty well with relative errors near machine precision.

```python
test_plot_sine(Xsmall, filename="sine_error_Xsmall.pdf")
```

But for larger $|x|$, the approximation breaks down and round-off error completely swamps the result:

```python
test_plot_sine(Xlarge, filename="sine_error_Xlarge.pdf")
```

#### More exploration

```python
test_plot_sine(np.pi*np.arange(-5, 5, 0.05))
```

```python
test_plot_sine(np.pi*np.arange(-15, 15, 0.05))
```

### Direct implementation
Let's see how the (supposedly) bad implementation holds up:

```python
test_plot_bad_sine(Xsmall, filename="sine_bad_error_Xsmall.pdf")
```

For the small range, the direct implementation performs similarly to the recursion.

```python

```

The `Xlarge` range overflows with an `OverflowError`

```python
test_plot_bad_sine(Xlarge, filename="sine_bad_error_Xlarge.pdf")
```

so we choose a smaller one:

```python
X10 = np.pi * np.arange(-15, 15.1, 0.05)
test_plot_bad_sine(X10, filename="sine_bad_error_X10.pdf")
```

In this range, the direct implementation and the recursion perform similarly badly (although the direct one is much slower).

### Speed comparison

The recursion implementation is faster:

```python
%timeit sin_series(-10/3*np.pi)
```

    10.3 µs ± 129 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
%timeit sin_series_bad(-10/3*np.pi)
```

    32.8 µs ± 571 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

```python
sin_series(-10/3*np.pi)
```

    (0.866025403783719, 28)

```python
sin_series_bad(-10/3*np.pi)
```

    (0.8660254037851793, 28)

## Conclusion

The recursion-based implementation is better because it is faster and does not suffer from overflows. However, in terms of accuracy it is as bad as the direct implementation. Both fail for $|x| > 10\pi$.

```python

```