## Gaussian Integration

So far, we have focussed on integration schemes where the function is sampled at regular intervals.  These are very useful for data that comes to us in this form, but if we wish to integrate a known function as accurately as possible it is better to take advantage of the freedom we have to choose our nodes, just as we did for interpolation where selecting Chebyshev points was optimal.

Consider that quadrature formulas are typically of the form

$$
\int_a^b f(x) dx = \sum_{i=0}^{n-1} c_i f(x_i) + error,
$$

which has $2n$ parameters (note for this section we are stopping the sum at $n-1$ rather than $n$ as this is conventional for Gaussian quadrature formulas).  The idea of Gaussian integration is that with these $2n$ parameters it should be possible to chose them to ensure that we can integrate all polynomials of degree less than or equal to $2n-1$ *exactly*.

Before we do this, it is useful to first transform to a standard interval, namely $y$ in $[-1,1]$, from the original $x$ in $[a,b]$.  Similar to what we did with the Chebyshev interpolation, we use the change of variables

$$ x = \frac{b-a}{2}y+\frac{b+a}{2}. $$

We must also remember to perform the appropriate change of the integral measure,

$$\int_a^b f(x) dx = \int_{-1}^1 f\left(\frac{b-a}{2}y+\frac{b+a}{2}\right) \frac{dx}{dy} dy,$$

with $\frac{dx}{dy}=\frac{b-a}{2}$.  The $n-$point Gaussian quadrature routine will then result in

$$\int_a^b f(x) dx \approx \frac{b-a}{2}\sum_{i=0}^{n-1} c_i f\left(\frac{b-a}{2}y_i+\frac{b+a}{2}\right).$$

So, we will focus on integration over $[-1,1]$ below.

**Theorem** Suppose $\{x_0,x_1,\cdots,x_{n-1}\}$ are the roots of the $n$th Legendre polynomial $P_n(x)$ and

$$ c_i=\int_{-1}^1 L_i^{n-1}(x) dx,\quad i=0,1,\cdots,n-1$$

where $L_i^{n-1}(x)$ is the Lagrange polynomial (from interpolation).  Then if $p(x)$ is any polynomial of degree less than or equal to $2n-1$, then

$$ \int_{-1}^1 p(x) dx = \sum_{i=0}^{n-1} c_i p(x_i),$$

exactly.

**Proof:**  First look at the case where the degree of $p(x)$ is less than $n$.  Then

$$ p(x) = \sum_{i=0}^{n-1} L_i^n(x) p(x_i) $$

is an exact interpolation of $p(x)$ (the [interpolation error](../InterpFit/InterpErrors) involves $\frac{d^n p(x)}{dx^n}$ which is zero for a polynomial of degree less than $n$).  Hence,

$$\int_{-1}^1 p(x)dx = \int_{-1}^1 \sum_{i=0}^{n-1} L_i^n(x) p(x_i) = \sum_{i=0}^{n-1} c_i p(x_i),$$

as claimed in the theorem.  Now suppose that the degree of $p(x)$ is at least $n$ but less than or equal to $2n-1$.  Then we can write

$$p(x) = q(x)P_n(x) + r(x),$$

where $P_n(x)$ is the $n$th Legendre polynomial (which has degree $n$) and $q(x)$ and $r(x)$ are polynomials of degree less than $n$.  (To construct $q(x)$ you pick its coefficients so that $q(x)P_n(x)$ has coefficients that match coefficients of $p(x)$ for terms $x^{2n-1}$ to $x^n$.  $r(x)$ is then whatever is left over).  Now,

$$ p(x_i) = q(x_i)P_n(x_i) + r(x_i) = r(x_i),$$

where the last step follows from the fact that the $x_i$ are the roots of $P_n(x_i)$.  As a result,

$$\sum_{i=0}^{n-1} c_i p(x_i) = \sum_{i=0}^{n-1} c_i r(x_i).$$

Also,

$$
\begin{align}
\int_{-1}^1 p(x) dx &= \int_{-1}^1\left(q(x)P_n(x) + r(x)\right)dx,\\
&= \int_{-1}^1 q(x)P_n(x) dx + \int_{-1}^1  r(x) dx.
\end{align}
$$

Now $q(x)$ has degree less than $n$ so can be written using the Legendre polynomials as a basis (they are an orthogonal basis set for polynomials), i.e.

$$
q(x) = \sum_{i=0}^{n-1} a_i P_i(x),
$$

for some constants $a_i$.  As a result,

$$
\int_{-1}^1 q(x)P_n(x) dx = \sum_{i=0}^{n-1} a_i \int_{-1}^1 P_i(x)P_n(x) dx =0,
$$

as $P_n(x)$ is orthogonal to all the $P_i(x)$ with $i<n$.  As a result, we now have

$$
\begin{align}
\int_{-1}^1 p(x) dx&=\int_{-1}^1  r(x) dx,\\
&= \sum_{i=0}^{n-1} c_i r(x_i),
\end{align}
$$

where the last step is exact as $r(x)$ is of degree less than $n$ (and we showed this was exact for such polynomials in the first step of the proof above).

--------

A general formula for the weights and nodes for Gauss-Legendre integration is not available.  However, the first few have all been tabulated and are listed below for reference :

| Number of nodes, $n$ | Points, $x_i$ | weights, $w_i$  |
| :------------------: | :-----------: | :-------------: |
| 1                    | 0             | 2               |
| 2                    | $\pm \frac{1}{\sqrt{3}}$ | 1     |
| 3                    | 0,            | $\frac{8}{9},$    |
|                      | $\pm \sqrt{\frac{3}{5}}$ | $\frac{5}{9}$ |
| 4                    | $\pm \sqrt{\frac{3}{7}-\frac{2}{7}\sqrt{\frac{6}{5}}},$ | $\frac{18+\sqrt{30}}{36},$ |
|                      | $\pm \sqrt{\frac{3}{7}+\frac{2}{7}\sqrt{\frac{6}{5}}}$| $\frac{18-\sqrt{30}}{36}$ |
| 5                    | 0,             | $\frac{128}{225},$ |
|                      | $\pm \frac{1}{3}\sqrt{5-2\sqrt{\frac{10}{7}}},$ | $\frac{322+13\sqrt{70}}{900},$ |
|                      | $\pm \frac{1}{3}\sqrt{5+2\sqrt{\frac{10}{7}}}$ | $\frac{322-13\sqrt{70}}{900}$ |
| 7                    | 0,             | 0.41795 91836 73469, |
|                      | $\pm$0.40584 51513 77397, | 0.38183 00505 05119, |
|                      | $\pm$0.74153 11855 99394, | 0.27970 53914 89277, |
|                      | $\pm$0.94910 79123 42759  | 0.12948 49661 68870  |

SciPy has an implementation of Gauss-Legendre integration illustrated in the example below:

```python
import numpy as np
from scipy import integrate
normaldist = lambda x: 1/np.sqrt(np.pi) * np.exp(-x**2)
print("n=1 result = ",integrate.fixed_quad(normaldist, 0.0, 2.0, n=1))
print("n=2 result = ",integrate.fixed_quad(normaldist, 0.0, 2.0, n=2))
print("n=3 result = ",integrate.fixed_quad(normaldist, 0.0, 2.0, n=3))
print("n=4 result = ",integrate.fixed_quad(normaldist, 0.0, 2.0, n=4))
print("n=5 result = ",integrate.fixed_quad(normaldist, 0.0, 2.0, n=5))
print("n=7 result = ",integrate.fixed_quad(normaldist, 0.0, 2.0, n=7))
```

    n=1 result =  (0.4151074974205947, None)
    n=2 result =  (0.5187644892255823, None)
    n=3 result =  (0.4958462362382512, None)
    n=4 result =  (0.49774446622792934, None)
    n=5 result =  (0.49765922944191265, None)
    n=7 result =  (0.4976611363409025, None)

We also integrated this same function in our example for [Romberg integration](./HigherOrderInt).  The accuracy of our result for $n=7$ here (so 7 function evaluations) was only obtained in the row for 16 steps (17 function evaluations) using Romberg integration!

```python

```


```python
import sympy as sy
import numpy as np
from sympy.plotting import plot, plot_parametric
import scipy.optimize as sci_opt
```

```python
sy_t = sy.symbols('t')

def sy_straight(p0, d, sy_t=sy_t):
    v_p0 = sy.Matrix(p0)
    v_d = sy.Matrix(d)
    sy_f = v_p0 + v_d.multiply(sy_t)
    return sy_f

def sy_arc(p0, r, a0, d, sy_t=sy_t):
    v_p0 = sy.Matrix(p0)
    sy_tau = a0 + d*sy_t
    sy_g = sy.Matrix([sy.cos(sy_tau), sy.sin(sy_tau)])
    sy_f = v_p0 + sy_g.multiply(r)
    return sy.simplify(sy_f)

def sy_interp(sy_f, sy_g, sy_t=sy_t):
    return sy.simplify(sy_t*sy_f + (1.0-sy_t)*sy_g)

def sy_lin_map(sy_f, t0, t1, sy_t=sy_t):
    sy_g = t0+(t1-t0)*sy_t
    return sy.simplify(sy_f.subs({sy_t:sy_g}))

def sy_f_tangent(sy_f, sy_t=sy_t):
    return sy.simplify(sy_f.jacobian(sy.Matrix([sy_t])))

def sy_f_normal(sy_f, sy_t=sy_t):
    tng = sy_f_tangent(sy_f, sy_t)
    return sy.simplify(sy.Matrix([-tng[1], tng[0]]))

def sy_f_arc_length(sy_f, sy_t=sy_t):
    sy_f_jac = sy_f.jacobian(sy.Matrix([sy_t]))
    fx, fy = sy_f_jac[0], sy_f_jac[1]
    sy_l = sy.sqrt(fx*fx + fy*fy)
    return sy.simplify(sy.integrate(sy_l, sy_t))

def sy_f_arc_length_eval(sy_f_arc_length, t0, t1, sy_t=sy_t):
    return sy_f_arc_length.subs(sy_t, t1)-sy_f_arc_length.subs(sy_t, t0)

def closest_t(sy_f, pt, sy_t=sy_t):
    obj_f_vec = sy_f - sy.Matrix(pt)
    obj_f = obj_f_vec.dot(obj_f_vec)
    #print(obj_f)
    res = sci_opt.minimize(sy.lambdify(sy_t, obj_f), 0.5) # TODO jac
    return res.x if res.success else None

def dbg_print_sy_f(sy_f, sy_t=sy_t):
    print(sy_f)
    print(sy_f.jacobian(sy.Matrix([sy_t])))
    # print(sy.lambdify(sy_t, sy_f))
    print(sy.lambdify(sy_t, sy_f)(0.5))
    plot_parametric((sy_f[0], sy_f[1]), (sy_t, 0,1), aspect_ratio=(1,1));

def dbg_plot_sy_f_batch(sy_f_batch, colors=['blue','green', 'purple', 'pink', 'red', 'black'], sy_t=sy_t):
    plts = [plot_parametric((sy_f_batch[i][0], sy_f_batch[i][1]), (sy_t, 0,1), aspect_ratio=(1,1), line_color=colors[i%len(colors)], show=False) for i in range(len(sy_f_batch))]
    for plt in plts[1:]:
        plts[0].append(plt[0])
    plts[0].show()

def dbg_plot_closest(sy_f, pt, sy_t=sy_t):
    t = closest_t(sy_f, pt)
    print(t)
    closest_pt = sy_f.subs(sy_t, t)
    print(list(closest_pt.vec()))
    closest_dbg_seg = sy_straight(pt, closest_pt-sy.Matrix(pt))
    print(closest_dbg_seg)
    dbg_plot_sy_f_batch([sy_f, closest_dbg_seg])
```

```python
straight_0 = sy_straight([1,1], [10,20])
dbg_print_sy_f(straight_0)
```

    Matrix([[10*t + 1], [20*t + 1]])
    Matrix([[10], [20]])
    [[ 6.]
     [11.]]

    <Figure size 640x480 with 1 Axes>

```python
arc_0 = sy_arc([1,1], 10, 0.1, -4)
dbg_print_sy_f(arc_0)
```

```python
dbg_plot_sy_f_batch([straight_0, arc_0])
```

```python
arc_1 = sy_arc([0,0], 10, 0, np.pi/2)
arc_2 = sy_arc([0,0], 11, 0, np.pi/2)
arc_1_2 = sy_interp(arc_1, arc_2)
dbg_plot_sy_f_batch([arc_1, arc_2, arc_1_2])
```

```python
straight_1 = sy_lin_map(straight_0, 0.5, 1)
dbg_print_sy_f(straight_1)
```

```python
print(sy_f_tangent(arc_1))
print(sy_f_normal(arc_1))
print(sy_f_normal(arc_1_2))
dbg_print_sy_f(sy_f_normal(arc_1_2))
```

```python
print(sy_f_arc_length(straight_1))
arc_1_length = sy_f_arc_length(arc_1)
print(arc_1_length)
print(sy_f_arc_length_eval(arc_1_length, 0,1.0))
print(sy_f_arc_length(arc_1_2))
```

    11.1803398874989*t
    15.707963267949*t
    15.7079632679490
    17.3076726665648*Integral(1.0*sqrt(0.00823687372577165*t**2 - 0.181211221966976*t + 1.0), t)

```python
dbg_plot_closest(arc_1_2, [8,2])
dbg_plot_closest(arc_1_2, [8,5])
dbg_plot_closest(arc_1_2, [6,10])
```

```python
#TODO: chain
```


# Time Series - IV (ARMA Models)

We have seen how Auroregressive (AR) and Moving average (MA) models explains the autocorrelation in data and we have also seen that not all of it can be explained by any one simple model. ARMA models or Autoregressive Moving average models are the combination of AR and MA models.

ARMA(p,z) is a combination of an AR model of order p and a MA model of order q. In the combination:
-  AR(p) model explains the momentum and the mean reversal effect of the market.
-  MA(q) model explains the shock effects observed.

ARMA(1,1) model looks like this:
\begin{align}
x_{t} = \alpha * x_{t-1} + \beta * e_{t-1} + e_{t}
\end{align}

where e(t) is white noise with mean = 0

```python
# importing the necessary packages
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt

%matplotlib inline

import statsmodels.tsa.api as smt
import statsmodels.api as sm
import scipy.stats as scs
import statsmodels.stats as sms

import warnings
warnings.filterwarnings('ignore')
```

```python
#Defining a function to visualize and analyze the time series

def tsplot(y, lags=None, figsize=(15, 10), style='bmh'):
    '''
    Prepares a (3,2) dimensional plot for the visualization of time series values, autocorrelation and partial
    autocorrelation plots and QQ and probability plots for comparision with normal distribution.

    Args:
    y: time series values
    lags: How many lagging values are to be considered.
    '''
    if not isinstance(y, pd.Series):
        y = pd.Series(y)
    with plt.style.context(style):
        fig = plt.figure(figsize=figsize)

        layout = (3, 2)
        ts_ax = plt.subplot2grid(layout, (0, 0), colspan=2)
        acf_ax = plt.subplot2grid(layout, (1, 0))
        pacf_ax = plt.subplot2grid(layout, (1, 1))
        qq_ax = plt.subplot2grid(layout, (2, 0))
        pp_ax = plt.subplot2grid(layout, (2, 1))

        y.plot(ax=ts_ax)
        ts_ax.set_title('Time Series Analysis Plots')
        smt.graphics.plot_acf(y, lags=lags, ax=acf_ax, alpha=0.05)
        smt.graphics.plot_pacf(y, lags=lags, ax=pacf_ax, alpha=0.05)
        sm.qqplot(y, line='s', ax=qq_ax)
        qq_ax.set_title('QQ Plot')
        scs.probplot(y, sparams=(y.mean(), y.std()), plot=pp_ax)

        plt.tight_layout()
    return
```

**Simulating an ARMA(2,2) model with alphas = [0.5, -0.25] and betas = [0.5, -0.3] **

```python
max_lag = 30
n = int(5000)
burn = int(n/10) #no of samples to disgard before fit

alphas = np.array([0.5, -0.25])
betas = np.array([0.5, -0.3])

ar = np.r_[1, -alphas]
ma = np.r_[1, betas]

arma22 = smt.arma_generate_sample(ar = ar, ma = ma, nsample = n, burnin = burn)
_ = tsplot(arma22, lags = max_lag)
```

Now that we have our ARMA(2,2) data, we will fit a model on this and see how well is our model able to estimate the alphas and betas.

```python
model = smt.ARMA(arma22, order = (2,2)).fit( maxlag = max_lag,
                                           method = 'mle',
                                           trend = 'nc',
                                           burnin = burn)

print(model.summary())
```

                                  ARMA Model Results
    ==============================================================================
    Dep. Variable:                      y   No. Observations:                 5000
    Model:                     ARMA(2, 2)   Log Likelihood               -7042.946
    Method:                           mle   S.D. of innovations              0.990
    Date:                Tue, 26 Jun 2018   AIC                          14095.892
    Time:                        00:10:19   BIC                          14128.478
    Sample:                             0   HQIC                         14107.312

    ==============================================================================
                     coef    std err          z      P>|z|      [0.025      0.975]
    ------------------------------------------------------------------------------
    ar.L1.y        0.4555      0.053      8.521      0.000       0.351       0.560
    ar.L2.y       -0.2699      0.015    -17.781      0.000      -0.300      -0.240
    ma.L1.y        0.5684      0.055     10.355      0.000       0.461       0.676
    ma.L2.y       -0.2348      0.050     -4.654      0.000      -0.334      -0.136
                                        Roots
    =============================================================================
                      Real          Imaginary           Modulus         Frequency
    -----------------------------------------------------------------------------
    AR.1            0.8439           -1.7300j            1.9249           -0.1778
    AR.2            0.8439           +1.7300j            1.9249            0.1778
    MA.1           -1.1820           +0.0000j            1.1820            0.5000
    MA.2            3.6025           +0.0000j            3.6025            0.0000
    -----------------------------------------------------------------------------

**How do we decide the value of p and q to be choosen? **

We use AIC or Akaike Information Criteria to find the relative quality of a model on a given data. In the process, we fit multiple ARMA models or better ARMA models with multiple orders and then compare the values of the AIC. We finally select the model with minimum AIC value and perform Ljung-Box test if the model is a good fit or not.

**Ljung-Box test: ** This is a type of statistical test in which we try to reject the null hypothesis that the data into consideration is randomly distributed and exhibits no serial autocorrelation in the favor of Alternate hypothesis that there is serial correlation in the data/residuals. It is applied to the residuals of the fitted ARMA model and not to the original time series. Since the alternate hypothesis say that there are correlations in the residuals, this means it represents lack of fit of the model.

**Simulating an ARMA(3,2) model **

```python
alphas = np.array([0.5, -0.4, 0.25])
betas = np.array([0.5, -0.3])

max_lag = 30
n = int(5000)
burn = 2000

ar = np.r_[1, -alphas]
ma = np.r_[1, betas]

arma32 = smt.arma_generate_sample(ar = ar, ma = ma, nsample = n, burnin = burn)
_ = tsplot(arma32, lags = max_lag)
```

**Fitting the model and selecting the best AIC **

```python
best_aic = np.inf
best_order = None
best_model = None

for i in range(5):
    for j in range(5):
        #print(i,j)
        try:
            tmp_model = smt.ARMA(arma32, order = (i,j)).fit(trend = 'nc',
                                                            method = 'mle')
            tmp_aic = tmp_model.aic
            if tmp_aic < best_aic:
                best_aic = tmp_aic
                best_order = (i,j)
                best_model = tmp_model
        except:
            continue

print('aic: %s, order: %s'%(best_aic, best_order))

```

    aic: 14211.29997664779, order: (2, 4)

```python
# Ljung-Box test

sms.diagnostic.acorr_ljungbox(best_model.resid, lags = [20], boxpierce=False)
```

    (array([5.34614925]), array([0.99953549]))

Here the p-value obtained from Ljung-Box test is ~0.34, which is way higher than the general threshold of 0.05. Hence we reject the alternative hypothesis that there is serial correlation in the residuals and accept the null hypothesis that the residuals are independently distributed, indicating a good fit of the data in the model.

**Plotting the residuals: **

```python
_ = tsplot(best_model.resid, lags=max_lag)
```

```python

```


**HW 10 - EC 414 - Prof. Kulis - Spring 2019**

Due Wednesday, May 1st, 2019.

## Problem 1: Universal approximation power of ReLU networks

A two layer NN with sigmoid activation function is a universal approximator, i.e: with sufficient hidden units, it can approximate any real function with desired accuracy. In this problem we want to demonstrate universal approximation power of NNs using ReLU activation units.

**Q1.1** Show that by composing only 2 hidden units in a ReLU network $-\sum_{i=1}^2a_i\ max(0,b_ix+c_i) -$ we can build an approximation to the step function $1[x>0]$.  Write some code to showcase this.

**Q1.2** Write some code to show that by composing 4 hidden units in a ReLU network, we can build an approximation to the unit impulse function of duration $\delta$

\begin{equation}
u_\delta(x) = 1[0\leq x\leq \delta]
\end{equation}

**Q1.3** Using your approximator for the unit impulse function in Q1.2, write code to draw the approximator for different duration values $\delta = [1,0.5,0.1]$.

```python
import numpy as np
import matplotlib.pyplot as plt

# step function
domain = np.arange(-4.0, 4.0, 0.1)
r1 = np.maximum(0, domain)
r2 = np.maximum(0, domain - 0.1)
step_function_approx = 10*(r1 - r2)
plt.plot(domain, step_function_approx)
```

```python
def plot_pluse_func_approx(delta, label = ""):
    im_r1 = np.maximum(0, domain)
    im_r2 = np.maximum(0, domain - 0.1)
    im_r3 = np.maximum(0, domain - delta)
    im_r4 = np.maximum(0, domain - (delta + 0.1))
    im = 10*(im_r1 - im_r2 - (im_r3 - im_r4))
    plt.plot(domain, im, label = label)
plot_pluse_func_approx(1)
```

```python
plot_pluse_func_approx(1, "$\delta$ = 1")
plot_pluse_func_approx(0.5, "$\delta$ = 0.5")
plot_pluse_func_approx(0.1, "$\delta$ = 0.1")
plt.legend(loc = 'right', bbox_to_anchor=(1.3, 0.5))
```

## Problem 2: Backprop in a simple MLP
Here, we ask you to derive all the steps of the backpropagation algorithm for a simple classification network. Consider a fully-connected neural network, also known as a multi-layer perceptron (MLP), with a single hidden layer and a one-node output layer. The hidden and output nodes use an elementwise sigmoid activation function and the loss layer uses cross-entropy loss:

$f(z)=\frac{1}{1+exp(-z))}$

$L(\hat{y},y)=-yln(\hat{y}) - (1-y)ln(1-\hat{y})$

The computation graph for an example network is shown below. Note that it has an equal number of nodes in the input and hidden layer (3 each), but, in general, they need not be equal. Also, to make the application of backprop easier, we show the computation graph which shows the dot product and activation functions as their own nodes, rather than the usual graph showing a single node for both.

The backpropagation algorithm for an MLP is displayed below. For simplicity, we will assume no regularization on the weights, so you can ignore the terms involving $\Omega$. The forward step is:

and the backward step is:

Write down each step of the backward pass explicitly for all layers, i.e. for the loss and $k=2,1$, compute all gradients above, expressing them as a function of variables $x, y, h, W, b$. Hint: you should substitute the updated values for the gradient $g$ in each step and simplify as much as possible.  Specifically, compute the following (we have replaced the superscript notation $u^{(i)}$ with $u^i$):

**Q2.1**: $\nabla_{\hat{y}}L(\hat{y},y)$

= $\nabla_{\hat{y}}(-yln(\hat{y}) - (1-y)ln(1-\hat{y}))$

= $\frac{1-y}{1-\hat{y}} - \frac{y}{\hat{y}}$

**Q2.2**: $\nabla_{a^2}J$

= $g \odot f^{'}(a^{2})$

= $g \odot (\sigma(a^2)(1-\sigma(a^2)))$

= $(\frac{1-y}{1-\hat{y}} - \frac{y}{\hat{y}}) \odot (\sigma(b^2 + W^2\sigma(b^1 + W^1x))(1-\sigma(b^2 + W^2\sigma(b^1 + W^1x)))) $

= $(\frac{1-y}{1-\hat{y}} - \frac{y}{\hat{y}})(\sigma(b^2 + W^2\sigma(b^1 + W^1x))(1-\sigma(b^2 + W^2\sigma(b^1 + W^1x)))) $

**Q2.3**: $\nabla_{b^2}J$

= $(\frac{1-y}{1-\hat{y}} - \frac{y}{\hat{y}})(\sigma(b^2 + W^2\sigma(b^1 + W^1x))(1-\sigma(b^2 + W^2\sigma(b^1 + W^1x)))) $

**Q2.4**: $\nabla_{W^2}J$
Hint: this should be a vector, since $W^2$ is a vector.

= $gh^{1T}$

= $gf(a^1)$

= $g\sigma(a^1)^T$

= $(\frac{1-y}{1-\hat{y}} - \frac{y}{\hat{y}})(\sigma(b^2 + W^2\sigma(b^1 + W^1x))(1-\sigma(b^2 + W^2\sigma(b^1 + W^1x))))\sigma(b^1 + W^1x)$

**Q2.5**: $\nabla_{h^1}J$

= $W^{2T}g$

$ W^{2T}(\frac{1-y}{1-\hat{y}} - \frac{y}{\hat{y}})(\sigma(b^2 + W^2\sigma(b^1 + W^1x))(1-\sigma(b^2 + W^2\sigma(b^1 + W^1x)))) $

**Q2.6**: $\nabla_{b^1}J$, $\nabla_{W^1}J$

$\nabla_{b^1}J$

$= g \odot f^{'}(a^1)$

$= (W^{2T}(\frac{1-y}{1-\hat{y}} - \frac{y}{\hat{y}})(\sigma(b^2 + W^2\sigma(b^1 + W^1x))(1-\sigma(b^2 + W^2\sigma(b^1 + W^1x))))) \odot (\sigma(b^1 + W^1x)(1-\sigma(b^1 + W^1x)))$

$\nabla_{W^1}J$

$= gh^{0T}$

$= ((W^{2T}(\frac{1-y}{1-\hat{y}} - \frac{y}{\hat{y}})(\sigma(b^2 + W^2\sigma(b^1 + W^1x))(1-\sigma(b^2 + W^2\sigma(b^1 + W^1x))))) \odot (\sigma(b^1 + W^1x)(1-\sigma(b^1 + W^1x))))x^T$

**Q2.7** Briefly, explain how would the computational speed of backpropagation be affected if it did not include a forward pass?

If there is no forward pass, then we will need to explecity use the formula above to calculate the gradient in each step in backpropagation that will slow down the process.

homework/hw10/HW 10 - EC 414 - Spring 2019.ipynb


# Support Vector Machine

A *Support Vector machine* (SVM) is a very powerful and versatile supervised machine learning model, capable of performing linear or nonlinear classification and regression tasks. To separate the two classes of data points, there are many possible hyperplanes that could be chosen. The key idea behind SVM is that the algorithm finds an optimal hyperplane that has the **maximum margin**, i.e the maximum distance between data points of both classes.

This fundamental idea is explained with the plots above. The plot on the left shows many hyperplanes that can separate the circles from the squares. These hyperplanes work perfectly on this training dataset, but their decision boundaries come so close to the instances that they will probably not perform as well on new instances.  However, the right plot shows the optimal hyperplane that maximize the margin between these two classes. Maximizing the margin distance provides some reinforcement so that future data points can be classified with more confidence.

In this lab assignment, you will learn to use and fine-tune linear SVM, the Polynomial kernel SVM and the RBF kernel SVM, and understand their pros and cons by comparing their prediction performance on a synthetic dataset.

```python
# import packages
%matplotlib inline
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
import seaborn as sns
import pandas as pd
from sklearn.datasets import make_moons
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LogisticRegression
from sklearn.svm import LinearSVC
from sklearn.svm import LinearSVR
from sklearn.svm import SVC
from sklearn.svm import SVR
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import GridSearchCV

# make this notebook's output stable across runs
np.random.seed(0)
```

```python
# helper functions used in this lab
def plot_decision_boundary(clf, X, y, axes=[-1.5, 2.5, -1, 1.5], alpha=0.5, contour=True):
    """
    Plot the decision boundary of a learnt classifier
    """
    x1s = np.linspace(axes[0], axes[1], 100)
    x2s = np.linspace(axes[2], axes[3], 100)
    x1, x2 = np.meshgrid(x1s, x2s)
    X_new = np.c_[x1.ravel(), x2.ravel()]
    y_pred = clf.predict(X_new).reshape(x1.shape)
    custom_cmap = ListedColormap(['#fafab0','#9898ff','#a0faa0'])
    plt.contourf(x1, x2, y_pred, alpha=0.3, cmap=custom_cmap)
    if contour:
        custom_cmap2 = ListedColormap(['#7d7d58','#4c4c7f','#507d50'])
        plt.contour(x1, x2, y_pred, cmap=custom_cmap2, alpha=1)
    plt.plot(X[:, 0][y==0], X[:, 1][y==0], "yo", alpha=alpha)
    plt.plot(X[:, 0][y==1], X[:, 1][y==1], "bs", alpha=alpha)
    plt.axis(axes)
    plt.xlabel(r"$x_1$", fontsize=18)
    plt.ylabel(r"$x_2$", fontsize=18, rotation=0)

def evaluate_model(model, X, y):
    # make prediction and evaluate the model performance on test data
    z = model.predict(X)
    z_prob = model.predict_proba(X)[:,1]

    print("model accuracy: {}".format(accuracy_score(y, z)))
    print("model ROC AUC: {}".format(roc_auc_score(y, z_prob)))
```

### Synthetic Moon dataset
In this lab, we will classify a moon shaped synthetic dataset with two features (_x1_ and _x2_) and two classes (0 or 1) with some added noise. We use this synthetic dataset because it's easier to help demonstrate the pros and cons of different SVM models covered in this lab.

```python
# load the moon train and test data from CSV files
train = pd.read_csv("moon_train.csv")
test = pd.read_csv("moon_test.csv")

train_x = train.iloc[:,0:2]
train_y = train.iloc[:,2]

test_x = test.iloc[:,0:2]
test_y = test.iloc[:,2]

print("Number of train data: {}".format(len(train_y)))
print("Number of test data: {}".format(len(test_y)))
```

```python
# plot the train dataset
plt.scatter(train_x.x1, train_x.x2, s=40, c=train_y, cmap=plt.cm.Spectral)
plt.title('Scatter plot of the train dataset')
plt.xlabel('x1')
plt.ylabel('x2')
```

```python
# plot the test dataset
plt.scatter(test_x.x1, test_x.x2, s=40, c=test_y, cmap=plt.cm.Spectral)
plt.title('Scatter plot of the test dataset')
plt.xlabel('x1')
plt.ylabel('x2')
```

## Linear SVM

**Coding Part 1: Learn to use and fine-tune the Linear Support Vector Machine.**

### Pipeline

Unlike tree-based models (e.g. CART, Random Forest and Gradient Boosted Trees), SVMs are sensitive to the feature scales. It is best practice to apply feature scaling with [StandardScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) function in scikit-learn.

[Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) in scikit-learn allows us to sequentially apply a list of transformations and a final estimator. For example, in the following code we first apply the feature scaling before feeding the data to train a linear SVM model. Note that you need to set the argument _probability_ to be True so that the SVC model outputs probabilities as well as the predictions.

```python
# define the pipeline
linear_svc_pipe = Pipeline(
    (
        ("scaler", StandardScaler()),
        ("csv", SVC(kernel="linear", probability=True, random_state=0))
    )
)
linear_svc_pipe.fit(train_x, train_y)
```

```python
# plot the decision boundary
plot_decision_boundary(linear_svc_pipe, train_x.values, train_y.values)
```

As we can see from the plot above, a linear SVM can produce only a linear hyperplane as the decision boundary and is not capable of modeling this moon-shaped dataset and generating the decision boundaries close to the ground truth.

```python
# make prediction and evaluate the model performance on test data
evaluate_model(linear_svc_pipe, test_x, test_y)
```

#### Penalty parameter C

SVM's objective function consists of two terms: the first term measures how large the margin is (we want to maximize the margin, which corresponds to minimizing $W^T W$ in the follow equation), and the second term measures the violations of the hyperplane ($\sum_i^m \xi^i$). This is where the hyper-parameter $C$ comes in. It is used to tradeoff between these two objectives.

\begin{align}
\min_{W, b} \frac{1}{2}W^T W & + C \sum_i^m \xi^i \\
S.T. y^i (W^T X^i + b) & \geqslant 1 - \xi^i \\
\xi^i & \geqslant 0
\end{align}

```python
# TODO: Use grid search with cross-validation on penalty parameter C to fine-tune the SVC model.
#       Note that you need to set probability to be True so that the trained model outputs both predictions as well as probabilities.
parameters = {
    "C": [0.01, 0.1, 1, 10, 100]
}
linear_svc_grid = # TODO: Use grid search with cross-validation on penalty parameter C to fine-tune the SVC model.

# define the pipeline
linear_svc_grid_pipe = # TODO: build the pipeline for training the model and fit the model to the training dataset.

# summarize the results of the grid search
print("The best score is {}".format(linear_svc_grid.best_score_))
print("The best hyper parameter setting is {}".format(linear_svc_grid.best_params_))
```

```python
# plot the decision boundary
plot_decision_boundary(linear_svc_grid_pipe, train_x.values, train_y.values)
```

After fine-tuning the model, we should see the model performance improve slightly as compared to the SVC with the default setting.

```python
evaluate_model(linear_svc_grid_pipe, test_x, test_y)
```

---

## Polynomial Kernel SVM

**Coding Part 2: Understand the limitation of linear SVM and learn how to use kernel trick to extend linear SVM to handle non-linear scenarios.**

### Polynomial Features

Although linear SVM classifiers are efficient and work surprisingly well in many cases, many datasets are not linearly separable. One approach to handling nonlinear datasets is to add more non-linear features, such as polynomial features, and hope the dataset is linear separable in the higher dimensional space with the help of non-linear features.

In the following example, we apply polynomial functions with degree of 3 to the original features before feeding the data to the linear SVC.

```python
poly_feature_svm_pipe = Pipeline(
    (
        ("poly_features", PolynomialFeatures(degree=3)),
        ("scaler", StandardScaler()),
        ("svm_clf", SVC(kernel="linear", probability=True))
    )
)
poly_feature_svm_pipe.fit(train_x, train_y)
```

```python
# plot the decision boundary
plot_decision_boundary(poly_feature_svm_pipe, train_x.values, train_y.values)
```

```python
# make prediction and evaluate the model performance on test data
evaluate_model(poly_feature_svm_pipe, test_x, test_y)
```

### Polynomial SVM
Instead of explicitly applying the polynomial functions to each data point, kernel trick allows us to get the same result as if you added many polynomial features (even with very high-degree polynomials) without actually having to add them to each data point. So the number of features won't grow combinatorially.

To train the SVC with polynomial kernel, we need to specify argument _kernel_ to be "poly". Like linear SVC, we need to fine tune the penalty parameter *C* to tradeoff minimizing margin and minimizing violations. Also, we need to specify the degree of the polynomial functions as well as a hyper-parameter _coef0_ which controls how much the model is influenced by high-degree polynomials vs. low-degree polynomials. Please refer to the [SVC](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html) API doc for more details.

```python
# TODO: Use grid search with cross-validation on penalty parameter C, polynomial degree, and coef0 to
#       fine-tune the model.
parameters = {
    "C": [0.1, 1, 10],
    "degree": [3, 5],
    "coef0": [0.5, 1.0]
}
poly_svc_grid = # TODO: Use grid search with cross-validation on hyper-parameters to fine-tune the SVC model.

# define the pipeline
poly_svc_grid_pipe = # TODO: build the pipeline for training the model and fit the model to the training dataset.

# summarize the results of the grid search
print("The best score is {}".format(poly_svc_grid.best_score_))
print("The best hyper parameter setting is {}".format(poly_svc_grid.best_params_))
```

```python
# plot the decision boundary
plot_decision_boundary(poly_svc_grid_pipe, train_x.values, train_y.values)
```

```python
# make prediction and evaluate the model performance on test data
evaluate_model(poly_svc_grid_pipe, test_x, test_y)
```

Both the polynomial SVC and adding polynomial features in a linear SVC result in very similar performance and significantly improve the model performance on the test dataset. The decision boundaries are no longer linear in the original feature space and better capture the ground truth of this moon-shaped dataset.

## Gaussian Radial Basis Function (RBF) SVM
**Coding Part 3: Learn to use Gaussian Radial Basis Function (RBF) SVM.**

Another very popular kernel used in SVM is called the Gaussian Radial Basis Function (RBF) kernel. The idea of RBF kernel is to allow each training instance to resemble a particular landmark. The RBF kernel on two samples x and x', represented as feature vectors in some input space, is defined as

\begin{align}
K(x, x') = \exp(-\gamma||x - x'||^2)
\end{align}

RBF Kernel has a hyper-parameter $\gamma$, which controls the shape of the Gaussian function. The parameter $\gamma$ is the inverse of the standard deviation of the RBF kernel (Gaussian function), which is used as similarity measure between two points. Intuitively, a small gamma value define a Gaussian function with a large variance. In this case, two points can be considered similar even if are far from each other. In the other hand, a large gamma value means define a Gaussian function with a small variance and in this case, two points are considered similar just if they are close to each other.

To demonstrate this effect of hyper-parameter $\gamma$, we train two separate SVCs, one with a large value of $\gamma$ and one with a small value of $\gamma$.

```python
# train RBF SVC with a small gamma and underfit the training set
rbf_svc_pipe = Pipeline(
    (
        ("scaler", StandardScaler()),
        ("svc", SVC(kernel="rbf", gamma=0.01, probability=True))
    )
)
rbf_svc_pipe.fit(train_x, train_y)
plot_decision_boundary(rbf_svc_pipe, train_x.values, train_y.values)
```

```python
# train RBF SVC with a large gamma and overfit the training set
rbf_svc_pipe = Pipeline(
    (
        ("scaler", StandardScaler()),
        ("svc", SVC(kernel="rbf", gamma=100, probability=True))
    )
)
rbf_svc_pipe.fit(train_x, train_y)
plot_decision_boundary(rbf_svc_pipe, train_x.values, train_y.values)
```

As we can see from the previous two plots, $\gamma$ acts like a regularization hyper-parameter. A small value of $\gamma$ leads to model underfitting (the upper figure) and a large value of $\gamma$ leads to model overfitting (the lower figure). Like other hyper-parameters, the value of $\gamma$ needs to be fine-tuned with a validation set.

```python
# TODO: use grid search with cross-validation on penalty parameter C to fine-tune the model
parameters = {
    "C": [0.01, 0.1, 1, 10, 100],
    "gamma": [0.1, 1, 10, 100]
}
rbf_svc_grid = # TODO: Use grid search with cross-validation on hyper-parameters to fine-tune the SVC model.

# define the pipeline
rbf_svc_grid_pipe = # TODO: build the pipeline for training the model and fit the model to the training dataset.

# summarize the results of the grid search
print("The best score is {}".format(rbf_svc_grid.best_score_))
print("The best hyper parameter setting is {}".format(rbf_svc_grid.best_params_))
```

```python
# plot the decision boundary
plot_decision_boundary(rbf_svc_grid_pipe, train_x.values, train_y.values)
```

```python
# make prediction and evaluate the model performance on test data
evaluate_model(rbf_svc_grid_pipe, test_x, test_y)
```

### End of ML 310 Lab Assignment 4
---


```python
import matplotlib.pyplot as plt

%matplotlib inline

import numpy as np
from scipy import linalg
from scipy import optimize
import sympy as sm
from sympy import *
```

# Model description: The Keyniesian cross and the IS-LM model

## Keyniesian cross
The Keynasian cross is a part of Keynes general theory. It proposed that an economy’s total income was, in the short run, determined largely by the spending plans of households, businesses, and government. The more people want to spend, the more goods and services firms can sell. Keynes believed that the problem during recessions and depressions was inadequate spending. The Keynesian cross is an attempt to model this insight.

**Structure of the model**

The planned expenditure is determined as:

$$ AD = C + I + G + NX $$

To this equation we add the consumption function:

$$ C = a + b(Y-T),    a>0,    0<b<1 $$

This equation states that concumption depends on disposable income, and is a linear function. b is the marginal consumption rate. Further more we take planned investment as exogenously fixed, and negatively related to the rate of interest:

$$ I = \bar{I} - di $$
Because of the free capital movement, the real domestic interest rate equals the real foreign interest rate, $r^*$

$$ r = \bar{r^*} $$

Furter more we assume that governments puchases and taxes are fixed

$$ T = \bar{T} $$

$$ G = \bar{G} $$

Combining these eqations we get

$$ AD = a + b(Y- \bar{T}) + \bar{I} - di + \bar{G} + NX $$

This equation shows that planned expenditure is a function of income Y, the
level of planned investment I , the fiscal policy variables G and T and the net export NX.
The Keynesian cross is in equilibrium when actual expenditure equals planned expenditure

$$ Y = AD $$

### Keynesian cross equilibrium analysis

```python
# we define the symbols
Y = sm.symbols('Y')
C = sm.symbols('C')
PE = sm.symbols('PE')
T = sm.symbols('T')
I = sm.symbols('I')
G = sm.symbols('G')
NX = sm.symbols('NX')
d = sm.symbols('d')
i = sm.symbols('i')
a = sm.symbols('a')
b = sm.symbols('b')
```

```python
# We now set Y=AD to solve for Y
eq_AD = sm.Eq(Y, a + b*(Y-T) + I-(d*i) + G + NX)
eq = sm.solve(eq_AD, Y)[0]

yeq = sm.factor(eq)
print('Y =')
yeq
```

    Y =

$\displaystyle - \frac{G + I + NX - T b + a - d i}{b - 1}$

We have now found the equilibrium for Y. We now want to plot the keynesian cross

```python
# Define the values for our parameters
T = 30
I = 40
G = 30
NX = 10
a = 30
b = 0.3
d = 5
i = 5

# The data for production and AD is plotted
Y_arrey = np.linspace(0,300)
AD_arrey = (a + b * (Y_arrey - T) + I - d*i + G + NX)
degree = Y_arrey

# The figure
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(1,1,1)

ax.plot(Y_arrey, degree, label="45-degree line", color='lightblue',linewidth=3)
ax.plot(Y_arrey, AD_arrey, label="AD=C+I+G+NX", color='darkorange',linewidth=3)

ax.set_xlabel("Y")
ax.set_ylabel("AD")
ax.legend(loc="upper left")

ax.grid()
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['bottom'].set_visible(False)
ax.spines['left'].set_visible(False)
```

```python
Y = -(G + I + NX - T*b + a - d*i)/(b-1)
print ('The equilibrium for the Keynesian cross is')
Y
```

    The equilibrium for the Keynesian cross is

    108.57142857142858

### An Increase in Government Purchases in the Keynesian Cross
We now want to examine how goverment purchase effects the  equilibrium of the economy, because higher government expenditure results in higher planned ecpenditure. We'll therefore examine hov big of a change a movement in G will make in Y.

```python
del G
G = sm.symbols('G')
diff_Y = sm.diff(yeq, G)
print('Y will change by')
diff_Y
```

    Y will change by

$\displaystyle - \frac{1}{b - 1}$

Where b is the marginal propensity to consume.

```python
#the increase is set:
G_change = -(1/(b-1))
print('This means when G rises by 1 amount, Y will rise by')
G_change
```

    This means when G rises by 1 amount, Y will rise by

    1.4285714285714286

We now want to compare our old equilibrium with our new equilibrium  (higher public expenditure)

```python
# New G:
G = 30
#Public expenditure rises by amount 20
delta_G = 20
G_new = G + delta_G

# The data for production and AD is plotted
Y_arrey = np.linspace(0,300)
AD_arrey_new = (a + b * (Y_arrey - T) + (I - d*i) + G_new + NX)
degree = Y_arrey

# The figure
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(1,1,1)

ax.plot(Y_arrey, degree, label="45-degree line", color='lightblue',linewidth=3)
ax.plot(Y_arrey, AD_arrey, label="AD=C+I+G+NX", color='darkorange',linewidth=3)
ax.plot(Y_arrey, AD_arrey_new, label="AD_2=C+I+G'+NX", color='red',linewidth=3)

ax.set_xlabel("Y")
ax.set_ylabel("AD")
ax.legend(loc="upper left")

ax.grid()
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['bottom'].set_visible(False)
ax.spines['left'].set_visible(False)
```

```python
Y_G = -(G_new + I + NX - T*b + a - d*i)/(b-1)
print('The equilibrium have risen to')
Y_G
```

    The equilibrium have risen to

    137.14285714285714

```python
change_G = Y_G - Y
print('The equlibrium have changed by')
change_G
```

    The equlibrium have changed by

    28.571428571428555

### A Decrease in Taxes in the Keynesian Cross
We now want to examine how a decrease in taxes in the Keynesian cross will affect the equilibrium of the economy, because lower taxes results in higher planned ecpenditure. We'll therefore examine hov big of a change a movement in T will make in Y.

```python
del T
T = sm.symbols('T')
diff_Y = sm.diff(yeq, T)
print('Y will change by')
diff_Y
```

    Y will change by

$\displaystyle \frac{b}{b - 1}$

Higher taxes has a negative effect on Y, because 0<b<1

```python
#the increase is set:
T_change = -(b/(b-1))
print('This means when T falls by 1 amount, Y will rise by')
T_change
```

    This means when T falls by 1 amount, Y will rise by

    0.4285714285714286

```python
# New T:
T = 30
#Taxs falls by amount 20
delta_T = -20
T_new = T + delta_T

# The data for production and AD is plotted
Y_arrey = np.linspace(0,300)
AD_arrey_new_2 = (a + b*(Y_arrey - T_new) + (I - d*i) + G + NX)
degree = Y_arrey

# The figure
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(1,1,1)

ax.plot(Y_arrey, degree, label="45-degree line", color='lightblue',linewidth=3)
ax.plot(Y_arrey, AD_arrey, label="AD=C(Y-T)+I+G+NX", color='darkorange',linewidth=3)
ax.plot(Y_arrey, AD_arrey_new_2, label="AD_2=C*(Y-T')+I+G + NX", color='red',linewidth=3)

ax.set_xlabel("Y")
ax.set_ylabel("PE")
ax.legend(loc="upper left")

ax.grid()
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['bottom'].set_visible(False)
ax.spines['left'].set_visible(False)
```

```python
Y = -(G + I + NX - T_new*b + a - d*i)/(b-1)
print('The equilibrium have risen to')
Y
```

    The equilibrium have risen to

    117.14285714285715

It's clear to see that a rise in public expenditure has a bigger effect on equilibrium, since 117.1 < 137.1

## The IS-curve
The Keynesian cross is only a stepping-stone on our path to the IS–curve, which explains the economy’s aggregate demand curve. The Keynesian cross is useful because it shows how the spending plans of households, firms, and the government determine the economy’s income. We'll now derive the IS-curve from the AD-curve

$$Y-bY=a+b\bar{T}+\bar{I}-di+\bar{G}+NX$$

$$Y(1-b)=a+b\bar{T}+\bar{I}-di+\bar{G}+NX$$

$$Y=\frac{1}{1-b}(a+\bar{I}+\bar{G}+NX)-\frac{1}{1-b}(b\bar{T}+di)$$

Our function for IS depends on the variables from the Keynesian cross. We can therefore define our function.

```python
# We'll now define our function for the IS-curve
del i
i = sm.symbols('i')
Y_IS = (1/(1-b))*(a+I+G+NX)-(1/(1-b))*(b*T+d*i)
print('The function for the IS-curve =')
Y_IS
```

    The function for the IS-curve =

$\displaystyle 144.285714285714 - 7.14285714285714 i$

## The LM-curve
Having derived algebraically equation for IS curve we now turn to the derivation of equation for LM curve. It will be recalled that LM curve is a curve that shows combinations of interest rates and levels of income at which money market is in equilibrium, that is, at which demand for money equals supply of money. Let us assume that money demand function is linear. Then:

$$ L(Y,i)=kY-hik, h > 0 $$

Parameter k represents how much demand for real money balances increases when level of income rises. Parameter h represents how much demand for real money balances decreases when rate of interest rises.
The equilibrium in the money market is established where demand for real money balances equals supply of real money balances and is given by

$$ M/P = kY – hi $$

Solving equation above for interest rate, we'll get

$$i = 1/h (kY – M/P) $$

The above equation describes the equation for the LM-curve. To be precise it gives us the equilibrium interest rate for any given value of level of income (Y) and real money balances. For the money market to be in equilibrium we have that:

$$M_d=M_s$$

Where $M_d$ is the demand for money, and $M_s$ is the rate of interest. We have that:

$$M_d=Y–2i$$

$$M_s=20$$

The solution is:

$$Y=20+2i$$

```python
# We'll now define our function for the LM-curve
Y_LM = 20 + 2*i
print('The function for the LM-curve =')
Y_LM
```

    The function for the LM-curve =

$\displaystyle 2 i + 20$

## The IS-LM model
We'll now put the IS- and LM curve together. The IS-LM model shows the relationship between interest rates and output.

```python
# The functions
I_arrey = np.linspace(0,25)
IS_arrey = 144-7*I_arrey
LM_arrey = 20+2*I_arrey

# The figure
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(1,1,1)

ax.plot(I_arrey, IS_arrey, label="IS-curve", color='darkorange',linewidth=3)
ax.plot(I_arrey, LM_arrey, label="LM-curve", color='blue',linewidth=3)

ax.set_xlabel("Y")
ax.set_ylabel("I")
ax.legend(loc="upper left")

ax.grid()
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['bottom'].set_visible(False)
ax.spines['left'].set_visible(False)
```

The equilibrium is 13.7. This means that the economy are in equiibrium when output is equal to 13.7.

## An Increase in Government Purchases in the IS-LM model
We'll now examine how a change in public expenditure will effect the IS-LM curve. We'll change it with the amount 20 - the same for the keynisian cross

```python
# We'll now define our new function for the IS-curve when we change the public expenditure
del i
i = sm.symbols('i')
Y_IS_G = (1/(1-b))*(a+I+G_new+NX)-(1/(1-b))*(b*T+d*i)
print('The function for the new IS-curve =')
Y_IS_G
```

    The function for the new IS-curve =

$\displaystyle 172.857142857143 - 7.14285714285714 i$

```python
# The functions
I_arrey = np.linspace(0,25)
IS_arrey_G = 172-7*I_arrey
LM_arrey_G = 20+2*I_arrey

# The figure
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(1,1,1)

ax.plot(I_arrey, IS_arrey_G, label="IS_2-curve", color='darkorange',linewidth=3)
ax.plot(I_arrey, LM_arrey_G, label="LM-curve", color='blue',linewidth=3)
ax.plot(I_arrey, IS_arrey, label="IS-curve", color='red',linewidth=3)

ax.set_xlabel("Y")
ax.set_ylabel("I")
ax.legend(loc="upper left")

ax.grid()
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['bottom'].set_visible(False)
ax.spines['left'].set_visible(False)
```

The new equilibrium is 16.8. We can therefore conclude that a rise in public expenditure rises output and the interest rate.

# Extension
We will now analyze how much a change in public expenditure makes on output in Keynes cross, and the IS-LM model, when we have respectively a lump sum tax, and taxes levied as proportion of income.
For lump sum taxes we assume that the consumption function is defined as:

$$C=a+b(Y-T+R)$$

Where R is the lump sum. If we assume proportionate income tax, then consumption is defined as:

$$C=a+b(Y-tY)$$

## Lump sum

### Lump Sum equilibrium analysis

```python
del a
del b
del Y
del T
del I
del d
del G
del NX
a = sm.symbols('a')
b = sm.symbols('b')
Y = sm.symbols('Y')
T = sm.symbols('T')
I = sm.symbols('I')
d = sm.symbols('d')
G = sm.symbols('G')
NX = sm.symbols('NX')
R = sm.symbols('R')

#We now set Y=AD to solve for Y
eq_AD_Lump = sm.Eq(Y, a + b*(Y-T+R) + I-(d*i) + G + NX)
eq_Lump = sm.solve(eq_AD_Lump, Y)[0]

yeq_Lump = sm.factor(eq_Lump)
print('Y =')
yeq_Lump
```

    Y =

$\displaystyle - \frac{G + I + NX + R b - T b + a - d i}{b - 1}$

We have now found the equilibrium for Y when implementing a lump sum tax. We want solve for how much a public expentuture makes in Y

```python
diff_Y_Lump = sm.diff(yeq_Lump, G)
print('Y will change by')
diff_Y_Lump
```

    Y will change by

$\displaystyle - \frac{1}{b - 1}$

This is exactly the same the change as with no lump sum tax. We'll therefore look at the change in Y for a proportionel income tax.

## Proportionate income tax

### Proportionate income tax equilirium analysis

```python
del a
del b
del Y
del T
del I
del d
del G
del NX
a = sm.symbols('a')
b = sm.symbols('b')
Y = sm.symbols('Y')
T = sm.symbols('T')
I = sm.symbols('I')
d = sm.symbols('d')
G = sm.symbols('G')
NX = sm.symbols('NX')
R = sm.symbols('R')

#We now set Y=PE to solve for Y
t = sm.symbols('t')
eq_AD_Prop = sm.Eq(Y, a + b*(Y-t*Y) + I-(d*i) + G + NX)
eq_Prop = sm.solve(eq_AD_Prop, Y)[0]

yeq_Prop = sm.factor(eq_Prop)
print('Y =')
yeq_Prop
```

    Y =

$\displaystyle \frac{G + I + NX + a - d i}{b t - b + 1}$

We have now found the equilibrium for Y when implementing a proportionate income tax. We now want solve for how big a change public expentuture makes in Y.

```python
diff_Y_Prop = sm.diff(yeq_Prop, G)
print('Y will change by')
diff_Y_Prop
```

    Y will change by

$\displaystyle \frac{1}{b t - b + 1}$

We can see that the new slope is less steep. This means that the multiplicator effect is smaller.

### An Increase in Government Purchases in the Keynesian Cross with proportionate income tax

```python
# Define the values for our parameters
T = 30
I = 40
G = 30
NX = 10
a = 30
b = 0.3
d = 5
i = 5
t = 0.3

Y_arrey = np.linspace(0,300)
AD_arrey_Prop = (a + b*(Y_arrey - t*Y_arrey) + I - d*i + G + NX)
AD_arrey_Prop_new = (a + b*(Y_arrey - t*Y_arrey) + I - d*i + G_new + NX)
degree = Y_arrey

# The figure
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(1,1,1)

ax.plot(Y_arrey, degree, label="45-degree line", color='lightblue',linewidth=3)
ax.plot(Y_arrey, AD_arrey_Prop, label="AD=C(Y-tY)+I+G+NX", color='darkorange',linewidth=3)
ax.plot(Y_arrey, AD_arrey_Prop_new, label="AD_2=C*(Y-tY)+I+G' + NX", color='red',linewidth=3)

ax.set_xlabel("Y")
ax.set_ylabel("PE")
ax.legend(loc="upper left")

ax.grid()
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['bottom'].set_visible(False)
ax.spines['left'].set_visible(False)
```

```python
Y_Prop = (G + I + NX + a - d*i)/(b*t-b+1)
Y_Prop_new = (G_new + I + NX + a - d*i)/(b*t-b+1)
print('The old equilibrium for the economy with proportionate income tax, but without public expenditure was')
Y_Prop
```

    The old equilibrium for the economy with proportionate income tax, but without public expenditure was

    107.59493670886076

```python
print('The new equilibrium for the economy with proportionate income tax, and with public expenditure is')
Y_Prop_new
```

    The new equilibrium for the economy with proportionate income tax, and with public expenditure is

    132.91139240506328

```python
change_prop = Y_Prop_new - Y_Prop
print('The change is')
change_prop
```

    The change is

    25.316455696202524

Because of the dependent tax, the shift in the total production is smaller 25.3 < 28.57.

### An Increase in Government Purchases in the IS-LM model with proportionate income tax
We'll now find our new function for the IS-curve.

```python
# We'll now define our function for the IS-curve
del i
i = sm.symbols('i')
Y_IS = (1/(b*t-b+1))*(a+I+G+NX-d*i)
print('The function for the IS-curve =')
Y_IS
```

    The function for the IS-curve =

$\displaystyle 139.240506329114 - 6.32911392405063 i$

```python
# The functions
I_arrey = np.linspace(0,25)
IS_arrey_prop = 139-6*I_arrey
LM_arrey_prop = 20+2*I_arrey

# The figure
fig = plt.figure(figsize=(10,5))
ax = fig.add_subplot(1,1,1)

ax.plot(I_arrey, IS_arrey_prop, label="IS_prop-curve", color='darkorange',linewidth=3)
ax.plot(I_arrey, LM_arrey_prop, label="LM_prop-curve", color='blue',linewidth=3)

ax.set_xlabel("Y")
ax.set_ylabel("I")
ax.legend(loc="upper left")

ax.grid()
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.spines['bottom'].set_visible(False)
ax.spines['left'].set_visible(False)
```

The equlibrium is equal to 14.8. It's clear to see that a proportionate income tax have a smaller effect on the economy when changing the public expenditure, because 14.8 < 16.8

# Conclusion
We can therefore conclude that a rise in government purchase have a bigger effect on production than a fall in taxes. Furthermore we can conclude that implementation of a lump sum tax will not have any effect, when changing the public expenditure. However a proportionate income tax have an effect. By including a proportionate income tax, the economy gets more stable and shifts in exogenous variables have a smaller effect on the economy.


Inverse Kinematics

Interactive Demo below

Based on Marc Toussaint's robotics lecture.

```python
%matplotlib inline
import numpy as np
import numpy.linalg
import numpy.random
import pandas as pd
import matplotlib.pyplot as py
import threading
import IPython.display as disp
from ipywidgets import widgets
```

kinematics differ from dynamics in that they control the generalized coordinates $q$ directly, instead of generalized accelerations $\ddot{q}$.

Kinematic Chain

A simple model is the kinematic chain (subset of a kinematic tree) where each joint is attached to the previous joint.
In this example I'm using simple rotational joints with two DoFs: the angle and extension $q = (\alpha,\ l)$. This is formulated using affine transformations:

$\begin{align}
A(\alpha,\ l) &= R(\alpha)\ T(l) \\
\\
R(\alpha) &= \left\lgroup\matrix{
\cos(\alpha) & -\sin(\alpha) & 0 \\
\sin(\alpha) &  \cos(\alpha) & 0 \\
0 & 0 & 1}\right\rgroup \\
\\
T(l) &= \left\lgroup\matrix{
1 & 0 & l \\
0 & 1 & 0 \\
0 & 0 & 1}\right\rgroup
\end{align}$

This could be multiplied out but for readability in the code I'll keep them seperated.

The chain TODO:

```python
# Simple rotational joint
def linkTransformation(angle, length):
    R = np.matrix([
        [np.cos(angle), -np.sin(angle), 0],
        [np.sin(angle),  np.cos(angle), 0],
        [0,              0,             1],
    ])

    T = np.matrix([
        [1, 0, length],
        [0, 1, 0],
        [0, 0, 1],
    ])

    return R * T

# Returns a set of transformations for a chain of joints
# attached to one another.
def kinematicChain(angles, lenghts):
    assert(len(angles) == len(lenghts))
    A = np.matrix(np.eye(3))
    trafos = np.zeros((len(angles), 3, 3))
    for i in range(0, len(angles)):
        linkA = linkTransformation(angles[i], lenghts[i])
        A = np.dot(A, linkA)
        trafos[i, :, :] = A
    return trafos

def effectorPositions(transformations):
    effPositions = np.zeros((len(transformations) + 1, 2))
    for i in range(0, len(transformations)):
        A = transformations[i]
        effPositions[i + 1, :] =  np.dot(A, [0, 0, 1])[:-1]
    return effPositions
```

Derivation of a Gauss-Newton IK Solver

Linearization of the transformation function (first order Taylor):

$\begin{align}
\phi(q - q_0) &= y_0 + \frac{\partial \phi}{\partial q} \bigg\rvert_{q_0} (q - q_0) + \mathcal{O}((q-q_0)^2) \\
              &\approx y_0 + J\rvert_{q_0} (q - q_0)
\end{align}$

Here a Thikonov regularized iterative least squares solver is used.

$\begin{align}
q^* &= \underset{q}{\text{argmin}} \ \lVert \phi(q) - y^* \rVert^2_C + \lVert q_0 - q \rVert^2_W &  \\
&= \underset{q}{\text{argmin}} \ (\phi(q) - y^*)^\top C (\phi(q) - y^*) + (q_0 - q)^\top W (q_0 - q) & \text{(Tikhonov regularization)} \\
&\approx \underset{q}{\text{argmin}} \ (y_0 + J\rvert_{q_0} (q - q_0) - y^*)^\top C (y_0 + J\rvert_{q_0} (q - q_0) - y^*) + (q_0 - q)^\top W (q_0 - q) &
\end{align}$

$\begin{align}
q^* = \underset{q}{\text{argmin}} \ L(q) &\Rightarrow \frac{\partial \ L(q^*)}{\partial q} = 0 & \text{(gradient vanishes at minimum)}
\end{align}$

$\begin{align}
\frac{\partial \ L(q)}{\partial q} &= \frac{\partial}{\partial q}
[y_0 + J (q - q_0) - y^*]^\top \ C \ [y_0 + J (q - q_0) - y^*]
+ (q_0 - q)^\top W (q_0 - q) \\
%
&= \frac{\partial}{\partial q}
[J (q - q_0) + (y_0 - y^*)]^\top \ C \ [J (q - q_0) + (y_0 - y^*)]
+ (q_0 - q)^\top W (q_0 - q) \\
%
&= \frac{\partial}{\partial q}
(q - q_0)^\top J^\top \ C \ J (q - q_0)
+ 2 (y_0 - y^*)^\top \ C \ J (q - q_0)
+ (q_0 - q)^\top W (q_0 - q) \\
%
&= 2(q - q_0)^\top J^\top \ C \ J
+  2 (y_0 - y^*)^\top \ C \ J
+  2 (q_0 - q)^\top W = 0
\end{align}$

$\begin{align}
2(q - q_0) J^\top \ C \ J  2 (q_0 - q)^\top W
&= -2 (y_0 - y^*)^\top \ C \ J \\
%
(J^\top \ C^\top \ J + W^\top) (q - q_0)
&= J^\top C^\top (y_0 - y^*) \\
%
q &= q_0 + (J^\top \ C^\top \ J + W^\top)^{-1} J^\top C^\top (y_0 - y^*)
\end{align}$

But since the whole thing is based on only a linear approximation evaluated at $q_0$ the result is expected to be off the farther away the optimal solution $q^*$ is from $q_0$.
A solution to alleviate this problem is to compute the solution incrementally in small steps and reevaluate the Jacobian at each step. A.k.a the Gaussâ€“Newton algorithm

$q_{t+1} = q_t + \ ({J_t}^\top \ C^\top \ J_t + W^\top)^{-1} {J_t}^\top C^\top \epsilon \ (y_t - y^*)$

with some $\epsilon < 1$ and make sure to recompute $J_t$ and $y_t$ at each time step.

 $\rightarrow$ The great thing about this method is that only the link transformations and the Jacobian have to be specified!

(But even the Jacobian could be computed using symbolic differntiation or approximated using finite differences.)

A slitghty faster method

The bottleneck of the above iteration equation is the matrix inversion which is in $\mathcal{O}(n^3)$ whereas direct linear system solvers are in $\mathcal{O}(n^2)$.
Therefore solving the linear system for the increment $\delta q_t = q_{t+1} - q_t$ is faster than actually computing the inverse and can be formulated as follows:

$\underbrace{({J_t}^\top \ C^\top \ J_t + W^\top)}_{A} \underbrace{(q_{t+1} - q_t)}_{\delta q} = \underbrace{\ {J_t}^\top C^\top \epsilon \ (y_t - y^*)}_{b}$

$\Rightarrow$ solve $\ A \ \delta q_t = b \ $ then update $\ q_{t + 1} = q_t + \delta q_t \ \ \ \text{(see inverseKinematic() below)}$

```python
# TODO: make explicit that only for angles, not lengths
def Jacobian(q):
    J = np.zeros((2, len(q['angles'])))
    trafos = kinematicChain(q['angles'], q['lenghts'])
    eff = effectorPositions(trafos)
    for i in range(0, len(q['angles'])):
        d = eff[i + 1] - eff[i]
        J[:, i] = [-d[1], d[0]]
    return J

# Approximate the Jacobian directly from the forward kinematics using central differences
def Jacobian_finiteDifferences(q, h=1.0e-6):
    J = np.zeros((2, len(q['angles'])))
    for d in range(0, len(q['angles'])):
        qAngle = np.copy(q['angles'])

        qAngle[d] += h
        trafos = kinematicChain(qAngle, q['lenghts'])
        effLeft = effectorPositions(trafos)[-1]

        qAngle[d] -= 2.0 * h
        trafos = kinematicChain(qAngle, q['lenghts'])
        effRight = effectorPositions(trafos)[-1]

        J[:, d] = (effLeft - effRight) / (2.0 * h)
    return J
```

```python
def inverseKinematic(q, J, endeffectorGoal, cVal = 50.0, wVal = 1.0, epsilon=0.2, maxSpeed=0.25):
    # TODO: check dimensions
    # TODO: check invertibility
    trafos = kinematicChain(q['angles'], q['lenghts'])
    eff = effectorPositions(trafos)

    diff = (endeffectorGoal - eff[-1])
    dist = np.sqrt(np.dot(diff, diff))

    C = cVal * np.eye(2)
    W = wVal * np.eye(len(q['angles']))

    A = np.dot(np.dot(J.T, C.T), J) + W.T
    b = np.dot(J.T, np.dot(C.T, epsilon * (diff / dist) * min(maxSpeed, dist)))
    q['angles'] = q['angles'] + np.linalg.solve(A, b)

    return q
```

The Effects of $W$ and $C$

The Woodbury identity can be applied to make the effects of $W$ and $C$ more transparent:

$\begin{align}
q &= q_0 + (J^\top \ C^\top \ J + W^\top)^{-1} J^\top C^\top (y_0 - y^*) \\
  &= q_0 + W^{-1} J^\top (C^{-1} + J W^{-1} J^\top)^{-1} (y_0 - y^*)
\end{align}$

Choosing an invertible $C$ ensures the inverse can always be solved even if the target position is not even reachable.

As $C$ approaches $\infty \ I$ and $W$ approaches $I$ the solution becomes the exact (left) pseudo inverse.

 Interactive Demo: Inverse Kinematic

(Requires Jupyter Notebook)
The arm follows uses inverse kinematics to align the endeffector with the target position.
Use the mouse to specify the target position.
Use the sliders to try out the effect of the Tikhonov regularization parameters $C$ and $W$ as well as the iteration step size $\epsilon$ and see when this method becomes unstable.

```python
import IPython.display as disp
from ipywidgets import widgets
import json

js = '''

<!-- Interaction with the python code -->

<!-- CSS constants for node and links style -->
<style>
node {

}

link {

}
</style>

<!-- svg area and mouse hover event forward declaration -->

<svg id="canvas" onmousemove="svgMouseMove(evt)" width="800" height="600">
<rect id="bg" width="100%" height="100%" fill="#808080"/>
</svg>

'''

def toJson(data):
    return pd.DataFrame(data).to_json(orient='split')

class RobotState():
    def __init__(self, q):
        self.out = widgets.Output()
        self.q = q
        self.goal = [0.5, 0.5]
        self.cVal = 50.0
        self.wVal = 1.0
        self.epsVal = 0.25

    def mouse(self, parameters):
        with self.out:
            self.goal = np.asarray([parameters['xRelative'], parameters['yRelative']])
            return '{"x":%f,"y":%f}' % (self.goal [0], self.goal[1])

    def update(self, parameters):
        with self.out:
            if not self.goal is None:
                J = Jacobian(self.q)
                # alternatively the jacobian could be approximated
                #J = Jacobian_finiteDifferences(self.q)

                self.q = inverseKinematic(self.q, J, self.goal, self.cVal, self.wVal, self.epsVal)
            return toJson([])

    # computes and returns the current effector positions in JSON format
    def requestEffectors(self, parameters):
        with self.out:
            trafos = kinematicChain(self.q['angles'], self.q['lenghts'])
            eff = effectorPositions(trafos)
            return toJson(eff)

linkCount = 7
linkLength = 0.15
robot = RobotState(q={'angles': np.ones(linkCount) / linkCount, 'lenghts': np.ones(linkCount) * linkLength})

def cValSlider(C_Value):
    robot.cVal = C_Value
widgets.interact(cValSlider, C_Value=widgets.FloatSlider(min=0.01, max=100.0, value=robot.cVal))

def wValSlider(W_Value):
    robot.wVal = W_Value
widgets.interact(wValSlider, W_Value=widgets.FloatSlider(min=0.01, max=10.0, value=robot.wVal))

def epsValSlider(epsilon_Value):
    robot.epsVal = epsilon_Value
widgets.interact(epsValSlider, epsilon_Value=widgets.FloatSlider(min=0.01, max=1.0, value=robot.epsVal))

disp.display(robot.out)
disp.display(disp.HTML(js))
# set the initial goal
disp.display(disp.HTML('' % (robot.goal[0], robot.goal[1])))
```

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type interactive.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

Failed to display Jupyter Widget of type Output.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

<svg id="canvas" onmousemove="svgMouseMove(evt)" width="800" height="600">
<rect id="bg" width="100%" height="100%" fill="#808080"/>
</svg>


Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur purus mi, sollicitudin ac justo a, dapibus ultrices dolor. Curabitur id eros mattis, tincidunt ligula at, condimentum urna. Morbi accumsan, risus eget porta consequat, tortor nibh blandit dui, in sodales quam elit non erat. Aenean lorem dui, lacinia a metus eu, accumsan dictum urna. Sed a egestas mauris, non porta nisi. Suspendisse eu lacinia neque. Morbi gravida eros non augue pharetra, condimentum auctor purus porttitor.

## Header 2

```python
a = 1
```

```python
a
```

    1

```python
b = 'pew'
```

```python
b
```

    'pew'

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
```

```python
from pylab import *
```

```python
x = linspace(0, 5, 10)
y = x ** 2
```

```python
figure()
plot(x, y, 'r')
xlabel('x')
ylabel('y')
title('title')
show()
```

```python
import numpy as np
```

```python
num_points = 130
y = np.random.random(num_points)
plt.plot(y)
```

This is some text, here comes some latex

```latex
%%latex
\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}
```

\begin{align}
\nabla \times \vec{\mathbf{B}} -\, \frac1c\, \frac{\partial\vec{\mathbf{E}}}{\partial t} & = \frac{4\pi}{c}\vec{\mathbf{j}} \\
\nabla \cdot \vec{\mathbf{E}} & = 4 \pi \rho \\
\nabla \times \vec{\mathbf{E}}\, +\, \frac1c\, \frac{\partial\vec{\mathbf{B}}}{\partial t} & = \vec{\mathbf{0}} \\
\nabla \cdot \vec{\mathbf{B}} & = 0
\end{align}

Apos?

```python
import re
```

```python
text = 'foo bar\t baz \tqux'
```

```python
re.split('\s+', text)
```

    ['foo', 'bar', 'baz', 'qux']

```python

```


 Circuit Analysis Using Sympy

 Assignment 7

 M V A Suhas kumar,EE17B109

March 16,2019

# Introduction
In this assignment, we use Sympy to analytically solve a matrix equation governing an analog circuit. We look at two circuits, an active low pass filter and an active high pass filter. We create matrices using node equations for the circuits in sympy, and then solve the equations analytically. We then convert the resulting sympy solution into a numpy function which can be called. We then use the signals toolbox we studied in the last assignment to understand the responses of the two circuits to various inputs.

Importing required packages

```python

from sympy import *
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sp
from pylab import *
from IPython.display import *

```

# Low pass Filter

where G  =1.586 and R1 = R2 = 10kΩ and C1=C2=10pF. This gives a 3dB Butter-worth filter with cutoff frequency of 1/2πMHz.

Circuit Equations are as follows:
$$V_{m}=\frac{V_{o}}{G}$$
$$ V_{p} =V_{1} \frac{1}{1+s R_{2}C_{2}}$$
$$ V_{o} = G(V_{p} - V_{m})$$
$$\frac{V_{i}-V_{1}}{R_{1}} + \frac{V_{p}-V_{1}}{R_{2}} + s C_{1}(V_{0}-V_{1}) = 0$$
Solving the above equations with approxmtion gives

$$ V_{o} \approx \frac{V_{i}}{s R_{1} C_{1}}$$

We would like to solve this in Python and also get (and plot) the exact result.  For this we need the sympy module.

To solve the equtions exactly we use matrix method of solving:

```python
init_printing()
R1,R2,C1,C2,G = symbols("R1 R2 C1 C2 G")
V1,Vp,Vm,Vo,Vi = symbols("V1 Vp Vm Vo Vi")
s = symbols("s")
A = Matrix([[0,0,1,-1/G],
            [-1/(1+s*R2*C2),1,0,0],
            [0,-G,G,1],
          [-1/R1-1/R2-s*C1,1/R2,0,s*C1]])
M = Matrix([V1,Vp,Vm,Vo])
b = Matrix([0,0,0,Vi/R1])
display(Eq(MatMul(A,M),b))
```

Solving the above matrix yield exact result

Function defining low pass filter:

```python
def lowpass(R1=10**4,R2=10**4,C1=10**-11,C2=10**-11,G=1.586,Vi=1):
    s=symbols("s")
    A=Matrix([[0,0,1,-1/G],
              [-1/(1+s*R2*C2),1,0,0],
              [0,-G,G,1],
              [-1/R1-1/R2-s*C1,1/R2,0,s*C1]])
    b=Matrix([0,0,0,Vi/R1])
    V = A.inv()*b
    return(A,b,V)

```

Function which can take input in laplace domain or time domain and give the output of low pass filter:

```python
def low_pass_output(laplace_fn = None,time_fn=None,t=np.linspace(0,1e-5,1e5),C=10**-11):
    A,b,V = lowpass(C1=C,C2=C)
    v_low_pass = V[-1]
    temp = expand(simplify(v_low_pass))
    n,d = fraction(temp)
    n,d = Poly(n,s),Poly(d,s)
    num,den = n.all_coeffs(),d.all_coeffs()
    H_v_low_pass = sp.lti([-float(f) for f in num],[float(f) for f in den])
    if laplace_fn !=None:
        temp = expand(simplify(laplace_fn))
        n,d = fraction(temp)
        n,d = Poly(n,s),Poly(d,s)
        num,den = n.all_coeffs(),d.all_coeffs()
        lap = sp.lti([float(f) for f in num],[float(f) for f in den])
        t,u = sp.impulse(lap,None,t)
    else:
        u = time_fn
    t,V_out,svec = sp.lsim(H_v_low_pass,u,t)
    return (t,V_out)
```

    /home/suhas/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:1: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.
      """Entry point for launching an IPython kernel.

# High pass filter

values you can use are R1=R3=10kΩ, C1=C2=1nF, and G=1.586

Circuit Equations are as follows:
$$V_{n}=\frac{V_{o}}{G}$$
$$ V_{p} =V_{1} \frac{s R_{3}C_{2}}{1+s R_{3}C_{2}}$$
$$ V_{o} = G(V_{p} - V_{n})$$
$$(V_{1}-V_{i})sC_{1} + \frac{(V_{1}-V_{o})}{R_{1}} + (V_{i}-V_{p})sC_{2} = 0 $$

```python
R1, R3, C1, C2, G, Vi = symbols('R_1 R_3 C_1 C_2 G V_i')
V1,Vn,Vp,Vo = symbols('V_1 V_n V_p V_o')
x=Matrix([V1,Vn,Vp,Vo])

A=Matrix([[0,-1,0,1/G],
        [s*C2*R3/(s*C2*R3+1),0,-1,0],
        [0,G,-G,1],
        [-s*C2-1/R1-s*C1,0,s*C2,1/R1]])

b=Matrix([0,0,0,-Vi*s*C1])
init_printing
display(Eq(MatMul(A,x),b))
```

Function defining high pass filter:

```python
def highpass(R1=10**4,R3=10**4,C1=10**-9,C2=10**-9,G=1.586,Vi=1):
    s= symbols("s")
    A=Matrix([[0,-1,0,1/G],
        [s*C2*R3/(s*C2*R3+1),0,-1,0],
        [0,G,-G,1],
        [-s*C2-1/R1-s*C1,0,s*C2,1/R1]])

    b=Matrix([0,0,0,-Vi*s*C1])
    V =A.inv() * b
    return (A,b,V)

```

Function which can take input in laplace domain or time domain and give the output of high pass filter:

```python

def high_pass_output(laplace_fn = None,time_fn=None,t=np.linspace(0,1e-4,1e5),C=10**-11):
    A,b,V = highpass(C1=C,C2=C)
    v_high_pass = V[-1]
    temp = expand(simplify(v_high_pass))
    n,d = fraction(temp)
    n,d = Poly(n,s),Poly(d,s)
    num,den = n.all_coeffs(),d.all_coeffs()
    H_v_high_pass = sp.lti([float(f) for f in num],[float(f) for f in den])
    if laplace_fn !=None:
        temp = expand(simplify(laplace_fn))
        n,d = fraction(temp)
        n,d = Poly(n,s),Poly(d,s)
        num,den = n.all_coeffs(),d.all_coeffs()
        lap = sp.lti([float(f) for f in num],[float(f) for f in den])
        t,u = sp.impulse(lap,None,t)
    else:
        u = time_fn
    t,V_out,svec = sp.lsim(H_v_high_pass,u,t)
    return (t,V_out)
```

    /home/suhas/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

# Question1

Step Response for low pass filter

```python
t,V_low_step = low_pass_output(laplace_fn=1/s)
```

```python
plt.plot(t,V_low_step)
plt.grid(True)
plt.xlabel("t ------>",size=14)
plt.ylabel(r"$Step\ Response\ V_{o}(t)$",size=14)
plt.title("Step Response When Capacitance = 10pF in low pass filter")
plt.show()
```

Step response is starting from zero and reaching 0.793 at steady state.This is because DC gain oftransfer function  is 0.793.Initial value is 0 because AC gain of low pass filter is zero(impulse can be assumed as High frequency signal and we know low pass filter dosen't pass high frequency signal).

# Question2

Finding Output when input signal is $$(sin(2000πt)+cos(2×106πt))u_{o}(t)$$

```python
t = np.linspace(0,1e-3,1e5)
plt.plot(t,np.sin(2000*np.pi*t)+np.cos(2e6*np.pi*t))
plt.grid(True)
plt.xlabel("t ------>",size=14)
plt.ylabel(r"$V_{i}(t)$",size=14)
plt.title("Mixed frequency input")
plt.show()
```

Band is high frequency wave and envolope is the low frequency wave

```python

t = linspace(0,1e-5,1e5)
t,vout = low_pass_output(time_fn=np.sin(2000*np.pi*t)+np.cos(2e6*np.pi*t),t=t,C=10**-9)
```

    /home/suhas/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

```python
plt.plot(t,vout)
plt.grid(True)
plt.xlabel("t ------>",size=14)
plt.ylabel(r"$V_{o}(t)$",size=14)
plt.title("Output for mixed frequency Sinusoid in lowpass filter in transient time")
plt.show()
```

From above we can clearly see that Output is superposition of High Amplitude low frequency wave and Low amplitude High frquency wave(Since Low pass filter attenuates the High frequencies)

```python

t = linspace(0,1e-5,1e5)
t,vout = high_pass_output(time_fn=np.sin(2000*np.pi*t)+np.cos(2e6*np.pi*t),t=t,C=10**-9)
```

    /home/suhas/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

```python
plt.plot(t,vout)
plt.grid(True)
plt.xlabel("t ------>",size=14)
plt.ylabel(r"$V_{o}(t)$",size=14)
plt.title("Output for mixed frequency Sinusoid in High pass filter in transient time")
plt.show()
```

The plot which is appearing to be band(closely placed lines) is superposition of High Amplitude High frequency wave and Low amplitude Low frquency wave(Since High pass filter attenuates the Low frequencies) which inturn appears to be non distorted sine wave.

```python

t = linspace(0,1e-3,1e5)
t,vout = low_pass_output(time_fn=np.sin(2000*np.pi*t)+np.cos(2e6*np.pi*t),t=t,C=10**-9)
```

    /home/suhas/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

```python
plt.plot(t,vout)
plt.grid(True)
plt.xlabel("t ------>",size=14)
plt.ylabel(r"$V_{o}(t)$",size=14)
plt.title("Output for mixed frequency Sinusoid in lowpass filter in steady time")
plt.show()
```

From graph we can see frequency is close to 1000Hz(which is low frquency input)

```python

t = linspace(0,1e-4,1e5)
t,vout = high_pass_output(time_fn=np.sin(2000*np.pi*t)+np.cos(2e6*np.pi*t),t=t,C=10**-9)
```

    /home/suhas/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

```python
plt.plot(t,vout)
plt.grid(True)
plt.xlabel("t ------>",size=14)
plt.ylabel(r"$V_{o}(t)$",size=14)
plt.title("Output for mixed frequency Sinusoid in High pass filter in steay time")
plt.show()
```

From graph we can see frequency is close to 1000KHz(which is high frquency input)

# Question 3,4

Damped Sinusoid -----> $exp(-300t)sin(10^{6}t)$

```python

t = linspace(0,1e-3,1e6)
f = np.exp(-3000*t) * np.sin(10**6 *t)
```

    /home/suhas/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

```python
plt.title("High frequency damped sinusoid")
plt.xlabel("$t$")
plt.ylabel("$v_i(t)$",size=20)
plt.plot(t,f)
plt.grid()
plt.show()
```

```python

t = linspace(0,1e-3,1e6)
t,vout = high_pass_output(time_fn=f,t=t,C=10**-9)
```

    /home/suhas/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

```python
plt.plot(t,vout)
plt.grid(True)
plt.xlabel("t ------>",size=14)
plt.ylabel(r"$V_{o}(t)$",size=14)
plt.title("Output for High frequency damped input in High pass filter")
plt.show()
```

From above graph we can clearly see that High pass filter passed high frequency sinusoid with out attenuating much.(Since property of high pass filter)

```python

t = linspace(0,1e-3,1e6)
t,vout = low_pass_output(time_fn=f,t=t,C=10**-9)
```

    /home/suhas/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: DeprecationWarning: object of type <class 'float'> cannot be safely interpreted as an integer.

```python
plt.plot(t,vout)
plt.grid(True)
plt.xlabel("t ------>",size=14)
plt.ylabel(r"$V_{o}(t)$",size=14)
plt.title("Output for High frequency damped input in low pass filter")
plt.show()
```

From above graph Low pass filter quickly attenuated the High frequency Sinusoid and gives distorted Output

# Question 5

```python
t,V_high_step = high_pass_output(laplace_fn=1/s,C=10**-9)
```

```python
plt.plot(t,V_high_step)
plt.grid(True)
plt.xlabel("t ------>",size=14)
plt.ylabel(r"$Step\ Response\ V_{o}(t)$",size=14)
plt.title("Step Response When Capacitance = 1nF in high pass filter")
plt.show()
```

Step response here saturates at zero and this is because DC gain of High pass filter is 0. We can clearly see from graph that it starts from 0.793 and this because AC gain of transfer function  at high frequencies is 0.793(Step can assumed as infinite frequency signal and we know high pass filter only allows high frequency signals)

step response overshoots the steady state value of 0, reaches an
extremum, then settles back to 0, unlike the response of the low pass filter which steadily
approaches the steady state value with no extrema. This occurs because of the presence of
zeros at the origin in the transfer function of the high pass filter(which imply that the DC
gain is 0). Since the steady state value of the step response is 0, the total signed area under
the curve of the impulse response must also be 0. This means that the impulse response must
equal zero at one or more time instants. Since the impulse response is the derivative of the
step response, this therefore means that the step response must have at least one extremum.
This explains the behaviour of the step response of the high pass filter.

# Conclusions:

The low pass filter responds by letting the low frequency sinusoid pass through without
much additional attenuation. The output decays as the input also decays.

The high pass filter responds by quickly attenuating the input. Notice that the time scales
show that the high pass filter response is orders of magnitudes faster than the low pass
response. This is because the input frequency is below the cutoff frequency, so the output
goes to 0 very fast.

In conclusion, the sympy module has allowed us to analyse quite complicated circuits by
analytically solving their node equations. We then interpreted the solutions by plotting time
domain responses using the signals toolbox. Thus, sympy combined with the scipy.signal
module is a very useful toolbox for analyzing complicated systems like the active filters in
this assignment.