### Gradient Descent

* Gradient Descent is a popular optimization technique in Machine Learning and Deep Learning, and it can be used with most, if not all, of the learning algorithms.
* A gradient is the slope of a function. It measures the degree of change of a variable in response to the changes of another variable.
* Mathematically, Gradient Descent is a convex function whose output is the partial derivative of a set of parameters of its inputs. The greater the gradient, the steeper the slope.
* Starting from an initial value, Gradient Descent is run iteratively to find the optimal values of the parameters to find the minimum possible value of the given cost function.

#### Types of Gradient Descent:

Typically, there are three types of Gradient Descent:

- Batch Gradient Descent
- Stochastic Gradient Descent
- Mini-batch Gradient Descent

```
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

```

```
plt.style.use(['ggplot'])
```

# Create Data

 Generate some data with:
\begin{equation} \theta_0= 4 \end{equation}
\begin{equation} \theta_1= 3 \end{equation}

Add some Gaussian noise to the data

```
X = 2 * np.random.rand(100,1)
y = 4 +3 * X+np.random.randn(100,1)
```

Let's plot our data to check the relation between X and Y

```

plt.plot(X,y,'b.')
plt.xlabel("$x$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
_ =plt.axis([0,2,0,15])
```

#  Analytical way of Linear Regression

```
X_b = np.c_[np.ones((100,1)),X]
theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)
print(theta_best)
```

    [[3.9073148 ]
     [3.01648259]]

This is close to our real thetas 4 and 3. It cannot be accurate due to the noise I have introduced in data

```
X_new = np.array([[0],[2]])
X_new_b = np.c_[np.ones((2,1)),X_new]
y_predict = X_new_b.dot(theta_best)
y_predict
```

    array([[3.9073148 ],
           [9.94027997]])

Let's plot prediction line with calculated:theta

```
plt.plot(X_new,y_predict,'r-')
plt.plot(X,y,'b.')
plt.xlabel("$x_1$", fontsize=18)
plt.ylabel("$y$", rotation=0, fontsize=18)
plt.axis([0,2,0,15])

```

# Gradient Descent

## Cost Function & Gradients

 The equation for calculating cost function and gradients are as shown below. Please note the cost function is for Linear regression. For other algorithms the cost function will be different and the gradients would have  to be derived from the cost functions

Cost
\begin{equation}
J(\theta) = 1/2m \sum_{i=1}^{m} (h(\theta)^{(i)} - y^{(i)})^2
\end{equation}

Gradient

\begin{equation}
\frac{\partial J(\theta)}{\partial \theta_j} = 1/m\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_j^{(i)}
\end{equation}

Gradients
\begin{equation}
\theta_0: = \theta_0 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}
\begin{equation}
\theta_1: = \theta_1 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_1^{(i)})
\end{equation}
\begin{equation}
\theta_2: = \theta_2 -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_2^{(i)})
\end{equation}

\begin{equation}
\theta_j: = \theta_j -\alpha . (1/m .\sum_{i=1}^{m}(h(\theta^{(i)} - y^{(i)}).X_0^{(i)})
\end{equation}

```

def  cal_cost(theta,X,y):
    '''

    Calculates the cost for given X and Y. The following shows and example of a single dimensional X
    theta = Vector of thetas
    X     = Row of X's np.zeros((2,j))
    y     = Actual y's np.zeros((2,1))

    where:
        j is the no of features
    '''

    m = len(y)

    predictions = X.dot(theta)
    cost = (1/2*m) * np.sum(np.square(predictions-y))
    return cost

```

```
def gradient_descent(X,y,theta,learning_rate=0.01,iterations=100):
    '''
    X    = Matrix of X with added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    '''
    m = len(y)
    cost_history = np.zeros(iterations)
    theta_history = np.zeros((iterations,2))
    for it in range(iterations):

        prediction = np.dot(X,theta)

        theta = theta -(1/m)*learning_rate*( X.T.dot((prediction - y)))
        theta_history[it,:] =theta.T
        cost_history[it]  = cal_cost(theta,X,y)

    return theta, cost_history, theta_history

```

 Let's start with 1000 iterations and a learning rate of 0.01. Start with theta from a Gaussian distribution

```
lr =0.01
n_iter = 1000

theta = np.random.randn(2,1)

X_b = np.c_[np.ones((len(X),1)),X]
theta,cost_history,theta_history = gradient_descent(X_b,y,theta,lr,n_iter)

print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    Theta0:          3.718,
    Theta1:          3.173
    Final cost/MSE:  5052.950

 Let's plot the cost history over iterations

```
fig,ax = plt.subplots(figsize=(12,8))

ax.set_ylabel('J(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(n_iter),cost_history,'b.')
```

 After around 150 iterations the cost is flat so the remaining iterations  are not needed or will not result in any further optimization. Let us zoom in till iteration 200 and see the curve

```

fig,ax = plt.subplots(figsize=(10,8))
ax.set_ylabel('J(Theta)')
ax.set_xlabel('Iterations')
_=ax.plot(range(200),cost_history[:200],'b.')
```

It is worth while to note that the cost drops faster initially and then the gain in cost reduction is not as much

### It would be great to see the effect of different learning rates and iterations together

### Let us  build a function which can show the effects together and also show how gradient decent actually is working

```

def plot_GD(n_iter,lr,ax,ax1=None):
     """
     n_iter = no of iterations
     lr = Learning Rate
     ax = Axis to plot the Gradient Descent
     ax1 = Axis to plot cost_history vs Iterations plot

     """
     _ = ax.plot(X,y,'b.')
     theta = np.random.randn(2,1)

     tr =0.1
     cost_history = np.zeros(n_iter)
     for i in range(n_iter):
        pred_prev = X_b.dot(theta)
        theta,h,_ = gradient_descent(X_b,y,theta,lr,1)
        pred = X_b.dot(theta)

        cost_history[i] = h[0]

        if ((i % 25 == 0) ):
            _ = ax.plot(X,pred,'r-',alpha=tr)
            if tr < 0.8:
                tr = tr+0.2
     if not ax1== None:
        _ = ax1.plot(range(n_iter),cost_history,'b.')
```

### Plot the graphs for different iterations and learning rates combination

```
fig = plt.figure(figsize=(30,25))
fig.subplots_adjust(hspace=0.4, wspace=0.4)

it_lr =[(2000,0.001),(500,0.01),(200,0.05),(100,0.1)]
count =0
for n_iter, lr in it_lr:
    count += 1

    ax = fig.add_subplot(4, 2, count)
    count += 1

    ax1 = fig.add_subplot(4,2,count)

    ax.set_title("lr:{}".format(lr))
    ax1.set_title("Iterations:{}".format(n_iter))
    plot_GD(n_iter,lr,ax,ax1)

```

 See how useful it is to visualize the effect of learning rates and iterations on gradient descent. The red lines show how the gradient descent starts and then slowly gets closer to the final value

## You can always plot Indiviual graphs to zoom in

```
_,ax = plt.subplots(figsize=(14,10))
plot_GD(100,0.1,ax)
```

# Stochastic Gradient Descent

```
def stocashtic_gradient_descent(X,y,theta,learning_rate=0.01,iterations=10):
    '''
    X    = Matrix of X with added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    '''
    m = len(y)
    cost_history = np.zeros(iterations)

    for it in range(iterations):
        cost =0.0
        for i in range(m):
            rand_ind = np.random.randint(0,m)
            X_i = X[rand_ind,:].reshape(1,X.shape[1])
            y_i = y[rand_ind].reshape(1,1)
            prediction = np.dot(X_i,theta)

            theta = theta -(1/m)*learning_rate*( X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta,X_i,y_i)
        cost_history[it]  = cost

    return theta, cost_history
```

```
lr =0.1
n_iter = 200

theta = np.random.randn(2,1)

X_b = np.c_[np.ones((len(X),1)),X]
theta,cost_history = stocashtic_gradient_descent(X_b,y,theta,lr,n_iter)

print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    Theta0:          3.827,
    Theta1:          3.086
    Final cost/MSE:  46.802

```
fig,ax = plt.subplots(figsize=(10,8))

ax.set_ylabel('{J(Theta)}',rotation=0)
ax.set_xlabel('{Iterations}')
theta = np.random.randn(2,1)

_=ax.plot(range(n_iter),cost_history,'b.')
```

# Mini Batch Gradient Descent

```
def minibatch_gradient_descent(X,y,theta,learning_rate=0.01,iterations=10,batch_size =20):
    '''
    X    = Matrix of X without added bias units
    y    = Vector of Y
    theta=Vector of thetas np.random.randn(j,1)
    learning_rate
    iterations = no of iterations

    Returns the final theta vector and array of cost history over no of iterations
    '''
    m = len(y)
    cost_history = np.zeros(iterations)
    n_batches = int(m/batch_size)

    for it in range(iterations):
        cost =0.0
        indices = np.random.permutation(m)
        X = X[indices]
        y = y[indices]
        for i in range(0,m,batch_size):
            X_i = X[i:i+batch_size]
            y_i = y[i:i+batch_size]

            X_i = np.c_[np.ones(len(X_i)),X_i]

            prediction = np.dot(X_i,theta)

            theta = theta -(1/m)*learning_rate*( X_i.T.dot((prediction - y_i)))
            cost += cal_cost(theta,X_i,y_i)
        cost_history[it]  = cost

    return theta, cost_history
```

```
lr =0.1
n_iter = 200

theta = np.random.randn(2,1)

theta,cost_history = minibatch_gradient_descent(X,y,theta,lr,n_iter)

print('Theta0:          {:0.3f},\nTheta1:          {:0.3f}'.format(theta[0][0],theta[1][0]))
print('Final cost/MSE:  {:0.3f}'.format(cost_history[-1]))
```

    Theta0:          3.835,
    Theta1:          3.077
    Final cost/MSE:  999.096

```
fig,ax = plt.subplots(figsize=(10,8))

ax.set_ylabel('{J(Theta)}',rotation=0)
ax.set_xlabel('{Iterations}')
theta = np.random.randn(2,1)

_=ax.plot(range(n_iter),cost_history,'b.')
```


# Least-Squares Regression Notebook
 Using the Normal Equations
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

```python
import numpy as np
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
```

```python
def rsquared(xi,yi,ymodel):
    '''
    xi: vector of length n representing the known x values.
    yi: vector of length n representing the known y values that correspond to xi.
    ymodel: a python function (of x only) that can be evaluated at xi and represents a model fit of
            the data (e.g. a regressed curve).
    '''
    ybar = np.average(yi)
    fi = ymodel(xi)
    result = 1 - np.sum( (yi - fi)**2 )/np.sum( (yi - ybar)**2 )
    return result
```

The purpose of this workbook is to develop a model that predicts the university GPA given a high-school GPA. We will use regression analysis to illustrate how we can do a best fit in the least-squares sense. We will use the normal equations to show this and also compare to direct regression as well as `numpy's` `polyfit` function.

First, load GPA data from gpa_data.txt. This data file contains the following columns:

high_GPA math_SAT verb_SAT comp_GPA univ_GPA

where

* high_GPA	High school grade point average
* math_SAT	Math SAT score
* verb_SAT	Verbal SAT score
* comp_GPA	Computer science grade point average
* univ_GPA	Overall university grade point average

This data was obtained from http://onlinestatbook.com/2/case_studies/sat.html. "The data examine the SAT and GPA information of 105 students who graduated from a state university with a B.S. in computer science. Using the grades and test scores from high school"

```python
data = np.loadtxt('gpa_data.txt').T
```

```python
# high school gpa
hgpa = data[0]
# university gpa
ugpa = data[4]
```

As usual, the first thing to do is plot the data

```python
plt.plot(hgpa,ugpa,'o')
plt.xlabel('high-shool gpa')
plt.ylabel('university gpa')
plt.title('University GPA')
plt.grid()
```

We will now develop a regression model for this using the normal equaitons.

## Straight Line Fit
 For a straight line model, we have
\begin{equation}
y = a_0 + a_1 x
\end{equation}
For this model, the normal equations are given by the system
\begin{equation}
\left[ {\begin{array}{*{20}{c}}
1& x_1\\
1& x_2 \\
 \vdots & \vdots \\
1& x_n
\end{array}} \right]\left( \begin{array}{l}
a_0\\
a_1
\end{array} \right) = \left( {\begin{array}{*{20}{c}}
y_1\\
y_2\\
 \vdots \\
y_n
\end{array}} \right)
\end{equation}
To solve this system in the least-squares sense, we have to solve the following system
\begin{equation}
[\mathbf{A}]^\text{T} [\mathbf{A}]  \mathbf{a} = [\mathbf{A}]^\text{T} \mathbf{b}
\end{equation}

For simplicity, let's rename of input data

```python
xi = hgpa
yi = ugpa
```

We now develop the matrix A as a numpy array of column vectors

```python
# get the size of the input data
N = len(xi)
# note that we have to take the transpose to make this work
A = np.array([np.ones(N),xi]).T
```

Now construct the normal equations

```python
ATA =  A.T @ A
b   = A.T@yi
```

solve them using the `linalg` package

```python
sol = np.linalg.solve(ATA,A.T@yi)
print(sol)
```

    [1.09682328 0.6748299 ]

Recall that the solution contains the coefficients of the linear fit. To plot the fit on top of the input data, we simply use the coefficients to create a straight line and plot it

```python
a0 = sol[0]
a1 = sol[1]
fit = lambda x: a0 + a1*x
plt.plot(xi,yi,'o')
plt.plot(xi,fit(xi))
plt.xlabel('high-shool gpa')
plt.ylabel('university gpa')
plt.title('University GPA')
plt.grid()
```

Let's check the R2 value

```python
r2 = rsquared(xi,yi,fit)
print(r2)
```

    0.5036754689173208

## Quadratic Fit
 For a quadratic model, we have
\begin{equation}
y = a_0 + a_1 x + a_2 x^2
\end{equation}
For this model, the normal equations are given by the system
\begin{equation}
\left[ {\begin{array}{*{20}{c}}
1&x_1 & x_1^2\\
1&x_2 & x_2^2\\
 \vdots & \vdots & \vdots\\
1 & x_n & x_n^2
\end{array}} \right]\left( \begin{array}{l}
{a_0}\\
{a_1} \\
a_2
\end{array} \right) = \left( {\begin{array}{*{20}{c}}
y_1\\
y_2\\
 \vdots \\
y_n
\end{array}} \right)
\end{equation}
To solve this system in the least-squares sense, we have to solve the following system
\begin{equation}
[\mathbf{A}]^\text{T} [\mathbf{A}]  \mathbf{a} = [\mathbf{A}]^\text{T} \mathbf{b}
\end{equation}

```python
# note that we have to take the transpose to make this work
A = np.array([np.ones(N),xi, xi**2]).T
```

Now construct the normal equations

```python
ATA =  A.T @ A
b   = A.T@yi
```

solve them using the `linalg` package

```python
sol = np.linalg.solve(ATA,A.T@yi)
print(sol)
```

    [ 0.97314251  0.75950032 -0.01406161]

Recall that the solution contains the coefficients of the linear fit. To plot the fit on top of the input data, we simply use the coefficients to create a straight line and plot it

```python
a0 = sol[0]
a1 = sol[1]
a2 = sol[2]
fit = lambda x: a0 + a1*x + a2*x**2
plt.plot(xi,yi,'o')

# here we must generate a linear space so that the polynomial
# doesn't go through duplicate x values from the input data
xx = np.linspace(2,4)
plt.plot(xx,fit(xx))

plt.xlabel('high-shool gpa')
plt.ylabel('university gpa')
plt.title('University GPA')
plt.grid()
```

Finally, the R2 value is

```python
r2 = rsquared(xi,yi,fit)
print(r2)
```

    0.6077867092534325

## Using Polyfit

```python
coefs = np.polyfit(xi,yi,1)
print(coefs)
```

    [0.6748299  1.09682328]

```python
p = np.poly1d(coefs)
plt.plot(xi,yi,'o')
x = np.linspace(2,4)
plt.plot(x,p(x))
print(rsquared(xi,yi,p))
```

    0.6378477159771265

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

CSS style adapted from https://github.com/barbagroup/CFDPython. Copyright (c) Barba group
<link href='http://fonts.googleapis.com/css?family=Merriweather' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Bitter' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Oxygen' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Lora' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```

topics/regression/Normal Equations Example - GPA.ipynb


# Center of Mass and Moment of Inertia
Renato Naville Watanabe

### Center of mass

The center of mass of a set of particles is formally defined as the point where the sum of the vectors linking this point to each particle, weighted by its mass, is zero. The sum of the vectors linking the center of mass to each particle is:

\begin{equation}
   \sum\limits_{i = 1}^nm_i\left({\bf\vec{r_i}} - {\bf\vec{r}_{cm}}\right) = \sum\limits_{i = 1}^nm_i{\bf\vec{r_i}} - \sum\limits_{i = 1}^nm_i{\bf\vec{r}_{cm}} = \sum\limits_{i = 1}^nm_i{\bf\vec{r_i}} - {\bf\vec{r}_{cm}}\sum\limits_{i = 1}^nm_i
\end{equation}
where $n$ is the number of particles.

Now, we equal this sum to zero and isolate ${\bf\vec{r_{cm}}}$:

\begin{equation}
     \sum\limits_{i = 1}^nm_i{\bf\vec{r_i}} - {\bf\vec{r}_{cm}}\sum\limits_{i = 1}^nm_i = 0 \longrightarrow {\bf\vec{r}_{cm}} = \frac{\sum\limits_{i = 1}^nm_i{\bf\vec{r_i}}}{\sum\limits_{i = 1}^nm_i}  = \frac{\sum\limits_{i = 1}^nm_i{\bf\vec{r_i}}}{m_T}
\end{equation}

where $m_T$ is the total mass of the particles.

For a rigid body, the center of mass is defined as the point where the integral of the vectors  linking this point to each differential part of mass, weighted by this differential mass, is zero. This integral is:

\begin{equation}
    \int\limits_{B} {\bf{\vec{r}_B}} dm = \int\limits_{B} {\bf{\vec{r}}} - {\bf{\vec{r}_{cm}}}  dm = \int\limits_{B} {\bf{\vec{r}}}\,dm - \int\limits_{B}{\bf{\vec{r}_{cm}}}  dm = \int\limits_{B} {\bf{\vec{r}}}\,dm - {\bf{\vec{r}_{cm}}}\int\limits_{B}\, dm
\end{equation}

Now we equal this integral to zero and isolate ${\bf\vec{r_{cm}}}$:

\begin{equation}
    \int\limits_{B} {\bf{\vec{r}}}\,dm - {\bf{\vec{r}_{cm}}}\int\limits_{B}\, dm  = 0 \longrightarrow  {\bf{\vec{r}_{cm}}} = \frac{ \int\limits_{B} {\bf{\vec{r}}}\,dm}{\int\limits_{B}\, dm}  = \frac{ \int\limits_{B} {\bf{\vec{r}}}\,dm}{m_B}
\end{equation}
where $m_B$ is the mass of the body.

### Center of gravity

Center of gravity of a body is the point where the moment caused by the gravitational force in the whole body relative to this point is zero.

\begin{equation}
    \vec{\bf{M_0}} = \int\limits_{B}  \vec{\bf{r_B}}\, \times\,\vec{\bf{g}}\,dm
\end{equation}

If the acceleration of gravity being applied to the whole body is the same (for all practical purposes, in Biomechanics we can consider it the same in the whole body), the gravity vector can go out of the integral:

\begin{equation}
  \vec{\bf{M_0}}= \int\limits_{B}\vec{\bf{r_B}}\,dm\, \times\,\vec{\bf{g}} =  \int\limits_{B}(\vec{\bf{r}} - \vec{\bf{r_{cm}}})\,dm\, \times\,\vec{\bf{g}} =  \left(\int\limits_{B}\vec{\bf{r}}\,dm -\int\limits_{B}\vec{\bf{r_{G}}}\,dm\,\right) \times\,\vec{\bf{g}}
\end{equation}

Now, we equal this moment to zero and isolate $\vec{\bf{r_{G}}}$:

\begin{equation}
            \left(\int\limits_{B}\vec{\bf{r}}\,dm -\int\limits_{B}\vec{\bf{r_{G}}}\,dm\right) \times\,\vec{\bf{g}} = 0 \longrightarrow \int\limits_{B}\vec{\bf{r}}\,dm -\int\limits_{B}\vec{\bf{r_{G}}}\,dm = 0 \longrightarrow \int\limits_{B}\vec{\bf{r}}\,dm -\vec{\bf{r_{G}}}\int\limits_{B}\,dm = 0\,\,\,\,\,\, \longrightarrow \vec{\bf{r_{G}}} = \frac{ \int\limits_{B}\vec{\bf{r}}\,dm}{\int\limits_{B}\,dm} = \frac{ \int\limits_{B}\vec{\bf{r}}\,dm}{m_B}
\end{equation}
where $m_B$ is the mass of the body.

Note that in this case, that the gravity acceleration is constant, the center of gravity $\vec{\bf{r_{G}}}$ is equal to the center of mass $\vec{\bf{r_{cm}}}$.

### Moment of inertia

The moment of inertia is a measure of how the mass of the body distributes relative to a given axis passing to a given point O. If we define the versor $\hat{\bf{n_a}}$ as a versor in the direction of the axis, then the moment of inertia relative to the point O in the direction of the versor $\hat{\bf{n_a}}$ is defined as:

\begin{equation}
    J_a = \int\limits_B (\vec{\bf{r_{/O}}} \times \hat{\bf{n_a}}).(\vec{\bf{r_{/O}}} \times \hat{\bf{n_a}})\,dm
    \label{eq:inmomgen}
\end{equation}

For planar movements, we normally compute the moment of inertia relative to the z-axis. So, the Eq.~\eqref{eq:inmomgen} simplifies to:

\begin{equation}

\end{equation}
### Parallel axis theorem

### Matrix of Inertia

```python

```

### Problems
3.2.5, 3.2.7,

```python

```

notebooks/.ipynb_checkpoints/CenterOfMassAndMomentOfInertia-checkpoint (rnwatanabe-Inspiron-5457's conflicted copy 2018-03-18).ipynb


#### Jupyter notebooks

This is a [Jupyter](http://jupyter.org/) notebook using Python.  You can install Jupyter locally to edit and interact with this notebook.

# Differential Equations

We now consider the problem of computing $y(t)$ from an *ordinary differential equation* (ODE)

$$ y'(t) = f(t,y) $$

and initial condition $y(0)$.
In the following, $y$ may be a scalar or vector value.
For convenience of notation, we may suppress the time dependence by writing $y$ instead of $y(t)$.

### Second Order Equations

A second order system such as

$$ y'' = f(t, y, y') $$

may always be converted to a first order system by introducing a new variable

$$ \begin{bmatrix} y_0 \\ y_1 \end{bmatrix}' = \begin{bmatrix} y_1 \\ f(t, y_0, y_1) \end{bmatrix} . $$
Therefore, without loss of generality, we will focus on first order systems.

### Implicit formulations

We have chosen the explicit representation $y' = f(t,y)$, but it is more general to write $h(t,y,y') = 0$.
If $\partial h/\partial y'$ is singular, then this describes a *differential algebraic equation* (DAE).  DAE are more challenging to solve and beyond the scope of this course.

## Linear Equations

If $f(y,t)$ is a linear function of $y$ then we have a linear ODE
$$ y' = A(t) y + \text{source}(t) . $$
If $A(t)$ is independent of $t$ and $\text{source}(t) = 0$ then we have a linear, constant, autonomous ODE.

When $y$ is a scalar then $A = a$ is a scalar and the solution is
$$ y = y(0) e^{at} . $$

#### Question
What qualitative dynamics does this imply for
* $a > 0$?
* $a < 0$?
* $a$ imaginary?

#### Question
* What if $A$ is diagonal?
* What if $A = X \Lambda X^{-1}$?

#### Matrix exponential
The general solution can be written in terms of the matrix exponential.
$$ y(t) = e^{At} y(0) . $$
The matrix exponential is defined by its Taylor series
$$ e^A = 1 + A + \frac{A^2}{2} + \frac{A^3}{3!} + \dotsb $$
and there are many [practical ways to compute it](http://www.cs.cornell.edu/cv/ResearchPDF/19ways+.pdf).

#### Question
Suppose that the diagonalization $A = X \Lambda X^{-1}$ exists and derive a finite expression for the matrix exponential using the scalar `exp` function.

## Forward Euler Method

The simplest method for solving $y'(t) = f(t,y)$ is
to use numerical differentiation to write
$$ y' \approx \frac{y(h) - y(0)}{h} $$
which yields the solution estimate
$$ \tilde y(h) = y(0) + h f(0, y(0)) $$
where $h$ is the step size.

Let's try this on a scalar problem
$$ y' = -k (y - \cos t) $$
where $k$ is a parameter controlling the rate at which the solution $u(t)$ is pulled toward the curve $\cos t$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')

def ode_euler(f, y0, tfinal=1, h=0.1):
    y = np.array(y0)
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        tnext = min(t+h, tfinal)
        h = tnext - t
        y += h * f(t, y)
        t = tnext
        thist.append(t)
        yhist.append(y.copy())
    return np.array(thist), np.array(yhist)

tests = []

class cosine:
    def __init__(self, k=5):
        self.k = k
    def __repr__(self):
        return 'cosine(k={:d})'.format(self.k)
    def f(self, t, y):
        return -self.k * (y - np.cos(t))
    def y(self, t, y0):
        k2p1 = self.k**2+1
        return (y0 - self.k**2/k2p1) * np.exp(-self.k*t) + self.k*(np.sin(t) + self.k*np.cos(t))/k2p1

tests.append(cosine(k=2))
tests.append(cosine(k=1000))
```

```python
y0 = np.array([.2])
for test in tests:
    thist, yhist = ode_euler(test.f, y0, h=.2, tfinal=20)
    plt.plot(thist, yhist, '.', label=repr(test)+' Forward Euler')
    plt.plot(thist, test.y(thist, y0), label=repr(test)+' exact')
plt.plot(thist, np.cos(thist), label='cos')
plt.legend(loc='upper right');
```

#### Question

* What happens when $h$ is increased?
* What if $k$ is increased?
* What if the final time is increased?

#### Example: Linear system

Now we consider linear systems

$$ y' = A y $$

which have an exact solution $y(t) = e^{At} y(0)$ in terms of the matrix exponential.

```python
def expm(A):
    """Compute the matrix exponential"""
    L, X = np.linalg.eig(A)
    return X @ np.diag(np.exp(L)) @ np.linalg.inv(X)

class linear:
    def __init__(self, A):
        self.A = A.copy()
    def f(self, t, y):
        return self.A @ y
    def y(self, t, y0):
        return [np.real_if_close(expm(self.A*s) @ y0) for s in t]

test = linear(np.array([[0, 1],
                        [-1, 0]]))
y0 = np.array([.5, 0])
thist, yhist = ode_euler(test.f, y0, h=.1, tfinal=10)
plt.figure()
plt.plot(thist, yhist, '.', label='Euler')
plt.plot(thist, test.y(thist, y0), label='exact')
plt.legend(loc='upper right');
```

#### Questions
* Does shrinking $h$ make this more accurate?
* What if `tfinal` is extended?
* What are the eigenvalues of $A$?

#### Runge-Kutta 4
Let's try a different method.  (We'll look back at where this comes from later.)

```python
def ode_rk4(f, y0, tfinal=1, h=0.1):
    y = np.array(y0)
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        h = min(h, tfinal - t)
        k1 = f(t, y)
        k2 = f(t+h/2, y + k1*h/2)
        k3 = f(t+h/2, y + k2*h/2)
        k4 = f(t+h, y + k3*h)
        y += h/6 * (k1 + 2*k2 + 2*k3 + k4)
        t += h
        thist.append(t)
        yhist.append(y.copy())
    return np.array(thist), np.array(yhist)

thist, yhist = ode_rk4(test.f, y0, h=.5, tfinal=50)
plt.figure()
plt.plot(thist, yhist, '.', label=repr(test)+' RK4')
plt.plot(thist, test.y(thist, y0), label=repr(test)+' exact');
```

### Linear Stability Analysis

Why did Euler diverge (even if slowly) while RK4 solved this problem accurately?
And why do both methods diverge if the step size is too large?
We can understand the convergence of methods by analyzing the test problem
$$ y' = \lambda y $$
for different values of $\lambda$ in the complex plane.
One step of the Euler method with step size $h$ maps
$$ y \to y + h \lambda y = \underbrace{(1 + h \lambda)}_{R(h \lambda)} y $$
where we have introduced the complex-valued function $R(z)$.

* When does this map cause solutions to "blow up" and when is it stable?

```python
def plot_stability(x, y, Rz, label):
    plt.figure()
    levels = np.linspace(0, 2, 21)
    C = plt.contourf(xx, yy, np.abs(Rz), levels, cmap=plt.cm.coolwarm)
    cbar = plt.colorbar(C, ticks=np.linspace(0, 2, 5))
    plt.axvline(x=0, linewidth=1, color='grey')
    plt.axhline(y=0, linewidth=1, color='grey')
    plt.contour(xx, yy, np.abs(Rz), [.5, 1, 1.5], colors='k')
    plt.title(label)

x = np.linspace(-2,2)
xx, yy = np.meshgrid(x, x)
zz = xx + 1j*yy
R = 1 + zz
plot_stability(xx, yy, R, 'Forward Euler')
```

Evidently the forward Euler method is stable if $z = h\lambda$ is in the unit circle centered at $z=-1$, but not stable otherwise.

#### Implicit methods
Recall that forward Euler is the step
$$ \tilde y(h) = y(0) + h f(0, y(0)) . $$
This can be evaluated **explicitly**; all the terms on the right hand side are known so the approximation $\tilde y(h)$ is computed merely by evaluating the right hand side.
Let's consider an alternative, **backward Euler** (or "implicit Euler"),
$$ \tilde y(h) = y(0) + h f(h, \tilde y(h)) . $$
This is a (generally) nonlinear equation for $\tilde y(h)$.
For the test equation $y' = \lambda y$, the backward Euler method is
$$ \tilde y(h) = y(0) + h \lambda \tilde y(h) $$
or
$$ \tilde y(h) = \underbrace{\frac{1}{1 - h \lambda}}_{R(h\lambda)} y(0) . $$

```python
plot_stability(xx, yy, 1/(1-zz), 'Backward Euler')
```

Evidently backward Euler is stable in the entire left half plane (where the exact solution is also stable) and also in some significant portions of the right half plane.  Let's test it on the oscillator problem.

```python
def solve_newtonfd(f, x0):
    def fdjacobian(x):
        J = np.eye(len(x),len(x))
        base = f(x)
        for col in J.T:
            col[:] = (f(x + 1e-8*col) - base) / 1e-8
        return J
    x = x0.copy()
    while True:
        res = f(x)
        if np.linalg.norm(res) < 1e-6:
            return x
        x -= np.linalg.solve(fdjacobian(x), res)

def ode_beuler(f, y0, tfinal=1, h=0.1):
    y = np.array(y0)
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        h = min(h, tfinal - t)
        # Solve x = y + h f(x)
        def residual(ytilde):
            return ytilde - (y + h * f(t+h, ytilde))
        y = solve_newtonfd(residual, y)
        t += h
        thist.append(t)
        yhist.append(y.copy())
    return np.array(thist), np.array(yhist)

y0 = np.array([.5, 0])
thist, yhist = ode_beuler(test.f, y0, h=.1, tfinal=40)
plt.plot(thist, yhist, '.')
plt.plot(thist, test.y(thist, y0))
plt.title('Backward Euler');
```

```python
y0 = np.array([.2])
for tst in tests:
    thist, yhist = ode_beuler(tst.f, y0, h=.3, tfinal=10)
    plt.plot(thist, yhist, '.', label=repr(tst)+' Backward Euler')
    plt.plot(thist, tst.y(thist, y0), label=repr(tst)+' exact')
plt.plot(thist, np.cos(thist), label='cos')
plt.legend(loc='upper right');
```

#### Observations
* We need to solve a system of equations on each time step.
* The cost to solve the linear system is $O(n^3)$ for a system of $n$ equations.
* The Jacobian matrix is $n\times n$, so requires $n^2$ storage when everything else requires $O(n)$.
* I used finite differencing to compute the Jacobian, but many problems have efficient ways to compute an exact Jacobian (it just takes some programming).
* Backward Euler is stable (as predicted by theory), but not very accurate.
* What happens when you make the time step smaller or larger?

### Midpoint Method

What if instead of evaluating the function at the end of the time step, we evaluated in the middle of the time step using the average of the endpoint values.  After all, something similar improved accuracy for our numerical integration...

$$ \tilde y(h) = y(0) + h f\left(\frac h 2, \frac{\tilde y(h) + y(0)}{2} \right) $$

```python
def ode_midpoint(f, y0, tfinal=1, h=0.1):
    y = y0.copy()
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        h = min(h, tfinal - t)
        # Solve x = y + h f(x)
        def residual(x):
            return x - h * f(t+h/2, (x + y)/2) - y
        y = solve_newtonfd(residual, y)
        t += h
        thist.append(t)
        yhist.append(y.copy())
    return np.array(thist), np.array(yhist)

test = linear(np.array([[0, 1],[-1, 0]]))
y0 = np.array([.5, 0])
thist, yhist = ode_midpoint(test.f, y0, h=1, tfinal=50)
plt.plot(thist, yhist, '.')
plt.plot(thist, test.y(thist, y0))
plt.title('Midpoint');
```

```python
y0 = np.array([.2])
for test in [cosine(k=5), cosine(k=500)]:
    thist, yhist = ode_midpoint(test.f, y0, h=.1, tfinal=5)
    plt.plot(thist, yhist, '.', label=repr(test)+' Midpoint')
    plt.plot(thist, test.y(thist, y0), label=repr(test)+' exact')
plt.legend(loc='upper right')
plt.title('Cosine');
```

#### Observations/Questions

* This appears to be quite accurate.
* We still have to solve a system of equations on each time step.
* What happens when the step size changes?
* What is the stability function $R(z)$ for the midpoint rule?

### Stability function

When applied to the test problem $y' = \lambda y$, the midpoint method is
\begin{align}
\tilde y(h) &= y(0) + h \lambda \frac{\tilde y(h) + y(0)}{2} \\
(1 - h \lambda/2) y(h) &= (1 + h\lambda/2) y(0) \\
y(h) &= \underbrace{\frac{1 + h\lambda/2}{1 - h \lambda/2}}_{R(h \lambda} y(0)
\end{align}
so the stability function is
$$ R(z) = \frac{1 + z/2}{1 - z/2} .$$

```python
plot_stability(xx, yy, (1+zz/2)/(1-zz/2), 'Midpoint')
```

### Runge-Kutta Methods

All of the methods we have seen thus far can be represented as Runge-Kutta methods, which can be arranged as a series of $s$ "stage" equations (possibly coupled) and a completion formula.

$$\begin{split}
Y_i = y(0) + h \sum_j a_{ij} f(t+c_j h, Y_j) \\
y(h) = y(0) + h \sum_j b_j f(t+c_j h, Y_j)
\end{split}$$

where $c$ is a vector of *abscissa*, $A$ is a table of coefficients, and $b$ is a vector of completion weights.
These coefficients are typically expressed in a Butcher Table
$$ \left[ \begin{array}{c|cc}
c_0 & a_{00} & a_{01} \\
c_1 & a_{10} & a_{11} \\
\hline
& b_0 & b_1
\end{array} \right] . $$

If the matrix $A$ is strictly lower triangular, then the method is **explicit** (does not require solving equations).  We have seen forward Euler

$$ \left[ \begin{array}{c|c}
0 & 0  \\
\hline
& 1
\end{array} \right] ,$$
backward Euler
$$ \left[ \begin{array}{c|c}
1 & 1 \\
\hline
& 1
\end{array} \right] ,$$
and Midpoint
$$ \left[ \begin{array}{c|c}
\frac 1 2 & \frac 1 2 \\
\hline
& 1
\end{array} \right], $$
and that explicit method we called RK4
$$ \left[ \begin{array}{c|cccc}
0 & 0 & 0 & 0 & 0 \\
\frac 1 2 & \frac 1 2 & 0 & 0 & 0 \\
\frac 1 2 & 0 & \frac 1 2 & 0 & 0 \\
1 & 0 & 0 & 1 & 0 \\
\hline
& \frac 1 6 & \frac 1 3 & \frac 1 3 & \frac 1 6
\end{array} \right] . $$

```python
def Rstability(A, b, z):
    s = len(b)
    def R(z):
        return 1 + z * (b @ np.linalg.solve(np.eye(s) - z*A, np.ones(s)))
    f = np.vectorize(R)
    return f(z)

def plot_rkstability(A, b, name):
    x = np.linspace(-4,4)
    xx, yy = np.meshgrid(x, x)
    zz = xx + 1j*yy
    R = Rstability(A, b, zz)
    plot_stability(xx, yy, R, name)

A_rk4 = np.array([[0,0,0,0],
                    [.5,0,0,0],
                    [0,.5,0,0],
                    [0,0,1,0]])
b_rk4 = np.array([1/6, 1/3, 1/3, 1/6])

plot_rkstability(A_rk4, b_rk4, 'RK4')
```

```python
plot_rkstability(np.eye(1)/2, [1], 'RK Midpoint')
```

## Ballistics

The velocity of a particle in the $x$-$z$ plane is the time derivative of its position,
$$ \begin{bmatrix} v_x \\ v_z \end{bmatrix} = \begin{bmatrix} p_x \\ p_z \end{bmatrix}' . $$
The acceleration (derivative of velocity) depends on the force applied,
$$ \begin{bmatrix} v_x \\ v_z \end{bmatrix}' = \frac 1 m \begin{bmatrix} F_x \\ F_z \end{bmatrix} $$
where $m$ is the mass of the particle.
The gravitational force is
$$ \mathbf F^{\text{grav}} = m \begin{bmatrix} 0 \\ -g \end{bmatrix} $$
where $g = 9.8\ \text{meter}/\text{second}^2 $.
The drag force is often approximated as
$$ F^{\text{drag}} = \frac 1 2 \rho v^2 c_d A $$
where $\rho$ is the density of the fluid (assume $1\ \text{kilogram}/\text{meter}^3$),
$v =\sqrt{v_x^2  + v_z^2}$ is the velocity, $c_d$ is the drag coefficient (assume 0.2 for this projectile), and $A = \pi r^2$  is the cross-sectional area of the projectile (assume the radius is $r = .05\ \text{meter}$).
The direction of the drag force is opposite the velocity, thus the vector drag is
$$ \mathbf F^{\text{drag}}(v_x, v_z) = - \frac{\pi}{2} v c_d r^2 \begin{bmatrix} v_x \\ v_z \end{bmatrix} . $$
This produces the system of equations

$$ \begin{bmatrix} p_x \\ p_z \\ v_x \\ v_z \end{bmatrix}' =
\begin{bmatrix} v_x \\ v_z \\ \frac 1 m F_x^{\text{drag}}  \\ \frac 1 m F_z^{\text{drag}} - 9.8 \end{bmatrix} . $$

For the ballistics computation, you should solve this differential equation with an $m=8$ kilogram projectile and initial condition of a velocity  $v_0 = 300\ \text{meter}/\text{second}$ at angle $\theta$ above the horizontal, i.e.,
$$ \begin{bmatrix} 0 \\ 0 \\ v_0 \cos \theta \\ v_0 \sin \theta \end{bmatrix} . $$

The homework question is to compute the angle needed to hit a target on a distant slope of angle $\phi = 40^\circ = 40 \pi / 180$.  For that, you should adapt one of the ODE solvers to determine where the projectile crosses the plane of the target slope and adjust the angle (either by hand or using a rootfinder) to hit the target.  Don't aim too high or your explosive will detonate on the other side of the mountain!  Make sure you use a time step that you are confident is accurate enough.

Your code should print the angle $\theta$ and the time at which your projectile hits the slope.

### Chemical reactions

The [Oregonator](http://www.scholarpedia.org/article/Oregonator) mechanism in chemical kinetics describes an oscillatory chemical system.  It consists of three species with concentrations $\mathbf x = [x_0,x_1,x_2]^T$ (scaled units) and the evolution equations
$$ \mathbf {x'} = \begin{bmatrix} 77.27 \big(x_1 + x_0 (1 - 8.375\cdot 10^{-6} x_0 - x_1) \big) \\
                    \frac{1}{77.27} \big(x_2 - (1 + x_0) x_1 \big) \\
                    0.161 (x_0 - x_2)
                    \end{bmatrix} . $$
Starting with the initial conditions $\mathbf x_0 = [1, 2, 3]^T$, this produces the time evolution below.

This calculation was performed using 287 steps of an adaptive implicit time integrator, where the time step size was adapted to preserve accuracy.  The time step size in shown in the first panel and the other three are the evolution of concentrations over time.

#### 1. How to measure accuracy?

A differential equation solver produces a time history of chemical concentrations
$$ x(t), \quad 0 \le t \le T_{\text{final}} .$$
We would like a measure of accuracy that does not explicitly depend on the solver or time steps that are used.  If a different method produces the evolution $\tilde x(t)$, then we would like to be able to measure the difference as
$$ \lVert \phi(\tilde x) - \phi(x) \rVert . $$
The function $\phi(\cdot)$ could return the solution at a particular time, it could measure the time between key events, or could be something else.
Choose a metric and explain why you chose it.

#### 2. Implicit versus Explicit

Is an implicit method better than an explicit method for solving the Oregonator?
Solve the Oregonator problem using an explicit method and using an implicit method (you can adapt code from the notebook or implement a method that is not in the notebook).
The Oregonator system has no known analytic solution, so you'll have to numerically compute a reference solution.
Do this using a method you believe to be highly accurate and explain why you believe it is very accurate.
(A common justification would be to refine the time step and compare using the metric $\phi$ above.)

Different methods (implicit versus explicit, Euler versus RK4, etc.) have different costs per time step.  Choose a way to measure cost and explain your choice and any shortcomings that it may have.

Compare the cost (using the metric you chose above) and accuracy (using $\phi$) of the implicit method to the cost and accuracy of the explicit method.  (This is often plotted as *accuracy versus cost* with a log-log scale.)  Explain what factors might go into your decision.

#### 3. Eigenvalues

Calculate the eigenvalues of the Jacobian matrix at different times.  (You can compute it analytically or using the `fdjacobian` code in this notebook.)  Do you notice any qualitative differences?  For example, are there any times when there are eigenvalues with positive real part?

#### 4. Generalization

Realistic chemical mechanisms can have hundreds or thousands of species (the Oregonator has only three).  Explain how the choice of methods might change when solving a large reaction mechanism.


```python
%pylab inline
import numpy as np
from scipy import *
```

## Solving a system of linear equations

## Finding a line passing through 2 points

We are given two points in $R^2$: $[-1,2]$ and $[1,1]$, and we want to find the line passing through these points.

```python

figure(figsize=[8,6])
p=np.array([[-1,2],[1,1]])
plot(p[:,0],p[:,1],'ro')
xlim([-1.1,1.1])
ylim([0,2.1])
grid()
title('we are looking for a line going through these 2 points');
```

Any line (other than vertical) can be expressed as $y=w_0+w_1x$ where $[x,y]$ are points along the line $w_0$ y-axis intercept and $w_1$ is the slope.

To identify the line that passes through the two points we need to find $w_0,w_1$ that satisfy two constraints:

* The point $[x,y]=[-1,2]$ implies the constraint $w_0-w_1=2$
* The point $[x,y]=[1,1]$ implies the constraint $w_0+w_1=1$

### Writing equations in matrix form
There are many ways to solve the equations, i.e. find $w_x,w_y$ that satisfy both constraints.

Here we show two ways to find the solution using `numpy`

We can write the two equations in matrix form as follows:
$$
\begin{equation}
\begin{bmatrix} 1 & -1 \\
	   1 & 1
\end{bmatrix}
\begin{bmatrix} w_0 \\ w_1
\end{bmatrix}
=
\begin{bmatrix} 2 \\ 1
\end{bmatrix}
\end{equation}
$$
Which we can write succinctly as ${\bf A w = b}$

* The coefficients matrix ${\bf A} = \begin{equation}
\begin{bmatrix} 1 & -1 \\
	   1 & 1
\end{bmatrix}
\end{equation} $
* The *ordinate* or *dependent variable* vector: ${\bf b}=\begin{bmatrix} 2 \\ 1	\end{bmatrix}$
* The parameter vector ${\bf w} = \begin{bmatrix} w_0 \\ w_1	\end{bmatrix}$

**Goal:** Find ${\bf w}$ such that  ${\bf A w = b}$

As ${\bf A}$ is invertible we can multiply both sides by ${\bf A^{-1}}$ and get:

${\bf w =  A^{-1} A w = A^{-1} b }$

### Solving using `numpy`

```python
A = np.array([[1,-1], [1,1]])
print('A =\n',A)
b = np.array([[2],[1]])
print('b =\n',b)
```

```python
from numpy.linalg import inv
print("find the solution using inv(A)")
Ainv=inv(A)
print('inv(A) = \n',Ainv)
print('check the Ainv is indeed the inverse of A')
print('A * inv(A)=\n',A.dot(Ainv)) #
w=inv(A).dot(b)
print('Solution: w= inv(A)*b = \n',w)
```

```python
print("alternatively, you can use 'solve()'")
from numpy.linalg import solve
w = solve(A, b)
print('solution: w =\n',w)
```

```python
def F(x,w):
    return w[0]+w[1]*x
print('F(-1)=',F(-1,w))
print('F(1)=',F(1,w))
```

```python
figure(figsize=[8,6])
plot(p[:,0],p[:,1],'ro')
_xlim=[-1.1,1.1]
xlim(_xlim)
ylim([0,2.1])
plot(_xlim,[F(_xlim[0],w),F(_xlim[1],w)])
grid()
title('plotting the line going through the two points');

```

## More than 2 points

Suppose we have three points, rather than two.

There is usually no straight line going through these points.

```python
figure(figsize=[8,6])
p=np.array([[-1,2],[1,1],[0,1.25]])
plot(p[:,0],p[:,1],'ro')
xlim([-1.1,1.1])
ylim([0,2.1])
grid()
title('No straight line goes through these three points');
```

## next video: minimizing square error
When the number of points is larger than the number of dimensions we say that the
system is **over-determined**.

In this case we want to find a line that passes **close to** the points.

### When can you use `numpy.linlag.solve` ?
The function `numpy.linlag.solve` can be used to solve a set of $n$ linear equations with $n$ unknowns.

You simply express the set of equation in a matrix format ${\bf Ax=b}$ and call `solve` using `x=solve(A,b)` the returned value is a vector with the solution for ${\bf x}$.

However the number of equations and the number of unknowns must be equal.
* If the number of equations is **smaller** that the number of unknowns we say that the system is **underdetermined** or **singular**. It means that we don't have enough information to specify a unique solution. In data analysis this would rarely be the case.
* If the number of equations is **larger** than the number of unknowns we say that the system if **overdetermined**. It means that there is no solution that satisfies all of the constraints. In data analysis this is a very common occurance, and we will discuss it in the next notebook.

### Example
Find the solution for the following set of 3 equations:
$$
x+y-z=5 \\
z+2y=3 \\
y-x=4
$$

```python
# Solution
A=np.array([[1,1,-1],[0,2,1],[-1,1,0]])
b=np.array([5,3,4])
solve(A,b)
```

Which means that the solution is $x=-1,y=3,z=-3$

Check that the solution is correct, i.e. that it satisfies the three constraints.

Week 12 _ Regression and PCA/more_lectures/3.Solving_a_system_of_Equations.ipynb


# 画图

sympy可以方便的绘制图形,如果有安装matplotlib,那么它将会使用matplotlib绘制图形

可以通过 `plot()`函数来实例化一个绘图类,实现绘图他可以带如下参数:

+ adaptive: Boolean.
    默认值设置为True。将自适应设置为False，如果需要均匀采样，请指定nb_of_points。

+ depth: int
    自适应算法的递归深度。值n的深度最多取样$2^n$个点。

+ nb_of_points: int.
    当自适应设置为False时使用。
    该函数在nb_of_points点数处统一采样。

+ line_color: float. 指定绘图的颜色。

每个plot对象又有如下参数可以设置

+ title : str 标题
+ xlabel : str x轴标签
+ ylabel : str y轴标签
+ legend : bool 是否使用图例
+ xscale : {‘linear’, ‘log’} x轴坐标按线性还是指数扩展
+ yscale : {‘linear’, ‘log’} y轴坐标按线性还是指数扩展
+ axis : bool 是否画出坐标轴
+ axis_center : tuple of two floats or {‘center’, ‘auto’} 坐标轴中心位置
+ xlim : tuple of two floats
+ ylim : tuple of two floats
+ aspect_ratio : tuple of two floats or {‘auto’}
+ autoscale : bool 是否自动扩展
+ margin : float in [0, 1] 边的比例

```python
%matplotlib inline
```

## 2维图形

+ ### 画单一函数

`plot(expr, range, **kwargs)`

```python
from sympy import symbols
from sympy.plotting import plot
x,y = symbols('x y')
p1 = plot(x*x)
```

```python
p2 = plot(x)
```

+ ### 多图合并

plot对象可以将多张图合并到一起,单独一张合并使用`append()`,多个可以使用`extend`

```python
p1.append(p2[0])
p1.axis=True
p1.show()
```

+ ### 参数范围

plot也可以指定参数范围

```python
plot(x**2,(x,-5,5,))
```

+ ### 同时画多个

`plot(expr1, expr2, ..., range, **kwargs)`

```python
plot(x,x**2,x**3,(x,-5,5,))
```

+ ### 多个但不同取值范围

`plot((expr1, range), (expr2, range), ..., **kwargs)`

```python
plot((x**2, (x, -6, 6)), (x, (x, -5, 5)))
```

+ ### 自适应采样

```python
plot(x**2, adaptive=False, nb_of_points=400)
```

+ ### 设定颜色

sympy的颜色设定有点类似matplotlib的,可以指定特定字符作为标记,比如红色就是"r"

```python
y = symbols("y")
fz = x**2-y**2
```

```python
plot(x**2,line_color='r')
```

## 绘制3d图形

同时,sympy也可以绘制3d图形,与2d图形形式差不多

+ `plot3d(expr, range_x, range_y, **kwargs)` 绘制一个算式的图形

+ `plot3d(expr1, expr2, range_x, range_y, **kwargs)` 绘制一个取值范围中的多个图形

+ `plot3d((expr1, range_x, range_y), (expr2, range_x, range_y), ..., **kwargs)`绘制多取值范围内的多图形

```python
from sympy.plotting import plot3d
```

```python
p3d = plot3d(x**2+y**2)
```


# How to Break Math
## Numerical Computing with Floating-Point Numbers

Real numbers may have an infinite number of decimal digits: for example, $\frac{1}{3} = 0.333\dots$, and $\pi = 3.14159\dots$. However, computers do not have an infinite amount of storage, and for efficiency, most computer programs represent real numbers using floating-point values.

Floating point is similar to scientific notation. For example, the speed of light can be written simply as $299792458\,ms^{-1}$ or in scientific notation, $2.99792458 \times 10^8\,ms^{-1}$. A number in scientific notation has three parts: the *significand* or *mantissa*, the *base*, and the *exponent*:
\begin{equation}
299792458 = \underbrace{2.99792458}_{\text{significand}} \times {\underbrace{10}_{\text{base}}}^{\hspace{-0.6em}\overbrace{8}^{\text{exponent}}}
\end{equation}

By convention, the significand has one non-zero digit before the point (unless the number is exactly zero).

In binary floating point, the base is always 2, and the signficand is made up entirely of the digits 0 and 1. For example, the number 42 can be represented as a binary integer as 101010, or in binary floating point:
\begin{equation}
42 = 1.01010 \times 2^5
\end{equation}

Computers use the [IEEE-754 standard floating point format](https://en.wikipedia.org/wiki/IEEE_754), which defines standard "single" and "double" precision floating-point numbers. The bits which represent an IEEE-754 value are divided into three fields: the sign bit, the exponent, and the significand.

A sign bit of `0` represents a positive number, while a sign bit of `1` represents a negative number.

The leading '1' in the significand is not stored, so the value `0b0100...` represents the signficand '1.0100...'.

The exponent is *biased* - it is read as an integer value, from which the value of the bias is subtracted. For example, in single precision the bias is 127, so the exponent `0b10000100` (decimal 132) is read as (132-127) = 5, and the exponent `0b01111100` (decimal 124) is read as (124-127) = -3.

```julia
function print_bitstring(a::Float32)
    s = bitstring(a)
    printstyled(s[1], color=:blue)
    print(" ")
    printstyled(s[2:9], color=:green)
    print(" ")
    printstyled(s[10:32], color=:red)
    println()
end

function print_bitstring(a::Float64)
    # TODO complete - print the bits of the double-precision floating-point number a, separated into fields
end

function exponent_ieee754(a::Float32)
    s = bitstring(a)
    e = parse(Int32,s[2:9],base=2)
    return e - (2^7-1)
end

function exponent_ieee754(a::Float64)
    # TODO complete - extract exponent field from the double-precision floating point number a
end

x = Float32(42)
print(x," = ")
print_bitstring(x)
println("exponent = ", exponent_ieee754(x))

y = Float32(0.15625)
print(y," = ")
print_bitstring(y)
println("exponent = ", exponent_ieee754(y))

```

### Rounding Error

Rounding error occurs whenever an input value or the result of a calculation cannot be represented exactly in the chosen floating-point format, for example:

```julia
print_bitstring(0.1)
print_bitstring(Float64(pi))
print_bitstring(1.0/3.0)
```

### Cancellation

Cancellation occurs when two close-together values are subtracted, leading to a large relative error in the result. For example:

```julia
epsilon = 1e-8
x = (1+epsilon)
print("x ")
print_bitstring(x)
y = (1.0)
print("y ")
print_bitstring(y)
difference = epsilon
z = x-y
print("z ")
print_bitstring(z)
println("Correct result is ", difference, " but got ", z, " (relative error ", (difference-z)/difference, ")")
```

## Rump's Example

Siegfried Rump created a [nasty example](http://www.ti3.tu-harburg.de/paper/rump/Ru88a.pdf) of floating point error. He asks us to compute the value of the function:

$333.75 b^6 + a^2 (11 a^2 b^2 – b^6 – 121 b^4 – 2) + 5.5 b^8 + \frac{a}{(2b)}$

for $a = 77617.0$ and $b = 33096.0$ . We can define a Julia function to compute the value using floating-point numbers of type `T`:

```julia
function rump(T)
  a = T(77617.0)
  b = T(33096.0)
  a2 = a*a
  b2 = b*b
  b4 = b2*b2
  b6 = b4*b2
  b8 = b4*b4
  return T(333.75)*b6 + a2 * (11*a2*b2 - b6 - 121*b4 - 2) + T(5.5)*b8 + (a/(2*b))
end
```

Let's compute that function in single-precision floating point:

```julia
rump(Float32)
```

To check the correctness, let's compute it again in double-precision:

```julia
rump(Float64)
```

Oh dear! Our results aren't even remotely the same - they're out by more than eight orders of magnitude.

Let's try increasing the precision still further. Julia supports arbitrary-precision arithmetic using [GNU MPFR](https://www.mpfr.org/) with a special type `BigFloat`. We can use this to create a function that will compute Rump's example using a chosen number of bits of precision in the significand:

```julia
function rump(numbits::Int)
  Base.MPFR.setprecision(numbits)
  return rump(BigFloat)
end
```

First, let's replicate our previous results using BigFloat. First single-precision, which in [IEEE-754](https://en.wikipedia.org/wiki/IEEE_754) has 24 bits in the significand:

```julia
rump(24)
```

Next, double precision, which has 53 bits in the significand:

```julia
rump(53)
```

Now let's try computing in quadruple precision, which in IEEE-754 has 113 bits in the significand:

```julia
rump(113)
```

This looks even worse - now even the sign of the result has changed!

```julia
for i = 113:130
    println("Precision ", i, " result is ", rump(i))
end
```

Rump gives the exact answer as $\frac{-54767}{66192} = -0.8273960599\dots$

Why do we need such high precision to correctly compute Rump's example? Use the `bitstring_ieee754(Float64)` function you created earlier to print the representation of some of the subexpressions that are computed in the function, to see if you can understand why the error is so great.

```julia
function rump_print(T)
  a = T(77617.0)
  b = T(33096.0)
  a2 = a*a
  b2 = b*b
  b4 = b2*b2
  b6 = b4*b2
  b8 = b4*b4
  # TODO print some of the subexpressions computed below
  intermediate_value1 = # ?
  intermediate_value2 = # ?
  print_bitstring(intermediate_value1)
  print_bitstring(intermediate_value2)
  return T(333.75)*b6 + a2 * (11*a2*b2 - b6 - 121*b4 - 2) + T(5.5)*b8 + (a/(2*b))
end

result = rump_print(Float64)
print_bitstring(result)
```

# Patriot vs. Scud

The [Patriot missile battery](http://www-users.math.umn.edu/~arnold//disasters/patriot.html) counted time using a clock which ticked ten times each second. To compute the expected location of an incoming target, the number of ticks was divided by 10 to produce a fixed-point value representing the number of seconds since the system started up. We can simulate this using a single-precision floating point value.

Modify the code below to compute the actual time in seconds given the current value of the clock. Why is `current_time_seconds` different? Hint: try printing out the bitstring representation of `tick_length`.

```julia
using Printf
const TICKS_PER_SECOND = 10
const SCUD_MISSILE_SPEED_METRES_PER_SECOND = 1676

function on_the_hour(ticks)
  return (ticks % (TICKS_PER_SECOND * 60 * 60) == 0)
end

Base.MPFR.setprecision(20)
tick_internal = BigFloat(1 / TICKS_PER_SECOND)
tick_length = Float32(tick_internal)

current_ticks = 0
current_time_seconds = 0.0
one_hundred_hours = 100 * 60 * 60 * TICKS_PER_SECOND
for i=0:one_hundred_hours
    current_ticks += 1
    current_time_seconds += tick_length
    if (on_the_hour(current_ticks))
        actual_time_seconds = # TODO compute correct time in seconds
        h = actual_time_seconds / 3600
        error_time_seconds = actual_time_seconds - current_time_seconds
        @printf("hour = %3d; error (time) = %.2e s; error (target position) %.2e m\n", h,
            error_time_seconds, error_time_seconds*SCUD_MISSILE_SPEED_METRES_PER_SECOND)
    end
end
```


1. Adapt the Gradient_univ for polynomial 2x**2 +2x +1
2. Adapt the program for x**4

```python
from sympy import *
from sympy.matrices import Matrix
import numpy as np
from matplotlib import pyplot as plt

x = Symbol('x')

# Function
y = 2*x**2 +2*x + 1
# First derivative with respect to x
yprime = y.diff(x)

# Initial sequence
theta = 2
theta2 = 0

# Initiale the step size
alpha = .001

iterations = 0
check = 0
precision = 1/100000
iterationsMax = 100000

while True:
    theta2 = theta - alpha*yprime.evalf(subs={x:theta})
    iterations += 1

    # If we make too much iterations our program
    # stops and we need to check it to be sure the
    # parameters are correct and it is working properly
    if iterations > iterationsMax:
        print("Too many iterations")
        break

    if abs(theta - theta2) < precision:
        break

    theta = theta2

print("Number of iterations:",iterations,"value of theta:",theta2,sep=" ")
plt.plot(theta,y.evalf(subs={x:theta}),marker='o',color='r')
space = np.linspace(-5,5,100)
data = np.array([y.evalf(subs={x: value}) for value in space])
plt.plot(space, data)
plt.show()
```

```python
# Function
y = x**4
# First derivative with respect to x
yprime = y.diff(x)

# Initial sequence
theta = 2
theta2 = 0

# Initiale the step size
alpha = .0001

iterations = 0
check = 0
precision = 1/100000
iterationsMax = 100000

while True:
    theta2 = theta - alpha*yprime.evalf(subs={x:theta})
    iterations += 1

    # If we make too much iterations our program
    # stops and we need to check it to be sure the
    # parameters are correct and it is working properly
    if iterations > iterationsMax:
        print("Too many iterations")
        break

    if abs(theta - theta2) < precision:
        break

    theta = theta2

print("Number of iterations:",iterations,"value of theta:",theta2,sep=" ")
plt.plot(theta,y.evalf(subs={x:theta}),marker='o',color='r')
space = np.linspace(-5,5,100)
data = np.array([y.evalf(subs={x: value}) for value in space])
plt.plot(space, data)
plt.show()
```

1. Make a 2 by 2 matrix of second partial derivatives (Hessian)
2. Adapt the program for a function of 2 variables. Precision?

```python
x,y = symbols('x y')
#function
z = x**2 + y**2

# Hessian
f11 = z.diff(x).diff(x)
f12 = z.diff(x).diff(y)
f21 = z.diff(y).diff(x)
f22 = z.diff(y).diff(y)
hessian = np.array([[f11, f12], [f21, f22]])
H = sympy.Matrix(hessian)
det = H.det()
print(hessian)
print('The determinant is', det)

if det >= 0:
    print('The matrix positive semidefinite!')
else:
    print('The matrix is not positive semidefinite!')
#det = f11 * f22 - f12 * f21
```

    [[2 0]
     [0 2]]
    The determinant is 4
    The matrix positive semidefinite!

```python
# Function
z = x**2 + y**2 + 1

zprimex = z.diff(x)
zprimey = z.diff(y)

# Initial sequence
thetaX = 2
thetaY = 2

theta2X = 0
theta2Y = 0

# Initiale the step size
alpha = .0001

iterations = 0
check = 0
precision = 1/100000
iterationsMax = 100000

while True:
    theta2X = thetaX - alpha*zprimex.evalf(subs={x:thetaX, y:thetaY})
    theta2Y = thetaY - alpha*zprimey.evalf(subs={x:thetaX, y:thetaY})

    iterations += 1

    # If we make too much iterations our program
    # stops and we need to check it to be sure the
    # parameters are correct and it is working properly
    if iterations > iterationsMax:
        print("Too many iterations")
        break

    if ((thetaX - theta2X)**2 + (thetaY - theta2Y)**2) < precision**2:
        break

    thetaX = theta2X
    thetaY = theta2Y

print("Number of iterations:",iterations,"value of theta:", theta2X, theta2Y,sep=" ")
#plt.plot(theta,y.evalf(subs={x:theta}),marker='o',color='r')
#space = np.linspace(-5,5,100)
#data = np.array([y.evalf(subs={x: value}) for value in space])
#plt.plot(space, data)
#plt.show()
```

    Number of iterations: 20177 value of theta: 0.0353429474378087 0.0353429474378087

```python
# Crazy Function
z = log(1-y**2 + x**4)

zprimex = z.diff(x)
zprimey = z.diff(y)

# Initial sequence
thetaX = 2
thetaY = 2

theta2X = 0
theta2Y = 0

# Initiale the step size
alpha = .0001

iterations = 0
check = 0
precision = 1/100000
iterationsMax = 100000

while True:
    theta2X = thetaX - alpha*zprimex.evalf(subs={x:thetaX, y:thetaY})
    theta2Y = thetaY - alpha*zprimey.evalf(subs={x:thetaX, y:thetaY})

    iterations += 1

    # If we make too much iterations our program
    # stops and we need to check it to be sure the
    # parameters are correct and it is working properly
    if iterations > iterationsMax:
        print("Too many iterations")
        break

    if ((thetaX - theta2X)**2 + (thetaY - theta2Y)**2) < precision**2:
        break

    thetaX = theta2X
    thetaY = theta2Y

print("Number of iterations:",iterations,"value of theta:", theta2X, theta2Y,sep=" ")
#plt.plot(theta,y.evalf(subs={x:theta}),marker='o',color='r')
#space = np.linspace(-5,5,100)
#data = np.array([y.evalf(subs={x: value}) for value in space])
#plt.plot(space, data)
#plt.show()
```

    Too many iterations
    Number of iterations: 100001 value of theta: 13.6925203998118 -8.48404427992362

```python

```

    -2.0000000000000004

```python

```


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of a Damped Spring Pendulum

The damped [spring pendulum](https://en.wikipedia.org/wiki/Spring_pendulum) is an example for a physical system that can be modeled by a linear ordinary differential equation (ODE) with constant coefficients. In view of the theory of signals and systems it hence can be interpreted as a linear time-invariant (LTI) system. The mechanical properties of the damped spring pendulum are analyzed by using the theory of LTI systems. The underlying mechanical setup is depicted in the following

A rigid body of mass $m$ is mounted on a spring with stiffness $k$ which is connected to the ground. A damper with viscous damping coefficient $c$ is mounted parallel to the spring to model the friction present in the system. It is assumed that the movement of the mass over time is restricted to the vertical axis, here denoted by $y$. It is assumed further that the mass is in its idle position for $t<0$. The pretension of the spring by the mass can be neglected this way. It is additionally assumed that the mass is not moving for $t<0$. Denoting the displacement of the mass over time with $y(t)$, these initial conditions are formulated as $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$.

The normalized values $m = 0.1$, $c = 0.1$, $k = 2.5$ are used for illustration in the following.

### Differential Equation

The differential equation of the mechanical system is derived by considering the force equilibrium at the mass

\begin{equation}
F_\text{S}(t) + F_\text{F}(t) + F_\text{I}(t) = F_\text{E}(t)
\end{equation}

where $F_\text{E}(t)$ denotes an external force acting onto the mass, the other forces are derived in the sequel. The force $F_\text{S}(t)$ induced by the spring is given by [Hooke's law](https://en.wikipedia.org/wiki/Hooke%27s_law)

\begin{equation}
F_\text{S}(t) = k y(t)
\end{equation}

Its common to model the frictional force $F_\text{F}(t)$ as being proportional to the velocity of the mass

\begin{equation}
F_\text{F}(t) = c \frac{d y(t)}{dt}
\end{equation}

The inertial force $F_\text{I}(t)$ due to the acceleration of the mass is given as

\begin{equation}
F_\text{I}(t) = m \frac{d^2 y(t)}{dt^2}
\end{equation}

Introducing the forces into the force equilibrium yields the differential equation describing the displacement of the damped spring pendulum

\begin{equation}
m \frac{d^2 y(t)}{dt^2} + c \frac{d y(t)}{dt} + k y(t) = F_\text{E}(t)
\end{equation}

as a consequence of the external force.
Above equation constitutes an ODE with constant coefficients. It can be interpreted as an LTI system with the external force as input signal $x(t) = F_\text{E}(t)$ and the displacement of the mass as output signal $y(t)$.

### Comparison to Passive Electrical Networks

Comparing the ODEs of the damped spring pendulum and the [second-order analog low-pass](http://localhost:8888/notebooks/systems_time_domain/network_analysis.ipynb#Differential-Equation) yields that both constitute second-order ODEs with constant coefficients. Dividing the ODE of the second-order analog low pass by $C$ results in

\begin{equation}
L \frac{d^2 u_\text{o}(t)}{dt^2} + R \frac{d u_\text{o}(t)}{dt} + \frac{1}{C} u_\text{o}(t) = \frac{1}{C} u_\text{i}(t)
\end{equation}

where $u_\text{i}(t)$ and $u_\text{o}(t)$ denote the in- and output voltage of the analog circuit.
Comparison with above ODE of the spring pendulum yields the [equivalence of both systems](https://en.wikipedia.org/wiki/System_equivalence) for

| | 2nd-order low-pass | spring pendulum |
|:---|:---|:---|
| input signal $x(t)$ | $u_\text{i}(t) = F_\text{E}(t) C$ | $F_\text{E}(t) = \frac{u_\text{i}(t)}{C}$ |
| output signal $y(t)$ | $u_\text{o}(t)$ | $y(t)$ |
| | $L = m$ | $m = L$ |
| | $R = c$ | $c = R$ |
| | $C = \frac{1}{k} $ | $k = \frac{1}{C}$ |

Note, the equivalence between mechanical systems described by ODEs with constant coefficients and analog circuits was used to simulate such systems by [analog computers](https://en.wikipedia.org/wiki/Analog_computer).

### Impulse Response

The LTI system corresponding to the pendulum can be characterized by its [impulse response](impulse_response.ipynb) $h(t)$. It is defined as the output of the system for a Dirac impulse $x(t) = \delta(t)$ at the input. Physically this can be approximated by hitting the mass. The impulse response characterizes the movement $y(t)$ of the mass after such an event.

First the ODE of the spring pendulum is defined in `SymPy`

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t, m, c, k = sym.symbols('t m c k', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(m*y.diff(t, 2) + c*y.diff(t) + k*y, x)
ode
```

The normalized values of the physical constants are stored in a dictionary for ease of later substitution

```python
mck = {m: 0.1, c: sym.Rational('.1'), k: sym.Rational('2.5')}
mck
```

The impulse response is calculated by explicit solution of the ODE.

```python
solution_h = sym.dsolve(ode.subs(x, sym.DiracDelta(t)).subs(y, sym.Function('h')(t)))
solution_h
```

The integration constants $C_1$ and $C_2$ have to be determined from the initial conditions $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$.

```python
integration_constants = sym.solve( (solution_h.rhs.limit(t, 0, '-'), solution_h.rhs.diff(t).limit(t, 0, '-')), ['C1', 'C2'] )
integration_constants
```

Substitution of the values for the integration constants $C_1$ and $C_2$ into the result from above yields the impulse response of the spring pendulum

```python
h = solution_h.subs(integration_constants).rhs
h
```

The impulse response is plotted for the specific values of $m$, $c$ and $k$ given above

```python
sym.plot(h.subs(mck), (t,0,12), ylabel=r'h(t)');
```

### Transfer Function

For an exponential input signal $x(t) = e^{s t}$, the [transfer function](eigenfunctions.ipynb#Transfer-Function) $H(s)$ represents the weight of the exponential output signal $y(t) = H(s) \cdot e^{s t}$. The transfer function is derived by introducing $x(t)$ and $y(t)$ into the ODE and solving for $H(s)$

```python
s = sym.symbols('s')
H = sym.Function('H')(s)

H, = sym.solve(ode.subs(x, sym.exp(s*t)).subs(y, H*sym.exp(s*t)).doit(), H)
H
```

The transfer characteristics of an LTI system for harmonic exponential signals $e^{j \omega} = \cos(\omega t) + j \sin(\omega t)$ are of special interest in the analysis of resonating systems. It can be derived from $H(s)$ by substituting the complex frequency $s$ with $s = j \omega$. The resulting transfer function $H(j \omega)$ provides the attenuation and phase the system adds to a harmonic input signal.

```python
w = sym.symbols('omega', real=True)

Hjw = H.subs(s, sym.I * w)
Hjw
```

The magnitude of the transfer function $|H(j \omega)|$ is plotted for the specific values of the elements given above

```python
sym.plot(abs(Hjw.subs(mck)), (w, -15, 15), ylabel=r'$|H(j \omega)|$', xlabel=r'$\omega$');
```

When inspecting the magnitude of the transfer function it becomes evident that the damped spring pendulum shows resonances (maxima) for two specific angular frequencies. These resonance frequencies $\omega_0$ are calculated by inspecting the extreme values of $|H(j \omega)|$. First the derivative of $|H(j \omega)|$ with respect to $\omega$ is computed and set to zero

```python
extrema = sym.solve(sym.Eq(sym.diff(abs(Hjw), w),0),w)
extrema
```

For the maxima of the transfer function only the 2nd and 3rd extrema are of interest

```python
w0 = extrema[1:3]
w0
```

The resonance frequencies are computed for the specific values of $m$, $c$ and $k$ given above

```python
[w00.subs(mck) for w00 in w0]
```

The phase of the transfer function $\varphi(j \omega)$ is computed and plotted for the specific values of the elements given above

```python
phi = sym.arg(Hjw)
sym.plot(phi.subs(mck), (w, -15, 15), ylabel=r'$\varphi(j \omega)$', xlabel=r'$\omega$');
```

**Exercise**

* Change the viscous damping coefficient $c$ of the spring pendulum and investigate how the magnitude and phase of the transfer function $H(j \omega)$ changes.

* How does the frequency of the damped harmonic oscillation in the impulse response relate to the resonance frequency?

### Application: Vibration Isolation

An application of above example is the design of [vibration isolation](https://en.wikipedia.org/wiki/Vibration_isolation) by a damped spring pendulum. An typical example is a rotating machinery with mass $m$ which has some sort of imbalance. Assuming that the imbalance can be modeled as a rotating mass, the external force $F_\text{E}(t)$ is given by the vertical component of its [centrifugal force](https://en.wikipedia.org/wiki/Centrifugal_force)

\begin{equation}
F_\text{E}(t) = F_0 \sin(\omega t) = F_0 \cdot \Im \{e^{j \omega t} \}
\end{equation}

where $\omega$ denotes the angular frequency of the rotating machinery and

\begin{equation}
F_0 = m_\text{I} r \omega^2
\end{equation}

the amplitude of the force with $m_\text{I}$ denoting the mass of the imbalance and $r$ the radius of its circular orbit. Since $e^{j \omega t}$ is an eigenfunction of the LTI system, the resulting displacement is then given as

\begin{equation}
y(t) = F_0 \cdot \Im \{e^{j \omega t} H(j \omega) \}
\end{equation}

The aim of vibration isolation is to keep the magnitude of the displacement as low as possible.

**Exercise**

* Compute and plot the displacement for given $m_\text{I}$ and $r$.

* For which angular frequencies $\omega$ is the magnitude of the displacement largest? How is the phase relation between the external force $F_\text{E}(t)$ and displacement $y(t)$ at these frequencies?

* How should the resonance frequencies $\omega_0$ of the spring pendulum be chosen in order to get a good vibration isolation for a machine rotating with angular frequency $\omega$? How is the phase relation between the external force $F_\text{E}(t)$ and displacement $y(t)$ at this frequency?

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# Optimización

```python
%matplotlib notebook
from sympy import *
import numpy as np
import matplotlib.pyplot as plt
init_printing()
```

```python
#problema 1
var ("x y f")
f = 4*(x**2) - y**2 +80*x +20*y -10

dfx = f.diff(x)
dfy = f.diff(y)

pc = solve([dfx,dfy],[x,y])

dfxx =dfx.diff(x)
dfyy =dfy.diff(y)
dfxy =dfx.diff(y)
D = dfxx*dfyy - dfxy
D.subs(pc),dfxx.subs(pc), "punto silla"
```

    (-16, 8, 'punto silla')

```python
#problema2
var("x y f")
f = x**2 + x*y -5*x - 2*(y**2)+ 2*y

dfx = f.diff(x)
dfy = f.diff(y)

pc = solve([dfx,dfy],[x,y])

dfxx =dfx.diff(x)
dfyy =dfy.diff(y)
dfxy =dfx.diff(y)
D = dfxx*dfyy - dfxy
D.subs(pc),dfxx.subs(pc), "punto silla"
```

    (-9, 2, 'punto silla')

```python
#problema3
var("x y f")

f = (x**3)/3 -5*(x**2)/2 + 3*(y**2) -12*y

nablaf =[f.diff(var) for var in [x, y]]

pcs = solve(nablaf)

H =hessian(f,[x,y])

for pc in pcs:
    eig = H.subs(pc).eigenvals()
    print ("punto critico:",pc, "sus eigenvalores son:",eig)
print ("1º punto critico es punto silla")
print ("2º punto critico es un minimo")

```

    punto critico: {x: 0, y: 2} sus eigenvalores son: {6: 1, -5: 1}
    punto critico: {x: 5, y: 2} sus eigenvalores son: {6: 1, 5: 1}
    1º punto critico es punto silla
    2º punto critico es un minimo

```python
#problema4
var("x y f")

f = -8*(x**2) +12*x*y +44*x -12*(y**2) +12*y

dfx = f.diff(x)
dfy = f.diff(y)

pc = solve([dfx,dfy],[x,y])

dfxx =dfx.diff(x)
dfyy =dfy.diff(y)
dfxy =dfx.diff(y)
D = dfxx*dfyy - dfxy
D.subs(pc),dfxx.subs(pc), "máximo"
```

    (372, -16, 'máximo')

```python
#problema5

var("x y f")

f = 3*(x**2) -4*x*y +3*(y**2) + 8*x -17*y +5

dfx = f.diff(x)
dfy = f.diff(y)

pc = solve([dfx,dfy],[x,y])

dfxx =dfx.diff(x)
dfyy =dfy.diff(y)
dfxy =dfx.diff(y)
D = dfxx*dfyy - dfxy
D.subs(pc),dfxx.subs(pc), "minimo"
```

    (40, 6, 'minimo')

  6. Una compañía fabrica y venden dos modelos de lámpara L1 y L2. Para su fabricación se necesita un trabajo manual de 20 minutos para el modelo L1 y de 30 minutos para el L2; y un trabajo de máquina para L1 y de 10 minutos para L2. Se dispone para el trabajo manual de 100 horas al mes y para la máquina 80 horas al mes. Sabiendo que el beneficio por unidad es de 15 y 10 euros para L1 y L2, respectivamente, planificar la producción para obtener el máximo beneficio.

```python
!pip install pulp
from pulp import *
```

    Requirement already satisfied: pulp in /anaconda3/lib/python3.6/site-packages (1.6.8)
    Requirement already satisfied: pyparsing>=2.0.1 in /anaconda3/lib/python3.6/site-packages (from pulp) (2.2.0)
    distributed 1.21.8 requires msgpack, which is not installed.
    You are using pip version 10.0.1, however version 18.0 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.

```python
problema = LpProblem("Problema lamparas",LpMaximize)
x=LpVariable("Lampara 1",0,None,LpInteger)
y=LpVariable("Lampara 2",0,None,LpInteger)

problema+= 15*x + 10*y

problema+= (1/3)*x + (1/2)*y <= 100
problema+= (1/3)*x + (1/6)*y <= 80

problema.solve()
print(LpStatus[problema.status])

for i in problema.variables():
    print(i.name,"=",i.varValue)
```

    Optimal
    Lampara_1 = 210.0
    Lampara_2 = 60.0

7. Con el comienzo del curso se va a lanzar unas ofertas de material escolar. Unos almacenes quieren ofrecer 600 cuadernos, 500 carpetas y 400 bolígrafos para la oferta, empaquetándolo de dos formas distintas; en el primer bloque pondrá 2 cuadernos, 1 carpeta y 2 bolígrafos; en el segundo, pondrán 3 cuadernos, 1 carpeta y 1 bolígrafo. Los precios de cada paquete serán 6.5 y 7 €, respectivamente. ¿Cuántos paquetes le conviene poner de cada tipo para obtener el máximo beneficio?

```python
prob = LpProblem("Problema almacenes",LpMaximize)
a=LpVariable("Paquete 1",0,None,LpInteger)
b=LpVariable("Paquete 2",0,None,LpInteger)

prob+= 6.5*a + 7*b

prob+= 2*a + 3*b <= 600
prob+= a + b <=500
prob+= 2*a + b <=400

prob.solve()
print(LpStatus[prob.status])

for i in prob.variables():
    print(i.name,"=",i.varValue)
```

    Optimal
    Paquete_1 = 150.0
    Paquete_2 = 100.0

8. En una granja de pollos se da una dieta, para engordar, con una composición mínima de 15 unidades de una sustancia A y otras 15 de una sustancia B. En el mercado sólo se encuentra dos clases de compuestos: el tipo X con una composición de una unidad de A y 5 de B, y el otro tipo, Y, con una composición de cinco unidades de A y una de B. El precio del tipo X es de 10 euros y del tipo Y es de 30 €. ¿Qué cantidades se han de comprar de cada tipo para cubrir las necesidades con un coste mínimo?

```python
proble = LpProblem("Pollos",LpMinimize)
c=LpVariable("X",0,None,LpInteger)
d=LpVariable("Y",0,None,LpInteger)

proble+= 10*c + 30*d

proble+= c +5*d >= 15
proble+= 5*c + d >= 15

proble.solve()
print(LpStatus[proble.status])

for e in proble.variables():
    print(e.name,"=",e.varValue)
```

    Optimal
    X = 5.0
    Y = 2.0

9. Se dispone de 600 g de un determinado fármaco para elaborar pastillas grandes y pequeñas. Las grandes pesan 40 g y las pequeñas 30 g. Se necesitan al menos tres pastillas grandes, y al menos el doble de pequeñas que de las grandes. Cada pastilla grande proporciona un beneficio de 2 € y la pequeña de 1 €. ¿Cuántas pastillas se han de elaborar de cada clase para que el beneficio sea máximo?

```python
pro = LpProblem("Pastillas",LpMaximize)
g=LpVariable("Pasilla Grande",0,None,LpInteger)
p=LpVariable("Pastilla Pequeña",0,None,LpInteger)

pro+=2*g + p

pro+= 40*g +30*p <= 600
pro+= g >=3
pro+= p >= 2*g

pro.solve()
print(LpStatus[pro.status])

for f in pro.variables():
    print(f.name,"=",f.varValue)
```

    Optimal
    Pasilla_Grande = 6.0
    Pastilla_Pequeña = 12.0

10. Una escuela prepara una excursión para 400 alumnos. La empresa de transporte tiene 8 autobuses de 40 plazas y 10 de 50 plazas, pero sólo dispone de 9 conductores. El alquiler de un autocar grande cuesta 800 € y el de uno pequeño 600 €. Calcular cuántos autobuses de cada tipo hay que utilizar para que la excursión resulte lo más económica posible para la escuela.

```python
problemita = LpProblem("Autobuses",LpMinimize)
ag=LpVariable("Autobus Grande",0,None,LpInteger)
ap=LpVariable("Autobus Pequeña",0,None,LpInteger)

problemita+= 800*ag + 600*ap

problemita+= ag + ap <= 9
problemita+= 40*ap + 50*ag >= 400

problemita.solve()
print(LpStatus[problemita.status])

for g in problemita.variables():
    print(g.name,"=",g.varValue)
```

    Optimal
    Autobus_Grande = 4.0
    Autobus_Pequeña = 5.0

11. Nota: El peso de una persona se puede adivinar, en parte, por su altura. Los siguientes datos muestran su descriptiva en 30 individuos (ejemplo simulado).

- Genera una grafica de la mejor curva que se ajuste a esos datos.
- ¿Cuál es el peso aproximado de una persona con una altura de 195 cm?

```python
import numpy as np

np.random.seed(12345)
altura = np.random.normal(171,10,30) # rnorm(30,171,8) # Generar la altura
peso = altura-100+ np.random.normal(0,10,30) # Generar el peso

var("x y")
from scipy.optimize import curve_fit
from scipy.stats import linregress

aprox =linregress(altura,peso)
linregress(altura,peso)

x=float(input("Ingrese altura"))
y= aprox.intercept+aprox.slope*x
print("peso:",y)
```

    Ingrese altura150
    peso: 48.22166748493679

```python
plt.plot(altura,peso,"o")
plt.plot(altura,aprox.intercept+aprox.slope*altura,"-")
plt.plot(x,y,"o")
plt.grid()
plt.show()
```

    <IPython.core.display.Javascript object>

12. El conjunto de datos proviene del repositorio UCI Machine Learning y está relacionado con campañas de marketing directo (llamadas telefónicas) de una institución bancaria portuguesa.

El objetivo de la clasificación es predecir si el cliente se suscribirá (1/0) a un depósito a plazo (variable y). El conjunto de datos se puede descargar desde aquí.

https://raw.githubusercontent.com/madmashup/targeted-marketing-predictive-engine/master/banking.csv

```python
!ls
```

    ExamenMod.ipynb suscripcion.csv

```python
import os
import pandas as pd

suscripcion =pd.read_csv(os.path.join("suscripcion.csv"))
print(suscripcion.columns)

labels= suscripcion.y.values
features=suscripcion[["age","default","housing","loan","cons_price_idx","cons_conf_idx"]]

features.head()
pd.get_dummies(features).head()
features_dummies=pd.get_dummies(features,columns=["default","housing","loan"])
features_dummies.head(n=16)

data=features_dummies.values
data[0]

import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import Imputer

train_data,test_data,train_labels,test_labels =train_test_split(data,labels,random_state=0)
imp=Imputer()
imp.fit(train_data)
train_data_finite=imp.transform(train_data)
test_data_finite=imp.transform(test_data)

np.isnan(train_data_finite).any()

from sklearn.linear_model import LogisticRegression
lr=LogisticRegression().fit(train_data_finite,train_labels)

print("Logistic Regression Score:%f"% lr.score(test_data_finite, test_labels))

```

    Index(['age', 'job', 'marital', 'education', 'default', 'housing', 'loan',
           'contact', 'month', 'day_of_week', 'duration', 'campaign', 'pdays',
           'previous', 'poutcome', 'emp_var_rate', 'cons_price_idx',
           'cons_conf_idx', 'euribor3m', 'nr_employed', 'y'],
          dtype='object')
    Logistic Regression Score:0.889191

### Input variables

- age (numeric)
- job : type of job (categorical: “admin”, “blue-collar”, “entrepreneur”, “housemaid”, “management”, “retired”, “self-employed”, “services”, “student”, “technician”, “unemployed”, “unknown”)
- marital : marital status (categorical: “divorced”, “married”, “single”, “unknown”)
- education (categorical: “basic.4y”, “basic.6y”, “basic.9y”, “high.school”, “illiterate”, “professional.course”, “university.degree”, “unknown”)
- default: has credit in default? (categorical: “no”, “yes”, “unknown”)
- housing: has housing loan? (categorical: “no”, “yes”, “unknown”)
- loan: has personal loan? (categorical: “no”, “yes”, “unknown”)
- contact: contact communication type (categorical: “cellular”, “telephone”)
- month: last contact month of year (categorical: “jan”, “feb”, “mar”, …, “nov”, “dec”)
- day_of_week: last contact day of the week (categorical: “mon”, “tue”, “wed”, “thu”, “fri”)
- duration: last contact duration, in seconds (numeric). Important note: this attribute highly affects the output target (e.g., if duration=0 then y=’no’). The duration is not known before a call is performed, also, after the end of the call, y is obviously known. Thus, this input should only be included for benchmark purposes and should be discarded if the intention is to have a realistic predictive model
- campaign: number of contacts performed during this campaign and for this client (numeric, includes last contact)
- pdays: number of days that passed by after the client was last contacted from a previous campaign (numeric; 999 means client was not previously contacted)
- previous: number of contacts performed before this campaign and for this client (numeric)
- poutcome: outcome of the previous marketing campaign (categorical: “failure”, “nonexistent”, “success”)
- emp.var.rate: employment variation rate — (numeric)
- cons.price.idx: consumer price index — (numeric)
- cons.conf.idx: consumer confidence index — (numeric)
- euribor3m: euribor 3 month rate — (numeric)
- nr.employed: number of employees — (numeric)

### Predict variable (desired target):

y — has the client subscribed a term deposit? (binary: “1”, means “Yes”, “0” means “No”)

```python

```