# Principal Component Analysis (PCA)

| concept | description |
|---------|-------------|
| dimensionality reduction | replacing features by a smaller subset  |
| principal components | new features the data is transformed to |
| vanilla PCA | use if the data fits into memory |
| incremental PCA | PCA variant that works in batches (for lots of data or frequent updates) |
| randomized PCA | faster when reducing number of dimensions heavily |
| Kernel PCA | PCA variant for non-linear transformations |

## Background
**Principal Component Analysis (PCA)** is an **unsupervised learning** method used for **dimensionality reduction**.

For multivariate data sets, it often happens that a few measured variables already contain a large part of the information and other metrics contribute little. PCA is a procedure that allows to condense such data sets to the “most informative” dimensions.

PCA transforms the feature space X into a new feature space T, the principal components W:

\begin{align}
T = X * W
\end{align}

## Principal Components
Each principal component is a linear combination of the original features that explains the **maximum variance** in the data.

The main difference to other dimensionality reduction methods is that the **principal components are ranked** – the **first principal component always explains the highest proportion of the variance**. The second principal component explains the next highest proportion and so on.

The **maximum number of principal components is the number of original features**.
- In that case, no information is lost in the transformation.

With less principal components than original features, the new feature space is an approximation of the original space.

\begin{align}
T \approx X * W
\end{align}

**Note:** The components work similarly as in NMF – the more components you have, the better you represent the data. However, in NMF each component tends to explain an **equal** part of the variance.

---

## Algorithm
### Step 1: Calculate the covariance matrix
It is assumed that the variables that have the greatest variance also contain the most information.

To measure the redundancy in the data, we calculate the covriance of each measured variable with each other. This then results in the covariance matrix C:

\begin{align}
C_{ij}=\frac{1}{n-1}\sum^n_{k=1}\left(x_{ik}-\bar{x}_i\right)\left(x_{jk}-\bar{x}_j\right)
\end{align}

where i and j are the features and n the number of data points.

**Note:** The features may have very different value ranges. Before calculating the covariance it is not only advisable to de-mean the data but also scale it.

### Step 2: Calculate the eigenvectors
Calculate the eigenvectors of the matrix C and sort them according to their corresponding eigenvalues (largest first). The eigenvectors are the so-called principal components and the associated eigenvalues indicate how much variance (“information”) is included in this principal component is included.

The cumulative sum of the eigenvalues gives the percentage of variance that is explained by the first two principal components.

### Step 3: Transform the data
Project the original data set by multiplying it with the desired number of principal components:

\begin{align}
T=X⋅W
\end{align}

where X is the d-dimensional (normalized) data set and W is a matrix, containing the principal components in columns.

Applications
reduce calculation time
reduce overfitting
preprocessing step for other unsupervised methods (e.g. clustering or t-SNE)

### What is it, and what does it have to do with this week's project?
#### Let's start with some familiar code:

```python
from keras.datasets import mnist
from matplotlib import pyplot as plt
%matplotlib inline

(xtrain, ytrain), (xtest, ytest) = mnist.load_data()
```

    Using TensorFlow backend.

#### And let's write a function that takes in numpy arrays of images and renders/plots the first 40 of them:

```python
def draw_array(x):
    plt.figure(figsize=(12,7))
    for i in range(40):
        plt.subplot(5, 8, i+1)
        plt.imshow(x[i], cmap=plt.cm.Greys)
        plt.axis('off')

draw_array(xtrain)
```

---

### 1. Need to reshape the data so that it can be properly handled in sklearn

```python
xtrain.shape
```

    (60000, 28, 28)

```python
xtrain = xtrain.reshape(-1, 28*28)
```

```python
xtrain.shape
```

    (60000, 784)

### Standard Scalar
- Some people recommend using standard scalar to 'demean' the data

```python
from sklearn.preprocessing import StandardScaler
```

```python
ss = StandardScaler()
ss_xtrain = ss.fit_transform(xtrain)
```

```python
ss_xtrain.shape
```

    (60000, 784)

### 2. Split training data even further (to speed up the calculation)

```python
xsmall = xtrain[:1000]
ysmall = ytrain[:1000]
```

```python
xsmall.shape
```

    (1000, 784)

### 3. Initialize PCA from Scikit-Learn and fit on X data
- By how many components would we like to decompose our data?

```python
from sklearn.decomposition import PCA
```

```python
m = PCA(n_components=40)
```

```python
m.fit(xtrain)
```

    PCA(copy=True, iterated_power='auto', n_components=40, random_state=None,
        svd_solver='auto', tol=0.0, whiten=False)

---

- Notice that each of the components gives us 784 coefficients:

```python
m.components_.shape
```

    (40, 784)

---

- And we can see the percentage of the overall variation in the data that each principal contributes.

```python
m.explained_variance_ratio_
```

    array([0.09704664, 0.07095924, 0.06169089, 0.05389419, 0.04868797,
           0.04312231, 0.0327193 , 0.02883895, 0.02762029, 0.02357001,
           0.0210919 , 0.02022991, 0.01715818, 0.01692111, 0.01578641,
           0.01482953, 0.01324561, 0.01276897, 0.01187263, 0.01152684,
           0.01066166, 0.01006713, 0.00953573, 0.00912543, 0.00883404,
           0.00839317, 0.00812579, 0.00786364, 0.00744733, 0.00690853,
           0.00658087, 0.00648136, 0.00602485, 0.00586286, 0.00569746,
           0.00543428, 0.00505236, 0.0048757 , 0.00480015, 0.00471926])

### 4. Use the trained PCA model to transform the data to a lower number of features.
- new data - with less features

```python
#new data - with less features
xt = m.transform(xsmall)
```

```python
xsmall.shape
```

    (1000, 784)

```python
xt.shape
```

    (1000, 40)

---

### 5. Use the inverse_transform() method to expand our reduced data back into its original shape.

```python
draw_array(xsmall.reshape(-1,28,28))
```

---

### 6. Visualize what our data looks after it has been reduced to N features.

```python
xback = m.inverse_transform(xt)
draw_array(xback.reshape(-1,28,28))
```

---

### 7. Can we actually see what the components look like, as well?

```python
m.explained_variance_ratio_
```

    array([0.09704664, 0.07095924, 0.06169089, 0.05389419, 0.04868797,
           0.04312231, 0.0327193 , 0.02883895, 0.02762029, 0.02357001,
           0.0210919 , 0.02022991, 0.01715818, 0.01692111, 0.01578641,
           0.01482953, 0.01324561, 0.01276897, 0.01187263, 0.01152684,
           0.01066166, 0.01006713, 0.00953573, 0.00912543, 0.00883404,
           0.00839317, 0.00812579, 0.00786364, 0.00744733, 0.00690853,
           0.00658087, 0.00648136, 0.00602485, 0.00586286, 0.00569746,
           0.00543428, 0.00505236, 0.0048757 , 0.00480015, 0.00471926])

```python
comp = m.components_.reshape(-1,28,28)
draw_array(comp)
```

```python
msmall = PCA(n_components=1)
msmall.fit(xsmall)
```

    PCA(copy=True, iterated_power='auto', n_components=1, random_state=None,
        svd_solver='auto', tol=0.0, whiten=False)

```python
xs = msmall.transform(xsmall)
xs = msmall.inverse_transform(xs)
```

```python
draw_array(xs.reshape(-1,28,28))
```

---

### 8. How can this be applied in practice (e.g. on this week's project)?
- Increase up the training speed of your models - save time
- Upsampling technique - use pca to create new data points

---

### 9. Other interesting analysis:
- Examine the false positives using a confusion matrix.

```python
m
```

    PCA(copy=True, iterated_power='auto', n_components=40, random_state=None,
        svd_solver='auto', tol=0.0, whiten=False)

```python
xt.shape
```

    (1000, 40)

```python
from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=5, max_depth=3)
rf.fit(xt,ysmall)
print(rf.score(xt,ysmall))
```

    0.656

```python
xtest.shape
```

    (10000, 28, 28)

```python
xtest = m.transform(xtest.reshape(-1,784))
```

```python
ypred = rf.predict(xtest)
```

```python
from sklearn.metrics import confusion_matrix
```

```python
conf = confusion_matrix(ytest, ypred)
conf
```

    array([[ 520,   61,   23,   69,    5,   11,  100,   48,  136,    7],
           [   1, 1042,   13,   36,    0,   17,    7,    3,   16,    0],
           [  25,   68,  560,  101,   42,   15,  105,   51,   59,    6],
           [  97,   36,   16,  635,    5,   73,   17,   64,   61,    6],
           [   1,   15,   26,   16,  601,   19,   42,  175,   27,   60],
           [  33,  100,   64,  108,   42,  199,   35,   94,  198,   19],
           [  10,   70,   48,   79,   27,   23,  615,   56,   13,   17],
           [  11,   68,    5,    8,   52,   21,    8,  693,   31,  131],
           [   5,   35,   32,   61,   12,   32,   20,   79,  649,   49],
           [   6,   26,    3,   11,  253,   25,   17,  295,   35,  338]])

```python
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline

plt.figure(figsize=(12,10))
sns.heatmap(conf, cmap='Oranges', annot=True, fmt='g')
```

---

### 10. Other interesting analysis:
- We can see how much of an impact adding more components has on capturing the overall variance of the original data set.
- We can do this by transforming the data with another PCA model that captures the maximum number of components (which in our case is 784) and plotting the cumulative sum of the explained variance ratio.

```python
xtrain.shape
```

    (60000, 784)

```python
m784 = PCA(n_components=10)
x = m784.fit_transform(xtrain)
```

```python
import numpy as np
```

```python
ex_var = m784.explained_variance_ratio_
ex = range(784)
y = np.array(ex_var).cumsum()
```

```python
plt.plot(ex,y)
```

```python

```


# Differential Drive Robotics Platform

I've been using [SymPy](http://sympy.org) in my research and coursework for a while now. For those that don't know, SymPy is a computer algebra system, capable of performing symbolic calculations that would be too complicated to do by hand. Which makes it perfect for solving the equations needed to generate the equations of motion (EOM) of multibody systems! In this post, I'll demonstrate a simple workflow for generating the EOM for a differential drive robot.

## Problem Setup

A common robot platform is that of the [differential drive](http://en.wikipedia.org/wiki/Differential_wheeled_robot). The robot is propelled using two wheels, each with their own motor. Usually a caster is used as a third wheel to provide stability. By varying the speeds of the two motors, the robot can change direction.

Differential drive platforms are highly maneuverable, as they have a very small turn radius. They are also extremely simple to design. In this example, the dynamics of a differential drive platform will be explored. The resulting model is highly accurate, but makes a few assumptions:

**Assumptions**

- Wheels roll without slipping
- Body is symmetric

**The setup is as follows:**

```
from IPython.display import SVG
SVG(filename='Robot.svg')
```

## Equations of Motion

We’ll start by generating the equations of motion for the robot with `sympy.physics.mechanics`. The `mechanics` module provides a clean, intuitive workflow for deriving the equations of motion for multibody dynamic systems using either `KanesMethod` or `LagrangesMethod` objects. In this example we'll make use of the `LagrangesMethod` object, which generates the equations of motion using [Lagrangian Mechanics](http://en.wikipedia.org/wiki/Lagrangian_mechanics) with constraints and generalized forces.

First we import the necessary functionality from SymPy.

```
from sympy.physics.mechanics import *
from sympy import sin, cos, symbols, Matrix, solve
```

Next we need to define an inertial reference frame, as well as a coordinate origin. All coordinates and reference frames defined later will be in reference to the world coordinate frame.

```
#Inertial Reference Frame
N = ReferenceFrame('N')

#Define a world coordinate origin
O = Point('O')
O.set_vel(N, 0)
```

Lagrange's Method requires a state vector $\vec{q}$ that contains all the generalized coordinates required to define the system in space. In the case of the robot:

$$
\vec{q} = [\matrix{x && y && \theta && \phi_1 && \phi_2}]^T
$$

As these are all time varying variables, they need to be created as `dynamicsymbols`:

```
theta = dynamicsymbols('theta')                      #Rotation about N.z
x, y = dynamicsymbols('x, y')                        #Coordinates of robot in World Frame
phi1, phi2 = dynamicsymbols('phi_1, phi_2')          #Angular displacement of wheel

#Create q and dq vectors
q = Matrix([x, y, theta, phi1, phi2])
dq = q.diff()
```

We'll also define some constant parameters that define the specific robot:

```
#Constants for the wheels
r = symbols('r')                                     #Radius of wheel
m_w = symbols('m_w')                                 #Mass of the wheel
t_w = symbols('t_w')                                 #Thickness of the wheel

#Constants for the Robot Body
w = symbols('w')                                     #2*w is the width of the wheel base
d = symbols('d')                                     #Distance between axel and center of mass
m_b = symbols('m_b')                                 #Mass of the body
Ixx, Iyy, Izz = symbols('Ixx, Iyy, Izz')             #Moments of inertia of body
```

Next we need to define a reference frame relative to the robot, and a point located at the center of the wheel axis. This will define the robot's location in the world coordinate frame. The velocity of this point is just the derivative of its x and y coordinates.

```
#Robot Reference Frame
R = N.orientnew('R', 'Axis', [theta, N.z])

#Center of wheel base
Cw = O.locatenew('Cw', x*N.x + y*N.y)

#Set the velocity of point Cw
Cw.set_vel(N, x.diff()*N.x + y.diff()*N.y)
```

Coordinates of the 2 wheel hubs then need to be specified. Luckily sympy mechanics makes this easy to do with the `locatenew` method. The hubs 1 and 2 are located at $-w\vec{R_y}$ and $w\vec{R_y}$ away from the center of the wheel base $C_w$ respectively.

The velocity of the hubs is then found using the `v2pt_theory` method, which calculates the velocity of a point based off the velocity of another point in the same frame, and the angular velocity of that frame ([reference](http://en.wikipedia.org/wiki/Rotating_reference_frame#Relation_between_velocities_in_the_two_frames)).

```
#Points at wheel hubs
H1 = Cw.locatenew('H1', -w*R.y)
H2 = Cw.locatenew('H2', w*R.y)

#Set the velocity of points H1 and H2
H1.v2pt_theory(Cw, N, R)
H2.v2pt_theory(Cw, N, R);
```

Rotating reference frames fixed to each wheel are then created. These rotate about the $\vec{R_y}$ axis with the angular position of the wheels, $\phi_1$ and $\phi_2$.

```
#Create reference frames for wheels 1 and 2
W1 = R.orientnew('W1', 'Axis', [phi1, R.y])
W2 = R.orientnew('W2', 'Axis', [phi2, R.y])
```

`sympy.physics.mechanics` provides classes for rigid bodies. These make calculating the lagrangian of a multibody system easier, as each body can be handled separately. To do this, each body needs the following things specified:

- A name
- A center of mass
- A reference frame
- A mass
- A tuple of (Inertia, Point of rotation)

Modeling the wheels as solid cylinders, the inertia can be calculated about the hubs, and `RigidBody` objects can be created for each wheel.

```
#Calculate inertia of the wheel
Iw = inertia(R, m_w*(3*r**2 + t_w**2)/12, m_w*r**2/2, m_w*(3*r**2 + t_w**2)/12)

#Create rigid bodies for wheels
Wheel1 = RigidBody('Wheel1', H1, W1, m_w, (Iw, H1))
Wheel2 = RigidBody('Wheel2', H2, W2, m_w, (Iw, H2))
```

The same can be done for the body of the robot. In this case we model the body as a symmetric object with inertias $I_{xx}$, $I_{yy}$, and $I_{zz}$. The center of mass of the body is located a distance $d$ in front of the wheel axel.

```
#Calculate inertia of body
Ib = inertia(R, Ixx, Iyy, Izz)

#Center of mass of body
Cm = Cw.locatenew('Cm', d*R.x)
Cm.v2pt_theory(Cw, N, R)

#Create a rigid body object for body
Body = RigidBody('Body', Cm, R, m_b, (Ib, Cm))
```

## Nonholonomic Constraints
At this point, the lagrangian of the system can be calculated. However, the system still isn't fully defined; the constraints still need to be specificed.

At this instance in time, the velocity of the ground contact points must have a velocity of zero. To specify this in the lagrange equations, a constraint matrix needs to be created.

First, two points are created where the wheels contact the ground, and their velocities calculated.

```
#Create two points, where the wheels contact the ground
C1 = H1.locatenew('C1', -r*R.z)
C2 = H2.locatenew('C2', -r*R.z)
#Calculate velocity of points
C1.v2pt_theory(H1, N, W1)
C2.v2pt_theory(H2, N, W2);
```

Using these velocities, a system of equations can be created. As the velocities must be zero, this expression = 0.

```
#Express the velocity of points in the inertial frame
con1 = C1.vel(N).express(N).args[0][0]
con2 = C2.vel(N).express(N).args[0][0]
#Create a matrix of constraints
constraints = con1.col_join(con2)
mprint(constraints)
```

    Matrix([
    [(-r*phi_1' + w*theta')*cos(theta) + x'],
    [(-r*phi_1' + w*theta')*sin(theta) + y'],
    [                                     0],
    [(-r*phi_2' - w*theta')*cos(theta) + x'],
    [(-r*phi_2' - w*theta')*sin(theta) + y'],
    [                                     0]])

There are 6 equations with 3 unknowns ($\dot{x}$, $\dot{y}$, and $\dot{theta}$), there are duplicate equations. Using solve, and reconfiguring the expression, this can be reduced to the required set:

```
#Solve for dx, dy, and dtheta in terms of dphi1 and dphi2
sol = solve(constraints, dq[:3])

#Split the resulting dict into a rhs and lhs, that are equivalent
sol_rhs = Matrix(sol.values())
sol_lhs = Matrix(sol.keys())

#Since sol_rhs = sol_lhs --> sol_rhs - sol_lhs = 0
#This forms the basis of our constraint matrix.
#Combining, and solving for a linear representation:
c = (sol_rhs - sol_lhs).jacobian(dq[:5])
mprint(c)
```

    Matrix([
    [-1,  0,  0, r*cos(theta)/2, r*cos(theta)/2],
    [ 0,  0, -1,        r/(2*w),       -r/(2*w)],
    [ 0, -1,  0, r*sin(theta)/2, r*sin(theta)/2]])

This is our constraint matrix. The `LagrangesMethod` class requires that the constraints be specified as

`coneqs` = $ C\dot{\vec{q}} = 0 $

Thus:

```
#Constraint Equations
coneqs = (c*dq)
mprint(coneqs)
```

    Matrix([
    [r*cos(theta)*phi_1'/2 + r*cos(theta)*phi_2'/2 - x'],
    [          r*phi_1'/(2*w) - r*phi_2'/(2*w) - theta'],
    [r*sin(theta)*phi_1'/2 + r*sin(theta)*phi_2'/2 - y']])

## Generalized Forces

The inputs of system are the torques at the wheels. Using the right-hand-rule, it can be seen that a positive torque will exert force on the system in the $R_x$ direction. With a wheel radius of $r$, the force on the wheel hubs is:

$\vec{F_{hub}} = \tau*\vec{R_x}$

```
#Define forces on system:
T1, T2 = symbols('tau_1, tau_2')              #Torques from the wheels
fl = [(H1, r*T1*R.x),
      (H2, r*T2*R.x)]
```

## Solving the System Dynamics

We are now ready to solve the for the equations of motion. First, calculate the lagrangian of the system $L = T - V$:

```
Lag = Lagrangian(N, Wheel1, Wheel2, Body)
```

Next, a `LagrangesMethod` object is created. This takes the following parameters:

- The system lagrangian
- The generalized coordinate vector $\vec{q}$
- The constraint equations
- A list force tuples: (Point, Force at point)
- An inertial reference frame

```
lm = LagrangesMethod(Lag, q, coneqs=coneqs, forcelist=fl, frame=N)
```

The equations of motion can then be found:

```
le = lm.form_lagranges_equations()
mprint(le)
```

    Matrix([
    [                                                                                m_b*(-2*d*sin(theta)*theta'' - 2*d*cos(theta)*theta'**2 + 2*x'')/2 + m_w*(-2*w*sin(theta)*theta'**2 + 2*w*cos(theta)*theta'' + 2*x'')/2 + m_w*(2*w*sin(theta)*theta'**2 - 2*w*cos(theta)*theta'' + 2*x'')/2 - r*tau_1*cos(theta) - r*tau_2*cos(theta) - lam1],
    [                                                                                m_b*(-2*d*sin(theta)*theta'**2 + 2*d*cos(theta)*theta'' + 2*y'')/2 + m_w*(-2*w*sin(theta)*theta'' - 2*w*cos(theta)*theta'**2 + 2*y'')/2 + m_w*(2*w*sin(theta)*theta'' + 2*w*cos(theta)*theta'**2 + 2*y'')/2 - r*tau_1*sin(theta) - r*tau_2*sin(theta) - lam3],
    [Izz*theta'' - m_b*(d*(-sin(theta)*y' - cos(theta)*x')*theta' - d*sin(theta)*theta'*y' - d*cos(theta)*theta'*x')/2 + m_b*(2*d**2*theta'' + d*(-sin(theta)*theta'*y' - sin(theta)*x'' - cos(theta)*theta'*x' + cos(theta)*y'') - d*sin(theta)*theta'*y' - d*sin(theta)*x'' - d*cos(theta)*theta'*x' + d*cos(theta)*y'')/2 + m_w*(3*r**2 + t_w**2)*theta''/6 - m_w*(-w*(-sin(theta)*x' + cos(theta)*y')*theta' + w*sin(theta)*theta'*x' - w*cos(theta)*theta'*y')/2 - m_w*(w*(-sin(theta)*x' + cos(theta)*y')*theta' - w*sin(theta)*theta'*x' + w*cos(theta)*theta'*y')/2 + m_w*(2*w**2*theta'' - w*(-sin(theta)*theta'*x' + sin(theta)*y'' + cos(theta)*theta'*y' + cos(theta)*x'') + w*sin(theta)*theta'*x' - w*sin(theta)*y'' - w*cos(theta)*theta'*y' - w*cos(theta)*x'')/2 + m_w*(2*w**2*theta'' + w*(-sin(theta)*theta'*x' + sin(theta)*y'' + cos(theta)*theta'*y' + cos(theta)*x'') - w*sin(theta)*theta'*x' + w*sin(theta)*y'' + w*cos(theta)*theta'*y' + w*cos(theta)*x'')/2 - r*tau_1*w + r*tau_2*w - lam2],
    [                                                                                m_w*r**2*phi_1''/2 + r*lam1*cos(theta)/2 + r*lam3*sin(theta)/2 + r*lam2/(2*w)],
    [                                                                                m_w*r**2*phi_2''/2 + r*lam1*cos(theta)/2 + r*lam3*sin(theta)/2 - r*lam2/(2*w)]])

Sympy makes this super easy: no need to do this by hand, and no more mistakes in algebra. However, these are rather complicated equations. Luckily, they can be converted into a function allowing numerical simulation.

## Simulation

To gain insight into the system dynamics, numerical simulation will be used. First, we need to import a few more functions:

```
from sympy import lambdify, Dummy
from scipy import array, hstack, linspace, ones
from scipy import random, interp, vstack
from scipy.integrate import odeint
import matplotlib.pyplot as plt
%matplotlib inline
```

The vector $\ddot{\vec{q}}$ can be calculated in a closed form expression using the `rhs` method. This method takes quite a long time to solve entirely symbolically (it took over an hour on my machine), so a simplifying assumption will be made: $d = 0$. This is actually pretty reasonable, as the motors and battery will be the heaviest part, and should be situated around the axle.

```
#Substitute in d = 0
lm.eom = lm.eom.subs({d: 0})

#Solve for the rhs:
rhs = lm.rhs()
```

We're almost done, all we need to do is model the dynamics of the motor. In the future there may be dynamics classes for electronics, allowing a piecewise construction like we did for the mechanics (using `RigidBody`). Defining the motor dynamics using the current code-base isn't terribly tricky though.

In general, we can model the current dynamics of the motor as follows:

$$
\dot{i} = -K/L \dot{\phi} - R/L i + V/L
$$

$$
\tau = K i
$$

where $i$ and $V$ are the current and voltage through the motor, $K$ is the motor constant, $R$ is the coil resistance, and $L$ is the coil inductance. Creating this model in sympy is fairly straightforward:

```
#Create dynamic symbols for current and voltage
i_1, i_2 = dynamicsymbols('i_1, i_2')        #Currents through motor 1 and 2
V_1, V_2 = symbols('V_1, V_2')               #Voltages across the motor terminals

#Define some motor constants.
#Assuming motor 1 and 2 are the same:
R = symbols('R')                             #Coil resistance
L = symbols('L')                             #Coil inductance
K1, K2 = symbols('K1, K2')                   #Motor constant

#Define the motor dynamics
di = Matrix([[-K1/L*phi1.diff() - R/L*i_1 + V_1/L],
             [-K2/L*phi2.diff() - R/L*i_2 + V_2/L]])
```

Now we just need to combine the motor model with the model for the robot:

```
#Define consts:
params = [Izz,  t_w,  m_w,    r,  m_b,   w,    R,      L,   K1,  K2]
values = [  5, 0.15,  2.0, 0.15, 50.0, 0.6, 0.05, 0.0001,  1.0, 1.0]       #Values of constants

#Create a list of dynamic symbols for simulation
dynamics = q.T.tolist()[0] + dq.T.tolist()[0] + lm.lam_vec.T.tolist()[0] + [i_1, i_2]

#Set the inputs to be the motor voltages
inputs = [V_1, V_2]

#Add the motor model to the rhs equations
aug_rhs = rhs.col_join(di)

#Substitute in K*i for T in the rhs equations
aug_rhs = aug_rhs.subs({T1: K1*i_1, T2: K2*i_2})
```

This can now be converted into a function to solve the rhs based on current state using `lambdify`:

```
#Create a list of dynamic symbols for simulation
dummys = [Dummy() for i in dynamics]
dummydict = dict(zip(dynamics, dummys))
#Sub in the dummy symbols
rhs_dummy = aug_rhs.subs(dummydict)
#Lambdify function
rhs_func = lambdify(dummys + inputs + params, rhs_dummy)
```

```
#Create a function in the required format for odeint
def right_hand_side(x, t, ts, us, values):
    """Calculate the rhs of the integral, at
    time t, state x.
    ts, us are used to get the current input
    values are constants in the integral"""

    #Interp is needed to get u for timestep
    u1 = interp(t, ts, us[:,0])
    u2 = interp(t, ts, us[:,1])

    arguments = hstack((x, u1, u2, values))

    #Need call to array and reshape, as odeint
    #requires state vector to be 1d
    dx = array(rhs_func(*arguments))
    return dx.reshape(dx.size)
```

Now we have everything we need to simulate the motion. First, let's see what happens if we input a constant, unbalanced voltage signal. Motor 1 gets 24 volts, Motor 2 gets 20 volts. Intuitively, this should cause the robot to drive in a circle.

```
ts = linspace(0, 30, 3000)
us = vstack((24*ones(len(ts)), 20*ones(len(ts)))).T
#Run the simulation
x0 = [0,] * 15                         #Start out at origin, unmoving
xs = odeint(right_hand_side, x0, ts, args=(ts, us, values))

plt.plot(xs[:,5], xs[:,6])
plt.title('Robot Position')
plt.xlabel('x (m)')
plt.ylabel('y (m)');
```

As seen by the simulation, the results line up with our intuition. The reason it doesn't immediately start out in a circle is due to the time it takes for it to accelerate from rest.

Next, lets see what happens when the input voltages are matched:

```
ts = linspace(0, 10, 1000)
us = 20*ones((len(ts),2))
#Run the simulation
x0 = [0,] * 15                         #Start out at origin, unmoving
xs = odeint(right_hand_side, x0, ts, args=(ts, us, values))
plt.plot(xs[:,0], xs[:,1])
plt.title('Robot Position')
plt.xlabel('x (m)')
plt.ylabel('y (m)');
plt.ylim(-2, 2);
```

As expected, the robot drives in a straight line.

## Simulation - Need for control

Currently our robot is driven in "open-loop" control. This means that an input signal is sent, but no output is measured. This is fine if our model is perfect, and there are no disturbances. But the real world isn't like that. For example, what if our motors aren't perfectly balanced, and had slightly different K values?

To see, we'll change the K value of Motor 2 to 0.99. Just a small difference.

```
#Change the value of K_2
values = [5, 0.15, 2.0, 0.15, 50.0, 0.6, 0.05, 0.0001, 1.0, 0.99]

#Re-run the straight line simulation
ts = linspace(0, 10, 1000)
us = 24*ones((len(ts),2))

#Run the simulation
x0 = [0,] * 15                         #Start out at origin, unmoving
xs = odeint(right_hand_side, x0, ts, args=(ts, us, values))
plt.plot(xs[:,0], xs[:,1])
plt.title('Robot Position')
plt.xlabel('x (m)')
plt.ylabel('y (m)');
```

After 10 seconds, the robot is already 2.5 m off it's intended course! Luckily, a simple feedback controller should be able to solve this.

As our model is highly nonlinear, the first thing that should be done is to linearize it about our setpoint. However, currently the `LagrangesMethod` class doesn't have a linearization method. And because there are constraints, simply computing the Jacobian of the RHS isn't a valid solution. So an LQR controller, or other traditional state space design methods are out.

To develop a controller for this robot, a simple linear model would need to be developed. This will be covered in part 2, but for now, let's just save the model to file so it can be opened later without rederiving:

```
#srepr returns the python code needed to recreate a
#sympy expression
from sympy.printing import srepr

def save_sol(expr, file_path):
    """Save a sympy expression to file

    To reload the expression:

       import file_path
       expression = file_path.load()

    """

    fil = open(file_path, "w")
    fil.write("from sympy import *\n")
    fil.write("def load():\n    expression=")
    fil.write(srepr(expr))
    fil.write("\n    return expression")
    fil.close()
```

```
#Save the rhs for later evaluation
save_sol(rhs, "robot_rhs.py")
```

This can later be reloaded by calling:

    import robot_rhs
    rhs = robot_rhs.load()

In a later example, a controller will be developed using this model for simulation, so it's nice not to have to rederive the expression.

Overall, the `sympy.physics.mechanics` workflow for modeling multibody dynamics is easy to use. As shown above, the tools for generating a model based off of its components (wheels, body, etc...), solving for the system dynamics, and then simulating the resulting expression are provided in an intuitive manner. While larger/more complicated models may take some time (I've gotten the [n-pendulum example](https://github.com/pydy/pydy_examples/tree/master/npendulum) to solve for up to 25 links), the resulting expression in symbolic form allows for simulation and analysis in a way that traditional numerical methods cannot provide. With some improvements to the simplification routines, and the addition of a linearization routine the hope is that the complicated expressions shown here can be simplified to "human readable form". This would allow for fast run-time due to simplified expressions, as well as easier analysis.

*Note: The code used here was developed as part of the [pydy](https://pydy.org/) project. Other awesome portions of the `pydy` workflow have not been demonstrated here. Some work is currently being done on [code generation](https://github.com/pydy/pydy-code-gen) to generate code in `C`, `FORTRAN`, or other language from a `sympy` expression. This would allow for faster simulations, and easy integration into embedded controllers or estimators. There is also some work being done on [visualization](https://github.com/pydy/pydy-viz), resulting in an animation of the system dynamics running in a browser.*

*This IPython notebook may be downloaded from [here](https://github.com/jcrist/pydy_examples/tree/master/differential_drive)*


# Python implementation of elastic net regularized GLMs using proximal batch gradient descent and coordinate descent

Jerome Friedman, Trevor Hastie and Rob Tibshirani. (2010).
Regularization Paths for Generalized Linear Models via Coordinate Descent.
Journal of Statistical Software, Vol. 33(1), 1-22 [[pdf](https://core.ac.uk/download/files/153/6287975.pdf)]

```python
import numpy as np
from scipy.special import expit
import scipy.sparse as sps
import matplotlib.pyplot as plt
%matplotlib inline
```

## GLM with elastic net penalty

### Poisson-like GLM

The ```pyglmnet``` implementation comes with ```poisson```, ```binomial``` and ```normal``` distributions, but for illustration, I will walk you through a particular adaptation of the canonical Poisson generalized linear model (GLM).

For the Poisson GLM, $\lambda_i$ is the rate parameter of an inhomogeneous linear-nonlinear Poisson (LNP) process with instantaneous mean given by:

$$\lambda_i = \exp(\beta_0 + \beta^T x_i)$$

where $x_i \in \mathcal{R}^{p \times 1}, i = \{1, 2, \dots, n\}$ are the observed independent variables (predictors), $\beta_0 \in \mathcal{R}^{1 \times 1}$, $\beta \in \mathcal{R}^{p \times 1}$ are linear coefficients. $\lambda_i$ is also known as the conditional intensity function, conditioned on $(\beta_0, \beta)$ and $q(z) = \exp(z)$ is the nonlinearity.

For numerical reasons, let's adopt a stabilizing non-linearity, known as the softplus or the smooth rectifier [[Dugas et al., 2001](http://papers.nips.cc/paper/1920-incorporating-second-order-functional-knowledge-for-better-option-pricing.pdf)], and adopted by Jonathan Pillow's and Liam Paninski's groups for neural data analysis. See for instance: [[Park et al., 2014](http://www.nature.com/neuro/journal/v17/n10/abs/nn.3800.html)].

$$q(z) = \log(1+\exp(z))$$

The softplus prevents $\lambda$ in the canonical inverse link function from exploding when the argument to the exponent is large. In this modification, the formulation is no longer an exact LNP, nor an exact GLM, but $\pm\mathcal{L}(\beta_0, \beta)$ is still concave (convex) and we can use gradient ascent (descent) to optimize it.

$$\lambda_i = q(\beta_0 + \beta^T x_i) = \log(1 + \exp(\beta_0 + \beta^T x_i))$$

[There is more to be said about this issue; ref. Sara Solla's GLM lectures concerning moment generating functions and strict definitions of GLMs]

```python
# Let's define the conditional intensity function
def qu(z):
    eps = np.spacing(1)
    return np.log(1+eps+np.exp(z))

def lmb(beta0, beta, x):
    eps = np.spacing(1)
    z = beta0 + np.dot(x, beta)
    return np.log(1+eps+np.exp(z))
```

### Log-likelihood
The likelihood of observing the spike count $y_i$ under the Poisson likelihood function with inhomogeneous rate $\lambda_i$ is given by:

\begin{equation}
\prod_i P(y = y_i) = \prod_i \frac{e^{-\lambda_i} \lambda_i^{y_i}}{y_i!}
\end{equation}

The log-likelihood is given by:

$$\mathcal{L} = \sum_i \bigg\{y_i \log(\lambda_i) - \lambda_i - log(y_i!)\bigg\}$$

However, we are interested in maximizing the log-likelihood as a function of $\beta_0$ and $\beta$. Thus, we can drop the factorial term:

$$\mathcal{L}(\beta_0, \beta) = \sum_i \bigg\{y_i \log(\lambda_i) - \lambda_i\bigg\}$$

```python
# Let's define the log likelihood
def logL(beta0, beta, x, y):
    l = lmb(beta0, beta, x)
    logL = np.sum(y*np.log(l) - l)
    return logL
```

### Elastic net penalty
For large models we need to penalize the log likelihood term in order to prevent overfitting.
The elastic net penalty is given by:

$$\mathcal{P}_\alpha(\beta) = (1-\alpha)\frac{1}{2} \|\beta\|^2_{\mathcal{l}_2} + \alpha\|\beta\|_{\mathcal{l}_1}$$

The elastic net interpolates between two extremes. $\alpha = 0$ is known as ridge regression and $\alpha = 1$ is known as Lasso. Note that we do not penalize the baseline term $\beta_0$.

```python
# Let's define the penalty term
def penalty(alpha, beta):
    P = 0.5*(1-alpha)*np.linalg.norm(beta,2) + alpha*np.linalg.norm(beta,1)
    return P
```

### Objective function
We minimize the objective function:

$$J(\beta_0, \beta) = -\mathcal{L}(\beta_0, \beta) + \lambda \mathcal{P}_\alpha(\beta)$$

where $\mathcal{L}(\beta_0, \beta)$ is the Poisson log-likelihood and $\mathcal{P}_\alpha(\beta)$ is the elastic net penalty term and $\lambda$ and $\alpha$ are regularization parameters.

```python
# Let's define the objective function with elastic net regularization
def loss(beta0, beta, alpha, reg_lambda, x, y):
    L = logL(beta0, beta, x, y)
    P = penalty(alpha, beta)
    J = -L + reg_lambda*P
    return J
```

## Optimization

### Gradients
To calculate the gradients of the cost function with respect to $\beta_0$ and $\beta$, let's plug in the definitions for the log likelihood and penalty terms from above.

$$
J(\beta_0, \beta) = \sum_i \bigg\{ \log(1 + \exp(\beta_0 + \beta^T x_i)) - y_i \log(\log(1 + \exp(\beta_0 + \beta^T x_i)))\bigg\} + \lambda(1-\alpha)\frac{1}{2} \|\beta\|^2_{\mathcal{l_2}} + \lambda\alpha\|\beta\|_{\mathcal{l_1}}
$$

Since we will apply co-ordinate descent, let's rewrite this cost in terms of each scalar parameter $\beta_j$

$$
J(\beta_0, \beta) = \sum_i \bigg\{ \log(1 + \exp(\beta_0 + \sum_j \beta_j x_{ij})) - y_i \log(\log(1 + \exp(\beta_0 + \sum_j \beta_j x_{ij})))\bigg\} + \lambda(1-\alpha)\frac{1}{2} \sum_j \beta_j^2 + \lambda\alpha\sum_j \mid\beta_j\mid
$$

Let's take the derivatives of some big expressions using chain rule. Define $z_i = \beta_0 + \sum_j \beta_j x_{ij}$.

For the nonlinearity in the first term $y = q(z) = \log(1+e^{z(\theta)})$,

$$\frac{\partial y}{\partial \theta} = \frac{\partial q}{\partial z}\frac{\partial z}{\partial \theta} = \frac{e^z}{1+e^z}\frac{\partial z}{\partial \theta} = \sigma(z)\frac{\partial z}{\partial \theta}$$

For the nonlinearity in the second term $y = \log(q(z)) = \log(\log(1+e^{z(\theta)}))$,

$$\frac{\partial y}{\partial \theta} = \frac{1}{q(z)}\frac{\partial q}{\partial z}\frac{\partial z}{\partial \theta} = \frac{\sigma(z)}{q(z)}\frac{\partial z}{\partial \theta}$$

where $\dot q(z)$ happens to be be the sigmoid function,
$$\sigma(z) = \frac{e^z}{1+e^z}$$

Putting it all together, we have,

$$
\frac{\partial J}{\partial \beta_0} = \sum_i \sigma(z_i) - \sum_i y_i\frac{\sigma(z_i)}{q(z_i)}
$$

$$
\frac{\partial J}{\partial \beta_j} = \sum_i \sigma(z_i) x_{ij} - \sum_i y_i \frac{\sigma(z_i)}{q(z_i)}x_{ij}
+ \lambda(1-\alpha)\beta_j + \lambda\alpha \text{sgn}(\beta_j)
$$

```python
# Let's define these gradients
def grad_loss(beta0, beta, alpha, reg_lambda, x, y):
    z = beta0 + np.dot(x, beta)
    q = qu(z)
    s = expit(z)
    grad_beta0 = np.sum(s) - np.sum(y*s/q)
    # This is a matrix implementation
    grad_beta = np.transpose(np.dot(np.transpose(s), x) - np.dot(np.transpose(y*s/q), x)) \
        + reg_lambda*(1-alpha)*beta# + reg_lambda*alpha*np.sign(beta)
    return grad_beta0, grad_beta
```

Note that this is all we need for a classic batch gradient descent implementation. However, let's also derive the Hessian terms that will be useful for second-order optimization methods.

### Hessian terms
Second-order derivatives can accelerate convergence to local minima by providing optimal step sizes. However, they are expensive to compute.

This is where co-ordinate descent shines. Since we update only one parameter $\beta_j$ per step, we can simply use the $j^{th}$ diagonal term in the Hessian matrix to perform an approximate Newton update as:

$$\beta_j^{t+1} = \beta_j^{t} - \bigg\{\frac{\partial^2 J}{\partial \beta_j^2}\bigg\}^{-1} \frac{\partial J}{\partial \beta_j}$$

Let's use calculus again to compute these diagonal terms. Recall that:

$$\dot q(z) = \sigma(z)$$ and
$$\dot\sigma(z) = \sigma(z)(1-\sigma(z))$$

Using these, and applying the product rule
$$
\frac{\partial}{\partial z}\bigg\{ \frac{\sigma(z)}{q(z)} \bigg\} = \frac{\sigma(z)(1-\sigma(z))}{q(z)} - \frac{\sigma(z)}{q(z)^2}
$$

Plugging all these in, we get
$$
\frac{\partial^2 J}{\partial \beta_0^2} = \sum_i \sigma(z_i)(1 - \sigma(z_i)) - \sum_i y_i \bigg\{ \frac{\sigma(z_i) (1 - \sigma(z_i))}{q(z_i)} - \frac{\sigma(z_i)}{q(z_i)^2} \bigg\}
$$

$$
\frac{\partial^2 J}{\partial \beta_j^2} = \sum_i \sigma(z_i)(1 - \sigma(z_i)) x_{ij}^2
- \sum_i y_i \bigg\{ \frac{\sigma(z_i) (1 - \sigma(z_i))}{q(z_i)} - \frac{\sigma(z_i)}{q(z_i)^2} \bigg\}x_{ij}^2 + \lambda(1-\alpha)
$$

```python
# Let's define these Hessian terms
def hessian_loss(beta0, beta, alpha, reg_lambda, x, y):
    z = beta0 + np.dot(x, beta)
    q = qu(z)
    s = expit(z)
    grad_s = s*(1-s)
    grad_s_by_q = grad_s/q - s/(q*q)

    hess_beta0 = np.sum(grad_s) - np.sum(y*grad_s_by_q)

    # This is a matrix implementation
    hess_beta = np.transpose(np.dot(np.transpose(grad_s), x*x) - np.dot(np.transpose(y*grad_s_by_q), x*x))\
                             + reg_lambda*(1-alpha)
    return hess_beta0, hess_beta
```

## Cyclical co-ordinate descent

### Parameter update step
In cylical coordinate descent with elastic net, we store an active set, $\mathcal{K}$, of parameter indices that we update. Since the $\mathcal{l}_1$ terms $|\beta_j|$ are not differentiable at zero, we use the gradient without the $\lambda\alpha \text{sgn}(\beta_j)$ term to update $\beta_j$. Let's call these gradient terms $\tilde{g}_k$.

We start by initializing $\mathcal{K}$ to contain all parameter indices
Let's say only the $k^{th}$ parameter is updated at time step $t$.

$$\beta_k^{t} = \beta_k^{t-1} - (h_k^{t-1})^{-1} \tilde{g}_k^{t-1}$$
$$\beta_j^{t} = \beta_j^{t-1}, \forall j \neq k $$

Next, we apply a soft thresholding step for $k \neq 0$ after every update iteration, as follows.
$$\beta_k^{t} = \mathcal{S}_{\lambda\alpha}(\beta_k^{t})$$

where
$$S_\lambda(x) =
\begin{cases}
0 & \text{if} & |x| \leq \lambda\\
\text{sgn}(x)||x|-\lambda| & \text{if} & |x| > \lambda
\end{cases}
$$

If $\beta_k^{t}$ has been zero-ed out, we remove $k$ from the active set.
$$
\mathcal{K} = \mathcal{K} \setminus \left\{k\right\}
$$

```python
# Define the proximal operator
def prox(x,l):
    #sx = [0. if np.abs(y) <= l else np.sign(y)*np.abs(abs(y)-l) for y in x]
    #return np.array(sx).reshape(x.shape)
    return np.sign(x) * (np.abs(x) - l) * (np.abs(x) > l)
```

### Efficient z update

Next, we want to update $\beta_{k+1}$ at the next time step $t+1$. For this we need the gradient and Hessian terms, $\tilde{g}_{k+1}$ and $h_{k+1}$. If we update them instead of recalculating them, we can save on a lot of multiplications and additions. This is possible because we only update one parameter at a time. Let's calculate how to make these updates.

$$z_i^{t} = z_i^{t-1} - \beta_k^{t-1}x_{ik} + \beta_k^{t}x_{ik}$$

$$z_i^{t} = z_i^{t-1} - (h_k^{t-1})^{-1} \tilde{g}_k^{t-1}x_{ik}$$

### Gradient update

If $k = 0$,

$$\tilde{g}_{k+1}^t = \sum_i \sigma(z_i^t) - \sum_i y_i \frac{\sigma(z_i^t)}{q(z_i^t)}$$

If $k > 0$,

$$\tilde{g}_{k+1}^t = \sum_i \sigma(z_i^t) x_{i,k+1} - \sum_i y_i \frac{\sigma(z_i^t)}{q(z_i^t)}x_{i,k+1} + \lambda(1-\alpha)\beta_{k+1}^t$$

```python
# Gradient update for a single coordinate
def grad_loss_k(z, beta_k, alpha, rl, xk, y, k):
    q = qu(z)
    s = expit(z)

    if(k == 0):
        gk = np.sum(s) - np.sum(y*s/q)
    else:
        gk = np.sum(s*xk) - np.sum(y*s/q*xk) + rl*(1-alpha)*beta_k
    return gk
```

### Hessian update

If $k = 0$,

$$
h_{k+1}^t = \sum_i \sigma(z_i^t)(1 - \sigma(z_i^t))
- \sum_i y_i \bigg\{ \frac{\sigma(z_i^t) (1 - \sigma(z_i^t))}{q(z_i^t)} - \frac{\sigma(z_i^t)}{q(z_i^t)^2} \bigg\}
$$

If $k > 0$,

$$
h_{k+1}^t = \sum_i \sigma(z_i^t)(1 - \sigma(z_i^t)) x_{i,k+1}^2
- \sum_i y_i \bigg\{ \frac{\sigma(z_i^t) (1 - \sigma(z_i^t))}{q(z_i^t)} - \frac{\sigma(z_i^t)}{q(z_i^t)^2} \bigg\}x_{i,k+1}^2 + \lambda(1-\alpha)
$$

```python
# Hessian update for a single coordinate
def hess_loss_k(z, alpha, rl, xk, y, k):
    q = qu(z)
    s = expit(z)
    grad_s = s*(1-s)
    grad_s_by_q = grad_s/q - s/(q*q)

    if(k == 0):
        hk = np.sum(grad_s) - np.sum(y*grad_s_by_q)
    else:
        hk = np.sum(grad_s*xk*xk) - np.sum(y*grad_s_by_q*xk*xk) + rl*(1-alpha)
    return hk
```

### Regularization paths and warm restarts

We often find the optimal regularization parameter $\lambda$ through cross-validation. Thus, in practice, we fit the model several times over a range of $\lambda$'s $\{ \lambda_{max} \geq \dots \geq \lambda_0\}$.

Instead of re-fitting the model each time, we can solve the problem for the most-regularized model ($\lambda_{max}$) and then initialize the subsequent model with this solution. The path that each parameter takes through the range of regularization parameters is known as the regularization path, and the trick of initializing each model with the previous model's solution is known as a warm restart.

In practice, this significantly speeds up convergence.

### Algorithm

Give the algorithm in an algorithm table [coming soon].

## Implementation

Note that this section is a work in progress.

The functions below are for illustration only. The more up to date version of these functions can be found in ```pyglmnet.py```. To have a working example, look at the demo in ```pyglmnet_example.ipynb```.

### Pathwise cyclical coordinate descent for elastic net regularized GLM

```python
# Native python function
def fitmodel(x, y, reg_lambda, alpha):

    n = x.shape[0]
    p = x.shape[1]

    beta0_hat = np.random.normal(0.0,1.0,1)
    beta_hat = np.random.normal(0.0,1.0,[p,1])
    fit = []

    # Outer loop with descending lambda
    for l,rl in enumerate(reg_lambda):
        fit.append({'beta0': 0., 'beta': np.zeros([p,1])})
        print('Lambda: {}\n').format(rl)

        # Warm initialize parameters
        if(l == 0):
            fit[-1]['beta0'] = beta0_hat
            fit[-1]['beta'] = beta_hat
        else:
            fit[-1]['beta0'] = fit[-2]['beta0']
            fit[-1]['beta'] = fit[-2]['beta']

        #---------------------------
        # Iterate until convergence
        #---------------------------
        no_convergence = 1
        convergence_threshold = 1e-3
        c = 0

        # Initialize parameters
        beta_prev = np.zeros([p+1,1])
        beta = np.zeros([p+1,1])

        beta_prev[0] = beta0_hat[:]
        beta_prev[1:] = beta_hat[:]
        z_prev = beta_prev[0] + np.dot(x, beta_prev[1:])
        z = z_prev

        # Initialize gradient/ Hessian
        g = np.zeros([p+1,1])
        h = np.zeros([p+1,1])

        grad_beta0, grad_beta = grad_loss(beta0_hat, beta_hat, alpha, rl, x, y)
        g[0] = grad_beta0
        g[1:] = grad_beta

        hess_beta0, hess_beta = hessian_loss(beta0_hat, beta_hat, alpha, rl, x, y)
        h[0] = hess_beta0
        h[1:] = hess_beta

        # Middle loop for cycles
        L = []
        DL = []

        # Initialize active set
        K = range(p)

        while(no_convergence):
            # Inner loop for coordinate descent
            print('  Cycle: {}\n'.format(c))
            for k in K:
                if np.mod(k,10) == 0:
                    print('    parameter: {}'.format(k))

                # Update the parameters
                #TODO: include a line search in place of h
                #TODO: include momentum
                beta[k] = beta_prev[k] - g[k]/h[k]

                #print 'beta_k: {}'.format(beta[k])
                #print 'lambda*alpha: {}'.format(rl*alpha)

                # Soft threshold for k > 0
                if k>0:
                    beta[k] = prox(beta[k], rl*alpha)

                #print 'soft thresholded beta_k: {}'.format(beta[k])

                #Update z
                z = z_prev - g[k]/h[k]*x[:,k]

                #Calculate gradient
                g[k] = grad_loss_k(z, beta[k], alpha, rl, x[:,k], y, k)

                #Calculate Hessian
                h[k] = hess_loss_k(z, alpha, rl, x[:,k], y, k)

                # Housekeeping
                beta_prev[k] = beta[k]
                z_prev = z

            # Calculate loss and convergence criteria
            L.append(loss(beta[0], beta[1:], alpha, rl, x, y))
            print('    Loss function: {}').format(L[c])

            # Update active set
            K = list(set(K) - set([i for (i, val) in enumerate(beta[1:]) if val == 0]))
            print "    Active set size:{}".format(len(K))

            # Delta loss and convergence criterion
            if c > 0:
                DL.append(L[c] - L[c-1])
                print('    dL/L: {}').format(DL[-1]/L[-1])
                if(np.abs(DL[-1]/L[-1]) < convergence_threshold):
                    no_convergence = 0
                    print('Converged\n')
            c = c+1

        #Store the parameters after convergence
        fit[-1]['beta0'] = beta[0]
        fit[-1]['beta'] = beta[1:]

    return fit
```

### Pathwise proximal batch gradient descent for elastic net regularized GLM

```python
# Native python function
def fitmodel_graddesc(x, y, reg_params, opt_params):

    # Implements batch gradient descent (i.e. vanilla gradient descent by computing gradient over entire training set)

    # Dataset shape
    n = x.shape[0]
    p = x.shape[1]

    # Regularization parameters
    reg_lambda = fit_params['reg_lambda']
    alpha = fit_params['alpha']

    # Optimization parameters
    max_iter = opt_params['max_iter']
    e = opt_params['learning_rate']
    p1 = opt_params['p1'] #first order moment decay rate
    p2 = opt_params['p2'] #second order moment decay rate

    # Initialize parameters
    beta0_hat = np.random.normal(0.0,1.0,1)
    beta_hat = np.random.normal(0.0,1.0,[p,1])
    fit = []

    # Outer loop with descending lambda
    for l,rl in enumerate(reg_lambda):
        fit.append({'beta0': 0., 'beta': np.zeros([p,1])})
        print('Lambda: {}\n').format(rl)

        # Warm initialize parameters
        if(l == 0):
            fit[-1]['beta0'] = beta0_hat
            fit[-1]['beta'] = beta_hat
        else:
            fit[-1]['beta0'] = fit[-2]['beta0']
            fit[-1]['beta'] = fit[-2]['beta']

        #---------------------------
        # Iterate until convergence
        #---------------------------
        no_convergence = 1
        convergence_threshold = 1e-3
        t = 0

        # Initialize parameters
        beta = np.zeros([p+1,1])
        beta[0] = beta0_hat[:]
        beta[1:] = beta_hat[:]

        # Initialize moment parameters
        #s = np.zeros([p+1,1])
        #r = np.zeros([p+1,1])
        g = np.zeros([p+1,1])

        # Initialize cost
        L = []
        DL = []

        while(no_convergence and t < max_iter):

            #Calculate gradient
            grad_beta0, grad_beta = grad_loss(beta[0], beta[1:], alpha, rl, x, y)
            g[0] = grad_beta0
            g[1:] = grad_beta

            # Update time step
            t = t+1

            # Update moment terms
            #s = p1*s + (1-p1)*g
            #r = p2*r + (1-p2)*g*g

            # Correct bias in moment terms
            #s = s/(1-p1**t)
            #r = r/(1-p2**t)

            # Update parameters
            #delta_beta = -np.divide(s, np.sqrt(r) + 1e-6)
            delta_beta = -e*g
            beta = beta + delta_beta

            # Apply proximal operator for L1-regularization
            beta[1:] = prox(beta[1:], rl*alpha)

            # Calculate loss and convergence criteria
            L.append(loss(beta[0], beta[1:], alpha, rl, x, y))
            #print('    Loss function: {}').format(L[c])

            # Delta loss and convergence criterion
            if t > 1:
                DL.append(L[-1] - L[-2])
                #print('    dL/L: {}').format(DL[-1]/L[-1])
                if(np.abs(DL[-1]/L[-1]) < convergence_threshold):
                    no_convergence = 0
                    print('Converged')
                    print('    Loss function: {}').format(L[-1])
                    print('    dL/L: {}\n').format(DL[-1]/L[-1])

        #Store the parameters after convergence
        fit[-1]['beta0'] = beta[0]
        fit[-1]['beta'] = beta[1:]

    return fit
```

```python

```


# ★ Partial Differential Equations ★

```python
# Import modules
import numpy as np
import scipy
import sympy as sym
from scipy import sparse
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import axes3d

from IPython.display import Math
from IPython.display import display

sym.init_printing(use_latex=True)
```

# 8.1 Parabolic Equations

## Forward Difference Method

```python
def heatfd(xl, xr, yb, yt, M, N):
    f = lambda x : np.power(np.sin(2 * np.pi * x), 2)
    l = lambda t : 0 * t
    r = lambda t : 0 * t
    D = 1
    h = (xr - xl) / M
    k = (yt - yb) / N
    m = M - 1
    n = N
    sigma = D * k / np.power(h, 2)
    A = np.diag(1 - 2 * sigma * np.ones(m)) + \
        np.diag(sigma * np.ones(m - 1), 1) + \
        np.diag(sigma * np.ones(m - 1), -1)

    lside = l(yb + np.arange(n) * k)
    rside = r(yb + np.arange(n) * k)

    x = sym.Symbol('x')
    expr = sym.sin(2 * sym.pi * x) ** 2
#     expr = sym.diff(expr, x)

    w = np.zeros(n * m).reshape(n, m).astype(np.float128)
    for i in range(m):
        w[0, i] = expr.subs(x, xl + (i + 1) * h).evalf()

    for j in range(n - 1):
        ww = np.zeros(m)
        ww[0] = lside[j]
        ww[-1] = rside[j]
        v = np.matmul(A, w[j]) + sigma * ww
        w[j + 1,:] = v

    w = np.column_stack([lside, w, rside])
    x = np.arange(0, m+2) * h
    t = np.arange(0, n) * k

    X, T = np.meshgrid(x, t)

    fig = plt.figure()
    ax = fig.gca(projection='3d')
    ax.plot_surface(X, T, w)
    plt.show()
    plt.clf()

heatfd(0, 1, 0, 1, 30, 2000)
```

## Backward Difference Method

```python
def heatbd(xl, xr, yb, yt, M, N):
    f = lambda x : np.sin(2 * np.pi * x) ** 2
    l = lambda t : 0 * t
    r = lambda t : 0 * t
    h = (xr - xl) / M
    k = (yt - yb) / N
    m = M - 1
    n = N
    D = 1 # diffusion coefficient
    sigma = D * k / (h ** 2)

    A = np.diag(1 + 2 * sigma * np.ones(m)) + \
        np.diag(-sigma * np.ones(m - 1), 1) + \
        np.diag(-sigma * np.ones(m - 1), -1)

    lside = l(yb + np.arange(n) * k)
    rside = r(yb + np.arange(n) * k)

    ''' Initial conditions '''
    x = sym.Symbol('x')
    expr = sym.sin(2 * sym.pi * x) ** 2
#     expr = sym.diff(expr, x)

    w = np.zeros(n * m).reshape(n, m).astype(np.float128)
    for i in range(m):
        w[0, i] = expr.subs(x, xl + (i + 1) * h).evalf()

    for j in range(n - 1):
        ww = np.zeros(m)
        ww[0] = lside[j]
        ww[-1] = rside[j]
        v = np.matmul(np.linalg.inv(A), w[j,:] + sigma * ww)
        w[j + 1,:] = v

    w = np.column_stack([lside, w, rside])
    x = np.arange(0, m+2) * h
    t = np.arange(0, n) * k

    X, T = np.meshgrid(x, t)

    fig = plt.figure()
    ax = fig.gca(projection='3d')
    ax.plot_surface(X, T, w)
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show()
    plt.clf()

heatbd(0, 1, 0, 1, 30, 20)
```

### Example

Apply the Backward Difference Method to solve the heat equation

$$
\left\{\begin{matrix}\begin{align*}
& u_t = 4u_{xx}\: & for\:all\:0 \leq x \leq 1\:,0 \leq t \leq 1 \\
& u(x,0) = e^{-x/2}\: & for\:all\:0 \leq x \leq 1 \\
& u(0,t) = e^t\: & for\:all\:0 \leq t \leq 1 \\
& u(1,t) = e^{t-1/2}\: & for\:all\:0 \leq t \leq 1
\end{align*}\end{matrix}\right.
$$

```python
def heatbd(xl, xr, yb, yt, M, N):
    l = lambda t : np.exp(t)
    r = lambda t : np.exp(t - 0.5)
    h = (xr - xl) / M
    k = (yt - yb) / N
    m = M - 1
    n = N
    D = 4 # diffusion coefficient
    sigma = D * k / (h ** 2)

    A = np.diag((1 + 2 * sigma) * np.ones(m)) + \
        np.diag(-sigma * np.ones(m - 1), 1) + \
        np.diag(-sigma * np.ones(m - 1), -1)

    lside = l(yb + np.arange(n) * k)
    rside = r(yb + np.arange(n) * k)

    ''' Initial conditions '''
    x = sym.Symbol('x')
    expr = sym.exp(-x / 2)
#     expr = sym.diff(expr, x)

    w = np.zeros(n * m).reshape(n, m).astype(np.float128)
    for i in range(m):
        w[0, i] = expr.subs(x, xl + (i + 1) * h).evalf()

    for j in range(n - 1):
        ww = np.zeros(m)
        ww[0] = lside[j]
        ww[-1] = rside[j]
        v = np.matmul(np.linalg.inv(A), w[j,:] + sigma * ww)
        w[j + 1,:] = v

    w = np.column_stack([lside, w, rside])
    x = np.arange(0, m+2) * h
    t = np.arange(0, n) * k

    X, T = np.meshgrid(x, t)

    fig = plt.figure()
    ax = fig.gca(projection='3d')
    stride = 0
    ax.plot_surface(X[stride:], T[stride:], w[stride:])
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show()
    plt.clf()

heatbd(0, 1, 0, 1, 20, 100)
```

### Example

Apply the Backward Difference Method to solve the heat equation with homogeneous Neumann boundary conditions

$$
\left\{\begin{matrix}\begin{align*}
& u_t = u_{xx}\: & for\:all\:0 \leq x \leq 1\:,0 \leq t \leq 1 \\
& u(x,0) = \sin^2{2\pi x}\: & for\:all\:0 \leq x \leq 1 \\
& u(0,t) = 0\: & for\:all\:0 \leq t \leq 1 \\
& u(1,t) = 0\: & for\:all\:0 \leq t \leq 1
\end{align*}\end{matrix}\right.
$$

```python
def heatbdn(xl, xr, yb, yt, M, N):
    h = (xr - xl) / M
    k = (yt - yb) / N
    m = M - 1
    n = N
    D = 1 # diffusion coefficient
    sigma = D * k / (h ** 2)

    A = np.diag((1 + 2 * sigma) * np.ones(m)) + \
        np.diag(-sigma * np.ones(m - 1), 1) + \
        np.diag(-sigma * np.ones(m - 1), -1)

    A[0,:3] = np.array([-3, 4, -1])
    A[-1,-3:] = np.array([-1, 4, -3])

    ''' Initial conditions '''
    x = sym.Symbol('x')
    expr = sym.sin(2 * sym.pi * x) ** 2
#     expr = sym.diff(expr, x)

    w = np.zeros(n * m).reshape(n, m).astype(np.float128)
    for i in range(m):
        w[0, i] = expr.subs(x, xl + (i + 1) * h).evalf()

    for j in range(n - 1):
        b = w[j,:]
        b[0] = 0
        b[-1] = 0
        w[j + 1,:] = np.matmul(np.linalg.inv(A), b)

    x = np.arange(0, m) * h
    t = np.arange(0, n) * k

    X, T = np.meshgrid(x, t)

    fig = plt.figure()
    ax = fig.gca(projection='3d')
    stride = 0
    ax.plot_surface(X[stride:], T[stride:], w[stride:])
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show()
    plt.clf()

heatbdn(0, 1, 0, 1, 20, 20)
```

## Crank-Nicolson Method

```python
def crank_nicolson_heat(xl, xr, yb, yt, M, N):
    l = lambda t : 0 * t
    r = lambda t : 0 * t
    D = 1
    h = (xr - xl) / M
    k = (yt - yb) / N
    m = M - 1
    n = N
    sigma = D * k / (h ** 2)

    A = np.diag((2 + 2 * sigma) * np.ones(m)) + \
        np.diag(-sigma * np.ones(m - 1), 1) + \
        np.diag(-sigma * np.ones(m - 1), -1)

    B = np.diag((2 - 2 * sigma) * np.ones(m)) + \
        np.diag(sigma * np.ones(m - 1), 1) + \
        np.diag(sigma * np.ones(m - 1), -1)

    lside = l(yb + np.arange(n) * k)
    rside = r(yb + np.arange(n) * k)

    ''' Initial conditions '''
    x = sym.Symbol('x')
    expr = sym.sin(2 * sym.pi * x) ** 2
#     expr = sym.diff(expr, x)

    w = np.zeros(n * m).reshape(n, m).astype(np.float128)
    for i in range(m):
        w[0, i] = expr.subs(x, xl + (i + 1) * h).evalf()

    for j in range(n - 1):
        s = np.zeros(m)
        s[0] = lside[j] + lside[j+1]
        s[-1] = rside[j] + rside[j+1]
        w[j + 1,:] = np.matmul(np.linalg.inv(A), np.matmul(B, w[j,:]) + sigma * s)

    w = np.column_stack([lside, w, rside])
    x = xl +np.arange(0, m+2) * h
    t = yb + np.arange(0, n) * k

    X, T = np.meshgrid(x, t)

    fig = plt.figure()
    ax = fig.gca(projection='3d')
    stride = 0
    ax.plot_surface(X[stride:], T[stride:], w[stride:])
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show()
    plt.clf()

crank_nicolson_heat(0, 0.5, 0, 1, 30, 100)
```

### Example

Apply the Crank-Nicolson Method to the heat equation

$$
\left\{\begin{matrix}\begin{align*}
& u_t = Du_{xx} + Cu \\
& u(x,0) = \sin^2{(\frac{\pi}{L} x)}\: &for\:all\:0 \leq x \leq L \\
& u(0,t) = 0\: &for\:all\:t \geq 0 \\
& u(L,t) = 0\: &for\:all\:t \geq 0
\end{align*}\end{matrix}\right.
$$

```python
def crank_nicolson_growth(xl, xr, yb, yt, M, N):
    l = lambda t : 0 * t
    r = lambda t : 0 * t
    D = 1
    L = 1
    C = 9.5
    h = (xr - xl) / M
    k = (yt - yb) / N
    m = M - 1
    n = N
    sigma = D * k / h ** 2

    A = np.diag((2 - k * C + 2 * sigma) * np.ones(m)) + \
        np.diag(-sigma * np.ones(m - 1), 1) + \
        np.diag(-sigma * np.ones(m - 1), -1)

    B = np.diag((2 + k * C - 2 * sigma) * np.ones(m)) + \
        np.diag(sigma * np.ones(m - 1), 1) + \
        np.diag(sigma * np.ones(m - 1), -1)

    lside = l(yb + np.arange(n) * k)
    rside = r(yb + np.arange(n) * k)

    ''' Initial conditions '''
    f = lambda x : np.power(np.sin(np.pi * x / L), 2)

    w = np.zeros(n * m).reshape(n, m).astype(np.float128)
    for i in range(m):
        w[0, i] = f(xl + (i + 1) * h)

    for j in range(n - 1):
        s = np.zeros(m)
        s[0] = lside[j] + lside[j+1]
        s[-1] = rside[j] + rside[j+1]
        w[j + 1,:] = np.matmul(np.linalg.inv(A), np.matmul(B, w[j,:]) + sigma * s)

    w = np.column_stack([lside, w, rside])
    x = xl + np.arange(0, m+2) * h
    t = yb + np.arange(0, n) * k

    X, T = np.meshgrid(x, t)

    fig = plt.figure()
    ax = fig.gca(projection='3d')
    stride = 0
    ax.plot_surface(X[stride:], T[stride:], w[stride:])
    plt.xlabel('x')
    plt.ylabel('t')
    plt.show()
    plt.clf()

crank_nicolson_growth(0, 1, 0, 1, 20, 20)
```

# 8.2 Hyperbolic Equations

### Example

Apply the explicit Finite Difference Method to the wave equation with wave speed $c = 2$ and initial conditions $f(x) = \sin{\pi x}$ and $g(x) = l(x) = r(x) = 0$

```python
def wavefd(xl, xr, yb, yt, M, N):
    c = 1
    h = (xr - xl) / M
    k = (yt - yb) / N
    m = M - 1
    n = N

    sigma = c * k / h

    f = lambda x : np.sin(x * np.pi)
    l = lambda x : 0 * x
    r = lambda x : 0 * x
    g = lambda x : 0 * x

    lside = l(yb + np.arange(n) * k)
    rside = r(yb + np.arange(n) * k)

    A = np.diag((2 - 2 * sigma ** 2) * np.ones(m)) + \
        np.diag((sigma ** 2) * np.ones(m - 1), 1) + \
        np.diag((sigma ** 2) * np.ones(m - 1), -1)

    '''Initial condition'''
    w = np.zeros(n * m).reshape(n, m).astype(np.float128)
    xv = np.linspace(0, 1, M + 1)[1:-1]
    w[0, :] = f(xv)
    w[1, :] = 0.5 * np.matmul(A, w[0, :]) + \
              k * g(xv) + \
              0.5 * np.power(sigma, 2) * np.array([lside[0], *np.zeros(m - 2), rside[0]])

    for i in range(2, n - 1):
        w[i,:] = np.matmul(A, w[i-1,:]) - w[i-2,:] + np.power(sigma, 2) * \
                 np.array([lside[i-1], *np.zeros(m - 2), rside[i-1]])

    w = np.column_stack([lside, w, rside])
    x = xl + np.arange(0, m + 2) * h
    t = yb + np.arange(0, n) * k

    X, T = np.meshgrid(x, t)

    fig = plt.figure()
    ax = fig.gca(projection='3d')
    stride = 0
    ax.plot_wireframe(X[stride:], T[stride:], w[stride:])
#     ax.plot_surface(X[stride:], T[stride:], w[stride:])
    ax.view_init(azim=60, elev=30)
    plt.xlabel('x')
    plt.ylabel('t')
    plt.show()
    plt.clf()

wavefd(0, 1, 0, 1, 20, 20)
```

## The CFL condition

The finite Difference Method is applied to the wave equation with wave speed $c > 0$ is stable if $\sigma = \frac{ck}{h} \leq 1$

```python
def wavefd_cfl(xl, xr, yb, yt, M, N, C = 1):
    c = C
    h = (xr - xl) / M
    k = (yt - yb) / N

    if c * k > h:
        raise ValueError("CFL condition 'c * k <= h' is not satisfied, c * k is %f and h is %f" %(c * k, h) )

    m = M - 1
    n = N

    sigma = c * k / h

    f = lambda x : np.sin(x * np.pi)
    l = lambda x : 0 * x
    r = lambda x : 0 * x
    g = lambda x : 0 * x

    lside = l(yb + np.arange(n) * k)
    rside = r(yb + np.arange(n) * k)

    A = np.diag((2 - 2 * sigma ** 2) * np.ones(m)) + \
        np.diag((sigma ** 2) * np.ones(m - 1), 1) + \
        np.diag((sigma ** 2) * np.ones(m - 1), -1)

    '''Initial condition'''
    w = np.zeros(n * m).reshape(n, m).astype(np.float128)
    xv = np.linspace(0, 1, M + 1)[1:-1]
    w[0, :] = f(xv)
    w[1, :] = 0.5 * np.matmul(A, w[0, :]) + \
              k * g(xv) + \
              0.5 * np.power(sigma, 2) * np.array([lside[0], *np.zeros(m - 2), rside[0]])

    for i in range(2, n - 1):
        w[i,:] = np.matmul(A, w[i-1,:]) - w[i-2,:] + np.power(sigma, 2) * \
                 np.array([lside[i-1], *np.zeros(m - 2), rside[i-1]])

    w = np.column_stack([lside, w, rside])
    x = xl + np.arange(0, m + 2) * h
    t = yb + np.arange(0, n) * k

    X, T = np.meshgrid(x, t)

    fig = plt.figure()
    ax = fig.gca(projection='3d')
    stride = 0
    ax.plot_wireframe(X[stride:], T[stride:], w[stride:])
#     ax.plot_surface(X[stride:], T[stride:], w[stride:])
    ax.view_init(azim=20, elev=20)
    plt.xlabel('x')
    plt.ylabel('t')
    plt.show()
    plt.clf()

wavefd_cfl(0, 1, 0, 1, 20, 200, 6)
```

# 8.3 Elliptic Equations

### Example

Apply the Finite Difference Method with m = n = 5 to approximate the solution of  the Laplace equation $\Delta{u} = 0$ on $[0,1] \times [1,2]$ with the following Dirichlet boundary conditions :

$$
\begin{matrix}\begin{align*}
u(x,1) &= \ln{(x^2 + 1)} \\
u(x,2) &= \ln{(x^2 + 4)} \\
u(0,y) &= 2\ln{y} \\
u(1,y) &= \ln{(y^2 + 1)}
\end{align*}\end{matrix}
$$

```python
def poisson(xl, xr, yb, yt, M, N):
    f = lambda x, y : 0
    g1 = lambda x : np.log(pow(x, 2) + 1)
    g2 = lambda x : np.log(pow(x, 2) + 4)
    g3 = lambda y : 2 * np.log(y)
    g4 = lambda y : np.log(pow(y, 2) + 1)

    m, n = M + 1, N + 1
    mn = m * n
    h, k = (xr - xl) / M, (yt - yb) / N
    h2, k2 = pow(h, 2), pow(k, 2)
    x = xl + np.arange(M + 1) * h
    y = yb + np.arange(N + 1) * k
    A = np.zeros((mn, mn))
    b = np.zeros((mn, 1))

    ''' interior points '''
    for i in range(2, m):
        for j in range(2, n):
            A[i+(j-1)*m - 1][i-1+(j-1)*m - 1] = 1 / h2
            A[i+(j-1)*m - 1][i+1+(j-1)*m - 1] = 1 / h2
            A[i+(j-1)*m - 1][i+(j-1)*m - 1] = - 2 / h2 - 2 / k2
            A[i+(j-1)*m - 1][i+(j-2)*m - 1] = 1 / k2
            A[i+(j-1)*m - 1][i+j*m - 1] = 1 / k2
            b[i+(j-1)*m - 1] = f(x[i], y[j])

    ''' bottom and top boundary points '''
    for i in range(1, m + 1):
        j = 1
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g1(x[i - 1])
        j = n
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g2(x[i - 1])

    ''' left and right boundary points '''
    for j in range(2, n):
        i = 1
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g3(y[j - 1])
        i = m
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g4(y[j - 1])

    v = np.matmul(np.linalg.inv(A), b)
    w = v.reshape(n, m).T

    X, Y = np.meshgrid(x, y)

    fig = plt.figure()
    ax =fig.gca(projection='3d')
    ax.view_init(azim=225)
    ax.plot_surface(X, Y, w)
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show(fig)
    plt.close()

poisson(0, 1, 1, 2, 4, 4)
poisson(0, 1, 1, 2, 10, 10)
```

### Example

Find the electrostatic potential on the square $[0,1] \times [0,1]$, assuming no charge in the interior and assuming the following boundary conditions:

$$
\begin{matrix}\begin{align*}
u(x,0) &= \sin{\pi x} \\
u(x,1) &= \sin{\pi x} \\
u(0,y) &= 0 \\
u(1,y) &= 0
\end{align*}\end{matrix}
$$

```python
def poisson(xl, xr, yb, yt, M, N):
    f = lambda x, y : 0
    g1 = lambda x : np.sin(x * np.pi)
    g2 = lambda x : np.sin(x * np.pi)
    g3 = lambda y : 0
    g4 = lambda y : 0

    m, n = M + 1, N + 1
    mn = m * n
    h, k = (xr - xl) / M, (yt - yb) / N
    h2, k2 = pow(h, 2), pow(k, 2)
    x = xl + np.arange(M + 1) * h
    y = yb + np.arange(N + 1) * k
    A = np.zeros((mn, mn))
    b = np.zeros((mn, 1))

    ''' interior points '''
    for i in range(2, m):
        for j in range(2, n):
            A[i+(j-1)*m - 1][i-1+(j-1)*m - 1] = 1 / h2
            A[i+(j-1)*m - 1][i+1+(j-1)*m - 1] = 1 / h2
            A[i+(j-1)*m - 1][i+(j-1)*m - 1] = - 2 / h2 - 2 / k2
            A[i+(j-1)*m - 1][i+(j-2)*m - 1] = 1 / k2
            A[i+(j-1)*m - 1][i+j*m - 1] = 1 / k2
            b[i+(j-1)*m - 1] = f(x[i], y[j])

    ''' bottom and top boundary points '''
    for i in range(1, m + 1):
        j = 1
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g1(x[i - 1])
        j = n
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g2(x[i - 1])

    ''' left and right boundary points '''
    for j in range(2, n):
        i = 1
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g3(y[j - 1])
        i = m
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g4(y[j - 1])

    v = np.matmul(np.linalg.inv(A), b)
    w = v.reshape(n, m).T

    X, Y = np.meshgrid(x, y)

    fig = plt.figure()
    ax =fig.gca(projection='3d')
    ax.view_init(azim=225)
    ax.plot_surface(X, Y, w)
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show(fig)
    plt.close()

poisson(0, 1, 0, 1, 10, 10)
```

## Finite Element Method for elliptic equations

```python
def poissonfem(xl, xr, yb, yt, M, N):
    f = lambda x, y : 0
    r = lambda x, y : 0
    g1 = lambda x : np.log(pow(x, 2) + 1)
    g2 = lambda x : np.log(pow(x, 2) + 4)
    g3 = lambda y : 2 * np.log(y)
    g4 = lambda y : np.log(pow(y, 2) + 1)

    m, n = M + 1, N + 1
    mn = m * n
    h, k = (xr - xl) / M, (yt - yb) / N
    hk = h * k
    h2, k2 = pow(h, 2), pow(k, 2)
    x = xl + np.arange(M + 1) * h
    y = yb + np.arange(N + 1) * k
    A = np.zeros((mn, mn))
    b = np.zeros((mn, 1))

    B1 = lambda i, j : (x[i] - 2 * h / 3, y[j] - k / 3)
    B2 = lambda i, j : (x[i] - h / 3, y[j] - 2 * k / 3)
    B3 = lambda i, j : (x[i] + h / 3, y[j] - k / 3)
    B4 = lambda i, j : (x[i] + 2 * h / 3, y[j] + k / 3)
    B5 = lambda i, j : (x[i] + h / 3, y[j] + 2 * k / 3)
    B6 = lambda i, j : (x[i] - h / 3, y[j] + k / 3)

    ''' interior points '''
    for i in range(2, m):
        for j in range(2, n):
            rsum = r(*B1(i,j)) + r(*B2(i,j)) + r(*B3(i,j)) + r(*B4(i,j)) + r(*B5(i,j)) + r(*B6(i,j))
            fsum = f(*B1(i,j)) + f(*B2(i,j)) + f(*B3(i,j)) + f(*B4(i,j)) + f(*B5(i,j)) + f(*B6(i,j))
            A[i+(j-1)*m - 1][i+(j-1)*m - 1] = 2 * (h2 + k2) / hk - hk * rsum / 18
            A[i+(j-1)*m - 1][i-1+(j-1)*m - 1] = -k/h - hk * (r(*B1(i,j)) + r(*B6(i,j))) / 18
            A[i+(j-1)*m - 1][i-1+(j-2)*m - 1] = -hk * (r(*B1(i,j)) + r(*B2(i,j))) / 18
            A[i+(j-1)*m - 1][i+(j-2)*m - 1] = -h/k - hk * (r(*B2(i,j)) + r(*B3(i,j))) / 18
            A[i+(j-1)*m - 1][i+1+(j-1)*m - 1] = -k/h - hk * (r(*B3(i,j)) + r(*B4(i,j))) / 18
            A[i+(j-1)*m - 1][i+1+j*m - 1] = -hk * (r(*B4(i,j)) + r(*B5(i,j))) / 18
            A[i+(j-1)*m - 1][i+j*m - 1] = - h / k - hk * (r(*B5(i,j)) + r(*B6(i,j))) / 18
            b[i+(j-1)*m - 1] = - h * k * fsum / 6

    ''' bottom and top boundary points '''
    for i in range(1, m + 1):
        j = 1
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g1(x[i - 1])
        j = n
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g2(x[i - 1])

    ''' left and right boundary points '''
    for j in range(2, n):
        i = 1
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g3(y[j - 1])
        i = m
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g4(y[j - 1])

    v = np.matmul(np.linalg.inv(A), b)
    w = v.reshape(n, m).T

    X, Y = np.meshgrid(x, y)

    fig = plt.figure()
    ax =fig.gca(projection='3d')
    ax.view_init(azim=225)
    ax.plot_surface(X, Y, w)
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show(fig)
    plt.close()

poissonfem(0, 1, 1, 2, 4, 4)
poissonfem(0, 1, 1, 2, 10, 10)
```

### Example

Apply the Finite Element Method with M = N = 16 to approximate the solution of the elliptic Dirichlet problem

$$
\left\{\begin{matrix}\begin{align*}
 & \Delta{u} + 4\pi^2u = 2 \sin{2\pi y} \\
 & u(x,0) = 0\text{ for } 0 \le x \le 1 \\
 & u(x,1) = 0\text{ for } 0 \le x \le 1 \\
 & u(0,y) = 0\text{ for } 0 \le y \le 1 \\
 & u(1,y) = \sin{2\pi y}\text{ for } 0 \le y \le 1
\end{align*}\end{matrix}\right.
$$

```python
def poissonfem(xl, xr, yb, yt, M, N):
    f = lambda x, y : 2 * np.sin(2 * np.pi * y)
    r = lambda x, y : 4 * pow(np.pi, 2)
    g1 = lambda x : 0
    g2 = lambda x : 0
    g3 = lambda y : 0
    g4 = lambda y : np.sin(2 * np.pi * y)

    m, n = M + 1, N + 1
    mn = m * n
    h, k = (xr - xl) / M, (yt - yb) / N
    hk = h * k
    h2, k2 = pow(h, 2), pow(k, 2)
    x = xl + np.arange(M + 1) * h
    y = yb + np.arange(N + 1) * k
    A = np.zeros((mn, mn))
    b = np.zeros((mn, 1))

    B1 = lambda i, j : (x[i] - 2 * h / 3, y[j] - k / 3)
    B2 = lambda i, j : (x[i] - h / 3, y[j] - 2 * k / 3)
    B3 = lambda i, j : (x[i] + h / 3, y[j] - k / 3)
    B4 = lambda i, j : (x[i] + 2 * h / 3, y[j] + k / 3)
    B5 = lambda i, j : (x[i] + h / 3, y[j] + 2 * k / 3)
    B6 = lambda i, j : (x[i] - h / 3, y[j] + k / 3)

    ''' interior points '''
    for i in range(2, m):
        for j in range(2, n):
            rsum = r(*B1(i,j)) + r(*B2(i,j)) + r(*B3(i,j)) + r(*B4(i,j)) + r(*B5(i,j)) + r(*B6(i,j))
            fsum = f(*B1(i,j)) + f(*B2(i,j)) + f(*B3(i,j)) + f(*B4(i,j)) + f(*B5(i,j)) + f(*B6(i,j))
            A[i+(j-1)*m - 1][i+(j-1)*m - 1] = 2 * (h2 + k2) / hk - hk * rsum / 18
            A[i+(j-1)*m - 1][i-1+(j-1)*m - 1] = -k/h - hk * (r(*B1(i,j)) + r(*B6(i,j))) / 18
            A[i+(j-1)*m - 1][i-1+(j-2)*m - 1] = -hk * (r(*B1(i,j)) + r(*B2(i,j))) / 18
            A[i+(j-1)*m - 1][i+(j-2)*m - 1] = -h/k - hk * (r(*B2(i,j)) + r(*B3(i,j))) / 18
            A[i+(j-1)*m - 1][i+1+(j-1)*m - 1] = -k/h - hk * (r(*B3(i,j)) + r(*B4(i,j))) / 18
            A[i+(j-1)*m - 1][i+1+j*m - 1] = -hk * (r(*B4(i,j)) + r(*B5(i,j))) / 18
            A[i+(j-1)*m - 1][i+j*m - 1] = - h / k - hk * (r(*B5(i,j)) + r(*B6(i,j))) / 18
            b[i+(j-1)*m - 1] = - h * k * fsum / 6

    ''' bottom and top boundary points '''
    for i in range(1, m + 1):
        j = 1
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g1(x[i - 1])
        j = n
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g2(x[i - 1])

    ''' left and right boundary points '''
    for j in range(2, n):
        i = 1
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g3(y[j - 1])
        i = m
        A[i+(j-1)*m - 1][i+(j-1)*m - 1]=1
        b[i+(j-1)*m - 1] = g4(y[j - 1])

    v = np.matmul(np.linalg.inv(A), b)
    w = v.reshape(n, m).T

    X, Y = np.meshgrid(x, y)

    fig = plt.figure()
    ax =fig.gca(projection='3d')
    ax.view_init(azim=225)
    ax.plot_surface(X, Y, w)
    plt.xlabel('x')
    plt.ylabel('y')
    plt.show(fig)
    plt.close()

poissonfem(0, 1, 0, 1, 16, 16)
```

# 8.4 Nonlinear partial differential equations

### Example

Use the Backward Difference Equation with Newton iteration to solve Burgers' equation

$$
\left\{\begin{matrix}\begin{align*}
& u_t + uu_x = Du_{xx} \\
& u(x,0) = \frac{2D\beta\pi\sin{\pi x}}{\alpha + \beta\cos{\pi x}}\text{ for } 0 \le x \le 1 \\
& u(0,t) = 0\text{ for all } t \ge 0 \\
& u(1,t) = 0\text{ for all } t \ge 0
\end{align*}\end{matrix}\right.
$$

```python
def burgers(xl, xr, tb, te, M, N):
    alpha = 5
    beta = 4
    D = 0.05
    f = lambda x : 2 * D * beta * np.pi * np.sin(x * np.pi) / (alpha + beta * np.cos(np.pi * x))
    l = lambda t : 0 * t
    r = lambda t : 0 * t

    h, k = (xr - xl) / M, (te - tb) / N
    m, n = M + 1, N
    sigma = D * k / (h * h)

    w = np.zeros((M + 1) * (n + 1)).reshape(M + 1, n + 1)
    w[:, 0] = f(xl + np.arange(M + 1) * h)
    w1 = np.copy(w[:, 0])

    for j in range(0, n):
        for it in range(3):
            DF1 = np.diag(1 + 2 * sigma * np.ones(m)) + np.diag(-sigma * np.ones(m-1), 1) \
                + np.diag(-sigma * np.ones(m-1), -1)
            DF2 = np.diag([0,*(k * w1[2:m] / (2 * h)),0]) - np.diag([0,*(k * w1[0:m - 2] / (2 * h)),0]) \
                + np.diag([0,*(k * w1[1:m - 1] / (2 * h))], 1) - np.diag([*(k * w1[1:m - 1] / (2 * h)), 0], -1)
            DF = DF1 + DF2;
            F = -w[:,j] + np.matmul((DF1 + DF2 / 2), w1)
            DF[0,:] = np.array([1, *np.zeros(m-1)])
            F[0] = w1[0] - l(j)
            F[m-1] = w1[m-1] - r(j)
            w1 -= np.matmul(np.linalg.inv(DF), F)
        w[:, j + 1] = w1

    # 3-D Plot
    x = xl + np.arange(M + 1) * h
    t = tb + np.arange(n + 1) * k
    X, T = np.meshgrid(x, t)
    fig = plt.figure()
    ax = fig.gca(projection='3d')
    ax.view_init(azim=225)
    ax.plot_surface(X, T, w.T)
    plt.xlabel('x')
    plt.ylabel('t')
    plt.show()
    plt.close()

burgers(0, 1, 0, 2, 20, 40)
```


```python
import sympy as sym
from sympy.physics.units.quantities import Quantity
from IPython.display import display, Math, Latex
import pandas as pd
import numpy as np
def func_str(function):
    return str(function)
    function_str = str(function)
    function_str = function_str.replace("**", "^")
    return function_str
class Lagrange:
    def __init__(self, function, *args):
        self.lamb = Quantity("λ")
        self.variables = []
        self.function = function
        for variable in args:
            self.variables.append(variable)
        self.make_delta()
        self.partial_deravative()

    def make_delta(self):
        summation = 0
        for variable in self.variables:
            summation += variable**2
        self.delta =  summation**0.5 + self.lamb * self.function
        display(Math(r"min(\Delta) =  " + func_str(self.delta)))

    def partial_deravative(self):
        self.partial_equations = []
        for i in range(len(self.variables) + 1):
            if i == len(self.variables):
                der_function = sym.diff(self.delta, self.lamb)
                self.partial_equations.append(sym.Eq(der_function, 0))
                display(Math(r"\frac{\partial min(\Delta)}{\partial \lambda} = " + func_str(der_function) + str( " = 0")))
            else:
                der_function = sym.diff(self.delta, self.variables[i])
                self.partial_equations.append(sym.Eq(der_function, 0))
                display(Math(r"\frac{\partial min(\Delta)}{\partial y_%d} = " % (i+1) + func_str(der_function) + str( " = 0")))

    def solve_equations(self):
        self.unknows = []
        for i in range(len(self.variables) + 1):
            if i == len(self.variables):
                self.unknows.append(self.lamb)
            else:
                self.unknows.append(self.variables[i])
        self.result = sym.solve(self.partial_equations, self.unknows)
        return self.result

    def get_beta(self):
        if hasattr(self, "result"):
            return ((float(self.result[0][0])**2) + float((self.result[0][1])**2))**0.5
        else:
            self.result = self.solve_equations()
            return self.get_beta()

y_1, y_2 = sym.symbols('y_1 y_2')

f = 10*y_1 - 4*y_2 + 12

lagrange = Lagrange(f, y_1, y_2)
lagrange.get_beta()
```

$\displaystyle min(\Delta) =  λ*(10*y_1 - 4*y_2 + 12) + (y_1**2 + y_2**2)**0.5$

$\displaystyle \frac{\partial min(\Delta)}{\partial y_1} = 10*λ + 1.0*y_1*(y_1**2 + y_2**2)**(-0.5) = 0$

$\displaystyle \frac{\partial min(\Delta)}{\partial y_2} = -4*λ + 1.0*y_2*(y_1**2 + y_2**2)**(-0.5) = 0$

$\displaystyle \frac{\partial min(\Delta)}{\partial \lambda} = 10*y_1 - 4*y_2 + 12 = 0$

    1.1141720290623114

```python
print("hi")
```


# Fourier coefficient intuition

last revised: 18-Jan-2019 by Dick Furnstahl [furnstahl.1@osu.edu]

Make plots of products of sines and cosines to gain intuition on how Fourier coefficients are projected out.

The Fourier expansion of a periodic function $f(t+\tau) = f(t)$ takes the form

$\begin{align}
  f(t) = \sum_{n=0} a_n \cos(n\omega t) + b_n \sin(n\omega t)
\end{align}$

where $\omega = 2\pi/\tau$.
The coefficients are found from integrals of cosines and sines over $f(t)$:

$\begin{align}
  a_0 = \frac{1}{\tau} \int_{-\tau/2}^{\tau/2} f(t)\, dt  \qquad
  b_0 = 0
\end{align}$

$\begin{align}
  a_m = \frac{2}{\tau} \int_{-\tau/2}^{\tau/2} \cos(m\omega t) f(t)\, dt  \qquad
  b_m = \frac{2}{\tau} \int_{-\tau/2}^{\tau/2} \sin(m\omega t) f(t)\, dt
\end{align}$

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import quad
from ipywidgets import interact

```

```python
def Fourier_integrand_plot(n, m):
    """Plot the integrands of sine and cosines of different orders"""
    tau = 1.
    omega = 2.*np.pi / tau
    t_pts = np.arange(-tau/2., tau/2., 0.01)

    cos_n_pts = np.cos(n * omega * t_pts)
    cos_m_pts = np.cos(m * omega * t_pts)
    sin_n_pts = np.sin(n * omega * t_pts)
    sin_m_pts = np.sin(m * omega * t_pts)

    fig = plt.figure(figsize=(12,4))

    ax1 = fig.add_subplot(1,3,1)
    ax1.plot(t_pts, cos_n_pts*cos_m_pts, color='red')
    ax1.plot(t_pts, cos_n_pts, color='blue', linestyle='dashed', alpha=0.5)
    ax1.plot(t_pts, cos_m_pts, color='green', linestyle='dotted', alpha=0.5)
    ax1.set_ylim(-1.1, 1.1)
    ax1.set_title('cosine * cosine')

    ax2 = fig.add_subplot(1,3,2)
    ax2.plot(t_pts, sin_n_pts*sin_m_pts, color='red')
    ax2.plot(t_pts, sin_n_pts, color='blue', linestyle='dashed', alpha=0.5)
    ax2.plot(t_pts, sin_m_pts, color='green', linestyle='dotted', alpha=0.5)
    ax2.set_ylim(-1.1, 1.1)
    ax2.set_title('sine * sine')

    ax3 = fig.add_subplot(1,3,3)
    ax3.plot(t_pts, cos_n_pts*sin_m_pts, color='red')
    ax3.plot(t_pts, cos_n_pts, color='blue', linestyle='dashed', alpha=0.5)
    ax3.plot(t_pts, sin_m_pts, color='green', linestyle='dotted', alpha=0.8)
    ax3.set_ylim(-1.1, 1.1)
    ax3.set_title('cosine * sine')

```

The plots below show the products of cosines and sines

```python
interact(Fourier_integrand_plot, m=(0,4), n=(0,4));
```

    interactive(children=(IntSlider(value=2, description='n', max=4), IntSlider(value=2, description='m', max=4), …

```python

```


# Creating Unit Models

To demonstrate creation of a new unit model, we will create a constant-heat-capacity ideal-gas isentropic compressor. This will be a simple textbook model. We will utilize the mass and energy balances provided by IDAES control volumes, but we will write our own isentropic constraint based off of equations 7.18 and 7.23 from "Introduction to Chemical Engineering Thermodynamics" by J.M. Smith, H.C. Van Ness, and M.M. Abbott.

The outlet temperature of an ideal gas undergoing isentropic compression is given by
\begin{align}
& t_{out} = t_{in} + \frac{1}{\eta} \left(t_{in} \left(\frac{p_{out}}{p_{in}}\right)^{\frac{\gamma - 1}{\gamma}} - t_{in}\right)
\end{align}
where $p$ is pressure, $t$ is temperature, and $\gamma$ is the ratio of constant pressure heat capacity to constant volume heat capacity.

We will begin with relevant imports. We will need

- Pyomo for writing our energy balance constraints
- ConfigBlocks for specifying options for our compressor
- ControlVolume0DBlocks for creating the appropriate state blocks for the inlet and outlet and for defining mas balances
- IdealParameterBlock which provides a simple ideal-gas property package.
- A few other helpful functions and enums from IDAES

```python
import pyomo.environ as pe
from pyomo.common.config import ConfigBlock, ConfigValue, In
from idaes.core import (ControlVolume0DBlock,
                        declare_process_block_class,
                        EnergyBalanceType,
                        MomentumBalanceType,
                        MaterialBalanceType,
                        UnitModelBlockData,
                        useDefault,
                        FlowsheetBlock)
from idaes.core.util.config import is_physical_parameter_block
from methanol_param_VLE import PhysicalParameterBlock
from idaes.core.util.misc import add_object_reference
```

Now, we can write a function to create a control volume for our compressor. The control volume will define the inlet and outlet streams along with the appropriate state variables (specified by the property package). We will also use the control volume to create mass and energy balance constraints.

Our function will take the compressor unit model object, the name of the control volume, and configuration options as arguments. Our compressor will only support steady-state models, so we will first ensure that ``dynamic`` and ``has_holdup`` are both ``False``.

Next, we will create a 0D control volume. We are using a 0D control volume because our model does not depend on space. We then

1. Attach the control volume to the compressor
2. Create the appropriate state blocks with the control volume (for the inlet and outlet streams)
3. Use the control volume to add mass balance constraints
4. Use the control volume to add energy balance constraints

```python
def make_control_volume(unit, name, config):
    if config.dynamic is not False:
        raise ValueError('IdealGasIsentropcCompressor does not support dynamics')
    if config.has_holdup is not False:
        raise ValueError('IdealGasIsentropcCompressor does not support holdup')

    control_volume = ControlVolume0DBlock(default={"property_package": config.property_package,
                                                   "property_package_args": config.property_package_args})

    setattr(unit, name, control_volume)

    control_volume.add_state_blocks(has_phase_equilibrium=config.has_phase_equilibrium)
    control_volume.add_material_balances(balance_type=config.material_balance_type,
                                         has_phase_equilibrium=config.has_phase_equilibrium)
    control_volume.add_total_enthalpy_balances(has_heat_of_reaction=False,
                                               has_heat_transfer=False,
                                               has_work_transfer=True)
```

Next, we will write a function to add constraints to specify that the compressor is isentropic.
1. Create a ``pressure_ratio`` variable to represent $p_{out}/p_{in}$. The lower bound is $1$, because we only want to allow compression (and not expansion).
2. Create a ``ConstraintList`` to hold the constraints.
3. Add the ``ConstraintList`` to the compressor
4. Create the local variables ``inlet`` and ``outlet`` to reference the inlet and outlet state blocks.
5. Add a constraint relating the inlet pressure, outlet pressure, and pressure ratio variables:
\begin{align}
p_{in} p_{ratio} = p_{out}
\end{align}
6. Add a constraint relating the inlet and outlet temperatures:
\begin{align}
& t_{out} = t_{in} + \frac{1}{\eta} \left(t_{in} p_{ratio}^{\frac{\gamma - 1}{\gamma}} - t_{in}\right)
\end{align}

```python
def add_isentropic(unit, name, config):
    unit.pressure_ratio = pe.Var(initialize=1.0, bounds=(1, None))
    cons = pe.ConstraintList()
    setattr(unit, name, cons)
    inlet = unit.control_volume.properties_in[0.0]
    outlet = unit.control_volume.properties_out[0.0]
    gamma = inlet._params.gamma
    cons.add(inlet.pressure * unit.pressure_ratio == outlet.pressure)
    cons.add(outlet.temperature ==
             (inlet.temperature +
              1/config.compressor_efficiency *
              (inlet.temperature * unit.pressure_ratio**((gamma - 1) / gamma) -
               inlet.temperature)))
```

We also need a function to specify configuration options for the compressor.

```python
def make_compressor_config_block(config):
    config.declare("material_balance_type", ConfigValue(default=MaterialBalanceType.componentPhase, domain=In(MaterialBalanceType)))
    config.declare("energy_balance_type", ConfigValue(default=EnergyBalanceType.enthalpyTotal, domain=In([EnergyBalanceType.enthalpyTotal])))
    config.declare("momentum_balance_type", ConfigValue(default=MomentumBalanceType.none, domain=In([MomentumBalanceType.none])))
    config.declare("has_phase_equilibrium", ConfigValue(default=False, domain=In([False])))
    config.declare("has_pressure_change", ConfigValue(default=False, domain=In([False])))
    config.declare("property_package", ConfigValue(default=useDefault, domain=is_physical_parameter_block))
    config.declare("property_package_args", ConfigBlock(implicit=True))
    config.declare("compressor_efficiency", ConfigValue(default=0.75, domain=float))
```

Finally, we can define the ideal-gas isentropic compressor. To do so, we create a class called ``IdealGasIsentropicCompressorData`` and use the ``declare_process_block_class`` decorator. For now, just consider the decorator to be boiler-plate. We then need to define the config block and write the ``build`` method. The ``build`` method should alwasy call ``super``. Next, we simply call the functions we wrote to build the control volume, energy balance, and electricity requirement performance equation. Finally, we need to call ``self.add_inlet_port()`` and ``self.add_outlet_port()``. These methods need to be called in order to create the ports which are used for connecting the unit to other units.

```python
@declare_process_block_class("IdealGasIsentropicCompressor")
class IdealGasIsentropicCompressorData(UnitModelBlockData):
    CONFIG = UnitModelBlockData.CONFIG()
    make_compressor_config_block(CONFIG)

    def build(self):
        super(IdealGasIsentropicCompressorData, self).build()

        make_control_volume(self, "control_volume", self.config)
        add_isentropic(self, "isentropic", self.config)

        self.add_inlet_port()
        self.add_outlet_port()

        add_object_reference(self, 'work', self.control_volume.work[0.0])
```

The compressor model is complete and can now be used like other IDAES unit models:

```python
m = pe.ConcreteModel()
m.fs = fs = FlowsheetBlock(default={"dynamic": False})
fs.properties = props = PhysicalParameterBlock(default={'Cp': 0.038056, 'valid_phase': 'Vap'})

fs.compressor = IdealGasIsentropicCompressor(default={"property_package": props,
                                                      "has_phase_equilibrium": False})
fs.compressor.inlet.flow_mol.fix(1)
fs.compressor.inlet.mole_frac_comp[0, 'CH3OH'].fix(0.25)
fs.compressor.inlet.mole_frac_comp[0, 'CH4'].fix(0.25)
fs.compressor.inlet.mole_frac_comp[0, 'H2'].fix(0.25)
fs.compressor.inlet.mole_frac_comp[0, 'CO'].fix(0.25)
fs.compressor.inlet.pressure.fix(0.14)
fs.compressor.inlet.temperature.fix(2.9315)
fs.compressor.outlet.pressure.fix(0.56)

opt = pe.SolverFactory('ipopt')
opt.options['linear_solver'] = 'ma27'
res = opt.solve(m, tee=False)
print(res.solver.termination_condition)
fs.compressor.outlet.display()
print('work: ', round(fs.compressor.work.value, 2), ' MJ')
```

```python

```