# PHY321: Harmonic Oscillations

**[Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/)**, Department of Physics and Astronomy and Facility for Rare Ion Beams (FRIB), Michigan State University, USA and Department of Physics, University of Oslo, Norway

Date: **Mar 3, 2021**

Copyright 1999-2021, [Morten Hjorth-Jensen](http://mhjgit.github.io/info/doc/web/). Released under CC Attribution-NonCommercial 4.0 license

## Aims and Overarching Motivation

### Monday

Summary on forces and conservation laws, with additional examples.
Begin harmonic oscillations.

**Reading suggestion**: Taylor chapters 3 and 4 for the summary on forces and conservation laws. Taylor section 5.1 for start harmonic oscillations.

### Wednesday

Harmonic oscillations, basic equations and formalism
**Reading suggestions**: Taylor sections 5.1-5.2

### Friday

Harmonic oscillations, damped motion.
**Reading suggestion**: Taylor sections 5.3-5.4.

## Harmonic Oscillator

The harmonic oscillator is omnipresent in physics. Although you may think
of this as being related to springs, it, or an equivalent
mathematical representation, appears in just about any problem where a
mode is sitting near its potential energy minimum. At that point,
$\partial_x V(x)=0$, and the first non-zero term (aside from a
constant) in the potential energy is that of a harmonic oscillator. In
a solid, sound modes (phonons) are built on a picture of coupled
harmonic oscillators, and in relativistic field theory the fundamental
interactions are also built on coupled oscillators positioned
infinitesimally close to one another in space. The phenomena of a
resonance of an oscillator driven at a fixed frequency plays out
repeatedly in atomic, nuclear and high-energy physics, when quantum
mechanically the evolution of a state oscillates according to
$e^{-iEt}$ and exciting discrete quantum states has very similar
mathematics as exciting discrete states of an oscillator.

## Harmonic Oscillator, deriving the Equations

The potential energy for a single particle as a function of its
position $x$ can be written as a Taylor expansion about some point
$b$ (we are considering a one-dimensional problem here)

$$
\begin{equation}
V(x)=V(b)+(x-b)\frac{dV(x)}{dx}\vert_{b}+\frac{1}{2!}(x-b)^2\frac{d^2V(x)}{dx^2}\vert_{b}
+\frac{1}{3!}(x-b)^3V(x)^{(3)}\vert_{b}+\cdots
\label{_auto1} \tag{1}
\end{equation}
$$

If the position $b$ is at the minimum of the resonance, the first two non-zero terms of the potential are

$$
\begin{equation}
V(x)\approx V(b)+\frac{1}{2!}(x-b)^2\frac{d^2V(x)}{dx^2}\vert_{b},
\label{_auto2} \tag{2}
\end{equation}
$$

$$
\nonumber
=V(b)+\frac{1}{2}k(x-b)^2,~~~~k\equiv \frac{d^2V(x)}{dx^2}\vert_{b},
$$

$$
\nonumber
F=-\frac{dV(x)}{dx}=-k(x-b).
$$

## Analyzing the equations

Our equation of motion is, with the only force given by the one-dimensional spring force,

$$
m\frac{d^2x}{dt^2}=-kx.
$$

Defining the natural frequency $\omega_0^2=k/m$ we can rewrite this equation as

$$
\frac{d^2x}{dt^2}=-\omega_0^2x.
$$

We call this a natural frequency since it is defined by the constants that describe our system, the spring constant $k$ and the mass $m$ of the object.

We can as usual split this equation of motion into one equation for the derivative of the velocity and

$$
\frac{dv}{dt}=-\omega_0^2x,
$$

and

$$
\frac{dx}{dt}=v.
$$

The solution to the equations of motion is given by

$$
x(t) = A\cos{(\omega_0 t)}+B\sin{(\omega_0 t)},
$$

where $A$ and $B$ are in general complex constants to be determined by the initial conditions.

## Checking the Solution

Inserting the solution into the equation of motion we have

$$
\frac{d^2x}{dt^2}=-\omega_0^2x,
$$

we have

$$
\frac{d^2x}{dt^2} = -A\omega_0^2\cos{(\omega_0 t)}-B\omega_0^2\sin{(\omega_0 t)},
$$

and the right-hand side is just $-\omega_0^2 x(t)$. Thus, inserting the solution into the differential equation shows that we obtain the same original differential equation.

## Initial condition example

Let us assume that our initial time $t_0=0$s and that the initial position $x(t_0)=x_0$ and that $v_0=0$ (we skip units here).
This gives us

$$
x(t=0) = x_0 =A,
$$

and it leaves $B$ undetermined. Taking the derivative of $x$ we obtain the velocity

$$
v(t) = -A\omega_0\sin{(\omega_0 t)}+B\omega_0\cos{(\omega_0 t)},
$$

and with

$$
v(t=0) = 0=B,
$$

we see that our solution with these initial conditions becomes

$$
x(t) = x_0\cos{(\omega_0 t)}.
$$

## Math Digression

From our first homework (exercise 1) we have that (we switch to $\omega$ instead of $\omega_0$)

$$
\cos{(\omega t)} = \sum_{n=0}^{\infty}\left(-1\right)^n \frac{(\omega t)^{2n}}{(2n)!},
$$

and

$$
\sin{(\omega t)} = \sum_{n=0}^{\infty}\left(-1\right)^n \frac{(\omega t)^{2n+1}}{(2n+1)!},
$$

and that we could  write

$$
\exp{(\pm\imath\omega t)} = \cos{(\omega t)}+\pm\imath\sin{(\omega t)}.
$$

This means (show this) that we can write our solution in terms of new constant $C$ and $D$ as

$$
x(t)=C\exp{(\imath\omega t)}+D\exp{(-\imath\omega t)}.
$$

To see the relation between these two forms we note that we can write our original
solution $x(t) = A\cos{(\omega t)}+B\sin{(\omega t)}$ as

$$
x(t) = (C+D)\cos{(\omega t)}+\imath(C-D)\sin{(\omega t)},
$$

meaning that we have $A=C+D$ and $B=\imath(C-D)$.

## More Math Manipulations

We can also rewrite the solution in a simpler way.  We define a new
constant $A=\sqrt{B_1^2+B_2^2}$ which can be thought as the hypotenuse
of a right-angle triangle with sides $B_1$ and $B_2$ and
$B_1=A\cos{(\delta)}$ and $B_2=A\sin{(\delta)}$.

We have then

$$
x(t) = A\left[\frac{B_1}{A}\cos{(\omega t)}+\frac{B_2}{A}\sin{(\omega t)}\right],
$$

which becomes

$$
x(t) = A\left[\cos{(\delta)}\cos{(\omega t)}+\sin{(\delta)}\sin{(\omega t)}\right],
$$

and using the trigonometric relations for addition of angles we have

$$
x(t) = A\cos{(\omega t-\delta)},
$$

where $\delta$ is a so-called phase shift.

## Energy Conservation

Our energy is given by the kinetic energy and the harmonic oscillator potential energy, that is we have (for a one-dimensional harmonic oscillator potential)

$$
E=\frac{1}{2}mv^2+\frac{1}{2}kx^2.
$$

We assume that we have initial conditions $v_0=0$ (no kinetic energy) and $x(t=0)=x_0$.
With these initial conditions we have

$$
x(t) = x_0\cos{(\omega_0 t)},
$$

and the velocity is given by

$$
v(t) = -x_0\omega_0\sin{(\omega_0 t)},
$$

The energy is conserved (as we have discussed before) and at $t=t_0=0$ we have thus

$$
E_0=\frac{1}{2}kx_0^2.
$$

At a time $t\ne 0$ we have

$$
E(t)=\frac{1}{2}mv^2+\frac{1}{2}kx^2=\frac{1}{2}mx_0^2\omega_0^2\sin^2{(\omega_0 t)}+\frac{1}{2}kx_0^2\cos^2{(\omega_0 t)},
$$

Recalling that $\omega_0^2=k/m$ we get

$$
E(t)=\frac{1}{2}kx_0^2\sin^2{(\omega_0 t)}+\frac{1}{2}kx_0^2\cos^2{(\omega_0 t)}=\frac{1}{2}kx_0^2=E_0.
$$

Energy is thus conserved

## The mathematical pendulum

**Note**: Figure to be inserted.

We consider a pendulum of length $l$ attached to the roof as illustrated in the figure (see [handwritten notes from Wednesday Feb 24](https://github.com/mhjensen/Physics321/blob/master/doc/HandWrittenNotes/Spring2021/NotesFeb24.pdf)).

The pendulum consists of a rod and a small object attached to the rod. The mass of this object is $m$ and it is the motion of this object we are concerned with. The distance from the object to the roof is $\boldsymbol{r}$ and we have $\vert \boldsymbol{r}\vert =l$.

The angle between the $y$-axis and the rod is $\phi$. The forces at play are the gravitational force and a tension force from the rod to the object. The net for is

$$
\boldsymbol{F}^{\mathrm{net}}=\boldsymbol{T}+\boldsymbol{G}=T\sin{(\phi)}\boldsymbol{e}_1+T\cos{(\phi)}\boldsymbol{e}_2-mg\boldsymbol{e}_2,
$$

and with

$$
\boldsymbol{r}=l\sin{(\phi)}\boldsymbol{e}_1+l\cos{(\phi)}\boldsymbol{e}_2,
$$

the equation of motion becomes

$$
m\frac{d^2\boldsymbol{r}}{dt^2}=\boldsymbol{T}+\boldsymbol{G}=T\sin{(\phi)}\boldsymbol{e}_1+T\cos{(\phi)}\boldsymbol{e}_2-mg\boldsymbol{e}_2.
$$

## Finding the equations for the $x$- and $y$-directions

Using the chain rule we can find the first derivative of $\boldsymbol{r}$

$$
\frac{d\boldsymbol{r}}{dt}=l\frac{d\phi}{dt}\cos{(\phi)}\boldsymbol{e}_1-l\frac{d\phi}{dt}\sin{(\phi)}\boldsymbol{e}_2,
$$

and thereafter the second derivative in the $x$-direction as

$$
\frac{d^2\boldsymbol{r}}{dt^2}\boldsymbol{e}_1=l\frac{d^2\phi}{dt^2}\cos{(\phi)}-l(\frac{d\phi}{dt})^2\sin{(\phi)},
$$

and in the $y$ direction

$$
\frac{d^2\boldsymbol{r}}{dt^2}\boldsymbol{e}_2=-l\frac{d^2\phi}{dt^2}\sin{(\phi)}-l(\frac{d\phi}{dt})^2\cos{(\phi)}.
$$

## Collecting terms

We can now set up the equations of motion in the $x$ and $y$ directions and get for the $x$-direction

$$
ml\frac{d^2\phi}{dt^2}\cos{(\phi)}-ml(\frac{d\phi}{dt})^2\sin{(\phi)}=T\sin{(\phi)},
$$

and for the $y$-direction

$$
-ml\frac{d^2\phi}{dt^2}\sin{(\phi)}-ml(\frac{d\phi}{dt})^2\cos{(\phi)}=T\cos{(\phi)}-mg.
$$

This looks ugly!

Let us rewrite

$$
ml\frac{d^2\phi}{dt^2}\cos{(\phi)}=\left[ml(\frac{d\phi}{dt})^2+T\right]\sin{(\phi)},
$$

and

$$
-ml\frac{d^2\phi}{dt^2}\sin{(\phi)}+mg=\left[ml(\frac{d\phi}{dt})^2+T\cos{(\phi)}\right].
$$

Still not so nice.

## Simple trick

How can we simplify the above equations, rewritten here

$$
ml\frac{d^2\phi}{dt^2}\cos{(\phi)}=\left[ml(\frac{d\phi}{dt})^2+T\right]\sin{(\phi)},
$$

and

$$
-ml\frac{d^2\phi}{dt^2}\sin{(\phi)}+mg=\left[ml(\frac{d\phi}{dt})^2+T\right]\cos{(\phi)}.
$$

We multiply the first equation with $\cos\phi$ and the second one with $\sin\phi$ and then subtract the two equations.
We get then

$$
-ml\frac{d^2\phi}{dt^2}(\cos{(\phi)})^2-ml\frac{d^2\phi}{dt^2}(\sin{(\phi)})^2+mg\sin{(\phi)}=0,
$$

leading to

$$
ml\frac{d^2\phi}{dt^2}=-mg\sin{(\phi)}.
$$

We are almost there.

## Last step

We divide by $m$ and $l$ and we have the famous non-linear in $\phi$ (due to the sine function) equation for the pendulumn

$$
\frac{d^2\phi}{dt^2}=-\frac{g}{l}\sin{(\phi)}.
$$

Introducing the natural frequency $\omega_0^2=g/l$ we can rewrite the equation as

$$
\frac{d^2\phi}{dt^2}=-\omega_0^2\sin{(\phi)}.
$$

If we now assume that the angle is very small, we can approximate $\sin{(\phi)}\approx \phi$ and we have essentially the same equation as we had for harmonic oscillations, that is

$$
\frac{d^2\phi}{dt^2}=-\omega_0^2\phi.
$$

The solution to this equation is again given by

$$
\phi(t) = A\cos{(\omega_0 t)}+B\sin{(\omega_0 t)}.
$$

For the general case, we have to resort to numerical solutions.

## Damped Oscillators

We consider only the case where the damping force is proportional to
the velocity. This is counter to dragging friction, where the force is
proportional in strength to the normal force and independent of
velocity, and is also inconsistent with wind resistance, where the
magnitude of the drag force is proportional the square of the
velocity. Rolling resistance does seem to be mainly proportional to
the velocity. However, the main motivation for considering damping
forces proportional to the velocity is that the math is more
friendly. This is because the differential equation is linear,
i.e. each term is of order $x$, $\dot{x}$, $\ddot{x}\cdots$, or even
terms with no mention of $x$, and there are no terms such as $x^2$ or
$x\ddot{x}$. The equations of motion for a spring with damping force
$-b\dot{x}$ are

$$
\begin{equation}
m\ddot{x}+b\dot{x}+kx=0.
\label{_auto3} \tag{3}
\end{equation}
$$

## Harmonic Oscillator,  Damping

Just to make the solution a bit less messy, we rewrite this equation as

$$
\begin{equation}
\label{eq:dampeddiffyq} \tag{4}
\ddot{x}+2\beta\dot{x}+\omega_0^2x=0,~~~~\beta\equiv b/2m,~\omega_0\equiv\sqrt{k/m}.
\end{equation}
$$

Both $\beta$ and $\omega$ have dimensions of inverse time. To find solutions (see appendix C in the text) you must make an educated guess at the form of the solution. To do this, first realize that the solution will need an arbitrary normalization $A$ because the equation is linear. Secondly, realize that if the form is

$$
\begin{equation}
x=Ae^{rt}
\label{_auto4} \tag{5}
\end{equation}
$$

that each derivative simply brings out an extra power of $r$. This
means that the $Ae^{rt}$ factors out and one can simply solve for an
equation for $r$. Plugging this form into Eq. ([4](#eq:dampeddiffyq)),

$$
\begin{equation}
r^2+2\beta r+\omega_0^2=0.
\label{_auto5} \tag{6}
\end{equation}
$$

## Harmonic Oscillator,  Solutions of Damped Motion

Because this is a quadratic equation there will be two solutions,

$$
\begin{equation}
r=-\beta\pm\sqrt{\beta^2-\omega_0^2}.
\label{_auto6} \tag{7}
\end{equation}
$$

We refer to the two solutions as $r_1$ and $r_2$ corresponding to the
$+$ and $-$ roots. As expected, there should be two arbitrary
constants involved in the solution,

$$
\begin{equation}
x=A_1e^{r_1t}+A_2e^{r_2t},
\label{_auto7} \tag{8}
\end{equation}
$$

where the coefficients $A_1$ and $A_2$ are determined by initial
conditions.

The roots listed above, $\sqrt{\omega_0^2-\beta_0^2}$, will be
imaginary if the damping is small and $\beta<\omega_0$. In that case,
$r$ is complex and the factor $e{rt}$ will have some oscillatory
behavior. If the roots are real, there will only be exponentially
decaying solutions. There are three cases:

## Underdamped: $\beta<\omega_0$

$$
\begin{eqnarray}
x&=&A_1e^{-\beta t}e^{i\omega't}+A_2e^{-\beta t}e^{-i\omega't},~~\omega'\equiv\sqrt{\omega_0^2-\beta^2}\\
\nonumber
&=&(A_1+A_2)e^{-\beta t}\cos\omega't+i(A_1-A_2)e^{-\beta t}\sin\omega't.
\end{eqnarray}
$$

Here we have made use of the identity
$e^{i\omega't}=\cos\omega't+i\sin\omega't$. Because the constants are
arbitrary, and because the real and imaginary parts are both solutions
individually, we can simply consider the real part of the solution
alone:

$$
\begin{eqnarray}
\label{eq:homogsolution} \tag{9}
x&=&B_1e^{-\beta t}\cos\omega't+B_2e^{-\beta t}\sin\omega't,\\
\nonumber
\omega'&\equiv&\sqrt{\omega_0^2-\beta^2}.
\end{eqnarray}
$$

## Critical dampling: $\beta=\omega_0$

In this case the two terms involving $r_1$ and $r_2$ are identical
because $\omega'=0$. Because we need to arbitrary constants, there
needs to be another solution. This is found by simply guessing, or by
taking the limit of $\omega'\rightarrow 0$ from the underdamped
solution. The solution is then

$$
\begin{equation}
\label{eq:criticallydamped} \tag{10}
x=Ae^{-\beta t}+Bte^{-\beta t}.
\end{equation}
$$

The critically damped solution is interesting because the solution
approaches zero quickly, but does not oscillate. For a problem with
zero initial velocity, the solution never crosses zero. This is a good
choice for designing shock absorbers or swinging doors.

## Overdamped: $\beta>\omega_0$

$$
\begin{eqnarray}
x&=&A_1\exp{-(\beta+\sqrt{\beta^2-\omega_0^2})t}+A_2\exp{-(\beta-\sqrt{\beta^2-\omega_0^2})t}
\end{eqnarray}
$$

This solution will also never pass the origin more than once, and then
only if the initial velocity is strong and initially toward zero.

Given $b$, $m$ and $\omega_0$, find $x(t)$ for a particle whose
initial position is $x=0$ and has initial velocity $v_0$ (assuming an
underdamped solution).

The solution is of the form,

$$
\begin{eqnarray*}
x&=&e^{-\beta t}\left[A_1\cos(\omega' t)+A_2\sin\omega't\right],\\
\dot{x}&=&-\beta x+\omega'e^{-\beta t}\left[-A_1\sin\omega't+A_2\cos\omega't\right].\\
\omega'&\equiv&\sqrt{\omega_0^2-\beta^2},~~~\beta\equiv b/2m.
\end{eqnarray*}
$$

From the initial conditions, $A_1=0$ because $x(0)=0$ and $\omega'A_2=v_0$. So

$$
x=\frac{v_0}{\omega'}e^{-\beta t}\sin\omega't.
$$

## Sinusoidally Driven Oscillators

Here, we consider the force

$$
\begin{equation}
F=-kx-b\dot{x}+F_0\cos\omega t,
\label{_auto8} \tag{11}
\end{equation}
$$

which leads to the differential equation

$$
\begin{equation}
\label{eq:drivenosc} \tag{12}
\ddot{x}+2\beta\dot{x}+\omega_0^2x=(F_0/m)\cos\omega t.
\end{equation}
$$

## Harmonic Oscillator,  Solutions

Consider a single solution with no arbitrary constants, which we will
call a {\it particular solution}, $x_p(t)$. It should be emphasized
that this is {\bf A} particular solution, because there exists an
infinite number of such solutions because the general solution should
have two arbitrary constants. Now consider solutions to the same
equation without the driving term, which include two arbitrary
constants. These are called either {\it homogenous solutions} or {\it
complementary solutions}, and were given in the previous section,
e.g. Eq. ([9](#eq:homogsolution)) for the underdamped case. The
homogenous solution already incorporates the two arbitrary constants,
so any sum of a homogenous solution and a particular solution will
represent the {\it general solution} of the equation. The general
solution incorporates the two arbitrary constants $A$ and $B$ to
accommodate the two initial conditions. One could have picked a
different particular solution, i.e. the original particular solution
plus any homogenous solution with the arbitrary constants $A_p$ and
$B_p$ chosen at will. When one adds in the homogenous solution, which
has adjustable constants with arbitrary constants $A'$ and $B'$, to
the new particular solution, one can get the same general solution by
simply adjusting the new constants such that $A'+A_p=A$ and
$B'+B_p=B$. Thus, the choice of $A_p$ and $B_p$ are irrelevant, and
when choosing the particular solution it is best to make the simplest
choice possible.

## Harmonic Oscillator,  Particular Solution

To find a particular solution, one first guesses at the form,

$$
\begin{equation}
\label{eq:partform} \tag{13}
x_p(t)=D\cos(\omega t-\delta),
\end{equation}
$$

and rewrite the differential equation as

$$
\begin{equation}
D\left\{-\omega^2\cos(\omega t-\delta)-2\beta\omega\sin(\omega t-\delta)+\omega_0^2\cos(\omega t-\delta)\right\}=\frac{F_0}{m}\cos(\omega t).
\label{_auto9} \tag{14}
\end{equation}
$$

One can now use angle addition formulas to get

$$
\begin{eqnarray}
D\left\{(-\omega^2\cos\delta+2\beta\omega\sin\delta+\omega_0^2\cos\delta)\cos(\omega t)\right.&&\\
\nonumber
\left.+(-\omega^2\sin\delta-2\beta\omega\cos\delta+\omega_0^2\sin\delta)\sin(\omega t)\right\}
&=&\frac{F_0}{m}\cos(\omega t).
\end{eqnarray}
$$

Both the $\cos$ and $\sin$ terms need to equate if the expression is to hold at all times. Thus, this becomes two equations

$$
\begin{eqnarray}
D\left\{-\omega^2\cos\delta+2\beta\omega\sin\delta+\omega_0^2\cos\delta\right\}&=&\frac{F_0}{m}\\
\nonumber
-\omega^2\sin\delta-2\beta\omega\cos\delta+\omega_0^2\sin\delta&=&0.
\end{eqnarray}
$$

After dividing by $\cos\delta$, the lower expression leads to

$$
\begin{equation}
\tan\delta=\frac{2\beta\omega}{\omega_0^2-\omega^2}.
\label{_auto10} \tag{15}
\end{equation}
$$

## Solving with Driven Oscillations

Using the identities $\tan^2+1=\csc^2$ and $\sin^2+\cos^2=1$, one can also express $\sin\delta$ and $\cos\delta$,

$$
\begin{eqnarray}
\sin\delta&=&\frac{2\beta\omega}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}},\\
\nonumber
\cos\delta&=&\frac{(\omega_0^2-\omega^2)}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}}
\end{eqnarray}
$$

Inserting the expressions for $\cos\delta$ and $\sin\delta$ into the expression for $D$,

$$
\begin{equation}
\label{eq:Ddrive} \tag{16}
D=\frac{F_0/m}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}}.
\end{equation}
$$

For a given initial condition, e.g. initial displacement and velocity,
one must add the homogenous solution then solve for the two arbitrary
constants. However, because the homogenous solutions decay with time
as $e^{-\beta t}$, the particular solution is all that remains at
large times, and is therefore the steady state solution. Because the
arbitrary constants are all in the homogenous solution, all memory of
the initial conditions are lost at large times, $t>>1/\beta$.

The amplitude of the motion, $D$, is linearly proportional to the
driving force ($F_0/m$), but also depends on the driving frequency
$\omega$. For small $\beta$ the maximum will occur at
$\omega=\omega_0$. This is referred to as a resonance. In the limit
$\beta\rightarrow 0$ the amplitude at resonance approaches infinity.

## Alternative Derivation for Driven Oscillators

Here, we derive the same expressions as in Equations ([13](#eq:partform)) and ([16](#eq:Ddrive)) but express the driving forces as

$$
\begin{eqnarray}
F(t)&=&F_0e^{i\omega t},
\end{eqnarray}
$$

rather than as $F_0\cos\omega t$. The real part of $F$ is the same as before. For the differential equation,

$$
\begin{eqnarray}
\label{eq:compdrive} \tag{17}
\ddot{x}+2\beta\dot{x}+\omega_0^2x&=&\frac{F_0}{m}e^{i\omega t},
\end{eqnarray}
$$

one can treat $x(t)$ as an imaginary function. Because the operations
$d^2/dt^2$ and $d/dt$ are real and thus do not mix the real and
imaginary parts of $x(t)$, Eq. ([17](#eq:compdrive)) is effectively 2
equations. Because $e^{\omega t}=\cos\omega t+i\sin\omega t$, the real
part of the solution for $x(t)$ gives the solution for a driving force
$F_0\cos\omega t$, and the imaginary part of $x$ corresponds to the
case where the driving force is $F_0\sin\omega t$. It is rather easy
to solve for the complex $x$ in this case, and by taking the real part
of the solution, one finds the answer for the $\cos\omega t$ driving
force.

We assume a simple form for the particular solution

$$
\begin{equation}
x_p=De^{i\omega t},
\label{_auto11} \tag{18}
\end{equation}
$$

where $D$ is a complex constant.

From Eq. ([17](#eq:compdrive)) one inserts the form for $x_p$ above to get

$$
\begin{eqnarray}
D\left\{-\omega^2+2i\beta\omega+\omega_0^2\right\}e^{i\omega t}=(F_0/m)e^{i\omega t},\\
\nonumber
D=\frac{F_0/m}{(\omega_0^2-\omega^2)+2i\beta\omega}.
\end{eqnarray}
$$

The norm and phase for $D=|D|e^{-i\delta}$ can be read by inspection,

$$
\begin{equation}
|D|=\frac{F_0/m}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}},~~~~\tan\delta=\frac{2\beta\omega}{\omega_0^2-\omega^2}.
\label{_auto12} \tag{19}
\end{equation}
$$

This is the same expression for $\delta$ as before. One then finds $x_p(t)$,

$$
\begin{eqnarray}
\label{eq:fastdriven1} \tag{20}
x_p(t)&=&\Re\frac{(F_0/m)e^{i\omega t-i\delta}}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}\\
\nonumber
&=&\frac{(F_0/m)\cos(\omega t-\delta)}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}.
\end{eqnarray}
$$

This is the same answer as before.
If one wished to solve for the case where $F(t)= F_0\sin\omega t$, the imaginary part of the solution would work

$$
\begin{eqnarray}
\label{eq:fastdriven2} \tag{21}
x_p(t)&=&\Im\frac{(F_0/m)e^{i\omega t-i\delta}}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}\\
\nonumber
&=&\frac{(F_0/m)\sin(\omega t-\delta)}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}.
\end{eqnarray}
$$

## Damped and Driven Oscillator

Consider the damped and driven harmonic oscillator worked out above. Given $F_0, m,\beta$ and $\omega_0$, solve for the complete solution $x(t)$ for the case where $F=F_0\sin\omega t$ with initial conditions $x(t=0)=0$ and $v(t=0)=0$. Assume the underdamped case.

The general solution including the arbitrary constants includes both the homogenous and particular solutions,

$$
\begin{eqnarray*}
x(t)&=&\frac{F_0}{m}\frac{\sin(\omega t-\delta)}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}
+A\cos\omega't e^{-\beta t}+B\sin\omega't e^{-\beta t}.
\end{eqnarray*}
$$

The quantities $\delta$ and $\omega'$ are given earlier in the
section, $\omega'=\sqrt{\omega_0^2-\beta^2},
\delta=\tan^{-1}(2\beta\omega/(\omega_0^2-\omega^2)$. Here, solving
the problem means finding the arbitrary constants $A$ and
$B$. Satisfying the initial conditions for the initial position and
velocity:

$$
\begin{eqnarray*}
x(t=0)=0&=&-\eta\sin\delta+A,\\
v(t=0)=0&=&\omega\eta\cos\delta-\beta A+\omega'B,\\
\eta&\equiv&\frac{F_0}{m}\frac{1}{\sqrt{(\omega_0^2-\omega^2)^2+4\beta^2\omega^2}}.
\end{eqnarray*}
$$

The problem is now reduced to 2 equations and 2 unknowns, $A$ and $B$. The solution is

$$
\begin{eqnarray}
A&=& \eta\sin\delta ,~~~B=\frac{-\omega\eta\cos\delta+\beta\eta\sin\delta}{\omega'}.
\end{eqnarray}
$$

## Resonance Widths; the $Q$ factor

From the previous two sections, the particular solution for a driving force, $F=F_0\cos\omega t$, is

$$
\begin{eqnarray}
x_p(t)&=&\frac{F_0/m}{\sqrt{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}}\cos(\omega_t-\delta),\\
\nonumber
\delta&=&\tan^{-1}\left(\frac{2\beta\omega}{\omega_0^2-\omega^2}\right).
\end{eqnarray}
$$

If one fixes the driving frequency $\omega$ and adjusts the
fundamental frequency $\omega_0=\sqrt{k/m}$, the maximum amplitude
occurs when $\omega_0=\omega$ because that is when the term from the
denominator $(\omega_0^2-\omega^2)^2+4\omega^2\beta^2$ is at a
minimum. This is akin to dialing into a radio station. However, if one
fixes $\omega_0$ and adjusts the driving frequency one minimize with
respect to $\omega$, e.g. set

$$
\begin{equation}
\frac{d}{d\omega}\left[(\omega_0^2-\omega^2)^2+4\omega^2\beta^2\right]=0,
\label{_auto13} \tag{22}
\end{equation}
$$

and one finds that the maximum amplitude occurs when
$\omega=\sqrt{\omega_0^2-2\beta^2}$. If $\beta$ is small relative to
$\omega_0$, one can simply state that the maximum amplitude is

$$
\begin{equation}
x_{\rm max}\approx\frac{F_0}{2m\beta \omega_0}.
\label{_auto14} \tag{23}
\end{equation}
$$

$$
\begin{eqnarray}
\frac{4\omega^2\beta^2}{(\omega_0^2-\omega^2)^2+4\omega^2\beta^2}=\frac{1}{2}.
\end{eqnarray}
$$

For small damping this occurs when $\omega=\omega_0\pm \beta$, so the $FWHM\approx 2\beta$. For the purposes of tuning to a specific frequency, one wants the width to be as small as possible. The ratio of $\omega_0$ to $FWHM$ is known as the {\it quality} factor, or $Q$ factor,

$$
\begin{equation}
Q\equiv \frac{\omega_0}{2\beta}.
\label{_auto15} \tag{24}
\end{equation}
$$

## Our Sliding Block Code
Here we study first the case without additional friction term and scale our equation
in terms of a dimensionless time $\tau$.

Let us remind ourselves about the differential equation we want to solve (the general case with damping due to friction)

$$
m\frac{d^2x}{dt^2} + b\frac{dx}{dt}+kx(t) =0.
$$

We divide by $m$ and introduce $\omega_0^2=\sqrt{k/m}$ and obtain

$$
\frac{d^2x}{dt^2} + \frac{b}{m}\frac{dx}{dt}+\omega_0^2x(t) =0.
$$

## Harmonic Oscillator, Sliding Block

Thereafter we introduce a dimensionless time $\tau = t\omega_0$ (check
that the dimensionality is correct) and rewrite our equation as

$$
\frac{d^2x}{d\tau^2} + \frac{b}{m\omega_0}\frac{dx}{d\tau}+x(\tau) =0,
$$

which gives us

$$
\frac{d^2x}{d\tau^2} + \frac{b}{m\omega_0}\frac{dx}{d\tau}+x(\tau) =0.
$$

## Harmonic Oscillator, Sliding Block, Numerical Aspects

We then define $\gamma = b/(2m\omega_0)$ and rewrite our equations as

$$
\frac{d^2x}{d\tau^2} + 2\gamma\frac{dx}{d\tau}+x(\tau) =0.
$$

This is the equation we will code below. The first version employs the Euler-Cromer method.

```
%matplotlib inline

# Common imports
import numpy as np
import pandas as pd
from math import *
import matplotlib.pyplot as plt
import os

# Where to save the figures and data files
PROJECT_ROOT_DIR = "Results"
FIGURE_ID = "Results/FigureFiles"
DATA_ID = "DataFiles/"

if not os.path.exists(PROJECT_ROOT_DIR):
    os.mkdir(PROJECT_ROOT_DIR)

if not os.path.exists(FIGURE_ID):
    os.makedirs(FIGURE_ID)

if not os.path.exists(DATA_ID):
    os.makedirs(DATA_ID)

def image_path(fig_id):
    return os.path.join(FIGURE_ID, fig_id)

def data_path(dat_id):
    return os.path.join(DATA_ID, dat_id)

def save_fig(fig_id):
    plt.savefig(image_path(fig_id) + ".png", format='png')

from pylab import plt, mpl
plt.style.use('seaborn')
mpl.rcParams['font.family'] = 'serif'

DeltaT = 0.001
#set up arrays
tfinal = 20 # in dimensionless time
n = ceil(tfinal/DeltaT)
# set up arrays for t, v, and x
t = np.zeros(n)
v = np.zeros(n)
x = np.zeros(n)
# Initial conditions as simple one-dimensional arrays of time
x0 =  1.0
v0 = 0.0
x[0] = x0
v[0] = v0
gamma = 0.0
# Start integrating using Euler-Cromer's method
for i in range(n-1):
    # Set up the acceleration
    # Here you could have defined your own function for this
    a =  -2*gamma*v[i]-x[i]
    # update velocity, time and position
    v[i+1] = v[i] + DeltaT*a
    x[i+1] = x[i] + DeltaT*v[i+1]
    t[i+1] = t[i] + DeltaT
# Plot position as function of time
fig, ax = plt.subplots()
#ax.set_xlim(0, tfinal)
ax.set_ylabel('x[m]')
ax.set_xlabel('t[s]')
ax.plot(t, x)
fig.tight_layout()
save_fig("BlockEulerCromer")
plt.show()
```

When setting up the value of $\gamma$ we see that for $\gamma=0$ we get the simple oscillatory motion with no damping.
Choosing $\gamma < 1$ leads to the classical underdamped case with oscillatory motion, but where the motion comes to an end.

Choosing $\gamma =1$ leads to what normally is called critical damping and $\gamma> 1$ leads to critical overdamping.
Try it out and try also to change the initial position and velocity. Setting $\gamma=1$
yields a situation, as discussed above, where the solution approaches quickly zero and does not oscillate. With zero initial velocity it will never cross zero.


```
figsize(12.5,4)
np.set_printoptions(precision=3, suppress= True)
import scipy.stats as stats
```

Chapter 2
======
______

This chapter introduces more PyMC syntax and design patterns, and ways to think about how to model a system like a Bayesian.

## A little more on PyMC

### Parent and Child relationships

To assist with terminology, and to be consistent with PyMC's documentation, we introduce *parent and child* variables.

*  *parent variables* are variables that influence another variable.

*  *child variable* are variables that are affected by other variables, i.e. are the subject of parent variables.

Variables can be both parents and children. For example, consider the PyMC code below.

```
import pymc as mc

poi_parameter = mc.Exponential( "Poisson_param", 1 )

data_generator = mc.Poisson("data_generator", poi_parameter )

output = data_generator + 1
```

`poi_parameter` controls the parameter of `data_generator`, hence influences its values. The former is a parent of the latter. By symmetry, `data_generator` is a child of `poi_parameter`.

Likewise, `data_generator` is a parent to the variable `output` (hence making `data_generator` both a parent and child variable). Although it does not look like one, `output` should be treated as a PyMC variable as it is a function of another PyMC variable.

This nomenclature is introduced to help us describe relationships in PyMC modeling. You can access a variables children and parent variables using the `children` and `parents` attributes attached to variables.

```
print poi_parameter.children
print
print data_generator.parents
print
print data_generator.children
```

    set([<pymc.distributions.Poisson 'data_generator' at 0x000000000827BA20>])

    {'mu': <pymc.distributions.Exponential 'Poisson_param' at 0x000000000827BFD0>}

    set([<pymc.PyMCObjects.Deterministic '(data_generator_add_1)' at 0x000000000827BEB8>])

Of course a child can  have more than one parent, and a parent can have many children.

### PyMC Variables

All PyMC variables also expose a `value` attribute. This method produces the *current* (possibly random) value of the variable, given the variable's parents. To use the same variables from before:

```
print "poi_parameter.value =",poi_parameter.value
print "data_generator.value =",data_generator.value
print "output.value =", output.value
```

    poi_parameter.value = 0.500067918512
    data_generator.value = 1
    output.value = 2

PyMC is concerned with two types of programming variables: `stochastic` and `deterministic`.

*  *stochastic variables* are variables that are not deterministic, i.e., even if you knew all the values of the variables' parents (if it even has any parents), it would still be random. Included in this catagory are instances of classes `Poisson`, `DiscreteUniform`, and `Exponential`.

*  *deterministic variables* are variables that are not random if the variables' parents are not random, ie. the parent's values are known. This might be confusing at first: a quick mental check is *if I knew all of variable `foo`'s parent variables, I could determine completely what `foo` is.*

#### Initializing Stochastic variables

Initializing a stochastic variable requires a `name` argument, plus additional parameters that are class specific. For example:

`some_variable = mc.DiscreteUniform( "discrete_uni_var", 0, 4 )`

where 0,4 are the `DiscreteUniform`-specific upper and lower bound on the random variable. The [PyMC docs](http://pymc-devs.github.com/pymc/distributions.html) contain the specific parameters for stochastic variables. (Or use `??` if you are using IPython!)

The name attribute is used to retrieve the posterior distribution later in the analysis, so best to use a clear name. Typically, I use the Python variable's name as the name.

Rather than creating a Python array of stochastic variables, addressing the `size` keyword in the call to a `Stochastic` variable creates multivariate array of (independent) stochastic variables. The array behaves like a Numpy array when used like one, and references to its `value` attribute return Numpy arrays.

The `size` argument also solves the annoying case where you may have many variables $\beta_i, \; i = 1,...,N$ you wish to model. Instead of creating arbitrary names and variables for each one, like:

    beta_1 = mc.Uniform( "beta_1", 0, 1)
    beta_2 = mc.Uniform( "beta_2", 0, 1)
    ...

we can instead wrap them into a single variable:

    betas = mc.Uniform( "betas", 0, 1, size = N )

#### Calling `random()`
We can also call on a stochastic variable's `random()` method, which (given the parent values) will generate a new, random value. Below we demonstrate this using the texting example from the previous chapter.

```
lambda_1 = mc.Exponential( "lambda_1",  1 ) #prior on first behaviour
lambda_2 = mc.Exponential( "lambda_2", 1 ) #prior on second behaviour
tau = mc.DiscreteUniform( "tau", lower = 0, upper = 10 ) #prior on behaviour change

print "lambda_1.value = %.3f"%lambda_1.value
print "lambda_2.value = %.3f"%lambda_2.value
print "tau.value = %.3f"%tau.value
print

lambda_1.random(), lambda_2.random(), tau.random()

print "After calling random() on the variables..."
print "lambda_1.value = %.3f"%lambda_1.value
print "lambda_2.value = %.3f"%lambda_2.value
print "tau.value = %.3f"%tau.value

```

    lambda_1.value = 0.795
    lambda_2.value = 0.507
    tau.value = 0.000

    After calling random() on the variables...
    lambda_1.value = 1.212
    lambda_2.value = 4.961
    tau.value = 10.000

The call to `random` stores a new value into the variables `value` attribute. In fact, this new value is stored in the computer's cache for faster recall and efficiency.

### Warning: *Don't update stochastic variables' values in-place.*

Straight from the PyMC docs, we quote [4]:

> 'Stochastic' objects' values should be values should not be updated in-place. This confuses PyMC’s caching scheme... The only way a stochastic variable’s value should be updated is using statements of the following form:

        A.value = new_value

> The following are in-place updates and should **never** be used:

        A.value += 3
        A.value[2,1] = 5
        A.value.attribute = new_attribute_value

#### Determinstic variables

Since most variables you will be modeling are stochastic, we distinguish deterministic variables with a `pymc.deterministic` wrapper. If you are unfamiliar with Python wrappers (also called decorators), that's no problem. Just preppend the `pymc.deterministic` decorator and you're good to go. No need to know know more. Preprending with the wrapper is the easist way, but not the only way, to create deterministic variables. This is not completely true: elementary operations, like addition, exponentials etc. implicity create determinsitic variables.

```
type( lambda_1 + lambda_2 )
```

    pymc.PyMCObjects.Deterministic

The use of the `deterministic` wrapper was seen in the previous text-message example.  Recall the model for $\lambda$ looked like:

$$
\lambda =
\cases{
\lambda_1  & \text{if } t \lt \tau \cr
\lambda_2 & \text{if } t \ge \tau
}
$$

And in PyMC code:

```
@mc.deterministic
def lambda_( tau = tau, lambda_1 = lambda_1, lambda_2 = lambda_2 ):
    out = np.zeros( 5 )
    out[:tau] = lambda_1 #lambda before tau is lambda1
    out[tau:] = lambda_2 #lambda after tau is lambda1
    return out
```

Clearly, if $\tau, \lambda_1$ and $\lambda_2$ are known, then $\lambda$ is known completely, hence it is a deterministic variable.

Notice in the creation of the deterministic function we added defaults to each variable used in the function. This is a necessary step, and all variables *must* have default values.

### Including observations in the Model

At this point, it may not look like it, but we have fully specified our priors. For example, we can ask and answer questions like "What does my prior distribution of $\lambda_1$ look like?"

```
figsize( 12.5, 3)
samples = [ lambda_1.random() for i in range( 20000) ]
hist( samples, bins = 70, normed=True )
plt.title( "Prior distribution for $\lambda_1$")
plt.xlim( 0, 8)
```

To frame this in the notation of the first chapter, though this is a slight abuse of notation, we have specified $P(A)$. Our next goal is to include data/evidence/observations $X$ into our model.

PyMC stochastic variables have a keyword argument `observed` which accepts a boolean (`False` by default). The keyword `observed` has a very simple role: fix the variables current value, i.e. make `value` immutable. For this to make sense, we have to specify an intial `value` in the class call. For example:

```
fixed_variable = mc.Poisson( "fxd", 1, value = 10, observed = True )
print "value: ",fixed_variable.value
print "calling .random()"
fixed_variable.random()
print "value: ",fixed_variable.value
```

    value:  10
    calling .random()
    value:  10

This is how we include data into our models: initializing the stochastic variable to have a *fixed value*. wait--

> ...initializing the stochastic variable to have a *fixed value*.

That might seem strange to hear at first. Shouldn't a stochastic variable be random? In fact, Bayesian analysis sees observed data as simply fixed variables. Taking this to its logical conclusion, any predictions made by Bayesian analysis is seen as fitting another variable, or parameter, in the model.

To complete our text message example, we fix the PyMC variable `observations` to the observed dataset:

```
#fake data
data = np.array( [ 10, 25, 15, 20, 35] )
obs = mc.Poisson( "obs", lambda_, value = data, observed = True )
```

### Finally...

We wrap all the created variables into a `mc.Model` class. With this `Model` class, we can analysis the variables as single unit.

```
model = mc.Model( [obs, lambda_, lambda_1, lambda_2, tau] )
```

We leave implementing a model here, and will return later with how the `Model` class is used.

## Modeling approaches

A good starting place to modeling is to think about *how your data might have been generated*. Position yourself from a god-like position, and try to imagine about how *you* would recreate the dataset. For example, in the last chapter, we investigated text message data:

1.  We started by thinking "what is the best random variable to describe this data?" A Poisson random variable is a good candidate because it assigns probabilities to count data.

2.  Next, we think, "Ok, assuming texts are Poisson-distributed, what do I need for the Poisson distribution?" Well, the Poisson distribution has a parameters $\lambda$.

3.  Do we know $\lambda$? No. In fact, we have a suspicion that there are *two* $\lambda$ values, one for the earlier behaviour and one for the latter behaviour. We don't know when the behaviour switches though.

4. What is a good distribution for the two $\lambda$s? The exponential is good, as it assigns probabilites to positive real numbers. Well the exponential distribution has a parmeter too, call it $\alpha$.

5.  Do we know what the parameter $\alpha$ might be? No. We could continue and assign a distribution to $\alpha$, but it's better to stop once we reach a set level of ignorrance: whereas we have a prior belief about $\lambda$, ("it probably changes over time", "it's likely between 10 and 30", etc.), we don't really have any strong beliefs about $\alpha$. So it's best to stop here.

6. What is a good value for $\alpha$ then? We think that the $\lambda$s are between 10-30, so if we set $\alpha$ really low (which corresponds to larger probabilty on high values) we are not reflecting our prior well. Similar, a too-high alpha misses our prior belief as well. A good idea for $\alpha$ as to reflect our belief is to set the value so that the mean of $\lambda$, given $\alpha$, is equal to our observed mean. This was shown in the last chapter.

In this way, we are thinking about how the data might have been created.

## Setting up a model

Likely the most common statistical task is estimating the frequency of events. Knowing the frequency (probability) of events like baseball home runs, frequency of social attributes, computer failures etc. are common requests we ask of Nature. Unfortunately because of our finite information about the world we must *infer* the true frequencies, and can never observe it directly.

Social data is really interesting as *people are not always honest*, which adds a further complication into inference. For example, simpley asking individuals "Have you ever cheated on a test?" will surely contain dishonesty. All you can say is the the true rate is less than you observed (for who would lie and admit "Yes" when in fact they hadn't).

To present an elegant solution to the dishonesty problem, and to show-off some Bayesian modeling, we first need to introduce the binomial distribution.

### The Binomial Distribution

The binomial distribution is one of the most popular distributions, mostly because of its simplicity and usefulness. Unlike the other distributions we have encountered thus far in the book, the binomial distribution has 2 parameters: $N$, a positive integer representing $N$ trials or number of instances of potential events, and $p$, the probability of an event occuring in a trial. Like the Poisson distribution, it is a discrete distribution, but unlike the Poisson distribution, it only weighs integers from $0$ to $N$. The mass distribution looks like:

$$P( X = k ) =  {{N}\choose{k}}  p^k(1-p)^{N-k}$$

If $X$ is a binomial random variable with parameter $p$ and $N$, denoted $X \sim \text{Bin}(N,p)$, then $X$ is the number of events that occured in the $N$ trials. The larger $p$ is (while still remaining between 0 and 1), the more events are likely to occur. The expected value of a binomial is equal to $Np$. Below we plot the mass probability distribution for varying parameters.

```
figsize( 12.5, 4)

binomial = stats.binom

parameters = [ (10, .4) , (10, .9) ]
colors = ["#348ABD", "#A60628", "#7A68A6"]

for i in range(2):
    N, p = parameters[i]
    _x = np.arange( N+1 )
    plt.bar( _x - 0.5,  binomial.pmf( _x, N, p ), color = colors[i],
             alpha = 0.7, label = "$N$: %d, $p$: %.1f"%(N,p))

plt.legend(loc="upper left")
plt.xlim(0, 10.5)
plt.xlabel("$k$")
plt.ylabel("$P(X = k)$")
plt.title("Probability mass distributions of binomial random variables")
```

##### Example: Cheating amoung students

------
We will use the binomial distribution to determine the frequency of students cheating during an exam. If we let $N$ be the total number of students who took the exam, and assuming each student is interviewed either in person or anonymously (answering without consequence), we will recieve $X$ "Yes I did cheat" answers. We then find the posterior distribution of $p$, given $N$, a prior on $p$, and observed data $X$.

This is a completely absurd model. No student, even with a free pass against punishment, would admit to cheating. What we need is a better *algorithm* to ask students if they had cheated. Ideally the algorithm should encourage individuals to be honest while preseving privacy. The following proposed algorithm is a solution I greatly admire for its enginuity and effectivness:

> In the interview process for each student, the student flips a coin, hidden from the interviewer. The student agrees to answer honestly if the coin comes up heads. Otherwise, if the coin is tails, the student flips the coin (secretly) again, and answer "Yes I did cheat" if the coin flip lands heads, and "No I did not cheat", if the coin flip lands tails. This way, the interviewer does not know where a "Yes" was the result of a guilty plea or a Heads on a second toss. Thus privacy is preserved and the researchers receive honest answers.

One could of course argue with the last statement above and say the interviewers are receiving false data, but more accuratly they are losing approximately half of there original dataset. They have gained though a systematic data generation process that can be modeled. Furthermore, they do not have to incorporate (perhaps somewhat naively) the possibility of deceitful answers. We can use PyMC to dig through this model, and easily find a posterior distribution for the true frequency of liars.

To begin modelling this in PyMC, we first need a prior on $p$, the frequency of cheaters in the exam.  We will be naive and suppose a uniform prior on [0,1].

```
import pymc as mc

p = mc.Uniform( "freq_cheating", 0, 1)
```

In our data-generation model, we imagine some higher-power picks $p$ randomly from [0,1].

Next we need a dataset. Suppose 100 students took the test, and after performing our coin-flip skewed interviews we received 30 "Yes" responses. To put this into a relative perspective, if there truely were no cheaters, we should expect to see on average 1/4 of all responses being a "Yes" (half chance of having first coin land Tails, and another 1/2 chance of having second coin land Heads), so about 25 responses give or take some randomness. On the other hand, if *all students cheated*, we should expected to see on approximately 3/4 of all response be "Yes". In fact, given a value for $p$ (which from our god-like position we know), we can find the probability the student will answer yes:

\begin{align}
P(\text{"Yes"}) = & P( \text{Heads on first coin} )P( \text{cheater} ) + P( \text{Tails on first coin} )P( \text{Heads on second coin} ) \\\\
& = \frac{1}{2}p + \frac{1}{2}\frac{1}{2}\\\\
& = \frac{p}{2} + \frac{1}{4}
\end{align}

Thus, knowing $p$ we know the probability a student will respond "Yes". In PyMC, we can create a deterministic function (see above if unclear on why a determinsitic function can return a probability) to evaluate this new probability of responding "Yes":

```
@mc.deterministic
def p_skewed( p =  p ):
    return 0.5*p + 0.25
```

Of course, I could have typed `p_skewed  = 0.5*p + 0.25` instead for a one-liner, as the elementary operations of plus and scalar multiplication will implicity create a `deterministic` variable, but I wanted to make the deterministic notation explicit for clarity's sake.

So, if we know the probability of respondents saying "Yes", and we have $N=25$ students, the number of "Yes" responses is a binomial with paramters `N` and `p_skewed`.

This is were we include our data. We actually observed 30 "Yes" responses. So in the declaration of the `mc.Binomial`, we include `value = 30` and `observed = True`.

```
yes_responses = mc.Binomial( "number_cheaters", 100, p_skewed,
                                value = 30, observed = True )
```

Below we add all the variables of interest to a `Model` container and run our black-box algorithm over the model.

```
### To Be Explained in Chapter 3!
model = mc.Model( [yes_responses, p_skewed, p ] )
mcmc = mc.MCMC(model)
mcmc.sample( 20000, 7500 )

p_trace = mcmc.trace("freq_cheating")[:]
```

    [****************100%******************]  20000 of 20000 complete

```

plt.hist( p_trace, histtype="stepfilled" , normed = True,
    alpha = 0.85, bins = 30, label = "posterior distribution",
    color = "#348ABD")
plt.xlim(0,1)
plt.legend()
```

We are still pretty unconfident with what the true frequency of cheaters might be, but we have narrowed it down to a range between 0.10 to 0.30. This is pretty good, as *a priori* we had no idea how many students might have cheated (hence the uniform distribution). On the other hand, it is also pretty bad since there is a 20% window the true value the true value most likely lives in. Have we even gained anything, or are we still too uncertain about the true frequency?

I would argue, yes, we have discovered something. It is implausbile, according to our posterior that there are *no cheaters*, i.e. the posterior assigns 0 probability to $p=0$. Also, we started with a uniform prior, so we gave every $p$ a plausible chance, but the data ruled out $p=0$ as a possibility. We can be confident that there were cheaters.

This kind of algorithm can be used the gather private information from users and be *reasonablly* confident that the data, though noisy, is truthful.

#####Example: Challenger Space Shuttle Disaster
_______

On January 28, 1986, the twenty-fifth flight of the U.S. space shuttle pro-gram ended in disaster when one of the rocket boosters of the Shuttle Challenger exploded shortly after lift-off, killing all seven crew members. The presidential commission on the accident concluded that it was caused by the failure of an O-ring in a field joint on the rocket booster, and that this failure was due to a faulty design that made the O-ring unacceptably sensitive to a number of factors including outside temperature. Of the previous 24 flights, data were available on failures of O-rings on 23, (one was lost at sea), and these data were discussed on the evening preceding the Challenger launch, but unfortunately only the data corresponding to the 7 flights on which there was a damage incident were considered important and these were thought to show no obvious trend. The data are shown below (see [1]):

```
figsize( 12.5, 3.5 )
challenger_data = np.genfromtxt("chp2data/challenger_data.csv", skip_header = 1, \
                               usecols=[1,2], missing_values="NA", delimiter=",")
#drop the NA values
challenger_data = challenger_data[ ~np.isnan(challenger_data[:,1]) ]

#plot it, as a function of tempature (the first column)
print "Temp (F), O-Ring failure?"
print challenger_data

plt.scatter( challenger_data[:,0], challenger_data[:,1], s = 75, color="k",
 alpha = 0.5)
plt.yticks([0,1])
plt.ylabel("Damage Incident?")
plt.xlabel("Outside temperature (Farhenhit)" )
plt.title("Defects of the Space Shuttle O-Rings vs temperature")
print

```

It looks clear that *the probability* of damage incidents occuring increases as the outside temperature decreases. We are interested in modeling the probability here because it does not look like there is a strict cutoff point between temperature and a damage incident ocurring. The best we can do is ask "At temperature $t$, what is the probability of a damage indicident?". The goal of this example is to answer that question.

We need a function of temperature, call it $p(t)$, that is bounded between 0 and 1 (so as to model a probability) and changes from 1 to 0 as we increase temperature. There are actually many such functions, but the most popular choice is the *logistic function.*

$$p(t) = \frac{1}{ 1 + e^{ \;\beta t } } $$

In this model, $\beta$ is the variable we are uncertain about. Below is the function plotted for $\beta = 1, 3, -5$.

```
figsize(12,3)

def logistic( x, beta):
    return 1.0/( 1.0 + np.exp( beta*x) )
x = np.linspace( -4, 4, 100 )
plt.plot(x, logistic( x, 1), label = r"$\beta = 1$")
plt.plot(x, logistic( x, 3), label = r"$\beta = 3$")
plt.plot(x, logistic( x, -5), label = r"$\beta = -5$")
plt.legend()
```

But something is missing. In the plot above, the probability changes quickly only near zero, but in our data above the probability changes around 65 to 70. We need to add a *bias* term to our logistic function:

$$p(t) = \frac{1}{ 1 + e^{ \;\beta t + \alpha } } $$

Some plots are below, with differing $\alpha$.

```
def logistic( x, beta, alpha=0):
    return 1.0/( 1.0 + np.exp( np.dot(beta, x) + alpha) )

x = np.linspace( -4, 4, 100 )

plt.plot(x, logistic( x, 1), label = r"$\beta = 1$", ls= "--", lw =1 )
plt.plot(x, logistic( x, 3), label = r"$\beta = 3$", ls= "--", lw =1)
plt.plot(x, logistic( x, -5), label = r"$\beta = -5$", ls= "--", lw =1)

plt.plot(x, logistic( x, 1,1), label = r"$\beta = 1, \alpha = 1$",
    color = "#348ABD")
plt.plot(x, logistic( x, 3, -2), label = r"$\beta = 3, \alpha = -2$",
    color="#A60628")
plt.plot(x, logistic( x, -5, 7), label = r"$\beta = -5, \alpha = 7$",
    color = "#7A68A6")

plt.legend(loc = "lower left")
```

Adding a constant term $\alpha$ amounts to shifting the curve left or right (hence why it called a *bias*. )

Let's start modeling this in PyMC. The $\beta, \alpha$ paramters have no reason to be positive, bounded or relativly large, so there are best modeled by a Normal random variable.

### Normal distributions

A Normal random variable, denoted $X \sim N(\mu, 1/\tau)$, has a distribution with two parameters: the mean, $\mu$, and the *precision*, $\tau$. Those familar with the Normal distribution already have probably seen $\sigma^2$ instead of $\tau$. They are infact recipricals of each other. The change was motivated by easier mathematics and is an artifact of Bayesian methods. Just remember: The smaller $\tau$, the larger the spread of the distribution (i.e. we are more uncertain); the larger $\tau$, the tighter the distribution (i.e. we are more certain). Regardless, $\tau$ is always positive.

The probability density function of a $N( \mu, 1/\tau)$ random variable is:

$$ f(x | \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau}{2} (x-\mu)^2 \right) $$

We plot some different density functions below.

```
import scipy.stats as stats
nor = stats.norm
x = np.linspace( -8, 7, 150 )
mu = (-2, 0, 3)
tau = (.7, 1, 2.8 )
colors = ["#348ABD", "#A60628", "#7A68A6"]
parameters = zip( mu, tau, colors )

for _mu, _tau, _color in parameters:
    plt.plot( x, nor.pdf( x, _mu , scale = 1./_tau ), \
        label ="$\mu = %d,\;\\tau = %.1f$"%(_mu, _tau), color = _color )
    plt.fill_between( x, nor.pdf( x, _mu, scale =1./_tau ), color = _color, \
         alpha = .33)

plt.legend(loc = "upper right")
plt.xlabel("$x$")
plt.ylabel("density function at $x$")
plt.title( "Probability distribution of three different Normal random variables" )
```

A Normal random variable can be any real number, but the variable is very likely to be relatively close to $\mu$. In fact, the expected value of a Normal is equal to its $\mu$ parameter:

$$ E[ X | \mu, \tau] = \mu$$

and it's variance is equal to the inverse of $\tau$:

$$Var( X | \mu, \tau ) = \frac{1}{\tau}$$

Below we continue our modeling of the defects:

```
import pymc as mc

temperature = challenger_data[:,0]
D = challenger_data[:,1] #defect or not?

beta = mc.Normal( "beta", 0, 0.001, value = 0 )
alpha = mc.Normal( "alpha", 0, 0.001, value = 0 )

@mc.deterministic
def p( temp = temperature, alpha = alpha, beta = beta):
    return 1.0/( 1. + np.exp( beta*temperature + alpha) )

```

We have our probabilities, but how do we connect them to our observed data? A *Bernoulli* random variable with parameter $p$, denoted $\text{Ber}(p)$, is a random variable that takes value 1 with probability $p$, and 0 else. Thus, our model can look like:

$$ \text{Defect Incident, $D_i$} \sim \text{Ber}( \;p(t_i)\; ), \;\; i=1..N$$

where $p(t)$ is our logistic function and $t_i$ are the temperatures we have observations about. Notice in the above code we had to set the values of `beta` and `alpha` to 0. The reason for this is that if `beta` and `alpha` are very large, they make `p` equal to 1 or 0. Unfortunatly, `mc.Bernoulli` does not like probabilities of 0 or 1, though they are mathematically well-defined probabilties. So by setting the values to `0`, we set the variable `p` to be a reasonable starting value. This has no effect on our results, nor does it mean we are including information in our prior. It is simply a computational caveat in PyMC.

```
p.value
```

    array([ 0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,
            0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,  0.5,
            0.5])

```
observed = mc.Bernoulli( "bernoulli_obs", p, value = D, observed=True)

model = mc.Model( [observed, beta, alpha] )

#mysterious code to be explained in Chapter 3
map_ = mc.MAP(model)
map_.fit()
mcmc = mc.MCMC( model )
mcmc.sample( 260000, 220000, 2 )

```

    [****************100%******************]  260000 of 260000 complete

We have trained our model on the observed data, now we can sample values from the positerior. Let's look at the positerior distributions for $\alpha$ and $\beta$:

```
alpha_samples = mcmc.trace( 'alpha' )[:, None] #best to make them 1d
beta_samples = mcmc.trace( 'beta' )[:, None]

figsize(12.5, 6)

#histogram of the samples:
plt.subplot(211)
plt.title(r"Posterior distributions of the variables $\alpha, \beta$")
plt.hist( beta_samples, histtype='stepfilled', bins = 45, alpha = 0.85, \
        label = r"positerior of $\beta$", color = "#7A68A6",normed = True )
plt.legend()

plt.subplot(212)
plt.hist( alpha_samples, histtype='stepfilled', bins = 45, alpha = 0.85, \
        label = r"positerior of $\alpha$", color = "#A60628",normed = True )
plt.legend()

```

Regarding the spread of the data, we are very uncertain about what the true parameters might be (though considering the low sample size and the large overlap of defects-to-nondefects this behaviour is perhaps expected).

Next, let's look at the *expected probability* for a specific value of the temperature. That is, we average over all samples from the posterior to get a likely value for $p(t_i)$.

```
t = np.linspace( temperature.min() - 5, temperature.max()+5, 50 )[:,None]
p = logistic( t.T, beta_samples, alpha_samples )

mean_prob_t = p.mean(axis=0)
```

```
figsize( 12.5, 4)
plt.plot( t, mean_prob_t, lw = 3, label = "average posterior \nprobability \
of defect")
plt.plot( t, p[0, :], ls="--",label="realization from posterior" )
plt.plot( t, p[-2, :], ls="--", label="realization from posterior"  )
plt.scatter( temperature, D, color = "k", s = 50, alpha = 0.5 )
plt.title("Posterior expected value of probability of defect; plus realizations")
plt.legend(loc= "lower left")
plt.ylim( -0.1, 1.1 )
plt.xlim( t.min(), t.max() )
plt.ylabel("probability")
plt.xlabel("temperature")
```

Above we also plotted two possible realizations of what the actual underlying system might be. Both are equally likely as any other draw. The blue line is what occurs when we average all the 20000 possible dotted lines together.

An interesting question to ask is for what temperatures are we most uncertain about the defect-probability? Below we plot the expected value line **and** the associated distribution at each temperature.

```
n = 2500
for i, _t in enumerate(t[:-1]):
    plt.scatter( _t*ones( n ) + 0.12*np.random.randn(n), p[:n,i],
                alpha = 0.005,
                color = "#7A68A6")

#what's a better way to only have a for loop and include it in the legend? and
# have it somewhat show. You can see an artifact at t = 85.
plt.scatter( t[-1]*ones( n ) + 0.20*np.random.randn(n), p[:n,-1],
        alpha = 0.3,
        color = "#7A68A6",
        label = "samples from posterior at temp $t$")

plt.plot( t, mean_prob_t, lw = 1, ls= "--", color = "k",
        label = "average posterior \nprobability of defect")

plt.xlim( t.min(), t.max() )
plt.ylim( -0.02, 1.02 )
plt.legend(loc="lower left")
plt.scatter( temperature, D, color = "k", s = 50, alpha = 0.5 )
plt.xlabel("temp, $t$")

plt.ylabel("probability estimate" )
plt.title( "Posterior probability estimates given temp. $t$" )
```

We can see that as the temperature nears 60 degrees, the distributions spread out over [0,1] quickly. As we pass 70 degrees, the distributions tighten again. This can give us insight about how to proceed next: we should probably test more O-rings around 60-65 temperature to get a better estimate of probabilities in that range. Similarly, when reporting to scientists your estimates, you should be very cautious about simply telling them the expected probability, as we can see this does not reflect how wild the posterior distribution is.

### What about the day of the Challenger disaster?

On the day of the Challenger disaster, the outside temperature was 31 degrees fahrenheit. What is the positerior distribution of a defect occuring,  given this temperature? The distribution it plotted below. It looks almost garunteed that the Challenger was going to be subject to defective O-rings.

```
figsize(8, 2.5)
prob_31 = logistic( 31, beta_samples, alpha_samples )

plt.xlim( 0.95, 1)
plt.hist( prob_31, bins = 300, normed = True, histtype='stepfilled' )
plt.title( "Posterior distribution of probability of defect, given $t = 31$")
plt.xlabel( "probability of defect occuring in O-ring" )

print
```

### More PyMC Tricks

#### Protip: *Lighter* deterministic variables with `Lambda` class

Sometimes writing a deterministic function using the `@mc.deterministic` dectorator can seem like a chore, especially for small function. I have already mentioned that simple elementary math operations *can* produce deterministic variables implicitly, but what about operations like indexing or slicing? Built-in `Lambda` functions can achieve the elegance desired plus that simplicity required. For example,

    beta = mc.Normal( "coefficients", 0, size=(N,1) )
    x = np.random.randn( (N,1) )
    linear_combination = mc.Lambda( lambda x=x, beta = beta, np.dot( x.T, beta ) )

#### Protip: Arrays of PyMC variables
There is no reason why we cannot store multiple heterogeneous PyMC variables in a Numpy array. Just remember to set the `dtype` of the array to `object` upon initialization. For example:

```
N = 10
x = np.empty( N , dtype=object )
for i in range(0, N):
        x[i] = mc.Exponential('x_%i' % i, (i+1)**2)
```

### Bonus: Returning to the Challenger data
#####Is there really a relationship between failure and temperature?

An critism of our above analysis is that *assumed* that the relationship followed a logistic model, this we implictly assumed that the probabilities change over temperature. Let's look at the data again. (Top figure)

Could it be that infact this particular sequence of defects occured by chance?  After all, I can produce similar plots. (Bottom figure) This might explain the large overlap in temperatures.

```
figsize( 12.5, 6)
subplot(211)
plt.scatter( challenger_data[:,0], challenger_data[:,1], s = 75, \
    color="k", alpha = 0.5)
plt.yticks([0,1])
plt.ylabel("Damage Incident?")
plt.title("(Real) Defects of the Space Shuttle O-Rings vs temperature")

subplot(212)
n = challenger_data.shape[0]
plt.scatter( challenger_data[:,0], stats.bernoulli.rvs(0.6, size = n) ,
     s = 75, color="k", alpha = 0.5)
plt.yticks([0,1])
plt.ylabel("Damage Incident?")
plt.xlabel("Outside temperature (Farhenhit)" )
plt.title("(Artificial) Defects of the Space Shuttle O-Rings vs \
temperature")

```

#### Introducing the Bayes factor

The *Bayes factor* is a measure comparing two models. In our example, on the one hand we believe that temperature plays an important role in the probability of O-ring failures. On the other hand, we believe that there is no significant connection, and the pattern appears by coincidence. We can compare these model using the ratio of the *probabilities of observing the data, given the model*:

$$ \frac{ P( \text{observations} | \text{Model}_1 ) }{  P( \text{observations} | \text{Model}_2 ) }$$

Calculating this is not at all obvious. For simplicity, let's select a set of parameters from the posterior distribution (which is tantamount to selecting a particular model).

```
alpha_hat = alpha_samples[0,0] #select the first alpha sample
beta_hat = beta_samples[0,0] #select the first beta sample

phat = logistic( temperature, beta_hat, alpha_hat)
print "estimates of probability at observed temperature, defects: "
print np.array(zip(p_hat, temperatures, D) )[:3, :]
```

    estimates of probability at observed temperature, defects:
    [[  0.35   66.      0.   ]
     [  0.166  70.      1.   ]
     [  0.204  69.      0.   ]]

To calculate the numerator in the Bayes factor, we start by **assuming a model**, in our case assume `alpha_hat`, `beta_hat` are true, and caluculate the probability of the defects we observe: equal to the product of:

$$ P(\; \text{Ber}(\; p(t_i \; | \; \hat{\beta}, \hat{\alpha} )\; ) = D_i \; ),\;\; i=1..N $$

For example, using the output above, the first computation, $i=0$, would look like

\begin{align}
& p = p(t_1 \; | \; \hat{\beta}, \hat{\alpha} ) = 0.667 \\\\
& d = D_0 = 0 \\\\
& \Rightarrow \; P( \; \text{Ber}(p) = d ) = ??
\end{align}

The probability of this ocurring is $(1-0.667) = 0.333\; $ (Recall the definition of a Bernoulli random variable $\text{Ber}(p)$: it is equal to $1$ with probability $p$ and equal to 0 with probability $1-p$). As each observation is independent, we multiply all the observations together. A clever way to do this is for a specific $i$ is, recalling the $D_i$ can only be 1 or 0:

$$\left( p(t_1 \; | \; \hat{\beta}, \hat{\alpha} )\right)^{D_i} \left( 1 - p(t_1 \; | \; \hat{\beta}, \hat{\alpha} ) \right)^{(1-D_i)}$$

(it is possible that the quantity can overflow, so it is recommended to take the $\log$ of the above::

$$ D_i\log( p(t_1 \; | \; \hat{\beta}, \hat{\alpha} ) ) + (1-D_i)\log( 1- p(t_1 \; | \; \hat{\beta}, \hat{\alpha} ) ) $$

and sum the terms instead of multiplying. Be sure use this transform for both models you are comparing.)

```
print p_hat
_product = p_hat**( D )*(1-p_hat)**(1-D)
prob_given_model_1 = _product.prod()
print "probability of observations, model 1: %.10f"%prob_given_model_1
```

    [ 0.372  0.372  0.372  0.372  0.372  0.372  0.372  0.372  0.372  0.372
      0.372  0.372  0.372  0.372  0.372  0.372  0.372  0.372  0.372  0.372
      0.372  0.372  0.372]
    probability of observations, assuming model 1: 0.0000005776

The second model we are comparing against is that $\beta=0$, i.e. there is no relationship between probabilites and temperature. We perform the same calculations as above. Notice that `beta=0` here in the below PyMC code:

```
beta = 0
alpha = mc.Normal( "alpha", 0, 0.001, value = 0 )

@mc.deterministic
def p( temp = temperature, alpha = alpha, beta = beta):
    return 1.0/( 1. + np.exp( beta*temperature + alpha) )

observed = mc.Bernoulli( "bernoulli_obs", p, \
                        value = D, observed=True)

model = mc.Model( {"observed":observed, "beta":beta, "alpha":alpha} )

#mysterious code to be explained in Chapter 3
map_ = mc.MAP(model)
map_.fit()
mcmc = mc.MCMC( model )
mcmc.sample( 260000, 220000, 3 )
######

alpha_samples_model_2 = mcmc.trace("alpha")[:, None]
alpha_hat = alpha_samples_model_2[0] #use the first 'model'
beta_hat = 0

p_hat = logistic( temperature, beta_hat, alpha_hat )
print "estimates of probability at observed temperature, defects: "
print np.array(zip(p_hat, temperature, D ) )[:3, :]
print
print "Notice the probability is constant for all temperatures?"
```

    [****************100%******************]  260000 of 260000 complete estimates of probability at observed temperature, defects:
    [[  0.397  66.      0.   ]
     [  0.397  70.      1.   ]
     [  0.397  69.      0.   ]]
    Notice the probability is constant for all temperatures?

```
#compute the probability of observations given the model_2

_product = p_hat**( D )*(1-p_hat)**(1-D)
prob_given_model_2 = _product.prod()
print "probability of observations, model 2: %.10f"%prob_given_model_2
```

    probability of observations, model 2: 0.0000004761

```
print "Bayes factor = %.3f"%(prob_given_model_1/prob_given_model_2)
```

    Bayes factor = 40.075

Is this good? Below is a chart that can be used to compare the computed Bayes factors to a degree of confidence in M1.

<table><tbody><tr><th>Bayes factor</th><th>Supporting M1</th></tr><tr><td><1:1  </td><td>Negative (supports M2)</td></tr><tr><td>  1:1 to 3:1 </td><td>Barely worth mentioning</td></tr><tr><td>  3:1 to 10:1 </td><td>Substantial</td></tr><tr><td>  10:1 to 30:1 </td><td>Strong</td></tr><tr><td>  30:1 to 100:1</td><td> Very strong</td></tr><tr><td>  > 100:1</td><td> Decisive</td></tr></tbody></table>

We are not done yet. If you recall, we selected only one model, but recall we actually have a distribution of possible models (sequential pairs of ($\beta, \alpha$) from the posterior disitributions). So to be correct we need to average over samples from the posterior:

```
p = logistic( temperature[None,:], beta_samples, alpha_samples )
_product = p**( D )*(1-p)**(1-D)
prob_given_model_1 = _product.prod(axis=1).mean()
print "expected prob. of obs., given model 1: %.10f"%prob_given_model_1

p_model_2 = logistic( temperature[:,None],
                np.zeros_like(temperature), alpha_samples_model_2)

_product = p_model_2**( D )*(1-p_model_2)**(1-D)
prob_given_model_2 = _product.prod(axis=1).mean()
print "expected prob. of obs., given model 2: %.10f"%prob_given_model_2
print
print "Bayes factor: %.3f"%(prob_given_model_1/prob_given_model_2)

```

    expected prob. of obs., given model 1: 0.0000190809
    expected prob. of obs., given model 2: 0.0000005072

    Bayes factor: 37.618

It looks we have a pretty good model, and temperature *is* significant. Look at you, you're a rocket scientist now.

##### Exercises

1\. How would add a prior belief that smoking causes more deaths in first example?

2\. Try plotting $\alpha$ samples versus $\beta$ samples.  Why might it look like this?

```
#type your code here.

plt.scatter( alpha_samples, beta_samples, alpha = 0.1 )
plt.title( "Why does the plot look like this?" )
plt.xlabel( r"$\alpha$")
plt.ylabel( r"$\beta$")
```

### References

-  [1] Dalal, Fowlkes and Hoadley (1989),JASA, 84, 945-957.
-  [2] German Rodriguez. Datasets. In WWS509. Retrieved 30/01/2013, from http://data.princeton.edu/wws509/datasets/#smoking.
-  [3] Reference bayes book by Dr. Cyntha
-  [4] Fonnesbeck, Christopher. "Building Models." PyMC-Devs. N.p., n.d. Web. 26 Feb 2013. <http://pymc-devs.github.com/pymc/modelbuilding.html>.

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    h1 {
        font-family: "Charis SIL", Palatino, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 120%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 16pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

```

```


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Class 13: Introduction to Real Business Cycle Modeling

Real business cycle (RBC) models are extensions of the stochastic Solow model. RBC models replace the ad hoc assumption of a constant saving rate in the Solow model with the solution to an intertemporal utility maximization problem that gives rise to a variable saving rate. RBC models also often feature some sort of household labor-leisure tradeoff that produces endogenous labor varation.

In this notebook, we'll consider a baseline RBC model that does not have labor. We'll use the model to compute impulse responses to a one percent shock to TFP.

## The Baseline RBC Model without Labor

The equilibrium conditions for the RBC model without labor are:

\begin{align}
\frac{1}{C_t} & = \beta E_t \left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1} +1-\delta }{C_{t+1}}\right]\\
K_{t+1} & = I_t + (1-\delta) K_t\\
Y_t & = A_t K_t^{\alpha}\\
Y_t & = C_t + I_t\\
\log A_{t+1} & = \rho \log A_t + \epsilon_{t+1}
\end{align}

where $\epsilon_{t+1} \sim \mathcal{N}(0,\sigma^2)$.

The objective is use `linearsolve` to simulate impulse responses to a TFP shock using the following parameter values for the simulation:

| $\rho$ | $\sigma$ | $\beta$ | $\alpha$ | $\delta $ | $T$ |
|--------|----------|---------|----------|-----------|-----|
| 0.75   | 0.006    | 0.99    | 0.35     |  0.025    | 26  |

## Model Preparation

Before proceding, let's recast the model in the form required for `linearsolve`. Write the model with all variables moved to the lefthand side of the equations and dropping the expecations operator $E_t$ and the exogenous shock $\epsilon_{t+1}$:

\begin{align}
0 & = \beta\left[\frac{\alpha A_{t+1}K_{t+1}^{\alpha-1} +1-\delta }{C_{t+1}}\right] - \frac{1}{C_t}\\
0 & = A_t K_t^{\alpha} - Y_t\\
0 & = I_t + (1-\delta) K_t - K_{t+1}\\
0 & = C_t + I_t - Y_t\\
0 & = \rho \log A_t - \log A_{t+1}
\end{align}

Remember, capital and TFP are called *state variables* because they're $t+1$ values are predetermined. Output, consumption, and investment are called a *costate* or *control* variables. Note that the model as 5 equations in 5 endogenous variables.

## Initialization, Approximation, and Solution

The next several cells initialize the model in `linearsolve` and then approximate and solve it.

```python
# Create a variable called 'parameters' that stores the model parameter values in a Pandas Series
parameters = pd.Series()
parameters['rho'] = .75
parameters['sigma_squared'] = 0.006**2
parameters['beta'] = 0.99
parameters['alpha'] = 0.35
parameters['delta'] = 0.025

# Print the model's parameters
print(parameters)
```

    rho              0.750000
    sigma_squared    0.000036
    beta             0.990000
    alpha            0.350000
    delta            0.025000
    dtype: float64

```python
# Create variable called 'varNames' that stores the variable names in a list with state variables ordered first
varNames = ['a','k','y','c','i']

# Create variable called 'shockNames' that stores an exogenous shock name for each state variable.
shockNames = ['e_a','e_k']
```

```python
# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # Euler equation
    euler_equation = p.beta*(p.alpha*fwd.a*fwd.k**(p.alpha-1)+1-p.delta)/fwd.c - 1/cur.c

    # Production function
    production_function = cur.a*cur.k**p.alpha - cur.y

    # Capital evolution
    capital_evolution = cur.i + (1 - p.delta)*cur.k - fwd.k

    # Market clearing
    market_clearing = cur.c+cur.i - cur.y

    # Exogenous tfp
    tfp_process = p.rho*np.log(cur.a) - np.log(fwd.a)

    # Stack equilibrium conditions into a numpy array
    return np.array([
        euler_equation,
        production_function,
        capital_evolution,
        market_clearing,
        tfp_process
        ])
```

Next, initialize the model using `ls.model` which takes the following required arguments:

* `equations`
* `nstates`
* `varNames`
* `shockNames`
* `parameters`

```python
# Initialize the model into a variable named 'rbc_model'
rbc_model = ls.model(equations = equilibrium_equations,
                 nstates=2,
                 varNames=varNames,
                 shockNames=shockNames,
                 parameters=parameters)
```

```python
# Compute the steady state numerically using .compute_ss() method of rbc_model
guess = [1,4,1,1,1]
rbc_model.compute_ss(guess)

# Print the computed steady state
print(rbc_model.ss)
```

    a     1.000000
    k    34.398226
    y     3.449750
    c     2.589794
    i     0.859956
    dtype: float64

```python
# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of rbc_model
rbc_model.approximate_and_solve()
```

### Impulse Responses

Compute a 26 period impulse responses of the model's variables to a 0.01 unit shock to TFP in period 5.

```python
# Compute impulse responses
rbc_model.impulse(T=26,t0=5,shocks=[0.01,0])

# Print the first 10 rows of the computed impulse responses to the TFP shock
print(rbc_model.irs['e_a'].head(10))
```

        e_a         a         k         y         c         i
    0  0.00  0.000000  0.000000  0.000000  0.000000  0.000000
    1  0.00  0.000000  0.000000  0.000000  0.000000  0.000000
    2  0.00  0.000000  0.000000  0.000000  0.000000  0.000000
    3  0.00  0.000000  0.000000  0.000000  0.000000  0.000000
    4  0.00  0.000000  0.000000  0.000000  0.000000  0.000000
    5  0.01  0.010000  0.000000  0.010000  0.001253  0.036342
    6  0.00  0.007500  0.000909  0.007818  0.001493  0.026865
    7  0.00  0.005625  0.001557  0.006170  0.001654  0.019772
    8  0.00  0.004219  0.002013  0.004923  0.001755  0.014465
    9  0.00  0.003164  0.002324  0.003978  0.001812  0.010499

Construct a $2\times2$ grid of plots of simulated TFP, output, consumption, and investment. Be sure to multiply simulated values by 100 so that vertical axis units are in "percent deviation from steady state."

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,8))

# Create upper-left axis. PROVIDED
ax = fig.add_subplot(2,2,1)
ax.plot(rbc_model.irs['e_a']['a']*100,'b',lw=5,alpha=0.75)
ax.set_title('TFP')
ax.set_ylabel('% dev from steady state')
ax.set_ylim([-0.2,1.2])
ax.grid()

# Create upper-right axis. PROVIDED
ax = fig.add_subplot(2,2,2)
ax.plot(rbc_model.irs['e_a']['y']*100,'b',lw=5,alpha=0.75)
ax.set_title('Output')
ax.set_ylabel('% dev from steady state')
ax.set_ylim([-0.2,1.2])
ax.grid()

# Create lower-left axis. PROVIDED
ax = fig.add_subplot(2,2,3)
ax.plot(rbc_model.irs['e_a']['c']*100,'b',lw=5,alpha=0.75)
ax.set_title('Consumption')
ax.set_ylabel('% dev from steady state')
ax.set_ylim([-0.05,0.30])
ax.grid()

# Create lower-right axis. PROVIDED
ax = fig.add_subplot(2,2,4)
ax.plot(rbc_model.irs['e_a']['i']*100,'b',lw=5,alpha=0.75)
ax.set_title('Investment')
ax.set_ylabel('% dev from steady state')
ax.set_ylim([-1,6])
ax.grid()

fig.tight_layout()
```


```python
#programacionlineal
```

```python
#lamba es una palabra reservada para definir una funcion
```

```python
%matplotlib notebook
from sympy import *
import numpy as np
```

```python
var("x y")
```
!pip install pulp

```python
from pulp import *
```

```python
pulp.LpVariable?
```

```python
mi_lp_problema = pulp.LpProblem("Mi LP problema",pulp.LpMinimize)

x= pulp.LpVariable("x",lowBound=0,cat="Continuous")
y= pulp.LpVariabñe("y",lowBound=0,cat="Continuous")

mi_lp_problema += x +2*y

mi_lp_problema +=3*x +4*y >= 1
mi_lp_problema +=2*x +5*y >= 2
```

```python
mi_lp_problema
```

    Mi LP problema:
    MINIMIZE
    None
    VARIABLES

```python
mi_lp_problema.solve()
pulp.LpStatus[]
```


# Training and Testing

**DIVE into Deep Learning**
___

```python
%load_ext tensorboard
%matplotlib inline
from util import *
```

## Optimization Theory

**Why can we learn from examples?**$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}
$

For the problem to be meaningful, $(\RM{x},\R{y})$ is assumed to be random with some unknown joint distribution $p_{\RM{x},\R{y}}$:

- If we always had $\R{y}=y$ instead, then a perfect classifier can just return $y$ without even looking at $\RM{x}$.

- If $p_{\RM{x},\R{y}}$ were known instead, then $p_{\R{y}|\RM{x}}$ would also be known and therefore needed not be estimated.

Examples are often collected randomly and independently from a population, i.e., as [i.i.d. samples](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables) of $(\RM{x},\R{y})$:

$$(\RM{x}_1,\R{y}_1), (\RM{x}_2,\R{y}_2), \dots\sim_{\text{i.i.d.}} p_{\RM{x},\R{y}}.$$

- If all the examples were the same instead, the examples could not show the pattern of how $\R{y}$ depends on $\RM{x}$.

- The observed distribution of i.i.d. samples converge to the unknown distribution by the [law of large number](https://en.wikipedia.org/wiki/Law_of_large_numbers).

**How to determine if a classifier is good?**

Ultimately, we desire a classifier with the maximum accuracy in predicting $\R{y}$ but doing so is [computationally too difficult](https://en.wikipedia.org/wiki/Loss_functions_for_classification).

Instead, we regard a classification algorithm to be reasonably good if
- it can achieve the maximum possible accuracy
- as the number of training samples goes to $\infty$.

**Definition** A probabilistic classifier for the input feature $\RM{x}$ and label $\R{y}$ with unknown joint distribution  is a conditional pmf

$$
\R{q}_{\R{y}|\RM{x}}(y|\RM{x})\qquad \text{for }\M{x}\in \mc{X}, y\in \mc{Y}
$$

defined as a function of the i.i.d. samples

$$\{(\RM{x}_i,\R{y}_i)\}_{i=1}^N$$

of $(\RM{x},\R{y})$ (but independent of $(\RM{x},\R{y})$).
The classifier is said to be a *consistent* estimate (of $p_{\R{y}|\M{x}}$) if

$$
\begin{align}
\lim_{N\to \infty} \Pr\Set{\R{q}_{\R{y}|\RM{x}}(y|\RM{x})=p_{\R{y}|\RM{x}}(y|\RM{x})\text{ for all } y\in \mc{Y}}=1,\tag{consistency}
\end{align}
$$

namely, $\R{q}_{\R{y}|\RM{x}}(y|\RM{x})$ converges almost surely (a.s.) to $p_{\R{y}|\RM{x}}$. $\square$

A consistent probabilistic classifier gives rise to an asymptotically optimal hard-decision classifier that achieves the maximum accuracy.

**Proposition** If for some $\epsilon\geq 0$ that

$$\Pr\Set{\R{q}_{\R{y}|\RM{x}}(y|\RM{x})=p_{\R{y}|\RM{x}}(y|\RM{x}) \text{ for all } y\in \mc{Y}}=1-\epsilon,$$

the (hard-decision) classifier

$$\begin{align}\R{f}(\RM{x}):=\arg\max_{y\in \mc{Y}} \R{q}_{\R{y}|\RM{x}}(y|\RM{x})\tag{hard decision}\end{align}$$

achieves an accuracy

$$
\begin{align}
\sup_{f:\mc{X}\to \mc{Y}} \Pr(\R{y}= f(\RM{x})) &\geq \E\left[\max_{y\in \mc{Y}} p_{\R{y}|\M{x}}(y|\RM{x})\right] - \epsilon,\tag{accuracy lower bound}
\end{align}
$$

where the expectation is the maximum possible accuracy. $\square$

*Proof:* For any classifier $f$,

$$ \begin{align*}
\Pr(\R{y}= f(\RM{x}))
&= \E[p_{\R{y}|\M{x}}(f(\RM{x})|\RM{x})] \\
&\leq \E\left[\max_{y\in \mc{Y}} p_{\R{y}|\M{x}}(y|\RM{x})\right]\end{align*}
$$

where the last inequality is achievable with equality with the hard-decision classifier and $\R{q}$ replaced by $p$. This implies the desired accuracy lower bound for the case $\epsilon=0$. The more general case with $\epsilon\geq 0$ can be derived similarly. $\blacksquare$

**How can we obtain a consistent classifier?**

We train a neural network to minimize certain *loss* function.

A common loss function for classification uses the [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) measure in information theory.

The theoretical underpinning is the following identity that relates three information quantities:

$$
\begin{align}
\overbrace{\E\left[\log \frac{1}{q_{\R{y}|\RM{x}}(\R{y}|\RM{x})}\right]}^{ \text{Cross entropy}\\ H(p_{\R{y}|\RM{x}}\|q_{\R{y}|\RM{x}}|p_{\RM{x}}):=} &\equiv \overbrace{\E\left[\log \frac{1}{p_{\R{y}|\RM{x}}(\R{y}|\RM{x})}\right]}^{\text{Conditional entropy}\\ H(\R{y}|\RM{x}):=} + \overbrace{\E\left[\log \frac{p_{\R{y}|\RM{x}}(\R{y}|\RM{x})}{q_{\R{y}|\RM{x}}(\R{y}|\RM{x})}\right].}^{\text{Divergence}\\ D(p_{\R{y}|\RM{x}}\|q_{\R{y}|\RM{x}}|p_{\RM{x}}):=}\tag{identity}
\end{align}
$$

The identity can be proved quite easily using the linearity of expectation

$$ \E[\R{u}+\R{v}]=\E[\R{u}]+\E[\R{v}],$$

and a property of logarithm that

$$\log uv = \log u+ \log v.$$

```python
%%html
<b>Information Identity</b><br>

```

**Proposition** With $q_{\R{y}|\RM{x}}(y|\M{x})$ being a valid pmf of a random variable taking values from $\mc{Y}$ conditioned on a random variable taking values from $\mc{X}$,

$$\begin{align}
\min_{q_{\R{y}|\RM{x}}} H(p_{\R{y}|\RM{x}}\|q_{\R{y}|\RM{x}}|p_{\RM{x}})
&= H(\R{y}|\RM{x})
\end{align}$$

and the optimal solution equals $p_{\R{y}|\RM{x}}(y|\RM{x})$ a.s. for all $y\in \mc{Y}$. $\square$

Hence, a neural network that minimizes the cross entropy equals $p_{\R{y}|\RM{x}}(y|\RM{x})$ a.s. for all $y\in \mc{Y}$ and any possible input image $\RM{x}$.

**Exercise** Prove the above proposition using the information identity and the property of divergence that

$$\begin{align}
D(p_{\R{y}|\RM{x}}\|q_{\R{y}|\RM{x}}|p_{\RM{x}})\geq 0 \tag{positivity of divergence}
\end{align}$$

with equality iff $q_{\R{y}|\RM{x}}(y|\RM{x})=p_{\R{y}|\RM{x}}(y|\RM{x})$ a.s. (This, in turn, can be proved using the [log-sum inequality](https://en.wikipedia.org/wiki/Log_sum_inequality):

$$\begin{align}
\sum_{i} a_i \log\left(\frac{a_i}{b_i}\right) \geq (\textstyle \sum_{i} a_i) \log\left(\frac{\sum_{i} a_i}{\sum_{i} b_i}\right)\tag{log-sum inequality}
\end{align}$$

for any sequences $\{a_i\}$, $\{b_i\}$, and $\{c_i\}$.)

YOUR ANSWER HERE

The cross entropy cannot be computed exactly without knowing the joint distribution $p_{\RM{x}\R{y}}$. Nevertheless, it can be estimated from a batch of i.i.d. samples $(\RM{x}_{\R{j}_i},\R{y}_{\R{j}_i})$ for $1\leq i\leq n$:

$$\begin{align}
\R{L}(\theta)&:=\frac1n \sum_{i=1}^n \log \frac{1}{q_{\R{y}|\RM{x}}(\R{y}_{\R{j}_i}|\RM{x}_{\R{j}_i})}\tag{empirical loss}
\end{align}$$

where

$$\theta := \operatorname{flat}(\M{W}^{(\ell)},\M{b}^{(\ell)}\mid 0\leq \ell \leq L)$$

is the vector of parameters of the neural network defined in (net).

A mini-batch [gradient descent algorithm](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) is often used to reduce the loss. It iteratively updates/trains the neural network parameters:

$$\theta \leftarrow \theta -s\nabla \R{L}(\theta)$$

by computing the gradient $\nabla \R{L}(\theta)$ on a randomly selected minibatch of examples and choosing an appropriate learning rate $s$.

```python
%%html
<div>
<b>What is gradient descent?</b><br>

</div>
<div>
<b>How to choose the step size?</b><br>

</div>
```

- The gradient can be computed systematically using a technique called *backpropagation* due to the structure of the neural network in (net).
- The learning rate can affect the convergence rate of the loss to a local minima:
    - $\theta$ may overshoot its optimal value if $s$ is too large, and
    - the convergence can be very slow if $\theta$ is too small.

A more advanced method called [Adam (Adaptive Momentum Estimation)](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Adam) can adaptively choose $s$ to speed up the convergence.

## Training

The [loss function](https://keras.io/api/losses/), gradient descent algorithm, and the performance metrics can be specified using the [`compile` method](https://keras.io/api/losses/).

```python
def compile_model(model):
    model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(),
                  optimizer=tf.keras.optimizers.Adam(0.001),
                  metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
    return model

compile_model(model)
model.loss, model.optimizer
```

We can train the neural network using the method [`fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) of the compiled model:

```python
if input('Train? [Y/n]').lower() != 'n':
    model.fit(ds_b["train"])
```

**Exercise** By default, the neural network is trained for 1 epoch. What happens to the training accuracy if you rerun the above cell to train the model for another epoch?

YOUR ANSWER HERE

We can set the parameter `epochs` to train the neural network for multiple epochs since it is quite unlikely to train a neural network well with just one epoch.

To determine whether the neural network is well-trained (when to stop training), we should also use a separate validation set to evaluate the performance of the neural network. The validation set can be specified using the parameter `validation_set` as follows:

```python
if input('Train? [Y/n]').lower() != 'n':
    model.fit(ds_b["train"], epochs=6, validation_data=ds_b["test"])
```

**Exercise** Is the maximum validation accuracy `val_sparse_categorical_accuracy` (over different epoches) an unbiased estimate of the performance of deep learning for the given dataset? If not, how to avoid the bias?

YOUR ANSWER HERE

## Deployment

Once you are satisfied with the result, you can deploy the model as a web application.

The `mnist` folder contain the webpage `index.html` that
- presents an HTML5 canvas for users to input a handwritten digit,
- loads a trained model using [`tensorflow.js`](https://www.tensorflow.org/js),
- passes the handwritten digit to the model to predict the distribution of the digit types, and
- display the most likely digit type.

```python
display.Code('mnist/index.html')
```

First, save the model as follows in [HDF5](https://www.h5py.org/) format:

```python
if input('Execute? [Y/n]').lower() != 'n':
    model.save('mnist/model.h5')
```

Then, convert the model to files that can be loaded by [`tensorflow.js`](https://www.tensorflow.org/js):

```python
if input('Execute? [Y/n]').lower() != 'n':
    !tensorflowjs_converter --input_format keras 'mnist/model.h5' 'mnist/model'
```

To host the web application, run the following command:

```python
if input('Execute? [Y/n]').lower() != 'n':
    !cp -rf mnist ~/www/
    !zip -r mnist.zip mnist/*
```

- Preview you web application at [\<JUPYTERHUB_URL\>/user-redirect/www/mnist/](/user-redirect/www/mnist/).
- Download [`mnist.zip`](./mnist.zip) and extract it to a static web server of your choice.