# Path Planning for a Simple Car

The following equations describe a simple model of a car

\begin{align}
\frac{dx}{dt} & = u \cos(\theta) \\
\frac{dy}{dt} & = u \sin(\theta) \\
\frac{d\theta}{dt} & = \frac{u}{L}\tan(\phi) \\
\end{align}

where $x$ and $y$ denote the position of the center of the rear axle, $\theta$ is the angle of the car axis to the horizontal, and $phi$ is the angle of the front steering wheels to the car axis. The length $L$ is the distance from the center of the rear axle to the center of the front axle.

The velocity $u$ is controlled by acceleration of the car, the position of the wheels is controlled by the rate limited steering input $v$.

\begin{align}
\frac{du}{dt} & = a \\
\frac{d\phi}{dt} & = v
\end{align}

The state of the car is determined by the value of the five state variables $x$, $y$, $\theta$, $u$, and $\phi$.

The path planning problem is to find find values of the functions $a(t)$ and $v(t)$ on a time interval $0 \leq t \leq t_f$ to steer the car from an initial condition $\left[x(0), y(0), \theta(0), u(0), \phi(0)\right]$ to a specified final condition $\left[x(t_f), y(t_f), \theta(t_f), u(t_f), \phi(t_f)\right]$ that minimizes an objective function:

\begin{align}
J  = \min \int_0^{t_f} \left( \phi(t)^2 + \alpha a(t)^2 + \beta v(t)^2\right)\,dt
\end{align}

## Pyomo Model

```python
from pyomo.environ import *
from pyomo.dae import *

L = 2
tf = 50

# create a model object
m = ConcreteModel()

# define the independent variable
m.time = ContinuousSet(bounds=(0, tf))

# define control inputs
m.a = Var(m.time)
m.v = Var(m.time, domain=Reals, bounds=(-0.1,0.1))

# define the dependent variables
m.x = Var(m.time)
m.y = Var(m.time)
m.t = Var(m.time)
m.u = Var(m.time)
m.p = Var(m.time, domain=Reals, bounds=(-0.5,0.5))

m.dxdt = DerivativeVar(m.x)
m.dydt = DerivativeVar(m.y)
m.dtdt = DerivativeVar(m.t)
m.dudt = DerivativeVar(m.u)
m.dpdt = DerivativeVar(m.p)

# define the differential equation as a constraint
m.ode_x = Constraint(m.time, rule=lambda m, time: m.dxdt[time] == m.u[time]*cos(m.t[time]))
m.ode_y = Constraint(m.time, rule=lambda m, time: m.dydt[time] == m.u[time]*sin(m.t[time]))
m.ode_t = Constraint(m.time, rule=lambda m, time: m.dtdt[time] == m.u[time]*tan(m.p[time])/L)
m.ode_u = Constraint(m.time, rule=lambda m, time: m.dudt[time] == m.a[time])
m.ode_p = Constraint(m.time, rule=lambda m, time: m.dpdt[time] == m.v[time])

# path constraints
m.path_x1 = Constraint(m.time, rule=lambda m, time: m.x[time] >= 0)
m.path_y1 = Constraint(m.time, rule=lambda m, time: m.y[time] >= 0)

# initial conditions
m.ic = ConstraintList()
m.ic.add(m.x[0]==0)
m.ic.add(m.y[0]==0)
m.ic.add(m.t[0]==0)
m.ic.add(m.u[0]==0)
m.ic.add(m.p[0]==0)

# final conditions
m.fc = ConstraintList()
m.fc.add(m.x[tf]==0)
m.fc.add(m.y[tf]==20)
m.fc.add(m.t[tf]==0)
m.fc.add(m.u[tf]==0)
m.fc.add(m.p[tf]==0)

# define the optimization objective
m.integral = Integral(m.time, wrt=m.time,
                      rule=lambda m, time: 0.2*m.p[time]**2 + m.a[time]**2 + m.v[time]**2)
m.obj = Objective(expr=m.integral)

# transform and solve
TransformationFactory('dae.collocation').apply_to(m, wrt=m.time, nfe=3, ncp=12, method='BACKWARD')
solver = SolverFactory('ipopt')
solver.options['linear_solver'] = 'MA57'
solver.solve(m, tee=True).write()
```

    Ipopt 3.12.10: linear_solver=MA57

    ******************************************************************************
    This program contains Ipopt, a library for large-scale nonlinear optimization.
     Ipopt is released as open source code under the Eclipse Public License (EPL).
             For more information visit http://projects.coin-or.org/Ipopt
    ******************************************************************************

    This is Ipopt version 3.12.10, running with linear solver ma57.

    Number of nonzeros in equality constraint Jacobian...:     3011
    Number of nonzeros in inequality constraint Jacobian.:       74
    Number of nonzeros in Lagrangian Hessian.............:      222

    Total number of variables............................:      444
                         variables with only lower bounds:        0
                    variables with lower and upper bounds:       74
                         variables with only upper bounds:        0
    Total number of equality constraints.................:      375
    Total number of inequality constraints...............:       74
            inequality constraints with only lower bounds:       74
       inequality constraints with lower and upper bounds:        0
            inequality constraints with only upper bounds:        0

    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
       0  0.0000000e+00 2.00e+01 1.00e+00  -1.0 0.00e+00    -  0.00e+00 0.00e+00   0
       1  2.3944631e-01 3.14e-01 1.70e+06  -1.0 1.99e+01    -  5.00e-04 9.90e-01f  1
       2  2.3944740e-01 3.14e-01 1.70e+06  -1.0 1.81e+04   6.0 5.47e-06 1.07e-08h 10
       3  2.3946389e-01 3.14e-01 1.70e+06  -1.0 1.81e+04   5.5 5.48e-06 4.27e-08h  8
       4  2.3957001e-01 3.14e-01 1.70e+06  -1.0 1.79e+04   5.0 5.85e-06 8.56e-08h  7
       5r 2.3957001e-01 3.14e-01 9.99e+02  -0.5 0.00e+00   5.5 0.00e+00 3.39e-07R  5
       6r 2.4107197e-01 3.17e-01 9.97e+02  -0.5 2.28e+02    -  8.24e-03 2.27e-03f  1
       7r 2.4810995e-01 1.85e+00 9.88e+02  -0.5 1.80e+02    -  2.63e-03 1.07e-02f  1
       8r 2.5615278e-01 2.64e+00 9.75e+02  -0.5 6.43e+01    -  1.94e-02 1.22e-02f  1
       9r 2.7422847e-01 2.86e+00 9.47e+02  -0.5 3.06e+01    -  5.74e-02 2.79e-02f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
      10r 3.1764375e-01 2.85e+00 8.82e+02  -0.5 5.08e+00    -  6.87e-01 6.94e-02f  1
      11r 7.5452012e-01 3.00e+00 2.41e+02  -0.5 2.41e+00    -  1.00e+00 7.27e-01f  1
      12r 7.5887866e-01 3.00e+00 7.50e+00  -0.5 3.50e-02   2.0 9.65e-01 1.00e+00f  1
      13r 7.8728499e-01 3.03e+00 7.17e+00  -1.2 5.74e-02   1.5 8.15e-01 9.05e-01f  1
      14r 8.2720582e-01 3.03e+00 3.13e+00  -1.2 3.71e-02   1.9 1.00e+00 1.00e+00f  1
      15r 8.4724902e-01 3.04e+00 3.40e+00  -1.9 1.46e-02   2.4 1.00e+00 1.00e+00f  1
      16r 8.6660180e-01 3.04e+00 5.53e+01  -1.9 5.27e-02   1.9 5.21e-01 3.21e-01f  1
      17r 8.8510940e-01 3.04e+00 3.59e+00  -1.9 1.71e-02   2.3 1.00e+00 9.62e-01f  1
      18r 9.5689371e-01 3.05e+00 2.05e+01  -1.9 7.73e-02   1.8 6.49e-01 8.48e-01f  1
      19r 9.5993600e-01 3.05e+00 2.60e+01  -1.9 2.77e+00   1.4 1.52e-02 1.80e-03f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
      20r 1.0046251e+00 3.05e+00 8.19e+01  -1.9 1.01e-01   1.8 1.00e+00 6.15e-01f  1
      21r 1.0194045e+00 3.05e+00 1.51e+02  -1.9 1.57e+00   1.3 8.01e-02 1.17e-02f  1
      22r 1.0825627e+00 3.05e+00 6.07e+01  -1.9 1.39e-01   1.7 1.00e+00 5.08e-01f  1
      23r 1.0865506e+00 3.05e+00 7.72e+01  -1.9 4.31e-02   2.2 6.24e-01 1.51e-01f  1
      24r 1.2649305e+00 3.05e+00 1.47e+01  -1.9 1.64e-01   1.7 8.31e-01 9.76e-01f  1
      25r 1.3861553e+00 3.05e+00 9.98e+00  -1.9 7.99e-02   2.1 8.26e-01 7.84e-01f  1
      26r 1.5217519e+00 3.05e+00 1.16e+02  -1.9 8.00e-01   1.6 3.38e-01 6.48e-02f  1
      27r 1.6713449e+00 3.05e+00 5.94e+01  -1.9 1.16e-01   2.1 1.00e+00 4.23e-01f  1
      28r 1.7126657e+00 3.05e+00 2.46e+01  -1.9 4.26e-02   2.5 1.00e+00 3.74e-01f  1
      29r 1.7720112e+00 3.05e+00 9.02e+01  -1.9 1.52e-01   2.0 8.73e-01 1.52e-01f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
      30r 1.8910752e+00 3.05e+00 1.31e+01  -1.9 4.73e-02   2.4 1.00e+00 1.00e+00f  1
      31r 2.0314643e+00 3.05e+00 9.71e+01  -1.9 1.83e-01   2.0 1.00e+00 2.71e-01f  1
      32r 2.0624158e+00 3.05e+00 6.35e+01  -1.9 5.83e-02   2.4 3.18e-01 1.78e-01f  1
      33r 2.2521504e+00 3.05e+00 6.49e+01  -1.9 2.45e-01   1.9 4.47e-01 2.33e-01f  1
      34r 2.3947446e+00 3.05e+00 2.93e+01  -1.9 7.60e-02   2.3 4.77e-01 5.87e-01f  1
      35r 2.7675424e+00 3.05e+00 3.57e+01  -1.9 3.16e-01   1.9 3.76e-01 3.22e-01f  1
      36r 2.9279502e+00 3.05e+00 5.97e+01  -1.9 1.11e-01   2.3 1.00e+00 3.64e-01f  1
      37r 3.0498793e+00 3.05e+00 4.15e+01  -1.9 3.99e-02   2.7 1.00e+00 7.57e-01f  1
      38r 3.1611925e+00 3.05e+00 1.88e+02  -1.9 1.49e-01   2.2 1.00e+00 1.89e-01f  1
      39r 3.3722800e+00 3.05e+00 2.43e+01  -1.9 5.27e-02   2.7 1.00e+00 1.00e+00f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
      40r 3.5376710e+00 3.05e+00 1.45e+02  -1.9 1.90e-01   2.2 1.00e+00 2.10e-01f  1
      41r 3.8133908e+00 3.05e+00 2.69e+01  -1.9 6.57e-02   2.6 1.00e+00 1.00e+00f  1
      42r 4.0978198e+00 3.05e+00 1.88e+02  -1.9 2.42e-01   2.1 1.00e+00 2.66e-01f  1
      43r 4.4143536e+00 3.05e+00 3.00e+01  -1.9 8.29e-02   2.6 1.00e+00 8.49e-01f  1
      44r 4.7007309e+00 3.05e+00 1.13e+02  -1.9 3.08e-01   2.1 5.35e-01 1.98e-01f  1
      45r 4.7912801e+00 3.05e+00 5.89e+01  -1.9 1.03e-01   2.5 1.00e+00 1.86e-01f  1
      46r 5.7813782e+00 3.04e+00 5.17e+01  -1.9 3.64e-01   2.0 2.80e-01 5.40e-01f  1
      47r 6.2653737e+00 3.04e+00 5.56e+01  -1.9 1.30e-01   2.5 1.00e+00 6.86e-01f  1
      48r 6.5241911e+00 3.04e+00 4.84e+01  -1.9 5.65e-01   2.0 8.64e-02 9.21e-02f  1
      49r 6.7481247e+00 3.04e+00 7.05e+01  -1.9 1.75e-01   2.4 6.29e-01 2.60e-01f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
      50r 6.7778153e+00 3.04e+00 1.02e+02  -1.9 7.54e-01   1.9 6.51e-02 9.25e-03f  1
      51r 7.1899470e+00 3.04e+00 6.95e+01  -1.9 2.07e-01   2.4 5.23e-01 4.07e-01f  1
      52r 7.5401885e+00 3.03e+00 4.56e+01  -1.9 7.55e-02   2.8 1.00e+00 9.06e-01f  1
      53r 7.9588459e+00 3.03e+00 4.92e+01  -1.9 2.84e-01   2.3 2.18e-01 3.06e-01f  1
      54r 8.4684734e+00 3.02e+00 5.42e+01  -1.9 1.00e-01   2.7 8.20e-01 1.00e+00f  1
      55r 9.6075613e+00 3.00e+00 3.31e+02  -1.9 4.94e-01   2.3 9.88e-02 4.96e-01f  1
      56r 1.0033493e+01 2.99e+00 7.01e+02  -1.9 2.45e-01   2.7 3.06e-02 2.87e-01f  1
      57r 1.0529050e+01 2.98e+00 4.93e+02  -1.9 9.44e-02   3.1 1.75e-01 1.00e+00f  1
      58r 1.0786193e+01 2.98e+00 4.87e+02  -1.9 4.38e+00   2.6 7.48e-03 2.01e-02f  1
      59r 1.1137189e+01 2.98e+00 2.98e+02  -1.9 2.29e-01   3.1 7.58e-01 3.69e-01f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
      60r 1.1140408e+01 2.98e+00 2.98e+02  -1.9 8.56e+00   2.6 1.05e-03 1.08e-04f  1
      61r 1.1470039e+01 2.97e+00 1.71e+02  -1.9 2.32e-01   3.0 4.84e-01 4.24e-01f  1
      62r 1.1993402e+01 2.96e+00 2.19e+02  -1.9 2.18e+00   2.5 7.17e-02 1.04e-01f  1
      63r 1.2000689e+01 2.96e+00 2.19e+02  -1.9 3.48e-01   3.0 2.67e-01 6.34e-03f  1
      64r 1.2001008e+01 2.96e+00 2.18e+02  -1.9 7.71e-01   2.5 4.62e-03 1.10e-02f  1
      65r 1.2173444e+01 2.96e+00 3.33e+02  -1.9 1.66e-01   2.9 4.35e-02 3.40e-01f  1
      66r 1.2176191e+01 2.96e+00 3.07e+02  -1.9 6.44e-02   3.3 6.17e-01 1.17e-02f  1
      67r 1.2230847e+01 2.95e+00 1.58e+02  -1.9 2.09e-01   2.9 1.04e-02 1.26e-01f  1
      68r 1.2291329e+01 2.95e+00 3.73e+02  -1.9 7.26e-02   3.3 1.77e-01 2.93e-01f  1
      69r 1.2299721e+01 2.95e+00 3.65e+02  -1.9 2.68e-01   2.8 2.26e-02 1.89e-02f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
      70r 1.2519464e+01 2.94e+00 1.64e+02  -1.9 1.10e-01   3.2 1.00e+00 7.79e-01f  1
      71r 1.2521387e+01 2.94e+00 1.30e+02  -1.9 3.09e-01   2.7 2.73e-01 2.78e-03f  1
      72r 1.2667769e+01 2.93e+00 1.05e+02  -1.9 6.94e-01   2.3 1.06e-01 4.70e-02f  1
      73r 1.2683551e+01 2.93e+00 4.21e+02  -1.9 1.54e-01   2.7 1.00e+00 1.64e-02f  1
      74r 1.2895987e+01 2.92e+00 3.86e+02  -1.9 5.76e-01   2.2 1.40e-01 8.69e-02f  1
      75r 1.3442459e+01 2.90e+00 1.98e+02  -1.9 1.79e-01   2.6 1.00e+00 5.10e-01f  1
      76r 1.3461504e+01 2.90e+00 2.03e+02  -1.9 7.41e-01   2.2 9.19e-02 8.76e-03f  1
      77r 1.3674754e+01 2.88e+00 1.56e+02  -1.9 2.19e-01   2.6 9.17e-02 2.18e-01f  1
      78r 1.3968591e+01 2.85e+00 1.31e+02  -1.9 9.20e-01   2.1 5.39e-02 1.17e-01f  1
      79r 1.3997769e+01 2.85e+00 1.60e+02  -1.9 2.56e-01   2.5 9.27e-01 2.50e-02f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
      80r 1.4283667e+01 2.82e+00 1.55e+02  -1.9 9.87e-01   2.1 2.22e-01 1.16e-01f  1
      81r 1.5287094e+01 2.75e+00 9.37e+01  -1.9 3.07e-01   2.5 1.00e+00 7.66e-01f  1
      82r 1.5424386e+01 2.73e+00 9.59e+01  -1.9 1.33e+00   2.0 1.25e-01 5.07e-02f  1
      83r 1.5589962e+01 2.72e+00 1.53e+02  -1.9 3.95e-01   2.4 9.10e-01 1.17e-01f  1
      84r 1.5798251e+01 2.68e+00 2.07e+02  -1.9 1.53e+00   2.0 1.69e-01 8.31e-02f  1
      85r 1.5864673e+01 2.68e+00 2.89e+02  -1.9 4.58e-01   2.4 3.02e-01 4.39e-02f  1
      86r 1.5871499e+01 2.67e+00 5.33e+02  -1.9 1.43e-01   2.8 5.58e-01 1.47e-02f  1
      87r 1.6332858e+01 2.63e+00 1.51e+02  -1.9 4.27e-01   2.3 1.04e-02 3.64e-01f  1
      88r 1.6764829e+01 2.59e+00 2.27e+02  -1.9 1.71e-01   2.8 1.03e-01 7.70e-01f  1
      89r 1.6858590e+01 2.59e+00 1.78e+02  -1.9 5.58e-01   2.3 1.45e-01 5.50e-02f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
      90r 1.6885729e+01 2.58e+00 1.00e+02  -1.9 1.97e-01   2.7 6.93e-01 4.30e-02f  1
      91r 1.7028327e+01 2.57e+00 1.00e+02  -1.9 5.90e-01   2.2 1.86e-02 7.64e-02f  1
      92r 1.7220396e+01 2.55e+00 3.54e+02  -1.9 2.09e-01   2.7 6.66e-01 2.98e-01f  1
      93r 1.7880555e+01 2.49e+00 1.64e+02  -1.9 6.74e-01   2.2 1.25e-01 3.01e-01f  1
      94r 1.8011846e+01 2.48e+00 1.48e+02  -1.9 2.51e-01   2.6 2.44e-01 1.62e-01f  1
      95r 1.8026497e+01 2.48e+00 1.46e+02  -1.9 8.08e-01   2.1 2.52e-02 8.12e-03f  1
      96r 1.8061637e+01 2.48e+00 1.39e+02  -1.9 2.88e-01   2.6 1.80e-01 4.53e-02f  1
      97r 1.8229229e+01 2.43e+00 1.56e+02  -1.9 8.68e-01   2.1 1.85e-02 1.93e-01f  1
      98r 1.8382445e+01 2.43e+00 1.28e+02  -1.9 3.36e-01   2.5 1.41e-01 1.90e-01f  1
      99r 1.8383071e+01 2.43e+00 1.20e+02  -1.9 1.10e+00   2.0 2.75e-02 2.99e-03f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     100r 1.8417316e+01 2.43e+00 1.12e+02  -1.9 3.68e-01   2.5 3.67e-01 8.44e-02f  1
     101r 1.8385445e+01 2.43e+00 1.06e+02  -1.9 1.43e+00   2.0 2.20e-02 7.74e-02f  1
     102r 1.8543858e+01 2.43e+00 1.08e+02  -1.9 4.29e-01   2.4 3.40e-01 4.57e-01f  1
     103r 1.8594229e+01 2.43e+00 1.08e+02  -1.9 1.63e-01   2.8 7.43e-01 2.12e-01f  1
     104r 1.8789845e+01 2.43e+00 1.11e+02  -1.9 5.15e-01   2.4 1.11e-01 2.85e-01f  1
     105r 1.8790260e+01 2.43e+00 1.11e+02  -1.9 3.64e+00   1.9 1.65e-02 2.38e-04f  1
     106r 1.9058189e+01 2.43e+00 1.13e+02  -1.9 5.80e-01   2.3 6.58e-01 3.24e-01f  1
     107r 1.9114799e+01 2.43e+00 1.13e+02  -1.9 2.07e+00   1.8 2.30e-03 2.10e-02f  1
     108r 1.9467546e+01 2.43e+00 1.16e+02  -1.9 6.76e-01   2.3 2.57e-01 3.47e-01f  1
     109r 1.9729927e+01 2.43e+00 1.18e+02  -1.9 2.39e+00   1.8 1.91e-02 6.68e-02f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     110r 1.9738007e+01 2.43e+00 1.72e+02  -1.9 7.88e-01   2.2 2.21e-01 7.23e-03f  1
     111r 2.0188345e+01 2.43e+00 1.26e+02  -1.9 2.85e+00   1.7 1.90e-02 9.58e-02f  1
     112r 2.0278748e+01 2.43e+00 2.19e+02  -1.9 9.22e-01   2.2 2.79e-01 6.90e-02f  1
     113r 2.0332047e+01 2.43e+00 3.07e+02  -1.9 3.31e-01   2.6 6.07e-01 1.24e-01f  1
     114r 2.1064536e+01 2.43e+00 1.28e+02  -1.9 1.05e+00   2.1 1.09e-01 5.45e-01f  1
     115r 2.1076998e+01 2.43e+00 1.28e+02  -1.9 3.94e-01   2.5 5.70e-01 2.26e-02f  1
     116r 2.1091284e+01 2.42e+00 1.30e+02  -1.9 1.65e+00   2.0 1.04e-03 3.18e-02f  1
     117r 2.1645070e+01 2.41e+00 1.38e+02  -1.9 4.61e-01   2.5 1.00e+00 1.00e+00f  1
     118r 2.1853550e+01 2.41e+00 1.40e+02  -1.9 1.76e-01   2.9 8.97e-01 9.89e-01f  1
     119r 2.1929536e+01 2.41e+00 1.45e+02  -1.9 5.99e-01   2.4 3.30e-01 2.67e-01f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     120r 2.2011057e+01 2.40e+00 1.48e+02  -1.9 2.14e-01   2.9 1.72e-01 4.65e-01f  1
     121r 2.1995724e+01 2.40e+00 1.51e+02  -1.9 1.93e+00   2.4 6.89e-02 6.04e-03f  1
     122r 2.2043377e+01 2.40e+00 1.53e+02  -1.9 2.52e-01   2.8 6.61e-02 3.93e-01f  1
     123r 2.2045208e+01 2.40e+00 1.53e+02  -1.9 9.07e-02   3.2 1.00e+00 2.21e-02f  1
     124r 2.2012821e+01 2.39e+00 1.58e+02  -1.9 3.28e-01   2.7 5.25e-02 4.98e-01f  1
     125r 2.2013085e+01 2.39e+00 1.58e+02  -1.9 1.36e-01   3.2 4.72e-01 2.09e-02f  1
     126r 2.1936777e+01 2.39e+00 1.60e+02  -1.9 8.82e-01   2.7 3.52e-03 5.88e-02f  1
     127r 2.1936191e+01 2.39e+00 1.60e+02  -1.9 1.72e-01   3.1 2.62e-01 9.70e-03f  1
     128r 2.1769055e+01 2.38e+00 1.83e+02  -1.9 1.12e+00   2.6 4.35e-02 8.89e-02f  1
     129r 2.1768437e+01 2.38e+00 3.63e+02  -1.9 2.28e-01   3.1 1.00e+00 5.39e-03f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     130r 2.1769609e+01 2.38e+00 5.52e+02  -1.9 4.39e-01   2.6 3.35e-01 2.45e-02f  1
     131r 2.1639326e+01 2.38e+00 3.53e+02  -1.9 1.81e+00   2.1 7.56e-04 4.90e-02f  1
     132r 2.1628215e+01 2.38e+00 3.16e+02  -1.9 6.10e-01   2.5 5.51e-02 6.88e-02f  1
     133r 2.1631226e+01 2.38e+00 2.53e+02  -1.9 2.16e-01   3.0 1.45e-01 1.66e-01f  1
     134r 2.1571901e+01 2.37e+00 3.55e+02  -1.9 9.18e-01   2.5 6.82e-03 2.59e-01f  1
     135r 2.1576184e+01 2.37e+00 3.02e+02  -1.9 3.00e-01   2.9 1.46e-01 1.41e-01f  1
     136r 2.1573400e+01 2.37e+00 3.00e+02  -1.9 1.48e+00   2.4 2.63e-03 1.93e-03f  1
     137r 2.1500100e+01 2.36e+00 2.50e+02  -1.9 4.00e-01   2.9 2.33e-01 3.00e-01f  1
     138r 2.1195455e+01 2.35e+00 4.21e+02  -1.9 2.09e+00   2.4 1.33e-03 1.43e-01f  1
     139r 2.1194076e+01 2.35e+00 3.32e+02  -1.9 5.72e-01   2.8 2.93e-01 5.22e-03f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     140r 2.1197582e+01 2.35e+00 4.00e+02  -1.9 1.67e-01   2.3 4.45e-01 5.32e-03f  1
     141r 2.1242747e+01 2.34e+00 3.91e+02  -1.9 8.96e-01   1.9 4.27e-02 2.95e-02f  1
     142r 2.1435316e+01 2.33e+00 4.52e+02  -1.9 2.19e-01   2.3 8.79e-01 2.58e-01f  1
     143r 2.1440677e+01 2.32e+00 4.55e+02  -1.9 1.75e+00   1.8 1.87e-02 3.71e-03f  1
     144r 2.1715426e+01 2.29e+00 2.66e+02  -1.9 2.28e-01   2.2 2.22e-02 3.85e-01f  1
     145r 2.1830875e+01 2.26e+00 2.59e+02  -1.9 1.97e+01   1.8 7.43e-04 4.88e-03f  1
     146r 2.1838825e+01 2.26e+00 2.75e+02  -1.9 2.89e-01   2.2 3.42e-01 7.72e-03f  1
     147r 2.1947904e+01 2.25e+00 2.47e+02  -1.9 1.06e-01   2.6 8.79e-01 2.88e-01f  1
     148r 2.2105525e+01 2.23e+00 2.17e+02  -1.9 3.98e-01   2.1 1.32e-01 1.26e-01f  1
     149r 2.2134956e+01 2.22e+00 3.02e+02  -1.9 1.38e-01   2.6 1.00e+00 6.83e-02f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     150r 2.2334035e+01 2.19e+00 2.63e+02  -1.9 5.78e-01   2.1 5.09e-02 1.56e-01f  1
     151r 2.2337543e+01 2.19e+00 2.12e+02  -1.9 1.73e-01   2.5 1.00e+00 7.30e-03f  1
     152r 2.2581253e+01 2.16e+00 1.59e+02  -1.9 4.31e-01   2.0 1.46e-01 2.07e-01f  1
     153r 2.2752927e+01 2.14e+00 2.64e+02  -1.9 1.51e-01   2.5 6.54e-01 3.63e-01f  1
     154r 2.3205495e+01 2.07e+00 6.95e+01  -1.9 6.05e-01   2.0 1.20e-02 3.08e-01f  1
     155r 2.3219945e+01 2.06e+00 1.07e+02  -1.9 2.19e-01   2.4 2.28e-01 2.48e-02f  1
     156r 2.3421671e+01 2.01e+00 1.07e+02  -1.9 1.25e+00   1.9 1.49e-03 1.04e-01f  1
     157r 2.4199799e+01 1.88e+00 1.69e+02  -1.9 3.38e-01   2.4 3.44e-01 1.00e+00f  1
     158r 2.4241509e+01 1.86e+00 1.66e+02  -1.9 5.67e+00   1.9 8.51e-03 8.24e-03f  1
     159r 2.4773601e+01 1.73e+00 2.25e+02  -1.9 5.21e-01   2.3 1.14e-01 5.55e-01f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     160r 2.5145959e+01 1.65e+00 1.05e+02  -1.9 1.97e-01   2.7 7.04e-01 9.49e-01f  1
     161r 2.6211664e+01 1.35e+00 2.30e+02  -1.9 7.19e-01   2.3 1.31e-01 8.62e-01f  1
     162r 2.6745008e+01 1.20e+00 1.33e+02  -1.9 2.79e-01   2.7 1.00e+00 1.00e+00f  1
     163r 2.7198606e+01 1.05e+00 1.36e+02  -1.9 9.05e-01   2.2 1.48e-01 2.98e-01f  1
     164r 2.7277269e+01 1.03e+00 1.37e+02  -1.9 3.28e-01   2.6 4.12e-01 1.28e-01f  1
     165r 2.7284835e+01 1.02e+00 1.37e+02  -1.9 1.00e+00   2.2 2.08e-03 6.43e-03f  1
     166r 2.7465903e+01 9.38e-01 1.35e+02  -1.9 3.43e-01   2.6 1.33e-01 2.72e-01f  1
     167r 2.7294620e+01 8.67e-01 1.36e+02  -1.9 6.01e+00   2.1 2.75e-03 1.69e-02f  1
     168r 2.7300145e+01 8.63e-01 1.36e+02  -1.9 4.07e-01   2.5 3.26e-01 7.96e-03f  1
     169r 2.7294898e+01 8.56e-01 1.36e+02  -1.9 2.21e+00   2.0 3.49e-02 3.48e-03f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     170r 2.7415792e+01 7.80e-01 1.35e+02  -1.9 4.60e-01   2.5 8.50e-03 1.65e-01f  1
     171r 2.7490404e+01 7.42e-01 1.34e+02  -1.9 1.64e-01   2.9 1.00e+00 2.30e-01f  1
     172r 2.8076067e+01 4.16e-01 1.35e+02  -1.9 5.61e-01   2.4 9.86e-02 5.81e-01f  1
     173r 2.8084890e+01 4.12e-01 1.33e+02  -1.9 2.09e-01   2.8 2.16e-01 1.83e-02f  1
     174r 2.8999650e+01 3.32e-01 3.31e+02  -1.9 6.79e-01   2.4 1.63e-02 6.01e-01f  1
     175r 2.9009728e+01 3.31e-01 1.47e+02  -1.9 2.74e-01   2.8 5.05e-01 1.49e-02f  1
     176r 2.9017103e+01 3.31e-01 1.90e+02  -1.9 6.56e-01   2.3 4.24e-01 8.21e-03f  1
     177r 2.9103410e+01 3.36e-01 1.69e+02  -1.9 2.30e+00   1.8 2.40e-03 3.36e-02f  1
     178r 2.9111063e+01 3.36e-01 2.93e+02  -1.9 6.15e-01   2.3 4.12e-01 8.44e-03f  1
     179r 2.9119692e+01 3.35e-01 3.47e+02  -1.9 7.83e-01   1.8 2.74e-01 4.51e-03f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     180r 2.9248334e+01 3.24e-01 3.25e+02  -1.9 1.42e-01   2.2 4.17e-01 2.39e-01f  1
     181r 2.9339205e+01 3.17e-01 3.84e+02  -1.9 5.76e-01   1.7 5.00e-01 5.19e-02f  1
     182r 2.9857064e+01 2.77e-01 1.18e+02  -1.9 1.63e-01   2.2 1.00e+00 8.12e-01f  1
     183  2.9536957e+01 2.61e-01 1.25e+01  -1.0 7.73e+00    -  4.93e-02 5.70e-02f  1
     184  2.4784538e+01 2.26e-01 4.15e+01  -1.0 2.34e+00    -  1.47e-01 1.35e-01f  1
     185  2.0960902e+01 1.90e-01 4.48e+02  -1.0 1.85e+00    -  2.51e-01 1.57e-01f  1
     186  1.3880219e+01 1.21e-01 1.57e+02  -1.0 2.38e+00    -  2.80e-01 3.64e-01f  1
     187  9.3255736e+00 1.15e-01 1.85e+03  -1.0 4.18e+00    -  1.21e-01 3.32e-01f  1
     188  5.4145707e+00 1.25e-01 2.89e+03  -1.0 2.88e+00    -  3.93e-01 5.61e-01f  1
     189  3.6426624e+00 1.57e-01 5.62e+03  -1.0 2.27e+00    -  4.21e-01 5.78e-01f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     190  3.2307772e+00 2.30e-01 1.81e+04  -1.0 4.68e+00    -  1.63e-01 3.86e-01f  1
     191  3.0902394e+00 1.55e-01 6.71e+04  -1.0 3.25e+00    -  1.54e-01 7.36e-01f  1
     192  2.9397007e+00 5.23e-02 6.59e+04  -1.0 1.93e+00    -  6.44e-01 7.73e-01f  1
     193  3.1497024e+00 1.01e-01 7.47e+04  -1.0 1.40e+01    -  2.72e-01 2.91e-01f  1
     194  3.2405928e+00 9.50e-02 6.41e+04  -1.0 1.11e+01    -  4.10e-01 3.50e-01f  1
     195  3.2427972e+00 7.10e-02 7.61e+04  -1.0 8.06e-02   5.0 2.00e-01 2.53e-01h  1
     196  3.2428487e+00 7.06e-02 2.10e+06  -1.0 5.91e-02   4.5 8.02e-01 5.40e-03h  1
     197  3.2658106e+00 2.12e-03 6.37e+02  -1.0 5.90e-02   4.0 1.00e+00 1.00e+00f  1
     198  3.2662774e+00 6.77e-05 9.64e+00  -1.7 2.63e-03   3.6 1.00e+00 1.00e+00h  1
     199  3.2652460e+00 9.40e-07 3.90e+02  -3.8 2.51e-03   3.1 7.65e-01 1.00e+00h  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     200  3.2648669e+00 2.32e-07 2.05e-01  -3.8 5.03e-04   2.6 1.00e+00 1.00e+00h  1
     201  2.7252592e+00 2.18e-02 1.86e-01  -5.7 1.85e+01    -  8.77e-02 9.30e-02f  1
     202  2.7120246e+00 2.52e-04 1.71e+02  -5.7 2.86e-02   2.1 1.52e-03 1.00e+00h  1
     203  2.7088596e+00 6.73e-07 1.19e+00  -5.7 1.80e-03   1.7 9.62e-01 1.00e+00h  1
     204  2.6993179e+00 1.51e-05 2.87e+00  -5.7 1.36e+01    -  2.12e-01 2.47e-03f  1
     205  2.0334704e+00 4.74e-02 6.03e+00  -5.7 1.15e+01    -  5.86e-01 2.66e-01f  1
     206  2.0275664e+00 4.41e-02 8.21e+00  -5.7 7.04e-02   1.2 4.65e-01 6.94e-02h  1
     207  1.9552605e+00 1.45e-03 1.55e+00  -5.7 6.50e-02   0.7 7.20e-01 1.00e+00h  1
     208  1.9274463e+00 6.67e-05 8.03e+00  -5.7 2.84e-02   0.2 4.68e-01 1.00e+00h  1
     209  1.8800274e+00 1.69e-04 3.81e-02  -5.7 6.84e-02  -0.3 1.00e+00 1.00e+00f  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     210  1.8387412e+00 2.63e-04 6.38e+00  -5.7 1.64e-01  -0.7 1.00e+00 5.32e-01f  1
     211  1.7047944e+00 2.69e-03 2.57e-02  -5.7 4.15e-01  -1.2 1.00e+00 1.00e+00f  1
     212  1.6560050e+00 2.28e-03 1.02e+01  -5.7 7.99e-01  -1.7 1.00e+00 2.63e-01h  1
     213  1.5754300e+00 2.42e-03 1.05e+01  -5.7 1.26e+00  -2.2 1.00e+00 3.05e-01f  1
     214  1.4871266e+00 6.78e-03 9.78e+00  -5.7 1.71e+00  -2.6 1.00e+00 2.82e-01f  1
     215  1.3593439e+00 5.60e-02 4.49e+00  -5.7 2.71e+00  -3.1 2.07e-01 4.98e-01f  1
     216  1.2724754e+00 3.92e-02 3.32e+00  -5.7 4.01e+00  -3.6 2.95e-02 5.44e-01h  1
     217  1.2646625e+00 3.79e-02 3.75e+00  -5.7 2.56e+00  -3.2 1.51e-01 4.12e-02h  1
     218  1.2563356e+00 6.29e-02 4.29e+00  -5.7 1.53e+01  -3.6 8.89e-02 7.62e-02h  1
     219  1.2513169e+00 6.23e-02 3.90e+00  -5.7 8.93e+00  -4.1 6.59e-03 1.42e-02h  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     220  1.2388335e+00 7.15e-02 9.71e+00  -5.7 4.11e+00  -3.7 3.03e-02 2.78e-01f  2
     221  1.2156177e+00 7.01e-02 9.49e+00  -5.7 4.78e+00  -3.3 2.76e-02 9.33e-02h  1
     222  1.1844158e+00 2.97e-02 2.32e+01  -5.7 1.21e+00  -3.7 4.42e-02 6.25e-01f  1
     223  1.1778575e+00 2.97e-02 2.36e+01  -5.7 1.48e+01  -3.3 5.04e-06 7.88e-03f  1
     224  1.1737085e+00 2.96e-02 2.51e+01  -5.7 2.43e+01  -2.0 2.40e-03 5.12e-03f  3
     225  1.1322622e+00 2.09e-02 1.64e+01  -5.7 9.96e-01  -2.5 3.11e-01 8.12e-01h  1
     226  1.1191471e+00 1.64e-02 2.38e+01  -5.7 8.53e-01  -2.9 1.00e+00 2.07e-01h  1
     227  1.1048699e+00 2.00e-02 1.59e+01  -5.7 2.87e+00  -3.4 6.32e-01 2.78e-01h  1
     228  1.1008121e+00 2.02e-02 1.59e+01  -5.7 2.32e+01  -3.9 9.67e-02 1.27e-02h  1
     229  9.9183899e-01 1.50e-01 1.77e+01  -5.7 3.35e+00  -3.5 2.09e-02 1.00e+00H  1
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     230  1.0014766e+00 1.21e-01 1.01e+01  -5.7 2.50e-01  -1.2 9.49e-01 1.92e-01h  1
     231  1.0018598e+00 1.17e-01 9.65e+00  -5.7 1.66e+00    -  6.76e-04 3.23e-02h  1
     232  1.0530709e+00 8.01e-03 1.61e+00  -5.7 4.10e-01  -1.7 6.52e-01 1.00e+00h  1
     233  1.0409443e+00 4.83e-03 8.42e-03  -5.7 3.26e-01  -2.2 1.00e+00 1.00e+00h  1
     234  1.0296045e+00 6.00e-03 6.47e+00  -5.7 5.00e-01  -2.7 5.23e-01 1.00e+00h  1
     235  1.0284511e+00 5.78e-03 1.19e+01  -5.7 1.40e+00  -3.2 9.47e-01 4.23e-02h  1
     236  9.9103227e-01 2.85e-02 1.41e+01  -5.7 3.75e+00    -  7.16e-02 1.00e+00F  1
     237  9.8654556e-01 2.34e-02 1.36e+01  -5.7 2.49e+00  -2.7 1.08e-01 1.79e-01h  1
     238  9.8775477e-01 1.07e-02 2.07e+01  -5.7 1.32e+00    -  2.36e-01 1.00e+00h  1
     239  9.8390913e-01 1.25e-02 1.86e+01  -5.7 4.41e+00    -  1.05e-01 1.10e-01h  3
    iter    objective    inf_pr   inf_du lg(mu)  ||d||  lg(rg) alpha_du alpha_pr  ls
     240  9.7112372e-01 2.10e-03 9.88e+00  -5.7 5.46e-01  -3.2 6.93e-01 1.00e+00h  1
     241  9.6632776e-01 2.86e-03 6.17e-04  -5.7 7.36e-01  -3.7 1.00e+00 1.00e+00h  1
     242  9.6330343e-01 1.13e-03 2.92e-04  -5.7 5.23e-01  -4.2 1.00e+00 1.00e+00h  1
     243  9.6243929e-01 6.99e-05 2.09e-05  -5.7 1.46e-01  -4.6 1.00e+00 1.00e+00h  1
     244  9.6241181e-01 6.93e-07 1.77e-07  -5.7 1.53e-02  -5.1 1.00e+00 1.00e+00h  1
     245  9.6240889e-01 4.24e-08 7.97e+00  -8.6 1.91e-03  -5.6 8.82e-01 9.94e-01h  1
     246  9.6240843e-01 5.00e-09 1.18e-08  -8.6 3.01e-05  -5.2 1.00e+00 1.00e+00f  1

    Number of Iterations....: 246

                                       (scaled)                 (unscaled)
    Objective...............:   9.6240842970768548e-01    9.6240842970768548e-01
    Dual infeasibility......:   1.1835675377269705e-08    1.1835675377269705e-08
    Constraint violation....:   5.0000000828016719e-09    5.0000000828016719e-09
    Complementarity.........:   1.4841224854874622e-08    1.4841224854874622e-08
    Overall NLP error.......:   5.0000000828016719e-09    1.4841224854874622e-08

    Number of objective function evaluations             = 290
    Number of objective gradient evaluations             = 71
    Number of equality constraint evaluations            = 290
    Number of inequality constraint evaluations          = 290
    Number of equality constraint Jacobian evaluations   = 248
    Number of inequality constraint Jacobian evaluations = 248
    Number of Lagrangian Hessian evaluations             = 246
    Total CPU secs in IPOPT (w/o function evaluations)   =      0.417
    Total CPU secs in NLP function evaluations           =      0.022

    EXIT: Optimal Solution Found.
    # ==========================================================
    # = Solver Results                                         =
    # ==========================================================
    # ----------------------------------------------------------
    #   Problem Information
    # ----------------------------------------------------------
    Problem:
    - Lower bound: -inf
      Upper bound: inf
      Number of objectives: 1
      Number of constraints: 449
      Number of variables: 444
      Sense: unknown
    # ----------------------------------------------------------
    #   Solver Information
    # ----------------------------------------------------------
    Solver:
    - Status: ok
      Message: Ipopt 3.12.10\x3a Optimal Solution Found
      Termination condition: optimal
      Id: 0
      Error rc: 0
      Time: 0.5478520393371582
    # ----------------------------------------------------------
    #   Solution Information
    # ----------------------------------------------------------
    Solution:
    - number of solutions: 0
      number of solutions displayed: 0

## Accessing Solution Data

```python
# access the results
time = [time for time in m.time]

a = [m.a[time]() for time in m.time]
v = [m.v[time]() for time in m.time]

x = [m.x[time]() for time in m.time]
y = [m.y[time]() for time in m.time]
t = [m.t[time]() for time in m.time]
u = [m.u[time]() for time in m.time]
p = [m.p[time]() for time in m.time]
```

## Visualizing Car Path

```python
% matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

scl=0.3

def draw_car(x=0, y=0, theta=0, phi=0):
    R = np.array([[np.cos(theta), -np.sin(theta)], [np.sin(theta), np.cos(theta)]])
    car = np.array([[0.2, 0.5], [-0.2, 0.5], [0, 0.5], [0, -0.5],
                    [0.2, -0.5], [-0.2, -0.5], [0, -0.5], [0, 0], [L, 0], [L, 0.5],
                    [L + 0.2*np.cos(phi), 0.5 + 0.2*np.sin(phi)],
                    [L - 0.2*np.cos(phi), 0.5 - 0.2*np.sin(phi)], [L, 0.5],[L, -0.5],
                    [L + 0.2*np.cos(phi), -0.5 + 0.2*np.sin(phi)],
                    [L - 0.2*np.cos(phi), -0.5 - 0.2*np.sin(phi)]])
    carz= scl*R.dot(car.T)
    plt.plot(x + carz[0], y + carz[1], 'k', lw=2)
    plt.plot(x, y, 'k.', ms=10)

plt.figure(figsize=(10,10))
for xs,ys,ts,ps in zip(x,y,t,p):
    draw_car(xs, ys, ts, scl*ps)
plt.plot(x, y, 'r--', lw=0.8)
plt.axis('square')
```

```python
plt.figure(figsize=(10,8))
plt.subplot(311)
plt.plot(time, a, time, v)
plt.legend(['Acceleration','Steering Input'])

plt.subplot(312)
plt.plot(time, p, time, t)
plt.legend(['Wheel Position','Car Direction'])

plt.subplot(313)
plt.plot(time, u)
plt.legend(['Velocity'])
```

```python

```

```python

```


# COURSE: Master math by coding in Python
## SECTION: Grahping conic sections

#### https://www.udemy.com/course/math-with-python/?couponCode=MXC-DISC4ALL
#### INSTRUCTOR: sincxpress.com

Note about this code: Each video in this section of the course corresponds to a section of code below. Please note that this code roughly matches the code shown in the live recording, but is not exactly the same -- the variable names, order of lines, and parameters may be slightly different.

```python

```

```python
import numpy as np
import sympy as sym
import matplotlib.pyplot as plt
```

# VIDEO: Graphing parabolas

```python
# parameters
n = 100
a = 1
h = 1
k = -2

# x-axis points to evaluate the function
x = np.linspace(-2,4,n)

# create the function
y = a*(x-h)**2 + k

# and plot it!
plt.plot(x,y)
plt.grid()
plt.axis('square')
plt.show()

```

```python
# parameters
n = 100
a = 1
h = 1
k = -2

# x-axis points to evaluate the function
x = np.linspace(-2,4,n)

# create the function
y = a*(x-h)**2 + k

# and plot it!
plt.plot(y,x)

plt.grid()
plt.axis('square')
plt.show()

```

### Exercise

```python
# draw the vertex, focus, and directrix

x = np.linspace(-6,6,40)
y = (x-2)**2/2 + 1

# 1) convert to standard parabola equation
# 4*p*(y-k) = (x-h)**2
# 2) write standard equation
h = 2
k = 1
p = 1/2 # 2=4p

# plot the parabola
plt.plot(x,y,label='Parabola')

# plot the vertex
plt.plot(h,k,'ro',label='Vertex')

# plot the focus
plt.plot(h,k+p,'go',label='focus')

# plot the directrix
d = k-p
plt.plot(x[[0,-1]],[d,d],label='directrix')

plt.legend()
plt.axis('square')
plt.axis([x[0],x[-1],d-.5,10])
plt.grid()
plt.show()
```

# VIDEO: Creating contours from meshes in Python

```python
X,Y = np.meshgrid(range(0,10),range(0,15))

plt.subplot(121)
plt.pcolormesh(X,edgecolors='k',linewidth=.1)
# plt.gca().set_aspect('equal')
plt.title('X')

plt.subplot(122)
plt.pcolormesh(Y,edgecolors='k',linewidth=.1)
plt.gca().set_aspect('equal')
plt.title('Y')

plt.show()
```

```python
x = np.linspace(0,2*np.pi,50)
y = np.linspace(0,4*np.pi,50)

X,Y = np.meshgrid(x,y)

F = np.cos(X) + np.sin(Y)
plt.imshow(F,extent=[x[0],x[-1],y[0],y[-1]])
plt.show()
```

### Exercise

```python
# Make a gaussian
x = np.linspace(-2,2,100)
s = 2

# create the gaussian
X,Y = np.meshgrid(x,x)
gauss2d = np.exp( -(X**2+Y**2)/s )

# and plot
plt.imshow(gauss2d,extent=[x[0],x[-1],x[0],x[-1]])
plt.axis('off')
plt.show()
```

# VIDEO: Graphing circles

```python
# circle parameters
a = 2
b = -3
r = 3

# grid space
axlim = r + np.max((abs(a),abs(b)))
x = np.linspace(-axlim,axlim,100)
y = np.linspace(-axlim,axlim,100)
X,Y = np.meshgrid(x,y)

# create the function
Fxy = (X-a)**2 + (Y-b)**2 - r**2

# draw it as a contour
plt.contour(X,Y,Fxy,0)

# draw a dot in the center
plt.plot(a,b,'go')

# draw guide lines
plt.plot([-axlim,axlim],[0,0],'k--')
plt.plot([0,0],[-axlim,axlim],'k--')

plt.gca().set_aspect('equal')
plt.show()
```

### Exercise

```python
# circle parameters
a = [-1.5,1.5]

# grid space
axlim = 5
x = np.linspace(-axlim,axlim,100)
y = np.linspace(-axlim,axlim,100)
X,Y = np.meshgrid(x,y)

for r in np.linspace(.5,3,15):
    for ai in a:
        Fxy = (X-ai)**2 + Y**2 - r**2
        plt.contour(X,Y,Fxy,0,colors=[(r/3,r/3,r/3)])

plt.gca().set_aspect('equal')
plt.plot(a,[0,0],'k',linewidth=3)
plt.axis('off')
plt.show()
```

# VIDEO: Graphing ellipses

```python
# parameters
a = 2
b = 3
h = 1
k = 2

# grid space
axlim = np.max((a,b)) + np.max((abs(h),abs(k)))
x = np.linspace(-axlim,axlim,100)
y = np.linspace(-axlim,axlim,100)
X,Y = np.meshgrid(x,y)

# create the function
Fxy = (X-h)**2/a**2 + (Y-k)**2/b**2 - 1

# draw it as a contour
plt.contour(X,Y,Fxy,0)

# draw a dot in the center
plt.plot(h,k,'go')

# draw guide lines
plt.plot([-axlim,axlim],[0,0],'--',color=[.8,.8,.8])
plt.plot([0,0],[-axlim,axlim],'k--',color=[.8,.8,.8]) # color overwrites k

plt.gca().set_aspect('equal')
plt.show()
```

### Exercise

```python
# parameters
n = 16
a = abs(np.linspace(4,-4,n))
b = 4
h = 0
k = np.linspace(-4,4,n)

# grid space
axlim = 8
x = np.linspace(-axlim,axlim,100)
y = np.linspace(-axlim,axlim,100)
X,Y = np.meshgrid(x,y)

for i in range(0,n):
    Fxy = (X-h)**2/a[i]**2 + (Y-k[i])**2/b**2 - 1
    plt.contour(X,Y,Fxy,0,colors=[(i/n,0,i/n)])

plt.gca().set_aspect('equal')
plt.axis('off')
plt.show()
```

# VIDEO: Graphing hyperbolas

```python
# parameters
a = 1
b = .5
h = 1
k = 2

# grid space
axlim = 2* (np.max((a,b)) + np.max((abs(h),abs(k))))
x = np.linspace(-axlim,axlim,100)
X,Y = np.meshgrid(x,x)

# create the function
Fxy = (X-h)**2/a**2 - (Y-k)**2/b**2 - 1

# draw it as a contour
plt.contour(X,Y,Fxy,0)

# draw a dot in the center
plt.plot(h,k,'go')

# draw guide lines
plt.plot([-axlim,axlim],[0,0],'--',color=[.8,.8,.8])
plt.plot([0,0],[-axlim,axlim],'k--',color=[.8,.8,.8]) # color overwrites k

plt.gca().set_aspect('equal')
plt.show()
```

### Exercise

```python
# parameters
n = 16
a = np.linspace(1,5,n)
b = np.linspace(1,5,n)

# grid space
axlim = 8
x = np.linspace(-axlim,axlim,100)
y = np.linspace(-axlim,axlim,100)
X,Y = np.meshgrid(x,y)

for i in range(0,n):
    Fxy = X**2/a[i]**2 - Y**2/b[i]**2 - 1
    plt.contour(X,Y,Fxy,0,colors=[(i/n,0,i/n)])

    Fxy = -X**2/a[i]**2 + Y**2/b[i]**2 - 1
    plt.contour(X,Y,Fxy,0,colors=[(0,i/n,i/n)])

plt.gca().set_aspect('equal')
plt.axis('off')
plt.show()
```

# VIDEO: Conics BUG HUNT!

```python
# Make a gaussian
x = np.linspace(-2,2,100)

# create the gaussian
X,Y = np.meshgrid(x,x)
gauss2d = np.exp( -(X**2+Y**2) )

# and plot
plt.imshow(gauss2d)
plt.axis('off')
plt.show()
```

```python
# draw a circle using meshgrid
r = 2

# grid space
x = np.linspace(-r,r,100)
y = np.linspace(-r,r,100)
X,Y = np.meshgrid(x,y)

# create the function
Fxy = X**2 + Y**2 - r**2

# draw it
plt.imshow(Fxy)
plt.contour(Fxy,0,colors='k')

plt.axis('off')
plt.show()
```

```python
# parameters
a = 1
b = 2
h = 2
k = -3

# grid space
axlim = np.max((a,b)) + np.max((abs(h),abs(k)))
x = np.linspace(-axlim,axlim,100)
y = np.linspace(-axlim,axlim,100)
X,Y = np.meshgrid(x,y)

# create the function
Fxy = (X-h)**2/a**2 + (Y-k)**2/b**2 - 1

# draw it as a contour
plt.contour(X,Y,Fxy,0)
plt.plot(h,k,'go')

# draw guide lines
plt.grid()
plt.title('Ellipse centered at (x,y)=(%s,%s)' %(h,k))
plt.gca().set_aspect('equal')
plt.show()
```

```python
# hyperbola! (not an "X")

# parameters
a = 1
b = .5
h = 1
k = 2

# grid space
axlim = 2* (np.max((a,b)) + np.max((abs(h),abs(k))))
x = np.linspace(-axlim,axlim,100)
y = np.linspace(-axlim,axlim,100)
X,Y = np.meshgrid(x,y)

# create the function
Fxy = (X-h)**2/a**2 - (Y-k)**2/b**2 - 1

# draw it as a contour
plt.contour(X,Y,Fxy,0)

# draw a dot in the center
plt.plot(h,k,'go')

# draw guide lines
plt.plot([-axlim,axlim],[0,0],'--',color=[.8,.8,.8])
plt.plot([0,0],[-axlim,axlim],'--',color=[.8,.8,.8])

plt.gca().set_aspect('equal')
plt.show()
```


```python
from IPython.display import HTML, display
```

# Simulating Planetary Orbits with a Symplectic Integrator

The name of this library in Fluxions in homage to Isaac Newton, whose early name for differential calculus was "the method of fluxions."  (For an entertaining work of  fiction that places the invention of calculus in historical context, I highly recommend Neal Stephenson's Baroque Cycle.)  The problem Newton was trying to solve was calculating the motion of the planets around the sun under the influence of gravity.  He succeeded in calculating a highly accuracte approximation based on all the planets moving around the sun in elliptical orbits.  This approximation effectively treats the sun as a very heavy stationary body and all the planets orbit around it.  It ignores the gravitational forces between planets and any movement of the sun itself.  It's a good approximation because the sun is much heavier than the planets put toghether (approximately 99.8% of the of the mass of the solar system), but it's not perfect because Jupiter in particular is heavy enough to throw things off.

**Kepler's Laws for Elliptical Orbits**

Citation: http://hyperphysics.phy-astr.gsu.edu/hbase/kepler.html

## Numerical Solution of Differential Equations with Symplectic Integrators
How do scientists and engineers today compute planetary orbits to the highest standards of precision suitable for launching space vehicles?  They use numerical integrators to solve the Newton's equations of motion.  In particular, they use a special class of numerical integrators called **symplectic integrators**.  Most students in STEM fields who encounter differential equations are likely to see an unrepresentative sample of ones that are analytically tractable, such as a simple harmonic oscillator.  It turns out that most differential equations of interest do **not** have analytical solutions, but have to be solved numerically instead.  This is a mature field of study in applied mathematics, and many methods exist.

A simple numerical method is the highly intuitive Forward Euler Method, in which the equation is discretized in time:

Citation: https://en.wikipedia.org/wiki/Euler_method#/media/File:Euler_method.svg

As the picture suggests, discretization errors can gradually accumulate and the approximated trajectory can drift away from the true solution.  These errors can be mitigated by using higher order methods with small step size, but unless an integrator is carefully constructed, it is likely to have a drift where it either gains or loses energy as the system evolves.  This destroys its ability to make accurate calculations over a large number of steps.

### Hamilton's Equations
No discussion of symplectic integrators would be complete without introducing Hamilton's equations.  This is a formalism for solving the equations of motion in a physical system.  The spatial coordinates are $q = (q_1, ... q_n)$ and their "conjugate momenta" are $p = (p_1, ... p_n)$.

$$\frac{d\mathcal{p}_i}{dt} = -\frac{\partial \mathcal{H}}{\partial q_i} \\
\frac{dq_i}{dt} = +\frac{\partial \mathcal{H}}{\partial p_i}$$

The highly symmetrical dual structure of these equations give Hamiltonian systems their special properties.
This can look like a bit much at first, but there's a physical intuition that helps to understand it:
* $q_i$ are the x, y, and z coordinates of different bodies in the problem
* $p_i$ are the x, y, and z coordinates of the momentum of the bodies; $p_i = m_i v_i$

For a classical mechanical system,
$$\mathcal{H} = T + U$$
where $T$ is the total kinetic energy of the system, and $U$ is the total potential energy of the system.
A very important special case is easier to solve: when the Hamiltonian is *separable* and *time invariant*.
In this case, the kinetic energy $T = T(p)$ depends only on the momenta $p$,
and the potential energy $U = U(q)$ depends only on the position.

It can be proven mathematically that any Hamiltonian system (i.e. a system that evolves according to these differential equations) has two special properties:
* It conserves volume in phase space, $dV = dp \; dq$
* It conserves energy $\mathcal{H}(p, q) = \mathcal{H}_0$

### Bad Idea: Simulating a Hamiltonian System with a Non-Symplectic Integrator
What happens if you try to simulate a Hamiltonian system with a non-symplectic integrator? It loses its special properties.  The simulated solution will not conserve energy and volume in phase space.

Citation: https://www.av8n.com/physics/symplectic-integrator.htm

### What is a Symplectic Integrator?
The figure above shows a typical behavior of a non-symplectic integrator that is gradually leaking energy.  If this were a planetary orbit, the simulation would show the planet crashing into the sun at a time when it should still enjoy a stable orbit.

A symplectic integrator on the other hand **respects** the two key symmetries of the Hamiltonian system.
* It conserves energy
* It conserves volume

### Conservation of Energy

Citation: https://www.av8n.com/physics/symplectic-integrator.htm

### Convervation of Volume in Phase Space

Here are visualizations of the fact that symplectic integrators conserve volume in phase space, but non-symplectic integrators do not:

Citation: https://www.av8n.com/physics/symplectic-integrator.htm

One way to think of a symplectic integrator is that it models the behavior of another symplectic system (one that conserves energy and volume in phase space) that is very close to the true system.
Whereas a non-symplectic integrator models a system close to the true system, but one that is not symplectic.

### Better Idea: Leapfrog Integration -- A Simple Symplectic Integrator
Fortunately there is a simple scheme for numerically solving separable Hamiltonian systems that is symplectic.  It is called Leapfrog Integration.  Here is a presentation in "traditional" coordinates $x$, $v$ and $a$ for position, velocity, and acceleration respectively.  Please note that velocities are indexed at half-integer time steps.
\begin{align}
x_i &= v_{i-1} + v_{i-1/2} \Delta t \\
a_i &= F(x_i) / m_i \\
v_{i+1/2} &= v_{i-1/2} + a_i \Delta t
\end{align}

Here is an equivalent version with only integer indices that is better suited to direct translation into computer code.
This version of the equations was used in the planetary simulation we wrote.
\begin{align}
x_{i+1} &= x_i + v_i \Delta t + \frac{1}{2} a_i \Delta t^2
\end{align}

citation: https://en.wikipedia.org/wiki/Leapfrog_integration

This embarrassingly simple code is at the heart of the planetary simulation presented below:
```python
# Perform leapfrog integration simulation
# https://en.wikipedia.org/wiki/Leapfrog_integration
print(f'Performing leapfrog integration with {N} steps...')
for i in tqdm(range(N-1)):
    # Positions at the next time step
    q[i+1,:] = q[i,:] + v[i,:] * dt + 0.5 * a[i,:] * dt2
    # Accelerations of each body in the system at the next time step
    a[i+1,:] = accel_func(q[i+1])
    # Velocities of each body at the next time step
    v[i+1,:] = v[i,:] + 0.5 * (a[i,:] + a[i+1,:]) * dt
    return q, v
```

## Modern Calculations of Planetary Orbits and Ephemerides

Planetary orbits can be described accurately with just a small number of parameters called **orbital elements**.  These quantities date back to Johannes Kepler and are sometimes referred to as Keplerian elements in his honor.  The reason an entire orbit can be decribed locally with just six parameters is that the orbits are very close to following elliptical paths as discovered by Kepler.  Here is a picture illustrating the definitions of the orbital elements.

Citation: By Lasunncty at the English Wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=8971052

Professionals working in astronomy and space exploration have collaborated over the years and built an excellent infrastructure for efficiently sharing data about the positions and orientations of celestial bodies.  Important ideas that feed into this include standardization of time measurements and reference frames.

### Julian Days and Astronomical Time Measurements
Astronomy is one of the oldest sciences and historical records date back millennia.  Astronomers have defined the concept of the Julian Day (https://en.wikipedia.org/wiki/Julian_day) as the number of days from the beginning of the Julian Period.  The Julian period began on January 1, 4713 BC on the proleptic Julian Calendar.  It's straightforward to find a conversion utility.  If you write your own be sure to check it against a reference implementation on the web!

### Ecliptic Coordinate Systems and Astronomical Frames of Reference
We take it for granted that we can describe the location of an object on the earth to a high degree of precision with just two numbers, latitude and longitude.  Pinning down the position of an object in the solar system is much more complicated because everything is always moving around.  The barycenter of the solar system (center of mass) makes a convenient origin, but you also need to define axes.  Astronomers have done this by standardizing on the ideas of an ecliptic coordinate system:

Citation: By Tfr000 (talk) 18:12, 20 June 2012 (UTC) - Own work, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=19971787

The most common frame in use today is called **J2000.0 epoch** and is based on the coordinate system above with the mean equinox of the year 2000.

### Ephemerides and jplephem library
An ephemeris in astronomy (plural ephemerides) comes from the Greek word for "diaries."  It refers to the positions of astronomical orbits at a moment in time.  Before the advent of computers, they were generated by a combination of astronomical observations and hand calculations.  In modern times, computer simulations can produce very accurate ephemerides for important objects like the planets in the solar system.  NASA and the JPL (Jet Propulsion Laboratory) at Caltech run a public service where they offer high quality ephemerides free to the public.  This is offered through an interface called Horizons which can be found here: https://ssd.jpl.nasa.gov/horizons.cgi
Fortunately there is a package available on PyPI called **jplephem** available here: https://pypi.org/project/jplephem/.
This is a very cool package that lets you get to work doing astronomical calculations in minutes.
The most convenient way to use it is to download a data file once and save it locally.  I downloaded the de430.bsp file here: https://naif.jpl.nasa.gov/pub/naif/generic_kernels/spk/
This file is 117 MB but includes positions of the planets between the years 1550 and 2650.  I excerpted it down to 40 years to reduce the size fo 4.26 MB so it would fit on GitHub.

Here is a quick showing how easy it is to use (citation: jplephem documentation on PyPI)

```python
from jplephem.spk import SPK
kernel = SPK.open('../solar_system/resources/planets.bsp')
print(kernel)
```

    File type DAF/SPK and format LTL-IEEE with 14 segments:
    2451544.50..2466160.50  Solar System Barycenter (0) -> Mercury Barycenter (1)
    2451536.50..2466160.50  Solar System Barycenter (0) -> Venus Barycenter (2)
    2451536.50..2466160.50  Solar System Barycenter (0) -> Earth Barycenter (3)
    2451536.50..2466160.50  Solar System Barycenter (0) -> Mars Barycenter (4)
    2451536.50..2466160.50  Solar System Barycenter (0) -> Jupiter Barycenter (5)
    2451536.50..2466160.50  Solar System Barycenter (0) -> Saturn Barycenter (6)
    2451536.50..2466160.50  Solar System Barycenter (0) -> Uranus Barycenter (7)
    2451536.50..2466160.50  Solar System Barycenter (0) -> Neptune Barycenter (8)
    2451536.50..2466160.50  Solar System Barycenter (0) -> Pluto Barycenter (9)
    2451536.50..2466160.50  Solar System Barycenter (0) -> Sun (10)
    2451544.50..2466156.50  Earth Barycenter (3) -> Moon (301)
    2451544.50..2466156.50  Earth Barycenter (3) -> Earth (399)
    2287184.50..2688976.50  Mercury Barycenter (1) -> Mercury (199)
    2287184.50..2688976.50  Venus Barycenter (2) -> Venus (299)

The documentation tells us that all coordinates are in the J2000.0 coordinate system.  Distances are given in kilometers.  The positions of the sun and 9 planets including Pluto are given vs. the barycenter of the solar system.  Here's how to get the position of the earth on February 8, 2015, which is Julian day 2457061.5:

```python
position = kernel[0,3].compute(2457061.5)
print(position)
```

    [-1.10369890e+08  8.93069592e+07  3.86931886e+07]

The system includes the ability to compute the velocity by differentiating a Chebyshev polynomial, as follows:

```python
position, velocity = kernel[0,3].compute_and_differentiate(2457061.5)
print(velocity)
```

    [-1740702.19738961 -1781622.22600993  -772367.63928313]

## Putting it Together: Strategy to Simulate the Solar System
Interested readers can of course review the code.  The highlights are:
* Install the jplephem package
* Download the de430.bsp data file with ephemerides from the JPL
* Write function to load the physical constants used: gravitational constant G, and masses of the sun and planets in kg (these are all available on Wikipedia)
* Write a utility function julian_day that converts a Python date to a julian day
* Write a function configuration(t0, t1, steps_per_day) that returns an array with time steps indexing the rows and positions and velocities across the columns.  For the sun and 8 planets, there are 27 positions and 27 velocities: sun_x, sun_y, sun_z, mercury ..., venus ..., earth, ... ... neptune.
* Write a function accel(q) that compute the acceleration applied to all 27 spatial coordinates (note this is time invariant).  This is for the "constructive" solution that doesn't use automatic differention, just Newton's equations directly
* Write a function make_force(q) that computes the forces applied to all the objects using the Fluxions library.  This is done by building a single Fluxion for the graviational potential energy U(q) once, and then differentiating it with respect to q.
* Write a function simulate_leapfrog() that takes among its arguments functions configuration_func and accel_func.  This allows the same integration back end to be used for both the constructive and Fluxions approach to the problem.
* Write a function energy(q, v) that computes the total energy in the system to check that it really conserves energy
* Write a function mse that computes the mean squared error between a simulated path and a reference path pulled directly from the JPL data
* Write functions to plot a still image and generate a movie clip of the orbit

That's it! The whole calculation is done in 271 lines of code in solar_sytem.py and 466 lines of code in eight_planets.py.  Much of that is constants, comments, testing, and making the movie.

Due to bugs with the bash magic command on Windows 10, it's best to demo running the program directly from the console.

We can see that this program works! It simulated the sun and 8 planets at 16 time steps per day for all of 2018 in just 3 seconds.  It has a mean squared error vs. the JPL simulation of 9.7E-5 astronomical units.  (That's actually very close on this scale; they're simulating other bodies e.g. the moon and the planets move around a lot.)  The energy checks out.  This system has an energy change on the order of 8.9e-10.  The JPL system shows a larger change on the order of 6.9E-7 because they are accounting for additional objects.  The orbits look essentially the same from the two sources.

```python
import io
import base64

video = io.open('../solar_system/movie/planets.mp4', 'r+b').read()
encoded = base64.b64encode(video)
HTML(data=''''''.format(encoded.decode('ascii')))
```

The soundtrack to this movie is from "The Planets" by Gustav Holst.  This is the opening of the piece, "Jupiter: The Bringer of Jollity"


Online-Kurs zu Modellierung und Simulation, Goethe-Universität-Frankfurt | [Startseite](../index.ipynb)

---

# Numerische Lösung der Lotka-Volterra Gleichungen

Populationen von Beute $x$ und Räubern $y$
\begin{align}
\dot{x} & = (a-b y)x \\
\dot{y} & = -(c -dx)y\\
\end{align}
mit $a,b,c,d>0$ zum Anfangswert $(x_0, y_0)^T$.

## Lösen der DGL
Definition von Vektoren Anfangswert
$$\vec u^* = \begin{pmatrix} x^*\\ y^*\end{pmatrix} = \begin{pmatrix} c/d \\ a/b\end{pmatrix}$$
sowie für den stationären Zustand
$$\vec u_0 = \begin{pmatrix} x_0\\ y_0\end{pmatrix} = \begin{pmatrix} 0.1 \\ 1\end{pmatrix}.$$
Zudem definieren wir noch eine Koeffizientenmatrix $$\begin{pmatrix} a & b\\ c& d \end{pmatrix}.$$ Da diese in den übrigen Funktionen bekannt sein soll, verwenden wir das Schlüsselwort 'global.

```octave
u0=[0.1; 1]
global Coeff =[2 1; 3 4];
ustat = [Coeff(2,1) / Coeff(2,2); Coeff(1,1)/Coeff(1,2)];
% Bemerke: Falls ein Kommando mit Semikolon beendet wird, erfolgt keine Ausgabe.
global Coeff =[2 1; 3 4];
```

    u0 =

       0.10000
       1.00000

Nun könnnen wir die rechte Seite der Differentialgleichung definieren.

```octave
function udot=f_lotka_volterra(u, t)
    global Coeff
    udot = zeros(2,1);

    udot(1) = u(1)*(Coeff(1,1) - Coeff(1,2)*u(2));
    udot(2) = - u(2)*(Coeff(2,1)- Coeff(2,2)*u(1));
endfunction
```

Definiere mittels `linspace` einen Vektor "times" mit 500 äquidistant verteilten Punkten im Intervall [0,20].

```octave
times = linspace(0.0, 20.0, 500);
```

Lösen der DGL mit `lsode`.

```octave
numsol  = lsode (@(y,t) f_lotka_volterra(y,t), u0, times);
```

## Erzeuge einige Plots

```octave
hold on   %
xlabel('Zeit t')
ylabel('Population')
plot(times, numsol(:,1), 'color', 'black')
plot(times, numsol(:,2), 'color', 'red')
```

Nun eine Darstellung von Hasen gegen Füchse.

```octave
hold on
xlabel('Hasen')
ylabel('Fuechse')
plot(numsol(:,1), numsol(:,2), 'color', 'blue')
plot(ustat(1), ustat(2), 'color', 'black')
plot(u0(1), u0(2), 'color', 'black')
```

Zuletzt das gleiche mit Vektorfeld:

```octave

% Erstelle N x N Raster
N = 17 % 17 Punkte = 16 Intervalle

xmax = max(numsol(:,1))
ymax = max(numsol(:,2))

xaxis= linspace(0.0, xmax, N);
yaxis= linspace(0.0, ymax, N);

% Initialisierung
dxRaster = zeros(N);
dyRaster = zeros(N);

% Erstelle eine Matrix mit i,j Indizierung
for i=1:length(yaxis)
for j=1:length(xaxis)
    f = f_lotka_volterra([xaxis(j); yaxis(i)], 0);
    dxRaster(i,j) = f(1);
    dyRaster(i,j) = f(2);
end
end

% Eine weitere Lösung zum Anfangswert (0.5, 1)
numsol2 = lsode (@(y,t) f_lotka_volterra(y,4,t), [0.5; 1], times);

% Eigentlicher Plot mit quiver (s. help quiver)
hold on
xlabel('x')
ylabel('y')
title ("Richtungsfeld dy/dx ");
quiver (xaxis, yaxis, dxRaster, dyRaster);
plot(numsol(:,1), numsol(:,2), 'color', 'black')
plot(numsol2(:,1), numsol2(:,2), 'color', 'green')
plot(ustat(1), ustat(2), 'color', 'black')
```

## Bemerkung
* Hilfe zu den Befehlen erhält man mit `help`, z.B. `help plot`.
* Codezellen unterstützen <TAB>-Expansion.
* Zuletzt: Viel Freude beim Experimentieren!

```octave
help plot
```

```octave

```

```octave

```


 Curso de Python para Ingenieros Mecánicos

 Por: Eduardo Vieira

  Sympy - Sistema de algebra simbólica

__SymPy es una biblioteca de Python para matemática simbólica__. Apunta a convertirse en un sistema de algebra computacional (__CAS__) con todas sus prestaciones manteniendo el código tan simple como sea posible para manterlo comprensible y fácilmente extensible. SymPy está __escrito totalmente en Python y no requiere bibliotecas adicionales__. _Este proyecto comenzó en 2005, fue lanzado al público en 2007 y a él han contribuido durante estos años cientos de personas._

_ Otros CAS conocidos son Mathematica y Maple, sin embargo ambos son software privativo y de pago. [Aquí](https://github.com/sympy/sympy/wiki/SymPy-vs.-Maple) puedes encontrar una comparativa de SymPy con Maple. _

Puede hacer cosas como:

* Manipular expresiones (simplificación, expansión...)
* Calcular derivadas e integrales.
* Límites y desarrollos en serie.
* Resolución de ecuaciones.
* Resolción de EDOs.
* Matrices

Sin embargo, SymPy no acaba aquí ni mucho menos...

```python
import sympy as sym
%matplotlib inline
```

Para que los resultados sean formateados en $\LaTeX$ podemos usar:

```python
sym.init_printing(use_latex=True)
```

## Variables simbólicas

En SymPy podemos crear símbolos para las variables con las que deseamos trabajar. Podemos crear un nuevo símbolo usando la clase `Symbol`:

```python
x = sym.Symbol('x')
```

```python
(sym.pi + x)**2
```

```python
# forma alternativa de definir (varios) símbolos
a, b, c = sym.symbols("a, b, c")
```

```python
type(a)
```

    sympy.core.symbol.Symbol

Podemos agregar algunas propiedades a los símbolos cuando son creados:

```python
x = sym.Symbol('x', real=True)
```

```python
x.is_imaginary
```

    False

```python
x = sym.Symbol('x', positive=True)
```

```python
x > 0
```

### Números complejos

La unidad imaginaria es denotada por `I` en Sympy.

```python
1+1*sym.I
```

```python
sym.I**2
```

```python
(x * sym.I + 1)**2
```

### Números racionales

Existen tres tipos distintos de números en SymPy: `Real`, `Rational`, `Integer`:

```python
r1 = sym.Rational(4,5)
r2 = sym.Rational(5,4)
```

```python
r1
```

```python
r1+r2
```

```python
r1/r2
```

### Evaluación numérica

SymPy usa una librería para trabajar con números con precisión arbitraria, y tiene expresiones SymPy  predefinidas para varias constantes matemáticas, tales como: `pi`, `e` y `oo` para el infinito.

Para evaluar numéricamente una expresión podemos usar la función `evalf` (o bien `N`). Ésta usa un argumento `n` que especifíca el número de cifras significativas.

```python
sym.pi.evalf(n=50)
```

```python
y = (x + sym.pi)**2
```

```python
sym.N(y, 5) # equivalente a evalf
```

Cuando evaluamos numéricamente expresiones a menudo deseamos substituir un símbolo por un valor numérico. En SymPy hacemos esto usando la función `subs`:

```python
y.subs(x, 1.5)
```

```python
sym.N(y.subs(x, 1.5))
```

La función `subs` también puede ser usada para substituir símbolos y expresiones:

```python
y.subs(x, a+sym.pi)
```

También podemos combinar la evaluación numérica de expresiones con arreglos NumPy:

```python
import numpy as np
```

```python
x_vec = np.arange(0, 10, 0.1)
```

```python
y_vec = np.array([sym.N(((x + sym.pi)**2).subs(x, xx)) for xx in x_vec])
```

```python
import matplotlib.pyplot as plt
plt.plot(x_vec, y_vec);
```

Sin embargo, este tipo de evaluación numérica puede ser muy lenta, y existe una forma mucho más eficiente de realizar la misma tarea: Usar la función `lambdify` para "mapear" una expresión de Sympy a una función que es mucho más eficiente para la evaluación numérica:

```python
f = sym.lambdify([x], (x + sym.pi)**2, 'numpy')  # el primer argumento es una lista de variables de las que la función f dependerá: en este caso sólo x -> f(x)
```

```python
type(f)
```

    function

```python
y_vec = f(x_vec)  # ahora podemos pasar directamente un arreglo Numpy. Así f(x) es evaluado más eficientemente
```

La mayor eficiencia de usar funciones "lambdificadas" en lugar de usar evalación numérica directa puede ser significativa, a menudo de varios órdenes de magnitud. Aún en este sencillo ejemplo obtenemos un aumento de velocidad importante:

```python
%%timeit

y_vec = np.array([sym.N(((x + sym.pi)**2).subs(x, xx)) for xx in x_vec])
```

    32.5 ms ± 1.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
%%timeit

y_vec = f(x_vec)
```

    7.26 µs ± 30.9 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

## Manipulaciones algebráicas

Uno de los usos principales de un sistema de cálculo simbólico es que realiza manipulaciones algebráicas de expresiones. Por ejemplo, si queremos expandir un producto, factorizar una expresión, o simplificar un resultado. En esta sección presentamos las funciones para realizar estas operaciones básicas en SymPy.

### Expand and factor

Primeros pasos en la manipulación algebráica

```python
(x+1)*(x+2)*(x+3)
```

```python
sym.expand((x+1)*(x+2)*(x+3))
```

La función `expand` acepta varias argumentos clave con los que se puede indicar qué tipo de expansión deseamos realizar. Por ejemplo, para expandir expresiones trigonométricas, usamos el argumento clave `trig=True`:

```python
sym.sin(a+b)
```

```python
sym.expand(sym.sin(a+b), trig=True)
```

Ver `help(expand)` para una descripción detallada de los distintos tipos de expansiones que la función `expand` puede realizar.

También podemos factorizar expresiones, usando la función `factor` de SymPy:

```python
sym.factor(x**3 + 6 * x**2 + 11*x + 6)
```

### Simplify

La función `simplify` intenta simplificar una expresión usando distinta técnicas. Existen también alternativas más específicas a la función `simplify`: `trigsimp`, `powsimp`, `logcombine`, etc.

El uso básico de estas funciones en el siguiente:

```python
# simplify expande un producto
sym.simplify((x+1)*(x+2)*(x+3))
```

```python
# simplify usa identidades trigonometricas
sym.simplify(sym.sin(a)**2 + sym.cos(a)**2)
```

```python
sym.simplify(sym.cos(x)/sym.sin(x))
```

## apart and together

Podemos también manipular expresiones simbólicas que involucran fracciones usando las funciones `apart` y `together`. La primera de estas funciones separa una fracción en sus correspondientes fracciones parciales; la segunda hace todo lo contrario.

```python
f1 = 1/((a+1)*(a+2))
```

```python
f1
```

```python
sym.apart(f1)
```

```python
f2 = 1/(a+2) + 1/(a+3)
```

```python
f2
```

```python
sym.together(f2)
```

Podemos usar también `Simplify`:

```python
sym.simplify(f2)
```

## Cálculo

Además de realizar manipulaciones algebráicas, SimPy puede realizar operaciones de cálculo, tales como derivar y derivar expresiones.

### Derivación

Derviar es usualmente algo simple. Usamos la función `diff`. El primer argumento es una expresión que será derivada, y el segundo argumento es el símbolo respecto al cual se realizará la derivada:

```python
y
```

```python
sym.diff(y**2, x)
```

Para calcular derivadas de orden superior podemos usar:

```python
sym.diff(y**2, x, x)
```

o bien

```python
sym.diff(y**2, x, 2) # hace lo mismo
```

Calculando la derivada de una función de varias variables:

```python
x, y, z = sym.symbols("x,y,z")
```

```python
f = sym.sin(x*y) + sym.cos(y*z)
```

$\frac{d^3f}{dxdy^2}$

```python
sym.diff(f, x, 1, y, 2)
```

### Integracion

La integración se realiza de la misma forma:

```python
f
```

```python
sym.integrate(f, x)
```

Si le damos los límites de integración calculamos la integral definida

```python
sym.integrate(f, (x, -1, 1))
```

Tambien podemos calcular integrales impropias

```python
from sympy import oo
# oo es el infinito
```

```python
sym.integrate(sym.exp(-x**2), (x, -oo, oo))
```

### Sumatoria y productos

Podemos evaluar sumatorias usando: 'Sum'

```python
n = sym.Symbol("n")
```

```python
sym.Sum(1/n**2, (n, 1, 10))
```

```python
sym.Sum(1/n**2, (n,1, 10)).evalf()
```

```python
sym.Sum(1/n**2, (n, 1, oo)).evalf()
```

Lo mismo con productos

```python
sym.Product(n, (n, 1, 10)) # 10!
```

### Límites

Los límites se evalúan con la función `limit`

```python
sym.limit(sym.sin(x)/x, x, 0)
```

Se puede usar el límite para verificar una derivada

```python
f
```

```python
sym.diff(f, x)
```

La derivada por definicion es
$$\frac{d}{{dx}}f\left( x \right) = \mathop {\lim }\limits_{h \to 0} \frac{{f\left( {x + h } \right) - f\left( x \right)}}{h }$$

```python
h = sym.Symbol("h")
```

```python
sym.limit((f.subs(x, x+h) - f)/h, h, 0)
```

OK!

Podemos calcular los limites laterales usando el argumento `dir`:

```python
sym.limit(1/x, x, 0, dir="+")
```

```python
sym.limit(1/x, x, 0, dir="-")
```

### Series

Las series tambien son una de las funciones mas útiles de un CAS. En SymPy se puede calcular la serie de una funcion usando `series`

```python
sym.series(sym.exp(x), x)
```

Por defecto, se calcula la expansión alrededor de $x=0$ (serie de Taylor), pero podemos expandir la serie al rededor de otro punto

```python
sym.series(sym.exp(x), x, 1)
```

Tambien podemos especificar el orden

```python
sym.series(sym.exp(x), x, 1, 10)
```

La expansion en serie incluye la aproximación de orden. Lo cual es útil para trabajar con series de diferente orden

```python
s1 = sym.cos(x).series(x, 0, 5)
s1
```

```python
s2 = sym.sin(x).series(x, 0, 2)
s2
```

```python
sym.expand(s1 * s2)
```

Si queremos eliminar la aproximación de orden usamos el método `removeO`

```python
sym.expand(s1.removeO() * s2.removeO())
```

Pero hay que tener en cuenta que esta no es la expansión correcta de $\cos(x)\sin(x)$ al $5$to orden:

```python
(sym.cos(x)*sym.sin(x)).series(x, 0, 6)
```

## Álgebra Lineal

### Matrices

Las matrices se definen usando la clase `Matrix`:

```python
m11, m12, m21, m22 = sym.symbols("m11, m12, m21, m22")
b1, b2 = sym.symbols("b1, b2")
```

```python
A = sym.Matrix([[m11, m12],[m21, m22]])
A
```

```python
b = sym.Matrix([[b1], [b2]])
b
```

Con la clase `Matrix` podemos realizar las operaciones matriciales típicas

```python
A**2
```

```python
A * b
```

Y tambien determinantes e inversas:

```python
A.det()
```

```python
A.inv()
```

## resolviendo ecuaciones

Para resolver ecuaciones podemos utilizar la función `solve`

```python
sym.solve(x**2 - 1, x)
```

```python
sym.solve(x**4 - x**2 - 1, x)
```

Sistemas de ecuaciones

```python
sym.solve([x + y - 1, x - y - 1], [x,y])
```

La ecuacion puede estar en términos de otras variables simbólicas

```python
sym.solve([x + y - a, x - y - c], [x,y])
```

## Lecturas complementarias

* http://sympy.org/en/index.html - The SymPy projects web page.
* https://github.com/sympy/sympy - The source code of SymPy.
* http://live.sympy.org - Online version of SymPy for testing and demonstrations.

```python
# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = './css/aeropython.css'
HTML(open(css_file, "r").read())
```

/* This template is inspired in the one used by Lorena Barba
in the numerical-mooc repository: https://github.com/numerical-mooc/numerical-mooc
We thank her work and hope you also enjoy the look of the notobooks with this style */

<link href='http://fonts.googleapis.com/css?family=Source+Sans+Pro|Josefin+Sans:400,700,400italic|Ubuntu+Condensed' rel='stylesheet' type='text/css'>

El estilo se ha aplicado =)

```python

```


Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

'''Y= np.exp(X)
y_noise = 10 * np.random.normal(size=X.size)
ydata = Y + y_noise
plt.plot(X, ydata,  'bo')'''
plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2020-07-22 11:00:47 URL:https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.447527, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

popt, pcov = curve_fit(sigmoid, train_x, train_y)

y_hat = sigmoid(test_x, *popt)

print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

print(y_hat[0],test_y[0])

```

    Mean absolute error: 0.18
    Residual sum of squares (MSE): 0.13
    R2-score: -534418111652364574525052944384.00
    2.2742181805891044e-16 0.004508540527225084

    /home/jupyterlab/conda/envs/python/lib/python3.6/site-packages/scipy/optimize/minpack.py:829: OptimizeWarning: Covariance of the parameters could not be estimated
      category=OptimizeWarning)

Double-click __here__ for the solution.

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

Thanks for completing this lesson!

Author:  Saeed Aghabozorgi

Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.


```python
import numpy as np
import scipy as sp
import scipy.stats
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
```

#  Bayesian Conjugates

In general Bayesian econometrics are computationally heavy, however in some special cases, we can derive posterior analytically. And these case all involves using **conjugate priors**, which are certain type of distributions and are able to derive posterior analytically.

The conjugate priors and posterior will have the same distribution function except parameters.

Some common combinations of conjugate priors are
\begin{align}
&\text{beta prior distribution } + \text{binomial data} \rightarrow \text{beta posterior distribution} \\
&\text{gamma prior distribution } + \text{Poisson data} \rightarrow \text{gamma posterior distribution} \\
&\text{normal prior distribution } + \text{normal data} \rightarrow \text{normal posterior distribution}
\end{align}

## The Beta-Binomial Conjugate

The beta distribution is usually used for modeling 'the probability of probability', due to its range $[0, 1]$.

$$
f(p;\alpha, \beta)=\frac{1}{\mathrm{~B}(\alpha, \beta)} p^{\alpha-1}(1-p)^{\beta-1}
$$
where $\alpha$ and $\beta$ are shape parameters both are positive numbers. Here are Beta distribution with different parameters.

```python
x = np.linspace(0, 1, 100)

fig = plt.figure(figsize=(9, 9))
ax = fig.add_subplot(111)

params = np.array([[[.5,.5]],
                  [[5,1.3]],
                  [[1,5]],
                  [[2,2]],
                  [[2,3]],
                  [[3,2]],
                  [[1,1]]])
for i in range(params.shape[0]):
    beta_pdf = sp.stats.beta.pdf(x, params[i][0][0], params[i][0][1])
    ax.plot(x, beta_pdf, lw = 3, label = '$a = %.1f, b =  %.1f$' % (params[i][0][0], params[i][0][1]))
    ax.legend()
ax.set_title('Beta Distribution')

ax.axis([0, 1, 0, 3])
plt.show()
```

The likelihood are obtained from a binomial distribution.
\begin{equation}
\mathcal{L}(k; n, p)=\binom{n}{k} p^k(1-p)^{n-k}
\end{equation}
The number of trials is denoted $n$. The number of observed successes is denoted as $y$. The probability of success is denoted as $p$.

The Bayes' Theorem in probability distribution form will do the work for us
$$
p(\theta \mid y)=\frac{p(y \mid \theta) p(\theta)}{\int_0^1p(y \mid \theta) p(\theta) d\theta}
$$

Substitute in prior and likelihood accordingly
$$
P(p \mid y)=\frac{\left(\begin{array}{l}
n \\
y
\end{array}\right) p^{k}(1-p)^{(n-k)}  \frac{1}{\mathrm{B}\left(\alpha_{0}, \beta_{0}\right)} p^{\alpha_{0}-1}(1-p)^{\beta_{0}-1}}{\int_{0}^{1}\left(\begin{array}{l}
n \\
y
\end{array}\right) p^{k}(1-p)^{(n-k)}  \frac{1}{\mathrm{B}\left(\alpha_{0}, \beta_{0}\right)} p^{\alpha_{0}-1}(1-p)^{\beta_{0}-1} d p}
$$

Cancel out the constant terms
$$
P(p \mid \text {y})=\frac{p^{k}(1-p)^{(n-k)}  p^{\alpha_{0}-1}(1-p)^{\beta_{0}-1}}{\int_{0}^{1} p^{k}(1-p)^{(n-k)}  p^{\alpha_{0}-1}(1-p)^{\beta_{0}-1} d p}
$$

Join terms
$$
P(p \mid y)=\frac{p^{\alpha_{0}+k-1}(1-p)^{\left(\beta_{0}+n-k-1\right)}}{\int_{0}^{1} p^{\alpha_{0}+k-1}(1-p)^{\left(\beta_{0}+n-k-1\right)} d p}
$$

To ensure the denominator integrates to $1$, we multiply an inverse of Beta function to both numerator and denominator.

$$
\frac{1}{B\left(\alpha_{0}+y, \beta_{0}+n-k\right)}
$$

$$
P(p \mid y)=\frac{\frac{1}{B\left(\alpha_{0}+k, \beta_{0}+n-k\right)}p^{\alpha_{0}+k-1}(1-p)^{\left(\beta_{0}+n-k-1\right)}}{\frac{1}{B\left(\alpha_{0}+k, \beta_{0}+n-k\right)}\int_{0}^{1} p^{\alpha_{0}+k-1}(1-p)^{\left(\beta_{0}+n-k-1\right)} d p}
$$

Now the denominator is $1$, which leaves us posterior
$$
P(p \mid y)=\frac{1}{B\left(\alpha_{0}+k, \beta_{0}+n-k\right)}p^{\alpha_{0}+k-1}(1-p)^{\left(\beta_{0}+n-k-1\right)}
$$

### An Example of Beta-Binomial Conjugate

Suppose we would to like to how estimate a basketball player's _free throw_ probability. For the prior, we don't much about him, he can be an excellent or awful player, so we choose $\alpha_0=\beta_0=.5$, the subscript zero represent priors. The plot tells our subjective view toward the player, he either has very high or very low probability of scoring.

```python
x = np.linspace(0, 1, 100)
params = [.5, .5]
beta_pdf = sp.stats.beta.pdf(x, params[0], params[1])

fig, ax = plt.subplots(figsize=(9, 9))
ax.plot(x, beta_pdf, lw = 3, label = r'$\alpha = %.1f, \beta =  %.1f$' % (params[0], params[1]))
ax.legend()

ax.axis([0, 1, 0, 3])
plt.show()
```

Suppose in a $10$ shots attempt, he made $2$ scores. How do we calculate the likelihood? We don't need to, because we have already the closed-form posterior derived.

Therefore, what we can do is to update the posterior by incorporating the likelihood parameters $n=10$ and $k=2$.
$$
\alpha_0+k = .5+2 = 2.5\\
\beta_0+n+k = .5+10+2 = 12.5
$$

```python
x = np.linspace(0, 1, 100)
params_prior = [.5, .5]
params_post = [2.5, 12.5]
beta_prior = sp.stats.beta.pdf(x, params_prior[0], params_prior[1])
beta_post = sp.stats.beta.pdf(x, params_post[0], params_post[1])

fig, ax = plt.subplots(figsize=(9, 9))
ax.plot(x, beta_prior, lw = 3, label = r'$\alpha_0 = %.1f, \beta_0 =  %.1f$' % (params_prior[0], params_prior[1]))
ax.plot(x, beta_post, lw = 3, label = r'$\alpha = %.1f, \beta =  %.1f$' % (params_post[0], params_post[1]))
ax.legend()

ax.axis([0, 1, 0, 5])
plt.show()
```

The posterior will be prior for next round of estimation, say $n= 10$ and $k=4$.
$$
\alpha_0+k = 2.5+4 = 6.5\\
\beta_0+n+k = 12.5+10+4 = 26.5
$$

```python
x = np.linspace(0, 1, 100)
params_prior = [2.5, 12.5]
params_post = [6.5, 26.5]
beta_prior = sp.stats.beta.pdf(x, params_prior[0], params_prior[1])
beta_post = sp.stats.beta.pdf(x, params_post[0], params_post[1])

fig, ax = plt.subplots(figsize=(9, 9))
ax.plot(x, beta_prior, lw = 3, label = r'$\alpha_0 = %.1f, \beta_0 =  %.1f$' % (params_prior[0], params_prior[1]))
ax.plot(x, beta_post, lw = 3, label = r'$\alpha = %.1f, \beta =  %.1f$' % (params_post[0], params_post[1]))
ax.legend()

ax.axis([0, 1, 0, 6])
plt.show()
```

And note that the mean of the beta distribution can be pinned with $\alpha$ and $\beta$, we can use this formula to characterize your beta priors.
\begin{align}
\text{mean:}&\quad\mathrm{E}[p]=\frac{\alpha}{\alpha+\beta}\\
\text{mode:}&\quad \frac{\alpha-1}{\alpha+\beta-2} \text { for } \alpha, \beta>1
\end{align}
For instance, if we want mean to equal $.88$, we can choose $\beta=3$, then solve the equation
$$
.88 =\frac{\alpha}{\alpha+3}
$$
In this case, we obtain $\alpha=22$. Therefore mode will be $.91$.

```python
x = np.linspace(0, 1, 100)
params = [22, 3]
beta_pdf = sp.stats.beta.pdf(x, params[0], params[1])

fig, ax = plt.subplots(figsize=(9, 9))
ax.plot(x, beta_pdf, lw = 3, label = r'$\alpha = %.1f, \beta =  %.1f$' % (params[0], params[1]))
ax.axvline(.88, color='r',ls='--',label='mean: .88')
ax.axvline(.91, color='tomato',ls='--',label='mode: .91')
ax.axis([0, 1, 0, 7])
ax.legend()
plt.show()
```

## The Gamma-Poisson Conjugate

The usually form of the Gamma distribution as a prior in Bayesian estimation has the following form
$$
p(\lambda ; \alpha, \beta)=\frac{\beta_{0}^{\alpha_{0}} \lambda^{\alpha_{0}-1} e^{-\beta_{0} \lambda}}{\Gamma\left(\alpha_{0}\right)}
$$
where $\alpha$ and $\beta$ are hyperparameters, the former determines the shape, the latter determines the scale.

```python
x = np.linspace(0, 15, 200)

fig = plt.figure(figsize=(9, 9))
ax = fig.add_subplot(111)
params = np.array([[[5,1.3]],
                  [[2,3]],
                  [[7.5,1]],
                  [[3,2]]])
for i in range(params.shape[0]):
    gamma_pdf = sp.stats.gamma.pdf(x, params[i][0][0], params[i][0][1])
    ax.plot(x, gamma_pdf, lw = 3, label = r'$\alpha = %.1f, \beta =  %.1f$' % (params[i][0][0], params[i][0][1]))
    ax.legend()
ax.set_title('Gamma Distribution')

plt.show()
```

The Poisson distribution will be used for likelihood function
$$
P(\operatorname{data} \mid \lambda)=\mathcal{L}\left(x_{1} \ldots x_{n} ; \lambda\right)=\prod_{i=1}^{n} \frac{\lambda^{x_{i}} e^{-\lambda}}{x_{i} !}=\frac{\lambda^{\Sigma x_{i}} e^{-n \lambda}}{\prod_{i=1}^{n} x_{i} !}
$$
$\lambda$ is the average number of successes and $x$ is the number of successes.

Use Bayes' Theorem
$$
P(\lambda \mid \text { data })=\frac{\frac{\lambda^{\Sigma x_{i}} e^{-n \lambda}}{\prod x_{i} !} \quad \frac{\beta_{0}^{\alpha_{0}} \lambda^{\alpha_{0}-1} e^{-\beta_{0} \lambda}}{\Gamma\left(\alpha_{0}\right)}}{\int_{0}^{\infty} \frac{\lambda^{\Sigma x_{i}} e^{-n \lambda}}{\prod x_{i} !} \quad \frac{\beta_{0}^{\alpha_{0}} \lambda^{\alpha_{0}-1} e^{-\beta_{0} \lambda}}{\Gamma\left(\alpha_{0}\right)} \quad d \lambda}
$$

After some canceling and joining term
$$
P(\lambda \mid \text { data })=\frac{\lambda^{\alpha_{0}+\Sigma x_{i}-1} e^{-\lambda\left(\beta_{0}+n\right)}}{\int_{0}^{\infty} \lambda^{\alpha_{0}+\Sigma x_{i}-1} e^{-\lambda\left(\beta_{0}+n\right)} d \lambda}
$$

To render denominator to $1$, multiply a common term both onto numerator and denominator. You can compare with prior.

$$
p(\lambda ; \alpha, \beta)=\frac{\beta_{0}^{\alpha_{0}} \lambda^{\alpha_{0}-1} e^{-\beta_{0} \lambda}}{\Gamma\left(\alpha_{0}\right)}
$$

$$
P(\lambda \mid \text { data })=\frac{\frac{(\beta_0+n)^{\alpha_{0}+\Sigma x_{i}}}{\Gamma\left(\alpha_{0}+\Sigma x_{i}\right)} \quad \lambda^{\alpha_{0}+\Sigma x_{i}-1} e^{-\lambda\left(\beta_{0}+n\right)}}{\frac{\left(\beta_{0}+n\right)^{\alpha_{0}+\Sigma x_{i}}}{\Gamma\left(\alpha_{0}+\Sigma x_{i}\right)} \quad \int_{0}^{\infty} \lambda^{\alpha_{0}+\Sigma x_{i}-1} e^{-\lambda\left(\beta_{0}+n\right)} d \lambda}
$$

Finally we obtain posterior
$$
P(\lambda \mid \text { data })=\frac{\left(\beta_{0}+n\right)^{\alpha_{0}+\Sigma x_{i}} \lambda^{\alpha_{0}+\Sigma x_{i}-1} e^{-\lambda\left(\beta_{0}+n\right)}}{\Gamma\left(\alpha_{0}+\Sigma x_{i}\right)}
$$

Posterior parameters are
$$
\alpha = \alpha_{0}+\sum_{i=1}^{n} \chi_{i}\\
\beta = \beta_0 + n
$$

### An Example of Gamma-Poisson Conjugate

The gamma distribution skews to the right and contain only positive values, we can use it for modeling number of hurricanes.
Suppose a country have in average $5$ times of hurricane, we can choose shape $\alpha_0= 10$ and and rate $\beta_0=2$, then variance is $\frac{10}{4}$.

The mean and variances are
$$
E(\lambda) = \frac{\alpha}{\beta}\\
\text{Var}(\lambda) = \frac{\alpha}{\beta^2}
$$

In ```Scipy``` context, the function uses _scale parameter_ rather than _rate parameter_, however they are just inverse of each other.

```python
x = np.linspace(0, 10, 100)
params = [10, 2]
gamma_pdf = sp.stats.gamma.pdf(x, a=params[0], scale=1/params[1])

fig, ax = plt.subplots(figsize=(9, 9))
ax.plot(x, gamma_pdf, lw = 3, label = r'$\alpha = %.1f, \beta =  %.1f$' % (params[0], params[1]))

mean = params[0]/params[1]
mode = (params[0]-1)/params[1]
ax.axvline(mean, color = 'tomato', ls='--', label='mean: {}'.format(mean))
ax.axvline(mode, color = 'red', ls='--', label='mode: {}'.format(mode))
ax.legend()
plt.show()
```

After $12$ year, the data records in total $29$ hurricanes. Update the prior
\begin{align}
\alpha_{\text {posterior }}&=\alpha_{0}+\sum_{i=1}^{n} x_{i} = 10+29=39\\
\beta_{\text {posterior }}&=\beta_{0}+n = 2+10=12
\end{align}

```python
x = np.linspace(0, 10, 100)
params_prior = [10, 2]
params_post = [39, 12]
gamma_pdf_prior = sp.stats.gamma.pdf(x, a=params_prior[0], scale=1/params_prior[1])
gamma_pdf_post = sp.stats.gamma.pdf(x, a=params_post[0], scale=1/params_post[1])

fig, ax = plt.subplots(figsize=(9, 9))
ax.plot(x, gamma_pdf_prior, lw = 3, label = r'$\alpha_0 = %.1f, \beta_0 =  %.1f$' % (params_prior[0], params_prior[1]))
ax.plot(x, gamma_pdf_post, lw = 3, label = r'$\alpha = %.1f, \beta =  %.1f$' % (params_post[0], params_post[1]))

ax.legend()
plt.show()
```

Another $5$ years, recorded $9$ hurricanes altogether. The previous posterior will be current prior
\begin{align}
\alpha_{\text {posterior }}&=\alpha_{0}+\sum_{i=1}^{n} x_{i} = 39+9=48\\
\beta_{\text {posterior }}&=\beta_{0}+n = 12+5=17
\end{align}

```python
x = np.linspace(0, 10, 100)
params_prior = [39, 12]
params_post = [48, 17]
gamma_pdf_prior = sp.stats.gamma.pdf(x, a=params_prior[0], scale=1/params_prior[1])
gamma_pdf_post = sp.stats.gamma.pdf(x, a=params_post[0], scale=1/params_post[1])

fig, ax = plt.subplots(figsize=(9, 9))
ax.plot(x, gamma_pdf_prior, lw = 3, label = r'$\alpha_0 = %.1f, \beta_0 =  %.1f$' % (params_prior[0], params_prior[1]))
ax.plot(x, gamma_pdf_post, lw = 3, label = r'$\alpha = %.1f, \beta =  %.1f$' % (params_post[0], params_post[1]))

ax.legend()
plt.show()
```

## The Normal-Normal Conjugate

The common normal form is
$$
P(x)=f(x ; \mu, \sigma)=\frac{1}{\sqrt{2 \pi} \sigma} e^{-(x-\mu)^{2} /\left(2 \sigma^{2}\right)}
$$
However as a Bayesian prior, the precision $\tau$ is preferred than variance $\sigma^2$, their relationship is $\tau=\frac{1}{\sigma^{2}}$
$$
P(\mu)=f\left(\mu ; \mu_{0}, \tau_{0}\right)=\frac{\sqrt{\tau_{0}}}{\sqrt{2 \pi}} e^{-\frac{1}{2} \tau_{0}\left(\mu-\mu_{0}\right)^{2}}
$$

Or equivalently
$$
P(\mu)=(2 \pi)^{-\frac{1}{2}} \tau_{0}^{\frac{1}{2}} e^{-\frac{1}{2} \tau_{0}\left(\mu-\mu_{0}\right)^{2}}
$$

The likelihood function is
$$
\mathcal{L}\left(x_{1}, \ldots, x_{n} ; \mu, \sigma\right)=\prod_{i=1}^{n} \frac{1}{\sqrt{2 \pi} \sigma} e^{-\left(x_{i}-\mu\right)^{2} /\left(2 \sigma^{2}\right)}
$$

Also replace $\sigma^2$ by $\tau$
$$
P(\text { data } \mid \mu)=\mathcal{L}\left(x_{1}, \ldots, x_{n} ; \mu, \tau\right)=(2 \pi)^{-\frac{n}{2}} \tau^{\frac{n}{2}} e^{-\frac{1}{2} \tau \Sigma\left(x_{i}-\mu\right)^{2}}
$$

Join terms by Bayes' Theorem
$$
P(\mu \mid \text { data })=\frac{(2 \pi)^{-\frac{n+1}{2}} \tau_{0}^{\frac{1}{2}} \tau^{\frac{n}{2}} e^{-\frac{1}{2}\left(\tau_{0}+n \tau\right)\left[\mu-\frac{\left(\tau_{0} \mu_{0}+\tau \Sigma x_{i}\right)}{\tau_{0}+n \tau}\qquad\right]^{2}+C}}{\int_{-\infty}^{\infty}(2 \pi)^{-\frac{n+1}{2}} \tau_{0}^{\frac{1}{2}} \tau^{\frac{n}{2}} e^{-\frac{1}{2}\left(\tau_{0}+n \tau\right)\left[u-\frac{\left(\tau_{0} \mu_{0}+\tau \Sigma x_{i}\right)}{\tau_{0}+n \tau}\qquad\right]^{2}+C} d u}
$$

Cancel out terms
$$
P(\mu \mid \text { data })=\frac{e^{-\frac{1}{2}\left(\tau_{0}+n \tau\right)\left[\mu-\frac{\left(\tau_{0} \mu_{0}+\tau \Sigma x_{i}\right)}{\tau_{0}+n \tau}\right]^{2}}}{\int_{-\infty}^{\infty} e^{-\frac{1}{2}\left(\tau_{0}+n \tau\right)\left[u-\frac{\left(\tau_{0} \mu_{0}+\tau \Sigma x_{i}\right)}{\tau_{0}+n \tau}\right]^{2}} d u}
$$

To render the denominator to $1$, multiply a common term onto both numerator and denominator
$$
P(\mu \mid \text { data })=\frac{(2 \pi)^{-\frac{1}{2}}\left(\tau_{0}+n \tau\right)^{\frac{1}{2}} e^{-\frac{1}{2}\left(\tau_{0}+n \tau\right)\left[\mu-\frac{\left(\tau_{0} \mu_{0}+\tau \Sigma x_{i}\right)}{\tau_{0}+n \tau}\right]^{2}}}{\int_{-\infty}^{\infty}(2 \pi)^{-\frac{1}{2}}\left(\tau_{0}+n \tau\right)^{\frac{1}{2}} e^{-\frac{1}{2}\left(\tau_{0}+n \tau\right)\left[u-\frac{\left(\tau_{0} \mu_{0}+\tau \Sigma x_{i}\right)}{\tau_{0}+n \tau}\right]^{2}} d u}
$$

The posterior is
$$
P(\mu)=(2 \pi)^{-\frac{1}{2}}\left(\tau_{0}+n \tau\right)^{\frac{1}{2}} e^{-\frac{1}{2}\left(\tau_{0}+n \tau\right)}\left(\mu-\frac{\tau_{0} \mu_{0}+\tau \Sigma x_{i}}{\tau_{0}+n \tau}\right)^{2}
$$

The posterior hyperparameters are
$$
\begin{aligned}
\mu_{\text {posterior }} &=\frac{\tau_{0} \mu_{0}+\tau \sum x_{i}}{\tau_{0}+n \tau} \\
\tau_{\text {posterior }} &=\tau_{0}+n \tau
\end{aligned}
$$

```python

```