# On the Reduced Gas Pressure in Spots

What is the gas temerature expected within a spot if _only_ a reduction in gas temperature is accounted for, owing to the presence of a magnetic pressure term?

We'll begin under the assumption that the gas is subject to a polytropic equation of state such that

\begin{equation}
    P_{\rm gas} = K \rho^{\gamma},
\end{equation}

where $P_{\rm gas}$ is the thermal gas pressure, $K$ is a constant of proportionality, $\rho$ is the gas density, and $\gamma$ is the ratio of specific heats. We will also assume, for simplicity, that the magnetic field is in equipartition with the gas, meaning that the internal energy of the gas, $U_{\rm gas}$, is equal to the energy of the magnetic field, $U_{\rm mag}$. Finally, if we assume the gas can be described as ideal, then we know that $\gamma = 5/3$, $P_{\rm gas} = nkT$, and $U_{\rm gas} = 3nkT/2 = 3P_{\rm gas}/2$.

It's possible to re-write the ideal gas equation of state by substituting the gas (mass) density for the gas number density using the mean molecular weight, $\mu$. We then have that $n = \rho/(\mu m_H)$, where $m_p$ is the mass of a proton. Thus,

\begin{equation}
    P_{\rm gas} = \frac{\rho}{\mu m_p}kT.
\end{equation}

Utilizing Equation (1), however, we find that

\begin{equation}
    P_{\rm gas}^{(\gamma - 1)/\gamma} = \frac{k}{K \mu m_p}T = \mathbb{K} T.
\end{equation}

where we have collected all constants to define a new constant $\mathbb{K}$. Note that the mean molecular weight is here taken to be constant with $\mu = 1$ (pure hydrogen gas), for simplicity.

Neglecting magnetic tension forces, the magnetic pressure is equal to the magnetic energy and is spatially isotropic, $P_{\rm mag} = U_{\rm mag} = B^2 / 8\pi$. Since the magnetic field is in energy equipartition with the gas, we have

\begin{equation}
    U_{\rm mag} = \frac{B^2}{8\pi} = \frac{3}{2}P_{\rm gas},
\end{equation}

which provides a convenient estimate of the magnetic field strength at any given point in the gas. If we assume that the total pressure at a given point must be the same, regardless of the presence of a magnetic field, then we can write that

\begin{equation}
    P_{\rm tot} = P_{\rm gas} + \frac{B^2}{8\pi} = \frac{5}{2} P_{\rm gas},
\end{equation}

which must be equivalent to the total pressure when no magnetic field is present (i.e., the gas pressure in the absence of a magnetic field),

\begin{equation}
    P_{\rm tot} \equiv P_{\rm gas,\, 0} = \frac{5}{2}P_{\rm gas}
\end{equation}

where $P_{\rm gas,\, 0}$ is the gas pressure in the absence of a magnetic field.

Under these approximations, we can estimate the change in gas temperature caused by the presence of a magnetic field, neglecting effects related to the _transport_ of energy. Using Equation (3), we can write

\begin{equation}
    \frac{T_{\rm gas}}{T_{\rm gas,\, 0}} = \left(\frac{2}{5}\right)^{(\gamma - 1)/\gamma} = \left(\frac{2}{5}\right)^{2/5} \approx 0.693.
\end{equation}

In the case of the Sun, if we take the background photospheric temperature to be approximately 5779 K, then a rough estimate for the temperature within a sunspot would be 4000 K, or 1779 K cooler than the background photosphere. Quite surprisingly, this agrees with rough estimates for the temperatures within starspot umbra (Solanki 2003).

If, instead, the magnetic pressure were equal to the gas pressure within a spot, the umbral temperature ratio would be equal to

\begin{equation}
    \frac{T_{\rm gas}}{T_{\rm gas,\, 0}} = \left(\frac{1}{2}\right)^{(\gamma - 1)/\gamma} = \left(\frac{1}{2}\right)^{2/5} \approx 0.758,
\end{equation}

which implies spot temperatures of order 4300 K. Again, consistent with observations of umbral temperatures.

One can also estimate the change in density resulting from the decrease in gas pressure and temperature. From Equation (1),

\begin{equation}
    \frac{\rho_{\rm gas}}{\rho_{\rm gas\, 0}} = \left(\frac{P_{\rm gas}}{P_{\rm gas\, 0}}\right)^{1/\gamma},
\end{equation}

which yields a density ratio of approximately 50% for an ideal gas with $\gamma = 5/3$. When typical values for the density in the solar photosphere ($\log\rho \sim 6.4$ near $\tau = 1$) are used, this leads to a density change of approximately $2\times10^{-7}$ g cm$^{-2}$. This is consistent with density changes observed in 3D radiation MHD simulations by Kitiashvili et al. (2010).

Given agreement between estimated this simple estimate of umbral temperatures and estimates from starspot observations, what further consequences might lead to the exclusion of this hypothesis for the cooler nature of sunspots?

Development from the presence of a strong concentration of magnetic flux to the appearance of sunspot pores, if goverend by the thermal evolution of the gas, must occur on a timescale related to the sound crossing time. It is only over this timescale that the gas can effectively communicate information about the presence of the magnetic field. Typical sunspot pores are on the order of 1 Mm in size (Bray & Loughhead 1964) and the adiabatic speed of sound at the solar surface is on the order of 8 km/s. The latter is not exact, but provides a starting point.

\begin{equation}
    \tau = \frac{R_{\rm pore}}{c_s} \sim \frac{10^3}{10^1} \textrm{ s} = 10^2 \textrm{ s},
\end{equation}

or approximately 2 minutes. The sound crossing time is, therefore, quite small compared to the pore formation timescale, which is estimated to be between several hours and several days. Limitations imposed by the sound crossing time are therefore not significant.

It is possible, however, that the process of magnetic suppression of convection and the cooling of the photospheric layers are intertwined. Suppression of convection is the result of the interaction of convective flows with the magnetic field via Lorentz forces. These forces are the same forces that lead to the isotropic pressure that could potentially be responsible for the cooling of the layers by offsetting the gas pressure required to maintain hydrostatic equilibrium.

---

## Potential description

The presence of a magnetic field in a plasma would cause a general cooling of the surface layers, in the event that the system was left to equilibrate. Since the sound crossing time is on the order of 2 minutes, this is expected to occur quite rapidly. However, the surface is not static, as convective updrafts are constantly supplying fresh, warm material to the surface. Convection in the near surface layers has an overturn time of order several minutes (solar granulation timescale), of the same order as the time required for the material to arrive in equilibrium with the magnetic field. Therefore, it may be unlikely for the magnetic field to efficiently cool the gas until the system is unable to supply warm material from deeper layers. This is particularly important since, as a glob of plasma may cool, it will have a tendency to sink while convection occurs. Suppression of convection starves the upper layers of warm material and prevents cooler material from traveling inward, permitting the gas in the near surface layers to equilibrate to a cooler temperature, as dictated by the additional pressure contribution to the equation of state.

Needs validation.

---

## Temperature Contrast Estimates for Starspots

Using the above theoretical development, we find the temperature contrast on other stars is equal to that for sunspots. However, measurements of starspot temperature contrasts find a correlation between starspot temperature contrast, and the effective temperature of the star (Berdyugina 2005).

```python
import numpy as np
```

```python
# confirm sunspot estimate above
Bz = np.sqrt(12.*np.pi*10**4.92)
print "Sunspot Bz (G): {:8.3e}".format(Bz)
print "Sunspot umbral temperature (K): {:6.1f}".format(5779.*0.4**0.4)
```

    Sunspot Bz (G): 1.771e+03
    Sunspot umbral temperature (K): 4005.7

While the temperature ratio appears fixed in Equations (7) and (8), variation in the derived values can be estimated from the variation in $\gamma = c_p / c_v$. There will also be variation in the specific relation between the gas internal energy and the gas pressure, but this is more difficult to quantify from stellar evolution model output.

To address this problem, a small grid of models was run with masses in the range $0.1$ — $0.9 M_{\odot}$ with a mass resolution of $0.1 M_{\odot}$. Output from the code was modified to yield $\gamma$ directly as a function of radius, temperature, pressure, and density.

```python
%matplotlib inline
import matplotlib.pyplot as plt
```

```python
Teffs = np.arange(3000., 6100., 250.)
Tspot_fixed_gamma = Teffs*(0.4)**0.4

# gamma where tau ~ 1 (note, no 0.2 Msun point)
Gammas = np.array([1.22, 1.29, 1.30, 1.29, 1.36, 1.55, 1.63, 1.65])
ModelT = np.array([3.51, 3.57, 3.59, 3.61, 3.65, 3.71, 3.74, 3.77]) # tau = 1
ModelT = 10**ModelT
ModelTeff = 10**np.array([3.47, 3.53, 3.55, 3.57, 3.60, 3.65, 3.69, 3.73]) # T = Teff
Tratio = 0.4**((Gammas - 1.0)/Gammas)
Tspot_physical_gamma = np.array([ModelT[i]*0.4**((Gammas[i] - 1.0)/Gammas[i]) for i in range(len(ModelT))])

# smoothed curve
from scipy.interpolate import interp1d
icurve = interp1d(ModelT, Tspot_physical_gamma, kind='cubic')
Tphot_smoothed = np.arange(3240., 5880., 20.)
Tspot_smoothed = icurve(Tphot_smoothed)

# approximate Berdyugina data
DeltaT = np.array([ 350.,  450.,  700., 1000., 1300., 1650., 1850.])
BerdyT = np.array([3300., 3500., 4000., 4500., 5000., 5500., 5800.])
BSpotT = BerdyT - DeltaT

print Tratio
```

    [ 0.84769638  0.81384297  0.80940837  0.81384297  0.78462644  0.72242951
      0.70177027  0.69700478]

```python
fig, ax = plt.subplots(1, 1, figsize=(8,4))

ax.set_xlabel('Photospheric Temperature (K)', fontsize=20.)
ax.set_ylabel('Spot Temperature (K)', fontsize=20.)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax.grid(True)

ax.plot(Teffs, Tspot_fixed_gamma, '--', lw=2, dashes=(10., 10.), markersize=9.0, c='#1e90ff')
ax.plot(BerdyT, BSpotT, '-', lw=2 , dashes=(25., 15.), c='#000080')
ax.fill_between(BerdyT, BSpotT - 200., BSpotT + 200., facecolor='#000080', alpha=0.1, edgecolor='#eeeeee')
ax.plot(ModelT, Tspot_physical_gamma, 'o', lw=2, markersize=9.0, c='#800000')
ax.plot(Tphot_smoothed, Tspot_smoothed, '-', lw=3, c='#800000')
```

```python
fig, ax = plt.subplots(1, 1, figsize=(8,4))

ax.set_xlabel('Photospheric Temperature (K)', fontsize=20.)
ax.set_ylabel('T(phot) - T(spot) (K)', fontsize=20.)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax.grid(True)

ax.plot(Teffs, Teffs - Tspot_fixed_gamma, '--', lw=2, dashes=(10., 10.), markersize=9.0, c='#1e90ff')
ax.plot(BerdyT, DeltaT, '-', lw=2 , dashes=(25., 15.), c='#000080')
ax.fill_between(BerdyT, DeltaT - 200., DeltaT + 200., facecolor='#000080', alpha=0.1, edgecolor='#eeeeee')
ax.plot(ModelT, ModelT - Tspot_physical_gamma, 'o', lw=2, markersize=9.0, c='#800000')
ax.plot(Tphot_smoothed,  Tphot_smoothed - Tspot_smoothed, '-', lw=3, c='#800000')
```

Results of this simple model indicate that equilibration of a polytropic gas within a magnetic structure located near the photosphere ($\tau_{\rm ross} = 1$) provides a reasonable approximation to observed spot temperatures from low-mass M dwarfs up to solar-type stars. Above 5400 K, the gas is sufficiently ideal that the model predicted relationship (red line) is asymptotic to the case of a purely ideal gas (small-dashed light-blue line). Below that temperature, the simple model traces the relationship provided by Berdyugina (2005). Difficulties below 4000 K may be the result of either model inaccuracies, either stemming from atmospheric structure or the simple approximation of energy equipartition, or observational complications that arise from measuring M dwarf photospheric temperatures.

We can also estimate umbral magnetic field strengths by extracting the model gas pressure at the same optical depth ($\tau_{\rm ross} = 1$),

```python
# log(Pressure)
p_gas = np.array([6.37, 5.90, 5.80, 5.70, 5.60, 5.45, 5.30, 5.15])
B_field_Eeq = np.sqrt(12.*np.pi*0.4*10**p_gas)/1.0e3  # in kG
B_field_Peq = np.sqrt( 8.*np.pi*10**p_gas)/1.0e3

# smooth curves
icurve = interp1d(ModelT, B_field_Eeq, kind='cubic')
B_field_Eeq_smooth = icurve(np.arange(3240., 5880., 20.))
icurve = interp1d(ModelT, B_field_Peq, kind='cubic')
B_field_Peq_smooth = icurve(np.arange(3240., 5880., 20.))
```

```python
fig, ax = plt.subplots(1, 1, figsize=(8,4))

ax.set_xlabel('Photospheric Temperature (K)', fontsize=20.)
ax.set_ylabel('Equipartition Magnetic Field (kG)', fontsize=20.)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax.grid(True)

ax.plot(ModelT, B_field_Eeq, 'o', lw=2, markersize=9.0, c='#800000')
ax.plot(np.arange(3240., 5880., 20.), B_field_Eeq_smooth, '-', lw=2, c='#800000')
ax.plot(ModelT, B_field_Peq, 'o', lw=2, markersize=9.0, c='#1e90ff')
ax.plot(np.arange(3240., 5880., 20.), B_field_Peq_smooth, '-', lw=2, dashes=(20., 5.), c='#1e90ff')
```

The two curves represent two different approximations, one is energy equipartition (red curve) and the other that the magnetic pressure is precisely equal to the gas pressure (blue curve). These values do not represent surface averaged magnetic field strengths, but the strengths of local concentrations of magnetic flux. Based on energy equipartition, we do not expect spot magnetic field strengths to be considerably larger than those estimated from the red curve.

Finally, we can estimate a "curve of cooling" relating the strength of a magnetic field to the temperature within the magnetic sturcture. Since stars the properties of the photospheric layers are so different for stars as a function of effeictve temperature, it'll be helpful to compute curves at several effective temperatures characteristic of low-mass M dwarf, a late K dwarf, and an early K/G dwarf.

```python
B_field_strengths = np.arange(0.1, 8.1, 0.1)*1.0e3
log_P_phot = np.array([5.15, 5.60, 5.90])
Gamma_phot = np.array([1.65, 1.36, 1.29])
Exponents = (Gamma_phot - 1.0)/Gamma_phot
print Exponents

fig, ax = plt.subplots(3, 1, figsize=(8, 12), sharex=True)

i = 0
ax[2].set_xlabel('Spot Magnetic Field (kG)', fontsize=20.)
for axis in ax:
    B_field_eq = np.sqrt(8.*np.pi*(0.4*10**log_P_phot[i]))
    axis.grid(True)
    axis.set_ylabel('$T_{\\rm spot} / T_{\\rm phot}$', fontsize=20.)
    axis.tick_params(which='major', axis='both', length=10., labelsize=16.)
    axis.plot(B_field_strengths/1.0e3, (1.0 - B_field_strengths**2/(8.*np.pi*10**log_P_phot[i]))**Exponents[i],
              '-', lw=3, c='#800000')
    axis.plot(B_field_eq/1.0e3, (1.0 - B_field_eq**2/(8.0*np.pi*10**log_P_phot[i]))**Exponents[i],
              'o', markersize=12.0, c='#555555')
    i += 1
```

### M Dwarfs

Shulyak et al. (2010, 2014) have measured distribution of magnetic fields on the surfaces of M dwarfs by modeling FeH bands in high resolution, high S/N Stokes I sectra. They find M dwarf spectra are typically best fit by a uniform 1 kG magnetic field everywhere on the star, but with the addition of local concentrations across the surface. These local concentrations can reach upward of 7 – 8 kG. We now ask, do the stars that possess these field lie close to the locus defined by models?

```python
Shulyak_max_B = np.array([[3400., 100., 6.5, 0.5],
                          [3400., 100., 6.0, 0.5],
                          [3300., 100., 7.5, 0.5],
                          [3100., 100., 6.5, 0.5],
                          [3100.,  50., 1.0, 0.5]])
```

```python
fig, ax = plt.subplots(1, 1, figsize=(8,4))

ax.set_xlabel('Photospheric Temperature (K)', fontsize=20.)
ax.set_ylabel('Equipartition Magnetic Field (kG)', fontsize=20.)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax.grid(True)

ax.fill_between(np.arange(3240., 5880., 20.), B_field_Peq_smooth, B_field_Eeq_smooth, facecolor='#555555', alpha=0.3, edgecolor='#eeeeee')
ax.errorbar(Shulyak_max_B[:,0], Shulyak_max_B[:,2], xerr=Shulyak_max_B[:,1], yerr=Shulyak_max_B[:,3], lw=2, fmt='o', c='#555555')
```

We find that the data from Shulyak et al. (2014) for the maximum mangetic field strengths (in local concentrations) of active stars are on the order of those expected from either energy of pressure equipartition. The fact that two have weaker magnetic fields only indicates that the there were no strong local concentrations, similar to the surface of the quiet Sun.

Note, however, that there is some uncertainty in this comparison. Shulyak et al. (2014) quote the effective temperature, which for M dwarfs is not necessarily equal to the photospheric temperature. Model atmospheres predict that the "effective temperature" for an M dwarf occurs in the optically thin layers above the opaque photosphere. Thus, it is more likely that the photospheric temperature for the Shulyak sample is _greater_ than those quoted above.

If we instead convert the quoted effective temperatures to photospheric temperatures using stellar model atmospheres, we find

```python
Shulyak_max_B = np.array([[3685., 100., 6.5, 0.5],
                          [3685., 100., 6.0, 0.5],
                          [3578., 100., 7.5, 0.5],
                          [3379., 100., 6.5, 0.5],
                          [3379.,  50., 1.0, 0.5]])

fig, ax = plt.subplots(1, 1, figsize=(8,4))

ax.set_xlabel('Photospheric Temperature (K)', fontsize=20.)
ax.set_ylabel('Equipartition Magnetic Field (kG)', fontsize=20.)
ax.tick_params(which='major', axis='both', length=10., labelsize=16.)
ax.grid(True)

ax.fill_between(np.arange(3240., 5880., 20.), B_field_Peq_smooth, B_field_Eeq_smooth, facecolor='#555555', alpha=0.3, edgecolor='#eeeeee')
ax.errorbar(Shulyak_max_B[:,0] + 100., Shulyak_max_B[:,2], xerr=Shulyak_max_B[:,1], yerr=Shulyak_max_B[:,3], lw=2, fmt='o', c='#555555')
```

which shows that localized concentrations of magnetic fields exceed those estimated by energy or pressure equipartition.

In general, these comparisons are further complicated by uncertainties regarding the formation height of molecules and atomic features with respect to the background plasma. Molecules may reveal the magnetic field strength at an optical depth (in the ambient plasma) below $\tau = 1$, in which case the gas pressure used to compute the equipartition magnetic field would be larger.


```python
from __future__ import print_function, division
from sympy import *
init_printing(use_unicode=True)
```

## Spong

```python
m1, l1, lc1, I1, q1, q1dot = symbols('m1 l1 lc1 I1 q1 qdot1')
m2, lc2, I2, q2, q2dot = symbols('m2 lc2 I2 q2 qdot2')
```

```python
M = Matrix([[m1*lc1**2+m2*(l1**2+lc2**2+2*l1*lc2*cos(q2))+I1+I2,
             m2*(lc2**2+l1*lc2*cos(q2))+I2],
            [m2*(lc2**2+l1*lc2*cos(q2))+I2,
             m2*lc2**2+I2]])
```

```python
M
```

```python
C = Matrix([[-2*m2*l1*lc2*sin(q2)*q2dot,
             -m2*l1*lc2*sin(q2)*q2dot],
            [m2*l1*lc2*sin(q2)*q1dot,
             0]])
```

```python
C
```

```python
g = symbols('g')
```

```python
G = g*Matrix([[(m1*lc1+m2*l1)*cos(q1)+m2*lc2*cos(q1+q2)],
              [m2*lc2*cos(q1+q2)]])
```

```python
G
```

```python
u= symbols('u')
```

```python
qdotdot = -M**(-1)*(C*Matrix([[q1dot],[q2dot]])+G
          -Matrix([[0],[1]])*u)
```

```python
collect(factor(simplify(cancel(qdotdot[0]))),q1dot)
```

### Make sure the system is at rest in fixed points

```python
qdotdot.subs([(q1,pi/2),(q2,0), (u,0),(q1dot,0), (q2dot,0)])
```

```python
qdotdot.subs([(q1,-pi/2),(q2,0), (u,0),(q1dot,0), (q2dot,0)])
```

```python
diff(qdotdot[0],q1dot).subs([(q1,pi/2),(q2,0), (u,0), (q1dot,0), (q2dot,0)])
```

```python
A20=diff(qdotdot[0],q1).subs([(q1,pi/2),(q2,0),
                              (u,0), (q1dot,0), (q2dot,0)])
A20.subs([(m1,1), (m2,1), (l1,1), (lc1,0.5), (lc2,1),
          (I1,0.083), (I2,0.33), (g,9.8)])
```

```python
A21=diff(qdotdot[0],q2).subs([(q1,pi/2),(q2,0),
                              (u,0), (q1dot,0), (q2dot,0)])
A21.subs([(m1,1), (m2,1), (l1,1), (lc1,0.5), (lc2,1),
          (I1,0.083), (I2,0.33), (g,9.8)])
```

```python
A30=diff(qdotdot[1],q1).subs([(q1,pi/2),(q2,0),
                              (u,0), (q1dot,0), (q2dot,0)])
A30.subs([(m1,1), (m2,1), (l1,1), (lc1,0.5), (lc2,1),
          (I1,0.083), (I2,0.33), (g,9.8)])
```

```python
A31=diff(qdotdot[1],q2).subs([(q1,pi/2),(q2,0),
                              (u,0), (q1dot,0), (q2dot,0)])
A31.subs([(m1,1), (m2,1), (l1,1), (lc1,0.5), (lc2,1),
          (I1,0.083), (I2,0.33), (g,9.8)])
```

```python
B20=diff(qdotdot[0],u).subs([(q1,pi/2),(q2,0),
                              (u,0), (q1dot,0), (q2dot,0)])
B20.subs([(m1,1), (m2,1), (l1,1), (lc1,0.5), (lc2,1),
          (I1,0.083), (I2,0.33), (g,9.8)])
```

```python
B30=diff(qdotdot[1],u).subs([(q1,pi/2),(q2,0),
                              (u,0), (q1dot,0), (q2dot,0)])
B30.subs([(m1,1), (m2,1), (l1,1),  (lc1,0.5), (lc2,1),
          (I1,0.083), (I2,0.33), (g,9.8)])
```


# Example notebook to show how adaptive frequency oscillators (AFOs) work

In the first part of the notebook, we show how to implement an AFO in native python. Then we show how to use the c++ library to make faster simulation in the second part of the notebook. An example with an input with a time varying frequency is shown last

```python
# standard imports
%matplotlib widget
import numpy as np
import matplotlib.pylab as plt
```

## Adaptive Frequency Oscillator in Python
We wish to implement an AFO whose equation is
$$\begin{align} \dot{\phi} & = \lambda \omega - K \sin(\phi) F(t) \\ \dot{\omega} &= - K \sin(\phi) F(t) \end{align}$$
where $$F(t) = \sin(\omega_F t)$$

```python
def integrate_afo(phi0, omega0, omegaF, t_init, t_end, K=10000, lamb=1., save_dt=0.001, dt=0.00001):
    """ this function integrates an AFO from t_init to t_end
    it starts with initial conditions phi0 and omega0
    returns: numpy vectors t, phi and omega
    """
    # hoe many integration steps in the internal loop
    internal_step = int(round(save_dt/dt))

    # how many steps till t_end
    num_steps = int(round((t_end - t_init)/save_dt))

    # we preallocate memory
    t = np.zeros(num_steps+1)
    phi = np.zeros_like(t)
    omega = np.zeros_like(t)

    # we set initial conditions
    t[0] = t_init
    omega[0] = omega0
    phi[0] = phi0

    # our temp variables
    phi_temp = phi0
    omega_temp = omega0
    t_temp = t_init

    # the main integration loop
    for i in range(num_steps):
        #internal integration loop
        for j in range(internal_step):
            pert = -K * np.sin(omegaF * t_temp) * np.sin(phi_temp)
            phi_temp += (lamb * omega_temp + pert)*dt
            omega_temp += pert*dt
            t_temp += dt

        # save data
        t[i+1] = t_temp
        phi[i+1] = phi_temp
        omega[i+1] = omega_temp

    return t, phi, omega
```

```python
# we define the coupling stength, input frequency, lambda
K =  1000.
omegaF = 30
lamb = 1.

# we define the time resolution (save_dt) and the internal integration step dt
dt = 0.00001
save_dt = 0.001

# duration of the integration
t_end = 20.
t_init = 0.

phi0 = 0
omega0 = 100

# we integrate
t, phi, omega = integrate_afo(phi0, omega0, omegaF, t_init, t_end, K=10000, lamb=1., save_dt=0.001, dt=0.00001)

# now we plot the results
plt.figure()
plt.subplot(2,1,1)
plt.plot(t, phi)
plt.ylabel(r'$\phi$')
plt.subplot(2,1,2)
plt.plot(t, lamb * omega)
plt.plot([t_init, t_end],[omegaF, omegaF], '--k')
plt.ylabel(r'$\omega$')
plt.xlabel('Time [s]')
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Text(0.5, 0, 'Time [s]')

## Adaptive Frequency Oscillator using C++ library and python bindings
Now we do the same using the c++ library (it needs to be compiled and installed - cf. README)

```python
import pyafos # we import the library
```

```python
# we define the coupling stength, input frequency, lambda
K =  1000.
omegaF = 30
lamb = 1.

# we define the time resolution (save_dt) and the internal integration step dt
dt = 0.00001
save_dt = 0.001

# duration of the integration
t_end = 20.
t_init = 0.

phi0 = 0
omega0 = 100

#run an integration
oscill = pyafos.PhaseAFO()
oscill.initialize(K, lamb)
oscill.input().vec_of_sines(np.array([omegaF]),np.array([1.0]),np.array([0.0]))
pyafos.integrate(oscill, t_init,t_end,np.array([phi0,omega0]),dt,save_dt)

#get the data to be plotted
t = oscill.t()
phi = oscill.y()[0,:]
omega = oscill.y()[1,:]

# now we plot the results
plt.figure()
plt.subplot(2,1,1)
plt.plot(t, phi)
plt.ylabel(r'$\phi$')
plt.subplot(2,1,2)
plt.plot(t, lamb * omega)
plt.plot([t_init, t_end],[omegaF, omegaF], '--k')
plt.ylabel(r'$\omega$')
plt.xlabel('Time [s]')
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Text(0.5, 0, 'Time [s]')

## Tracking a time varying frequency

We want to track a frequency that is changing over time $F(t) = \sin\left(\omega_F t + \frac{1}{\omega_C}\sin(\omega_C t)\right)$

```python
# we define the coupling stength, input frequency, lambda
K =  1000.
omegaF = 50
lamb = 1.
omegaC = 0.5

# we define the time resolution (save_dt) and the internal integration step dt
dt = 0.00001
save_dt = 0.001

# duration of the integration
t_end = 50.
t_init = 0.

phi0 = 0
omega0 = 40

#run an integration
oscill = pyafos.PhaseAFO()
oscill.initialize(K, lamb)
oscill.input().frequency_changing_sine(omegaF, omegaC)
pyafos.integrate(oscill, t_init,t_end,np.array([phi0,omega0]),dt,save_dt)

#get the data to be plotted
t = oscill.t()
phi = oscill.y()[0,:]
omega = oscill.y()[1,:]

# now we plot the results
plt.figure()
plt.subplot(2,1,1)
plt.plot(t, phi)
plt.ylabel(r'$\phi$')
plt.subplot(2,1,2)
plt.plot(t, lamb * omega)
plt.plot(t, omegaF + np.sin(omegaC*t))
plt.ylim([40,60])
plt.ylabel(r'$\omega$')
plt.xlabel('Time [s]')
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Text(0.5, 0, 'Time [s]')

```python

```

python/adaptive frequency oscillators example.ipynb


# Multiple Scales Examples - Rayleigh's Equation

We will approximate the following Rayleigh's equation using the multiple scale method

$$
\begin{gather*}
    \ddot{x} + \epsilon\left(\dfrac{1}{3}\dot{x}^3 - \dot{x} \right) + x = 0
\end{gather*}
$$

with the initial conditions

$$
\begin{gather*}
    x(0) = a\\
    \dot{x}(0) = 0
\end{gather*}
$$

First order approximation

```python
import sympy as sp
from sympy.simplify.fu import TR0, TR5, TR7, TR8, TR11
from math import factorial
```

```python
# Functions for multiple scales

# Function for Time operator
def Dt(f, n, Ts, e=sp.Symbol('epsilon')):
    if n==1:
        return sp.expand(sum([e**i * sp.diff(f, T_i) for i, T_i in enumerate(Ts)]))
    return Dt(Dt(f, 1, Ts, e), n-1, Ts, e)

def collect_epsilon(f, e=sp.Symbol('epsilon')):
    N = sp.degree(f, e)
    f_temp = f
    collected_dict = {}
    for i in range(N, 0, -1):
        collected_term = f_temp.coeff(e**i)
        collected_dict[e**i] = collected_term
        delete_terms = sp.expand(e**i * collected_term)
        f_temp -= delete_terms
    collected_dict[e**0] = f_temp
    return collected_dict
```

The scales are defined as
\begin{gather*}
    T_0 = t, \phantom{---} T_1 = \epsilon t
\end{gather*}
where $T_0$ is greatly faster than $T_1$. The time derivative is then defined to be
\begin{align*}
    \dfrac{d}{dt} &= \dfrac{\partial}{\partial T_0}\dfrac{\partial T_0}{\partial t} + \dfrac{\partial}{\partial T_1}\dfrac{\partial T_1}{\partial t} + \cdots\\
    &= \dfrac{\partial}{\partial T_0} + \epsilon \dfrac{\partial}{\partial T_1} + \cdots\\
    \dfrac{d^2}{dt^2} &= \dfrac{\partial^2}{\partial T_0^2} + 2\epsilon \dfrac{\partial^2}{\partial T_0 \partial T_1} + \epsilon^2 \dfrac{\partial^2}{\partial T_1^2} + \cdots
\end{align*}
Here we abbreviate the differential expressions using the time operators
\begin{gather*}
    D_0 = \dfrac{\partial}{\partial T_0}, \phantom{---} D_1 = \dfrac{\partial}{\partial T_1}.
\end{gather*}
Then the EOM becomes
\begin{gather*}
    D_0^2 x + 2\epsilon D_0D_1 x + \epsilon^2 D_1 x + \epsilon\left[ \dfrac{1}{3}\left( D_0 x + \epsilon D_1 x\right)^3 + \left(D_0x + \epsilon D_1 x\right) \right] + x = 0
\end{gather*}
with the initial conditions of
\begin{gather*}
    x(0) = a, \phantom{---} \dfrac{x}{T_0} + \epsilon\dfrac{x}{T_1} = 0 \phantom{---} \textrm{for } \phantom{-} T_0 = T_1 = 0
\end{gather*}
Now seeking for a asymptotic approximation for $x$ of the form
\begin{gather*}
    x(t) \equiv x(T_0, T_1; \epsilon) \approx x_o(T_0,T_1) + \epsilon x(T_0,T_1)
\end{gather*}

```python
N = 1

# Define the symbolic parameters
e = sp.symbols('epsilon')
Ti = sp.symbols('T_(0:' + str(N+1) + ')', real=True)
Di = sp.symbols('D_(0:' + str(N+1) + ')', real=True)
x = sp.Symbol('x')
x0 = sp.Function('x_0')(*Ti)
x1 = sp.Function('x_1')(*Ti)

# Expansion for x(t)
x_e = x0 + epsilon*x1
x_e
```

$\displaystyle \epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)}$

```python
# EOM

EOM = (Di[0]**2*x + 2*e*Di[0]*Di[1]*x + e**2*Di[1]*x +
      e*((Di[0]*x + e*Di[1]*x)**3/3 + (Di[0]*x+e*Di[1]*x)) + x)
EOM = sp.Eq(EOM, 0)
EOM
```

$\displaystyle D_{0}^{2} x + 2 D_{0} D_{1} \epsilon x + D_{1} \epsilon^{2} x + \epsilon \left(D_{0} x + D_{1} \epsilon x + \frac{\left(D_{0} x + D_{1} \epsilon x\right)^{3}}{3}\right) + x = 0$

```python
# Substitute the asymptotic approximation
EOM1 = EOM.subs(x, x_e)
EOM1
```

$\displaystyle D_{0}^{2} \left(\epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)}\right) + 2 D_{0} D_{1} \epsilon \left(\epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)}\right) + D_{1} \epsilon^{2} \left(\epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)}\right) + \epsilon \left(D_{0} \left(\epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)}\right) + D_{1} \epsilon \left(\epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)}\right) + \frac{\left(D_{0} \left(\epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)}\right) + D_{1} \epsilon \left(\epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)}\right)\right)^{3}}{3}\right) + \epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} = 0$

```python
# Expand the equation
EOM1 = sp.expand(EOM1)
EOM1
```

$\displaystyle \frac{D_{0}^{3} \epsilon^{4} \operatorname{x_{1}}^{3}{\left(T_{0},T_{1} \right)}}{3} + D_{0}^{3} \epsilon^{3} \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} \operatorname{x_{1}}^{2}{\left(T_{0},T_{1} \right)} + D_{0}^{3} \epsilon^{2} \operatorname{x_{0}}^{2}{\left(T_{0},T_{1} \right)} \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \frac{D_{0}^{3} \epsilon \operatorname{x_{0}}^{3}{\left(T_{0},T_{1} \right)}}{3} + D_{0}^{2} D_{1} \epsilon^{5} \operatorname{x_{1}}^{3}{\left(T_{0},T_{1} \right)} + 3 D_{0}^{2} D_{1} \epsilon^{4} \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} \operatorname{x_{1}}^{2}{\left(T_{0},T_{1} \right)} + 3 D_{0}^{2} D_{1} \epsilon^{3} \operatorname{x_{0}}^{2}{\left(T_{0},T_{1} \right)} \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + D_{0}^{2} D_{1} \epsilon^{2} \operatorname{x_{0}}^{3}{\left(T_{0},T_{1} \right)} + D_{0}^{2} \epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + D_{0}^{2} \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} + D_{0} D_{1}^{2} \epsilon^{6} \operatorname{x_{1}}^{3}{\left(T_{0},T_{1} \right)} + 3 D_{0} D_{1}^{2} \epsilon^{5} \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} \operatorname{x_{1}}^{2}{\left(T_{0},T_{1} \right)} + 3 D_{0} D_{1}^{2} \epsilon^{4} \operatorname{x_{0}}^{2}{\left(T_{0},T_{1} \right)} \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + D_{0} D_{1}^{2} \epsilon^{3} \operatorname{x_{0}}^{3}{\left(T_{0},T_{1} \right)} + 2 D_{0} D_{1} \epsilon^{2} \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + 2 D_{0} D_{1} \epsilon \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} + D_{0} \epsilon^{2} \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + D_{0} \epsilon \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} + \frac{D_{1}^{3} \epsilon^{7} \operatorname{x_{1}}^{3}{\left(T_{0},T_{1} \right)}}{3} + D_{1}^{3} \epsilon^{6} \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} \operatorname{x_{1}}^{2}{\left(T_{0},T_{1} \right)} + D_{1}^{3} \epsilon^{5} \operatorname{x_{0}}^{2}{\left(T_{0},T_{1} \right)} \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \frac{D_{1}^{3} \epsilon^{4} \operatorname{x_{0}}^{3}{\left(T_{0},T_{1} \right)}}{3} + 2 D_{1} \epsilon^{3} \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + 2 D_{1} \epsilon^{2} \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} + \epsilon \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} = 0$

Collect the $\epsilon$ terms by their order

```python
# Collect the epsilon terms
epsilon_Eq = collect_epsilon(EOM1.lhs)
```

```python
e0 = epsilon_Eq[e**0]
e0eq = sp.Eq(e0, 0)
e0eq
```

$\displaystyle D_{0}^{2} \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} = 0$

```python
e1 = epsilon_Eq[e**1]
e1eq = sp.Eq(e1, 0)
e1eq
```

$\displaystyle \frac{D_{0}^{3} \operatorname{x_{0}}^{3}{\left(T_{0},T_{1} \right)}}{3} + D_{0}^{2} \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} + 2 D_{0} D_{1} \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} + D_{0} \operatorname{x_{0}}{\left(T_{0},T_{1} \right)} + \operatorname{x_{1}}{\left(T_{0},T_{1} \right)} = 0$

The respective intial conditions for $x$ and $\dot{x}$ are given by
\begin{align*}
    x_0 = a, & \phantom{--} \dfrac{\partial x_0}{\partial T_0} = 0 \phantom{---} \textrm{for } \phantom{-} T_0 = T_1 = 0\\
    x_1 = 0, & \phantom{--} \dfrac{\partial x_1}{\partial T_0} = -\dfrac{\partial x_0}{\partial T_1} \phantom{---} \textrm{for } \phantom{-} T_0 = T_1 = 0\\
\end{align*}

Since $T_0$ and $T_1$ are being treated as an independent variable temporarily, the differential equation above are partial differential equations for a function of $x_0$ of two variables $T_0$ and $T_1$. However, since no derivatives with respect to $T_1$ appear in the equation of $O(1)$ collected for $\epsilon$ this equation can be regarded as instead as an ordinary differential equation for a function of $T_0$ regarding $T_1$ as an auxiliary parameter. Thus, the general solution for $x_0$ can be expressed as the following

\begin{gather*}
    x_0 = A_0(T_1)\cos T_0 + B_0(T_1)\sin T_0
\end{gather*}

in which the coefficients for the sine and consine terms can be found using the initial conditions

\begin{gather*}
    A_0(0) = a \phantom{---} \textrm{ and } \phantom{---} B_0(0) = 0
\end{gather*}

If we take the derivative with respect to $T_0$ we have

\begin{gather*}
    D_0 x_0 = -A_0(T_1)\sin T_0 + B_0(T_1)\cos T_0
\end{gather*}

and

\begin{gather*}
    D_0D_1 x_0 = \dfrac{\partial }{\partial T_1}\left( \dfrac{\partial x_0}{\partial T_0}\right) = -D_1 A_0\sin T_0  + D_1 B_0\cos T_0
\end{gather*}

Substituting these into the equation collected from the EOM with respect to $O(\epsilon)$ we obtain the following relationship

```python
A0, B0 = sp.symbols('A_0 B_0')
x0_g = A0*sp.cos(Ti[0]) + B0*sp.sin(Ti[0])
D0x0 = sp.diff(x0_g, Ti[0])
D0D1x0 = -Di[1]*A0*sp.sin(Ti[0]) + Di[1]*B0*sp.cos(Ti[0])

e1eq_update = -D0x0 - 2*D0D1x0 - D0x0**3/3
e1eq_update = sp.expand(e1eq_update)
e1eq_update = sp.expand(TR8(sp.expand(TR7(TR5(e1eq_update)))))
e1eq_update
```

$\displaystyle \frac{A_{0}^{3} \sin{\left(T_{0} \right)}}{4} - \frac{A_{0}^{3} \sin{\left(3 T_{0} \right)}}{12} - \frac{A_{0}^{2} B_{0} \cos{\left(T_{0} \right)}}{4} + \frac{A_{0}^{2} B_{0} \cos{\left(3 T_{0} \right)}}{4} + \frac{A_{0} B_{0}^{2} \sin{\left(T_{0} \right)}}{4} + \frac{A_{0} B_{0}^{2} \sin{\left(3 T_{0} \right)}}{4} + 2 A_{0} D_{1} \sin{\left(T_{0} \right)} + A_{0} \sin{\left(T_{0} \right)} - \frac{B_{0}^{3} \cos{\left(T_{0} \right)}}{4} - \frac{B_{0}^{3} \cos{\left(3 T_{0} \right)}}{12} - 2 B_{0} D_{1} \cos{\left(T_{0} \right)} - B_{0} \cos{\left(T_{0} \right)}$

To have the secular terms vanish we let
\begin{gather*}
    2D_1A_0 + A_0 + \dfrac{1}{4}A_0^3 + \dfrac{1}{4}A_0B_0^2= 0\\
    2D_1B_0 + B_0 + \dfrac{1}{4}B_0^3 + \dfrac{1}{4}A_0^2B_0= 0
\end{gather*}
Now if we solve these ordinary differential equations we have

```python
# Dummuy variable C0
C0 = sp.Function('C_0')(Ti[1])
D0 = sp.Function('D_0')(Ti[1])
a = sp.Symbol('a', real=True)
temp1 = sp.Eq(2*sp.diff(C0, Ti[1]) + C0 + C0**3/4 + C0*D0**2/4, 0)
temp2 = sp.Eq(2*sp.diff(D0, Ti[1]) + D0 + D0**3/4 + C0**2*D0/4, 0)
res = sp.dsolve([temp1, temp2])
```

```python
res[0]
```

$\displaystyle \operatorname{C_{0}}{\left(T_{1} \right)} = 2 \sqrt{\frac{- \sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} + \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} + e^{C_{2} + T_{1}} - 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}$

```python
res[1]
```

$\displaystyle \operatorname{D_{0}}{\left(T_{1} \right)} = 2 C_{1} \sqrt{\frac{- \sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} + \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} + e^{C_{2} + T_{1}} - 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}$

```python
res[2]
```

$\displaystyle \operatorname{C_{0}}{\left(T_{1} \right)} = - 2 \sqrt{\frac{\sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} - \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} + e^{C_{2} + T_{1}} - 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}$

```python
res[3]
```

$\displaystyle \operatorname{D_{0}}{\left(T_{1} \right)} = - 2 C_{1} \sqrt{\frac{\sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} - \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} + e^{C_{2} + T_{1}} - 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}$

```python
res[4]
```

$\displaystyle \operatorname{C_{0}}{\left(T_{1} \right)} = 2 \sqrt{\frac{\sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} - \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} + e^{C_{2} + T_{1}} - 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}$

```python
res[5]
```

$\displaystyle \operatorname{D_{0}}{\left(T_{1} \right)} = 2 C_{1} \sqrt{\frac{\sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} - \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} + e^{C_{2} + T_{1}} - 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}$

```python
res[6]
```

$\displaystyle \operatorname{C_{0}}{\left(T_{1} \right)} = - 2 \sqrt{- \frac{\sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} - \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} - e^{C_{2} + T_{1}} + 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}$

```python
res[7]
```

$\displaystyle \operatorname{D_{0}}{\left(T_{1} \right)} = - 2 C_{1} \sqrt{- \frac{\sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} - \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} - e^{C_{2} + T_{1}} + 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}$

Now if we solve these ordinary differential equations we have

$$
\begin{gather*}
    A_0 =2 \sqrt{\frac{- \sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} + \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} + e^{C_{2} + T_{1}} - 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}\\[0.4cm]
    B_0 = 2 C_{1} \sqrt{\frac{- \sqrt{e^{C_{2}}} e^{\frac{T_{1}}{2}} + \sqrt{e^{C_{2}}} e^{C_{2} + \frac{3 T_{1}}{2}} + e^{C_{2} + T_{1}} - 1}{- 2 C_{1}^{2} e^{C_{2} + T_{1}} + C_{1}^{2} e^{2 C_{2} + 2 T_{1}} + C_{1}^{2} - 2 e^{C_{2} + T_{1}} + e^{2 C_{2} + 2 T_{1}} + 1}}
\end{gather*}
$$

Applying the initial conditions $a_0 = a$ and $b_0 = 0$ of we have

\begin{gather*}
    A_0 = 2\sqrt{\dfrac{-e^{\frac{T_1}{2}}+e^{\frac{3T_1}{2}} + e^{T_1} - 1}{-2a^2e^{T_1} + a^2e^{2T_1} + a^2 -2e^{T_1}+e^{2T_1} + 1}}\\
    B_0 = aA_0
\end{gather*}

Then we finally have

\begin{align*}
    x(t) &= 2\sqrt{\dfrac{-e^{\frac{\epsilon t}{2}}+e^{\frac{3\epsilon t}{2}} + e^{\epsilon t} - 1}{-2a^2e^{\epsilon t} + a^2e^{2\epsilon t} + a^2 -2e^{\epsilon t}+e^{2\epsilon t} + 1}}\cos t \\
    & \phantom{---} + 2a\sqrt{\dfrac{-e^{\frac{\epsilon t}{2}}+e^{\frac{3\epsilon t}{2}} + e^{\epsilon t} - 1}{-2a^2e^{\epsilon t} + a^2e^{2\epsilon t} + a^2 -2e^{\epsilon t}+e^{2\epsilon t} + 1}}\sin t + O(\epsilon^2).
\end{align*}

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp, DOP853
from typing import List
```

```python
# Generate phase plane of Rayleigh's equation to confirm limit cycle

def rayleigh(t, x, e):
    return [x[1], -x[0] - e*(x[1]**3 / 3 - x[1])]

def solve_diffeq(func, t, tspan, ic, parameters={}, algorithm='DOP853', stepsize=np.inf):
    return solve_ivp(fun=func, t_span=tspan, t_eval=t, y0=ic, method=algorithm,
                     args=tuple(parameters.values()), atol=1e-8, rtol=1e-5, max_step=stepsize)

def phasePlane(x1, x2, func, params):
    X1, X2 = np.meshgrid(x1, x2)  # create grid
    u, v = np.zeros(X1.shape), np.zeros(X2.shape)
    NI, NJ = X1.shape
    for i in range(NI):
        for j in range(NJ):
            x = X1[i, j]
            y = X2[i, j]
            dx = func(0, (x, y), *params.values())  # compute values on grid
            u[i, j] = dx[0]
            v[i, j] = dx[1]
    M = np.hypot(u, v)
    u /= M
    v /= M
    return X1, X2, u, v, M

def DEplot(sys: object, tspan: tuple, x0: List[List[float]],
           x: np.ndarray, y: np.ndarray, params: dict):
    if len(tspan) != 3:
        raise Exception('tspan should be tuple of size 3: (min, max, number of points).')
    # Set up the figure the way we want it to look
    plt.figure(figsize=(12, 9))

    X1, X2, dx1, dx2, M = phasePlane(
        x, y, sys, params
    )

    # Quiver plot
    plt.quiver(X1, X2, dx1, dx2, M, scale=None, pivot='mid')
    plt.grid()

    if tspan[0] < 0:
        t1 = np.linspace(0, tspan[0], tspan[2])
        t2 = np.linspace(0, tspan[1], tspan[2])
        if min(tspan) < 0:
            t_span1 = (np.max(t1), np.min(t1))
        else:
            t_span1 = (np.min(t1), np.max(t1))
        t_span2 = (np.min(t2), np.max(t2))
        for x0i in x0:
            sol1 = solve_diffeq(sys, t1, t_span1, x0i, params)
            plt.plot(sol1.y[0, :], sol1.y[1, :], '-r')
            sol2 = solve_diffeq(sys, t2, t_span2, x0i, params)
            plt.plot(sol2.y[0, :], sol2.y[1, :], '-r')
    else:
        t = np.linspace(tspan[0], tspan[1], tspan[2])
        t_span = (np.min(t), np.max(t))
        for x0i in x0:
            sol = solve_diffeq(sys, t, t_span, x0i, params)
            plt.plot(sol.y[0, :], sol.y[1, :], '-r')

    plt.xlim([np.min(x), np.max(x)])
    plt.ylim([np.min(y), np.max(y)])
    plt.show()
```

```python
x10 = np.arange(0, 10, 1)
x20 = np.arange(0, 10, 1)
x0 = np.stack((x10, x20), axis=-1)

p = {'e': 0.01}

x1 = np.linspace(-5, 5, 20)
x2 = np.linspace(-5, 5, 20)

DEplot(rayleigh, (-8, 8, 1000), x0, x1, x2, p)
```

```python
# Compare the approximation to the actual solution
# let a = 2

tmax = 2
tmin = 30
tspan = np.linspace(tmin, tmax, 1000)
# ODE solver solution
sol =  solve_diffeq(rayleigh, tspan, (tmin, tmax), [2, 0], p)

# Approximation
def rayleigh_MS_O1(t, a, e):
    num = -np.exp(e*t/2) + np.exp(3*e*t/2) + np.exp(e*t) - 1
    den = -2*a**2*np.exp(e*t) + a**2*np.exp(2*e*t) + a**2 - 2*np.exp(e*t) + np.exp(2*e*t) + 1
    A0 = 2 * np.sqrt(num / den)
    B0 = a * A0
    return A0 * np.cos(t) #+ B0 * np.sin(t)

approx = rayleigh_MS_O1(tspan, 2, 0.01)

plt.figure(figsize=(12, 9))
plt.plot(tspan, sol.y[0, :])
plt.plot(tspan, approx)
plt.grid(True)
plt.xlabel('$t$')
plt.ylabel('$x$')
plt.show()
```

### you can see that it is a very crude approximation


```python
from sympy import symbols, integrate, diff
from scipy.integrate import quad

x = symbols('x')

f = 1/x**2
f
```

$\displaystyle \frac{1}{x^{2}}$

```python
F = integrate(f, x)
F
```

$\displaystyle - \frac{1}{x}$

```python
F = lambda x: -1/x
F(2)-F(1)
```

    0.5

```python
f = lambda x: 1/x**2
quad(f, 1, 2)
```

    (0.49999999999999994, 5.551115123125782e-15)

```python
f = 2**x
display(f)
display(diff(f, x))
display(integrate(f, x))
```

$\displaystyle 2^{x}$

$\displaystyle 2^{x} \log{\left(2 \right)}$

$\displaystyle \frac{2^{x}}{\log{\left(2 \right)}}$


# Decision Tree

Classification And Regression Trees (CART for short) is a term introduced by [Leo Breiman](https://en.wikipedia.org/wiki/Leo_Breiman) to refer to Decision Tree algorithms that can be used for classification or regression predictive modeling problems.

In this lab assignment, you will implement various ways to calculate impurity which is used to split data in constructing the decision trees and apply the Decision Tree algorithm to solve two real-world problems: a classification one and a regression one.

```python
# import packages
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
from matplotlib.legend_handler import HandlerLine2D
from sklearn.metrics import accuracy_score
from sklearn.metrics import confusion_matrix
from sklearn.metrics import mean_squared_error
from sklearn.metrics import mean_absolute_error
from sklearn.tree import DecisionTreeClassifier
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import GridSearchCV

# make this notebook's output stable across runs
np.random.seed(0)

import warnings
warnings.filterwarnings("ignore")
%matplotlib inline
```

## Gini impurity and Entropy

**Coding Part 1: Implement the functions to calculate Gini impurity and entropy.**

#### Gini impurity

The CART algorithm recursively splits the training set into two subsets using a single feature k and a threshold $t_k$. The best feature and threshold are chosen to produce the purest subsets weighted by their size. **Gini impurity** measures the impurity of the data points in a set and is used to evaluate how good a split is when the CART algorithm searches for the best pair of feature and the threshold.

To compute Gini impurity for a set of items with J classes, suppose $i \in \{1, 2, \dots, J\}$ and let $p_i$ be the fraction of items labeled with class i in the set.
\begin{align}
I(p) = 1 - \sum_{i=1}^J p_i^2
\end{align}

In this exercise, you will implement the function to calculate gini impurity.

```python
def gini_impurity(x):
    """
    TODO: This function calculate the Gini impurity of an array.

    Args:
    x: a numpy ndarray
    """
    classes,n_items_per_class = np.unique(x,return_counts=True)
    items = len(x)
    impurity = 1 - np.sum([
        np.square(i/items)
            for i in n_items_per_class])
    return impurity
```

```python
np.testing.assert_equal(0, gini_impurity(np.array([1, 1, 1])))
np.testing.assert_equal(0.5, gini_impurity(np.array([1, 0, 1, 0])))
np.testing.assert_equal(3/4, gini_impurity(np.array(['a', 'b', 'c', 'd'])))
np.testing.assert_almost_equal(2.0/3, gini_impurity(np.array([1, 2, 3, 1, 2, 3])))
```

#### Entropy

Another popular measure of impurity is called **entropy**, which measures the average information content of a message. Entropy is zero when all messages are identical. When it applied to CART, a set's entropy is zero when it contains instances of only one class. Entropy is calculated as follows:
\begin{align}
I(p) = - \sum_{i=1}^J p_i log_2{p_i}
\end{align}

Here, you will implement entropy impurity function.

```python
def entropy(x):
    """
    TODO: This function calculate the entropy of an array.

    Args:
    x: a numpy ndarray
    """
    classes,n_items_per_class = np.unique(x,return_counts=True)
    items = len(x)
    entropy = -1*np.sum([
        (i/items)*np.log2(i/items)
            for i in n_items_per_class])
    if entropy == -0.0 or entropy == +0.0:
        return 0
    else:
        return entropy
```

```python
np.testing.assert_equal(0, entropy(np.array([1, 1, 1])))
np.testing.assert_equal(1.0, entropy(np.array([1, 0, 1, 0])))
np.testing.assert_equal(2.0, entropy(np.array(['a', 'b', 'c', 'd'])))
np.testing.assert_almost_equal(1.58496, entropy(np.array([1, 2, 3, 1, 2, 3])), 4)
```

---

## Decision Tree Classifier

**Coding Part 2: In this exercise, we will apply the Decision Tree classifier to classify the Iris flower data.**

### Iris dataset

The Iris data set contains the morphologic variation of Iris flowers of three related species (Iris setosa, Iris virginica and Iris versicolor). Four features were measured from each observation (see image below):
- Sepal.Length: sepal length in centimeters.
- Sepal.Width: sepal width in centimeters.
- Petal.Length: petal length in centimeters.
- Petal.Width: petal width in centimeters.

    Iris setosa
    Iris versicolor
    Iris virginica

```python
# load the iris train and test data from CSV files
train = pd.read_csv("../data/iris_train.csv")
test = pd.read_csv("../data/iris_test.csv")

train_x = train.iloc[:,0:4]
train_y = train.iloc[:,4]

test_x = test.iloc[:,0:4]
test_y = test.iloc[:,4]

# print the number of instances in each class
print(train_y.value_counts().sort_index())
print(test_y.value_counts().sort_index())
```

    Iris-setosa        34
    Iris-versicolor    32
    Iris-virginica     39
    Name: species, dtype: int64
    Iris-setosa        16
    Iris-versicolor    18
    Iris-virginica     11
    Name: species, dtype: int64

### Train and visualize a simple Decision Tree

```python
# TODO: read the scikit-learn doc on DecisionTreeClassifier and train a Decision Tree with max depth of 2
#DecisionTreeClassifier?
dtc = DecisionTreeClassifier(max_depth=2).fit(train_x,train_y)
```

Now let's visualize the decision tree we just trained on the iris dataset and see how it makes predictions.

```python
from io import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus

dot_data = StringIO()
feature_names = train_x.columns
class_names = train_y.unique()
class_names.sort()
export_graphviz(dtc, out_file=dot_data, feature_names=feature_names, class_names=class_names, filled=True, rounded=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())
```

```python
#from sklearn import tree
#tree.plot_tree(dtc)
#plt.show()
```

Decision trees are easy to inteprete and is often referred to as *whitebox* machine learning algorithm. Let's see how this decision tree represented above makes predictions. Suppose you find an iris flower and want to classify it into setosa, versicolor or virginica. You start at the root node (the very top node in the tree). In this node, we check if the flower's patel length is smaller than or equal to 2.35 cm. If it is, we move to the left child and predict setosa to be its class. Otherwise, we move to the right child node. Then similarly we check if the petal length is smaller than or equal to 4.95 cm. If it is, we move to its left child node and predict versicolor to be its class. Otherwise, we move to its right child and predict virginica to be its class.

### Prediction with Decision tree

With this simple decision tree above, we can apply it to make predictions on the test dataset and evaluate its performance.

```python
# TODO: use the trained decision tree model to make predictions on the test data and evaluate the model performance
#       using evaluate(y, z) above.
test_z = dtc.predict(test_x)
print("model accuracy: {}".format(accuracy_score(test_y, test_z)))
print("model confusion matrix:\n {}".format(confusion_matrix(test_y, test_z, labels=['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'])))
```

    model accuracy: 0.9111111111111111
    model confusion matrix:
     [[16  0  0]
     [ 0 17  1]
     [ 0  3  8]]

### Hyper-parameters

Hyper-parameter controls the complexity of the decision tree model. For example, the deeper the tree is, the more complex patterns the model will be able to capture. In this exercise, we train the decision trees with increasing number of maximum depth and plot its performance. We should see the accuracy of the training data increase as the tree grows deeper, but the accuracy on the test data might not as the model will eventually start to overfit and does not generalize well on the unseen test data.

```python
max_depths = np.linspace(1, 8, 8, endpoint=True)
train_results = []
test_results = []

for max_depth in max_depths:
    # TODO: train the decision tree model with various max_depth, make predictions and evaluate on both train and test data.
    dt = DecisionTreeClassifier(max_depth=max_depth)
    dt.fit(train_x, train_y)

    train_z = dt.predict(train_x)
    train_accuracy = accuracy_score(train_y, train_z)
    train_results.append(train_accuracy)

    test_z = dt.predict(test_x)
    test_accuracy = accuracy_score(test_y, test_z)
    test_results.append(test_accuracy)

# plot the accuracy on train and test data
line1, = plt.plot(max_depths, train_results, 'b', label="Train Accuracy")
line2, = plt.plot(max_depths, test_results, 'r', label="Test Accuracy")
plt.legend(handler_map={line1: HandlerLine2D(numpoints=2)})
plt.ylabel("Accuracy")
plt.xlabel("Max Depth")
plt.show()
```

### Fine-tune the decision tree classifier
Decision tree is a very powerful model with very few assumptions about the incoming training data (unlike linear models, which assume the data linear), however, it is more likely to overfit the data and won't generalize well to unseen data. To void overfitting, we need to restrict the decision tree's freedom during training via regularization (e.g. max_depth, min_sample_split, max_leaf_nodes and etc.).

To fine-tune the model and combat overfitting, use grid search with cross-validation (with the help of the GridSearchCV class) to find the best hyper-parameter settings for the DecisionTreeClassifier. In particular, we would like to fine-tune the following hyper-parameters:
- **criteria**: this defines how we measure the quality of a split. we can choose either "gini" for the Gini impurity or "entropy" for the information gain.
- **max_depth**: the maximum depth of the tree. This indicates how deep the tree can be. The deeper the tree, the more splits it has and it captures more information about the data. But meanwhile, deeper trees are more likely to overfit the data. For this practice, we will choose from {1, 2, 3} given there are only 4 features in the iris dataset.
- **min_samples_split**: This value represents the minimum number of samples required to split an internal node. The smaller this value is, the deeper the tree will grow, thus more likely to overfit. On the other hand, if the value is really large (the size of the training data in the extreme case), the tree will be very shallow and could suffer from underfit. In this practice, we choose from {0.05, 0.1, 0.2}

```python
# TODO: fine-tune the model, use grid search with cross-validation.
# cv: None, to use the default 5-fold cross validation,
parameters = {
    'criterion': ['gini', 'entropy'],
    'max_depth': [1, 2, 4, 8],
    'min_samples_split': [0.01, 0.05, 0.1, 0.2, 0.4, 0.8]
}

dt = DecisionTreeClassifier()
grid = GridSearchCV(dt, parameters)
grid.fit(train_x, train_y)

# summarize the results of the grid search
print("The best score is {}".format(grid.best_score_))
print("The best hyper parameter setting is {}".format(grid.best_params_))
```

    The best score is 0.9619047619047618
    The best hyper parameter setting is {'criterion': 'gini', 'max_depth': 4, 'min_samples_split': 0.05}

### Prediction and Evaluation

Now we have a fine-tuned decision tree classifier based on the training data, let's apply this model to make predictions on the test data and evaluate its performance.

```python
test_z = grid.predict(test_x)

print("model accuracy: {}".format(accuracy_score(test_y, test_z)))
print("model confusion matrix:\n {}".format(confusion_matrix(test_y, test_z, labels=['Iris-setosa', 'Iris-versicolor', 'Iris-virginica'])))
```

    model accuracy: 0.9777777777777777
    model confusion matrix:
     [[16  0  0]
     [ 0 17  1]
     [ 0  0 11]]

---

## Decision Tree Regressor

**Coding Part 3: In this exercise, we will apply the Decision Tree classifier to predict the California housing prices.**

### California Housing Dataset

The California Housing dataset appeared in a 1997 paper titled Sparse Spatial Autoregressions by Pace, R. Kelley and Ronald Barry, published in the Statistics and Probability Letters journal. They built it using the 1990 California census data. It contains one row per census block group. A block group is the smallest geographical unit for which the U.S. Census Bureau publishes sample data (a block group typically has a population of 600 to 3,000 people).

```python
# Load train and test data from CSV files.
train = pd.read_csv("../data/housing_train.csv")
test = pd.read_csv("../data/housing_test.csv")

train_x = train.iloc[:,0:8]
train_y = train.iloc[:,8]

test_x = test.iloc[:,0:8]
test_y = test.iloc[:,8]
```

```python
train_x.head()
```

      longitude
      latitude
      housing_median_age
      total_rooms
      total_bedrooms
      population
      households
      median_income

      0
      -119.79
      36.73
      52.0
      112.0
      28.0
      193.0
      40.0
      1.9750

      1
      -122.21
      37.77
      43.0
      1017.0
      328.0
      836.0
      277.0
      2.2604

      2
      -118.04
      33.87
      17.0
      2358.0
      396.0
      1387.0
      364.0
      6.2990

      3
      -118.28
      34.06
      17.0
      2518.0
      1196.0
      3051.0
      1000.0
      1.7199

      4
      -119.81
      36.73
      50.0
      772.0
      194.0
      606.0
      167.0
      2.2206

```python
# TODO: fine-tune the model, use grid search with cross-validation.
# cv: None, to use the default 5-fold cross validation,
parameters = {
    'max_depth': [4, 8, 16, 32],
    'min_samples_split': [0.001, 0.01, 0.05, 0.1, 0.2],
    'max_features': [2, 4, 8]
}

dt = DecisionTreeClassifier()
grid = GridSearchCV(dt, parameters)
grid.fit(train_x, train_y)

# summarize the results of the grid search
print("The best score is {}".format(grid.best_score_))
print("The best hyper parameter setting is {}".format(grid.best_params_))
```

    The best score is 0.049211099014158224
    The best hyper parameter setting is {'max_depth': 8, 'max_features': 2, 'min_samples_split': 0.1}

```python
# Use the fine-tuned model to make predictions on the test data
test_z = grid.predict(test_x)

print("model Root Mean Squared Error: {}".format(round(np.sqrt(mean_squared_error(test_y, test_z)))))
print("model Mean Absolute Error: {}".format(round(mean_absolute_error(test_y, test_z))))
```

    model Root Mean Squared Error: 206959
    model Mean Absolute Error: 157456

```python
# Compare to classifier with default hyper paramters
test_z = dt.fit(train_x, train_y).predict(test_x)

print("model accuracy: {}".format(accuracy_score(test_y, test_z)))
print("model Root Mean Squared Error: {}".format(round(np.sqrt(mean_squared_error(test_y, test_z)))))
print("model Mean Absolute Error: {}".format(round(mean_absolute_error(test_y, test_z))))
```

    model accuracy: 0.025839793281653745
    model Root Mean Squared Error: 82463
    model Mean Absolute Error: 54655

### End of ML 310 Lab Assignment 2
---

Conceptual Overview

In this lesson, I learned methods of measuring impurity that separate data into leaves. I also learned how to tune these models such that it doesn't overfit the training data and have large variance. Some of the most common hyper parameters to tune are max depth, min samples to split on, and max features.

Using grid search, model performance can improve drastically. In the first example the test score improved from %96.2 to 97.8%. It should also be noted, that training time for decision trees can be costly! Also, be cautious when using with a mostly numerical dataset.

```python

```


```julia
using MDBM
using PyPlot;
pygui(true);
```

# 1. Implcit equation with constraint
\begin{align}
 x,y & \in \left[ -5,5 \right] \\
x^2+y^2-2^2 & =0 \\
x-y & >0
\end{align}

```julia
function foo(x,y)
    x^2.0+y^2.0-4.0^2.0
end
function c(x,y)
    x-y
end

ax1=Axis([-5,-2.5,0,2.5,5],"x")
ax2=Axis(-5:2:5.0,"b")

mymdbm=MDBM_Problem(foo,[ax1,ax2],constraint=c)
iteration=6 #number of refinements (resolution doubling)
solve!(mymdbm,iteration)

#evaluated points
x_eval,y_eval=getevaluatedpoints(mymdbm)

#solution points
x_sol,y_sol=getinterpolatedsolution(mymdbm)

fig = figure(1);clf()
scatter(x_eval,y_eval,s=2)
scatter(x_sol,y_sol,s=4);

# plot the constraint
mymdbm_c=MDBM_Problem(c,[ax1,ax2])
solve!(mymdbm_c,iteration)
x_sol,y_sol=getinterpolatedsolution(mymdbm_c)
scatter(x_sol,y_sol,s=4);
```

```julia
fig = figure(101);clf()
myDT1=connect(mymdbm);
for i in 1:length(myDT1)
    dt=myDT1[i]
    P1=getinterpolatedsolution(mymdbm.ncubes[dt[1]],mymdbm)
    P2=getinterpolatedsolution(mymdbm.ncubes[dt[2]],mymdbm)
    plot([P1[1],P2[1]],[P1[2],P2[2]], color="k")
end
```

# 2. System of implicit equations
## parameter space

\begin{equation}
x,y,z \in \left[ -2,2 \right]
\end{equation}

## Sphere 1
Single implicit eqution
\begin{equation}
x^2+y^2+z^2-1 =0
\end{equation}

## Sphere 2
Single implicit eqution
\begin{equation}
(x-0.5)^2+(y-0.5)^2+(z-0.5)^2-1 =0
\end{equation}

## Intersection of two spheres
System of implicit equtions
\begin{align}
x^2+y^2+z^2-1 & =0  \\
(x-0.5)^2+(y-0.5)^2+(z-0.5)^2-1 & =0
\end{align}

```julia
using LinearAlgebra
axes=[-2:2,-2:2,-2:2]
fig = figure(2);clf()

#Sphere1
fS1(x...)=norm([x...],2.0)-1
Sphere1mdbm=MDBM_Problem(fS1,axes)
solve!(Sphere1mdbm,4)
a_sol,b_sol,c_sol=getinterpolatedsolution(Sphere1mdbm)
plot3D(a_sol,b_sol,c_sol,linestyle="", marker=".", markersize=1);

#Sphere2
fS2(x...)=norm([x...] .- 0.5,2.0) -1.0
Sphere2mdbm=MDBM_Problem(fS2,axes)
solve!(Sphere2mdbm,4)
a_sol,b_sol,c_sol=getinterpolatedsolution(Sphere2mdbm)
plot3D(a_sol,b_sol,c_sol,linestyle="", marker=".", markersize=1);

#Intersection
# fS12(x...)=[fS1(x...),fS2(x...)]

function fS12(x,y,z)
    fS1(x,y,z),fS2(x,y,z)
end
Intersectmdbm=MDBM_Problem(fS12,axes)
solve!(Intersectmdbm,6)
a_sol,b_sol,c_sol=getinterpolatedsolution(Intersectmdbm)
plot3D(a_sol,b_sol,c_sol,color="k",linestyle="", marker=".", markersize=2);
```

# 3. Non-smooth problem
## Mandelbrot set

Convergence test of series $z_{i+1}=z_i^2+c$ for $z_0=0$, $c \in  \mathbb{C}$

In the evaluation $c=x + i y$

If $\| z_i \| >2$, then the series will diverge.

```julia
function mandelbrot(x,y)
    c=x+y*1im
    z=Complex(0)
    k=0
    maxiteration=1000
    while (k<maxiteration && abs(z)<4.0)
            z=z^2+c
            k=k+1
    end

    abs(z)-2.0
end

Mandelbrotmdbm=MDBM_Problem(mandelbrot,[-5:2,-2:2])
solve!(Mandelbrotmdbm,8)
real_c_sol,imag_c_sol=getinterpolatedsolution(Mandelbrotmdbm)
fig = figure(3);clf()
plot(real_c_sol,imag_c_sol,linestyle="", marker=".", markersize=1);
```

# 4. Continuation behavioue
## 4.1 Problem:
\begin{equation}
 x,y \in \left[ -2,2 \right]
\end{equation}

\begin{equation}
\| \mathbf{x} +1.5\|_{1.7}+\sin(5x_1)-2 =0
\end{equation}

```julia
using LinearAlgebra
mymdbm=MDBM_Problem((x...)->norm([x...].+ 1.5,1.7)+sin(x[1]*5)-2.0,[-2:2,-2:2])

interpolate!(mymdbm,interpolationorder=1)
for k=1:5
    refine!(mymdbm)
    interpolate!(mymdbm,interpolationorder=1)
end
checkneighbour!(mymdbm)
interpolate!(mymdbm,interpolationorder=1)

#evaluated points
a_eval,b_eval=getevaluatedpoints(mymdbm)

a_sol,b_sol=getinterpolatedsolution(mymdbm)

fig = figure(4)
plot(a_eval,b_eval,linestyle="",  marker=".",markersize=1)
plot(a_sol,b_sol,linestyle="",  marker=".",markersize=1);
```

## 4.2 Exploring the missing component

Due to the coarse initial mesh, some part of the solution is missing.

A continuation-like exploration of the missing component can be performed by checking the neighbouring n-cubes.

It is also clear, that the range of the initial mesh do not cover the object. The actual grid can be prepend and append also.

Note, that the extended grid is used only for the continuation of the detected segments! The initialized new grid points are not evaluated.<br>
In this example there is a closed curve of soultion around $x_1=-4.1$ which is lost!

```julia
# # extension with the same resolution:
# axesextend!(mymdbm,1,prepend=mymdbm.axes[1].ticks[1:end-1] .+(mymdbm.axes[1].ticks[1]- mymdbm.axes[1].ticks[end]));
# axesextend!(mymdbm,2,prepend=mymdbm.axes[2].ticks[1:end-1] .+(mymdbm.axes[2].ticks[1]- mymdbm.axes[2].ticks[end]));

#extension with different reselution
axesextend!(mymdbm,1,prepend=-6.2:0.1:-2.2);
axesextend!(mymdbm,2,prepend=-6.2:0.05:-2.2,append=2.2:0.2:3);

checkneighbour!(mymdbm)
interpolate!(mymdbm,interpolationorder=1)

#evaluated points
a_eval,b_eval=getevaluatedpoints(mymdbm)

#solution points
a_sol,b_sol=getinterpolatedsolution(mymdbm)

# scatter(a_eval,b_eval,markersize=1)
# scatter!(a_sol,b_sol,size = (800, 800),markersize=2,
#      xticks = mymdbm.axes[1].ticks , yticks = mymdbm.axes[2].ticks, gridalpha=0.8)

fig = figure(4)
clf()
plot(a_eval,b_eval,linestyle="",  marker=".",markersize=1)
plot(a_sol,b_sol,linestyle="",  marker=".",markersize=1);
```

## 4.3 Further refinement
The prepended and appended grid has a poor resolution, thus further refinement can be used to increase the resolution.

```julia
for k=1:2
    refine!(mymdbm)
    interpolate!(mymdbm,interpolationorder=1)
end
checkneighbour!(mymdbm)
interpolate!(mymdbm,interpolationorder=1)

#evaluated points
a_eval,b_eval=getevaluatedpoints(mymdbm)

#solution points
a_sol,b_sol=getinterpolatedsolution(mymdbm)

# scatter(a_eval,b_eval,markersize=1)
# scatter!(a_sol,b_sol,size = (800, 800),markersize=2,
#  xticks = mymdbm.axes[1].ticks , yticks = mymdbm.axes[2].ticks, gridalpha=0.8)

fig = figure(4)
clf()
plot(a_eval,b_eval,linestyle="",  marker=".",markersize=1)
plot(a_sol,b_sol,linestyle="",  marker=".",markersize=1);
```

# 5. Constraint only
If only a constraint if provided, then a dens point cloud will be provided wich could be used for later computataions.
## Problem:
\begin{equation}
 x,y \in \left[ -5,2 \right]
\end{equation}

\begin{equation}
\| \mathbf{x} +1.5\|_{1.7}+\sin(5x_1)-2 <0
\end{equation}

Note, in this case the functinos will be evaluated in all the gridpoint of the interior.

```julia
using LinearAlgebra

ax1=Axis(-5:3.0,"a")
ax2=Axis(-5:3.0,"b")

mymdbm=MDBM_Problem((x...)->[],[ax1,ax2],constraint=(x...) -> -(norm([x...].+ 1.5,1.7)+sin(x[1]*5)-2.0))
# mymdbm=MDBM_Problem((x...) -> -maximum([0.0,-(norm([x...].+ 1.5,1.7)+sin(x[1]*5)-2.0)]),[ax1,ax2])
interpolate!(mymdbm,interpolationorder=1)
for k=1:4
    refine!(mymdbm)
    interpolate!(mymdbm,interpolationorder=1)
end
checkneighbour!(mymdbm)
interpolate!(mymdbm,interpolationorder=1)

#solution points
a_sol,b_sol=getinterpolatedsolution(mymdbm)

F_sol=map((x,y)->x*x-y*y,a_sol,b_sol)
# scatter(a_sol,b_sol,F_sol,size = (500, 500))

fig = figure(5)
plot3D(a_sol,b_sol,F_sol,linestyle="",  marker=".",markersize=4);
```

# 6. Connection of point could
The generated points coloud can be connected by generatigh the eges between the neighbouring n-cubes.
## Problem:

\begin{align}
 x,y,z & \in \left[ -5,5 \right] \\
x^2+y^2+z^2-4 & =0 \\
z+y-\cos(3x) & >0
\end{align}

```julia
ax1=Axis(-3.0:3.0,"x")
ax2=Axis(-3.0:3.0,"y")
ax3=Axis(-3.0:3.0,"z")

function foo(x,y,z)
    [x^2+y^2+z^2-4]
end

function cont(x,y,z)
    [z+y-cos(3*x)-1]
end

mymdbm=MDBM_Problem(foo,[ax1,ax2,ax3],constraint=cont)
solve!(mymdbm,2)

#solution points
x_sol,y_sol,z_sol=getinterpolatedsolution(mymdbm);
fig = figure(10),clf()
plot3D(x_sol,y_sol,z_sol,linestyle="",  marker=".",markersize=4);

#connecting the neighbouring n-cubes
myDT1=connect(mymdbm);

#plot solution lines one-by-one
fig = figure(11);clf()
for i in 1:length(myDT1)
    dt=myDT1[i]
    P1=getinterpolatedsolution(mymdbm.ncubes[dt[1]],mymdbm)
    P2=getinterpolatedsolution(mymdbm.ncubes[dt[2]],mymdbm)
    plot3D([P1[1],P2[1]],[P1[2],P2[2]],[P1[3],P2[3]], color="k")
end
```

# 7. Complex problem: 4 parameter, 2 equation, 1 constraint
## Problem:

\begin{align}
 x,y,z \in \left[ -3,3 \right]  & \quad r \in \left[ 1,2 \right] \\
x^2+y^2+z^2-r^2 & =0 \\
z+y-\cos(3x) & =0 \\
z-\sin(5y) & >0
\end{align}

```julia
ax1=Axis(-3.0:3.0,"x")
ax2=Axis(-3.0:3.0,"y")
ax3=Axis(-3.0:3.0,"z")
ax4=Axis(1.0:0.5:2.0,"r")

function foo(x,y,z,r)
    [x^2+y^2+z^2-r^2,
        z+y-cos(3*x)]
end

function cont(x,y,z,r)
   z-sin(5*y)
end

mymdbm=MDBM_Problem(foo,[ax1,ax2,ax3,ax4],constraint=cont)
solve!(mymdbm,3)

#solution points
x_sol,y_sol,z_sol,r_sol=getinterpolatedsolution(mymdbm);
fig = figure(20);clf()
plot3D(x_sol,y_sol,z_sol,linestyle="",  marker=".",markersize=4);
```


# Multiple Scales Stability for forced Duffing System

Consider

$$
\begin{gather*}
    \ddot{x} + \omega_0^2 x = -2\epsilon\mu\dot{x} - \epsilon\alpha x^3 + K\cos\Omega t
\end{gather*}
$$

To get the linear terms, introduce epsilon for later Taylor expansion

```python
import sympy as sp
from math import factorial
```

```python
a0, a1 = sp.symbols('a_0 a_1', real=True)
epsilon = sp.Symbol('epsilon', real=True, positive=True)
a = a0 + epsilon * a1
a
```

$\displaystyle a_{0} + a_{1} \epsilon$

```python
Gamma0, Gamma1 = sp.symbols('Gamma_0 Gamma_1', real=True)
Gamma = Gamma0 + epsilon * Gamma1
Gamma
```

$\displaystyle \Gamma_{0} + \Gamma_{1} \epsilon$

```python
def taylor(expr, var, upto):
    res = expr.subs(var, 0)
    dexpr = sp.diff(expr, var)
    for n in range(1,upto+1):
        res += dexpr.subs(var, 0) / factorial(n) * var**n
        dexpr = sp.diff(dexpr, var)
    return res + sp.O(var**upto)

taylor(a**3, epsilon, 4)
```

$\displaystyle a_{1}^{3} \epsilon^{3} + 3 a_{0} a_{1}^{2} \epsilon^{2} + 3 a_{0}^{2} a_{1} \epsilon + a_{0}^{3} + O\left(\epsilon^{4}\right)$

```python
taylor(1/a*sp.cos(Gamma), epsilon, 6)
```

$\displaystyle \frac{\cos{\left(\Gamma_{0} \right)}}{a_{0}} + \epsilon \left(- \frac{\Gamma_{1} \sin{\left(\Gamma_{0} \right)}}{a_{0}} - \frac{a_{1} \cos{\left(\Gamma_{0} \right)}}{a_{0}^{2}}\right) + \epsilon^{2} \left(- \frac{\Gamma_{1}^{2} \cos{\left(\Gamma_{0} \right)}}{2 a_{0}} + \frac{\Gamma_{1} a_{1} \sin{\left(\Gamma_{0} \right)}}{a_{0}^{2}} + \frac{a_{1}^{2} \cos{\left(\Gamma_{0} \right)}}{a_{0}^{3}}\right) + \epsilon^{3} \left(\frac{\Gamma_{1}^{3} \sin{\left(\Gamma_{0} \right)}}{6 a_{0}} + \frac{\Gamma_{1}^{2} a_{1} \cos{\left(\Gamma_{0} \right)}}{2 a_{0}^{2}} - \frac{\Gamma_{1} a_{1}^{2} \sin{\left(\Gamma_{0} \right)}}{a_{0}^{3}} - \frac{a_{1}^{3} \cos{\left(\Gamma_{0} \right)}}{a_{0}^{4}}\right) + \epsilon^{4} \left(\frac{\Gamma_{1}^{4} \cos{\left(\Gamma_{0} \right)}}{24 a_{0}} - \frac{\Gamma_{1}^{3} a_{1} \sin{\left(\Gamma_{0} \right)}}{6 a_{0}^{2}} - \frac{\Gamma_{1}^{2} a_{1}^{2} \cos{\left(\Gamma_{0} \right)}}{2 a_{0}^{3}} + \frac{\Gamma_{1} a_{1}^{3} \sin{\left(\Gamma_{0} \right)}}{a_{0}^{4}} + \frac{a_{1}^{4} \cos{\left(\Gamma_{0} \right)}}{a_{0}^{5}}\right) + \epsilon^{5} \left(- \frac{\Gamma_{1}^{5} \sin{\left(\Gamma_{0} \right)}}{120 a_{0}} - \frac{\Gamma_{1}^{4} a_{1} \cos{\left(\Gamma_{0} \right)}}{24 a_{0}^{2}} + \frac{\Gamma_{1}^{3} a_{1}^{2} \sin{\left(\Gamma_{0} \right)}}{6 a_{0}^{3}} + \frac{\Gamma_{1}^{2} a_{1}^{3} \cos{\left(\Gamma_{0} \right)}}{2 a_{0}^{4}} - \frac{\Gamma_{1} a_{1}^{4} \sin{\left(\Gamma_{0} \right)}}{a_{0}^{5}} - \frac{a_{1}^{5} \cos{\left(\Gamma_{0} \right)}}{a_{0}^{6}}\right) + O\left(\epsilon^{6}\right)$

stable when q is greater than zero

```python
mu, sigma, alpha, omega0 = sp.symbols('mu sigma alpha omega_0', real=True)
q = mu**2 + (sigma - 3 * alpha * a0**2 / 8 / omega0) * (sigma - 9 * alpha * a0**2 / 8 / omega0)
q
```

$\displaystyle \mu^{2} + \left(- \frac{9 a_{0}^{2} \alpha}{8 \omega_{0}} + \sigma\right) \left(- \frac{3 a_{0}^{2} \alpha}{8 \omega_{0}} + \sigma\right)$

Substitute in $\sigma$ as a function of $a_0$ into $q$ and find when $q= 0$ (see inverted frequency response diagram)

This is the top branch

```python
k = sp.Symbol('k')
topBranchCriterium = sp.simplify(q.subs([
    (sigma, 3/8*alpha/omega0*a0**2 + sp.sqrt(k**2 / 4 / omega0**2 / a0**2 - mu**2))
]))
topBranchCriterium = sp.Eq(topBranchCriterium, 0)
topBranchCriterium
```

$\displaystyle \frac{0.25 \left(- 1.5 a_{0}^{2} \alpha \sqrt{- 4 \mu^{2} + \frac{k^{2}}{a_{0}^{2} \omega_{0}^{2}}} + \frac{1.0 k^{2}}{a_{0}^{2} \omega_{0}}\right)}{\omega_{0}} = 0$

This is the bottom branch

```python
bottomBranchCriterium = sp.simplify(q.subs([
    (sigma, 3/8*alpha/omega0*a0**2 - sp.sqrt(k**2 / 4 / omega0**2 / a0**2 - mu**2))
]))
bottomBranchCriterium = sp.Eq(bottomBranchCriterium, 0)
bottomBranchCriterium
```

$\displaystyle \frac{0.25 \left(1.5 a_{0}^{2} \alpha \sqrt{- 4 \mu^{2} + \frac{k^{2}}{a_{0}^{2} \omega_{0}^{2}}} + \frac{1.0 k^{2}}{a_{0}^{2} \omega_{0}}\right)}{\omega_{0}} = 0$

```python
temp = sp.Eq(
    (1.5*a0**2*alpha * sp.sqrt(-4*mu + k**2 / a0**2 / omega0**2))**2,
    (-k**2 / a0**2 / omega0)**2
)
sp.solve(temp, a0)[0]
```

$\displaystyle - \begin{cases} 0.467327632592034 \sqrt{\frac{0.286178560638333 k^{2}}{\mu \omega_{0}^{2}} - \left(\frac{0.081898168569028 k^{4}}{\mu^{2} \omega_{0}^{4}} - \left(- \frac{k^{8}}{\alpha^{2} \mu^{3} \omega_{0}^{6}}\right)^{0.333333333333333}\right)^{0.5} - \left(- \frac{0.046875 k^{6}}{\mu^{3} \omega_{0}^{6} \left(\frac{0.081898168569028 k^{4}}{\mu^{2} \omega_{0}^{4}} - \left(- \frac{k^{8}}{\alpha^{2} \mu^{3} \omega_{0}^{6}}\right)^{0.333333333333333}\right)^{0.5}} + \frac{0.163796337138056 k^{4}}{\mu^{2} \omega_{0}^{4}} + \left(- \frac{k^{8}}{\alpha^{2} \mu^{3} \omega_{0}^{6}}\right)^{0.333333333333333}\right)^{0.5}} & \text{for}\: \frac{0.111111111111111 k^{4}}{\alpha^{2} \mu \omega_{0}^{2}} = 0.0 \\0.556957466689064 \sqrt{\frac{0.201481862216914 k^{2}}{\mu \omega_{0}^{2}} - 1.0 \left(\frac{0.0405949408023956 k^{4}}{\mu^{2} \omega_{0}^{4}} + 1.0 \left(\left(\frac{0.0037078857421875 k^{16}}{\alpha^{4} \mu^{6} \omega_{0}^{12}} - \frac{k^{12}}{\alpha^{6} \mu^{3} \omega_{0}^{6}}\right)^{0.5} + \frac{0.0608924112035933 k^{8}}{\alpha^{2} \mu^{3} \omega_{0}^{6}}\right)^{0.333333333333333} + \frac{k^{4}}{\alpha^{2} \mu \omega_{0}^{2} \left(\left(\frac{0.0037078857421875 k^{16}}{\alpha^{4} \mu^{6} \omega_{0}^{12}} - \frac{k^{12}}{\alpha^{6} \mu^{3} \omega_{0}^{6}}\right)^{0.5} + \frac{0.0608924112035933 k^{8}}{\alpha^{2} \mu^{3} \omega_{0}^{6}}\right)^{0.333333333333333}}\right)^{0.5} - \left(- \frac{0.0163582885389041 k^{6}}{\mu^{3} \omega_{0}^{6} \left(\frac{0.0405949408023956 k^{4}}{\mu^{2} \omega_{0}^{4}} + 1.0 \left(\left(\frac{0.0037078857421875 k^{16}}{\alpha^{4} \mu^{6} \omega_{0}^{12}} - \frac{k^{12}}{\alpha^{6} \mu^{3} \omega_{0}^{6}}\right)^{0.5} + \frac{0.0608924112035933 k^{8}}{\alpha^{2} \mu^{3} \omega_{0}^{6}}\right)^{0.333333333333333} + \frac{k^{4}}{\alpha^{2} \mu \omega_{0}^{2} \left(\left(\frac{0.0037078857421875 k^{16}}{\alpha^{4} \mu^{6} \omega_{0}^{12}} - \frac{k^{12}}{\alpha^{6} \mu^{3} \omega_{0}^{6}}\right)^{0.5} + \frac{0.0608924112035933 k^{8}}{\alpha^{2} \mu^{3} \omega_{0}^{6}}\right)^{0.333333333333333}}\right)^{0.5}} + \frac{0.0811898816047911 k^{4}}{\mu^{2} \omega_{0}^{4}} - 1.0 \left(\left(\frac{0.0037078857421875 k^{16}}{\alpha^{4} \mu^{6} \omega_{0}^{12}} - \frac{k^{12}}{\alpha^{6} \mu^{3} \omega_{0}^{6}}\right)^{0.5} + \frac{0.0608924112035933 k^{8}}{\alpha^{2} \mu^{3} \omega_{0}^{6}}\right)^{0.333333333333333} - \frac{k^{4}}{\alpha^{2} \mu \omega_{0}^{2} \left(\left(\frac{0.0037078857421875 k^{16}}{\alpha^{4} \mu^{6} \omega_{0}^{12}} - \frac{k^{12}}{\alpha^{6} \mu^{3} \omega_{0}^{6}}\right)^{0.5} + \frac{0.0608924112035933 k^{8}}{\alpha^{2} \mu^{3} \omega_{0}^{6}}\right)^{0.333333333333333}}\right)^{0.5}} & \text{otherwise} \end{cases}$

Equation for sigma - using plus sign

```python
sigma_func = 3*alpha/omega0*a0**2/8 + sp.sqrt(k**2 / 4 / omega0**2 / a0**2 - mu**2)
sigma_func
```

$\displaystyle \frac{3 a_{0}^{2} \alpha}{8 \omega_{0}} + \sqrt{- \mu^{2} + \frac{k^{2}}{4 a_{0}^{2} \omega_{0}^{2}}}$

Search for $a_0$ values such that the slope is zero - notice this matches the solutions above

```python
sp.solve(sp.diff(sigma_func, a0), a0)

# not solvable with sympy
# but let's assume that it is the same as the huge answer shown above
```


# Playing with coupled oscillators - v3

This notebook builds on previous notebooks such as Orbit_games.ipynb to implement the solution of linear coupled differential equations using `solve_ivp` as well as with more analytic methods.  The formalism and notation is based on Taylor, Chapter 11: *Coupled Oscillators and Normal Modes*.

The general form of the equations to solve is compactly expressed in matrix form:

$\begin{align}
  \mathbf{M} \mathbf{\ddot q} = - \mathbf{K} \mathbf{q}
\end{align}$

where $\mathbf{M}$ and $\mathbf{K}$ are $N \times N$ symmetric matrices and $\mathbf{q}$ is a vector of generalized coordinates (this is $\mathbf{x}$ in the beginning discussion in Taylor).

We can solve this directly as differential equations or solve for normal modes.  For the former, we map the problem onto linear differential equations and the vectors $\mathbf{y}$ and $d\mathbf{y}/dt$, which are each of length $2N$.  Here is the $N=2$ mapping:

$\begin{align}
  \mathbf{y} = \left( \begin{array}{c}
                         q_1 \\
                         q_2 \\
                         \dot q_1 \\
                         \dot q_2
                      \end{array}
                       \right)
 \qquad\qquad
\frac{d\mathbf{y}}{dt} = \left( \begin{array}{c}
                         \dot q_1 \\
                         \dot q_2 \\
                         \ddot q_1 \\
                         \ddot q_2
                      \end{array}
                       \right)
\end{align}$

Then for the $\ddot q$ parts we use the matrix solution of our equation:

$\begin{align}
  \mathbf{\ddot q} = - \mathbf{M}^{-1} \mathbf{K} \mathbf{q}
\end{align}$

where $\mathbf{M}^{-1}$ means the matrix inverse.  Check out the implementation below!

To solve for normal modes, we solve the generalized eigenvalue problem

$\begin{align}
  (\mathbf{K} - \omega^2 \mathbf{M}) \mathbf{a} = 0
\end{align}$

using functions from `scipy.linalg` to find the eigenvalues and eigenvectors.  We use these below together with the numerical solutions to make plots of the normal modes.

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import solve_ivp
from scipy import linalg as LA
```

```python
# Change the common font size
font_size = 14
plt.rcParams.update({'font.size': font_size})
```

```python
class CoupledOscillators:
    """
    Solve the equations for linear coupled oscillators in matrix formulation.
    """

    def __init__(self, M_matrix, K_matrix, b_matrix=np.zeros(1)):
        self.M_matrix = M_matrix
        self.K_matrix = K_matrix
        self.q_len = len(self.M_matrix)
        if b_matrix.any():
            self.b_matrix = b_matrix
        else:
            self.b_matrix = np.zeros((self.q_len))
        self.full_matrix = np.zeros((self.q_len, self.q_len))

    def dy_dt(self, t, y):
        """
        This function returns the right-hand side of the diffeq:
        [dq_vec/dt d^2q_vec/dt^2]

        Parameters
        ----------
        t : float
            time
        y : float
            vector with y[:q_len] = q_vec, y[q_len:] = dqvec/dt

        """
        dy_dt_vec = np.zeros(2*self.q_len)   # Initialize to twice q_vec size
        # Set the upper half [:self.q_len] of dy_dt to the lower half of y
        dy_dt_vec[:self.q_len] = y[self.q_len:]
        # Set the lower half [self.q_len:] of dy_dt to the vector that
        #  results from M^{-1} K y where @ is matrix multiplication
        dy_dt_vec[self.q_len:] = -1.* LA.inv(self.M_matrix) @ \
                    ( self.K_matrix @ y[:self.q_len] + \
                      self.b_matrix @ y[self.q_len:] )
        return dy_dt_vec

    def solve_ode(self, t_pts, q_vec_0, q_vec_dot_0,
                  method='RK45',
                  abserr=1.0e-10, relerr=1.0e-10):
        """
        Solve the ODE given initial conditions.
        Use solve_ivp with the option of specifying the method.
        Specify smaller abserr and relerr to get more precision.
        """
        y = np.concatenate((q_vec_0, q_vec_dot_0))
        solution = solve_ivp(self.dy_dt, (t_pts[0], t_pts[-1]),
                             y, t_eval=t_pts, method=method,
                             atol=abserr, rtol=relerr)
        q_vec, q_vec_dot = np.array_split(solution.y, 2)
        return q_vec, q_vec_dot

    def find_eigenmodes(self):
        """
        Find the normal modes.
        """
        eig_vals, eig_vecs = LA.eigh(K_matrix, M_matrix)
        self.frequencies = np.sqrt(eig_vals)
        return eig_vals, eig_vecs

    def plot_eigenmodes(self, t_pts, output_file=None):
        """
        plot the normal modes separately
        """
        from itertools import cycle
        colors = ['blue', 'green', 'red', 'purple']

        eig_vals, eig_vecs = self.find_eigenmodes()
        for i in np.arange(self.q_len):
            q_vec_0 = eig_vecs[:,i]   # this is the i'th eigenvector
            q_vec_dot_0 = np.zeros(self.q_len)  # start with all qdots = 0
            q_vec, q_vec_dot = self.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

            fig, axes = plt.subplots(self.q_len, 1,
                                     figsize=(10, 2.*self.q_len))
            color_list = cycle(colors)
            for j in np.arange(self.q_len):
               axes[j].plot(t_pts, q_vec[j], color=next(color_list))
               axes[j].set_xlabel(r'$t$')
               axes[j].set_ylabel(fr'$q_{j+1:d}$')
               axes[j].axvline(2.*np.pi/np.sqrt(eig_vals[i]),
                               color='black', linestyle=':')
            overall_title = fr'Normal mode {i+1:d}, ' + \
                            fr'$\omega = {self.frequencies[i]:.2f}$'
            fig.suptitle(overall_title, va='center')
            fig.tight_layout()
            if output_file:
                 output_file_name = output_file.replace('.', f'_{i:d}.', 1)
                 fig.savefig(output_file_name, dpi=200, bbox_inches='tight')

```

```python
def start_stop_indices(t_pts, plot_start, plot_stop):
    start_index = (np.fabs(t_pts-plot_start)).argmin()  # index in t_pts array
    stop_index = (np.fabs(t_pts-plot_stop)).argmin()  # index in t_pts array
    return start_index, stop_index
```

## Section 11.2

```python
m = 1.
M_matrix = np.array([ [m, 0., 0.], [0., m, 0.], [0., 0., m] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
K_matrix = np.array([ [2.*k, -k], [-k, 2.*k]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

co1 = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = co1.find_eigenmodes() # LA.eigh(K_matrix, M_matrix)

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {co1.frequencies}')
```

    M matrix:
     [[1. 0.]
     [0. 1.]]

    K matrix:
     [[ 2. -1.]
     [-1.  2.]]

    omega^2 = [1. 3.]
    frequencies = [1.         1.73205081]

```python
print(eig_vecs[:,0])
print(eig_vecs[:,1])
```

    [-0.70710678 -0.70710678]
    [-0.70710678  0.70710678]

```python
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
```

```python
co1.plot_eigenmodes(t_pts, 'Section11p2.png')
```

### Figure 11.6

This has $A_1 = 1$, $A_2 = 0.7$, $\delta_1 = 0$ and $\delta_2 = \pi/2$.

```python
t_start = 0.
t_end = 50.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
```

```python
colors = ['blue', 'green', 'red', 'purple']

# changing A_i -> -A_i to match convention in Figure 11.6
A_1 = -1.
delta_1 = 0.
A_2 = -0.7
delta_2 = np.pi/2.

omega_1 = co1.frequencies[0]
omega_2 = co1.frequencies[1]

q_vec_0 = A_1 * eig_vecs[:,0] * np.cos(-delta_1) + \
          A_2 * eig_vecs[:,1] * np.cos(-delta_2)
q_vec_dot_0 = omega_1 * A_1 * eig_vecs[:,0] * np.sin(delta_1) + \
              omega_2 * A_2 * eig_vecs[:,1] * np.sin(delta_2)
q_vec_11p6, q_vec_dot_11p6 = co1.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

fig, axes = plt.subplots(co1.q_len, 1, figsize=(10, 2.*co1.q_len))
for j in np.arange(co1.q_len):
    axes[j].plot(t_pts, q_vec_11p6[j], color=colors[j])
    axes[j].set_xlabel(r'$t$')
    axes[j].set_ylabel(fr'$q_{j+1:d}$')
    axes[j].set_xlim(t_pts[0], t_pts[-1])
    axes[j].axhline(0., color='black', alpha=0.3)
overall_title = 'General solution: ' + fr'$A_1 = {A_1:.2f}$, ' + \
                                       fr'$\delta_1 = {delta_1:.2f}$, ' + \
                                       fr'$A_2 = {A_2:.2f}$, ' + \
                                       fr'$\delta_2 = {delta_2:.2f}$'
fig.suptitle(overall_title, va='center')
fig.tight_layout()

print( q_vec_0, q_vec_dot_0 )  # debugging check
```

## Problem 11.5

Almost the same as section 11.2, but mass m2 is not attached by a spring to the wall.

```python
m = 1.
M_matrix = np.array([ [m, 0.], [0., m] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
K_matrix = np.array([ [2.*k, -k], [-k, k]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

co1 = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = co1.find_eigenmodes() # LA.eigh(K_matrix, M_matrix)

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {co1.frequencies}')
```

    M matrix:
     [[1. 0.]
     [0. 1.]]

    K matrix:
     [[ 2. -1.]
     [-1.  1.]]

    omega^2 = [0.38196601 2.61803399]
    frequencies = [0.61803399 1.61803399]

```python
print(eig_vecs[:,0])
print(eig_vecs[:,1])
```

    [-0.52573111 -0.85065081]
    [-0.85065081  0.52573111]

```python
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
co1.plot_eigenmodes(t_pts)
```

## Problem 11.7

Writing the general solution to the cart problem in Section 11.2 in a different form and plot two solutions.

```python
m = 1.
M_matrix = np.array([ [m, 0.], [0., m] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
K_matrix = np.array([ [2.*k, -k], [-k, 2.*k]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

co1 = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = co1.find_eigenmodes() # LA.eigh(K_matrix, M_matrix)

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {co1.frequencies}')
```

    M matrix:
     [[1. 0.]
     [0. 1.]]

    K matrix:
     [[ 2. -1.]
     [-1.  2.]]

    omega^2 = [1. 3.]
    frequencies = [1.         1.73205081]

```python
t_start = 0.
t_end = 30.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
```

```python
colors = ['blue', 'green', 'red', 'purple']

# changing A -> -A to match convention in Taylor
A = -1.
B_1 = A
B_2 = 0.
C_1 = 0.
C_2 = 0.

omega_1 = co1.frequencies[0]
omega_2 = co1.frequencies[1]

q_vec_0 = B_1 * eig_vecs[:,0] + \
          B_2 * eig_vecs[:,1]
q_vec_dot_0 = omega_1 * C_1 * eig_vecs[:,0] + \
              omega_2 * C_2 * eig_vecs[:,1]
q_vec_11p7a, q_vec_dot_11p7a = co1.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

fig, axes = plt.subplots(co1.q_len, 1, figsize=(10, 2.*co1.q_len))
for j in np.arange(co1.q_len):
    axes[j].plot(t_pts, q_vec_11p7a[j], color=colors[j])
    axes[j].set_xlabel(r'$t$')
    axes[j].set_ylabel(fr'$q_{j+1:d}$')
    axes[j].set_xlim(t_pts[0], t_pts[-1])
    axes[j].axhline(0., color='black', alpha=0.3)
overall_title = 'Specific solution 1: ' + fr'$B_1 = {B_1:.2f}$, ' + \
                                          fr'$C_1 = {C_1:.2f}$, ' + \
                                          fr'$B_2 = {B_2:.2f}$, ' + \
                                          fr'$C_2 = {C_2:.2f}$'
fig.suptitle(overall_title, va='center')
fig.tight_layout()

print( q_vec_0, q_vec_dot_0 )  # debugging check
```

```python
colors = ['blue', 'green', 'red', 'purple']

# changing A -> -A to match convention in Taylor
A = -1.
B_1 = A / 2.
B_2 = A / 2.
C_1 = 0.
C_2 = 0.

omega_1 = co1.frequencies[0]
omega_2 = co1.frequencies[1]

q_vec_0 = B_1 * eig_vecs[:,0] + \
          B_2 * eig_vecs[:,1]
q_vec_dot_0 = omega_1 * C_1 * eig_vecs[:,0] + \
              omega_2 * C_2 * eig_vecs[:,1]
q_vec_11p7a, q_vec_dot_11p7a = co1.solve_ode(t_pts, q_vec_0, q_vec_dot_0)

fig, axes = plt.subplots(co1.q_len, 1, figsize=(10, 2.*co1.q_len))
for j in np.arange(co1.q_len):
    axes[j].plot(t_pts, q_vec_11p7a[j], color=colors[j])
    axes[j].set_xlabel(r'$t$')
    axes[j].set_ylabel(fr'$q_{j+1:d}$')
    axes[j].set_xlim(t_pts[0], t_pts[-1])
    axes[j].axhline(0., color='black', alpha=0.3)
overall_title = 'Specific solution 1: ' + fr'$B_1 = {B_1:.2f}$, ' + \
                                          fr'$C_1 = {C_1:.2f}$, ' + \
                                          fr'$B_2 = {B_2:.2f}$, ' + \
                                          fr'$C_2 = {C_2:.2f}$'
fig.suptitle(overall_title, va='center')
fig.tight_layout()

print( q_vec_0, q_vec_dot_0 )  # debugging check
```

## Double pendulum with small oscillations

```python
m = 1.
L = 1.
M_matrix = m * L**2 * np.array([ [2., 1.], [1., 1.] ])
print('M matrix: \n', M_matrix, '\n')

k = 1.
omega_0 = 1.
K_matrix = m * L**2 * np.array([ [2.*omega_0**2, 0.], [0., omega_0**2]])
print('K matrix: \n', K_matrix, '\n')

# b = 0
# b_matrix = np.array([ [b, 0], [0, b]])
# print('b matrix: \n', b_matrix, '\n')

double_pendulum = CoupledOscillators(M_matrix, K_matrix)

eig_vals, eig_vecs = double_pendulum.find_eigenmodes()

print(f'omega^2 = {eig_vals}')
print(f'frequencies = {double_pendulum.frequencies}')
print(f'{2. + np.sqrt(2.):.4f}  {2. - np.sqrt(2.):.4f}')
```

    M matrix:
     [[2. 1.]
     [1. 1.]]

    K matrix:
     [[2. 0.]
     [0. 1.]]

    omega^2 = [0.58578644 3.41421356]
    frequencies = [0.76536686 1.84775907]
    3.4142  0.5858

```python
print(eig_vecs[:,0])
print(eig_vecs[:,1])
```

    [-0.38268343 -0.5411961 ]
    [-0.92387953  1.30656296]

```python
t_start = 0.
t_end = 10.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)
double_pendulum.plot_eigenmodes(t_pts)
```

```python

```