# Approximate solvers for the shallow water equations

In this chapter we discuss approximate Riemann solvers for the shallow water equations:
\begin{align}
    h_t + (hu)_x & = 0 \label{SWA_mass} \\
    (hu)_t + \left(hu^2 + \frac{1}{2}gh^2\right)_x & = 0. \label{SWA_mom}
\end{align}
We investigated the exact solution of the Riemann problem for this system in [Shallow_water.ipynb](Shallow_water.ipynb); code for the exact solution is in [exact_solvers/shallow_water.py](exact_solvers/shallow_water.py).  We found that the system has two genuinely nonlinear characteristic fields, so the solution consists of two waves, each of which may be a shock or a rarefaction.

Recall that the flux Jacobian is
\begin{align}
f'(q) & = \begin{pmatrix} 0 & 1 \\ -(q_2/q_1)^2 + g q_1 & 2 q_2/q_1 \end{pmatrix}
        = \begin{pmatrix} 0 & 1 \\ -u^2 + g h & 2 u \end{pmatrix}.
\end{align}
Its eigenvalues are
\begin{align} \label{SWA:char-speeds}
    \lambda_1 & = u - \sqrt{gh} & \lambda_2 & = u + \sqrt{gh},
\end{align}
with corresponding eigenvectors
\begin{align} \label{SWA:fjac-evecs}
    r_1 & = \begin{bmatrix} 1 \\ u-\sqrt{gh} \end{bmatrix} &
    r_2 & = \begin{bmatrix} 1 \\ u+\sqrt{gh} \end{bmatrix}.
\end{align}

## Roe solver

To define a linearized Riemann solver, we consider the quasilinear form $q_t + f'(q)q_x=0$, and replace the Jacobian $f'(q)$ by some matrix $\hat{A}(q_l,q_r)$ depending on the left and right states.  This matrix should satisfy three properties:

1. Consistency: $\hat{A}(q_l,q_r) \to f'(q)$ as $q_l, q_r \to q$.
2. Hyperbolicity: $\hat{A}$ must be diagonalizable with real eigenvalues, so that we can define the waves and speeds needed in the approximate solution.
3. Conservation: $\hat{A}(q_l,q_r)(q_r - q_l) = f(q_r) - f(q_l)$.

The first two properties are essential for any linearized approximate Riemann solver.  As mentioned in [Burgers_approximate.ipynb](Burgers_approximate.ipynb), the third property is the defining property of a Roe solver.  It ensures that if $q_r$ and $q_l$ can be connected by a single shock (i.e., if they lie on a common Hugoniot locus), then the approximate solver will in fact yield the exact solution -- a single discontinuity propagating at the speed given by the Rankine-Hugoniot jump condition.  This property is quite useful, since at most points (in space and time) where a shock appears, it is an isolated shock corresponding to one characteristic family.  Only at isolated points in space and time do multiple shocks interact.

Here we derive the Roe solver for the shallow water equations in a simple and straightforward way by applying the conservation requirement.  Roe's original paper (Roe, 1981) (which dealt with the Euler equations rather than the shallow water equations) gave a different and possibly more general derivation that is also reviewed in Chapter 15 of (LeVeque, 2002).

One natural approach is to look for a matrix $\hat{A}$ that is the flux Jacobian evaluated at some average state:
$$\hat{A}(q_l,q_r) = f'(\hat{q}).$$
This guarantees property 2, and will ensure property 1 as long as $\hat{q}$ is chosen as some kind of average of $q_l$ and $q_r$.
For the shallow water system, this means that the linearized equations take the form
\begin{align} \label{SWA_quasilinear}
h_t + (hu)_x & = 0 \\
(hu)_t + (g\hat{h}-\hat{u}^2)h_x + 2\hat{u}(hu)_x & = 0. \nonumber
\end{align}
Observe that (\ref{SWA_quasilinear}) is linear *not in the physical variables $h, u$, but in the conserved variables $h, hu$*.  In the context of a Riemann solver, $\hat{q}$ will be some kind of average of the initial states $q_l, q_r$.

The characteristic speeds of the linear hyperbolic system (\ref{SWA_quasilinear}) are just the characteristic speeds of the original shallow water system, evaluated at the average state $\hat{q}$:
\begin{align} \label{SWA:avg-char-speeds}
    s_1 & = \hat{u} - \sqrt{g\hat{h}} & s_2 & = \hat{u} + \sqrt{g\hat{h}},
\end{align}
with corresponding eigenvectors
\begin{align} \label{SWA:avg-fjac-evecs}
    \hat{r}_1 = r_1(\hat{q}) & = \begin{bmatrix} 1 \\ \hat{u}-\sqrt{g\hat{h}} \end{bmatrix} &
    \hat{r}_2 = r_2(\hat{q}) & = \begin{bmatrix} 1 \\ \hat{u}+\sqrt{g\hat{h}} \end{bmatrix}.
\end{align}

The linearized solver simply decomposes the jump $q_r-q_l$ in terms
of the eigenvectors, to define the waves:
\begin{align} \label{SWA:wave-decomp}
q_r - q_l = \alpha_1 \hat{r}_1 + \alpha_2 \hat{r}_2 = {\mathcal W}_1 + {\mathcal W}_2
\end{align}
and uses the speeds given by (\ref{SWA:avg-char-speeds}).  We refer to $\alpha_1, \alpha_2$ as the
wave strengths; they can be found by solving the system above, which amounts to
\begin{align}
\alpha = \hat{R}^{-1}(q_r-q_l),
\end{align}
where $\hat{R} = (\hat{r}_1 | \hat{r}_2)$ is the matrix of right eigenvectors, and thus $\hat{R}^{-1}$ is the matrix whose rows are the left eigenvectors.  This leads to the expressions
\begin{align}
\alpha_1 & = \frac{(\hat{u}+ \hat{c}) \delta_1 - \delta_2}{2\hat{c}} \\
\alpha_2 & = \frac{-(\hat{u} - \hat{c}) \delta_1 + \delta_2}{2\hat{c}},
\end{align}
where $\hat{c} = \sqrt{g\hat{h}}$ and $\delta = q_r -q_l$.

### Derivation of Roe averages
It remains only to specify the average state $(\hat{h},\hat{u})$.  We do so by solving the conservation condition:
\begin{align} \label{SWA:roe_cons}
f'(\hat{q}) (q_r - q_l) & = f(q_r) - f(q_l).
\end{align}
For convenience we recall here the relevant quantities:
\begin{align}
f'(\hat{q}) & = \begin{pmatrix} 0 & 1 \\ -\hat{u}^2 + g \hat{h} & 2 \hat{u} \end{pmatrix},
\end{align}
\begin{align*}
q_r - q_l & = \begin{pmatrix} h_r - h_l \\ h_r u_r - h_l u_l \end{pmatrix},
& f(q_r) - f(q_l) & = \begin{pmatrix} h_r u_r - h_l u_l \\ h_r u_r^2 - h_l u_l^2 + \frac{g}{2}(h_r^2-h_l^2)\end{pmatrix}.
\end{align*}
The first equation of (\ref{SWA:roe_cons}) is an identity, satisfied no matter how $\hat{u}$ and $\hat{h}$ are chosen.  We therefore turn to the second equation:
\begin{align} \label{SWA:2nd}
    (-\hat{u}^2 + g\hat{h})(h_r-h_l) + 2\hat{u}(h_r u_r - h_l u_l) & = h_r u_r^2 - h_l u_l^2 + \frac{g}{2}(h_r^2-h_l^2).
\end{align}
Since (\ref{SWA:2nd}) must hold for any value of $g$, we can equate separately the terms involving $g$ and those not involving $g$.  For the terms with $g$ in them, we get:
\begin{align}  \label{SWA:h_roe}
\hat{h}(h_r-h_l) & = \frac{1}{2} (h_r^2 - h_l^2),
\end{align}
which gives the arithmetic average
$$\hat{h} = \frac{h_r + h_l}{2}.$$
Equating terms that do not involve $g$, we get a quadratic equation for $\hat{u}$:
$$(h_r-h_l)\hat{u}^2 - 2(h_r u_r - h_l u_l) \hat{u} + h_r u_r^2 - h_l u_l^2 = 0.$$
The two roots of this equation are
$$\hat{u}_\pm = \frac{h_r u_r - h_l u_l \mp \sqrt{h_r h_l} (u_l-u_r)}{h_r-h_l}.$$
Simplifying these expressions further, we find
\begin{align} \label{SWA:u_roe}
    \hat{u}_\pm = \frac{\sqrt{h_r}u_r \pm \sqrt{h_l}u_l}{\sqrt{h_r} \pm \sqrt{h_l}},
\end{align}
where the same sign (plus or minus) must be taken in both numerator and denominator.  Either of these expressions can be used for $\hat{u}$, as long as $h_r \ne h_l$.  The value $\hat{u}_+$ is preferable for two reasons: first, it is well defined even when $h_r = h_l$; second, it has the appearance of a weighted average of $u$, and its value always lies between $u_l$ and $u_r$.

The definitions (\ref{SWA:h_roe}) and (\ref{SWA:u_roe}) (with the plus sign) complete the definition of the approximate solver.  An implementation appears below.

```python
%matplotlib inline
```

```python
%config InlineBackend.figure_format = 'svg'

import numpy as np
import matplotlib.pyplot as plt
from exact_solvers import shallow_water as sw
from utils import riemann_tools as rt
from ipywidgets import interact
from ipywidgets import widgets
```

```python
def shallow_water_roe(q_l, q_r, grav=1.):
    """
    Approximate Roe solver for the shallow water equations.
    """

    h_l = q_l[0]
    hu_l = q_l[1]
    u_l = hu_l/h_l
    h_r = q_r[0]
    hu_r = q_r[1]
    u_r = hu_r/h_r

    delta = q_r - q_l

    # Roe averages
    h_hat = (h_r + h_l)/2.
    u_hat = (np.sqrt(h_r)*u_r + np.sqrt(h_l)*u_l) \
                / (np.sqrt(h_r) + np.sqrt(h_l))
    c_hat = np.sqrt(grav*h_hat)

    s1 = u_hat - c_hat
    s2 = u_hat + c_hat

    alpha1 = ( (u_hat+c_hat)*delta[0] - delta[1])/(2*c_hat)
    alpha2 = (-(u_hat-c_hat)*delta[0] + delta[1])/(2*c_hat)

    h_m = q_l[0] + alpha1
    hu_m = q_l[1] + alpha1 * (u_hat - c_hat)
    q_m = np.array([h_m, hu_m])

    states = np.column_stack([q_l,q_m,q_r])
    speeds = [s1, s2]
    wave_types = ['contact','contact']

    def reval(xi):
        h_out  = (xi<s1)*h_l + (s1<=xi)*(xi<=s2)*h_m + (s2<xi)*h_r
        hu_out = (xi<s1)*hu_l + (s1<=xi)*(xi<=s2)*hu_m + (s2<xi)*hu_r
        return h_out, hu_out

    return states, speeds, reval, wave_types
```

Here is an example showing the solution from the Roe solver, compared with the exact Riemann solution.

```python
# Dam-break example
h_l = 4; h_r = 1
u_l = 0; u_r = 0

def make_plot_functions(h_l, u_l, h_r, u_r):
    hu_l = h_l*u_l; hu_r = h_r*u_r
    q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])
    ex_states, ex_speeds, ex_reval, ex_wave_types = \
                                    sw.exact_riemann_solution(q_l,q_r)
    states, speeds, reval, wave_types = shallow_water_roe(q_l, q_r)

    pr = rt.make_plot_function([ex_states, states],
                               [ex_speeds, speeds],
                               [ex_reval, reval],
                               [ex_wave_types, wave_types],
                               variable_names=sw.conserved_variables)

    pp = sw.phase_plane_plot(q_l, q_r, g=1., y_axis='u',
                             approx_states=states)

    return pp, pr

phase_plane, comparison = make_plot_functions(h_l, u_l, h_r, u_r)
interact(comparison, t=widgets.FloatSlider(min=0,max=.9,step=.1));
```

Here and throughout this section, we plot the approximate solution in green.  As usual, Hugoniot loci and shock paths are plotted in red, while integral curves and rarefaction fans are plotted in blue.

Notice that the rarefaction wave is approximated by a discontinuity.

The next example illustrates the exactness property for isolated shocks.  Here we have chosen $q_l$ and $q_r$ such that they are related by a single 2-shock (up to rounding errors).

```python
h_l = 2.20698770767; h_r = 1.
u_l = 2.27057814896/h_l; hu_r = 0.

phase_plane, comparison = make_plot_functions(h_l, u_l, h_r, u_r)
interact(comparison, t=widgets.FloatSlider(min=0,max=.9,step=.1));
```

Because the exact and approximate solutions are the same, the exact solution is not visible in these plots.  The only exception is the 1-wave in the first plot; the speed of the 1-wave appears different for the Roe solution and the exact solution, but this is unimportant since the 1-wave has zero strength.

### Entropy fix

As discussed previously, the use of an entropy-violating discontinuity in place of a rarefaction can be problematic in the case of a transonic rarefaction.  For the Roe solver, we can again apply an entropy fix, splitting the single discontinuity into two waves, to avoid this problem.  Several different entropy fixes have been proposed; here we focus on one that is quite general and effective in practice.

Suppose that $\lambda_1(q_l) < 0$ while $\lambda_1(q_m)>0$.  In this case, it appears that the 1-wave should be a transonic rarefaction.  We take the 1-wave ${\mathcal W}^1$ of the Roe solver, with speed $s_1$, and split it into two waves, ${\mathcal W}^1_l = \alpha^1_l r_1$ and ${\mathcal W}^1_r = \alpha^1_r r_1$, with speeds $\lambda_1(q_l), \lambda_1(q_m)$.  Two requirements determine the strength of these waves.  First, the total jump across them should be equal to the jump across the original wave:

$${\mathcal W}_{1,l} + {\mathcal W}_{1,r} = {\mathcal W}_1.$$

Second, we must maintain conservation:

$$\lambda_1(q_l) {\mathcal W}_{1,l} + \lambda_1(q_m) {\mathcal W}_{1,r} = s_1 {\mathcal W}_1.$$

This leads to

\begin{align*}
{\mathcal W}_{1,l} & = \frac{\lambda_1(q_m)-s_1}{\lambda_1(q_m)-\lambda_1(q_l)}{\mathcal W}_1 \\
{\mathcal W}_{1,r} & = \frac{s_1 - \lambda_1(q_l)}{\lambda_1(q_m)-\lambda_1(q_l)}{\mathcal W}_1.
\end{align*}

We can check in a similar way for a transonic 2-rarefaction, and modify the 2-wave accordingly.  Note that only one of the two characteristic fields can be transonic in the Riemann solution for the shallow water equations.

This entropy fix is implemented in the solver below.

```python
def shallow_water_roe_with_efix(q_l, q_r, grav=1., efix=True):
    """
    Approximate Roe solver for the shallow water equations,
    with entropy fix.
    """

    h_l = q_l[0]
    hu_l = q_l[1]
    u_l = hu_l/h_l
    h_r = q_r[0]
    hu_r = q_r[1]
    u_r = hu_r/h_r

    delta = q_r - q_l

    # Roe averages
    h_hat = (h_r + h_l)/2.
    u_hat = (np.sqrt(h_r)*u_r + np.sqrt(h_l)*u_l) /  \
            (np.sqrt(h_r) + np.sqrt(h_l))
    c_hat = np.sqrt(grav*h_hat)

    s1 = u_hat - c_hat
    s2 = u_hat + c_hat

    alpha1 = ( (u_hat+c_hat)*delta[0] - delta[1])/(2*c_hat)
    #alpha2 = (-(u_hat-c_hat)*delta[0] + delta[1])/(2*c_hat)

    h_m = q_l[0] + alpha1
    hu_m = q_l[1] + alpha1 * (u_hat - c_hat)
    u_m = hu_m/h_m
    q_m = np.array([h_m, hu_m])

    transonic = None
    if efix:
        # Check if either wave appears transonic
        lambda_1_l = u_l - np.sqrt(grav*h_l)
        lambda_1_m = u_m - np.sqrt(grav*h_m)
        lambda_2_m = u_m + np.sqrt(grav*h_m)
        lambda_2_r = u_r + np.sqrt(grav*h_r)

        if lambda_1_l < 0 < lambda_1_m: # 1-wave appears transonic
            transonic = 1
        elif lambda_2_m < 0 < lambda_2_r: # 2-wave appears transonic
            transonic = 2

    if transonic == 1:
        beta = (lambda_1_m - s1)/(lambda_1_m - lambda_1_l)
        alpha1_l = beta*alpha1
        #alpha_1_r = (1-beta)*alpha1
        h_m_new = q_l[0] + alpha1_l
        hu_m_new = q_l[1] + alpha1_l * (u_hat-c_hat)
        q_m_new = np.array([h_m_new, hu_m_new])
        states = np.column_stack([q_l,q_m_new,q_m,q_r])
        speeds = [lambda_1_l, lambda_1_m, s2]
        wave_types = ['contact']*3

        def reval(xi):
            h_out  = (xi<speeds[0])*states[0,0] + \
                     (speeds[0]<=xi)*(xi<speeds[1])*states[0,1] + \
                     (speeds[1]<=xi)*(xi<speeds[2])*states[0,2] + \
                     (speeds[2]<=xi)*states[0,3]
            hu_out = (xi<speeds[0])*states[1,0] + \
                     (speeds[0]<=xi)*(xi<speeds[1])*states[1,1] + \
                     (speeds[1]<=xi)*(xi<speeds[2])*states[1,2] + \
                     (speeds[2]<=xi)*states[1,3]
            return h_out, hu_out

    elif transonic == 2:
        beta = (lambda_2_r - s2)/(lambda_2_r - lambda_2_m)
        #alpha2_l = beta*alpha2
        alpha2_r = (1-beta)*alpha2
        h_m_new = q_r[0] - alpha2_r
        hu_m_new = q_r[1] - alpha2_r*(u_hat+c_hat)
        q_m_new = np.array([h_m_new, hu_m_new])
        states = np.column_stack([q_l,q_m,q_m_new,q_r])
        speeds = [s1,lambda_2_, lambda_2_r]
        wave_types = ['contact']*3

        def reval(xi):
            h_out  = (xi<speeds[0])*states[0,0] + \
                     (speeds[0]<=xi)*(xi<speeds[1])*states[0,1] + \
                     (speeds[1]<=xi)*(xi<speeds[2])*states[0,2] + \
                     (speeds[2]<=xi)*states[0,3]
            hu_out = (xi<speeds[0])*states[1,0] + \
                     (speeds[0]<=xi)*(xi<speeds[1])*states[1,1] + \
                     (speeds[1]<=xi)*(xi<speeds[2])*states[1,2] + \
                     (speeds[2]<=xi)*states[1,3]
            return h_out, hu_out

    else:
        states = np.column_stack([q_l,q_m,q_r])
        speeds = [s1, s2]
        wave_types = ['contact','contact']

        def reval(xi):
            h  = (xi<s1)*h_l + (s1<=xi)*(xi<=s2)*h_m + (s2<xi)*h_r
            hu = (xi<s1)*hu_l + (s1<=xi)*(xi<=s2)*hu_m + (s2<xi)*hu_r
            return h, hu

    return states, speeds, reval, wave_types
```

An implementation of this solver for use in Clawpack can be found [here](https://github.com/clawpack/riemann/blob/c0a41c664e9a13d35a113409dea92f3c87648d09/src/rp1_shallow_roe_with_efix.f90).

Here are some examples of problems with transonic rarefactions, showing the behavior of the solver.

```python
h_l = 1; h_r = 1
u_l = 0.5; u_r = 2

def compare_efix(h_l,u_l,h_r,u_r):
    hu_l = h_l*u_l; hu_r = h_r*u_r
    q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])
    ex_states, ex_speeds, ex_reval, ex_wave_types = \
                        sw.exact_riemann_solution(q_l,q_r)
    states, speeds, reval, wave_types = \
                        shallow_water_roe_with_efix(q_l, q_r)
    no_efix_states, no_efix_speeds, no_efix_reval, no_efix_wave_types = \
                        shallow_water_roe(q_l, q_r)

    plot_function = \
        rt.make_plot_function([ex_states, no_efix_states, states],
                              [ex_speeds, no_efix_speeds, speeds],
                              [ex_reval, no_efix_reval, reval],
                              [ex_wave_types, no_efix_wave_types, wave_types],
                              ['Exact','Roe w/o fix','Roe with fix'],
                              layout='vertical',
                              variable_names=sw.conserved_variables)
    interact(plot_function, t=widgets.FloatSlider(min=0,max=.9,step=.1));
    return q_l, q_r, states, no_efix_states

q_l, q_r, states, no_efix_states = compare_efix(h_l,u_l,h_r,u_r)
```

```python
fig, ax = plt.subplots(1,2,figsize=(10,4))
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=no_efix_states,ax=ax[0])
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states,ax=ax[1],color='orange')
ax[0].set_title('Without entropy fix')
ax[1].set_title('With entropy fix');
```

```python
h_l = 4; h_r = 1
u_l = 1; u_r = 1

q_l, q_r, states, no_efix_states = compare_efix(h_l,u_l,h_r,u_r)
```

```python
fig, ax = plt.subplots(1,2,figsize=(10,4))
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=no_efix_states,ax=ax[0])
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states,ax=ax[1],color='orange')
ax[0].set_title('Without entropy fix')
ax[1].set_title('With entropy fix');
```

### Positivity

In the shallow water system, it is both physically and mathematically essential that the depth remain non-negative.  Unfortunately, as the next example shows, the Roe solver does not always guarantee this.

```python
h_l = 1; h_r = 1
u_l = -1.5; u_r = 1.5

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])
ex_states, ex_speeds, ex_reval, ex_wave_types = \
                sw.exact_riemann_solution(q_l,q_r)
states, speeds, reval, wave_types = \
                shallow_water_roe_with_efix(q_l, q_r)

plot_function = \
    rt.make_plot_function([ex_states, states],
                          [ex_speeds, speeds],
                          [ex_reval, reval],
                          [ex_wave_types, wave_types],
                          variable_names=sw.conserved_variables)
interact(plot_function, t=widgets.FloatSlider(min=0,max=.9,step=.1));
```

```python
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states, hmin = -1)
```

In fact, this is a feature of all linearized Riemann solvers: for some Riemann problems, they will produce negative intermediate states.  The HLL solver described next does guarantee a positive solution for all initial data.

## HLL solver

### Derivation and implementation

Recall that an HLL approximate solver is based on a fictitious solution consisting of two waves (discontinuities) with a single intermediate state $q_m$ between them.  The waves are thus
\begin{align}
{\mathcal W}_1 & = q_m - q_l \\
{\mathcal W}_2 & = q_r - q_m.
\end{align}
The middle state is determined by the requirement of conservation:
\begin{align}
s_1 {\mathcal W}_1 + s_2 {\mathcal W}_2 & = f(q_r) - f(q_l),
\end{align}
which can be solved for $q_m$ to find
\begin{align}  \label{SWA:hll_middle_state}
q_m = \frac{f(q_r) - f(q_l) - s_2 q_r + s_1 q_l}{s_1 - s_2}.
\end{align}

The key remaining ingredient is the choice of the wave speeds $s_1, s_2$.  These should approximate the wave speeds appearing in the true Riemann solution.  In general, it is desirable for the purpose of stability that the speeds used be approximations of the fastest (left- and right-going) waves in the Riemann solution.  Indeed, in the original paper of Harten, Lax, and van Leer (Harten, Lax, and van Leer) it was proposed to use lower and upper bounds on the true wave speeds.  However, overestimating the wave speeds leads to increased diffusion, and various refinements to the choice of wave speeds were subsequently proposed.  Here we choose the wave speeds based on a suggestion of Einfeldt (originally in the context of the Euler equations); the resulting solver is often referred to as an HLLE solver.

For the shallow water system, the 1-wave speed is always less than or equal to the 2-wave speed.  If the 1-wave is a rarefaction, then the minimum wave velocity is given simply by $\lambda_1(q_l)$, the characteristic speed at the left edge of the 1-rarefaction fan.  If the left-going wave speed is a shock, then the minimum wave velocity is the shock speed.  We cannot determine the shock speed exactly without solving the Riemann problem -- just what we hope to avoid.  Instead, we can use an approximate shock speed based on the Roe average:

$$\hat{s}_1 = \hat{u} - \hat{c}.$$

Since we don't know whether the 1-wave should be a shock or a rarefaction, we use a lower bound on these two speeds, by taking their minimum:

$$s_1 = \min\left(\lambda_1(q_l), \hat{u}-\hat{c}\right).$$

Similarly, the second wave speed is given by

$$s_2 = \max\left(\lambda_2(q_r), \hat{u}+\hat{c}\right).$$

This completes the definition of the solver. Notice that if the states $q_l, q_r$ are connected by a single 2-shock, then the value of $s_2$ will be the exact shock speed (since the Roe speed is exact in this case, and the shock speed will be faster than the characteristic speed ahead of it).  Furthermore, by substituting the Rankine-Hugoniot conditions into (\ref{SWA:hll_middle_state}), one sees that the the middle state in this case will be $q_m = q_l$.  Thus the HLL solver gives the exact solution for this case.  Similarly, it yields the exact solution when the initial states are connected by just a 1-shock.  It shares this nice property with the Roe solver (as long as the wave speeds are chosen as just described).

```python
def shallow_water_hll(q_l, q_r, grav=1.):
    """
    HLLE approximate solver for the shallow water equations.
    """
    h_l = q_l[0]
    hu_l = q_l[1]
    u_l = hu_l/h_l
    h_r = q_r[0]
    hu_r = q_r[1]
    u_r = hu_r/h_r

    # Roe averages
    h_hat = (h_r + h_l)/2.
    u_hat = (np.sqrt(h_r)*u_r + np.sqrt(h_l)*u_l) / \
            (np.sqrt(h_r) + np.sqrt(h_l))
    c_hat = np.sqrt(grav*h_hat)

    lambda_1_l = u_l - np.sqrt(grav*h_l)
    lambda_2_r = u_r + np.sqrt(grav*h_r)

    s1 = min(lambda_1_l, u_hat - c_hat)
    s2 = max(lambda_2_r, u_hat + c_hat)

    h_m = (hu_r - hu_l - s2*h_r + s1*h_l)/(s1-s2)
    hu_m = (hu_r*u_r - hu_l*u_l + 0.5*grav*(h_r**2 - h_l**2) \
            - s2*hu_r + s1*hu_l)/(s1-s2)
    q_m = np.array([h_m, hu_m])

    states = np.column_stack([q_l,q_m,q_r])
    speeds = [s1, s2]
    wave_types = ['contact','contact']

    def reval(xi):
        h_out  = (xi<s1)*h_l + (s1<=xi)*(xi<=s2)*h_m + (s2<xi)*h_r
        hu_out = (xi<s1)*hu_l + (s1<=xi)*(xi<=s2)*hu_m + (s2<xi)*hu_r
        return h_out, hu_out

    return states, speeds, reval, wave_types
```

An implementation of this solver for use in Clawpack can be found [here](https://github.com/clawpack/riemann/blob/c0a41c664e9a13d35a113409dea92f3c87648d09/src/rp1_shallow_hlle.f90).

### Examples

```python
def compare_solutions(q_l, q_r, solvers):
    outputs = []
    states = {}

    for solver in solvers:
        if solver.lower() == 'exact':
            outputs.append(sw.exact_riemann_solution(q_l,q_r))
        if solver.lower() == 'hll':
            outputs.append(shallow_water_hll(q_l, q_r))
            states['hll'] = outputs[-1][0]
        if solver.lower() == 'roe':
            outputs.append(shallow_water_roe_with_efix(q_l, q_r))
            states['roe'] = outputs[-1][0]
        if solver.lower() == 'roe no efix':
            outputs.append(shallow_water_roe(q_l, q_r))
            states['roe no efix'] = outputs[-1][0]

    plot_function = \
        rt.make_plot_function([val[0] for val in outputs],
                              [val[1] for val in outputs],
                              [val[2] for val in outputs],
                              [val[3] for val in outputs],
                              solvers, layout='horizontal',
                              variable_names=sw.conserved_variables)

    interact(plot_function,
             t=widgets.FloatSlider(min=0,max=0.9,step=0.1));

    return states
```

```python
# Dam-break example
h_l = 4; h_r = 1
u_l = 0; u_r = 0

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])

states = compare_solutions(q_l, q_r, ['Exact','HLL'])
```

```python
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',approx_states=states['hll'])
```

```python
# Isolated 2-shock example
h_l = 2.20698770767; h_r = 1.
u_l = 2.27057814896/h_l; hu_r = 0.

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])

states = compare_solutions(q_l, q_r, ['Exact','HLL'])
```

```python
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states['hll'])
```

```python
# Transonic rarefaction example a
h_l = 1; h_r = 1
u_l = 0.5; u_r = 2

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])

states = compare_solutions(q_l, q_r, ['Exact','HLL'])
```

```python
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',approx_states=states['hll'])
```

Notice that, by design, the HLLE wave always lies exactly on the leading edge of a rarefaction wave, when compared to the true solution.

The momentum solution above has the interesting feature that the solution in the 1-rarefaction is not monotone but appears as a "bump".  This can be explained by plotting the solution in the h-hu phase plane, where one sees that the integral curve has a local maximum between $q_l$ and $q_m$:

```python
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='hu',approx_states=states['hll'])
```

### Entropy and positivity

It can be shown that the HLLE solver always preserves positivity and does not require an entropy fix.  Both of these properties result in part from the fact that the HLLE solver is more diffusive than the Roe solver.

```python
# Transonic rarefaction example b
h_l = 4; h_r = 1
u_l = 4; u_r = 1

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])

states = compare_solutions(q_l, q_r, ['Exact','HLL'])
```

```python
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states['hll'])
```

The final example below illustrates that the HLLE solver maintains positivity when the same initial data is used as in the example with the Roe solver above where a negative intermediate depth was obtained.

```python
h_l = 1; h_r = 1
u_l = -1.5; u_r = 1.5

hu_l = h_l*u_l; hu_r = h_r*u_r
q_l = np.array([h_l, hu_l]); q_r = np.array([h_r, hu_r])

states = compare_solutions(q_l, q_r, ['Exact','HLL'])
```

```python
sw.phase_plane_plot(q_l,q_r,g=1.,y_axis='u',
                    approx_states=states['hll'])
```


```python
import sympy as sp
sp.init_printing()
```

# 03_StellarDynamics_HW_Chap06

# Question (1) Low Surface Density Part

```python
# initialize the symbols
kappa, kappa_0, P, T, F_rad, F_conv, a, b, mu = sp.symbols(
    'kappa kappa_0 P T F_rad F_conv a b mu')
```

```python
# kappa for opacity
exprK = sp.Eq(kappa, kappa_0 * P**a * T**b)
exprK
```

```python
# eq (6.121), pressure for photosphere with optical depth = 3/2
eqPh = sp.Eq(sp.simplify('P_ph'),
             sp.simplify('2/3 * (1+a) * g / (kappa)'))
eqPh
```

```python
# approx expression for F_rad in (6.123)
exprFr = sp.Eq(F_rad, sp.simplify('a * c / 4 * T^4'))
exprFr
```

```python
# (6.124)
exprFc = sp.Eq(F_conv, sp.simplify('1/2 * rho * c_P * v * T'))
exprFc
```

```python
# approx the radial-velocity fluctuation
v = sp.Symbol('v')
gamma = sp.Symbol('gamma')
exprV = sp.simplify('1 / 2 * ((r * k * T)/(mu * m_H))^(1/2)').subs(
    sp.sympify('r'), gamma)
sp.Eq(v, exprV)
```

```python
# (6.126) approx ρcT to internal energy
exprInternal = sp.simplify('3 / 2 * rho * (k * T / mu/ m_H)') * gamma
eqInternal = sp.Eq(sp.simplify('rho * c_P * T'), exprInternal)
eqInternal
```

```python
# ideal gas law
eqIdeal = sp.Eq(P, sp.simplify('rho * k * T'))
eqIdeal = sp.solve(eqIdeal, sp.simplify('rho'))[0]
sp.Eq(sp.simplify('rho'), eqIdeal)
```

```python
# substitute (6.124)
exprFc = exprFc.subs(
    eqInternal.lhs, eqInternal.rhs).subs(
    v, exprV).subs(sp.simplify('rho'), eqIdeal)
exprFc
# This is eq (6.127)
```

```python
# for energy conservation, the radiative flux which enters the
# sphere shold equals to the convective flux leaving the
# stellar interior
sp.Eq(exprFc.lhs, exprFr.lhs)
```

```python
# So we have
eqFcFr = sp.Eq(exprFc.rhs, exprFr.rhs)
eqP = sp.Eq(sp.simplify('P_ph'), sp.solve(eqFcFr, P)[0])
eqP
```

```python
# So, the effective temperature for the fully convective star
# in low surface density is
## here we substitute kappa and P_ph
eqP = eqP.subs(eqPh.lhs, eqPh.rhs).subs(exprK.lhs, exprK.rhs).subs(P, eqP.rhs)

## here we symbolically solve for T
sp.solve(eqP, T)
```

The above solution is awful.
But we can roughly see that the exponents of, e.g., $\kappa_0$ is the same as `eq (6.129)`.
There is come inconsistent between above solution and `eq (6.129)`. However, since `eq. (6.129)` is just a approximation, let's just take the approximation made by the textbook.

```python
# eq (6.129), effective temperature T for low surface density
eqTeff = sp.Eq(sp.simplify('T_eff'),
    sp.simplify('(M / mu^((a+1)/2) / kappa_0 / R^2)^(1/(b+3.5*(1+a)))'))
eqTeff
```

- Population 1

```python
# Population I effective temperature
eqTeff.subs(a, 0.7).subs(b, 5.3).subs(kappa_0, 6.9*10**-26)
```

- Population 2

```python
# Population II effective temperature
eqTeff.subs(a, 0.6).subs(b, 9.4).subs(kappa_0, 6.1*10**-40)
```

# Question (2) Low Surface Density Part

```python
eqL = sp.Eq(sp.simplify('L'),
            sp.simplify('4 * pi * sigma * R^2 * T_eff^4'))
# equaiton of luminosity
eqL = sp.Eq(sp.simplify('L'),
            sp.simplify('4 * pi * sigma * R^2 * T_eff^4'))
eqL
```

```python
# substitute for effective temperature
eqL = eqL.subs(eqTeff.lhs, eqTeff.rhs)
eqL
```

- Population 1

```python
# Population I luminosity
eqL.subs(a, 0.7).subs(b, 5.3).subs(kappa_0, 6.9*10**-26)
```

- Population 2

```python
# Population II luminosity
eqL.subs(a, 0.6).subs(b, 9.4).subs(kappa_0, 6.1*10**-40)
```

# Question (1) High Surface Density Part

For high surface density, the convective transport is efficient. The temperature gradient is adiabatic. So we may have...

$$
P \propto \rho^{\gamma} \propto \rho T
$$
$$
P \propto T^{\gamma / (\gamma - 1)}
$$

```python
K = sp.Symbol('K')
eqAdiabatic = sp.Eq(P, K * T**(gamma/(gamma-1)))
eqAdiabatic
```

So, we would have the relation between $P_{ph}$ and $P_{c}$

```python
Pc, Tc = sp.symbols('P_c T_c')
P_ph, T_eff = sp.symbols('P_ph T_eff')
eqAdiabaticHigh = sp.Eq(
    eqAdiabatic.subs(P, Pc).lhs / eqAdiabatic.subs(T, Tc).rhs * K,
    eqAdiabatic.subs(P, P_ph).lhs / eqAdiabatic.subs(T, T_eff).rhs * K
     ).subs(gamma, 5/3)
eqAdiabaticHigh
```

```python
# P_ph Substitute with (6.121)
eqHighSurface = eqAdiabaticHigh.subs(P_ph, eqPh.rhs)
eqHighSurface
```

```python
# substitute kappa
eqHighSurface = eqHighSurface.subs(kappa, exprK.rhs).subs(T, T_eff).subs(P, P_ph)
eqHighSurface
```

```python
# it is awuful that I need to replace P_ph with Pc/Tc again
eqHighSurface = eqHighSurface.subs(P_ph,
    sp.solve(eqAdiabaticHigh, P_ph)[0])
eqHighSurface
```

不太確定為什麼課本不給一下 core 的 profile ，
但是老闕上課好像有給。

$$
\frac{P_c}{T_c^{2.5}} \propto M^{-0.5} R^{1.5}
$$

```python
# substitute core profile with above relation provided by 老闕
M, R = sp.symbols('M R')
eqHighSurface = eqHighSurface.subs(Pc, M**-0.5 * R**1.5 * Tc**2.5)
eqHighSurface
```

```python
# it seems Tc should have zero exponent,
eqHighSurface = eqHighSurface.subs(Tc, 1)
eqHighSurface
```

```python
## subsitute for g = M / R^2
g = sp.Symbol('g')
eqHighSurface = eqHighSurface.subs(g, sp.simplify('M / R^2'))
eqHighSurface
```

```python
## solve for T_eff
eqHighSurface = sp.Eq(T_eff, sp.solve(eqHighSurface, T_eff)[0])
eqHighSurface
```

- Population 1

```python
# Population 1 effective temperature
eqHighSurface.subs(
    a, 0.7).subs(b, 5.3).subs(kappa_0, 6.9*10**-26)
```

這個結果也太好笑了XD  不知道為什麼程式不把指數項爆開，但是我們可以試著幫他算一下：

- exponent of M : 3.7 * 0.0523 = 0.1935, the same as (6.135)
- exponent of R : 9.1 * 0.0523 = 0.4759, not the same as (6.135)

R 的指數是錯的，有可能老闕給的那條式子是錯的或者我用錯地方，但是 M 的指數是對的

- Population 2

```python
# Populatoin II effective temperature
eqHighSurface.subs(
    a, 0.6).subs(b, 9.4).subs(kappa_0, 6.1*10**-40)
```

這次是兩個都錯了。好像不是很確定哪裡算錯了。

# Question (2) High Surface Density Part

```python
# eq of luminosity
eqLHighSurface = sp.Eq(
    sp.simplify('L'), sp.simplify('4 * pi * sigma * R^2 * T_eff^4'))
eqLHighSurface
```

```python
eqLHighSurface = eqLHighSurface.subs(T_eff, eqHighSurface.rhs)
eqLHighSurface
```

- Population 1

```python
# population I luminosity
eqLHighSurface.subs(
    a, 0.7).subs(b, 5.3).subs(kappa_0, 6.9*10**-26)
```

```python
# population II luminosity
eqLHighSurface.subs(
    a, 0.6).subs(b, 9.4).subs(kappa_0, 6.1*10**-40)
```

# Question (1) Ionization Zones
課本這邊寫得看不太懂，不過大概是叫我們硬解吧。

Use the same adiabatic condition.

```python
eqAdiabatic
```

The textbook says that the ionization zone is convective and the entropy across it is constant. So that $S(x=1) - S(x=0) = 0$, where x denotes the fraction of hydrogen ionized.

Also, argued in the textbook,
- x = 1 : the center of the star
- x = 0 : at the base of the photosphere

Yeah ... so, basically we would have

$$
S(x=1, P=P_c) = S(x=0, P=P_{ph})
$$

```python
# entropy per unit mass
exprEntropy = sp.simplify(
    '''
    X * k / m_H *
    (5 / 2 * (1 + x + delta) +
    chi / k / T +
    ln(2 * pi * m_H / h^2)^(3/2) +
    delta * ln(2 * pi * m_He / h^2)^(3/2) +
    x * ln(2 * pi * m_e / h^2)^(3/2) +
    (1 + x + delta) * ln(k * T)^(5/2) * (1 + x + delta) / P)
    ''')
S = sp.Symbol('S')
epEntropy = sp.Eq(S, exprEntropy)
epEntropy
# lol this is ugly
```

Get the relation here,
$$
S(x=1, P=P_c) = S(x=0, P=P_{ph})
$$

```python
# S(x=1, P=P_c) = S(x=0, P=P_{ph})
x, delta = sp.symbols('x delta')
eqBoundary = sp.Eq(epEntropy.subs(x, 1).subs(P, Pc).subs(T, Tc).rhs,
    epEntropy.subs(x, 0).subs(P, P_ph).subs(T, T_eff).rhs)

eqBoundary
# lol 超級複雜，用手算會瘋掉
```

```python
# substitute delta = Y / (4 X)
eqBoundary = eqBoundary.subs(delta, sp.simplify('Y / (4 * X)'))
eqBoundary
```

And then, substitute with $$
\frac{P_c}{T_c^{2.5}} \propto M^{-0.5} R^{1.5}
$$

```python
eqBoundary = eqBoundary.subs(Pc, M**-0.5 * R**1.5 * Tc**2.5)
eqBoundary
```

天啊，感覺 $T_c$ 是消不掉的。用 `simplify` 來試試看好了。

註： `simplify` 是 `sympy` 用來簡化方程式的懶人指令，他自動會幫你跑他自己覺得比較簡化的結果。

```python
eqBoundary = sp.simplify(eqBoundary)
eqBoundary
```

都做到這裡了也只能硬著頭皮做下去了：

- 用 (6.121) 替代掉 $P_{ph}$
- 替代掉 $\kappa$
- 再用 adiabatic equation 替代掉 多餘的 $P_{ph}$，留下$T_{eff}$ 可解

```python
# 用 (6.121)
eqBoundary = eqBoundary.subs(P_ph, eqPh.rhs)
eqBoundary
```

```python
# 換掉 kappa
eqBoundary = eqBoundary.subs(kappa, exprK.rhs.subs(P, P_ph).subs(T, T_eff))
eqBoundary
```

```python
# adiabatic condition Pc / Tc^2.5 = P_ph / T_eff^2.5
# 但是這邊就直接把 Pc / Tc^2.5 換成  M^{-0.5} R^{1.5} 好了
eqAdiabaticHigh.subs(eqAdiabaticHigh.lhs, M**-0.5 * R**1.5)
```

```python
eqBoundary = eqBoundary.subs(
    P_ph,
    sp.solve(
        eqAdiabaticHigh.subs(eqAdiabaticHigh.lhs, M**-0.5 * R**1.5),
        P_ph
        )[0]
    )
eqBoundary
```

感覺是很嚴重的大失敗。
但是還是讓他把答案寫完好了。

```python
sp.solve(eqBoundary, T_eff)
```

最後是算不出來的⋯⋯

（投降）

# Question 3

C)Explain the mechanism of Hayashi track, why we can roughly disassemble the Hayashi track and what is the difference among the three states?(10%)

- 一開始可能是一坨氣體開始收縮，因為還沒有核反應，所以溫度超低，幾乎沒有輻射、都是對流在掌控溫度。這時的這星體可以說是在 Hayashi track 上的 protostar。  由於才剛開始收縮，表面密度就會比較低，而且又是個完全對流（fully-convective）的星體，所以可以用 6.5 節的low surface density 的 近似來做。這時的表面溫度會稍微提高、luminosity 會迅速降低。
- 持續收縮之後，到了某種程度，表面密度變高了，這時就可以用 high surface density 來近似。溫度隨著收縮而降低一點點、luminosity 掉得更快了。
- 但是到了某種程度之後，內部溫度夠高了，輻射開始比對流還要重要，內部形成了一個輻射核。這時就可以用 ionization zone 的近似來解釋。接著，開始越來越高溫，在 H-R 圖上明顯往左移，到了某種高溫程度後，點燃氫核融合。

抄錄自課本 171 頁。


```python
%config InlineBackend.figure_format = 'retina'
import matplotlib.pyplot as plt
import numpy as np
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

# Machine Learning

## Binary Classification

The simplest, common and yet very practical problem in machine learning is __binary classification__. What does binary classification mean? Let's see an example. Let's say that you have a lot of images of cats and dogs. When you look at the picture, you know whether it contains a dog or a cat. But you want to teach the computer to recognise them. What you want to achieve is: you give a computer an image as an input. Its output should be: this is a dog, or this is a cat. The binary in binary classification means that computer can only tell you two answers, dog, or a cat, an nothing more.

To achieve this requires two things. You need the __data__. Data is usually represented in vectors. Later we will learn more about vectors, but for now, think of them as ordered list of numbers. So let's say that a picture of a dog or a cat can be encoded in 100 numbers. What else do you need? You need __labels__. A label defines the class associated with your data. In our case, a label could be a dog or a cat. But it is not necessary to think of a label as a string of text. Since we know that we only have two classes, we can say that we will label a picture of a dog with the number 1 and a picture of a cat with the number 0. This should work just as well.

Let's introduce an example that is easier to implement at this stage. We have a data set below, containing 6 data points, each containing a 2-dimensional vector associated with a label. For example, a vector `[1.2, 0.7]` is associated with the label `+1`.

| vector [x,y] 	| label 	|
|--------------	|-------	|
| [ 1.2, 0.7]  	| +1    	|
| [-0.3, -0.5] 	| -1    	|
| [3.0, 0.1]   	| +1    	|
| [-0.1, -1.0] 	| -1    	|
| [-1.0, 1.1]  	| -1    	|
| [2.1, -3.0]  	| +1    	|

To get a better understanding of the data, we can plot these two-dimensional vectors on a cartesian plane:

Here, the location on the point in the plane is given by the pair of coordinates, and its colour is defined by the label. Red points have the label `+1`, and the blue points have the label `-1`.  When we give the computer a pair of (x,y) coordinates, we want the computer to tell us should whether the point should be marked `-1`—red or `+1`—blue, according to what it knows from the data.  What does learning means in this case? The computer needs to define the boundary between these two given classes. This can be explained in more technical terms:

 Our __goal__ in binary classification is to learn a function that takes an arbitrary 2-dimensional vector and predicts the label according to the provided data. This function is usually parameterized by a certain set of parameters, and we will want to tune the parameters of the function so that its outputs are consistent with the labeling in the provided dataset. In the end we can discard the dataset and use the learned parameters to predict labels for previously unseen vectors.

### Classification by using a simple linear function

First we will use a simple linear function to implement binary classification:

\begin{align}
F = f(\mathbf{X},\mathbf{Y})=A\mathbf{X}+B\mathbf{Y}+C
\end{align}

Here, $\mathbf{X}$ and $\mathbf{Y}$ are the input vectors, and $A$, $B$ and $C$ are the parameters that we are trying to learn. This can be represented schematically:

So, if our data point is $(\mathbf{X},\mathbf{Y})=$ `[1.2,0.7]`, and if the chosen parameters are $A=$ `1`, $B=$ `-2`, $C=$ `-3`.

```python
X, Y, A, B, C = 1.2, 0.7, 1, -2, -3
```

then the output of the function $F = f(\mathbf{X},\mathbf{Y})$ is:

```python
A*X + B*Y + C
```

    -3.2

How can we use this setup to predict the label (`+1`, or `-1`) of an arbitrary vector, for example `[-2.1, 0.8]`, given only our data set?

| vector [x,y] 	| label 	|
|--------------	|-------	|
| [ 1.2, 0.7]  	| +1    	|
| [-0.3, -0.5] 	| -1    	|
| [3.0, 0.1]   	| +1    	|
| [-0.1, -1.0] 	| -1    	|
| [-1.0, 1.1]  	| -1    	|
| [2.1, -3.0]  	| +1    	|

The prediction has to do with how do we interpret the output $F$ of the function $f(\mathbf{X},\mathbf{Y})$.

__We can define that in our algorithm, the parameters $A$, $B$, and $C$ must be tuned such that the function $f$ outputs a number greater than 0 when it encounters a data point associated with the label `+1`. In the same way, the function should output a number less (or equal to) than 0 when it encounters a data point associated with the label `-1`.__

For example, when we plugged the vector $(\mathbf{X},\mathbf{Y})=$ `[1.2,0.7]` associated with the label `+1` into the function $f$ whose current parameters are
 $[A,B,C]=$ `1, -2, -3` we got the result $F = -3.2$.

Since the label associated with this vector is `+1`, we want our function to output a number larger than 0. Instead we got `-3.2` which is less than 0. This tells us what needs to be done. We need to tweak the parameters `A`, `B` and `C` such that we *maximize* the ouput of the function—push it towards a positive number. Luckily we know how to do this with derivatives!

#### Backpropagation

In order to increase the output of the function $f$ we need to find the partial derivatives $\frac{\partial F}{\partial A}$, $\frac{\partial F}{\partial B}$ and $\frac{\partial F}{\partial C}$. Note that we don't need to find the partials $\frac{\partial F}{\partial \mathbf{X}}$, and $\frac{\partial F}{\partial \mathbf{Y}}$ as these are our data points. They don't need to change!

By using the chain rule, rules of differentiation and consulting the diagram above we can compute:

\begin{align*}
\frac{\partial F}{\partial A} &=\frac{\partial F}{\partial G}*\frac{\partial G}{\partial A}; &\frac{\partial F}{\partial B} &=\frac{\partial F}{\partial G}*\frac{\partial G}{\partial B}; &\frac{\partial F}{\partial C} ;& \\\\
\frac{\partial F}{\partial A} &=1*X &\frac{\partial F}{\partial B} &=1*Y &\frac{\partial F}{\partial C} &=1 \\\\
\frac{\partial F}{\partial A} &=X &\frac{\partial F}{\partial B} &=Y &\frac{\partial F}{\partial C} &=1
\end{align*}

Since in this case we want to maximize our function we could translate this computation in code by simply writing `dA = X; dB = Y; dC = 1` and this would work. However, in some cases we will need to minimize our function. For that purpose it is useful to multiply `dA`, `dB` and `dC` with either `-1` or `+1` depending on the given situation. For that we can introduce a new variable `pull` that can be set to  `1` or `-1`:

```python
pull = 1 # since here we want to maximize the function

dA = X * pull
dB = Y * pull
dC = 1 * pull
print (f'dF/dA: {dA}\ndF/dB: {dB}\ndF/dC: {dC}')
```

    dF/dA: 1.2
    dF/dB: 0.7
    dF/dC: 1

In order to gradually maximize the function $f$ (step-by-step) towards our desired result, we need to update our parameters. This is done by adding to the parameter's value the value of its partial derivative. To achieve this gradually in small steps, we multiply the value of the partial derivative by a a small number (step).

```python
step_size = 0.01
A = A + dA * step_size
B = B + dB * step_size
C = C + dC * step_size
```

If we now evaluate the function $f(\mathbf{X},\mathbf{Y})$, with the updated parameters $A$, $B$, and $C$, we get:

```python
A*X + B*Y + C
```

    -3.1707

This result is better then the original result `-3.2` !

#### Regularization

In a machine learning algorithm like this, the parameters $A$, $B$, and $C$ are usually randomly initialised in the beginning. Sometimes with a given initialisation, it can happen during the training that a certain parameter(s) becomes much larger than the others. In our particular case $(F = A\mathbf{X}+B\mathbf{Y}+C)$ the parameters $A$ and $B$ can be especially problematic, since they multiply our data points $\mathbf{X}$ and $\mathbf{Y}$, and multiplication leads to a greater change of the output value $F$ than the addition ($C$).
 For this reason, we can slightly push the parameters $A$ and $B$ towards zero with a given force that is proportional to their value. This is called __regularization__.
 Let's re-initialise our parameters $A$, $B$, and $C$:

```python
A, B, C = 1, -2, -3
```

Let's recompute the derivatives:

```python
pull = 1 # since here we want to maximize the function
dA = X * pull
dB = Y * pull
dC = 1 * pull
print (f'dF/dA: {dA}\ndF/dB: {dB}\ndF/dC: {dC}')
```

    dF/dA: 1.2
    dF/dB: 0.7
    dF/dC: 1

When updating the parameters $A$, $B$, and $C$, regularization means that we now additionaly subtract the value of each parameter from its original value, multiplied by a constant that determines the strenght of regularization:

```python
step_size = 0.01
reg_strength = 0.2
A = A + step_size * (dA -A * reg_strength)
B = B + step_size * (dB -B * reg_strength)
C = C + step_size * pull;
```

If we again evaluate the function $f(\mathbf{X},\mathbf{Y})$, with the updated parameters $A$, $B$, and $C$, we get:

```python
A*X + B*Y + C
```

    -3.1703

This result is again better then the original result `-3.2` !

***

### Simple linear classification algorithm

Here, the previous steps are put together in an working algorithm.

We can represent the data as a 2-dimensional numpy array

```python
data = np.array([[ 1.2, 0.7],
                 [-0.3,-0.5],
                 [ 3.0, 0.1],
                 [-0.1,-1.0],
                 [-1.0, 1.1],
                 [ 2.1,-3.0]])
```

We can represent the labels as a simple numpy array of numbers

```python
labels = np.array([ 1,
                   -1,
                    1,
                   -1,
                   -1,
                   +1])
```

We can plot the data using the following function:

```python
def plot_data(data, labels):
    fig = plt.figure(figsize=(5,5))
    ax = fig.add_subplot(111)
    ax.scatter(data[:,0], data[:,1], c=labels, s=50,  cmap=plt.cm.bwr,zorder=50)
    nudge = 0.08
    for i in range(data.shape[0]):
        d = data[i]
        ax.annotate(f'{i}',(d[0]+nudge,d[1]+nudge))
    ax.set_aspect('equal', 'datalim')
    plt.show()
```

```python
plot_data(data,labels)
```

This is a function to evaluate the accuracy of the training

```python
def eval_accuracy(X,Y,A,B,C):
    num_correct = 0;
    data_len = data.shape[0]

    for i in range(data_len):
        X,Y = data[i]
        true_label = labels[i]

        output = A*X + B*Y + C
        predicted_label = 1 if output > 0 else -1

        if (predicted_label == true_label):
            num_correct += 1
    return num_correct / data_len

def create_meshgrid(data):
    h = 0.02
    x_min, x_max = data[:, 0].min() - 1, data[:, 0].max() + 1
    y_min, y_max = data[:, 1].min() - 1, data[:, 1].max() + 1
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    return (xx,yy,np.ones(xx.shape))

def plot_learning_simple(grid,data,A,B,C,iteration, accuracy):
    xx,yy,Z = grid

    for i in range(xx.shape[0]): # row
        for j in range(yy.shape[1]): #column
            X, Y = xx[i][j],yy[i][j]
            output = A*X + B*Y + C
            output = 1 if output > 0 else -1
            Z[i][j] = output

    fig = plt.figure(figsize=(5,5))
    ax = fig.add_subplot(111)
    plt.title(f'accuracy at the iteration {iteration}: {accuracy}')
    ax.contourf(xx, yy, Z, cmap=plt.cm.binary, alpha=0.1, zorder=15)
    ax.scatter(data[:, 0], data[:, 1], c=labels, s=50,  cmap=plt.cm.bwr,zorder=50)
    ax.set_aspect('equal')
    nudge = 0.08
    for i in range(data.shape[0]):
        d = data[i]
        ax.annotate(f'{i}',(d[0]+nudge,d[1]+nudge))
    plt.xlim(xx.min(), xx.max())
    plt.ylim(yy.min(), yy.max())
    plt.show()
```

Here is the main algorithm.

```python
def train_neural_network(data, labels, step_size, no_loops, iter_info):
    #A, B, and C are parameters of the function F. Here, they are set to 1, -2, -1
    A, B, C = 1, -2, -1
    # regularization strenght
    reg_strength = 0.2
    # this function is used for plotting, it can be ignored
    grid = create_meshgrid(data)

    # the main training loop
    for i in range(no_loops):
        # we randomly select the data point, and store its info into: x,y,label
        index = np.random.randint(data.shape[0])
        X,Y = data[index]
        label = labels[index]
        # we calculate the output of the function
        output = A*X + B*Y + C
        # We need to define how to affect parameters.
        # If the label is 1 but the output is smaller than 1, we want to push the score up.
        # If the label is -1 but the output is larger than -1, we want to push the score down.
        pull = 0.0
        if (label == 1 and output < 1):
            pull = 1.0
        if (label ==-1 and output > -1):
            pull = -1.0
        # here we update the values with the partial derivatives, and regularization.
        # partial derivative of dF/dA is X, dF/dB is Y, and of dF/dC is 1.
        # regularization means to pull the parameters A, B, and C towards zero (in the oposite direction)
        A += step_size * (X * pull - A*reg_strength) # -A*reg_strength is from the regularization
        B += step_size * (Y * pull - B*reg_strength) # -A*reg_strength is from the regularization
        C += step_size * pull;

        # after a number of iterations, show training accuracy and plot it
        if (i%iter_info==0):
            accuracy = eval_accuracy(X,Y,A,B,C)
            plot_learning_simple(grid,data,A,B,C,i,accuracy)
    # the algorithm returns the learned parameters A, B, and C
    return (A,B,C)
```

```python
train_1 = train_neural_network(data, labels, 0.01, 1000, 200)
```

We can inspect the result by comparing the real label of a data point and the predicted label:

```python
def show_prediction_SVM_simple(train, data, labels):
    a, b, c = train
    for i in range(data.shape[0]):
        x,y = data[i]
        label = labels[i]
        score = a*x + b*y + c
        score = 1 if score > 0 else -1
        print (f'data point {i}: real label : {label}, pred. label: {score}, {(score==label)}')
```

```python
show_prediction_SVM_simple(train_1,data,labels)
```

    data point 0: real label : 1, pred. label: 1, True
    data point 1: real label : -1, pred. label: -1, True
    data point 2: real label : 1, pred. label: 1, True
    data point 3: real label : -1, pred. label: -1, True
    data point 4: real label : -1, pred. label: -1, True
    data point 5: real label : 1, pred. label: 1, True

05. Binary Classification Using a Simple Linear Function.ipynb


# How to size your bets - The Kelly Rule

he Kelly rule has a long history in gambling because it provides guidance on how much to stake on each of an (infinite) sequence of bets with varying (but favorable) odds to maximize terminal wealth. It was published as A New Interpretation of the Information Rate in 1956 by John Kelly who was a colleague of Claude Shannon's at Bell Labs. He was intrigued by bets placed on candidates at the new quiz show The $64,000 Question, where a viewer on the west coast used the three-hour delay to obtain insider information about the winners.

Kelly drew a connection to Shannon's information theory to solve for the bet that is optimal for long-term capital growth when the odds are favorable, but uncertainty remains. His rule maximizes logarithmic wealth as a function of the odds of success of each game, and includes implicit bankruptcy protection since log(0) is negative infinity so that a Kelly gambler would naturally avoid losing everything.

## Imports

```python
%matplotlib inline
from sympy import symbols, solve, log, diff
from scipy.optimize import minimize_scalar, newton, minimize
from scipy.integrate import quad
from scipy.stats import norm
import numpy as np
import pandas as pd
from numpy.linalg import inv
import matplotlib.pyplot as plt
from numpy.random import dirichlet
import warnings
```

```python
warnings.filterwarnings('ignore')
plt.style.use('fivethirtyeight')
np.random.seed(42)
```

## The optimal size of a bet

Kelly began by analyzing games with a binary win-lose outcome. The key variables are:
- b: The odds define the amount won for a \\$1 bet. Odds = 5/1 implies a \\$5 gain if the bet wins, plus recovery of the \\$1 capital.
- p: The probability defines the likelihood of a favorable outcome.
- f: The share of the current capital to bet.
- V: The value of the capital as a result of betting.

The Kelly rule aims to maximize the value's growth rate, G, of infinitely-repeated bets (see Chapter 5 for background).
$$G=\lim_{N\rightarrow\infty}=\frac{1}{N}\log\frac{V_N}{V_0}$$

We can maximize the rate of growth G by maximizing G with respect to f, as illustrated using sympy as follows:

```python
share, odds, probability = symbols('share odds probability')
Value = probability * log(1 + odds * share) + (1 - probability) * log(1 - share)
solve(diff(Value, share), share)
```

    [(odds*probability + probability - 1)/odds]

```python
f, p = symbols('f p')
y = p * log(1 + f) + (1 - p) * log(1 - f)
solve(diff(y, f), f)
```

    [2*p - 1]

## Get S&P 500 Data

```python
with pd.HDFStore('../../data/assets.h5') as store:
    sp500 = store['sp500/prices'].close
```

### Compute Returns & Standard Deviation

```python
annual_returns = sp500.resample('A').last().pct_change().to_frame('sp500')
```

```python
return_params = annual_returns.sp500.rolling(25).agg(['mean', 'std']).dropna()
```

```python
return_ci = (return_params[['mean']]
                .assign(lower=return_params['mean'].sub(return_params['std'].mul(2)))
                .assign(upper=return_params['mean'].add(return_params['std'].mul(2))))
```

```python
return_ci.plot(lw=2, figsize=(14, 8));
```

### Kelly Rule for a Single Asset - Index Returns

In a financial market context, both outcomes and alternatives are more complex, but the Kelly rule logic does still apply. It was made popular by Ed Thorp, who first applied it profitably to gambling (described in Beat the Dealer) and later started the successful hedge fund Princeton/Newport Partners.

With continuous outcomes, the growth rate of capital is defined by an integrate over the probability distribution of the different returns that can be optimized numerically.
We can solve this expression (see book) for the optimal f* using the `scipy.optimize` module:

```python
def norm_integral(f, mean, std):
    val, er = quad(lambda s: np.log(1 + f * s) * norm.pdf(s, mean, std),
                               mean - 3 * std,
                               mean + 3 * std)
    return -val
```

```python
def norm_dev_integral(f, mean, std):
    val, er = quad(lambda s: (s / (1 + f * s)) * norm.pdf(s, mean, std), m-3*std, mean+3*std)
    return val
```

```python
def get_kelly_share(data):
    solution = minimize_scalar(norm_integral,
                        args=(data['mean'], data['std']),
                        bounds=[0, 2],
                        method='bounded')
    return solution.x
```

```python
annual_returns['f'] = return_params.apply(get_kelly_share, axis=1)
```

```python
return_params.plot(subplots=True, lw=2, figsize=(14, 8));
```

```python
annual_returns.tail()
```

      sp500
      f

      date

      2014-12-31
      0.113906
      1.999996

      2015-12-31
      -0.007266
      1.999996

      2016-12-31
      0.095350
      1.999996

      2017-12-31
      0.194200
      1.999996

      2018-12-31
      0.014815
      1.999996

### Performance Evaluation

```python
(annual_returns[['sp500']]
 .assign(kelly=annual_returns.sp500.mul(annual_returns.f.shift()))
 .dropna()
 .loc['1900':]
 .add(1)
 .cumprod()
 .sub(1)
 .plot(lw=2));

```

```python
annual_returns.f.describe()
```

    count    205.000000
    mean       1.303795
    std        0.585540
    min        0.000003
    25%        0.830339
    50%        1.227349
    75%        1.999996
    max        1.999996
    Name: f, dtype: float64

```python
return_ci.head()
```

      mean
      lower
      upper

      date

      1814-12-31
      0.074059
      -0.426468
      0.574586

      1815-12-31
      0.070621
      -0.429199
      0.570442

      1816-12-31
      0.064872
      -0.436591
      0.566336

      1817-12-31
      0.075920
      -0.418596
      0.570436

      1818-12-31
      0.074465
      -0.419380
      0.568309

### Compute Kelly Fraction

```python
m = .058
s = .216
```

```python
# Option 1: minimize the expectation integral
sol = minimize_scalar(norm_integral, args=(m, s), bounds=[0., 2.], method='bounded')
print('Optimal Kelly fraction: {:.4f}'.format(sol.x))
```

    Optimal Kelly fraction: 1.1974

```python
# Option 2: take the derivative of the expectation and make it null
x0 = newton(norm_dev_integral, .1, args=(m, s))
print('Optimal Kelly fraction: {:.4f}'.format(x0))
```

    Optimal Kelly fraction: 1.1974

## Kelly Rule for Multiple Assets

We will use an example with various equities. [E. Chan (2008)](https://www.amazon.com/Quantitative-Trading-Build-Algorithmic-Business/dp/0470284889) illustrates how to arrive at a multi-asset application of the Kelly Rule, and that the result is equivalent to the (potentially levered) maximum Sharpe ratio portfolio from the mean-variance optimization.

The computation involves the dot product of the precision matrix, which is the inverse of the covariance matrix, and the return matrix:

```python
with pd.HDFStore('../../data/assets.h5') as store:
    sp500_stocks = store['sp500/stocks'].index
    prices = store['quandl/wiki/prices'].adj_close.unstack('ticker').filter(sp500_stocks)
```

```python
prices.info()
```

    <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 14277 entries, 1962-01-02 to 2018-03-27
    Columns: 501 entries, MMM to ZTS
    dtypes: float64(501)
    memory usage: 54.7 MB

```python
monthly_returns = prices.loc['1988':'2017'].resample('M').last().pct_change().dropna(how='all').dropna(axis=1)
stocks = monthly_returns.columns
monthly_returns.info()
```

    <class 'pandas.core.frame.DataFrame'>
    DatetimeIndex: 359 entries, 1988-02-29 to 2017-12-31
    Freq: M
    Columns: 217 entries, MMM to XRX
    dtypes: float64(217)
    memory usage: 611.4 KB

### Compute Precision Matrix

```python
cov = monthly_returns.cov()
precision_matrix = pd.DataFrame(inv(cov), index=stocks, columns=stocks)
```

```python
kelly_allocation = monthly_returns.mean().dot(precision_matrix)
```

```python
kelly_allocation.describe()
```

    count    217.000000
    mean       0.213979
    std        3.493734
    min      -10.840354
    25%       -1.845360
    50%        0.002192
    75%        1.990041
    max       11.316506
    dtype: float64

```python
kelly_allocation.sum()
```

    46.433472367624404

### Largest Portfolio Allocation

The plot shows the tickers that receive an allocation weight > 5x their value:

```python
kelly_allocation[kelly_allocation.abs()>5].sort_values(ascending=False).plot.barh(figsize=(8, 10))
plt.yticks(fontsize=12)
plt.tight_layout()
```

### Performance vs SP500

The Kelly rule does really well. But it has also been computed from historical data..

```python
ax = monthly_returns.loc['2010':].mul(kelly_allocation.div(kelly_allocation.sum())).sum(1).to_frame('Kelly').add(1).cumprod().sub(1).plot();
sp500.loc[monthly_returns.loc['2010':].index].pct_change().add(1).cumprod().sub(1).to_frame('SP500').plot(ax=ax, legend=True);
```

```python

```