# Model Project 2021: Solow model with economical damage from climate change

The aim of the project is to solve the basic Solow model and then extend the model to account for economical damage from climate change.

**The project is organized as follows:**

1. Model project description
2. Model description
3. Classic Solow model
4. Extended Solow model (with economic damage from climate change)
5. Conclusion

**Group:** SSTL

**Members:** Louise Otte Arildsen (srb330), Signe Kolind (hmk792), Stine Fürst (qvj635) & Thomas Nielsen (qzw392)

**Imports and set magics:**

```python
import numpy as np
import matplotlib.pyplot as plt
from scipy import optimize
import sympy as sm
import ipywidgets as widgets
```

```python
import modelproject as mp
```

# Model description

$$
\begin{align}
Y_{t}&=[1-D] K_{t}^{\alpha}\left(A_{t} L_{t}\right)^{1-\alpha}, 0<\alpha<1 \\
D&=1-\frac{1}{1+\sigma(\Delta T)^{2}}, \sigma>0 \\
K_{t+1}&=s Y_{t}+(1-\delta) K_{t}, 0<s<1,0<\delta<1, K_{0}>0  \\
A_{t+1}&=(1+g) A_{t}, A_{0}>0  \\
L_{t+1}&=(1+n) L_{t}, L_{0}>0
\end{align}
$$

$$
\tilde{y}_{t} \equiv \frac{Y_{t}}{A_{t} L_{t}}=\frac{y_{t}}{A_{t}} ; \quad \tilde{k}_{t} \equiv \frac{K_{t}}{A_{t} L_{t}}=\frac{k_{t}}{A_{t}}
$$

Equation (1) is a Cobb-Douglas production function, which describes production ($Y_t$) as a function of capital ($K_t$), population size ($L_t$), and the level of technology ($A_t$). The model assumes that climate changes reduce production with a share D, meaning that a share of the production ($DY_t$) is damaged and can not be used. The share (1-D) of the production can be used for consumption and investment. Equation 2 describes the damage caused by climate change, where \DeltaT measures the temperature change. Equation (3) describes the development of capital, where s is the saving rate and delta is the depreciation rate. Equation (4) and (5) describes how the level of technology ($A_t$) and the population size develops over time.

# Classic Solow model (D=0)

When assuming $D=0$ the model reduces to the classic Solow model, which we will solve in this section.

## Analytical solution

The transition equation of the model for capital per effective worker is given by:

$$
\tilde{k}_{t+1}=\frac{1}{(1+n)(1+g)}\left(s \tilde{k}_{t}^{\alpha}+(1-\delta) \tilde{k}_{t}\right)
$$

Note that convergence requires that $n+g+\delta+ng >0$.

The intuition behind the convergence result is decreasing marginal product on capital and constant depletion/attrition. In steady-state this implies that the savings are just as large as they cover for depreciation and thinning of resources.

First we define all the **symbols** using sympy:

```python
y = sm.symbols('y')
k = sm.symbols('k')
K = sm.symbols('K')
alpha = sm.symbols('alpha')
sigma = sm.symbols('sigma')
deltaT = sm.symbols('delta T')
s = sm.symbols('s')
delta = sm.symbols('delta')
g = sm.symbols('g')
n = sm.symbols('n')
```

Then we define the **steady state equation**

```python
ss = sm.Eq(k,(s*k**alpha+(1-delta)*k)/((1+n)*(1+g)))
```

and **solve** it

```python
kss = sm.solve(ss,k)[0]
kss
```

$\displaystyle \left(\frac{\delta + g n + g + n}{s}\right)^{\frac{1}{\alpha - 1}}$

For later use, we turn the solution into a **Python funciton**

```python
#transforming the function to a python function
ss_func = sm.lambdify((s,g,n,alpha,delta),kss)
```

```python
#Defining parameters
s = 0.3
g = 0.02
n = 0.01
alpha = 1/3
delta = 0.05
```

```python
print(f'analytical solution is: {ss_func(s,g,n,alpha,delta):.3f}')
```

    analytical solution is: 7.235

In steady state the capital per effective worker equals 7.235.

## Numerical solution

We can re-write the equation for the steady state capital per capita as

$$
\tilde{k}_{t+1}-\frac{1}{(1+n)(1+g)}\left(s \tilde{k}_{t}^{\alpha}+(1-\delta) \tilde{k}_{t}\right)=0
$$

whereby it clearly becomes a **root-finding problem**. Such a problem can be solved by various methods.

In the section below, the problem will be solved from the **brent-** and **bisection method** in order to compare the robustness of our results.

### Bisection Method

```python
def solve_for_ss(s,g,n,alpha,delta):
    """ solve for the steady state level of capital

    Args:
        s (float): saving rate
        g (float): technological growth rate
        n (float): population growth rate
        alpha (float): cobb-douglas parameter
        delta (float): capital depreciation rate

    Returns:
        result (RootResults): the solution represented as a RootResults object

    """

    # a. define objective function
    obj_kss = lambda kss: kss - (s*kss**alpha + (1-delta)*kss)/((1+g)*(1+n))

    #. b. call root finder
    result = optimize.root_scalar(obj_kss,bracket=[0.1,100],method='bisect')

    return result

#Solve numerically for the steady state:
bisect_solution = solve_for_ss(s,g,n,alpha,delta)
print(f' The numerical solution, calculated from the bisect method is: {bisect_solution.root:}')
```

     The numerical solution, calculated from the bisect method is: 7.234696712204852

```python
def solve_for_ss(s,g,n,alpha,delta,method='bisect'):
    """ solve for the steady state level of capital

    Args:
        s (float): saving rate
        g (float): technological growth rate
        n (float): population growth rate
        alpha (float): cobb-douglas parameter
        delta (float): capital depreciation rate

    Returns:
        result (RootResults): the solution represented as a RootResults object

    """

    # a. define objective function
    obj_kss = lambda kss: kss - (s*kss**alpha + (1-delta)*kss)/((1+g)*(1+n))

    #. b. call root finder
    result = optimize.root_scalar(obj_kss,bracket=[0.1,100],method=method)

    return result

#Solve numerically for the steady state:
bisect_solution = solve_for_ss(s,g,n,alpha,delta)
print(f' The numerical solution, calculated from the bisect method is: {bisect_solution.root:}')
```

     The numerical solution, calculated from the bisect method is: 7.234696712204852

```python
#Solve numerically for the steady state:
brent_solution = solve_for_ss(s,g,n,alpha,delta,method='brentq')
print(f'The numerical solution, calculated from the brent method, is: {brent_solution.root:}')
```

    The numerical solution, calculated from the brent method, is: 7.234696712205987

### Brent Method

```python
def solve_for_brent(s,g,n,alpha,delta):
    """ solve for the steady state level of capital

    Args:
        s (float): saving rate
        g (float): technological growth rate
        n (float): population growth rate
        alpha (float): cobb-douglas parameter
        delta (float): capital depreciation rate

    Returns:
        result (RootResults): the solution represented as a RootResults object

    """

    # a. define objective function
    obj_kss = lambda kss: kss - (s*kss**alpha + (1-delta)*kss)/((1+g)*(1+n))

    #. b. call root finder
    result = optimize.root_scalar(obj_kss,bracket=[0.1,100],method='brentq')

    return result

#Solve numerically for the steady state:
brent_solution = solve_for_brent(s,g,n,alpha,delta)
print(f'The numerical solution, calculated from the brent method, is: {brent_solution.root:}')
```

    The numerical solution, calculated from the brent method, is: 7.234696712205987

When comparing our numerical and analytical results, it is clear that our results are quite robust, only differing by few decimals:

```python
print(f'Analytical solution : {ss_func(s,g,n,alpha,delta):}')
print(f'Bisect solution     : {bisect_solution.root:}')
print(f'Brent solution      : {brent_solution.root:}')

```

    Analytical solution : 7.234696712206231
    Bisect solution     : 7.234696712204852
    Brent solution      : 7.234696712205987

Because our results seem robust, we choose to rely on a single optimization method, when climate change is introduced from section 4 in the extended model.

## GDP per effective worker:

From equation 1 we divide both sides with $A_{t}L_{t}$ and obtain the following expression for GDP per effective worker:

$$
\begin{equation*}
\tilde{y}_{t}=\tilde{k}_{t}^\alpha
\end{equation*}
$$

We insert the expression for $\tilde{k}^*=\left(\frac{\delta + gn + g+n}{s}\right)^{\frac{1}{\alpha - 1}}$ found in section 3.1 in $\tilde{k}_{t}$'s  place.
We then get this expression for GDP per effective worker:

$$
\tilde{y}^{*}=\left(\frac{s}{n+g+\delta+n g}\right)^{\frac{\alpha}{1-\alpha}}
$$

We calculate $\tilde{y}^{*}$ given our parameter values:

```python
gdp_pr_worker=(s/(n+g+delta+n*g))**(alpha/(1-alpha))
print(f'The GDP per effective worker is: {gdp_pr_worker:.3f}')
```

    The GDP per effective worker is: 1.934

# The Solow model with economical damage from climate change (D>0)

In this section we consider an extension of the classic Solow model, which includes economical damage from climate change. D represents the reduction in production due to climate change and we now assume that D>0.

## Analytical solution

The transition equation of the model for capital per effective worker is given by:

$$
\tilde{k}_{t+1}=\frac{1}{(1+n)(1+g)}\left(s[1-D] \tilde{k}_{t}^{\alpha}+(1-\delta) \tilde{k}_{t}\right)
$$

Then we define the **steady state equation**

```python
# introducing climate effects
y = sm.symbols('y')
k = sm.symbols('k')
K = sm.symbols('K')
alpha = sm.symbols('alpha')
sigma = sm.symbols('sigma')
deltaT = sm.symbols('delta T')
s = sm.symbols('s')
delta = sm.symbols('delta')
g = sm.symbols('g')
n = sm.symbols('n')
D = sm.symbols('D')

# climate steady state
css = sm.Eq(k,(((s*(1-D)*(k**alpha))+((1-delta)*k)))/((1+n)*(1+g)))
```

and **solve** it:

```python
kcss = sm.solve(css,k)[0]
kcss
```

$\displaystyle \left(\frac{- \delta - g n - g - n}{s \left(D - 1\right)}\right)^{\frac{1}{\alpha - 1}}$

```python
#transforming the function to a python function
css_func = sm.lambdify((s,g,n,alpha,delta,D),kcss)
```

```python
#Defining parameters
s = 0.3
g = 0.02
n = 0.01
alpha = 1/3
delta = 0.05
D=0.175
print(f'analytical solution is: {css_func(s,g,n,alpha,delta,D):.3f}')
```

    analytical solution is: 5.421

In steady state the capital per effective worker equals 5.421.

## Numerical solution

```python
def solve_for_css(s,g,n,alpha,delta,D):
    """ solve for the steady state level of capital

    Args:
        s (float): saving rate
        g (float): technological growth rate
        n (float): population growth rate
        alpha (float): cobb-douglas parameter
        delta (float): capital depreciation rate

    Returns:
        result (RootResults): the solution represented as a RootResults object

    """

    # a. define objective function
    obj_kcss = lambda kcss: kcss - (s*(1-D)*kcss**alpha + (1-delta)*kcss)/((1+g)*(1+n))

    #. b. call root finder
    result = optimize.root_scalar(obj_kcss,bracket=[0.1,100],method='bisect')

    return result

```

```python
solution_2 = solve_for_css(s,g,n,alpha,delta,D)

print(f'analytical solution is: {css_func(s,g,n,alpha,delta,D):.3f}')
print(f' numerical solution is: {solution_2.root:.3f}')
```

    analytical solution is: 5.421
     numerical solution is: 5.421

## GDP per effective worker with climate change (D = 0.175)

There are different scenarios for the future temperature changes. A scenario with "little CO2 emission" will imply that the global average temperature increases by approx. 1 degree over the next 100 years. The worst case scenario, with "a lot of CO2 emissions", will imply that the global temperature rise at about 4 degrees over the next 100 years. A meta-study by Nordhaus og Moffat (2017) shows that the most pessimistic estimates in the literature suggests that temperature rises of 4 degress are associated with approx. 17.5% loss of production i.e. D = 0.175.

We want to compare the GDP per effective worker in the classic Solow model (D=0) with the GDP per effective in the extended Solow model when the worst possible climate scenario unfolds (D=0.175). We start by calculating the GDP per effecitive worker with D=0.175:

From equation 1 we divide both sides with $A_{t}L_{t}$ and obtain the following expression for GDP per effective worker:

$$
\begin{equation*}
\tilde{y}_{t}=(1-D)\tilde{k}_{t}^\alpha
\end{equation*}
$$

We insert the expression for $\tilde{k}^*=\left(\frac{\delta + gn + g+n}{s(1-D)}\right)^{\frac{1}{\alpha - 1}}$ found in section 4.1 in $\tilde{k}_{t}$'s  place.
We then get this expression for GDP per effective worker:

$$
\tilde{y}^{*}=(1-D)^{\frac{1}{1-\alpha}} \left(\frac{s}{n+g+\delta+n g}\right)^{\frac{\alpha}{1-\alpha}}
$$

We calculate $\tilde{y}^{*}$ given our parameter values:

```python
D=0.175
gdp_pr_worker_climate=(1-D)**(1/(1-alpha))*(s/(n+g+delta+n*g))**(alpha/(1-alpha))
print(f'The GDP per effective worker when accounting for climate change is: {gdp_pr_worker_climate:.3f}')
```

    The GDP per effective worker when accounting for climate change is: 1.449

### Compare GDP per effective worker with the model without climate change

```python
comparison= 1-gdp_pr_worker_climate/gdp_pr_worker
print(f'The reduction in GDP per effective worker when accounting for climate change: {comparison:.3f}')
```

    The reduction in GDP per effective worker when accounting for climate change: 0.251

This imply that GDP per efficient worker decreases by approx. 25 pct. This effect is greater than the initial decline of 17.5 pct. The reason for this difference is due to negative effects via capital accumulation.

```python
def Solaris(D,alpha,s,n,g,delta):
    return 1-((1-D)**(1/(1-alpha))*(s/(n+g+delta+n*g))**(alpha/(1-alpha)))/(s/(n+g+delta+n*g))**(alpha/(1-alpha))
```

We create an interactive widget where climate changes reduces the production with a share D and thus also reduces the GDP per effective worker:

```python
widgets.interact(Solaris,
                 alpha= widgets.fixed(1/3),
                 s=widgets.fixed(0.3),
                 g=widgets.fixed(0.02),
                 n=widgets.fixed(0.01),
                 delta=widgets.fixed(0.05),
                 D= widgets.FloatSlider(description ='$D$',
                                        min = 0.0 , max = 1 , step = 0.005, value = 0.17));
```

    interactive(children=(FloatSlider(value=0.17, description='$D$', max=1.0, step=0.005), Output()), _dom_classes…

We see that for any given level of D the reduction in GDP per capita is greater than the reduction in production due to the negative affect froms capital accumulation.

## Visulization

The Solow model attemps to explain how long-run economic growth is determined from capital accumulation, population(labor) growth and increase in productivity. The key takeway, is that the economy will grow towards a steady state, as long as per capita investments sf(k) is larger than what is needed to maintain the current level of capital. In order to maintain the population at the given level of k, both labor and capital must be “employed” and replaced by new units, as of regards to depreciation and population growth k(n+d+g) and in our case *climate change, D*. As shown in the transition equation, D, will essentially decrease the ammount of savings in the economy, which implies that the economy will converge towards a lower steady state, since less capital can be accumulated.

The process is illustrated in the below visualization of the Solow model.
Note that the default setting of widget paratemeter "D" is value=0.

In order to view the effects of climate changes and compare with our prior steady state result, please change the value of D to value = 0.175):

```python
def simulate_basic(k,alpha,delta,s,n,g,K,T,D):
    """
    Args:
    k      (int)  : initial value for physical capital per worker
    alpha  (float): return on physical capital
    delta  (float): depreciation rate on capital
    s      (float): savings/investments
    n      (float): increase in labour
    g             : growth
    K      (int)  : totalfactorproductivity
    T      (int)  : periods to simulate through
    D             : climate change

    Returns:
    A plot showing how capital per worker accumulates over time"""
```

```python
widgets.interact(mp.simulate_basic,
                k     = widgets.fixed(0),
                alpha = widgets.FloatSlider(description = '$\u03B1$' , min = 0 ,    max = 0.9 , step = 0.05 , value = 1/3),
                delta = widgets.FloatSlider(description = '$\u03B4$' , min = 0 ,    max = 0.5 , step = 0.01 , value = 0.05),
                s     = widgets.FloatSlider(description = 's' ,        min = 0.01 , max = 0.3 , step = 0.005 , value = 0.3),
                n     = widgets.FloatSlider(description ='$n$',        min = 0.01 , max = 0.1 , step = 0.005, value = 0.01),
                g     = widgets.FloatSlider(description ='$g$',        min = 0.01 , max = 0.1 , step = 0.005, value = 0.02),
                D     = widgets.FloatSlider(description ='$D$',        min = 0.0 , max = 1 , step = 0.005, value = 0),
                K     = widgets.fixed(1),
                T     = widgets.IntSlider(description='$T$' ,          min = 1,     max = 1000, step = 10,    value = 10))
```

    interactive(children=(FloatSlider(value=0.3333333333333333, description='$α$', max=0.9, step=0.05), FloatSlide…

    <function modelproject.simulate_basic(k, alpha, delta, s, n, g, K, T, D)>

## Simulation of the classic and extended Solow model

We want to simulate the classic and the extended Solow model, to show how the two models converges to their steady state, $\tilde{k}^*$.

We choose the initial capital and create a function that simulates the Solow model for $T=100$ periods.

```python
k0 =1
T = 100

def solow_equation(k,alpha,delta,s,n,g, D):
    """ calculate capital in the next-period

    Args:

        k (float): capital in this period
        alpha (float): cobb-douglas parameter
        delta (float): depreciation rate
        s (float): saving rate
        n : population growth
        g : technology growth

    Returns:

        k_plus (float): capital in next period

    """

    saving = s*k**alpha
    depreciation = delta*k
    k_plus = (k + saving*(1-D) - depreciation)/((1+n)*(1+g))

    return k_plus

def simulate_solow_model(k0,alpha,delta,s,n,g,T,D):
    """ simulate the solow model

    Args:

        k0 (float): initial level of kapital
        alpha (float): cobb-douglas parameter
        delta (float): depreciation rate
        s (float): saving rate
        T (int): number of periods to simulate

    Returns:

        k_path (list): capital path (length T)

    """

    # a. initialize
    k_path = [k0]

    # b. simulate forward
    for t in range(1,T):

        # i. apply solow equation
        k_plus = solow_equation(k_path[t-1],alpha,delta,s,n,g,D)

        # ii. save result
        k_path.append(k_plus)

    return k_path
```

This function represents the steady state value:

```python
def find_ssk_k(delta,g,n,alpha,s,D):
    return ((delta+g*n+g+n)/(s*(1-D)))**(1/(alpha-1))
```

**The classic Solow model (D=0)**

We create the Solow path and the steady state value:

```python
k_path = simulate_solow_model(k0,alpha,delta,s,n,g,T,D=0)
k_star = find_ssk_k(delta,g,n,alpha,s,D=0)
k_stars =[k_star for i in range(T)]
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(k_path[:T], lw=2, label='Solow path')
ax.plot(k_stars,lw=2, label="Steady state")
ax.set_title('Simulated convergence of capital to steady state')
ax.set_xlabel('t')
ax.set_ylabel('Capital per capita')
ax.set_xlim([0,100])
ax.set_ylim([0,10])
ax.legend(loc="lower right")
ax.grid(True);
```

**The extended Solow model (D=0.175)**

We create the Solow path and the steady state value:

```python
k_path_ext = simulate_solow_model(k0,alpha,delta,s,n,g,T,D=0.175)
k_star_ext = find_ssk_k(delta,g,n,alpha,s,D=0.175)
k_stars_ext =[k_star_ext for i in range(T)]
```

```python
fig = plt.figure()
ax = fig.add_subplot(1,1,1)

ax.plot(k_path_ext[:T], lw=2, label='Solow path')
ax.plot(k_stars_ext,lw=2, label="Steady state")
ax.set_title('Simulated convergence of capital to steady state')
ax.set_xlabel('t')
ax.set_ylabel('Capital per capita')
ax.set_xlim([0,100])
ax.set_ylim([0,10])
ax.legend(loc="lower right")
ax.grid(True);
```

We find that the convergence is little bit faster in the extended Solow model with economic damage from climate change.

# Conclusion

We have solved the basic Solow model and an extended Solow model, which account for economic damage from climate change. We have analytically solved the model using Pythons Sympy and numerically using two different methods.

For the classical Solow model, we find that in steady-state, the capital per effective worker equals 7.235. The result was constant despite the different approaches. Furthermore, we find that The GDP per effective worker is 1.934.

For the extended model, we conclude that the capital per effective worker equals 5.421 in the analytical and numerical solution, and the GDP per effective worker is 1.449.

Comparing the two models, we find that for any given level of D the reduction in GDP per capita is greater than the reduction in production due to the negative effect from capital accumulation.


Moreschi (1986) Eqs. (A53) — (A57) give the transformation law as
\begin{align}
  u' &= k(u - \alpha) + \Omega \Delta u \\
  \zeta' &= \frac{a \zeta + b} {c \zeta + d} + \Omega \Delta \zeta \\
  \bar{\zeta}' &= \frac{\bar{a} \bar{\zeta} + \bar{b}} {\bar{c} \bar{\zeta} + \bar{d}} + \Omega \overline{\Delta \zeta} \\
  \Omega' &= k\, \Omega
\end{align}
where $k$ is a function of $(\zeta, \bar{\zeta})$ and the first-order parts are
\begin{equation}
  \Delta u = - \frac{1}{k} \eth u' \bar{\eth} u'
  \qquad \text{and} \qquad
  \Delta \zeta =  - \frac{1}{k} \left( \eth u' \bar{\eth} \zeta' + \eth \zeta' \bar{\eth} u' \right).
\end{equation}

Note that with $P = \frac{1}{2} (1 + \zeta \bar{\zeta})$, we can also write
\begin{equation}
  \eth f = P^{1-s} \frac{\partial} {\partial \zeta} (P^s f)
\end{equation}
for functions of spin weight $s$.  [Here, we will be consistently interpreting Moreschi's $\eth$ differently from how he means it by a factor of $\sqrt{2}$, just as a consistency test.  The results should be the same, regardless.  See below for a derivation using his interpretation.]  In particular, for $s=0$, we have $\partial_\zeta = P^{-1} \eth$.

In particular, we see that — at least in the limit $\Omega \to 0$ — we have $\bar{\eth} \zeta' = 0$ and $\eth \bar{\zeta}' = 0$, so the expression for $\Delta \zeta$ simplifies to
\begin{equation}
  \Delta \zeta =  - \frac{1}{k} \eth \zeta' \bar{\eth} u'.
\end{equation}

We can see that
\begin{equation}
  \frac{\partial \zeta'} {\partial \zeta} = \frac{1} {(c\zeta + d)^2},
\end{equation}
and Moreschi's Eq. (A52) gives
\begin{equation}
  k = \frac{2P} {|a \zeta + b|^2 + |c \zeta + d|^2}
\end{equation}

A useful expression is
\begin{equation}
  \frac{P'}{P} \frac{1} {\frac{\partial \zeta'} {\partial \zeta}}
  = \frac{|a \zeta + b|^2 + |c \zeta + d|^2} {(1 + \zeta \bar{\zeta})(\bar{c} \bar{\zeta} + \bar{d})(c \zeta + d)} (c \zeta + d)^2
  = \frac{c \zeta + d} {k (\bar{c} \bar{\zeta} + \bar{d})}
  = \frac{\exp(i \lambda)} {k},
\end{equation}
where the last expression may as well define $\exp(i \lambda)$, and can also be written as $\frac{P'} {\eth\zeta'} = \frac{\exp(i\lambda)}{k}$.

$\Delta \zeta = - \frac{1}{k} P \frac{\partial \zeta'} {\partial \zeta} \bar{\eth} u'$

# Manual derivation

\begin{align}
  \left. \begin{pmatrix}
  \frac{\partial u} {\partial u'} &
  \frac{\partial \zeta} {\partial u'} &
  \frac{\partial \bar{\zeta}} {\partial u'} &
  \frac{\partial \Omega} {\partial u'}
  \\
  \frac{\partial u} {\partial \zeta'} &
  \frac{\partial \zeta} {\partial \zeta'} &
  \frac{\partial \bar{\zeta}} {\partial \zeta'} &
  \frac{\partial \Omega} {\partial \zeta'}
  \\
  \frac{\partial u} {\partial \bar{\zeta}'} &
  \frac{\partial \zeta} {\partial \bar{\zeta}'} &
  \frac{\partial \bar{\zeta}} {\partial \bar{\zeta}'} &
  \frac{\partial \Omega} {\partial \bar{\zeta}'}
  \\
  \frac{\partial u} {\partial \Omega'} &
  \frac{\partial \zeta} {\partial \Omega'} &
  \frac{\partial \bar{\zeta}} {\partial \Omega'} &
  \frac{\partial \Omega} {\partial \Omega'}
  \end{pmatrix} \right|_{\Omega=0}
  &=
  \left. \begin{pmatrix}
  \frac{\partial u'} {\partial u} &
  \frac{\partial \zeta'} {\partial u} &
  \frac{\partial \bar{\zeta}'} {\partial u} &
  \frac{\partial \Omega'} {\partial u}
  \\
  \frac{\partial u'} {\partial \zeta} &
  \frac{\partial \zeta'} {\partial \zeta} &
  \frac{\partial \bar{\zeta}'} {\partial \zeta} &
  \frac{\partial \Omega'} {\partial \zeta'}
  \\
  \frac{\partial u'} {\partial \bar{\zeta}} &
  \frac{\partial \zeta'} {\partial \bar{\zeta}} &
  \frac{\partial \bar{\zeta}'} {\partial \bar{\zeta}} &
  \frac{\partial \Omega'} {\partial \bar{\zeta}}
  \\
  \frac{\partial u'} {\partial \Omega} &
  \frac{\partial \zeta'} {\partial \Omega} &
  \frac{\partial \bar{\zeta}'} {\partial \Omega} &
  \frac{\partial \Omega'} {\partial \Omega}
  \end{pmatrix} \right|_{\Omega=0} ^{-1}
  \\ &=
  \begin{pmatrix}
  k &
  0 &
  0 &
  0
  \\
  P^{-1} \eth u' &
  \frac{\partial \zeta'} {\partial \zeta} &
  0 &
  0
  \\
  P^{-1} \bar{\eth} u' &
  0 &
  \frac{\partial \bar{\zeta}'} {\partial \bar{\zeta}} &
  0
  \\
  \Delta u &
  \Delta \zeta &
  \overline{\Delta \zeta} &
  k
  \end{pmatrix}^{-1}
  \\ &=
  \begin{pmatrix}
  1/k &
  0 &
  0 &
  0
  \\
  - \frac{\eth u'} {P k \frac{\partial \zeta'} {\partial \zeta}} &
  1 / \frac{\partial \zeta'} {\partial \zeta} &
  0 &
  0
  \\
  - \frac{\bar{\eth} u'} {P k \frac{\partial \bar{\zeta}'} {\partial \bar{\zeta}}} &
  0 &
  1 / \frac{\partial \bar{\zeta}'} {\partial \bar{\zeta}} &
  0
  \\
  - \frac{\Delta u} {k^2}
  + \frac{\Delta \zeta\, \eth u'} {P k^2 \frac{\partial \zeta'} {\partial \zeta}}
  + \frac{\overline{\Delta \zeta}\, \bar{\eth} u'} {P k^2 \frac{\partial \bar{\zeta}'} {\partial \bar{\zeta}}} &
  -\frac{\Delta \zeta} {k \frac{\partial \zeta'} {\partial \zeta}} &
  -\frac{\overline{\Delta \zeta}} {k \frac{\partial \bar{\zeta}'} {\partial \bar{\zeta}}} &
  1/k
  \end{pmatrix}
  \\ &=
  \begin{pmatrix}
  1/k &
  0 &
  0 &
  0
  \\
  - \frac{\eth u'} {P k \frac{\partial \zeta'} {\partial \zeta}} &
  1 / \frac{\partial \zeta'} {\partial \zeta} &
  0 &
  0
  \\
  - \frac{\bar{\eth} u'} {P k \frac{\partial \bar{\zeta}'} {\partial \bar{\zeta}}} &
  0 &
  1 / \frac{\partial \bar{\zeta}'} {\partial \bar{\zeta}} &
  0
  \\
  -\frac{\eth u'\, \bar{\eth} u'} {k^3} &
  \frac{P \bar{\eth} u'} {k^2} &
  \frac{P \eth u'} {k^2} &
  1/k
  \end{pmatrix}
\end{align}

\begin{align}
  \mathbf{n}'
  &=
  \frac{\partial} {\partial u'}
  \\ &=
  \frac{\partial u} {\partial u'} \frac{\partial} {\partial u} + \frac{\partial \zeta} {\partial u'} \frac{\partial} {\partial \zeta} + \frac{\partial \bar{\zeta}} {\partial u'} \frac{\partial} {\partial \bar{\zeta}} + \frac{\partial \Omega} {\partial u'} \frac{\partial} {\partial \Omega}
  \\ &=
  \frac{1} {k} \frac{\partial} {\partial u}
  \\ &=
  \frac{1} {k} \mathbf{n}
  \\
  \mathbf{m}'
  &=
  P' \frac{\partial} {\partial \zeta'}
  \\ &=
  P' \frac{\partial u} {\partial \zeta'} \frac{\partial} {\partial u} + P' \frac{\partial \zeta} {\partial \zeta'} \frac{\partial} {\partial \zeta} + P' \frac{\partial \bar{\zeta}} {\partial \zeta'} \frac{\partial} {\partial \bar{\zeta}} + P' \frac{\partial \Omega} {\partial \zeta'} \frac{\partial} {\partial \Omega}
  \\ &=
  - \frac{P'}{P} \frac{\eth u'} {k \frac{\partial \zeta'} {\partial \zeta}} \frac{\partial} {\partial u} + \frac{P'}{P} \frac{1} {\frac{\partial \zeta'} {\partial \zeta}} P \frac{\partial} {\partial \zeta}
  \\ &=
  - \frac{\exp(i\lambda) \eth u'} {k^2} \frac{\partial} {\partial u} + \frac{\exp(i\lambda)} {k} P \frac{\partial} {\partial \zeta}
  \\ &=
  \frac{\exp(i\lambda)} {k} \left( \mathbf{m} - \frac{\eth u'} {k} \mathbf{n}  \right)
  \\
  \mathbf{l}'
  &=
  -\frac{\partial} {\partial \Omega'}
  \\ &=
  -\frac{\partial u} {\partial \Omega'} \frac{\partial} {\partial u} - \frac{\partial \zeta} {\partial \Omega'} \frac{\partial} {\partial \zeta} - \frac{\partial \bar{\zeta}} {\partial \Omega'} \frac{\partial} {\partial \bar{\zeta}} - \frac{\partial \Omega} {\partial \Omega'} \frac{\partial} {\partial \Omega}
  \\ &=
  -\frac{\partial u} {\partial \Omega'} \mathbf{n} - \frac{\partial \zeta} {\partial \Omega'} P^{-1} \mathbf{m} - \frac{\partial \bar{\zeta}} {\partial \Omega'} P^{-1} \bar{\mathbf{m}} + \frac{\partial \Omega} {\partial \Omega'} \mathbf{l}
  \\ &=
  \frac{\eth u'\, \bar{\eth} u'} {k^3} \mathbf{n} - \frac{\bar{\eth} u'} {k^2} \mathbf{m} - \frac{\eth u'} {k^2} \bar{\mathbf{m}} + \frac{1} {k} \mathbf{l}
\end{align}

# Sympy derivation

```python
import sympy
from sympy import symbols, sin, cos, exp, cot, pi, I
from sympy.diffgeom import Manifold, Patch, CoordSystem, TensorProduct, vectors_in_basis
from sympy import init_printing
init_printing()

u, Ω = symbols("u, Omega", real=True)
ζ, ζ̄ = symbols(r"\zeta, \bar{\zeta}")
up, ðup, ð̄up = symbols(r"u', {\eth}u', \bar{\eth}u'", cls=sympy.Function, real=True)
Ωp = symbols("Omega'", real=True)
ζp, ζ̄p = symbols(r"\zeta', \bar{\zeta}'", cls=sympy.Function)
κ, ðκ, ð̄κ = symbols('κ, ðκ, ð̄κ', cls=sympy.Function, real=True)
Δu, Δζ = symbols('Δu, Δζ')  # See note below about why these aren't functions
P = symbols('P', cls=sympy.Function)
ðζp, ð̄ζ̄p = symbols(r"ðζ', \bar{\eth}\bar{ζ}'", cls=sympy.Function)
λ = symbols(r"\lambda", real=True)

m = Manifold('M', 4)
patch = Patch('P', m)
asymptotic_stereographic = CoordSystem('asymptotic_stereographic', patch, ["u", "Ω", "ζ", "ζ̄"])
asymptotic_stereographic_transformed = CoordSystem('asymptotic_stereographic_transformed', patch, ["u'", "Ω'", "ζ'", "ζ̄'"])
```

```python
# NOTE: Strictly speaking, Δu and Δζ should be functions of (u, ζ, ζ̄).  However, I'm only interested in
# the asymptotic behavior, which means that any derivatives of these variables would still be multiplied
# by Ω, and thus would disappear in the limit Ω → 0.

asymptotic_stereographic.connect_to(
    asymptotic_stereographic_transformed,
    [u, Ω, ζ, ζ̄],
    [
        up(u, ζ, ζ̄) - Ω * ðup(u, ζ, ζ̄) * ð̄up(u, ζ, ζ̄) / κ(ζ, ζ̄),
        κ(ζ, ζ̄) * Ω,
        ζp(ζ) - Ω * ð̄up(u, ζ, ζ̄) * ðζp(ζ) / κ(ζ, ζ̄),  # (a * ζ + b) / (c * ζ + d)
        ζ̄p(ζ̄) - Ω * ðup(u, ζ, ζ̄) * ð̄ζ̄p(ζ̄) / κ(ζ, ζ̄),  # (a.conjugate() * ζ̄ + b.conjugate()) / (c.conjugate() * ζ̄ + d.conjugate())
    ],
    inverse=False
)

J = sympy.simplify(asymptotic_stereographic.jacobian(asymptotic_stereographic_transformed, [u, Ω, ζ, ζ̄]))
```

```python
d = {
    sympy.Derivative(up(u, ζ, ζ̄), u): κ(ζ, ζ̄),
    sympy.Derivative(up(u, ζ, ζ̄), ζ): ðup(u, ζ, ζ̄) / sympy.sqrt(2) / P(ζ, ζ̄),
    sympy.Derivative(up(u, ζ, ζ̄), ζ̄): ð̄up(u, ζ, ζ̄) / sympy.sqrt(2) / P(ζ, ζ̄),
    sympy.Derivative(κ(ζ, ζ̄), ζ): ðκ(ζ, ζ̄) / sympy.sqrt(2) / P(ζ, ζ̄),
    sympy.Derivative(κ(ζ, ζ̄), ζ̄): ð̄κ(ζ, ζ̄) / sympy.sqrt(2) / P(ζ, ζ̄),
    sympy.Derivative(ζp(ζ), ζ): ðζp(ζ) / sympy.sqrt(2) / P(ζ, ζ̄),
    sympy.Derivative(ζ̄p(ζ̄), ζ̄): ð̄ζ̄p(ζ̄) / sympy.sqrt(2) / P(ζ, ζ̄),
    Δu: - ðup(u, ζ, ζ̄) * ð̄up(u, ζ, ζ̄) / κ(ζ, ζ̄),
    Δζ: - ð̄up(u, ζ, ζ̄) * ðζp(ζ) / κ(ζ, ζ̄),
    ð̄up(u, ζ, ζ̄).conjugate(): ðup(u, ζ, ζ̄),
    ðup(u, ζ, ζ̄).conjugate(): ð̄up(u, ζ, ζ̄),
#     ð̄ζ̄p(ζ̄).conjugate(): ðζp(ζ),
#     ðζp(ζ).conjugate(): ð̄ζ̄p(ζ̄)
}
```

```python
lᵃ = -asymptotic_stereographic.base_vector(1)
nᵃ = asymptotic_stereographic.base_vector(0)
mᵃ = sympy.sqrt(2) * P(ζ, ζ̄) * asymptotic_stereographic.base_vector(2)
m̄ᵃ = sympy.sqrt(2) * P(ζ, ζ̄) * asymptotic_stereographic.base_vector(3)
```

\begin{align}
  \sqrt{\frac{\partial \bar{\zeta}' / \partial \bar{\zeta}} {\partial \zeta' / \partial \zeta}}
  &=
  \sqrt{\frac{\bar{\eth} \bar{\zeta}'} {\eth \zeta'}}
  =
  \exp(i\lambda)
\end{align}
\begin{equation}
  \frac{P'} {\eth\zeta'} = \frac{\exp(i\lambda)}{\sqrt{2}k}
\end{equation}

```python
w1, w2, w3 = sympy.symbols('w1, w2, w3', cls=sympy.Wild)

limit_substitutions = {
    Ω: 0,
    Ωp: 0,
    asymptotic_stereographic_transformed.coord_function(1): 0,
    asymptotic_stereographic.coord_function(1): 0,
}

deriv_evaluations = [
    [sympy.Derivative(up(w1, w2, w3), (w1, 1)), κ(ζ, ζ̄)],
    [sympy.Derivative(up(w1, w2, w3), (w2, 1)), ðup(u, ζ, ζ̄) / sympy.sqrt(2) / P(ζ, ζ̄)],
    [sympy.Derivative(up(w1, w2, w3), (w3, 1)), ð̄up(u, ζ, ζ̄) / sympy.sqrt(2) / P(ζ, ζ̄)],
    [sympy.Derivative(ζp(w1), (w1, 1)), ðζp(ζ) / sympy.sqrt(2) / P(ζ, ζ̄)],
]

def replace_derivs(expr):
    for de in deriv_evaluations:
        expr = expr.replace(*de)
    return expr

basis_subs = {
    asymptotic_stereographic.base_vector(0): sympy.Dummy(r"n^a"),
    asymptotic_stereographic.base_vector(1): -sympy.Dummy(r"l^a"),
    asymptotic_stereographic.base_vector(2): sympy.Dummy(r"m^a") / sympy.sqrt(2) / P(ζ, ζ̄),
    asymptotic_stereographic.base_vector(3): sympy.Dummy(r"\bar{m}^a") / sympy.sqrt(2) / P(ζ, ζ̄),
    asymptotic_stereographic_transformed.base_vector(0): sympy.Dummy(r"n'^a"),
    asymptotic_stereographic_transformed.base_vector(1): -sympy.Dummy(r"l'^a"),
    asymptotic_stereographic_transformed.base_vector(2): sympy.Dummy(r"m'^a") / sympy.sqrt(2) / P(ζp(ζ), ζ̄p(ζ̄)),
    asymptotic_stereographic_transformed.base_vector(3): sympy.Dummy(r"\bar{m}'^a") / sympy.sqrt(2) / P(ζp(ζ), ζ̄p(ζ̄)),
}

pretty_subs = {
    P(ζp(ζ), ζ̄p(ζ̄)): ðζp(ζ) * sympy.exp(sympy.I * λ) / sympy.sqrt(2) / κ(ζ, ζ̄),
    ð̄ζ̄p(ζ̄): ðζp(ζ) * sympy.exp(2* sympy.I * λ),
    asymptotic_stereographic.coord_function(0): u,
    asymptotic_stereographic.coord_function(1): Ω,
    asymptotic_stereographic.coord_function(2): ζ,
    asymptotic_stereographic.coord_function(3): ζ̄,
}

def simplify(expr):
    expr = replace_derivs(expr.xreplace(d).xreplace(d).subs(limit_substitutions))
    return sympy.simplify(sympy.expand(sympy.simplify(sympy.expand(expr.subs(basis_subs).subs(pretty_subs).subs(pretty_subs)))))

nᵃᵖ = simplify(vectors_in_basis(nᵃ, asymptotic_stereographic_transformed))
display(sympy.Eq(sympy.Dummy(r"n^a"), nᵃᵖ))
mᵃᵖ = simplify(vectors_in_basis(mᵃ, asymptotic_stereographic_transformed))
display(sympy.Eq(sympy.Dummy(r"m^a"), mᵃᵖ))
lᵃᵖ = simplify(vectors_in_basis(lᵃ, asymptotic_stereographic_transformed))
display(sympy.Eq(sympy.Dummy(r"l^a"), sympy.expand(lᵃᵖ)))
```

Unfortunately, the direct transformation is a little too hard for sympy to invert on its own, so I've expressed the relationship between new and old bases in an awkward way here.  But it's easy enough to invert the relations I derived above, so that I expect the last result to be the same as this:
\begin{align}
  \mathbf{n}
  &=
  k \mathbf{n}'
  \\
  \mathbf{m}
  &=
  \frac{k} {\exp(i\lambda)} \mathbf{m}' + \eth u' \mathbf{n}'
  \\
  \mathbf{l}
  &=
  k \mathbf{l}'
  -\frac{\eth u'\, \bar{\eth} u'} {k} \mathbf{n}'
  +\frac{\bar{\eth} u'} {k} \left( \frac{k} {\exp(i\lambda)} \mathbf{m}' + \eth u' \mathbf{n}' \right)
  +\frac{\eth u'} {k} \left( \frac{k} {\exp(-i\lambda)} \bar{\mathbf{m}}' + \bar{\eth} u' \mathbf{n}' \right)
  \\&=
  k \mathbf{l}'
  +\frac{\eth u'\, \bar{\eth} u'} {k} \mathbf{n}'
  +\frac{\bar{\eth} u'} {\exp(i\lambda)} \mathbf{m}'
  +\frac{\eth u'} {\exp(-i\lambda)} \bar{\mathbf{m}}'
\end{align}
By inspection we see that these are, indeed, equal.

```python

```


# 15 Partial Differential Equations — 2
This notebook is based on `15_PDEs-2.ipynb` and includes additional code to make movies of the relaxation of the solutions. It is included in case students are interested in how to accomplish simple movie making.

## Solving Laplace's or Poisson's equation

Still solving **Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

General solution by iteration:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

#### Jacobi method
Do not change $\Phi_{i,j}$ until a complete sweep has been completed.

```python
def Laplace_Jacobi_slow(Phi):
    # Don't use, very slow AND inefficient
    Phi_new = Phi.copy()
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi_new[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                  + Phi[xi, yj+1] + Phi[xi, yj-1])
    Phi[:, :] = Phi_new
    return Phi
```

Fast implementation using numpy array operations (vectorized, run at C speed, not Python speed) (see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/16_PDEs/16_PDEs-2-LectureNotes.pdf) or [local PDF](16_PDEs-2-LectureNotes.pdf) for an explanation for how to set up the numpy array operations that give you a 100-fold speed up over `Laplace_Jacobi_slow()`):

```python
def Laplace_Jacobi(Phi):
    """One update in the Jacobi algorithm"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2])
    return Phi
```

#### Gauss-Seidel method
Immediately use updated new values for $\Phi_{i-1, j}$ and $\Phi_{i, j-1}$ (if starting from $\Phi_{1, 1}$).

Leads to *accelerated convergence* and therefore *less round-off error* (but distorts symmetry of boundary conditions... hopefully irrelevant when converged but check!)

```python
def Laplace_Gauss_Seidel(Phi):
    """One update in the Gauss-Seidel algorithm"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1])
    return Phi
```

#### Fast Gauss-Seidel-like

Divide the lattice into a checkerboard of black and white cells. Update the odd cells first (like Jacobi) but then use the odd cells to update the even ones (Gauss-Seidel-like; see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/16_PDEs/16_PDEs-2-LectureNotes.pdf) or [local PDF](16_PDEs-2-LectureNotes.pdf) ). This leads to faster convergence *and* can be done with numpy array operations.

```python
def Laplace_Gauss_Seidel_odd_even(Phi):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2] + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2])
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2] + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2])

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2] + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2])
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2] + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2])
    return Phi
```

### *Converged* solution of the wire-in-a-box problem

Solve the box-wire problem and **make sure that the solution is converged to  `tol = 1e-3`.**

Check convergence with the [_Frobenius Norm_](http://mathworld.wolfram.com/FrobeniusNorm.html):

$$
||\mathsf{\Phi}|| := \sqrt{\sum_{i,j}|\Phi_{ij}|^2}
$$

which is implemented as [numpy.linalg.norm()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html) when the argument is a matrix.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

Interactive plotting (with ipympl):

```python
# for interactive work
%matplotlib widget
```

Only execute the next line if you *don't* want interactive plotting (e.g., when exporting to LaTeX/PDF or html):

```python
%matplotlib inline
```

#### Convience plotting functions.
If you provide a filename then output is only written to a file and figures are close to conserve memory. This allows you to plot files in loops and later assemble them into movies using other programs such as ffmpeg, ImageMagick, mencoder, QuickTime 7, ...

```python
def plot_contour(Phi, filename=None, zlabel=r"potential $\Phi$ (V)",
                 cmap=plt.cm.coolwarm):
    """Plot Phi as a contour plot.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    cmap : colormap
          pick one from matplotlib.cm
    """
    fig = plt.figure(figsize=(5,4))
    ax = fig.add_subplot(111)

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]
    cset = ax.contourf(X, Y, Z, 20, cmap=cmap)
    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_aspect(1)

    cb = fig.colorbar(cset, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax

def plot_surf(Phi, filename=None, offset=-20, zlabel=r'potential $\Phi$ (V)',
             elevation=40, azimuth=-65, cmap=plt.cm.coolwarm):
    """Plot Phi as a 3D plot with contour plot underneath.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    offset : float, optional (default: 20)
          position the 2D contour plot by offset along the Z direction
          under the minimum Z value
    zlabel : string, optional
          label for the Z axis and color scale bar
    elevation : float, optional
          choose elevation for initial viewpoint
    azimuth : float, optional
          chooze azimuth angle for initial viewpoint
    cmap : colormap
          pick one from matplotlib.cm
    """

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2, linewidth=0.5, color="gray")
    surf = ax.plot_surface(X, Y, Z, cmap=cmap, alpha=0.6)
    cset = ax.contourf(X, Y, Z, 20, zdir='z', offset=offset+Z.min(), cmap=cmap)

    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_zlabel(zlabel)
    ax.set_zlim(offset + Z.min(), Z.max())

    ax.view_init(elev=elevation, azim=azimuth)

    cb = fig.colorbar(surf, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax
```

#### Wire problem

Use the fastest Poisson solver that we have at the moment: `Laplace_Gauss_Seidel_odd_even()`

```python
Max_iter=30000
tol = 1e-3
Nmax = 100
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("Laplace_Gauss_Seidel_odd_even converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
else:
    print("Laplace_Gauss_Seidel_odd_even did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    Laplace_Gauss_Seidel_odd_even converged in 7558 iterations to 0.0009995267090286367

Plot the result and visualy compare to what we had before:

```python
plot_contour(Phi)
```

```python
plot_surf(Phi)
```

#### Make a movie showing convergence

The following solution can also write out intermediate solution steps as PNG graphics (commented out for speed). Filenames and directory names are hard-coded.(Important: `plt.ioff()` to disable interactive plotting in the notebook frontend, otherwise it is very slow.)

Put all images into a directory `wire`:

```python
import os, shutil

def mk_new_dir(dirname):
    """Create new directory (remove existing one)"""
    if os.path.exists(dirname):
        shutil.rmtree(dirname)
        print("Removed directory {}".format(dirname))
    os.mkdir(dirname)
    print("Created empty directory {}".format(dirname))

mk_new_dir("./wire")
```

    Created empty directory ./wire

Make a movie by saving an image of the potential every 100 steps.

```python
plt.ioff()  # suppress interactive plotting

nsave = 100

Max_iter=30000
tol = 1e-3
Nmax = 100
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("Laplace_Gauss_Seidel_odd_even converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, filename="wire/phi_{0:08d}.png".format(n_iter))
        break
    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, filename="wire/phi_{0:08d}.png".format(n_iter))
else:
    print("Laplace_Gauss_Seidel_odd_even did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

plt.ion()
```

    Laplace_Gauss_Seidel_odd_even converged in 7558 iterations to 0.0009995267090286367

```python
ax = plot_surf(Phi, elevation=40, azimuth=-65)
```

## Successive Over Relaxation (SOR)
Accelerate convergence with the scheme

\begin{align}
r_{i, j} &= \Phi_{i,j}^\text{new} - \Phi_{i, j}^\text{old}\\
\Phi_{i,j}^\text{new} &= \Phi_{i,j}^\text{old} + \omega r_{i,j}
\end{align}

where the new solution is computed with the Gauss-Seidel scheme.

Values of $1 \leq \omega \leq 2$ may work well, $\omega > 2$ can lead to numerical instabilities. Experiment!

### Solve Wire-in-a-box with SOR

Run to convergence (tol = 1e-03). Record the number of steps required (and visually check the solution).
* Start with $\omega = 1$. What should you get?
* Change $\omega$ and try to get faster convergence? Who can get it to converge in the fewest number of steps?

```python
Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
residual = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual  # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    SOR converged in 3797 iterations to 0.0009987117944452034

```python
plot_surf(Phi)
```

### Movie of SOR

Create images in directory `./wire_SOR`

```python
mk_new_dir("./wire_SOR")
mk_new_dir("./wire_SOR_surf")
```

    Created empty directory ./wire_SOR
    Created empty directory ./wire_SOR_surf

```python
# %%timeit
plt.ioff()

nsave = 10

Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
residual = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, "wire_SOR/phi_{0:08d}.png".format(n_iter))
        plot_surf(Phi, "./wire_SOR_surf/phi_{0:08d}.png".format(n_iter),
                  elevation=40, azimuth=-65)
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old

    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, "./wire_SOR/phi_{0:08d}.png".format(n_iter))
        plot_surf(Phi, "./wire_SOR_surf/phi_{0:08d}.png".format(n_iter),
                  elevation=40, azimuth=-65)

else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

plt.ion()
```

    SOR converged in 3797 iterations to 0.0009987117944452034

## Simple example of Poisson's equation: Wire and charge
Add a positive charge and a negative charge in the box.

Now we need to solve **Poisson's equation**.

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

Finite difference solution:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

### Poisson Solver
Modify the Laplace solvers to now solve Poisson's equation:

```python
import numpy as np

def Poisson_Jacobi(Phi, rho, Delta=1.):
    """One update in the Jacobi algorithm for Poisson's equation"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2]) \
                      + np.pi * Delta**2 * rho[1:-1, 1:-1]
    return Phi

def Poisson_Gauss_Seidel(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm for Poisson's equation"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1]) \
                            + np.pi * Delta**2 * rho[xi, yj]
    return Phi

def Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    a = np.pi * Delta**2
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2]
                                + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2]) + a * rho[1:-2:2, 1:-2:2]
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2]
                                + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2]) + a * rho[2:-1:2, 2:-1:2]

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2]
                                + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2]) + a * rho[1:-2:2, 2:-1:2]
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2]
                                + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2]) + a * rho[2:-1:2, 1:-2:2]
    return Phi
```

### Solve wire with dipole

Solve the Poisson problem with SOR (as above), but now employing the Poisson solvers (we are using the fastest one that we have):

```python
Nmax = 100
Max_iter = 10000
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
rho = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V
rho[25:27, 39:41] = 5.0
rho[75:77, 39:41] = -5.0

Delta = 1.0

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=Delta)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    SOR converged in 3823 iterations to 0.0009991605868815044

```python
plot_surf(Phi, elevation=40, azimuth=20)
```

#### Make a movie

The following code can write partial solutions as images. To make it a bit more comfortable, we generate a directory for the images from Python:

```python
import os

def makedir(dirname):
    try:
        os.mkdir(dirname)
    except OSError:
        pass
    return dirname
```

```python
plt.ioff()

nsave = 100

Nmax = 100
Max_iter = 10000
omega = 1.99

dirname = makedir("dipole_wire")
filename = os.path.join(dirname, "phi_{0:08d}.png")

dirname3 = makedir("dipole_wire_3d")
filename3 = os.path.join(dirname3, "phi_{0:08d}.png")

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
rho = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V
rho[25:27, 39:41] = 5.0
rho[75:77, 39:41] = -5.0

Delta = 1.0

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=Delta)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, filename=filename.format(n_iter))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old

    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, filename=filename.format(n_iter))
        plot_surf(Phi, filename=filename3.format(n_iter),
                  elevation=40, azimuth=20)
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

print("Images in {0} and {1}".format(dirname, dirname3))
plt.ion()

```

    SOR converged in 3823 iterations to 0.0009991605868815044
    Images in dipole_wire and dipole_wire_3d

## Charge density
Given a potential $\Phi$, Poisson's equation gives the charge density:
$$
\rho(\mathbf{x}) = -\frac{1}{4\pi} \nabla^2\Phi(\mathbf{x})
$$

Discretized Laplacian
$$
\nabla^2 \Phi \approx \frac{\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\Phi(x,y+\Delta y) +\, \Phi(x,y-\Delta y) - 4\Phi(x, y)}{\Delta^2}
$$

### Laplacian in Python

```python
def laplacian2d(f, Delta=1):
    """Finite difference approximation of Del^2 f.

    Arguments
    ---------
    f : M x N matrix
    Delta : float

    Returns
    -------
    M x N matrix, boundaries set to 0
    """

    L = np.zeros_like(f, dtype=np.float64)
    L[1:-1, 1:-1] = f[1:-1, 2:] + f[1:-1, 0:-2] + f[2:, 1:-1] + f[0:-2, 1:-1] - 4*f[1:-1, 1:-1]
    return L/Delta**2

def laplacian2dsimple(f, Delta=1):
    L = np.zeros_like(f, dtype=np.float64)
    for i in range(1, L.shape[0]-1):
        for j in range(1, L.shape[1]-1):
            L[i, j] = f[i, j+1] + f[i, j-1] + f[i+1, j] + f[i-1, j] - 4*f[i, j]
    return L/Delta**2
```

#### Testing the implementation of Laplacian

Making sure that our fancy numpy-version of the Laplacian is producing the same results as the naive implementation with Python loops:

```python
import numpy as np

def test_laplacian2d():
    ftest = np.random.random((200, 200))
    assert np.allclose(laplacian2d(ftest), laplacian2dsimple(ftest))

test_laplacian2d()
```

### Charge density from potential

Compute the charge density from the converged potential
$$
\rho(x, y) = -\frac{1}{4\pi} \nabla^2 \Phi(x, y)
$$

```python
rhox = - laplacian2d(Phi)/(4*np.pi)
```

Does `rhox` show the charges of $+5$ and $-5$ that we introduced with the charge density $\rho$?

```python
print(rhox.min())
print(rhox.max())
```

    -5.000004186724989
    5.000000000000001

Indeed, the point charges are recovered from the potential.

Plot the charge density:

```python
plot_contour(rhox, zlabel=r"charge density $\rho$")
```

The position of the point charges are found correctly but the charge on the wire appear to be absent. This is expected because the Laplacian is not defined on the boundary. If you wanted to see the charge on the wire you would need to include it inside the integration domain and also assign a thickness (e.g., one or two grid cells).

The 3D plot shows the "point" charges as spikes of the size of one grid cell:

```python
plot_surf(rhox, zlabel=r"charge density $\rho$", elevation=20, azimuth=20, offset=-3)
```

The position of the point charges are found correctly but the charge on the wire appear to be absent. The problem is that the Laplacian is not defined on the boundary.

Also, if the charges are only defined on a single grid cell then the 3D visualization is sometimes not able to show it correctly.

```python

```


Fantasy Basketball Part Two

 Introduction

If you're familiar with fantasy basketball and worked through our **first example** you may have noticed that the lineup we solved for doesn't actually fit the criteria for many popular contests. It was meant as a ground-floor example into mathematical optimization, so make sure to dive into that one first for background on the problem. Additionally, we developed a predictive model to forecast each player's fantasy points, and we'll again use that forecast (without going through the modeling process again).

In this follow-up example, we will extend the optimization model to fulfill the requirements of a typical [DraftKings](https://www.draftkings.com/help/rules/nba) lineup. Specifically, we'll need to change the model to:
- Allow some players to be selected for up to two positions (e.g. LeBron James can fill the PF or SF position)
- Increase the roster from five to eight players
- Ensure the three new roster slots consist of one guard (SG or PG), one forward (SF or PF), and one utility player (any position)

A quick recap of what we saw in the prior example: When selecting a 5-player lineup, there were *a lot* of possibilities; staying under a salary cap wasn't all that easy; and implementing intuitive decision rules can lead to suboptimal lineups -- providing a glimpse into why mathematical optimization should be part of everyone's analytics toolkit.

**The repository for this project can be accessed by following this link:**

 Objective and Prerequisites

As with the first example, the goal here is to build upon our findings of the previous example and select the optimal lineup of players of the National Basketball Association (NBA) that would produce the highest total of fantasy points, which are composed of a player's in-game stats.

As mentioned above, we will make a lineup that meets DraftKings contest rules. There were over 2 million possible lineups when selecting a 5-player team that contained one of each position. Can you figure out the number of possible lineups for this example? Here's a hint: It's a lot. Remember that there are 25 PGs, 23 SFs, 22 SGs, 19 PFs, and 9 Cs. The answer (approximately) will be given a bit later.

Also as mentioned, this example will use the same forecasting results as our fantasy basketball beginner's example, which used historical data from the 2016-2017 and 2017-2018 seasons to predict each player's fantasy points for games on 12/25/2017.

This example assumes you have experience using Python for data manipulation and requires the installation of the following packages:

- **pandas**: for data analysis and manipulation
- **math**: for mathematical manipulations
- **gurobipy**: for utilizing Gurobi to formulate and solve the optimization problem

We'll also explore a few different ways to write summations and constraints in gurobipy, so you can find what works best for you. A quick note: Any output you see similar to <gurobi.Constr \*Awaiting Model Update\*> can be ignored and semi-colons have been added to cells to suppress that output.

 Problem Statement and Solution Approach

By formuating a model that allows some players to be assigned to more than one position and by also expanding the roster, we are making the model much more complex.

Our final lineup needs to include each of the following:
- Point Guard (PG)
- Shooting Guard (SG)
- Shooting Forward (SF)
- Power Forward (PF)
- Center (C)
- Guard (SG,PG)
- Forward (SF,PF)
- Utility (PG, SG, SF, PF, C)

The solution of the problem consists of two components: 1) **fantasy points forecast** and 2) **lineup optimization**.

We'll start by loading a dataset for the eligible players, showing their potential positions, salaries, as well as the forecasts for their upcoming performance. This information will act as an input to our optimization model which will guarantee that we satisfy the rules of the DraftKings contest while maximizing the total fantasy points of our team.

 Fantasy Points Forecast

This section will be short as we are using the output of the predictive model from the first part of this example. We'll see that in this version some players have a *Main Position* and *Alternative Position*, which will let players fill different position slots if eligible. For example, James Harden can fill the PG or SG position, though Russel Westbrook is only available as a PG.

We begin by loading the necessary libraries to solve our problem and importing the data from the predictive model in part one of the problem.

```python
%pip install gurobipy
```

```python
import pandas as pd                                       #importing pandas
import math                                               #importing math
from gurobipy import Model, GRB, quicksum                 #importing Gurobi
```

```python
player_predictions = pd.read_csv('https://raw.githubusercontent.com/yurchisin/modeling-examples/master/fantasy_basketball_1_2/results_target_advanced.csv')     #load processed dataset
player_predictions.sort_values(by='PredictedFantasyPoints',ascending=False).head(20)
```

      Player
      MainPosition
      AlternativePosition
      Team
      Opp
      Salary
      PredictedFantasyPoints
      Points/Salary Ratio

      4
      Joel Embiid
      C
      NaN
      PHI
      NYK
      9500
      51.313689
      5.401441

      0
      James Harden
      PG
      SG
      HOU
      OKC
      11100
      48.809577
      4.397259

      1
      LeBron James
      SF
      PF
      CLE
      GSW
      11000
      48.149718
      4.377247

      2
      Russell Westbrook
      PG
      NaN
      OKC
      HOU
      10900
      44.007224
      4.037360

      3
      Kevin Durant
      SF
      PF
      GSW
      CLE
      10500
      43.438575
      4.137007

      19
      Dario Saric
      PF
      C
      PHI
      NYK
      6200
      40.505486
      6.533143

      5
      Ben Simmons
      PG
      SF
      PHI
      NYK
      9300
      38.692817
      4.160518

      12
      Kyle Kuzma
      SF
      PF
      LAL
      MIN
      7300
      38.201774
      5.233120

      8
      Jimmy Butler
      SG
      NaN
      MIN
      LAL
      8400
      37.873164
      4.508710

      13
      Draymond Green
      PF
      C
      GSW
      CLE
      7200
      37.018949
      5.141521

      15
      Paul George
      SF
      NaN
      OKC
      HOU
      6800
      36.398212
      5.352678

      6
      Karl-Anthony Towns
      C
      NaN
      MIN
      LAL
      8900
      36.188913
      4.066170

      17
      Clint Capela
      C
      NaN
      HOU
      OKC
      6600
      33.490320
      5.074291

      10
      John Wall
      PG
      NaN
      WAS
      BOS
      8000
      33.197563
      4.149695

      38
      Jordan Bell
      PF
      C
      GSW
      CLE
      4900
      33.083296
      6.751693

      9
      Kyrie Irving
      PG
      NaN
      BOS
      WAS
      8300
      31.835863
      3.835646

      22
      Jeff Teague
      PG
      NaN
      MIN
      LAL
      6000
      31.460451
      5.243408

      27
      Steven Adams
      C
      NaN
      OKC
      HOU
      5500
      31.150428
      5.663714

      18
      Klay Thompson
      SG
      NaN
      GSW
      CLE
      6400
      29.283897
      4.575609

      16
      Al Horford
      PF
      C
      BOS
      WAS
      6700
      27.906772
      4.165190

Have you figured out the total number of possible lineups yet? It's around $3.6 \times 10^{11}$, which is a lot.

This is where mathematical optimization and Gurobi are best utilized: To efficiently explore a huge decision space and provide a much needed tool for optimal decision-making.

 Optimal DraftKings Lineup Selection

As we set up our optimization model, we first need to make some definitions. Some of this is the same as before, but as this example is a bit more complicated, we'll need to be a bit more thorough in some definitions.

**Sets and Indices**

$i$ is the index for the set of all players

$j$ is the index for the set of basketball positions (PG,SG,SF,PF,C)

**Input Parameters**

$p_{i}$: the predicted fantasy points of player $i$

$s_{i}$: the salary of player $i$

$S$: our total available salary

```python
players = player_predictions["Player"].tolist()
positions = player_predictions["MainPosition"].unique().tolist()
salaries = player_predictions["Salary"].tolist()
fantasypoints = player_predictions["PredictedFantasyPoints"].tolist()
S = 50000

salary_dict = {players[i]: salaries[i] for i in range(len(players))}
points_dict = {players[i]: fantasypoints[i] for i in range(len(players))}

m = Model()
```

    Restricted license - for non-production use only - expires 2023-10-25

**Decision Variables**

Since it is possible for certain players to fill one of two positions, we need to map each player to their eligible positions. Additionally, we need to add the position index to our decision variable. Instead of a binary variable (i.e. a variable that only takes the values of 0 or 1) $y_i$ we have $y_{i,j}$.

$y_{i,j}$: This variable is equal to 1 if player $i$ is selected at position $j$; and 0 otherwise.

```python
mainposition = list(zip(player_predictions.Player, player_predictions.MainPosition))
alternativeposition = list(zip(player_predictions.Player, player_predictions.AlternativePosition))
indices = mainposition+alternativeposition
player_pos_map = [t for t in indices if not any(isinstance(n, float) and math.isnan(n) for n in t)]

y = m.addVars(player_pos_map, vtype=GRB.BINARY, name="y")
```

**Objective Function**

The objective function of our problem is to maximize the total fantasy points of our lineup, same as the last time but using the differently indexed decision variable.

\begin{align}
 Max \hspace{0.2cm} Z = \sum_{i,j} p_{i} \cdot y_{i,j}
\end{align}

```python
m.setObjective(quicksum([points_dict[i]*y[i,j] for i,j in player_pos_map]), GRB.MAXIMIZE)
```

**Constraints**

Our model still requires each of the primary basketball positions (PG, SG, SF, PF, C) to be filled. Last time we required exactly one of each, so these constraints were equalities (note that this also implied our roster size of five players). In this version of our model, we need *at least one* of each position since it is possible to use, for example, only one power forward (PF).

For each position $j$:
\begin{align}
\sum_{i} y_{i,j} \geq 1
\end{align}

Here is where we have a couple of options in how to add these constraints. The first, intuitively, is to loop through the set of positions. The second uses a slightly different function to add the constraints (*addConstr* -> *addConstrs*). This function incorporates the for loop directly as an argument. **Running both will double up on the constraints in the model, which isn't best practice, so take a look at each approach and comment one out before running the cell.**

```python
# option 1 for writing the above constraints
for j in positions:
    m.addConstr(quicksum([y[i,j] for i, pos in player_pos_map if pos==j])>=1, name = "pos" + j)

# option 2, a slightly more compact way of adding the same set of constraints
m.addConstrs((quicksum([y[i,j] for i, pos in player_pos_map if pos==j])>=1 for j in positions), name = "pos");
```

Now let's work with the additional slots of our new roster: the guard, forward, and utility slots. Here it's worth mentioning that there are different ways to formulate the same problem in mathematical optimization. Some approaches can be better than others and writing efficient models will become an important skill as you tackle larger and more complex problems.

Let's first consider the additional slot for the guard position, which can be filled by a PG or a SG. Considering the overall roster we want to create, we have already guaranteed one PG and one SG (so two total guards). To make sure we get one more additional guard, a constraint needs to added that says the total number of guards needs to be *at least* three.

\begin{align}
\sum_{i} y_{i,j} \geq 3, position\space j \space is\space PG\space or \space SG
\end{align}

The same needs to be done for forwards (SF or PF)
\begin{align}
\sum_{i} y_{i,j} \geq 3, position\space j \space is\space SF\space or \space PF
\end{align}

It is important that we use inequalities here because of the utility position we'll address in a little bit.

```python
m.addConstr(quicksum([y[i,j] for i, j in player_pos_map if (j=='PG' or j=='SG')])>=3)
m.addConstr(quicksum([y[i,j] for i, j in player_pos_map if (j=='SF' or j=='PF')])>=3);
```

Now that we have a position index for our decision variable, we need to ensure each player is assigned to *at most* one position (either their primary or alternative, but not both). To do this we sum across each position $j$ for each player and limit that summation to one.

For each player $i$,
\begin{align}
\sum_{j} y_{i,j} \leq 1
\end{align}

Here we'll use another way to sum over one of the variable's indices by appending *.sum* to the end and replacing the index we want to sum over with "\*". This is useful because each player is not eligible for each position and this syntax automatically sums over the second index.

```python
m.addConstrs((y.sum(i, "*") <= 1 for i in players), name="max_one");
```

A good exercise would be to rewrite other summations using this syntax and to try and the above constraint using quicksum.

So far we have addressed seven of our eight lineup slots. Since the utility slot can have any position, all we need is to require the total number of players selected to be eight by setting the sum over all players and positions to that value.

\begin{align}
\sum_{i,j} y_{i,j} = 8
\end{align}

```python
m.addConstr(quicksum([y[i,j] for i,j in player_pos_map]) == 8, name="full_lineup");
```

Finally, we need to stay under the salary cap $S$, which in a typical DraftKings contest is $\$50,000$:

\begin{align}
\sum_{i,j} s_{i} \cdot y_{i,j} \leq S
\end{align}

```python
cap = m.addConstr(quicksum(salary_dict[i]*y[i,j] for i,j in player_pos_map) <= S, name="salary")
```

In this last constraint, we stored it to be able to easily get information about this constraint after the model runs and the model is updated just as if we didn't store the constraint. Time to find the optimal lineup.

```python
m.optimize()  # optimize our model
```

    Gurobi Optimizer version 9.5.1 build v9.5.1rc2 (mac64[rosetta2])
    Thread count: 8 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 112 rows, 167 columns and 975 nonzeros
    Model fingerprint: 0x9c23fd93
    Variable types: 0 continuous, 167 integer (167 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+04]
      Objective range  [7e+00, 5e+01]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 5e+04]
    Found heuristic solution: objective 206.2597683
    Presolve removed 34 rows and 0 columns
    Presolve time: 0.00s
    Presolved: 78 rows, 167 columns, 779 nonzeros
    Variable types: 0 continuous, 167 integer (167 binary)
    Found heuristic solution: objective 268.7822348

    Root relaxation: objective 2.864073e+02, 71 iterations, 0.00 seconds (0.00 work units)

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0  286.40727    0    2  268.78223  286.40727  6.56%     -    0s
    H    0     0                     283.6262215  286.40727  0.98%     -    0s
         0     0  285.84146    0    7  283.62622  285.84146  0.78%     -    0s
         0     0  285.84146    0    2  283.62622  285.84146  0.78%     -    0s

    Cutting planes:
      Cover: 1
      RLT: 1

    Explored 1 nodes (85 simplex iterations) in 0.03 seconds (0.00 work units)
    Thread count was 8 (of 8 available processors)

    Solution count 4: 283.626 283.626 268.782 206.26

    Optimal solution found (tolerance 1.00e-04)
    Best objective 2.836262214900e+02, best bound 2.836262214900e+02, gap 0.0000%

Notice the output above says we have 167 binary variables, which is much less than $98 \cdot 5 = 490$ variables we'd have if we mapped all players to all basketball positions. Additionally, we would need more constraints to eliminate ineligible player/position combinations. While this wouldn't make much difference in this small example, formulating efficient models is a valuable skill in mathematical optimization.

Let's display our optimal lineup.

```python
player_selections = []
for v in m.getVars():
    if (abs(v.x) > 1e-6):
        player_selections.append(tuple(y)[v.index])

df = pd.DataFrame(player_selections, columns = ['Player','Assigned Position'])
df = df.merge(pd.DataFrame(list(salary_dict.items()), columns=['Player', 'Salary']), left_on=['Player'], right_on=['Player'])
lineup = df.merge(pd.DataFrame(list(points_dict.items()), columns=['Player', 'Predicted Points']), left_on=['Player'], right_on=['Player'])

lineup.sort_values(by=['Assigned Position'])
```

      Player
      Assigned Position
      Salary
      Predicted Points

      0
      Joel Embiid
      C
      9500
      51.313689

      6
      Jordan Bell
      C
      4900
      33.083296

      2
      Draymond Green
      PF
      7200
      37.018949

      3
      Dario Saric
      PF
      6200
      40.505486

      4
      Jeff Teague
      PG
      6000
      31.460451

      5
      Jarrett Jack
      PG
      4600
      27.780012

      1
      Kyle Kuzma
      SF
      7300
      38.201774

      7
      Josh Hart
      SG
      3700
      24.262564

```python
print('Total fantasy score: ', round(m.objVal,2))
print('Remaining salary: ', cap.Slack)
```

    Total fantasy score:  283.63
    Remaining salary:  600.0

The last print statement uses the constraint we stored earlier. The *Slack* attribute of a constraint will show any gap between each side of the inequality. For this application, it's the gap between the salary cap and the amount of salary used in the optimal lineup, which shows there is $600 in unused salary.

 Conclusion

In this notebook we finished up a problem that started from a raw data set containing NBA player box score data and ended in, if our predictive model holds, an optimal fantasy basketball lineup.

Specifically in this part, we:
- Expanded the initial model to reflect the true complexity of creating a fantasy basketball lineup
- Discovered multiple ways to add a set of constraints to a model
- Utilized two summation commands
- Used attributes of a model to get more information about the optimal solution

Overall, this two-part example displayed how, even with the best predictive model, making optimal decisions is still a complicated exercise. Along with machine learning techniques, mathematical optimization is an essential part of a well-developed analytical toolbox.


# Intermediate representation (IR)

$$
\newcommand{\iv}{{\mathrm{i}\nu}}
\newcommand{\wmax}{{\omega_\mathrm{max}}}
\newcommand{\dd}{{\mathrm{d}}}
$$

## One-particle Green's function

We introduce a Green’s function with imaginary arguments in time and frequency.
This has no physical meaning but is merely a mathematical trick and makes calculations easier (to give another example of this: in Minsowski spacetime we take advantage of a similiar substitution)

The so-called imaginary-frequency (Matsubara) Green's functions are defined as followed:

$$
G_{ij}(\tau-\tau') = -\langle T_\tau [c_i(\tau){c}^\dagger_j(\tau')]\rangle,
$$

where $i$ and $j$ denote spin/orbital/band and $T_\tau$ is the time-ordering operator.
Here $\tau$ represents a imaginary time unit $\mathrm{i}t$,
while $c_i$/$c_j$ is a fermionic or bosonic annihilation/creation operator.

The Fourier Transformation of $G_{ij}(\tau)$ (with $\tau \in [0,\beta]$) reads

$$
G_{ij}(\iv_n) = \int_0^{\beta} \dd \tau e^{\iv_n\tau} G_{ij}(\tau),
$$

where $\nu_n = (2n+1)\pi/\beta$ (fermion) and $\nu_n = 2n\pi/\beta$ (boson) with $n$ being an integer.
The inverse temperature is denoted by $\beta$ (We take $\hbar=1$).
The inverse transformation is given by

$$
G_{ij}(\tau) = \frac{1}{\beta}\sum_{n=-\infty}^\infty e^{-\iv_n\tau}G_{ij}(\iv_n).
$$

Continuing $G_{ij}(\iv_n)$ to a holomorphic function in the upper half of the complex plane,
the imaginary-frequency (Matsubara) Green's function can be related to the "orginary" retarded Green's function as

$$
G_{ij}^\mathrm{R}(\omega)=G_{ij}(z \rightarrow \omega+\mathrm{i}0^{+}).
$$

In the following, we omit the symbols $i$, $j$, $n$ unless there is confusion.

## Lehmann representation

In the imaginary-frequency domain, the Lehmann representation reads

$$
\begin{align}
    G(\iv) &= \int_{-\infty}^\infty \dd\omega \underbrace{\frac{1}{\iv - \omega}}_{\equiv K(\iv, \omega)} A(\omega),
\end{align}
$$

where $A(\omega)$ is a spectral function.
$K(\iv, \omega)$ is the so-called analytic continuation kernel.
The Lehmann representation can be transformed to the imaginary-time domain as

$$
\begin{align}
    G(\tau) &= - \int_{-\infty}^\infty \dd\omega K(\tau, \omega) A(\omega),
\end{align}
$$ (lehmann-tau)

where $0 < \tau < \beta$ and

$$
\begin{align}
    K(\tau, \omega) &\equiv - \frac{1}{\beta} \sum_{\iv} e^{-\iv \tau} K(\iv, \omega) =
    \begin{cases}
        \frac{e^{-\tau\omega}}{1+e^{-\beta\omega}} & (\mathrm{fermion}),\\
        \frac{e^{-\tau\omega}}{1-e^{-\beta\omega}} & (\mathrm{boson})
    \end{cases}.
\end{align}
$$

The minus sign originates from our the convention $K(\tau, \omega) > 0$.
To avoid the divergence of the bosonic kernel at $\omega=0$, we reformulate Eq. {eq}`lehmann-tau` as

$$
\begin{equation}
    G(\tau)= - \int_{-\infty}^\infty\dd{\omega} K^\mathrm{L}(\tau,\omega) \rho(\omega),
\end{equation}
$$

where $K^\mathrm{L}(\tau, \omega)$ is the "logistic kernel" defined as

$$
K^\mathrm{L}(\tau, \omega) =  \frac{e^{-\tau\omega}}{1+e^{-\beta\omega}},
$$ (KL)

and $\rho(\omega)$ is the modified spectral function

$$
\begin{align}
    \rho(\omega) &\equiv
    \begin{cases}
        A(\omega) & (\mathrm{fermion}),\\
        \frac{A(\omega)}{\tanh(\beta \omega/2)} & (\mathrm{boson}).
    \end{cases}
\end{align}
$$

## Singular value expansion and basis functions

The singular value expnasion of the kernel {eq}`KL` reads {cite:p}`Shinaoka:2017ix`

$$
K^\mathrm{L}(\tau, \omega) = \sum_{l=0}^\infty U_l(\tau) S_l V_l(\omega),
$$

for $\omega \in [-\wmax, \wmax]$ with $\wmax$ (>0) being a cut-off frequency.
$U_l(\tau)$ and $V_l(\omega)$ are left and right singular functions and $S_l$ as the singular values (with $S_0>S_1>S_2>...>0$).
The two sets of singular functions $U$ and $V$ make up the basis functions of the so-called Intermediate Representation (IR), which depends on $\beta$ and the cutoff $\wmax$.
For the peculiar choice of the regularization for the bosonic kernel using $K^\mathrm{L}$, these basis functions do not depend on statistical properties.
The basis functions $U_l(\tau)$ are transformed to the imaginary-frequency axis as

$$
U_l(\iv) \equiv \int_0^\beta \dd \tau e^{\iv \tau} U_l(\tau).
$$

Some of the information regarding real-frequency properties of the system is often lost during transition into the imaginary-time domain, so that the imaginary-frequency Green's function does hold less information than the real-frequency Green's function. The reason for using IR lies within its compactness and ability to display that information in imaginary quantities.

The decay of the singular values depends on $\beta$ and $\wmax$ only through the dimensionless parameter $\Lambda\equiv \beta\wmax$.
The following plots show the singular values and the basis functions computed for $\beta=10$ and $\wmax=10$.

```
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import sparse_ir

plt.rcParams.update({
    "font.family": "serif",
    "font.size": 16,
})

lambda_ = 100
beta = 10
wmax = lambda_/beta
basis = sparse_ir.FiniteTempBasis('F', beta, wmax, eps=1e-10)

plt.semilogy(basis.s, marker='o', ls='')
plt.xlabel(r'$l$')
plt.ylabel(r'$S_l$')
plt.tight_layout()
plt.show()
```

```
fig = plt.figure(figsize=(6,12))
ax1 = plt.subplot(311)
ax2 = plt.subplot(312)
ax3 = plt.subplot(313)
axes = [ax1, ax2, ax3]

taus = np.linspace(0, beta, 1000)
omegas = np.linspace(-wmax, wmax, 1000)

beta = 10
nmax = 100

v = 2*np.arange(nmax)+1
iv = 1J * (2*np.arange(nmax)+1) * np.pi/beta

uhat_val = basis.uhat(v)
for l in range(4):
    ax1.plot(taus, basis.u[l](taus), label=f'$l={l}$')
    ax2.plot(omegas, basis.v[l](omegas), label=f'$l={l}$')
    y = uhat_val[l,:].imag if l%2 == 0 else uhat_val[l,:].real
    ax3.plot(iv.imag, y, label=f'$l={l}$')

ax1.set_xlabel(r'$\tau$')
ax2.set_xlabel(r'$\omega$')
ax1.set_ylabel(r'$U_l(\tau)$')
ax2.set_ylabel(r'$V_l(\omega)$')
ax1.set_xlim([0,beta])
ax2.set_xlim([-wmax, wmax])

ax3.plot(iv.imag, np.zeros_like(iv.imag), ls='-', lw=0.5, color='k')
ax3.set_xlabel(r'$\nu$')
ax3.set_ylabel(r'$\hat{U}_l(\mathrm{i}\nu)$')
ax3.set_xlim([0, iv.imag.max()])

for ax in axes:
    ax.legend(loc='best', frameon=False)

plt.tight_layout()
```

We expand $G(\tau)$ and $\rho(\omega)$ as

$$
G(\tau) = \sum_{l=0}^{L-1} G_l U_l(\tau) + \epsilon_L,
$$

$$
\hat{G}(\mathrm{i}\nu) = \sum_{l=0}^{L-1} G_l \hat{U}_l(\mathrm{i}\nu) + \hat{\epsilon}_L,
$$

where $\epsilon_L,~\hat{\epsilon}_L \approx S_L$. The expansion coefficients $G_l$ can be determined from the spectral function as

$$
G_l = -S_l \rho_l,
$$

where

$$
\rho_l = \int_{-\omega_\mathrm{max}}^{\omega_\mathrm{max}} \mathrm{d} \omega \rho(\omega) V_l(\omega).
$$

As a simple example, we consider a particle-hole-symmetric simple spectral function
$\rho(\omega) = \frac{1}{2} (\delta(\omega-1) + \delta(\omega+1))$ for fermions.
The expansion coefficients are given by

$$
\rho_l = \frac{1}{2}(V_l(1) + V_l(-1)).
$$

```
rho_l = 0.5 * (basis.v(1) + basis.v(-1))
gl = - basis.s * rho_l

ls = np.arange(basis.size)
plt.semilogy(ls[::2], np.abs(rho_l[::2]), label=r"$|\rho_l|$", marker="o", ls="")
plt.semilogy(ls[::2], np.abs(gl[::2]), label=r"$|G_l|$", marker="x", ls="")
plt.semilogy(ls, basis.s, label=r"$S_l$", marker="", ls="--")
plt.xlabel(r"$l$")
plt.legend(frameon=False)
plt.show()
#plt.savefig("plot.pdf")
```

## Remark: What happens if $\omega_\mathrm{max}$ is too small?

If you expand $G(\tau)$ using $U_l(\tau)$ for too small $\omega_\mathrm{max}$,
$|G_l|$ decays slower than $S_l$.
Let us demonstrate it with $\omega_\mathrm{max} = 0.5$.

```
def gtau_exact(tau):
    return -0.5*(np.exp(-tau)/(1 + np.exp(-beta)) + np.exp(tau)/(1 + np.exp(beta)))

plt.plot(taus, gtau_exact(taus))
plt.xlabel(r"$\tau$")
plt.ylabel(r"$G(\tau)$")
plt.show()
#plt.savefig("gtau.pdf")
```

```
from scipy.integrate import quad
from matplotlib.ticker import MaxNLocator

for wmax_bad in [10, 1, 0.5, 0.2]:
    basis_bad = sparse_ir.FiniteTempBasis("F", beta, wmax_bad, eps=1e-10)

    # We expand G(τ).
    gl_bad = [quad(lambda x: gtau_exact(x) * basis_bad.u[l](x), 0, beta)[0] for l in range(basis_bad.size)]

    ax = plt.figure().gca()
    ax.xaxis.set_major_locator(MaxNLocator(integer=True)) # Force xtics to be integer
    plt.semilogy(np.abs(gl_bad), marker="s", ls="", label=r"$|g_l|$")
    plt.semilogy(np.abs(basis_bad.s), marker="x", ls="--", label=r"$S_l$")
    plt.title(r"$\omega_\mathrm{max}$ = " + str(wmax_bad))
    plt.xlabel(r"$l$")
    plt.xlim([0, 30])
    plt.ylim([1e-5, 10])
    plt.legend(frameon=False)
    plt.show()
    #plt.savefig(f"coeff_bad_{wmax_bad}.pdf")
```

```

```


# Simple GP Regression

Gaussian processes are a powerful tool for nonlinear regression models.

Assume that we have **predictor** variables $\mathbf{X} = \{\mathbf{x_i}\}_{i=1}^N \in \mathbb{R}^d$ and **response** variables $\mathbf{y}=\{y_i \in \mathbb{R}\}_{i=1}^N$.

The response variables $\mathbf{y}$ are assumed to dependent on the predictors $\mathbf{X}$,

\begin{equation}
y_i \sim \mathcal{N}(f(\mathbf{x}_i),\sigma^2), \ i=1,\ldots,n,
\end{equation}

where $f$ is a mapping function. Treating $f$ as a random function, we assume that the distribution over $f$ is a **Gaussian process**,
$$
f \sim \mathcal{GP}(m(\mathbf{x}),k(\mathbf{x},\mathbf{x}')),
$$
where $m(\cdot)$ and $k(\cdot,\cdot)$ are the mean and kernel functions respectively.

## 1D regression example

We start by simulating some data

```julia
using GaussianProcesses
using Random

Random.seed!(20140430)
# Training data
n=10;                          #number of training points
x = 2π * rand(n);              #predictors
y = sin.(x) + 0.05*randn(n);   #regressors
```

The first step in modelling with Gaussian Processes is to choose mean functions and kernels which describe the process.

**Note** that all hyperparameters for the **mean** and **kernel** functions and $\sigma$ are given on the log scale. This is true for all strictly positive hyperparameters. Gaussian Processes are represented by objects of type 'GP' and constructed from observation data, a mean function and kernel, and optionally the amount of observation noise.

```julia
#Select mean and covariance function
mZero = MeanZero()                   #Zero mean function
kern = SE(0.0,0.0)                   #Squared exponential kernel (note that hyperparameters are on the log scale)

logObsNoise = -1.0                        # log standard deviation of observation noise (this is optional)
gp = GP(x,y,mZero,kern,logObsNoise)       #Fit the GP
```

    GP Exact object:
      Dim = 1
      Number of observations = 10
      Mean function:
        Type: MeanZero, Params: Float64[]
      Kernel:
        Type: SEIso{Float64}, Params: [0.0, 0.0]
      Input observations =
    [4.854610892030431 5.176527683588912 … 1.9941157865846477 3.4567590405289352]
      Output observations = [-0.9672931901680022, -1.0070469610508248, -1.0903962829715461, 0.8811208605233783, -0.3332131916909575, -0.9769651482106089, 0.9159341732111771, 0.7362180710829012, 0.9508490881368652, -0.3064317818816142]
      Variance of observation noise = 0.1353352832366127
      Marginal Log-Likelihood = -6.335

Once we've fit the `GP` function to the data, we can calculate the **predicted mean** and **variance** of the function at unobserved points $\{\mathbf{x}^\ast,y^\ast\}$, conditional on the observed data $\mathcal{D}=\{\mathbf{y},\mathbf{X}\}$. This is done with the `predict_y` function.

The `predict_y` function returns the mean vector $\mu(\mathbf{x}^\ast)$ and covariance matrix (variance vector if `full_cov=false`) $\Sigma(\mathbf{x}^\ast,\mathbf{x}^{\ast^\top})$ of the predictive distribution,
\begin{equation}
    y^\ast|\mathbf{x}^\ast,\mathcal{D} \sim \mathcal{N}(\mu(\mathbf{x}^\ast),\Sigma(\mathbf{x}^\ast,\mathbf{x}^{\ast^\top})+\sigma^2\mathbf{I}),
\end{equation}
where
\begin{eqnarray}
\mu(\mathbf{x}^\ast) &=& k(\mathbf{x}^\ast,\mathbf{X})(k(\mathbf{X}, \mathbf{X}) + \sigma_n^2 \mathbf{I})^{-1}\mathbf{y} \\ \ \mbox{and} \
\Sigma(\mathbf{x}^\ast,\mathbf{x}^{\ast}) &=& k(\mathbf{x}^\ast,\mathbf{x}^\ast) -k(\mathbf{x}^\ast,\mathbf{X})(k(\mathbf{X}, \mathbf{X})+ \sigma_n^2 \mathbf{I})^{-1} k(\mathbf{X},\mathbf{x}^\ast).
\end{eqnarray}

**Note** you can use the `predict_f` function to predict the latent function $\mathbf{f}^\ast$.

```julia
μ, σ² = predict_y(gp,range(0,stop=2π,length=100));
```

Plotting GPs is straightforward and utilises the recipes approach to plotting from the [Plots.jl](https://github.com/JuliaPlots/Plots.jl) package. More information about plotting GPs and the available functionality can be found in this [notebook](https://github.com/STOR-i/GaussianProcesses.jl/blob/master/notebooks/Plotting%20GPs.ipynb).

The default plot function `plot(gp)` outputs the predicted mean and variance of the function (i.e. uses `predict_f` in the background), with the uncertainty in the function represented by a confidence ribbon (set to 95% by default). All optional plotting arguments are given after `;`.

```julia
using Plots  #Load Plots.jl package

plot(gp; xlabel="x", ylabel="y", title="Gaussian process", legend=false, fmt=:png)      # Plot the GP
```

The hyperparameters are optimized using the [Optim.jl](https://github.com/JuliaOpt/Optim.jl) package. This offers users a range of optimization algorithms which can be applied to estimate the hyperparameters using type II maximum likelihood estimation. Gradients are available for all mean and kernel functions used in the package and therefore it is recommended that the user utilizes gradient based optimization techniques. As a default, the `optimize!` function uses the `L-BFGS` solver, however, alternative solvers can be applied.

```julia
using Optim
optimize!(gp; method=ConjugateGradient())   # Optimise the hyperparameters
```

     * Status: failure (objective increased between iterations) (line search failed)

     * Candidate solution
        Minimizer: [-2.99e+00, 4.64e-01, -2.29e-01]
        Minimum:   -3.275745e+00

     * Found with
        Algorithm:     Conjugate Gradient
        Initial Point: [-1.00e+00, 0.00e+00, 0.00e+00]

     * Convergence measures
        |x - x'|               = 4.62e-09 ≰ 0.0e+00
        |x - x'|/|x'|          = 1.54e-09 ≰ 0.0e+00
        |f(x) - f(x')|         = 1.35e-13 ≰ 0.0e+00
        |f(x) - f(x')|/|f(x')| = 4.12e-14 ≰ 0.0e+00
        |g(x)|                 = 5.69e-08 ≰ 1.0e-08

     * Work counters
        Seconds run:   0  (vs limit Inf)
        Iterations:    27
        f(x) calls:    66
        ∇f(x) calls:   41

```julia
plot(gp; legend=false, fmt=:png)   #Plot the GP after the hyperparameters have been optimised
```

By default all hyperparameters or optimized. But the function `optimize!` allows also to force hyperparameters to remain constant or optimize them in a box.

```julia
optimize!(gp; kern = false)   # Don't optimize kernel hyperparameters
optimize!(gp; kernbounds = [[-1, -1], [1, 1]]) # Optimize the kernel parameters in a box with lower bounds [-1, -1] and upper bounds [1, 1]
```

     * Status: success

     * Candidate solution
        Minimizer: [-2.99e+00, 4.64e-01, -2.29e-01]
        Minimum:   -3.275745e+00

     * Found with
        Algorithm:     Fminbox with L-BFGS
        Initial Point: [-2.99e+00, 4.64e-01, -2.29e-01]

     * Convergence measures
        |x - x'|               = 0.00e+00 ≤ 0.0e+00
        |x - x'|/|x'|          = 0.00e+00 ≤ 0.0e+00
        |f(x) - f(x')|         = 0.00e+00 ≤ 0.0e+00
        |f(x) - f(x')|/|f(x')| = 0.00e+00 ≤ 0.0e+00
        |g(x)|                 = 6.32e-08 ≰ 1.0e-08

     * Work counters
        Seconds run:   1  (vs limit Inf)
        Iterations:    1
        f(x) calls:    3
        ∇f(x) calls:   3

MCMC, specifically a Hamiltonian Monte Carlo in this instance, can be run on the GPE hyperparameters through the `mcmc` function. Priors for hyperparameters of the mean and kernel parameters can be set through the `set_priors!` function. The log noise parameter of the GPE is a Uniform(0,1) distribution and currently can't be changed.

```julia
using Distributions

set_priors!(kern, [Normal(), Normal()]) # Uniform(0,1) distribution assumed by default if priors not specified
@time chain = mcmc(gp, nIter=5000);
```

    Number of iterations = 5000, Thinning = 1, Burn-in = 1
    Step size = 0.100000, Average number of leapfrog steps = 10.131000
    Number of function calls: 50656
    Acceptance rate: 0.975200
      2.699465 seconds (3.62 M allocations: 218.559 MiB, 2.83% gc time)

There is additional support for inference to be done in a fully Bayesian fashion through the use of an [elliptical slice sampler](http://proceedings.mlr.press/v9/murray10a/murray10a.pdf). While a HMC sampler can often be shown to be highly efficient, the sampler's efficiency can be highly dependent upon good initial choice of the sampler's hyperparameters. Conversely, ESS has no free parameters and is designed to be highly efficient in tightly correlated Gaussian posteriors: a geometry commonly found in Gaussian process models. Currently, inference via an ESS is only supported when the likelihood is Gaussian.

For advice on how to fit Gaussian processes with non-Gaussian data, see our notebook on [Poisson regression](https://github.com/STOR-i/GaussianProcesses.jl/blob/master/notebooks/Poisson%20regression.ipynb) or [classification](https://github.com/STOR-i/GaussianProcesses.jl/blob/master/notebooks/Classification.ipynb).

```julia
set_priors!(gp.logNoise, [Normal()])
@time ess_chain = ess(gp, nIter=5000);
```

    Number of iterations = 5000, Thinning = 1, Burn-in = 1
    Number of function calls: 28804
    Acceptance rate: 0.210049
      0.353048 seconds (1.59 M allocations: 95.275 MiB, 12.60% gc time)

```julia
p1 = plot(chain', label=["Noise" "SE log length" "SE log scale"]; fmt=:png, size = (1200, 600), title="Hamiltonian Monte-Carlo", ylims=(-4, 2.5))
p2 = plot(ess_chain', label=["Noise" "SE log length" "SE log scale"]; fmt=:png, size = (1200, 600), title="Elliptical Slice Sampler", ylims=(-4, 2.5))
plot(p1, p2, xlabel="Iteration", ylabel="Parameter Value")
```

As can be seen in the timing outputs, Hamiltonian MC is considerably slower than elliptical slice sampling, however, this increase in computational cost is accompanied by chains with more desirable mixing.

## Multi-dimensional regression

The regression example above can be easily extended to higher dimensions. For the purpose of visualisation, and without loss of generality, we consider a 2 dimensional regression example.

```julia
#Training data
d, n = 2, 50;         #Dimension and number of observations
x = 2π * rand(d, n);                               #Predictors
y = vec(sin.(x[1,:]).*sin.(x[2,:])) + 0.05*rand(n);  #Responses
```

For problems of dimension>1 we can use isotropic (`Iso`) kernels or automatic relevance determination (`ARD`) kernels. For `Iso` kernels, the length scale parameter $\ell$ is the same for all dimensions. For `ARD` kernels, each dimension has different length scale parameter.

The `Iso` and `ARD` kernels are implemented automatically by replacing the single length scale parameter with a vector of parameters. For example, below we use the Matern 5/2 `ARD` kernel, if we wanted to use the `Iso` alternative then we would set the kernel as `kern=Mat(5/2,0.0,0.0)`.

In this example we use a composite kernel represented as the sum of a Matern 5/2 ARD kernel and a Squared Exponential isotropic kernel. This is easily implemented using the `+` symbol, or in the case of a product kernel, using the `*` symbol (i.e. `kern = Mat(5/2,[0.0,0.0],0.0) * SE(0.0,0.0)`).

```julia
mZero = MeanZero()                             # Zero mean function
kern = Matern(5/2,[0.0,0.0],0.0) + SE(0.0,0.0)    # Sum kernel with Matern 5/2 ARD kernel
                                               # with parameters [log(ℓ₁), log(ℓ₂)] = [0,0] and log(σ) = 0
                                               # and Squared Exponential Iso kernel with
                                               # parameters log(ℓ) = 0 and log(σ) = 0
```

    Type: SumKernel{Mat52Ard{Float64},SEIso{Float64}}
      Type: Mat52Ard{Float64}, Params: [-0.0, -0.0, 0.0]  Type: SEIso{Float64}, Params: [0.0, 0.0]

Fit the Gaussian process to the data using the prespecfied mean and covariance functions.

```julia
gp = GP(x,y,mZero,kern,-2.0)          # Fit the GP
```

    GP Exact object:
      Dim = 2
      Number of observations = 50
      Mean function:
        Type: MeanZero, Params: Float64[]
      Kernel:
        Type: SumKernel{Mat52Ard{Float64},SEIso{Float64}}
          Type: Mat52Ard{Float64}, Params: [-0.0, -0.0, 0.0]      Type: SEIso{Float64}, Params: [0.0, 0.0]
      Input observations =
    [3.05977 4.74752 … 2.82127 5.38224; 2.02102 4.27258 … 6.13114 1.56497]
      Output observations = [0.08509, 0.924505, 0.275745, -0.448035, -0.784758, -0.316803, -0.823483, -0.886726, 0.0059149, 0.414951  …  -0.413905, -0.347505, 0.46108, -0.204102, -0.538689, 0.554203, -0.874479, -0.0506017, -0.0215167, -0.745944]
      Variance of observation noise = 0.01831563888873418
      Marginal Log-Likelihood = -29.547

Using the [Optim](https://github.com/JuliaOpt/Optim.jl) package we have the option to choose from a range of optimize functions including conjugate gradients. It is also possible to fix the hyperparameters in either the mean function, kernel function or observation noise, by settting them to false in `optimize!` (e.g. `optimize!(...,domean=false)`).

```julia
optimize!(gp)                         # Optimize the hyperparameters
```

    Results of Optimization Algorithm
     * Algorithm: L-BFGS
     * Starting Point: [-2.0,-0.0,-0.0,0.0,0.0,0.0]
     * Minimizer: [-4.277211995773057,5.536664077544042, ...]
     * Minimum: -5.023830e+01
     * Iterations: 34
     * Convergence: false
       * |x - x'| ≤ 0.0e+00: false
         |x - x'| = 4.74e-02
       * |f(x) - f(x')| ≤ 0.0e+00 |f(x)|: false
         |f(x) - f(x')| = -1.36e-14 |f(x)|
       * |g(x)| ≤ 1.0e-08: false
         |g(x)| = 1.87e-05
       * Stopped by an increasing objective: true
       * Reached Maximum Number of Iterations: false
     * Objective Calls: 115
     * Gradient Calls: 115

A range of plotting options are availbe through the [Plots.jl](https://github.com/JuliaPlots/Plots.jl) package.

```julia
plot(contour(gp) ,heatmap(gp); fmt=:png)
```