# Polarization
Following [Detlefs[2012]](http://dx.doi.org/10.1140/epjst/e2012-01630-3) the electric field of a monochromatic plane wave can be described as

$$
\begin{equation}
\begin{split}
\vec{E}(t,\vec{x}) =& \Re[(V_0\hat{e}_0 + V_1\hat{e}_1)e^{i(\vec{k}\cdot\vec{x} - \omega t)}]\\
\hat{e}_0 =& \hat{e}_1\times\hat{e}_2\\
\hat{e}_1 =& \hat{e}_2\times\hat{e}_0\\
\hat{e}_2 =&\hat{k}
\end{split}
\end{equation}
$$

The complex random vector $\vec{V}=[V_0,V_1]^T$ is known as the Jones vector. The state of polarization of this field refers to its second order statistics, which can be expressed as the expectation of the outer product of the Jones vector with itself ($x^*$ denotes the complex conjugate of $x$ and $\langle x\rangle$ the expectation)

$$
\begin{equation}
C = \langle\vec{V}\otimes\vec{V}\rangle=
\begin{bmatrix} \langle V_0 V_0^*\rangle & \langle V_0 V_1^*\rangle\\
\langle V_1 V_0^*\rangle & \langle V_1 V_1^*\rangle
\end{bmatrix}
\end{equation}
$$

Under Stokes' formalism the coherency matrix $C$ is decomposed as follows

$$
\begin{equation}
C = \frac{1}{2}\begin{bmatrix} S_0+S_1 & S_2-iS_3\\
S_2+iS_3 & S_0-S_1
\end{bmatrix}
\end{equation}
$$

The Poynting vector can be written as function of the Jones vector:

$$
\begin{equation}
\begin{split}
\vec{P}(t,\vec{x})=& \frac{1}{\mu_0}\vec{E}(t,\vec{x})\times\vec{B}(t,\vec{x})\\
=& \frac{1}{c\mu_0}\left(\vec{E}(t,\vec{x})\cdot\vec{E}(t,\vec{x})\right)\hat{k}\\
=& \frac{c\epsilon_0}{2}\left( \Re[\vec{V}\cdot\vec{V}^*] + \Re[\vec{V}\cdot\vec{V}e^{2i(\vec{k}\cdot\vec{x} - \omega t)} ]  \right)\hat{k}\\
\Re[\vec{V}\cdot\vec{V}^*]=&\vec{V}\cdot\vec{V}^*=\left\lvert V_0\right\rvert^2 +\left\lvert V_1\right\rvert^2\\
\Re[\vec{V}\cdot\vec{V}e^{2i(\vec{k}\cdot\vec{x} - \omega t)}]=&
\left\lvert V_0\right\rvert^2 \cos(2(\phi_0+\alpha))
+\left\lvert V_1\right\rvert^2 \cos(2(\phi_1+\alpha))\\
\alpha=&\vec{k}\cdot\vec{x} - \omega t\\
\end{split}
\end{equation}
$$

where we used $\cos 2x = 2\cos^2 x -1$ and SI units $V/m$ for $\left\lvert V_j\right\rvert$.

For an isolated wave $\vec{V}\in\mathbb{C}^2$, the intensity (SI units: $W/m^2$) can be obtained by time averaging the Poynting vector (second term vanishes):

$$
\begin{equation}
\begin{split}
I(\vec{x})=&{\left\langle \left\lVert \vec{P}(t,\vec{x}) \right\rVert \right\rangle}_{t}\\
=&\frac{c\epsilon_0}{2}\vec{V}\cdot\vec{V}^*\quad\quad {\langle \cos(2(\phi_j+\alpha)) \rangle}_{t}=0\\
=&\frac{c\epsilon_0}{2}\mathrm{Tr}(C)\\
=&\frac{c\epsilon_0}{2}S_0
\end{split}
\end{equation}
$$

For an ensemble of waves (with the same angular frequency $\omega$) the Jones vector is a random vector and the intensity is the expectation value of the Poynting vector norm instead of the time average

$$
\begin{equation}
\begin{split}
I(\vec{x})=&{\left\langle \left\lVert \vec{P}(t,\vec{x}) \right\rVert \right\rangle}\\
=&\frac{c\epsilon_0}{2}\langle\vec{V}\cdot\vec{V}^*\rangle\quad\quad \langle \left\lvert V_j\right\rvert^2\cos(2(\phi_j+\alpha))\rangle = 0\\
=&\frac{c\epsilon_0}{2}\mathrm{Tr}(C)\\
=&\frac{c\epsilon_0}{2}S_0
\end{split}
\end{equation}
$$

where we assume that $\left\lvert V_j\right\rvert$ and $\phi_j$ are independent random variables and $\phi_j$ uniform, in which case the ensemble average is independent of time.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

from spectrocrunch.sources import polarization
```

## Fully polarized
Plot the electric field vector in a plane perpendicular to the propagation direction:

```python
# Define incident intensity and polarization
intensity = 1 # W/m^2
dolp = 0.4 # degree of linear polarization (in [0,1])
polangle = 40 # angle of semi-major axis with respect to the horizontal direction (in [-90,90])
handedness = "left"
phase0 = 0 # phase of V0 in degrees

# Describe with Jones' formalism because fully polarized
J1 = polarization.Jones.from_params(intensity=intensity,dolp=dolp,polangle=polangle,\
                                    handedness=handedness,phase0=phase0)

# We're looking upstream:
J1.plot_efield()

plt.show()
```

## Partially polarized
Plot the electric field vector in a plane perpendicular to the propagation direction:

```python
# Define incident intensity and polarization
intensity = 1 # W/m^2
dop = 0.9 # degree of polarization (in [0,1])
dolp = 0.5*dop # degree of linear polarization (in [0,dop])
polangle = -30 # angle of polarization ellipse with respect to the horizontal direction (in [-90,90])
handedness = "right"

# Describe with Stokes' formalism because not fully polarized
S1 = polarization.Stokes.from_params(intensity=intensity,dop=dop,dolp=dolp,\
                                    polangle=polangle,handedness=handedness)

# Plot the polarized component:
print("Intensity: {} (pol, W/m²), {} (unpol, W/m²)".format(S1.intensity_polarized,S1.intensity_unpolarized))
S1.decompose()["pol"].plot_efield()
plt.show()
```

## Thomson scattering

The scattering direction defined by the wave vector $\vec{k}_\mathrm{sc}$ in the coordinate system of the primary beam $\vec{k}$:

$$
\begin{equation}
\begin{split}
\hat{k}_\mathrm{sc}\cdot\hat{e}_0 =& \cos\phi\sin\theta\\
\hat{k}_\mathrm{sc}\cdot\hat{e}_1 =& \sin\phi\sin\theta\\
\hat{k}_\mathrm{sc}\cdot\hat{e}_2 =& \cos\theta\\
\end{split}
\end{equation}
$$

If we define the reference frame of the scattered beam as
$$
\begin{equation}
\begin{split}
\hat{e}_2^\prime =& \hat{k}_\mathrm{sc}\\
(\hat{k}_\mathrm{sc}\times\hat{k})\cdot\hat{e}_0^\prime=1\\
\hat{e}_1^\prime =&\hat{e}_2^\prime\times\hat{e}_0^\prime
\end{split}
\end{equation}
$$

which means that $\hat{e}_0^\prime$ is prependicular to the scattering plane defined by $\hat{k}_\mathrm{sc}$ and $\hat{k}$, then the Stokes parameters after Thomson scattering can be expressed in this reference frame as

$$
\begin{equation}
\begin{split}
S_{sc}=&M\cdot S\\
M =& \begin{bmatrix}
a&(1-a)\cos2\beta&-(1-a)\sin2\beta&0\\
(1-a)&a\cos2\beta&-a\sin2\beta&0\\
0&\cos\theta\sin2\beta&\cos\theta\cos2\beta&0\\
0&0&0&\cos\theta
\end{bmatrix}\\
\beta=&\frac{\pi}{2}-\phi=\arccos(\hat{e}_0\cdot\hat{e}_0^\prime)\\
a=&\frac{1+\cos^2\theta}{2}
\end{split}
\end{equation}
$$

The scattered intensity can be written as

$$
\begin{equation}
\begin{split}
I_{sc}=&I_0K(\phi,\theta)\\
K(\phi,\theta)=&a-(1-a)\left(\frac{S_1}{S_0}\cos2\phi+\frac{S_2}{S_0}\sin2\phi\right)
\end{split}
\end{equation}
$$

Two common examples are $S_1=S_2=S_3=0$ (unpolarized) and $S_1=S_0$, $S_2=S_3=0$ (linear polarized in the horizontal direction)

$$
\begin{equation}
\begin{split}
K_{un}=&\frac{1+\cos^2\theta}{2}\\
K_{linh}=&1-\sin^2\theta\cos^2\phi
\end{split}
\end{equation}
$$

```python
# Define incident intensity and polarization
intensity = 1 # W/m^2
dop = 0.4 # degree of polarization (in [0,1])
dolp = 0.8*dop # degree of linear polarization (in [0,dop])
polangle = 0 # angle of polarization ellipse with respect to the horizontal direction (in [-90,90])
handedness = "left"

# Fully polarized
J1 = polarization.Jones.from_params(intensity=intensity,dolp=dolp/dop,\
                                    polangle=polangle,handedness=handedness)

# Partially polarized
S1 = polarization.Stokes.from_params(intensity=intensity,dop=dop,dolp=dolp,\
                                    polangle=polangle,handedness=handedness)

# Scattering direction: spherical coordinates
azimuth = 20
polar = 80 # scattering angle

J2 = J1.thomson_scattering(azimuth,polar)
S2 = S1.thomson_scattering(azimuth,polar)

print("Incident (Jones):")
print(J1)
print("\nScattered (Jones:")
print(J2)

print("\n\nIncident (Stokes):")
print(S1)
print("\nScattered (Stokes):")
print(S2)

# We're looking upstream:
fig,axs = plt.subplots(1,2,figsize=(12,8))

print("\nIncident beam (polarized component):")
plt.sca(axs[0])
J1.plot_efield()
if S1.dop==0:
    axs[1].set_aspect('equal')
    axs[1].axis('off')
else:
    plt.sca(axs[1])
    S1.decompose()["pol"].plot_efield()
plt.show()

fig,axs = plt.subplots(1,2,figsize=(12,8))

print("Scattered beam (polarized component in scattered reference frame):")
plt.sca(axs[0])
J2.plot_efield()
if S2.dop==0:
    axs[1].set_aspect('equal')
    axs[1].axis('off')
else:
    plt.sca(axs[1])
    S2.decompose()["pol"].plot_efield()

plt.show()
```

```python
# Define incident intensity and polarization
intensity = 1 # W/m^2
dop = 0.7 # degree of polarization (in [0,1])
dolp = 0.95*dop # degree of linear polarization (in [0,dop])
polangle = 0 # angle of polarization ellipse with respect to the horizontal direction (in [-90,90])
handedness = "left"

S1 = polarization.Stokes.from_params(intensity=intensity,dop=dop,dolp=dolp,\
                                    polangle=polangle,handedness=handedness)

if S1.dop!=0:
    print("\nIncident beam (polarized component):")
    S1.decompose()["pol"].plot_efield()
    plt.show()

azimuth = np.linspace(0,360,100)
polar = np.linspace(0,180,50)
extent = [azimuth[0],azimuth[-1],polar[0],polar[-1]]
azimuth,polar = np.meshgrid(azimuth,polar)

print("Incident intensity: {} W/m²".format(S1.intensity))
print("Scattered intensity (phi=0deg,theta=90deg) = {} W/m²".format(S1.thomson_intensity(0,90)))
print("Scattered intensity (phi=90deg,theta=90deg) = {} W/m²".format(S1.thomson_intensity(90,90)))

img = S1.thomson_intensity(azimuth,polar)
plt.imshow(img,origin="lower",extent=extent)
plt.axhline(y=90)
for x in [90,180,270]:
    plt.axvline(x=x)
plt.xlabel("Azimuth (deg)")
plt.ylabel("Polar (deg)")
plt.title("Incident intensity = {} W/m$^2$".format(S1.intensity))
plt.colorbar(label="W/m$^2$")
plt.show()
```

### Thomson cross-section

The Thomson cross-section (elastic cross-section for a free electron) can be written as

$$
\begin{equation}
\begin{split}
\frac{d\sigma_T}{d\Omega} =& r_e^2\frac{I_{sc}(\phi,\theta)}{I_0}\\
=&r_e^2 K(\phi,\theta)\\
\sigma_T =& r_e^2\int_0^{2\pi}\int_0^\pi K(\phi,\theta) \sin\theta d\theta d\phi\\
=&\frac{8\pi}{3}r_e^2
\end{split}
\end{equation}
$$

```python
from scipy import integrate

K = S1.thomson_K
integrand = lambda azimuth,polar: K(azimuth,polar)*np.sin(polar)

thomsonsc = integrate.dblquad(integrand, 0, np.pi, lambda x:0, lambda x:2*np.pi)[0]
print(thomsonsc,8*np.pi/3) # units of r_e^2
```

    8.377580409572783 8.377580409572781

### Rayleigh cross-section

The differential Rayleigh cross-section (elastic scattering cross-section of an atom) is proportional to the differential Thomson cross-section and the squared atomic form factor

$$
\begin{equation}
\frac{d\mu_R}{d\Omega} = r_e^2 K(\phi,\theta) \frac{N_A}{M}f^2(E,\theta)
\end{equation}
$$

with SI units $cm^2/g/sr$, $M$ the molar mass of the atom ($g/mol$), $N_A$ the Avogadro constant ($1/mol$) and $f$ the atomic form factor.

```python
from spectrocrunch.materials import element
from spectrocrunch.sources import xray as xraysources

source = xraysources.factory("synchrotron")
print(source)
if source.stokes.dop!=0:
    print("\nPolarized component:")
    source.stokes.decompose()["pol"].plot_efield()
    plt.show()

el = element.Element("Fe")
polar = np.linspace(0,2*np.pi,500)[:-1]

fig,axs = plt.subplots(1,2,figsize=(15, 5),subplot_kw={"projection":"polar"})
for ax,azimuth in zip(axs,[0,np.pi/2]):
    plt.sca(ax)

    for energy in [2,5,10,20]:
        diffcs = el.diff_rayleigh_cross_section(energy,source)
        r = diffcs(azimuth,polar)
        ax.plot(polar, r, label="{}keV".format(energy))
    ax.set_title("Azimuth = {} deg".format(np.degrees(azimuth)))
    ax.legend()

energy = 10
diffcs = el.diff_rayleigh_cross_section(energy,source)
integrand = lambda azimuth,polar: diffcs(azimuth,polar)*np.sin(polar)
cs1 = integrate.dblquad(integrand, 0, np.pi, lambda x:0, lambda x:2*np.pi)[0]
cs2 = el.rayleigh_cross_section(energy)
print("Rayleigh cross-section ({} keV): {} cm²/g (integrated)".format(energy,cs1))
print("Rayleigh cross-section ({} keV): {} cm²/g (tabulated)".format(energy,cs2))

plt.show()
```

## Compton scattering

Using the same incident and reference frames as for Thomson scattering, the Stokes parameters are transformed as follows [(McMaster[1961])](https://link.aps.org/doi/10.1103/RevModPhys.33.8)

$$
\begin{equation}
\begin{split}
S_{sc}=&M\cdot S\\
M =& \frac{E_{sc}^2}{E^2}\begin{bmatrix}
a+b&(1-a)\cos2\beta&-(1-a)\sin2\beta&0\\
(1-a)&a\cos2\beta&-a\sin2\beta&0\\
0&\cos\theta\sin2\beta&\cos\theta\cos2\beta&0\\
0&0&0&(b+1)\cos\theta
\end{bmatrix}\\
\beta=&\frac{\pi}{2}-\phi=\arccos(\hat{e}_0\cdot\hat{e}_0^\prime)\\
a=&\frac{1+\cos^2\theta}{2}\\
b=&\frac{E-E_{sc}}{2 m_e c^2}(1-\cos\theta)\\
E_{sc}=&\frac{E}{1+\frac{E}{m_e c^2}(1-\cos\theta)}
\end{split}
\end{equation}
$$

where $E$ and $E_{sc}$ the incident and scattered photon energy. Note that when $E_{sc}=E$ we obtain the same expressions as for Thomson scattering. The scattered intensity can be written as

$$
\begin{equation}
\begin{split}
I_{sc}=&I_0K(\phi,\theta)\\
K(\phi,\theta)=&\frac{E_{sc}^2}{E^2}\left(a+b-(1-a)\left(\frac{S_1}{S_0}\cos2\phi+\frac{S_2}{S_0}\sin2\phi\right)\right)
\end{split}
\end{equation}
$$

Two common examples are $S_1=S_2=S_3=0$ (unpolarized) and $S_1=S_0$, $S_2=S_3=0$ (linear polarized in the horizontal direction)

$$
\begin{equation}
\begin{split}
K_{un}=&\frac{1}{2}\frac{E_{sc}^2}{E^2}\left(\frac{E}{E_{sc}}+\frac{E_{sc}}{E}-\sin^2\theta\right)\\
K_{linh}=&\frac{1}{2}\frac{E_{sc}^2}{E^2}\left(\frac{E}{E_{sc}}+\frac{E_{sc}}{E}-2\sin^2\theta\cos^2\phi\right)
\end{split}
\end{equation}
$$

```python
# Define incident intensity and polarization
intensity = 1 # W/m^2
dop = 0.4 # degree of polarization (in [0,1])
dolp = 0.7*dop # degree of linear polarization (in [0,dop])
polangle = 0 # angle of polarization ellipse with respect to the horizontal direction (in [-90,90])
handedness = "left"

# Partially polarized
S1 = polarization.Stokes.from_params(intensity=intensity,dop=dop,dolp=dolp,\
                                    polangle=polangle,handedness=handedness)

# Scattering direction: spherical coordinates
azimuth = 20
polar = 50 # scattering angle
energy = 5

S2 = S1.compton_scattering(azimuth,polar,energy)

# We're looking upstream:
if S1.dop!=0:
    print("\nIncident beam (polarized component):")
    S1.decompose()["pol"].plot_efield()
    plt.show()

if S2.dop!=0:
    print("Scattered beam (polarized component in diffraction reference frame):")
    S2.decompose()["pol"].plot_efield()
    plt.show()
```

```python
# Define incident intensity and polarization
intensity = 1 # W/m^2
dop = 0.7 # degree of polarization (in [0,1])
dolp = 0.95*dop # degree of linear polarization (in [0,dop])
polangle = 0 # angle of polarization ellipse with respect to the horizontal direction (in [-90,90])
handedness = "left"

S1 = polarization.Stokes.from_params(intensity=intensity,dop=dop,dolp=dolp,\
                                    polangle=polangle,handedness=handedness)

if S1.dop!=0:
    print("\nIncident beam (polarized component):")
    S1.decompose()["pol"].plot_efield()
    plt.show()

azimuth = np.linspace(0,360,100)
polar = np.linspace(0,180,50)
extent = [azimuth[0],azimuth[-1],polar[0],polar[-1]]
azimuth,polar = np.meshgrid(azimuth,polar)
energy = 5

print("Incident intensity: {} W/m²".format(S1.intensity))
print("Scattered intensity (phi=0deg,theta=90deg) = {} W/m²".format(S1.compton_intensity(0,90,energy)))
print("Scattered intensity (phi=90deg,theta=90deg) = {} W/m²".format(S1.compton_intensity(90,90,energy)))

img = S1.compton_intensity(azimuth,polar,energy)
plt.imshow(img,origin="lower",extent=extent)
plt.axhline(y=90)
for x in [90,180,270]:
    plt.axvline(x=x)
plt.xlabel("Azimuth (deg)")
plt.ylabel("Polar (deg)")
plt.title("Incident intensity = {} W/m$^2$".format(S1.intensity))
plt.colorbar(label="W/m$^2$")
plt.show()
```

### Klein-Nishina cross-section

The Klein-Nishina cross-section (inelastic cross-section for a free electron) can be written as

$$
\begin{equation}
\begin{split}
\frac{d\sigma_{KN}}{d\Omega} =& r_e^2\frac{I_{sc}(\phi,\theta)}{I_0}\\
=&r_e^2 K(\phi,\theta)\\
\sigma_{KN} =& r_e^2\int_0^{2\pi}\int_0^\pi K(\phi,\theta) \sin\theta d\theta d\phi\\
=&2\pi r_e^2 \left(
\frac{1+k}{k^2}\left(\frac{2+2k}{c}-\frac{\ln (1+2k)}{k}\right)+\frac{\ln (1+2k)}{2k}-\frac{1+3k}{(1+2k)^2}
\right)\\
k =& \frac{E}{m_e c^2}
\end{split}
\end{equation}
$$

```python
from spectrocrunch.patch.pint import ureg

K = S1.compton_K(energy)
integrand = lambda azimuth,polar: K(azimuth,polar)*np.sin(polar)

comptonsc = integrate.dblquad(integrand, 0, np.pi, lambda x:0, lambda x:2*np.pi)[0]

E = ureg.Quantity(energy,"keV").to("m_e*c^2").magnitude

c = 2*E+1
logc = np.log(c)
comptonsc1 = 2*np.pi*((1+E)/E**2*((1+c)/c-logc/E)+logc/(2*E)-(1+3*E)/c**2)

P0 = 2*E**4 + 18*E**3 + 16*E**2 + 4*E
P1 = 4*E**4 - 4*E**3 - 15*E**2 - 10*E - 2
P2 = 4*E**5 + 4*E**4 + E**3
comptonsc2 = np.pi*(P0 + P1*np.log(2*E + 1))/P2

print(comptonsc,comptonsc1,comptonsc2) # units of r_e^2
```

    8.21770414755146 8.217704148257559 8.217704148224696

### Compton cross-section

The differential Compton cross-section (inelastic scattering cross-section of an atom) is proportional to the differential Klein-Nishina cross-section and the incoherent scattering function

$$
\begin{equation}
\frac{d\mu_C}{d\Omega} = r_e^2 K(\phi,\theta) \frac{N_A}{M}S(E,\theta)
\end{equation}
$$

with SI units $cm^2/g/sr$, $M$ the molar mass of the atom ($g/mol$), $N_A$ the Avogadro constant ($1/mol$) and $S$ the incoherent scattering function of the atom.

```python
source = xraysources.factory("synchrotron")
print(source)
if source.stokes.dop!=0:
    print("\nPolarized component:")
    source.stokes.decompose()["pol"].plot_efield()
    plt.show()

el = element.Element("Fe")
polar = np.linspace(0,2*np.pi,500)

fig,axs = plt.subplots(1,2,figsize=(15, 5),subplot_kw={"projection":"polar"})
for ax,azimuth in zip(axs,[0,np.pi/2]):
    plt.sca(ax)

    for energy in [2,5,10,20]:
        diffcs = el.diff_compton_cross_section(energy,source)
        r = diffcs(azimuth,polar)
        ax.plot(polar, r, label="{}keV".format(energy))
    ax.set_title("Azimuth = {} deg".format(np.degrees(azimuth)))
    ax.legend()

energy = 10
diffcs = el.diff_compton_cross_section(energy,source)
integrand = lambda azimuth,polar: diffcs(azimuth,polar)*np.sin(polar)
cs1 = integrate.dblquad(integrand, 0, np.pi, lambda x:0, lambda x:2*np.pi)[0]
cs2 = el.compton_cross_section(energy)
print("Compton cross-section ({} keV): {} cm²/g (integrated)".format(energy,cs1))
print("Compton cross-section ({} keV): {} cm²/g (tabulated)".format(energy,cs2))
plt.show()
```

The abscence of forward scattering can be attributed to the incoherent scattering function.

Compare Rayleigh and Compton differential cross-sections:

```python
for source in ['synchrotron', 'tube']:
    source = xraysources.factory(source)
    print(source)
    if source.stokes.dop!=0:
        print("\nPolarized component:")
        source.stokes.decompose()["pol"].plot_efield()
        plt.show()

    el = element.Element("Fe")
    polar = np.linspace(0,2*np.pi,500)[:-1]
    energy = 5

    fig,axs = plt.subplots(1,2,figsize=(15, 5),subplot_kw={"projection":"polar"})
    for ax,azimuth in zip(axs,[0,np.pi/2]):
        plt.sca(ax)

        diffcs = el.diff_rayleigh_cross_section(energy,source)
        rr = diffcs(azimuth,polar)
        ax.plot(polar, rr, label="Rayleigh")
        diffcs = el.diff_compton_cross_section(energy,source)
        rc = diffcs(azimuth,polar)
        m = int(np.round(np.max(rr)/np.max(rc))/10)*10
        ax.plot(polar, rc*m, label="{}x Compton".format(m))

        ax.set_title("Azimuth = {} deg".format(np.degrees(azimuth)))
        ax.legend()
    plt.show()
```


### Plotting workbook
Plotting workbook to accompany the paper _Carbon Capture and Storage Energy Systems vs. Dispatchable Renewables for Climate Mitigation: A Bio-physical Comparison_ by Sgouridis, Dale, Csala, Chiesa & Bardi

```python
import matplotlib.pyplot as plt
import matplotlib as mpl
import matplotlib.font_manager as font_manager
plt.style.use('seaborn-darkgrid')
%matplotlib inline
prop = font_manager.FontProperties('Segoe UI')
from sympy import *
init_printing()
```

### Eq. 6
Define EROI-CCS function (Eq. 6 in the paper)

```python
var('EROI_CCS EROI f_op');
```

```python
eq6=Eq(EROI_CCS,EROI/(1+f_op*EROI))
eq6
```

### Eq. 2
Define dispatchable EROI for renewables with storage (Eq. 2 in the paper)

```python
phi = Symbol('phi')
eta = Symbol('eta')
var('EROI_disp EROI ESOI phi eta');
```

```python
eq2=Eq(EROI_disp,((1-phi)+(eta*phi))/((1/EROI)+(eta*phi/ESOI)))
eq2
```

### Figure 3 & Table 3
Now we generate the contour lines for _Figure 3_, using the examples from _Table 3_.

We have two tyes of renewable energy plants, _PV_ and _Wind_, each with two cases of EROI, taken from the lower and upper boundaries encountered in studies.

We have assumed a storage technology mixes taken from _Table 1_ of  [Barnhart et al 2013](http://pubs.rsc.org/en/content/articlehtml/2013/ee/c3ee41973h). Here we have the ESOI values reported for 7 energy storage storage technologies, 5 types of electrochemical storage (Batteries), Compressed air (CAES) and pumped hydro (PHS).

We have considered diffrent energy storage technology mixes for our PV and Wind examples. For PV, we have taken an equal mix of all 5 electrochemical storage technologies, without CAES or PHS (resulting in the pv share vector of `[0.2,0.2,0.2,0.2,0.2,0,0]`). For Wind, we have considered 50% battery storage (as an equal 10% from each technology), and 25-25% of CAES and PHS, resulting in the wind share vector of `[0.1,0.1,0.1,0.1,0.1,0.25,0.25]`).

```python
etas=[0.90,0.75,0.90,0.75,0.60,0.70,0.85] #from (Barnhart et al 2013)
S=[32,20,5,10,9,797,704]  #from (Barnhart et al 2013)
shr0=[0.2,0.2,0.2,0.2,0.2,0,0] #assumed shares for considered PV exmaples in this paper
shr1=[0.1,0.1,0.1,0.1,0.1,0.25,0.25]  #assumed shares for considered Wind exmaples in this paper
eta_pv=sum([etas[i]*shr0[i] for i in range(len(etas))]) #resulting composite eta for pv
eta_wind=sum([etas[i]*shr1[i] for i in range(len(etas))]) #resulting composite eta for wind
S_pv=sum([S[i]*shr0[i] for i in range(len(etas))])  #resulting composite S for pv
S_wind=sum([S[i]*shr1[i] for i in range(len(etas))])  #resulting composite S for wind
print 'Composite eta for pv',eta_pv
print 'Composite eta for wind',eta_wind
print 'Composite ESOI for pv',S_pv
print 'Composite ESOI for wind',S_wind
```

    Composite eta for pv 0.78
    Composite eta for wind 0.7775
    Composite ESOI for pv 15.2
    Composite ESOI for wind 382.85

### Construct test cases: _Renewables_

```python
#define pretty output formatter for sympy equation
def format_eq(eq):
    return [solve(eq[i])[0] for i in range(len(eq))]
    #return [float((repr(eq[i])[2:]).split(',')[1].strip()[:-1]) for i in range(len(eq))]
```

```python
eroi_pv=[10,40] #EROEIe for case [F, G] in Table 3 of the paper, PV plant
eroi_disp_pv=format_eq([eq2.subs([[EROI,i],[phi,0.4],[eta,eta_pv],[ESOI,S_pv]]) for i in eroi_pv])
eroi_wind=[10,33] #EROEIe for case [H, I] in Table 3 of the paper, Wind plant
eroi_disp_wind=format_eq([eq2.subs([[EROI,i],[phi,0.4],[eta,eta_wind],[ESOI,S_wind]]) for i in eroi_wind])
```

```python
print 'PV [F,G]'
eroi_disp_pv
```

```python
print 'Wind [H,I]'
eroi_disp_wind
```

### Construct test cases: _Fossils_

```python
eroi_gas=[9.6, 38.5] #EROEIe for case [A, B] in Table 3 of the paper, gas plant
eroi_ccs_gas=format_eq([eq6.subs([[EROI,i],[f_op,0.147]]) for i in eroi_gas])
eroi_coal=[8.9, 22.6] #EROEIe for case [D, E] in Table 3 of the paper, coal plant
eroi_ccs_coal=format_eq([eq6.subs([[EROI,i],[f_op,0.219]]) for i in eroi_coal])
```

```python
print 'gas [A,B]'
eroi_ccs_gas
```

```python
print 'coal [D,E]'
eroi_ccs_coal
```

### Figure 3
Produce contour lines for multiple $f_{op}$ and $\phi$ levels and place the example plants on the contours. Calculate the $EROI_{CCS}$ and $EROI_{disp}$, respectively, for multiple base $EROI$ values. Likewise we obtain a plot for comparison between dispatchable (i.e. with storage) EROI for renewables and CCS EROI for fossil fuel based power plants.

```python
plt.figure(figsize=(5,4))
n=51
erois=range(1,n)

### Renewables contours
colors=['#addd8e',
        '#78c679',
        '#31a354',
        '#006837']
plt.plot(0,0,lw=0,label='${\phi}$ (%)')
#plot EROI_disp contour lines for various EROI values and phi levels
#for eta and S assume a fully battery-based storage (worst case), i.e. the eta_pv and S_pv
pv=1.0 #(pv share in hypothetical renewable technologies)
phis=[0.1,0.2,0.3,0.4] #assumed levels for phi
for r in range(len(phis)):
    phir=phis[r]
    eroi_disps=format_eq([eq2.subs([[EROI,i],[phi,phir],[eta,pv*eta_pv+(1-pv)*eta_wind],[ESOI,pv*S_pv+(1-pv)*S_wind]]) for i in erois])
    plt.plot(erois,eroi_disps,c=colors[r],label=int(phir*100))

plt.xlabel('EROI')
a=plt.gca()
plt.text(-0.07, 0.65, 'EROI$_{disp}$',
        horizontalalignment='right',
        verticalalignment='bottom',rotation=90,#fontproperties=prop,
        size=11,color='#006837',transform=plt.gca().transAxes)
plt.text(-0.07, 0.27, 'EROI$_{CCS}$',
        horizontalalignment='right',
        verticalalignment='bottom',rotation=90,#fontproperties=prop,
        size=11,color='#bd0036',transform=plt.gca().transAxes)

### Fossils contours
colors=['#fed976',
        '#feb24c',
        '#f03b20',
        '#bd0036']
plt.plot(0,0,lw=0,label='$f_{op}$ (%)')
fops=[0.05,0.1,0.15,0.2]  #assumed levels for fop
for r in range(len(fops)):
    fopr=fops[r]
    eroi_ccss=format_eq([eq6.subs([[EROI,i],[f_op,fopr]]) for i in erois])
    plt.plot(erois,eroi_ccss,c=colors[r],label=int(fopr*100))

### Table 3 cases
cf="#f03b20"
cr="#31a354"
cg="#aaaaaa"
s=30 #marker size
plt.scatter(eroi_gas,eroi_ccs_gas,s,color=cf,marker='o')
plt.scatter(eroi_coal,eroi_ccs_coal,s,color=cf,marker='^')
plt.scatter(eroi_pv,eroi_disp_pv,s,color=cr,marker='d')
plt.scatter(eroi_wind,eroi_disp_wind,s,color=cr,marker='s')
plt.xlim(0,50)
plt.ylim(0,40)
#[F, G]
plt.text(eroi_pv[1], eroi_disp_pv[1]-1.5, 'PV +\nBattery',
        horizontalalignment='center',
        verticalalignment='top',
        size=9,color=cr)
#[H, I]
plt.text(eroi_wind[1], eroi_disp_wind[1]+1.5, 'Wind + Battery +\n PHS + CAES',
        horizontalalignment='center',
        verticalalignment='bottom',
        size=9,color=cr)
#[A, B]
plt.text(eroi_gas[1], eroi_ccs_gas[1]-2.2, 'Gas + CCS',
        horizontalalignment='center',
        verticalalignment='top',
        size=9,color=cf)
#[D, E]
plt.text(eroi_coal[1], eroi_ccs_coal[1]-1, 'Coal + CCS',
        horizontalalignment='center',
        verticalalignment='top',
        size=9,color=cf)

#Annotate cases
ap=0.7
plt.text(eroi_coal[0]+2, eroi_ccs_coal[0]-0.5, 'D',
        horizontalalignment='center',
        verticalalignment='top',
        size=10,color=cf,
        bbox=dict(facecolor='white', edgecolor=cf,  alpha=ap, boxstyle='round'))
plt.text(eroi_gas[0]+2, eroi_ccs_gas[0]+2, 'B',
        horizontalalignment='center',
        verticalalignment='top',
        size=10,color=cf,
        bbox=dict(facecolor='white', edgecolor=cf,  alpha=ap, boxstyle='round'))
plt.text(eroi_coal[1], eroi_ccs_coal[1]+3.3, 'E',
        horizontalalignment='center',
        verticalalignment='top',
        size=10,color=cf,
        bbox=dict(facecolor='white', edgecolor=cf,  alpha=ap, boxstyle='round'))
plt.text(eroi_gas[1], eroi_ccs_gas[1]+3.3, 'A',
        horizontalalignment='center',
        verticalalignment='top',
        size=10,color=cf,
        bbox=dict(facecolor='white', edgecolor=cf, alpha=ap, boxstyle='round'))

plt.text(eroi_pv[1], eroi_disp_pv[1]+3.3, 'G',
        horizontalalignment='center',
        verticalalignment='top',
        size=10,color=cr,
        bbox=dict(facecolor='white', edgecolor=cr, alpha=ap, boxstyle='round'))
plt.text(eroi_wind[0]+2.6, eroi_disp_wind[0], 'F',
        horizontalalignment='center',
        verticalalignment='top',
        size=10,color=cr,
        bbox=dict(facecolor='white', edgecolor=cr,  alpha=ap, boxstyle='round'))
plt.text(eroi_pv[0], eroi_disp_pv[0]+5.3, 'H',
        horizontalalignment='center',
        verticalalignment='top',
        size=10,color=cr,
        bbox=dict(facecolor='white', edgecolor=cr,  alpha=ap, boxstyle='round'))
plt.text(eroi_wind[1], eroi_disp_wind[1]-2, 'I',
        horizontalalignment='center',
        verticalalignment='top',
        size=10,color=cr,
        bbox=dict(facecolor='white', edgecolor=cr,  alpha=ap, boxstyle='round'))

plt.legend(framealpha=0,loc=2,fontsize=9)
plt.savefig('fig3.png',bbox_inches = 'tight', facecolor='w', pad_inches = 0.1, dpi=200)
plt.show()
```

### Figure 4
For this figure we are considering various levels of powerplant efficiency, and its effect on the EROI , under various capture ratio assumptions. This is described in Eq. 1 and Eq. 4 in the paper.

Define parameters

Electricity penalty (b) curve from Sanpasternich (2009) Figure 8 [kWh/kg CO2]

```python
#CCS_op #operating CCS penalty
#CR #capture ratio
var('CCS_op CR');
CCS_op_eq=Eq(CCS_op,25.957**6 - 85.031*CR**5 + \
114.5*CR**4 - 80.385*CR**3 + \
31.47*CR**2 - 6.7725*CR  + 1.1137)
CCS_op_eq
```

Operational cost of coal fuel cycle over plant lifetime output [kWh]

```python
#PP_op_L #operational cost of coal fuel cycle over plant lifetime
#PP_op #operational cost of coal fuel cycle
#PP_CF #powerplant capacity factor
#PP_L #powerplant lifetime
#eta #powerplant efficiency
var('PP_op_L PP_op PP_CF eta PP_L');
PP_op_L_eq=Eq(PP_op_L,8760*PP_L*PP_CF*PP_op/eta)
PP_op_L_eq
```

Carbon intensity of electricity [kg CO2/kWh]

```python
#C_CO2 #carbon dioxide content of coal [kg/MJ]
#Elec_CO2 #carbon intensity of electricity [kg CO2/kWh]
var('Elec_CO2 C_CO2');
Elec_CO2_eq=Eq(Elec_CO2,C_CO2*3.6/eta)
Elec_CO2_eq
```

Energy penalty of CCS [dmnl]

```python
#b #energy penalty of CCS
var('b');
b_eq=Eq(b,CCS_op*Elec_CO2*CR)
b_eq
```

Emissions

```python
#E #emissions
var('E');
E_eq=Eq(E,CCS_op*(1-CR))
E_eq
```

Energy cost of constructing CCS plant

```python
#CCS_cons_energy #energy cost of constructing CCS plant
#f_cap #energy cost share of constructing CCS plant
#PP_cons_energy #energy cost of power plant construction [MJ/MW], does not include energy embodied in materials
var('CCS_cons_energy PP_cons_energy f_cap');
CCS_cons_energy_eq=Eq(CCS_cons_energy,f_cap*PP_cons_energy)
CCS_cons_energy_eq
```

EROIs

Energy cost of constructing CCS plant

```python
#EROI_1 #EROI where CCS construction energy is a ratio of power plant construction energy [MJ_e/MW/(MJ_p/MW) = dmnl]
#CCS_L #CCS plant lifetime
var('EROI_1 CCS_L');
EROI_1_eq=Eq(EROI_1,8760*PP_L*PP_CF*(1-b)*3.6/\
                    (PP_cons_energy + PP_op_L + CCS_cons_energy*PP_L/CCS_L))
EROI_1_eq
```

```python
#EROI_1_adj #makes adjustment for electrical output [MJ_p/MJ_p = dmnl]
var('EROI_1_adj');
EROI_1_adj_eq=Eq(EROI_1_adj,EROI_1/0.3)
EROI_1_adj_eq
```

Energy cost of operating CCS plant

```python
#EROI_2 #where CCS construction energy is a ratio of CCS operation energy [MJ_e/MJ_p = dmnl]
var('EROI_2');
EROI_2_eq=Eq(EROI_2,8760*PP_L*PP_CF*(1-b)/\
                    (PP_cons_energy + PP_op_L + f_op*CCS_op + CCS_cons_energy*PP_L/CCS_L))
EROI_2_eq
```

```python
#EROI_2_adj #makes adjustment for electrical output [MJ_p/MJ_p = dmnl]
var('EROI_2_adj');
EROI_2_adj_eq=Eq(EROI_2_adj,EROI_1/0.3)
EROI_2_adj_eq
```

Translation to quantities as per defined in the paper

### Eq. 5

```python
#R #E_op/E_cap
var('R');
eq5=Eq(EROI_CCS,(R+1)*EROI/(R+1+f_cap+f_op*(R+1)*EROI))
eq5
```

```python
var('E_op E_cap');
R_eq=Eq(R,E_op/E_cap)
R_eq
```

### Eq. 4

```python
var('E_CCS E_out E_red');
eq4=Eq(EROI_CCS,E_out/((1+f_cap)*E_cap+E_op+f_op*E_out))
eq4
```

```python
f_op_eq=Eq(f_op,E_red/E_out)
f_op_eq
```

```python
f_cap_eq=Eq(f_cap,E_CCS/E_cap)
f_cap_eq
```

### Eq. 3

```python
eq3=Eq(EROI_CCS,E_out/(E_cap+E_op+E_red+E_CCS))
eq3
```

### Eq. 1

```python
var('E_in');
eq1=Eq(EROI,E_out/E_in)
eq1
```

```python
E_in_eq=Eq(E_in,E_cap+E_op)
E_in_eq
```

```python
solve(E_in_eq,E_in)[0]
```

```python
eq1.subs([[E_in,solve(E_in_eq,E_in)[0]]])
```

### Verification
Bactkrack to Eq. 3 from Eq. 5

```python
eq5
```

Get back Eq. 4 from Eq. 5

```python
eq5.subs([[R,solve(R_eq,R)[0]],\
          [EROI,solve(eq1.subs([[E_in,solve(E_in_eq,E_in)[0]]]),EROI)[0]]]).simplify()
```

```python
eq4
```

Get back Eq. 3 from Eq. 4

```python
eq4.subs([[f_op,solve(f_op_eq,f_op)[0]],\
         [f_cap,solve(f_cap_eq,f_cap)[0]]]).simplify()
```

```python
eq3
```

### Translate
Translate example notation into paper equation notations

```python
eq3
```

```python
E_out_eq=Eq(E_out,8760*PP_L*PP_CF*(1-b))
E_out_eq
```

```python
E_out_eq.subs([[b,solve(b_eq,b)[0]]])\
        .subs([[CCS_op,solve(CCS_op_eq,CCS_op)[0]]])\
        .subs([[Elec_CO2,solve(Elec_CO2_eq,Elec_CO2)[0]]])
```

... ?

Construct from bottom up

```python
eq3
```


# An introduction to Gaussian Processes

## The Gaussian Distribution

In this notebook, we'll go over the very basics of Gaussian Processes (GPs) and how to construct and draw samples from them. But first, let's review some stuff about the Gausssian distribution $-$ the familiar bell curve $-$ itself.

We're probably all familiar with ``numpy``'s built-in ``np.random.randn()`` function, which draws a sample from the standard normal $\mathcal{N}(0, 1)$, i.e., a Gaussian with zero mean and unit variance:

```python
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(1)
u = np.random.randn(100000)
plt.hist(u, bins=50, density=1);
plt.axvline(0, color="C1", ls="--")
y = np.exp(-0.5) / (np.sqrt(2 * np.pi))
plt.plot([-1, 1], [y, y], "C1-")
plt.text(0.1, 0.05, r"$\mu$", color="C1", fontsize=20);
plt.text(0.3, 0.26, r"$\sigma$", color="C1", fontsize=20);
plt.text(-0.5, 0.26, r"$\sigma$", color="C1", fontsize=20);
```

To draw from a Gaussian distribution with a different mean $\mu$, we simply *add* $\mu$ to $u$, and to draw from a distribution with different variance $\sigma^2$, we simply *multiply* $u$ by $\sigma$. We can therefore draw from a Gaussian with mean (say) $10.0$ and variance $5.0^2$ by running

```python
np.random.seed(1)
u = np.random.randn(100000)
plt.hist(10.0 + 5.0 * u, bins=50, density=1);
```

**Easy.** But what about Gaussians in multiple dimensions, where there can be correlations (covariances) between the various dimensions? Say we want to draw from the normal distribution $\mathcal{N}(0, \mathbf{\Sigma})$, where $\mathbf{\Sigma}$ is the *covariance matrix*:

$$
\mathbf{\Sigma} =
\begin{pmatrix}
 \mathrm{Var(x_0)} & \mathrm{Cov(x_0, x_1)} & \cdots & \mathrm{Cov(x_0, x_N)} \\ \\
 \mathrm{Cov(x_1, x_0)} & \mathrm{Var(x_1)} & \cdots & \mathrm{Cov(x_1, x_N)} \\ \\
 \vdots & \vdots & \ddots & \vdots \\ \\
 \mathrm{Cov(x_N, x_0)} & \mathrm{Cov(x_N, x_1)} & \cdots & \mathrm{Var(x_N)}
\end{pmatrix}
$$

Here, $\mathrm{Var(x_i)}$ is the variance, or square of the standard deviation, along the $i^\mathrm{th}$ dimension, and $\mathrm{Cov(x_i, x_j)}$ is the covariance $-$ a measure of how two variables vary jointly $-$ between the $i^\mathrm{th}$ and $j^\mathrm{th}$ dimensions.

It's hard to picture a multivariate Gaussian, but it helps to imagine what the contours of equal probability (or equipotential surfaces) look like. For a two dimensional standard normal with no covariance ($\mathbf{\Sigma} = \mathbf{I}$), these are just circles. As I add to the off-diagonal terms of the matrix, I introduce a preferred direction in the space, since now the dimensions are correlated. The circle becomes an ellipse whose elongation increases with the covariance. In the general case of a multidimensional Gaussian, the contours form a multidimensional ellipsoid, elongated by different amounts along different dimensions.

**So.** How do we draw samples from a multivariate Gaussian given $\mathbf{\Sigma}$? It turns out there's a simple trick. Take $u$ to be a variable drawn from the standard normal (as we did above with ``u = np.random.randn()``) in $N$ dimensions. If we transform $u$ by a yet-to-be-determined operator (matrix) $\mathbf{L}$,

$$
\begin{align}
x = \mathbf{L}u
\end{align}
$$

we now have a random variable $x$ whose variance is

$$
\begin{align}
\mathrm{Var}(x) &\equiv \left< x x^\top \right> \\
                &= \left< \left(\mathbf{L}u\right) \left(\mathbf{L}u \right)^\top \right> \\
                &= \left< \mathbf{L} uu^\top \mathbf{L}^\top \right> \\
                &= \mathbf{L} \left<uu^\top\right> \mathbf{L}^\top \\
                &= \mathbf{L} \mathrm{Var}(u) \mathbf{L}^\top \\
                &= \mathbf{L}\mathbf{L}^\top.
\end{align}
$$

In the expressions above, I used the fact that the variance of a random variable is defined as the expectation value (denoted by $\left<\,\right>$) of the squared deviation from the mean (zero, in this case). The expectation value of $\mathbf{L}$ is just $\mathbf{L}$ (since it is constant), and the variance of $u$ is unity (by construction).

Now, all we have to do is find a matrix $\mathbf{L}$ such that

$$
\begin{align}
\mathbf{\Sigma} &= \mathbf{L} \mathbf{L}^\top
\end{align}
$$

and we're done!

**But how do we do that?** As it happens, what we're trying to do has a name: it's the **Cholesky decomposition** of $\mathbf{\Sigma}$. Formally, all the Cholesky decomposition does is to find a factorization of a matrix $\mathbf{\Sigma}$ in the form given by the expression above, where $\mathbf{L}$ is a lower triangular matrix. You can (informally) think of this decomposition as finding the square root of $\mathbf{\Sigma}$, and it is *enormously* useful for GPs (and linear algebra in general). Note that the Cholesky decomposition only works for Hermitian (symmetric), positive-definite matrices, but those are the only kinds of matrices we'll be dealing with here anyways.

To perform the Cholesky decomposition in Python, use ``scipy``:

```python
from scipy.linalg import cho_factor
L, _ = cho_factor(S, lower=True)  # Compute the lower triangular matrix L
L = np.tril(L)                    # Ensure the upper triangle of L is zeroed out
```

Exercise 1a

Define a function ``draw_from_gaussian(mu, S, ndraws=1)`` that returns ``ndraws`` samples from a multivariate Gaussian with mean ``mu`` and covariance matrix ``S``. The shape of the output should be ``(ndraws, ndim)`` where ``ndim`` is the dimension of the Gaussian. Note that as ``S`` gets large, roundoff error can often accumulate in the Cholesky decomposition, potentially causing weird ``LinearAlgebraError``s. For this reason, it's good practice to always add a tiny number to the diagonal of a matrix (something like ``1e-12``) whenever decomposing it, factorizing it, inverting it, etc. You probably want to bake that into your code to compute ``L``.

Exercise 1b

Draw ``10,000`` samples from a zero-mean Gaussian with covariance

$$
\mathbf{\Sigma} =
\begin{pmatrix}
 1 & 0.5 \\
 0.5 & 1
\end{pmatrix}
$$

Plot the "corner" plot for your ``samples`` using the ``corner`` package (``!pip install corner``):

```python
from corner import corner
fig = corner(samples);
```

Vary the terms in the covariance matrix (recalling that it must be symmetric!) to get a sense of how they affect the joint distribution of samples.

## Parametrizing the covariance

As the number of dimensions increases, and in particular as it becomes *infinite* (which it will, when we get to actual Gaussian Processes), it's no longer convenient to describe the covariance in terms of every single entry in the covariance matrix. Instead, it's useful to introduce the notion of a *kernel*, a function of just a few parameters that describes the overall structure of the covariance matrix.

In general, the covariance matrix can have *any* structure, but quite often in timeseries analysis *data points close to each other in time are strongly correlated*. This is true whether your timeseries contains photometric variability from a rotating star, the rise and fall of a supernova light curve, a gravitational wave signal, or PSF shape changes due to temperature fluctuations on the detector. This is because each of these processes have characteristic timescales over which they operate, and on timescales shorter than that, all measurements you make are likely to be close to relatively each other. But if you wait long enough, what your system will do down the line will be pretty decoupled from what it is doing right now.

So we can imagine defining a kernel that looks something like this:

```python
%show_kernel
```

where the covariance peaks at zero time lag and drops smoothly to zero as the time lag increases. This particular kernel is extremely useful and has a name: the *Squared Exponential Kernel*, defined as

$$
\begin{align}
    k(t_i, t_j) = A^2 \exp\left(-\frac{\left(t_i - t_j\right)^2}{2l^2}\right)
\end{align}
$$

Exercise 2a

Define a function ``ExpSquaredKernel(t, A=1.0, l=1.0)`` that returns the covariance matrix described by the squared exponential kernel for a timeseries ``t``.

Exercise 2c

For ``t = np.linspace(0, 2, 11)``, draw ``10,000`` samples from the guassian described by the covariance matrix and plot the corresponding corner plot, labeling each subplot with the time point it represents. Comment on the structure of the plot and the correlations among the different dimensions.

## From Gaussians to Gaussian Processes

So far we've been taking about Gaussians *distributions* and how they can jointly model a few random variables at a time. But what happens as the dimensionality of the Gaussian increases and eventually becomes infinite? The Gaussian no longer represents a collection of random variables, but instead the behaviour of a continuous function. And that's exactly what a Gaussian Process is: a distribution over functions with infinitely many points. We can't technically model a function with infinitely many points on a computer, but we can investigate what happens as the number of points becomes very large.

Exercise 3a

Construct a ``1000 x 1000`` covariance matrix using the squared exponential kernel for a timeseries spanning 10 time units, and plot an image of it using ``plt.imshow``. Comment on the structure of the matrix.

Exercise 3b

Draw ``30`` samples from the zero-mean Gaussian described by the covariance matrix above. Each sample should be a vector of length ``1,000``. Plot each sample against time on the same plot. Also plot the mean function and use ``plt.fill_between`` to shade the 1- and 2-$\sigma$ levels of the Gaussian Process. Recall that the variance $\sigma^2$ at each point in time is given by the corresponding entry on the diagonal of $\mathbf{\Sigma}$.

Next, vary the hyperparameters of the GP, $A$ and $l$, and comment on how they change the behavior of the GP.

## Conditioning on data

If you completed **Exercise 3b**, you have actually implemented your own Gaussian Process! But it's still pretty simple, because so far we are drawing only from our *prior*: we know the Gaussian has a certain covariance and a certain mean (zero, in the case above), but nothing else about how it's supposed to behave. To apply this to a real dataset, we need to *condition* the GP on observations.

Say I make two perfectly noise-free observations:

$$
\begin{align}
    y(t = 2.5) &= 1.0 \\
    y(t = 7.5) &= -1.0
\end{align}
$$

Again, $y$ could be anything: the flux received from a star, the strain due to a gravitational wave, etc. I want to draw new samples from my GP, but not just *any* samples: I explicitly want to draw functions that go through those two data points. One (very inefficient) way to do this is to draw a *ton* of samples and discard any samples that don't agree with the data. (In this particular case, that would be literally impossible, since I know $y(t)$ at those two points *exactly*: I will never draw at random a function whose value is exactly what I want it to be. But note that *rejection sampling* can actually be useful in other cases. Anyways.)

A *better* way to do this is to consider the *joint distribution* of the observed data $y(t)$ (usually called the "training set") and the points where I'm trying to predict the value of the function $y_\star(t_\star)$ (the "test set"):

$$
\begin{align}
\begin{pmatrix}
 y \\
 y_\star
\end{pmatrix}
&=
\mathcal{N}
\left[
\mathbf{0},
\begin{pmatrix}
 \mathbf{\Sigma}(t, t) & \mathbf{\Sigma}(t, t_\star)\\
 \mathbf{\Sigma}(t_\star, t) & \mathbf{\Sigma}(t_\star, t_\star)
\end{pmatrix}
\right]
\end{align}
$$

where $\mathbf{\Sigma}(t, t)$ is the $N_\mathrm{train} \times N_\mathrm{train}$ covariance matrix evaluated at all pairs of training points, $\mathbf{\Sigma}(t_\star, t_\star)$ is the $N_\mathrm{test} \times N_\mathrm{test}$ covariance matrix evaluated at all pairs of test points, and the remaining two entries are the (rectangular) covariance matrices evaluated at all pairs of (test, training) points.

Given this joint distribution, we can compute the *conditional* distribution of $y_\star$ given $y$ with some linear algebra (for a derivation, see Appendix A.2 of [Rasmussen & Williams (2006)](http://www.gaussianprocess.org/gpml/chapters/RW.pdf) and references therein):

$$
\begin{align}
y_\star | y \sim \mathcal{N}\left(
    \mathbf{\Sigma}(t_\star, t) \mathbf{\Sigma}(t, t)^{-1}y,
    \mathbf{\Sigma}(t_\star, t_\star) -
        \mathbf{\Sigma}(t_\star, t) \mathbf{\Sigma}(t, t)^{-1} \mathbf{\Sigma}(t, t_\star)
\right)
\end{align}
$$

In other words, given $y$, the distribution for $y_\star$ is *still Gaussian*, but this time with mean equal to
$$
\begin{align}
    \mathbf{\Sigma}(t_\star, t) \mathbf{\Sigma}(t, t)^{-1}y
\end{align}
$$
and covariance
$$
\begin{align}
    \mathbf{\Sigma}(t_\star, t_\star) - \mathbf{\Sigma}(t_\star, t) \mathbf{\Sigma}(t, t)^{-1} \mathbf{\Sigma}(t, t_\star)
\end{align}
$$

Exercise 4

Show that when $t_\star = t$, the mean and covariance of the GP for $y_\star$ give you what you'd expect. When $t_\star \neq t$, what can you say about the variance of the GP after versus before data was collected?

Exercise 5a

Define a function ``compute_gp(t_train, y_train, t_test, **kwargs)`` that returns the mean and covariance of the Gaussian process for ``y_test(t_test)`` conditioned on ``y_train(t_train)``. The ``**kwargs`` are for additional optional keyword arguments passed directly to the kernel (in this simple case, ``A`` and ``l``). You will need to re-define the ``ExpSquaredKernel`` to take *two* timeseries as inputs so you can compute the covariance between $t$ and $t_\star$.

Exercise 5b

Given

```python
t_train = np.array([2.5, 7.5])
y_train = np.array([1.0, -1.0])
t_test = np.linspace(0, 10, 1000)
```

compute the mean and covariance of the GP. Draw ``30`` samples from it as before. Plot the mean function and shade the 1- and 2-$\sigma$ levels. Comment on the behavior of the GP. Does the GP correctly predict the training points? How does the variance behave close to and far from the training points? How does the GP behave very far from the training points?

Exercise 6

The last thing we'll do regarding drawing from a GP is to account for noise in the observations. If the measurement error on the data points is Gaussian and uncorrelated (white), as we usually assume it to be, we simply add a term to the joint distribution of the training and test data:

$$
\begin{align}
\begin{pmatrix}
 y \\
 y_\star
\end{pmatrix}
&=
\mathcal{N}
\left[
\mathbf{0},
\begin{pmatrix}
 \mathbf{\Sigma}(t, t) + \sigma_n^2 \mathbf{I} & \mathbf{\Sigma}(t, t_\star)\\
 \mathbf{\Sigma}(t_\star, t) & \mathbf{\Sigma}(t_\star, t_\star)
\end{pmatrix}
\right]
\end{align}
$$

where $\sigma_n$ is the standard deviation for the $n^\mathrm{th}$ data point in the training set and $\mathbf{I}$ is the identiy matrix. Figure out how this changes the expression for the mean and variance of the GP and re-plot the figure from **Exercise 5b**, this time given an observational uncertainty $\sigma = 0.25$ at both training points. Comment on the new behavior of the GP.


# Colun or semicolun ?

In this notebook, you are going to implement a logistic regression algrorithm.
- 1st, you'll build a dataset
- 2nd, you'll you are going do define a model
- 3rd, a backpropagation method
- 4th, a gradient descent method

---
### Dataset

We build a dataset to illustrate our purpose.

The dataset we build is supposed to help us converting a paper scan into a ASCII string. Lets imagine that, when a paper is scaned, we can detect, with high confidence that we are over a colun or a semicolun. Our objective here is to detect wether it's one or the other.

Therefore, our algorithm is fed with a vector $x_i \in [0,1]^5$ which represent the intensity of the pen stroke writting on the paper.

Here below, you have an example of 'perfect' strokes for $x_1$ an example of colun, and $x_2$ an example of semicolun.

```python
import matplotlib.pyplot as plt
import numpy as np

x_1 = np.array([0,1,0,1,0])
x_2 = np.array([0,1,0,1,1])

def to_img(vec):
    matrix = np.ones((5, 3))
    matrix[:, 1] = 1-vec
    return matrix

fig, axs = plt.subplots(1,2)
axs[0].imshow(to_img(x_1), cmap='gray')
axs[1].imshow(to_img(x_2), cmap='gray')
plt.show()
```

Whenever a sample $x_i$ belongs to the class *colun*, we'll label it with $y_i=0$.
Likewise, whenever a sample $x_i$ belongs to the class *semicolun*, we'll label it with $y_i=1$.

```python
y_1 = 0
y_2 = 1
```

### Dataset generation

```python
X = np.array([
    [.0, 1., 0., 1., 0.],
    [.0, .9, 0., .9, 0.],
    [.2, .8, 0., .8, .2],
    [.0, 1., 0., 1., 1.],
    [.0, 1., 0., .5, .5],
    [.2, .8, 0., .7, .7]])
y = np.array([0,0,0,1,1,1])

fig, axs = plt.subplots(1,6)
for i in range(len(X)):
    axs[i].imshow(to_img(X[i]), cmap='gray')
plt.show()
```

---
### Define a logistic regression model

(You may want to read this : http://cs229.stanford.edu/notes/cs229-notes1.pdf).

You're going to build a model which outputs a prediction value $p_i$ given an input $x_i$. This prediction $p_i$ will reflect the propability that your input $x_i$ belongs to class 1.
$$
\begin{align}
p_i &= P(Y=1 | W, x_i) \\
p(x_i,W) &= P(Y=1 | W, x_i)
\end{align}
$$

As $p_i$ is a probability, it must be in [0,1].

The model we'll consider perform a weighted sum of its input:
- Weighted sum : $ s = (W^t \cdot X + b) $

And then squizes the values between 0 and 1 (which is our prediction value):
- prediction : $ p(s) = \frac{1}{1 + e^{-s}} $

```python
# Write the code ^^
```

---
### Compare these predicted values ($p_i$) with the true output ($y_i$)

Overall, we would like to maximize the likelihood that we are right at predicting a label.

$$
\begin{align}
\max \text{likelihood} &= \text{argmax}_w \Pi_i P(Y | W, x_i) \\
&= \text{argmax}_w \Pi_i \big( P(Y=y_i | W, x_i) \big) \\
&= \text{argmax}_w \Pi_i \big( P(Y=1 | W, x_i)^{y_i} \cdot P(Y=0 | W, x_i)^{1-y_i} \big) \\
&= \text{argmax}_w \Pi_i \big( P(Y=1 | W, x_i)^{y_i} \cdot 1-P(Y=1 | W, x_i)^{1-y_i}\big) \\
&= \text{argmax}_w \Pi_i \big( p_i^{y_i} \cdot 1-p_i^{1-y_i}\big) \\
&= \text{argmax}_w \sum_i \big( y_i \ln(p_i) + (1-y_i) \ln(1-p_i) \big) \\
&= \text{argmin}_w - \sum_i \big( y_i \ln(p_i) + (1-y_i) \ln(1-p_i) \big) \\
\end{align}
$$
And this term is going to be our **loss** that we want to reduce:
$$
L(x_i, W, y_i) = - \sum_i \big( y_i \ln(p_i) + (1-y_i) \ln(1-p_i) \big)
$$
This is how you compare the prediction you made ($p_i$) to the true output you expected ($y_i$).

#### In our example :
In means of colun and semicolun : remember $x_0$, it's a colun, therefore it's label is $y_0=0$.
If your classifier is good you'ld expect it to predict it's a semicolun, hense have $p_i = $*"Something small like 0.1"*.

The error for this one sample is going to be:
$$
\begin{align}
L(X, W, y) &= - \sum_i \big( y_i \ln(p_i) + (1-y_i) \ln(1-p_i) \big) \\
&= y_0 \ln(p_0) + (1-y_0) \ln(1-p_0) \\
&= - 0 \ln(.9) + (1-0) \ln(1-.9) \\
&= - \ln(.1)
\end{align}
$$

### Find the minimum of the Loss function

To reduce the error, we have to find the minimum of $L(x, W, y)$.
Hense, we derive it with respect to $W$ and find the 'zeros'.

```python
# Pen, paper, and your code ^^
```

### Apply Stochastic gradient descent to solve this

We are going to solve this with Stochastic Gradient Descent (SGD), meaning that we start with some values for $W$ and update this values such that our loss value disminushes.
$$
W = W + \alpha \frac{\delta L(x, W, y)}{\delta W}
$$

```python
# Code
```


# Calcule numérique avec la méthode Monte-Carlo (première partie)

**TODO**
- traduire en anglais ce qu'il reste à traduire...

## Approximation numérique d'une surface avec la méthode Monte-Carlo

Conceptuellement, pour calculer la surface $\mathcal A_1$ d'un objet $O_1$ avec la methode Monte-Carlo, il suffit:

1. de placer cet objet $O_1$ entièrement dans une figure géométrique $O_2$ dont on connait la surface $\mathcal A_2$ (par exemple un carré ou un rectangle)
2. tirer aléatoirement un grand nombre de points dans cette figure  $O_2$ (tirage uniforme)
3. compter le nombre de points $N_1$ tombés dans l'objet  $O_1$ dont on veut calculer la surface
4. calculer le rapport $\frac{N_1}{N}$ où $N$ est le nombre total de points tiré aléatoirement (en multipliant par 100 ce rapport on obtient le pourcentage de points tombés dans l'objet $O_1$)
5. appliquer ce rapport à la surface $\mathcal A_2$ de figure englobante $O_2$ (le carré, rectangle, ... dont on connait la surface) pour obtenir la surface $\mathcal A_1$ recherchée: $\mathcal A_1 \simeq \frac{N_1}{N} \mathcal A_2$

```python
import numpy as np
import matplotlib.pyplot as plt

t = np.linspace(0., 2. * np.pi, 100)
x = np.cos(t) + np.cos(2. * t)
y = np.sin(t)

N = 100
rand = np.array([np.random.uniform(low=-3, high=3, size=N), np.random.uniform(low=-3, high=3, size=N)]).T

fig, ax = plt.subplots(1, 1, figsize=(6, 6))
ax.plot(rand[:,0], rand[:,1], '.k')
ax.plot(x, y, "-r", linewidth=2)
ax.plot([-3, -3, 3, 3, -3], [-3, 3, 3, -3, -3], "-b", linewidth=2)
ax.set_axis_off()
ax.set_xlim([-4, 4])
ax.set_ylim([-4, 4])

plt.show()
```

Le même principe peut être appliqué pour calculer un volume.

Cette methode très simple est parfois très utile pour calculer la surface (ou le volume) de figures géometriques complexes. En revanche, elle suppose l'existance d'une procedure ou d'une fonction permettant de dire si un point est tombé dans l'objet $O_2$ ou non.

## Application au calcul d'intégrales

Calculer l'intégrale d'une fonction, revient à calculer la surface entre la courbe décrivant cette fonction et l'axe des abscisses (les surfaces au dessus de l'axe des abscisses sont ajoutées et les surfaces au dessous sont retranchées).

### Exemple written in Python

```python
import sympy as sp
sp.init_printing()

%matplotlib inline
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

#### The function to integrate

```python
def f(x):
    return -x**2 + 3.
```

#### Random points

```python
N = 100000           # The number of random points

x_lower_bound = -4.0
x_upper_bound = 4.0
y_lower_bound = -16.0
y_upper_bound = 16.0

random_points = np.array([np.random.uniform(low=x_lower_bound, high=x_upper_bound, size=N),
                          np.random.uniform(low=y_lower_bound, high=y_upper_bound, size=N)]).T
```

#### Numerical computation of the integral with Monte-Carlo

```python
# Points between f and the abscissa
random_points_in_pos = np.array([p for p in random_points if 0 <= p[1] <= f(p[0])])
random_points_in_neg = np.array([p for p in random_points if 0 >  p[1] >= f(p[0])])
```

```python
ratio_pos = float(len(random_points_in_pos)) / float(N)
ratio_neg = float(len(random_points_in_neg)) / float(N)
print('Percentage of "positive" points between f and the abscissa: {:.2f}%'.format(ratio_pos * 100))
print('Percentage of "negative" points between f and the abscissa: {:.2f}%'.format(ratio_neg * 100))

s2 = (x_upper_bound - x_lower_bound) * (y_upper_bound - y_lower_bound)
print("Box surface:", s2)
```

```python
s1 = ratio_pos * s2 - ratio_neg * s2
print("Function integral (numerical computation using Monte-Carlo):", s1)
```

#### The actual integral value

```python
x = sp.symbols("x")

integ = sp.Integral(f(x), (x, x_lower_bound, x_upper_bound))
sp.Eq(integ, integ.doit())
```

#### The error ratio

```python
actual_s1 = float(integ.doit())

error = actual_s1 - s1
print("Error ratio = {:.6f}%".format(abs(error / actual_s1) * 100.))
```

#### Graphical illustration

```python
x_array = np.arange(x_lower_bound, x_upper_bound, 0.01)
y_array = f(x_array)

plt.axis([x_lower_bound, x_upper_bound, y_lower_bound, y_upper_bound])
plt.plot(random_points[:,0], random_points[:,1], ',k')
plt.plot(random_points_in_pos[:,0], random_points_in_pos[:,1], ',r')
plt.plot(random_points_in_neg[:,0], random_points_in_neg[:,1], ',r')
plt.hlines(y=0, xmin=x_lower_bound, xmax=x_upper_bound)
plt.plot(x_array, y_array, '-r', linewidth=2)

plt.show()
```

## Suite

Dans le document suivant, nous allons appliquer le même principe pour calculer la constante $\pi$ :
http://www.jdhp.org/docs/notebook/maths_montecarlo_compute_pi_en.html.


# Lorenz

In this notebook we are going to syncronize a subsystem of a Lorenz system with another Lorenz system according to the proposed method of Pecora and Carroll (1990).

```python
from plotly import offline as py
from plotly import graph_objs as go

py.init_notebook_mode(connected=True)
```

A first system governed by the Lorenz,$$
\begin{align}
\dot{x}_t&=\sigma(y_t-x_t)\\
\dot{y}_t&=x_t(r-z_t)-y_t\\
\dot{z}_t&=x_ty_t-bz_t
\end{align},$$ acts as a transmitter with $r=60,\sigma=10,b=8/3$.

A second system governed by, $$
\begin{align}
\dot{x}_r&=x_d\\
\dot{y}_r&=x_r(r-z_r)-y_r\\
\dot{z}_r&=x_ry_r-bz_r
\end{align}, $$acts as a receiver, using the $x_d$ from the transmitter.

```python
from numba import jit
from random import uniform

r = 60
b = 8 / 3
sigma = 10

@jit(nopython=True)
def simulate(N, h):
    xt = [uniform(0, 10)]
    yt = [uniform(-60, 60)]
    zt = [uniform(0, 140)]

    xr = xt
    yr = [uniform(-60, 60)]
    zr = [uniform(0, 140)]

    for i in range(1, N+1):
        xti = xt[-1]
        yti = yt[-1]
        zti = zt[-1]

        xt.append(xti + h * sigma * (yti - xti))
        yt.append(yti + h * (xti * (r - zti) - yti))
        zt.append(zti + h * (xti * yti - b * zti))

        yri = yr[-1]
        zri = zr[-1]

        yr.append(yri + h * (xti * (r - zri) - yri))
        zr.append(zri + h * (xti * yri - b * zri))

    return (xt, yt, zt), (xr, yr, zr)
```

Introducing the error measure $e_2=y_t-y_r,e_3=z_t-z_r$ and subtracting the receiver equations from the transmitter equations one finds, $$
\begin{align}
\dot{e}_1&=0\\
\dot{e}_2&=-e_2-x_de_3\\
\dot{e}_3&=x_de_2-be_3
\end{align}. $$
If we multiply the second equation by $e_2$, the third equation by $4e_3$ and add the results we obtain, $$
-2\left(e_2^2+e_3^2\right)
=
e_2\dot{e}_2+e_3\dot{e}_3
=
\frac{1}{2}\frac{d}{dt}\left(e_2^2+e_3^2\right), $$ we identify $V(e_2,e_3)=e_2^2+e_3^2$ and see that $V$ is a Lyapunov function as $V$ it is positive definite and $\dot{V}$ is negative definite, thus, $e_2=e_3=0$ is a stable equilibrium point which one approaches asymptopically for $t\to\infty$ regardless of the initial conditions chosen for transmitter and receiver system, thus, the receiver will syncronize with the transmitter.

```python
N = 1000
h = 1e-2

t = np.arange(N) * h

(xt, yt, zt), (xr, yr, zr) = simulate(1000, 1e-2)

(xt[0], xr[0]), (yt[0], yr[0]), (zt[0], zr[0])
```

    ((6.324940084560655, 6.324940084560655),
     (35.15227077484603, -16.044206306661586),
     (127.13535435788735, 126.39523401898253))

```python
layout = go.Layout(
    xaxis=dict(title='t'),
    yaxis=dict(title='y(t)'),
)

figure = go.Figure([
    go.Scatter(x=t, y=yt, mode='lines+markers', name='transmitter'),
    go.Scatter(x=t, y=yr, mode='lines+markers', name='receiver'),
], layout)

py.iplot(figure)
```

```python
layout = go.Layout(
    xaxis=dict(title='y(t)'),
    yaxis=dict(title='z(t)'),
)

figure = go.Figure([
    go.Scatter(x=yt, y=zt, mode='lines+markers', name='transmitter'),
    go.Scatter(x=yr, y=zr, mode='lines+markers', name='receiver'),
], layout)

py.iplot(figure)
```


# DGP fundamentals

This notebook will introduce you to the fundamentals of **disciplined geometric programming** (**DGP**), which lets you formulate and solve *log-log convex programs* (LLCPs) in CVXPY.

LLCPs are problems that become convex after the variables, objective functions, and constraint functions are replaced with their logs, an operation that we refer to as a log-log transformation. LLCPs generalize geometric programming.

```python
import cvxpy as cp
```

# 1. Log-log curvature

Just as every Expression in CVXPY has a curvature (constant, affine, convex, concave, or unknown), every Expression also has a log-log curvature.

A function $f : D \subseteq \mathbf{R}^{n}_{++} \to \mathbf{R}$ is said to be **log-log convex** if the function $F(u)=\log f(e^u)$, with domain $\{u \in \mathbf{R}^n : e^u \in D\}$, is convex (where $\mathbf{R}^{n}_{++}$ denotes the set of positive reals and the logarithm and exponential are meant elementwise); the function $F$ is called the log-log transformation of $f$. The function $f$ is **log-log concave** if $F$ is concave, and it is **log-log affine** if $F$ is affine.

Notice that if a function has log-log curvature, then its domain and range can only include positive numbers.

The log-log curvature of an `Expression` can be accessed via its `.log_log_curvature` attribute. For an `Expression` to have known log-log curvature, all of the `Constant`s, `Variable`s, and `Parameter`s it refers to must be elementwise positive.

```python
# Only elementwise positive constants are allowed in DGP.
c = cp.Constant(1.0)
print(c, c.log_log_curvature)

c = cp.Constant([1.0, 2.0])
print(c, c.log_log_curvature)

c = cp.Constant([1.0, 0.0])
print(c, c.log_log_curvature)

c = cp.Constant(-2.0)
print(c, c.log_log_curvature)
```

    1.0 LOG-LOG CONSTANT
    [1. 2.] LOG-LOG CONSTANT
    [1. 0.] UNKNOWN
    -2.0 UNKNOWN

```python
# Variables and parameters must be positive, ie, they must be constructed with the option `pos=True`
v = cp.Variable(pos=True)
print(v, v.log_log_curvature)

v = cp.Variable()
print(v, v.log_log_curvature)

p = cp.Parameter(pos=True)
print(p, p.log_log_curvature)

p = cp.Parameter()
print(p, p.log_log_curvature)
```

    var0 LOG-LOG AFFINE
    var1 UNKNOWN
    param2 LOG-LOG CONSTANT
    param3 UNKNOWN

## Functions from geometric programming

A function $f(x)$ is log-log affine if and only if it is given by

$$
f(x) = cx_1^{a_1}x_2^{a_2} \ldots x_n^{a_n},
$$

where $c > 0$ and the $a_i$ are real numbers. In the context of geometric programming, such a function is called
a monomial.

```python
x = cp.Variable(shape=(3,), pos=True, name="x")
c = 2.0
a = [0.5, 2.0, 1.8]

monomial = c * x[0] ** a[0] * x[1] ** a[1] * x[2] ** a[2]
# Monomials are not convex.
assert not monomial.is_convex()

# They are, however, log-log affine.
print(monomial, ":", monomial.log_log_curvature)
assert monomial.is_log_log_affine()
```

    2.0 * power(x[0], 1/2) * power(x[1], 2) * power(x[2], 9/5) : LOG-LOG AFFINE

A sum of monomial functions is log-log convex; in the context of geometric programming, such a function is called a posynomial. There are functions that are not posynomials that are still log-log convex.

```python
x = cp.Variable(pos=True, name="x")
y = cp.Variable(pos=True, name="y")

constant = cp.Constant(2.0)
monomial = constant * x * y
posynomial = monomial + (x ** 1.5) * (y ** -1)
reciprocal = posynomial ** -1
unknown = reciprocal + posynomial

print(constant, ":", constant.log_log_curvature)
print(monomial, ":", monomial.log_log_curvature)
print(posynomial, ":", posynomial.log_log_curvature)
print(reciprocal, ":", reciprocal.log_log_curvature)
print(unknown, ":", unknown.log_log_curvature)
```

    2.0 : LOG-LOG CONSTANT
    2.0 * x * y : LOG-LOG AFFINE
    2.0 * x * y + power(x, 3/2) * power(y, -1) : LOG-LOG CONVEX
    power(2.0 * x * y + power(x, 3/2) * power(y, -1), -1) : LOG-LOG CONCAVE
    power(2.0 * x * y + power(x, 3/2) * power(y, -1), -1) + 2.0 * x * y + power(x, 3/2) * power(y, -1) : UNKNOWN

# 2. Log-log curvature ruleset

CVXPY has a library of atomic functions with known log-log curvature and monotonicity. It uses this information to
tag every `Expression`, i.e., every composition of atomic functions, with a log-log curvature. In particular,

A function $f(expr_1,expr_2,...,expr_n)$ is log-log convex if  $f$ is a log-log convex function and for each expri one of the following conditions holds:

$f$ is increasing in argument i and $expr_i$ is log-log convex.
$f$ is decreasing in argument $i$ and $expr_i$ is log-log concave.
$expr_i$ is log-log affine.
A function $f(expr_1,expr_2,...,expr_n)$ is log-log concave if  $f$ is a log-log concave function and for each $expr_i$ one of the following conditions holds:

$f$ is increasing in argument $i$ and $expr_i$ is log-log concave.
$f$ is decreasing in argument $i$ and $expr_i$ is log-log convex.
$expr_i$ is log-log affine.
A function $f(expr_1,expr_2,...,expr_n)$ is log-log affine if $f$ is an log-log affine function and each $expr_i$ is log-log affine.

If none of the three rules apply, the expression $f(expr_1,expr_2,...,expr_n)$ is marked as having unknown curvature.

If an Expression satisfies the composition rule, we colloquially say that the `Expression` “is DGP.” You can check whether an `Expression` is DGP by calling the method `is_dgp()`.

```python
x = cp.Variable(pos=True, name="x")
y = cp.Variable(pos=True, name="y")

monomial = 2.0 * x * y
posynomial = monomial + (x ** 1.5) * (y ** -1)

print(monomial, "is dgp?", monomial.is_dgp())
print(posynomial, "is dgp?", posynomial.is_dgp())
```

    2.0 * x * y is dgp? True
    2.0 * x * y + power(x, 3/2) * power(y, -1) is dgp? True

# 3. DGP problems

An LLCP is an optimization problem of the form

$$
\begin{equation}
\begin{array}{ll}
\mbox{minimize} & f_0(x) \\
\mbox{subject to} & f_i(x) \leq \tilde{f_i}, \quad i=1, \ldots, m\\
& g_i(x) = \tilde{g_i}, \quad i=1, \ldots, p,
\end{array}
\end{equation}
$$

where the functions $f_i$ are log-log convex, $\tilde{f_i}$ are log-log concave, and the functions $g_i$ and $\tilde{g_i}$ are log-log affine. An optimization problem with constraints of the above form in which the goal is to maximize a log-log concave function is also an LLCP.

A problem is DGP if additionally all the functions are DGP. You can check whether a CVXPY `Problem` is DGP by calling
its `.is_dgp()` method.

```python
x = cp.Variable(pos=True, name="x")
y = cp.Variable(pos=True, name="y")
z = cp.Variable(pos=True, name="z")

objective_fn = x * y * z
constraints = [
  4 * x * y * z + 2 * x * z <= 10, x <= 2*y, y <= 2*x, z >= 1]
assert objective_fn.is_log_log_concave()
assert all(constraint.is_dgp() for constraint in constraints)
problem = cp.Problem(cp.Maximize(objective_fn), constraints)

print(problem)
print("Is this problem DGP?", problem.is_dgp())
```

    maximize x * y * z
    subject to 4.0 * x * y * z + 2.0 * x * z <= 10.0
               x <= 2.0 * y
               y <= 2.0 * x
               1.0 <= z
    Is this problem DGP? True

## Solving DGP problems

You can solve a DGP `Problem` by calling its `solve` method with `gp=True`.

```python
problem.solve(gp=True)
print("Optimal value:", problem.value)
print(x, ":", x.value)
print(y, ":", y.value)
print(z, ":", z.value)
print("Dual values: ", list(c.dual_value for c in constraints))
```

    Optimal value: 1.9999999926890524
    x : 0.9999999989968756
    y : 1.9999999529045318
    z : 1.000000020895385
    Dual values:  [1.1111111199586956, 1.94877846244994e-09, 0.1111111217156332, 0.11111112214962586]

If you forget to supply `gp=True`, an error will be raised.

```python
try:
    problem.solve()
except cp.DCPError as e:
    print(e)
```

    Problem does not follow DCP rules. However, the problem does follow DGP rules. Consider calling this function with `gp=True`.

# 4. Next steps

## Atoms
CVXPY has a large library of log-log convex functions, including common functions like $\exp$, $\log$, and the difference between two numbers. Check out the tutorial on our website for the full list of atoms: https://www.cvxpy.org/tutorial/dgp/index.html

## References
For a reference on DGP, consult the following paper: https://web.stanford.edu/~boyd/papers/dgp.html


# Problem set 6: Solving the Solow model

[](https://mybinder.org/v2/gh/NumEconCopenhagen/exercises-2020/master?urlpath=lab/tree/PS6/problem_set_6.ipynb)

```python
import numpy as np
from scipy import linalg
from scipy import optimize
import sympy as sm
```

# Tasks

## Solving matrix equations I

```python
np.random.seed(1900)
n = 5
A = np.random.uniform(size=(n,n))
b = np.random.uniform(size=n)
c = np.random.uniform(size=n)
d = np.random.uniform(size=n)
```

**Question A:** Find the determinant of $[A \cdot A]^{-1}$

```python
# write your code here
```

**Answer:** see A1.py

**Question B:** Solve the following equation systems directly using **scipy**.

$$
\begin{aligned}
Ax &= b \\
Ax &= c \\
Ax &= d
\end{aligned}
$$

**Answer:** A2.py

**Question C:** Solve the same equation systems as above using `linalg.lu_factor()` and `linalg.lu_solve()`. What is the benefit of this approach?

**Answer:** A3.py

## Solving matrix equations II

```python
F = np.array([[2.0, 1.0, -1.0], [-3.0, -1.0, 2], [-2.0, 1.0, 2.0]])
e = np.array([8.0, -11.0, -3.0])
```

**Question:** Use the function `gauss_jordan()` in the `numecon_linalg` module located in this folder to solve

$$
Fx = e
$$

```python
# write your code here
```

**Answer:** see A4.py

## Symbolic

**Question A:** Find

$$
\lim_{x \rightarrow 0} \frac{\sin(x)}{x}
$$

and

$$
\frac{\partial\sin(2x)}{\partial x}
$$

```python
# write your code here
```

**Answer:** A5.py

**Question B:** Solve the equation

$$
\frac{\sin(x)}{x} = 0
$$

```python
# write your code here
```

**Answer:** A6.py

# Problem: Solve the Solow model

## Introduction

Consider the **standard Solow-model** where:

1. $K_t$ is capital2
2. $L_t$ is labor (growing with a constant rate of $n$)
3. $A_t$ is technology (growing with a constant rate of $g$)
4. $Y_t = F(K_t,A_tL_t)$ is GDP

**Saving** is a constant fraction of GDP

$$
S_t = sY_t,\,s\in(0,1)
$$

such that **capital accumulates** according to

$$
K_{t+1}=S_{t}+(1-\delta)K_{t}=sF(K_{t},A_{t}L_{t})+(1-\delta)K_{t}, \delta \in (0,1)
$$

The **production function** has **constant-return to scale** such that

$$
\frac{Y_{t}}{A_{t}L_{t}}=\frac{F(K_{t},A_{t}L_{t})}{A_{t}L_{t}}=F(\tilde{k}_{t},1)\equiv f(\tilde{k}_{t})
$$

where $\tilde{k}_t = \frac{K_t}{A_{t}L_{t}}$ is the technology adjusted capital-labor ratio.

The **transition equation** then becomes

$$
\tilde{k}_{t+1}= \frac{1}{(1+n)(1+g)}[sf(\tilde{k}_{t})+(1-\delta)\tilde{k}_{t}]
$$

If the **production function** is **Cobb-Douglas** then

$$
F(K_{t},A_{t}L_{t})=K_{t}^{\alpha}(A_{t}L_{t})^{1-\alpha}\Rightarrow f(\tilde{k}_{t})=\tilde{k}_{t}^{\alpha}
$$

If it is **CES** (with $\beta < 1, \beta \neq 0$) then

$$
F(K_{t},A_{t}L_{t})=(\alpha K_{t}^{\beta}+(1-\alpha)(A_{t}L_{t})^{\beta})^{\frac{1}{\beta}}\Rightarrow f(\tilde{k}_{t})=(\alpha\tilde{k}_{t}^{\beta}+(1-\alpha))^{\frac{1}{\beta}}
$$

## Steady state

Assume the production function is **Cobb-Douglas**.

**Question A:** Use **sympy** to find an analytical expression for the steady state, i.e. solve

$$
\tilde{k}^{\ast}= \frac{1}{(1+n)(1+g)}[sf(\tilde{k}^{\ast})+(1-\delta)\tilde{k}^{\ast}]
$$

```python
k = sm.symbols('k')
alpha = sm.symbols('alpha')
delta = sm.symbols('delta')
s = sm.symbols('s')
g = sm.symbols('g')
n = sm.symbols('n')
```

```python
# write your code here
```

**Answer:** see A7.py

**Question B:** Turn you solution into a Python function called as `ss_func(s,g,n,delta,alpha)`.

```python
# write your code here
```

**Answer:** A8.py

**Question C**: Find the steady state numerically using root-finding with `optimize.root_scalar`.

```python
s = 0.2
g = 0.02
n = 0.01
alpha = 1/3
delta = 0.1

# write your code here
```

**Answer:** A9.py

**Question D:** Now assume the production function is CES. Find the steady state for $k$ for the various values of $\beta$ shown below.

```python
betas = [-0.5,-0.25,-0.1,-0.05,0.05,0.1,0.25,0.5]

# write your code here
```

**Answer:** A10.py


# Основные понятия линейной алгебры
В даннном материале мы рассмотрим основные понятия линейной алгебры, которые необходимы для понимания основ машинного обучения.
Основными понятиями линейной алгебры являются векторы и матрицы. Основные операции линейной алгебры определяются с помощью линейных отображений. Во многом именно линейные отображения делают линейную алгебру столь важной для прикладных задач.

## Почему линейная алгебра столь важна?
Окружающий нас мир очень сложен и многие его проблемы имеют нелинейную природу. Изучение и понимание этих нелинейных связей в природе и в технике крайне сложно. Из-за этого инженеры, вместо того чтобы напрямую работать с нелинейными моделями, часто прибегают к линейным приблежениям этих моделей. Для этого нелинейную модель разбивают на многие меньшие части, где каждая часть в отдельности очень хорошо апроксимируется (т.е. приближается) линейным отображением. Как только проблема сформулирована в виде линейных отображений, сразу появляется возможность решить её очень эффективно и быстро с помощью методов линейной алгебры.

В данном материале будут даны лишь общие понятия, описание терминов и некоторые теоремы без доказательств. Для более глубокого изучения нужно обратиться к специализированным курсам по линейной алгебре.

## Скалярная величина (scalar)
Скалярная величина — величина, каждое значение которой может быть выражено одним действительным числом. Другими словами любое действительное число является скалярным значением. Термин появился в физике для обозначения величин, значения которых может быть выражено одним числом (длина, площадь, время, температура и т. д.), в отличии от других величин, которые выражаются с помощью векторов (например, сила имеет не только величину, но и направление). Скалярное значение обозначается маленькими буквами греческого (например, $\alpha$, $\beta$, $\gamma$) или латинского (например, a, b, c) алфавитов.

## Вектор (vector)
Вектором размера n мы будем называть одномерный массив из n чисел. Для обозначения вектора используется буква латинского алфавита со стрелочкой $\vec{x}$. Для записи вектора вместе с элементами в столбец используется следующая нотация
$$
\begin{align}
\vec{x} &= \begin{bmatrix}
        x_{1} \\
        x_{2} \\
        \vdots \\
        x_{n}
     \end{bmatrix}
\end{align}
$$
Вместо записи в столбец, веткор может быть записан в виде строки следующим образом:
1. $\vec{x} = (x_1, x_2, \ldots, x_n)$
2. $\vec{x} = \left<x_1, x_2, \ldots, x_n\right>$

Каждый элемент вектора является действительным числом. Для обозначения этого свойства используется нотация $x_i \in \mathbb{R}$, где $\mathbb{R}$ обозначает множество действительных чисел, а символ $\in$ принадлежность к данному множеству. Для обозначения того, что вектор размера n состоит из действительных чисел исопльзуется нотация $\vec{x} \in \mathbb{R^n}$.

В python мы будем использовать библиотеку NumPy для работы с векторами. Как было сказано выше, вектор размера n это одномерный массив с n числами. В NumPy мы можем создать $\vec{x} = (1, 2, 3)$ следующим образом

```python
import numpy as np

np.array([1, 2, 3])
```

    array([1, 2, 3])

### Стандартный единичный вектор (standard unit vector)
Стандартным или базисным (basis) единичным вектором называются векторы размера n, у которых один элемент равен 1, а все остальные 0. У стандартного вектора $\vec{e_j}$ элемент на позиции j равен 1, а все остальные позиции равны 0:
$$
\begin{align}
\vec{e_j} &= \begin{bmatrix}
        0 \\
        \vdots \\
        0 \\
        1 \\
        0 \\
        \vdots \\
        0
     \end{bmatrix}
\end{align}
$$
Единичные векторы в двухмерном пространстве обозначаются как $\hat{i}$ и $\hat{j}$
$$
\begin{align}
\hat{i}=\vec{e_0} &= \begin{bmatrix}
        1 \\
        0
     \end{bmatrix} &
\hat{j}=\vec{e_1} &= \begin{bmatrix}
        0 \\
        1
     \end{bmatrix}
\end{align}
$$
Единичные векторы в трехмерном пространстве обозначаются как $\hat{i}$, $\hat{j}$ и $\hat{k}$
$$
\begin{align}
\hat{i}=\vec{e_0} &= \begin{bmatrix}
        1 \\
        0 \\
        0
     \end{bmatrix} &
\hat{j}=\vec{e_1} &= \begin{bmatrix}
        0 \\
        1 \\
        0
     \end{bmatrix} &
\hat{k}=\vec{e_2} &= \begin{bmatrix}
        0 \\
        0 \\
        1
     \end{bmatrix}
\end{align}
$$
Стандартный единичный вектор имеет важное значение, которое мы увидем далее. Ниже приведен пример создания единичных векторов в трехмерном прастранстве с помощью NumPy

```python
i = np.array([1, 0, 0])
j = np.array([0, 1, 0])
k = np.array([0, 0, 1])
i, j, k
```

    (array([1, 0, 0]), array([0, 1, 0]), array([0, 0, 1]))

### Произведение вектора на скалярное значение
При произведении вектора на скалярное значение каждый элемент этого вектора умножается на скалярное значение. Например, если мы хотим скалярное значение $\alpha$ умножить на вектор $\vec{x}=(x_1, x_2, \ldots, x_n)$, то в результате получим
$$
\begin{align}
\alpha\vec{x} &= \begin{bmatrix}
        \alpha x_1 \\
        \alpha x_2 \\
        \vdots \\
        \alpha x_n
     \end{bmatrix}
\end{align}
$$
Умножение вектора на скалярное значение также называется масштабированием вектора (vector scaling), так как при этом длина вектора увеличивается (или уменьшается если $0 < \alpha < 1$ ) на соответсвтвующее скалярное значение.

Следующий код релизует умножение вектора $\vec{x}=(1, 2, 3)$ на скалярное значение 5

```python
x = np.array([1, 2, 3])
y = np.zeros_like(x)
a = 5
for i in range(len(x)):
    y[i] = x[i] * a
print(y)
```

    [ 5 10 15]

Однако, на практике писать такой код не стоит. Как было показано в разделе про векторизованные вычисления в NumPy, данный код можно реализовать значительно короче и эффективнее

```python
a * x
```

    array([ 5, 10, 15])

### Сложение и вычитание векторов (addition and substraction of vectors)
При сложении двух векторов каждый элемент первого вектора складывается с соответствующим по индексу элементом второго вектора. Т.е. если нам даны два вектора $\vec{x}=(x_1, x_2, \ldots, x_n)$ и $\vec{y}=(y_1, y_2, \ldots, y_n)$, то при их сложении мы получим следующий вектор
$$
\begin{align}
\vec{x} + \vec{y} &= \begin{bmatrix}
        x_1 + y_1 \\
        x_2 + y_2 \\
        \vdots \\
        x_n + y_n
     \end{bmatrix}
\end{align}
$$
Чтобы сложить два вектора их размер должен быть одинаковым.

Следующий пример реализует сложение векторов на python

```python
x = np.array([1, 2, 3])
y = np.array([4, 5, 6])
z = np.zeros_like(x)
for i in range(len(x)):
    z[i] = x[i] + y[i]
print(z)
```

    [5 7 9]

Реализация с помощью NumPy

```python
x + y
```

    array([5, 7, 9])

Вычитание веткоров выполняется аналогично сложению, только вместо складывания каждого элемента они вычитаются. Если даны два вектора $\vec{x}=(x_1, x_2, \ldots, x_n)$ и $\vec{y}=(y_1, y_2, \ldots, y_n)$, то при их вычитании мы получим следующий вектор
$$
\begin{align}
\vec{x} - \vec{y} &= \begin{bmatrix}
        x_1 - y_1 \\
        x_2 - y_2 \\
        \vdots \\
        x_n - y_n
     \end{bmatrix}
\end{align}
$$
Чтобы сложить два вектора их размер должен быть одинаковым. Следующий пример реализует сложение веткоров на python

```python
for i in range(len(x)):
    z[i] = x[i] - y[i]
print(z)
```

    [-3 -3 -3]

Реализация с помощью NumPy

```python
x - y
```

    array([-3, -3, -3])

### Линейная комбинация векторов (linear combination)
Допустим, что нам даны два вектора размера m: $\vec{x}, \vec{y} \in \mathbb{R^m}$ и два скалярных значения: $\alpha, \beta \in \mathbb{R}$. Линейной комбинацией векторов $\vec{x}, \vec{y}$ с коэффициентами $\alpha, \beta$ называется следующий вектор
$$
\begin{align}
\alpha \vec{x} + \beta \vec{y} &= \begin{bmatrix}
        \alpha x_1 + \beta y_1 \\
        \alpha x_2 + \beta y_2 \\
        \vdots \\
        \alpha x_n + \beta y_n
     \end{bmatrix}
\end{align}
$$
Линейная комбинация является одной из центральных понятий в линейной алгебре и многие другие понятия определяются с помощью нее. Линейная комбинация может быть обобщена на случай большего количества веткоров. Допустим у нас есть n векторов размера m: $\vec{x_1}, \vec{x_2}, \ldots, \vec{x_n} \in \mathbb{R^m}$ и n скалярных значений: $a_1, a_2, \ldots, a_n \in \mathbb{R}$. Тогда линейной комбинацией этих векторов называется вектор, полученный следующим образом
$$
\begin{align}
a_1 \vec{x_1} + a_2 \vec{x_2} + \cdots + a_n \vec{x_n} &= \begin{bmatrix}
         a_1 x_{1,1} + a_2 x_{1,2} + \cdots + a_n x_{1,n}\\
         a_1 x_{2,1} + a_2 x_{2,2} + \cdots + a_n x_{2,n}\\
         \vdots \\
         a_1 x_{m,1} + a_2 x_{m, 2} + \cdots + a_n x_{m, n}\\
     \end{bmatrix} &= \sum_{i=1}^{n} a_i \vec{x_i}
\end{align}
$$
Значение $x_{i,j}$ соответствует элементу i вектора j. Символ $\sum_{i=1}^{n}$ означает сумму элементов от 1 до n. Это математический символ соотвествующий циклу в языках программирования. Линейная комбинация двух векторов может быть реализована в python следующим образом

```python
x = np.array([1, 1, 1])
y = np.array([2, 2, 2])
a = 2
b = 1
z = np.zeros_like(x)
for i in range(len(x)):
    z[i] = a * x[i] + b * y[i]
print(z)
```

    [4 4 4]

Реализация с помощью NumPy

```python
a * x + b * y
```

    array([4, 4, 4])

### Представление векторов в виде линейной комбинации стандартных базисных векторов
Любой вектор можно представить с помощью линейной комбинации стандартных единичных векторов. Допустим у нас есть вектор $\vec{x}=(1, 2, 3)$. Этот вектор можно записать следующим обрзом:
$$
\begin{align}
\vec{x} &= \begin{bmatrix}
        1 \\
        2 \\
        3
     \end{bmatrix} &= 1 \begin{bmatrix}
        1 \\
        0 \\
        0
     \end{bmatrix} + 2 \begin{bmatrix}
        0 \\
        1 \\
        0
     \end{bmatrix} + 3 \begin{bmatrix}
        0 \\
        0 \\
        1
     \end{bmatrix} &= 1 \vec{e_1} + 2 \vec{e_2} + 3 \vec{e_3}
\end{align}
$$
В более общем виде, когда у нас есть вектор размера n $\vec{x} \in \mathbb{R}^n$, он может быть выражен с помощью стандартных единичных векторов следующим образом:
$$
\begin{align}
\vec{x} &= \begin{bmatrix}
        x_1 \\
        x_2 \\
        \vdots \\
        x_n
     \end{bmatrix} &= x_1 \begin{bmatrix}
        1 \\
        0 \\
        \vdots \\
        0
     \end{bmatrix} + x_2 \begin{bmatrix}
        0 \\
        1 \\
        \vdots \\
        0
     \end{bmatrix} + \cdots + x_n \begin{bmatrix}
        0 \\
        0 \\
        \vdots \\
        1
     \end{bmatrix} = x_1 \vec{e_1} + x_2 \vec{e_2} + \cdots + x_n \vec{e_n} = \sum_{i=1}^{n} x_i \vec{e_i}
\end{align}
$$
Такая форма представления векторов понадобится нам при определении матриц.

```python
x = np.array([1, 2, 3])
i = np.array([1, 0, 0])
j = np.array([0, 1, 0])
k = np.array([0, 0, 1])

y = x[0] * i + x[1] * j + x[2] * k
x == y
```

    array([ True,  True,  True], dtype=bool)

### Скалярное произведение векторов (dot product)
Допустим, что нам даны два вектора размера m: $\vec{x}, \vec{y} \in \mathbb{R^m}$. Скалярным или внутренним произведением этих двух векторов называется **скалярное** значение полученное следующим образом
$$
\begin{align}
\operatorname{dot(\vec{x}, \vec{y})} = \vec{x} \cdot \vec{y} = x_1 y_1 + x_2 y_2 + \cdots + x_n y_n = \sum_{i=1}^{n} x_i y_i
\end{align}
$$
Часто для обозначения скалярного произведения векторов используется нотация $\vec{x}^T\vec{y}$. Символ $T$ означает транспозицию. При применении данного оператора из вектора в столбец получается вектор в строку:
$$
\begin{align}
\vec{x}^T\vec{y} &= \begin{bmatrix}
        x_1 \\
        x_2 \\
        \vdots \\
        x_n
     \end{bmatrix}^T \begin{bmatrix}
        y_1 \\
        y_2 \\
        \vdots \\
        y_n
     \end{bmatrix} \\
     &= [x_1, x_2, \cdots, x_n] \begin{bmatrix}
        y_1 \\
        y_2 \\
        \vdots \\
        y_n
     \end{bmatrix} \\
     &= \sum_{i=1}^{n} x_i y_i
\end{align}
$$
Скалярное произведение векторов можно реализовать на python следующим образом

```python
x = np.array([1, 2, 3])
y = np.array([4, 5, 6])
z = 0
for i in range(len(x)):
    z += x[i] * y[i]
print(z)
```

    32

Реализация с помощью NumPy

```python
x.dot(y)
```

    32

### Длина вектора (vector length)
Длина, величина или норма (length, magnitude, norm) вектора размера m: $\vec{x} \in \mathbb{R^m}$ обозначается как $||\vec{x}||$ и вычисляется следующим образом:
$$
\begin{align}
||\vec{x}|| = \sqrt{x_1^2 + x_2^2 + \cdots + x_n^2} = \sqrt{\sum_{i=1}^{n} x_i^2}
\end{align}
$$
Длина вектора, определенная таким образом, также называется евклидовой длиной. В двухмерном и трехмерном пространствах данное определение соответствует теореме пифагора для нахождения расстояния до точки от начала координат. Обратите внимание, что длина вектора также может быть определена через скалярное произведение вектора на самого себя:
$$
\begin{align}
||\vec{x}|| = \sqrt{\vec{x}^T\vec{x}}
\end{align}
$$
Вычисление длины вектора можно реализовать на python следующим образом

```python
x = np.array([3, 4])
l = 0
for i in range(len(x)):
    l += x[i] ** 2
l = l ** 0.5
print(l)
```

    5.0

Для вычисления длины вектора с помощью NumPy мы можем вомпользоваться функцией `norm` из модуля `numpy.linalg`

```python
np.linalg.norm(x)
```

    5.0

### Вектор-функция (vector function)
Вектор-функция - функция, значениями которой являются векторы двух, трёх или более измерений. Аргументами функции могут быть скалярные значения и векторы. Нас будут интересовать вектор-функции, которые на вход принимают вектор и на выходе выдают другой вектор. При этом размеры входного и выходного векторов могут быть разными. Вектор функция принимающая в качестве аргумента вектор размера m и отображающая его на вектор размера n обозначается следующим образом:
$$
\begin{align}
    f\colon \mathbb R^m \to \mathbb R^n
\end{align}
$$
Примерами векторных функций могут быть функция, которая выполняет сложение или вычитание векторов, функция, которая умножает скалярное значение на вектор и т.д. В разделе про векторизованные вычисления в NumPy мы видели примеры функций, которые принимают на вход массив и применяют функцию каждому элмементу этого массива. Данные функции также являются примерами вектор функций. Например, функция `np.exp` возводит число $e$ в степень каждого элемента вектора и возвращает вектор такого же размера

```python
x = np.array([1, 2, 3])
np.exp(x)
```

    array([  2.71828183,   7.3890561 ,  20.08553692])

### Линейное отображение (linear map)
Вектор функция $L\colon \mathbb R^m \to \mathbb R^n$ называется *линейным отображением* (или линейным оператором) если для всех $\vec{x}, \vec{y} \in \mathbb{R^m}$ и $\alpha \in \mathbb{R}$ удовлетворяет следующим двум условиям
1. Отображение произведения вектора на скалярное значение равно произведению скалярного значения на отображение вектора
$$L(\alpha \vec{x}) = \alpha L(\vec{x})$$
2. Отображение сложения векторов равно сложению отображений векторов
$$L(\vec{x} + \vec{y}) = L(\vec{x}) + L(\vec{y})$$

В качестве примера линейного отображения можно рассмотреть, функцию, которая вычисляет среднее значение всех элементов вектора. Например, если взять два вектора $\vec{x}=(1, 2, 3), \vec{y}=(4, 5, 6)$, то следующий код показывает свойства этого отображения

```python
x = np.array([1, 3, 5])
y = np.array([2, 4, 6])
a = 42

print('{0} = {1}'.format((a * x).mean(), a * x.mean()))
print('{0} = {1}'.format((x + y).mean(), x.mean() + y.mean()))
```

    126.0 = 126.0
    7.0 = 7.0

Из свойств линейного отображения следует равенство: $L(\alpha \vec{x} + \beta \vec{y}) = \alpha L(\vec{x}) + \beta L(\vec{y})$. Можно обобщить данное выражение на большее количество векторов. Допустим у нас есть n векторов размера m: $\vec{x_1}, \vec{x_2}, \ldots, \vec{x_n} \in \mathbb{R^m}$ и n скалярных значений: $a_1, a_2, \ldots, a_n \in \mathbb{R}$. Тогда верно следующее выражение, которое здесь приводится без доказательства:
$$L(a_1 \vec{x_1} + a_2 \vec{x_2} + \cdots + a_n \vec{x_n}) = a_1 L(\vec{x_1}) + a_2 L(\vec{x_2}) + \cdots + a_n L(\vec{x_n})$$

Можно записать данное выражение в более сжатом виде следующим образом:
$$L(\sum_{i=1}^n a_i \vec{x_i}) = \sum_{i=1}^n a_i L(\vec{x_i})$$

# Матрица
Выше мы видели, что любой вектор может быть записан в виде линейной комбинации базисных векторов:
$$\vec{x} = \sum_{i=1}^{m} x_i \vec{e_i}$$
Поэтому линейное отображение вектора эквивалентно выражению:
$$L(\vec{x}) = L(\sum_{i=1}^{m} x_i \vec{e_i})$$
Однако мы только что видели, что линейное отображение суммы векторов равно сумме линейных отображений каждого из этих векторов. Поэтому линейное отображение вектора можно записать следующим образом:
$$L(\vec{x}) = L(\sum_{i=1}^{m} x_i \vec{e_i}) = \sum_{i=1}^{m} x_i L(\vec{e_i})$$
Если определить вектор $\vec{u_i}=L(\vec{e_i})$, то любое линейное отображение может быть полностью определено через линейную комбинацию векторов $\vec{u_i}$:
$$L(\vec{x}) = \sum_{i=1}^m x_i \vec{u_i}$$
Эти векторы ${\vec{u_1}, \vec{u_2}, \ldots, \vec{u_n}}$ можно записать в двухмерном виде следующим образом:
$$
\begin{align}
A &= (\vec{u_1}, \vec{u_2}, \ldots, \vec{u_n}) &= \begin{pmatrix}
        u_{1,1} & u_{1,2} & \cdots & u_{1,n} \\
        u_{2,1} & u_{2,2} & \cdots & u_{2,n} \\
        \vdots & \vdots & \ddots & \vdots \\
        u_{m,1} & u_{m,2} & \cdots & u_{m,n} \\
     \end{pmatrix}
\end{align}
$$
Данная двухмерная структура называется матрицей линейного отображения. Любое линейное отображение может быть применено для каждого базисного вектора и результат может быть записан в виде матрицы. Линейное отображение может быть записано с помощью матрицы следующим образом:
$$L(\vec{x}) = A\vec{x}$$
## Умножение матрицы на вектор
Выражение $A\vec{x}$ называется умножением матрицы на вектор. Так как это выражение эквивалентно линейному отображению, то умножение матрицы на вектор можно получить из определения линейного отображения:
$$
\begin{align}
A\vec{x} &= L(\vec{x}) &= x_1\vec{u_1} + x_2\vec{u_2} + \cdots + x_n\vec{u_n} &= x_1 \begin{bmatrix}
        u_{1,1} \\
        u_{2,1} \\
        \vdots \\
        u_{m,1}
     \end{bmatrix} + x_2 \begin{bmatrix}
        u_{1,2} \\
        u_{2,2} \\
        \vdots \\
        u_{m,2}
     \end{bmatrix} + \cdots + x_n \begin{bmatrix}
        u_{1,n} \\
        u_{2,n} \\
        \vdots \\
        u_{m,n}
     \end{bmatrix} &= \begin{pmatrix}
        x_1 u_{1,1} + x_2 u_{1,2} + \cdots + x_n u_{1,n} \\
        x_1 u_{2,1} + x_2 u_{2,2} + \cdots + x_n u_{2,n} \\
        \vdots \\
        x_1 u_{m,1} + x_2 u_{m,2} + \cdots + x_n u_{m,n} \\
     \end{pmatrix}
\end{align}
$$
Умножение матрицы на вектор можно реализовать на python следующим образом

```python
A = np.array([[0, -1],
              [1, 0]])
x = np.array([1, 0])
y = np.zeros_like(x)
for i in range(len(x)):
    for j in range(len(x)):
        y[i] += A[i, j] * x[j]
y
```

    array([0, 1])

В NumPy для умножения матрицы на вектор можно использовать функцию `dot`

```python
A.dot(x)
```

    array([0, 1])

## Умножение матриц
Перед тем как определить умножение двух матриц рассмотрим последовательное применение двух линейных отображений. Пусть у нас есть два линейных отображения $L_A\colon \mathbb R^k \to \mathbb R^m$ и $L_B\colon \mathbb R^n \to \mathbb R^k$. Тогда последовательное применение этих двух линейных отображений даст нам третье линейное отображение $L_C\colon \mathbb R^n \to \mathbb R^m$:
$$L_C(\vec{x}) = L_A(L_B(\vec{x}))$$
Выше мы видели, что любое линейное отображение может быть представлено в виде соответствующей матрицы. Допустим, матрица $A \in \mathbb{R}^{m \times k}$ представляет $L_A$ и матрица $B \in \mathbb{R}^{k \times n}$ представляет $L_B$. Тогда матрица $C \in \mathbb{R}^{m \times n}$, которая представляет $L_C$, вычисляется с помощью умножения двух матриц: $C=AB$. Линейное отображение $L_C$ может быть записано с помощью этих матриц следующим образом:
$$L_C(\vec{x})=C\vec{x}=(AB)\vec{x}=(A(B\vec{x}))$$
Каждый элемент матрицы $C$ может быть вычислен по следующей формуле:
$$
c_{i,j} = \sum_{k=1}^{n} a_{i,k} b_{k,j}
$$
Рассмотрим следующий пример:
$$
\begin{align}
C &= \begin{pmatrix}
        0 & -1 \\
        1 & 0
     \end{pmatrix} \times \begin{pmatrix}
        -1 & 0 \\
        0 & -1
     \end{pmatrix} &= \begin{pmatrix}
        0 * (-1) + (-1) * 0 & 0 * 0 + (-1) * (-1) \\
        1 * (-1) + 0 * 0 & -1 * 0 + 0 * (-1)
     \end{pmatrix} &= \begin{pmatrix}
        0 & 1 \\
        -1 & 0
     \end{pmatrix}
\end{align}
$$
Умножение матриц можно реализовать на python следующим образом

```python
A = np.array([[0, -1],
              [1, 0]])
B = np.array([[-1, 0],
              [0, -1]])
C = np.zeros_like(A)
for i in range(A.shape[0]):
    for j in range(B.shape[1]):
        for k in range(A.shape[1]):
            C[i, j] += A[i, k] * B[k, j]
C
```

    array([[ 0,  1],
           [-1,  0]])

В NumPy для умножения матриц можно использовать функцию `dot`

```python
A.dot(B)
```

    array([[ 0,  1],
           [-1,  0]])

04.01 Linear Algebra - definitions and main concepts.ipynb


```python
import scipy as sp
import sympy
import numpy as np
import scipy as sp
import sklearn as sk

import matplotlib as mpl
mpl.use('Agg')
import matplotlib.pylab as plt
from mpl_toolkits.mplot3d import Axes3D

%matplotlib inline

```

    /home/kang/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:8: UserWarning:
    This call to matplotlib.use() has no effect because the backend has already
    been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
    or matplotlib.backends is imported for the first time.

    The backend was *originally* set to 'module://ipykernel.pylab.backend_inline' by the following code:
      File "/home/kang/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/home/kang/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/home/kang/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py", line 16, in <module>
        app.launch_new_instance()
      File "/home/kang/anaconda3/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
        app.initialize(argv)
      File "<decorator-gen-123>", line 2, in initialize
      File "/home/kang/anaconda3/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
        return method(app, *args, **kwargs)
      File "/home/kang/anaconda3/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 462, in initialize
        self.init_gui_pylab()
      File "/home/kang/anaconda3/lib/python3.6/site-packages/ipykernel/kernelapp.py", line 403, in init_gui_pylab
        InteractiveShellApp.init_gui_pylab(self)
      File "/home/kang/anaconda3/lib/python3.6/site-packages/IPython/core/shellapp.py", line 207, in init_gui_pylab
        r = enable(key)
      File "/home/kang/anaconda3/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2930, in enable_matplotlib
        pt.activate_matplotlib(backend)
      File "/home/kang/anaconda3/lib/python3.6/site-packages/IPython/core/pylabtools.py", line 307, in activate_matplotlib
        matplotlib.pyplot.switch_backend(backend)
      File "/home/kang/anaconda3/lib/python3.6/site-packages/matplotlib/pyplot.py", line 229, in switch_backend
        matplotlib.use(newbackend, warn=False, force=True)
      File "/home/kang/anaconda3/lib/python3.6/site-packages/matplotlib/__init__.py", line 1305, in use
        reload(sys.modules['matplotlib.backends'])
      File "/home/kang/anaconda3/lib/python3.6/importlib/__init__.py", line 166, in reload
        _bootstrap._exec(spec, module)
      File "/home/kang/anaconda3/lib/python3.6/site-packages/matplotlib/backends/__init__.py", line 14, in <module>
        line for line in traceback.format_stack()

```python
def f2p(x) :
    return -400.0 * x[0] * (-x[0] ** 2 + x[1]) + 2*x[0] - 2
```

```python
x
```

```python
result = sp.optimize.minimize(f2, (-2, -2), jac = f2p)
print(result)
```

```python
x, y = sympy.symbols(' x y ')
f2 = (1 - x) ** 2 + 100 * (y - x ** 2) ** 2
f2
```

    (-x + 1)**2 + 100*(-x**2 + y)**2

```python
sympy.diff(f2, x )
```

    -400*x*(-x**2 + y) + 2*x - 2

```python
f2 =
```

```python
def f2(x) :
    return (1 - x[0]) ** 2 + 100.0 * (x [1] - x [0] ** 2) ** 2
```

```python
def f2p(x) :
    return -400 * x[0] * ( -x[0] ** 2 + x[1]) + 2 * x[0] - 2
```

```python
def f2g(x):
    """gradient of f2(x)"""
    return np.array((2.0 * (x[0] - 1) - 400.0 * x[0] * (x[1] - x[0]**2), 200.0 * (x[1] - x[0]**2)))
```

```python
result = sp.optimize.minimize(f2, (3, 3), jac = f2g)
print(result)
x0 = result['x']
x0
```

          fun: 7.880245049711645e-20
     hess_inv: array([[ 0.48216528,  0.9656648 ],
           [ 0.9656648 ,  1.93889528]])
          jac: array([ -1.11665241e-08,   5.53674884e-09])
      message: 'Optimization terminated successfully.'
         nfev: 58
          nit: 43
         njev: 58
       status: 0
      success: True
            x: array([ 1.,  1.])

    array([ 1.,  1.])

```python
def f1logs(x) :
    return np.log(x[0]) + np.log(x[1])

def eq_constraint(x) :
    return x[0] + x[1] - 1

sp.optimize.fmin_slsqp(f1logs, np.array([(1, 1)]) , eqcons = [eq_constraint])
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -1.3862943611198901
                Iterations: 2
                Function evaluations: 8
                Gradient evaluations: 2

    array([ 0.5,  0.5])

```python

```

```python
def f2(x) :
    return np.sqrt((x[0] - 3))
```