# Potential flow

## Potencial functions

**Uniform flow**:

$w = U\exp{(-i\alpha)}z$

**Line source**:

$w = \dfrac{\dot{Q}}{2\pi}\ln z$

## Velocity fields

$\dfrac{dw}{dz} = u + iv$

**Uniform flow:**

$\dfrac{dw}{dz} = U\left(\cos{\alpha}-i\sin{\alpha}\right)$

**Line source:**

$\dfrac{dw}{dz} = \dfrac{\dot{Q}}{2\pi}\dfrac{1}{z}$

## Combine potentials

Consider a potential flow with a sink at $(0,0)$ and a source at $(r,0)$, under the influence of a horizontal uniform flow going from left to right ($\alpha = \pi$). The potential that describes that system is given by:

$w = w_{\rm uniform} + w_{\rm source} + w_{\rm sink}$

Hence,

$w = U\exp{(-i\alpha)}z + \dfrac{\dot{Q_{\rm in}}}{2\pi}\ln (z-r) - \dfrac{\dot{Q_{\rm out}}}{2\pi}\ln z$

$w = -Uz + \dfrac{\dot{Q_{\rm in}}}{2\pi}\ln (z-r) - \dfrac{\dot{Q_{\rm out}}}{2\pi}\ln z$

The uniform flow component will represent the regional flow, which can be described using Darcy's law as $U=-K\dot{I}$, where $K$ is the hydraulic conductivity (m/s) and $\dot{I}$ is the water table gradient (m/m). The source and sink wells represent injection and extraction points, characterized by flow rates per unit depth $\dot{Q_{\rm in}}$ and $\dot{Q_{\rm out}}$ (m²/s). For an aquifer of depth $H$, the volumetric flow rate $Q$ is $\dot{Q}=Q/H$. Now, the extraction rate can be expressed as a proportion to the injection rate as $Q_{\rm out} = fQ_{\rm in}$. With all this replacements, the potential function can be rewritten as:

$w = K\dot{I}z + \dfrac{Q_{\rm in}}{2\pi H} \left( \ln(z-r) - f \ln z \right)$

Notice that in our case, the gradient $\dot{I}$ must be negative so the regional flow goes indeed from right to left. To avoid confusion, let's rewrite it in terms of the abs value of the gradient $I$:

$w = -KIz + \dfrac{Q_{\rm in}}{2\pi H} \left( \ln(z-r) - f \ln z \right)$

## Velocity field

$\dfrac{dw}{dz} = \dfrac{d}{dz}\left(-KIz + \dfrac{Q_{\rm in}}{2\pi H} \left( \ln(z-r) - f \ln z \right)\right)$

$\dfrac{dw}{dz} = -KI + \dfrac{Q_{\rm in}}{2\pi H} \dfrac{d}{dz}\left( \ln(z-r) - f \ln z \right)$

$\dfrac{dw}{dz} = -KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{z-r} - \dfrac{f}{z} \right)$

## Velocity at the midpoint

We'll pick the velocity in the midpoint between the extraction and injection points as the characteristic velocity of the system. This means that we want to evaluate the velocity field at $z = r/2 + 0i$

\begin{equation}
\begin{array}{rl}
    \dfrac{dw}{dz}\bigg\rvert_{z=r/2+0i} =& -KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{r/2-r} - \dfrac{f}{r/2} \right)\\
    =& -KI - \dfrac{Q_{\rm in}}{\pi H} \left(\dfrac{1+f}{r}\right)
\end{array}
\end{equation}

Notice that this velocity can be split into the contribution of the regional flow and the contribution from the wells:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -KI \\
    w'_{\rm wells} =& -\dfrac{1}{\pi}\dfrac{Q_{\rm in} (1+f)}{r H}
\end{array}
\end{equation}

### Non-dimensional flow number ($\mathcal{F}_L$)

A comparison between the component in the velocity due regional flow and the component due the wells as a ratio between them two.

\begin{equation}
\begin{array}{rl}
\mathcal{F}_L =& \dfrac{\text{Velocity due regional flow}}{\text{Velocity due well couple}} \\
=& \dfrac{w'_{\rm regional}}{w'_{\rm wells}} \\
=& \dfrac{-KI}{-\dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1+f}{r}\right)} \\
=& \dfrac{\pi K I H r}{Q_{in}(1+f)}
\end{array}
\end{equation}

If $\mathcal{F}_L \to 0$, the flow induced by the wells rule the system, whereas if $\mathcal{F}_L \to \infty$, the regional flow rule the system.

## Minimum velocity along a streamline

The minimum velocity is found at:

\begin{equation}
\begin{array}{rl}
    \dfrac{d}{dx}\dfrac{dw}{dz}\bigg\rvert_{z=x+0i} =& 0 \\
    \dfrac{d}{dx}\left(-KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{x-r} - \dfrac{f}{x} \right)\right) =& 0\\
    \dfrac{Q_{\rm in}}{2 \pi H} \dfrac{d}{dx}\left(\dfrac{1}{x-r} - \dfrac{f}{x} \right) =& 0\\
    \dfrac{Q_{\rm in}}{2 \pi H} \left(\dfrac{f}{x^2} - \dfrac{1}{\left(x-r\right)^2}\right) =& 0\\
    \\
    \dfrac{f}{x^2} =& \dfrac{1}{\left(x-r\right)^2}\\
    x^2 =& f \left( x-r \right) ^2\\
    x =& \dfrac{\sqrt{f}}{\sqrt{f}\pm1}r
    \\
    x < r \Rightarrow x =& \dfrac{\sqrt{f}}{\sqrt{f}+1}r
\end{array}
\end{equation}

Defining $m = \dfrac{\sqrt{f}}{\sqrt{f}+1}$, evaluating at $z = mr + 0i$ gives the minimum velocity.

\begin{equation}
\begin{array}{rl}
    \dfrac{dw}{dz}\bigg\rvert_{z=mr+0i} =& -KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{mr-r} - \dfrac{f}{mr} \right)\\
    =& -KI + \dfrac{Q_{\rm in}}{2 \pi H} \left(\dfrac{m-fm+f}{m(m-1)}\right)\\
    =& -KI - \dfrac{Q_{\rm in}}{2 \pi H} \left(\dfrac{\left(\sqrt{f}+f\right)\left(\sqrt{f}+1\right)}{\sqrt{f}}\right)\\
    =& -KI - \dfrac{Q_{\rm in}}{2 \pi H} \left( \left( 1+ \sqrt{f}\right)\left( 1 + \dfrac{\sqrt{f}}{f} \right) \right)\\
    =& -KI - \dfrac{ Q_{\rm in}\left( 1 + \tfrac{\sqrt{f}}{f} \right) } {\pi H r}
\end{array}\\
\end{equation}

This velocity can also be split into the contribution of the regional flow and the contribution from the wells:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -KI \\
    w'_{\rm wells} =& -\dfrac{Q_{\rm in} }{\pi r H}\left(1+\tfrac{\sqrt{f}}{f}\right)
\end{array}
\end{equation}

## Mean velocity along a streamline

Consider the streamline that connects the sink and source points. It cannot be measured at those points because those are poles in the system, hence, some separation $\delta$ from those points has to be considered:  the well radius should be the straighforward election so we'll adopt it and assume that both injection and extraction wells have the same diameter.

Now, with $z = x+iy$, the streamline exists at $y=0$ and $\delta<x<r-\delta$, hence, the mean velocity can be derived as follows:

\begin{equation}
\begin{array}{rl}
\bar{u} =& \dfrac{1}{r-2\delta} \int_{\delta}^{r-\delta}\dfrac{dw}{dz}dx\\
=& \dfrac{1}{r-2\delta} \int_{\delta}^{r-\delta} \left(-KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{z-r} - \dfrac{f}{z} \right)\right)dx\\
=& -KI + \dfrac{Q_{\rm in}}{2\pi (r-2\delta) H}\int_{\delta}^{r-\delta}\left(\dfrac{1}{x-r} - \dfrac{f}{x} \right)dx\\
=& -KI + \dfrac{Q_{\rm in}}{2\pi H}\left(\ln({x-r}) - f\ln{x} \right)\bigg\rvert_{x=\delta}^{x=r-\delta}\\
=& -KI + \dfrac{Q_{\rm in}}{2\pi H}\left[ (1+f)(\ln(\delta) -\ln(r-\delta)  \right]\\
=& -KI - \dfrac{Q_{\rm in}(1+f)}{2\pi H}\ln\left(\dfrac{r-\delta}{\delta}\right)\\
\end{array}
\end{equation}

Notice that this velocity can be split into the contribution of the regional flow and the contribution from the wells:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -KI \\
    w'_{\rm wells} =& -\dfrac{Q_{\rm in}(1+f)}{2\pi H}\ln\left(\dfrac{r-\delta}{\delta}\right)
\end{array}
\end{equation}

### Non-dimensional flow number ($\mathcal{F}_L$)

A comparison between the component in the velocity due regional flow and the component due the wells as a ratio between them two.

\begin{equation}
\begin{array}{rl}
\mathcal{F}_L =& \dfrac{\text{Velocity due regional flow}}{\text{Velocity due well couple}} \\
=& \dfrac{w'_{\rm regional}}{w'_{\rm wells}} \\
=& \dfrac{-KI}{-\dfrac{Q_{\rm in}(1+f)}{2\pi H}\ln\left(\dfrac{r-\delta}{\delta}\right)} \\
=& \dfrac{2 \pi K I H (r-2\delta)}{Q_{in}(1+f)\ln\left(\frac{r-\delta}{\delta}\right)}
\end{array}
\end{equation}

If $\mathcal{F}_L \to 0$, the flow induced by the wells rule the system, whereas if $\mathcal{F}_L \to \infty$, the regional flow rule the system.

## Mass flow through an infinite line

The previous choice of characteristic velocity seems arbitrary in the sense it overestimates the effect of the potential induced by the wells on the system. This is because the streamline along the x-axis has the highest velocity magnitude (the sharpest potential gradient), if we were to pick a point elsewhere, the veloicty form the wells would be less. But in the same sense, for $f=1$, this point represents the lowest velocity along that streamline.

To avoid this apparent bias, let's consider the totality of the flow that crosses an infinite vertical line at $z=r/2+iy$. This won't give us a useful insight on the regional flow as that mass flow will diverge, but it will give us a point of comparison to the wells contribution to the total mass balance in the system.

So, the velocity field along that vertical line is

\begin{equation}
\begin{array}{rl}
\dfrac{dw}{dz}\bigg\rvert_{z=\tfrac{r}{2}+iy} =& -KI - \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{z-r} - \dfrac{f}{z} \right)\bigg\rvert_{z=\tfrac{r}{2}+iy}\\
=& - KI + \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{1}{\tfrac{r}{2}+iy-r} - \dfrac{f}{\tfrac{r}{2}+iy} \right)\\
=& -KI - \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{\tfrac{r}{2}(1+f) + iy(1-f)}{\tfrac{r^2}{4}-y^2}\right)
\end{array}
\end{equation}

The mass flow through that vertical line will require only the horizontal component of the velocity field $u$ at that point, meaning that we are only intereste in the real component of the function above:

$u = -KI - \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{\tfrac{r}{2}(1+f)}{\tfrac{r^2}{4}-y^2}\right) = -KI - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right) $

The mass flow will be described as:

\begin{equation}
\begin{array}{rl}
    \int_{-\infty}^{\infty} u \, dy =& \int_{-\infty}^{\infty}\left[ -KI - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right)\right] \, dy \\
    =& -\underbrace{\int_{-\infty}^{\infty}KI \, dy}_{\text{Diverges}} - \underbrace{\dfrac{Q_{\rm in}r(1+f)}{4\pi H} \int_{-\infty}^{\infty}\left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right) \, dy}_{\text{Converges}}\\
    =& -\int_{-\infty}^{\infty}KI \, dy - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left[ \dfrac{2}{r} \tan^{-1}{\left(\dfrac{2y}{r}\right)}\right]\bigg\rvert_{y \to -\infty}^{y \to \infty}\\
    =& -\int_{-\infty}^{\infty}KI \, dy - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left[ \dfrac{2\pi}{r} \right] \\
    =& -\int_{-\infty}^{\infty}KI \, dy - \dfrac{Q_{\rm in}(1+f)}{2 H}
\end{array}
\end{equation}

This end term, we can divide it by the lenght of the line $l_r$ in order to obtain a velocity, hence, the contribution from the wells are:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -\dfrac{1}{l_r}\int_{-\infty}^{\infty}KI \, dy \\
    w'_{\rm wells} =& -\dfrac{1}{2l_r}\dfrac{Q_{\rm in}(1+f)}{rH}
\end{array}
\end{equation}

### Non-dimensional flow number ($\mathcal{F}_L$)

In this case, this comparison don't make much sense as one of the integrals diverged.

\begin{equation}
\begin{array}{rl}
\mathcal{F}_L =& \dfrac{\text{Velocity due regional flow}}{\text{Velocity due well couple}} \\
=& \dfrac{w'_{\rm regional}}{w'_{\rm wells}} \\
=& \dfrac{-\dfrac{1}{l_r}\int_{-\infty}^{\infty}KI \, dy }{-\dfrac{1}{2l_r}\dfrac{Q_{\rm in}(1+f)}{rH}} \\
=& \dfrac{2rH \int_{-\infty}^{\infty}KI \, dy}{Q_{in}(1+f)}
\end{array}
\end{equation}

We could infer that in this approach, the regional flow will always dwarf the potential induced by the pair of wells. But this is not useful for our purpose :(

## Mass flow through an finite line

In order to obtain a convergent integral for the regional component, let's pick a vertical line that spans from $r/2(1-i)$ to $r/2(1+i)$, that way, the mass flow will be described as:

\begin{equation}
\begin{array}{rl}
    \int_{-r/2}^{r/2} u \, dy =& \int_{-r/2}^{r/2}\left[ -KI - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right)\right] \, dy \\
    =& \int_{-r/2}^{r/2}KI \, dy - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \int_{-r/2}^{r/2}\left(\dfrac{1}{\tfrac{r^2}{4}-y^2}\right) \, dy\\
    =& -KIy\bigg\rvert_{y = -r/2}^{y = r/2} - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left[ \dfrac{2}{r} \tan^{-1}{\left(\dfrac{2y}{r}\right)}\right]\bigg\rvert_{y = -r/2}^{y = r/2}\\
    =& -KIr - \dfrac{Q_{\rm in}r(1+f)}{4\pi H} \left[ \dfrac{\pi}{r} \right] \\
    =& -KIr - \dfrac{Q_{\rm in}(1+f)}{4 H}
\end{array}
\end{equation}

Jusy as before, this end term can be divided by $r$ in order to obtain a velocity, hence, both contributions are:

\begin{equation}
\begin{array}{rl}
    w'_{\rm regional} =& -KI \\
    w'_{\rm wells} =& -\dfrac{1}{4}\dfrac{Q_{\rm in}(1+f)}{rH}
\end{array}
\end{equation}

### Non-dimensional flow number (2) ($\mathcal{F}_L$)

From this analysis we could also draw a comparison between the two contributors of the system:

\begin{equation}
\begin{array}{rl}
\mathcal{F}_L =& \dfrac{\text{Velocity due regional flow}}{\text{Velocity due well couple}}\\
=& \dfrac{w'_{\rm regional}}{w'_{wells}}\\
=& \dfrac{-KI}{-\dfrac{Q_{\rm in}}{4 H} \left(\dfrac{1+f}{r}\right)} \\
=& \dfrac{4 K I H r}{Q_{in}(1+f)}
\end{array}
\end{equation}

There is not much of a difference in this "more conservative" approach, and it holds that if $\mathcal{F}_L \to 0$, the flow induced by the wells rule the system, whereas if $\mathcal{F}_L \to \infty$, the regional flow rule the system. We well conserve the first non-dimensional flow number definition, i.e.,

\begin{equation}
\mathcal{F}_L = \dfrac{4 K I H r}{Q_{in}(1+f)}
\end{equation}

## Characteristic velocity ($q_c$)

We found out before an expression for the velocity at the midpoint between the well couple. This will be used as our characteristic velocity, which can be rewritten in terms of the non-dimensional flow number:

\begin{equation}
\begin{array}{rl}
    q_c =& \dfrac{1}{r}\int_{-r/2}^{r/2}\dfrac{dw}{dz}dy\\
    =& -KI - \dfrac{Q_{\rm in}(1+f)}{4 r H}\\
    =& -KI \left( 1 + \dfrac{1}{\mathcal{F}_L} \right)\\
    =& - KI \left( \dfrac{\mathcal{F}_L}{\mathcal{F}_L + 1} \right)
\end{array}
\end{equation}

The actual pore-water velocity $u_c$ is found by dividing by the porosity $\theta$

\begin{equation}
\begin{array}{rl}
    u_c =& \dfrac{q_c}{\theta}\\
    =& - \dfrac{KI}{\theta} \left( \dfrac{\mathcal{F}_L}{\mathcal{F}_L + 1} \right)
\end{array}
\end{equation}

## Flow time scale ($\tau$)
Picking the half-distance between the two wells as our characteristic lenght $r/2$, a characteristic time $\tau$ can be derived from $u_c$

\begin{equation}
\begin{array}{rl}
    \tau =& \dfrac{r}{|u_c|}\\
    =& \dfrac{r\theta}{KI} \left( \dfrac{\mathcal{F}_L + 1}{\mathcal{F}_L} \right)\\
\end{array}
\end{equation}

## Decay rate and timescale comparison

Considering a tracer $C$ that is injected in the source point. This tracer follows a first-order decay reaction with a rate $\lambda$.

$\dfrac{dC}{dt} = -\lambda C$

With an initial condition $C(0) = C_0$, the concentration of that tracer as a function of space and time is,

$C(t) = C_{0}\exp{-\lambda t}$

$C_\tau = C_0 \exp{-\lambda \tau} =  C_0 \exp{\left(-\lambda\dfrac{r}{|u_c|}\right)}$

Hence, for our characteristic time ($t=\tau$), the relative concentration of that tracer will be:

$C_\tau = C_0\exp{ \left( - \lambda \dfrac{r \theta (\mathcal{F}_L + 1)}{KI \mathcal{F}_L} \right) }$

## Attachment rate

Considering a tracer $C$ that is injected in the source point. This tracer follows a first-order decay reaction with a rate $\lambda$.

\begin{equation}
\begin{array}{rl}
    \dfrac{dC}{dt} =& -k_{\rm att} C
\end{array}
\end{equation}

\begin{equation}
\begin{array}{rl}
    k_{\rm att} =& \dfrac{3(1-\theta)}{2d}\alpha\eta_0 |u_c|\\
\end{array}
\end{equation}

- $d$ : collector diameter (soil grain size)
- $\alpha$ : collision/attachment efficiency
- $\eta_0$ : collector efficiency

With an initial condition $C(0) = C_0$, the concentration of that tracer as a function of space and time is,

\begin{equation}
\begin{array}{rl}
    C(t) =& C_{0}\exp{(-k_{\rm att} t)}\\
    C_\tau =& C_{0}\exp{(-k_{\rm att} \tau)}\\
     =& C_0 \exp\left(-\dfrac{3(1-\theta)}{2d}\alpha\eta_0 |u_c| \dfrac{r}{\theta |u_c|}\right)\\
     =& C_0 \exp\left(-\dfrac{3(1-\theta)r}{2d\theta}\alpha\eta_0\right)\\
\end{array}
\end{equation}

## Dilution effect

Assuming that perfect mixing has between the injection point and the distance between the wells, we could calculate a relative concentration that takes into account this mass balance. Notice that this analysis cannot be done directly from our previous assumptions because there is no mixing in potential flow. However, we can keep the characteristic velocity.

\begin{equation}
\begin{array}{rl}
    C_0 u_c \Delta y \Delta z \theta=& C_{\rm in} Q_{\rm in} + C_{\rm reg} Q_{\rm reg}\\
    C_0 u_c \Delta y \Delta z \theta=& C_{\rm in} Q_{\rm in}\\
    C_0 =& \dfrac{Q_{\rm in}}{u_c \Delta y \Delta z \theta} C_{in} \\
    =& \dfrac{Q_{\rm in}}{u_c \Delta y \Delta z \theta} C_{\rm in}\\
    =& \dfrac{Q_{\rm in}}{- \dfrac{KI}{\theta} \left( \dfrac{\mathcal{F}_L}{\mathcal{F}_L + 1} \right) \Delta y \Delta z \theta} C_{\rm in}\\
    =& \dfrac{Q_{\rm in}(\mathcal{F}_L + 1)}{KI \mathcal{F}_L \Delta y \Delta z} C_{\rm in}
\end{array}
\end{equation}

$\Delta y$ and $\Delta z$ are just lenghts around the arbitrary control volume. For our 2D case, $\Delta z = H$ and $\Delta y = $ size of the element in PFLOTRAN

$C_0 =
\dfrac{Q_{\rm in}(\mathcal{F}_L + 1)}{KI \mathcal{F}_L \Delta y H} C_{\rm in}$

### Which process is more dominant?

The effects of each process is more enhanced dependind of the system. For example, if $I$ is big, our characteristic time will be small, then, decay won't be as important as dilution could be in the aquifer. We are interested in finding which set of parameters maximize $C_\tau$, or minimize the log-reductions of the tracer concentration.

\begin{equation}
\begin{array}{rl}
C_\tau =& C_0 \exp{\left(-\left(\lambda + k_{\rm att}\right)\dfrac{r}{|u_c|}\right)}\\
C_\tau =& \dfrac{C_{\rm in}Q_{\rm in}}{u_c \Delta y H \theta} \exp{\left(-\left(\lambda + k_{\rm att}\right)\dfrac{r}{|u_c|}\right)}
\end{array}
\end{equation}

# Potential flow visualization

## Well flow is dominant

\begin{equation}
\begin{array}{rl}
\dfrac{dw}{dz} =& -KI - \dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{f}{z} - \dfrac{1}{z-r}\right)\\
\\
\mathcal{F}_L =& \dfrac{K I}{\dfrac{Q_{\rm in}}{2\pi H} \left(\dfrac{f}{z} - \dfrac{1}{z-r}\right)}
\end{array}
\end{equation}

```python
%reset -f
import numpy as np
import matplotlib.pyplot as plt
import matplotlib

def ZFix(x,y):
    XX,YY = np.meshgrid(X,Y)
    return XX + (YY*1j)

def regionalFlow():
    return -K*I

def wellFlow(z):
    w = -(QIN/(2*PI*H))*(F/z - 1/(z-R))
    wAbs = np.abs(w)
    return (w, w.real, -w.imag, wAbs)

def totalFlow(z):
    reg  = regionalFlow()
    well = wellFlow(z)
    tot  = reg + well[0]
    totAbs = np.abs(tot)
    return (tot, tot.real,-tot.imag,totAbs)

def flowNumber(z):
    reg  = regionalFlow()
    well = wellFlow(z)[3]
    FL = np.abs(reg)/well
    return FL
```

```python
''' GLOBAL CONSTANTS '''
PI = 3.141592
K = 1.0E-2
#I = 6.6E-14
I = 1.0E-10
R = 40.
F = 1.5
QIN = 1.0/86400.
H = 1
#H = 20.

POINT_OUT = {"x":0,
             "y":0,
             "c":"r",
             "s":100,
             "linewidths":1,
             "edgecolors":"w",
             "label":"Extraction"}

POINT_QIN = {"x":R,
             "y":0,
             "c":"b",
             "s":80,
             "linewidths":1,
             "edgecolors":"w",
             "label":"Injection"}

X = np.linspace(-50,150,151)
Y = np.linspace(-100,100,33)
Z = ZFix(X,Y)
_,U,V,C = totalFlow(Z)
FL = flowNumber(Z)

fig, axs = plt.subplots(2,1,figsize=(12,10))
ax = axs[0]
st = ax.streamplot(X,Y,U,V,color="k",density=[0.8, 0.8])
fi = ax.pcolormesh(X,Y,C,shading='auto',vmin=0,vmax=5.0E-7,cmap="cool")
pi = ax.scatter(**POINT_QIN,zorder=3)
po = ax.scatter(**POINT_OUT,zorder=3)

cbar = ax.figure.colorbar(fi,ax=ax,orientation="vertical")
cbar.set_label(r'$\bf{|U|}$')

ax.legend(loc='lower right')

ax = axs[1]
fi = ax.pcolormesh(X,Y,np.log10(FL),shading='auto')
pi = ax.scatter(**POINT_QIN)
po = ax.scatter(**POINT_OUT)

ax.xaxis.set_tick_params(which="both",labelbottom=False,bottom=False)
ax.yaxis.set_tick_params(which="both",labelleft=False,left=False)
cbar = ax.figure.colorbar(fi,ax=ax,orientation="vertical")
ax.legend(loc='lower right')

plt.show()
```

```python
X = np.linspace(0.01,R-0.01,151)
Y = np.array([0.])
Z = ZFix(X,Y)
_,U,V,C = totalFlow(Z)
FL = flowNumber(Z)
fig, axs = plt.subplots(2,1,figsize=(8,6),sharex=True,\
                        gridspec_kw={"height_ratios":[3,1],"hspace":0})
ax = axs[0]
li = ax.plot(X,C[0],lw=5)
ax.set(xlim=[0,R],ylim=[0,1.0E-6],ylabel=r"$\bf{|U|}$ [m/s]")

ax = axs[1]
li = ax.plot(X,np.log10(FL[0]),lw=3,c="gray")
ax.set(ylim=[-7.5,-4.5],xlabel=r"$\bf{X}$ [m]",ylabel=r"$\bf{\log(\mathcal{F}_L)}$")

plt.show()
```

## Regional flow is dominant

```python
%reset -f
import numpy as np
import matplotlib.pyplot as plt
import matplotlib

def ZFix(x,y):
    XX,YY = np.meshgrid(X,Y)
    return XX + (YY*1j)

def regionalFlow():
    return -K*I

def wellFlow(z):
    w = -(QIN/(2*PI*H))*(F/z - 1/(z-R))
    wAbs = np.abs(w)
    return (w, w.real, -w.imag, wAbs)

def totalFlow(z):
    reg  = regionalFlow()
    well = wellFlow(z)
    tot  = reg + well[0]
    totAbs = np.abs(tot)
    return (tot, tot.real,-tot.imag,totAbs)

def flowNumber(z):
    reg  = regionalFlow()
    well = wellFlow(z)[3]
    FL = np.abs(reg)/well
    return FL
```

```python
''' GLOBAL CONSTANTS '''
PI = 3.141592
K = 1.0E-2
I = 6.6E-4
R = 40.
F = 10
QIN = 1.0/86400.
H = 20.

POINT_OUT = {"x":0,
             "y":0,
             "c":"r",
             "s":100,
             "linewidths":1,
             "edgecolors":"w",
             "label":"Extraction"}

POINT_QIN = {"x":R,
             "y":0,
             "c":"b",
             "s":80,
             "linewidths":1,
             "edgecolors":"w",
             "label":"Injection"}

X = np.linspace(-50,150,151)
Y = np.linspace(-100,100,33)
Z = ZFix(X,Y)
_,U,V,C = totalFlow(Z)
FL = flowNumber(Z)

fig, axs = plt.subplots(2,1,figsize=(12,10))
ax = axs[0]
st = ax.streamplot(X,Y,U,V,color="k",density=[0.8, 0.8])
fi = ax.pcolormesh(X,Y,C,shading='auto',vmin=0,vmax=2.0E-5,cmap="cool")
pi = ax.scatter(**POINT_QIN,zorder=3)
po = ax.scatter(**POINT_OUT,zorder=3)

cbar = ax.figure.colorbar(fi,ax=ax,orientation="vertical")
cbar.set_label(r'$\bf{|U|}$')

ax.legend(loc='lower right')

ax = axs[1]
fi = ax.pcolormesh(X,Y,np.log10(FL),shading='auto')
pi = ax.scatter(**POINT_QIN)
po = ax.scatter(**POINT_OUT)

ax.xaxis.set_tick_params(which="both",labelbottom=False,bottom=False)
ax.yaxis.set_tick_params(which="both",labelleft=False,left=False)
cbar = ax.figure.colorbar(fi,ax=ax,orientation="vertical")
ax.legend(loc='lower right')

plt.show()
```

```python
X = np.linspace(0.01,R-0.01,151)
Y = np.array([0.])
Z = ZFix(X,Y)
_,U,V,C = totalFlow(Z)
FL = flowNumber(Z)
fig, axs = plt.subplots(2,1,figsize=(8,6),sharex=True,\
                        gridspec_kw={"height_ratios":[3,1],"hspace":0})
ax = axs[0]
li = ax.plot(X,C[0],lw=5)
ax.set(xlim=[0,R],ylim=[0,1.0E-5],ylabel=r"$\bf{|U|}$ [m/s]")

ax = axs[1]
li = ax.plot(X,np.log10(FL[0]),lw=3,c="gray")
ax.set(xlabel=r"$\bf{X}$ [m]",ylabel=r"$\bf{\log(\mathcal{F}_L)}$")
ax.yaxis.set_tick_params(which="both",labelright=True,right=True,labelleft=False,left=False)

plt.show()
```


# Maximising the volume of a box

Let us consider a sheet of metal of dimensions $l\times L$

We can cut in to the sheet of metal a distance $x$ to create folds.

The volume of a box with these folds will be given by:

$V(x) = (l - 2x)\times (L-2x) \times x$

In this notebook we will use calculus (the study of continuous change) and `sympy` to identify the size of the cut to give the biggest volume.

## Defining a function

To help with writing our code, we will start by defining a python function for our volume.

```python
import sympy as sym
sym.init_printing()
x, l, L = sym.symbols("x, l, L")

def V(x=x, l=l, L=L):
    """
    Return the volume of a box as described.
    """
    return (l - 2 * x) * (L - 2 * x) * x
```

```python
V()
```

We can use this function to get our value as a function of $L$ and $l$:

```python
V(x=2)
```

Or we can pass values to all our variables and obtain a given volume:

```python
V(2, l=12, L=14)
```

### Exercises

- Define the mathematical function $m x ^ 2 - w$

## Plotting our function

Let us start by looking at the volume as a function of $x$ for the following values of $l=20$ and $L=38$:

```python
%matplotlib inline
```

```python
sym.plot(V(x,l=20, L=38), (x, 0, 10));  # We only consider x < min(l, L) / 2
```

We see that our function has one stationary points (where the graph is flat).

### Exercises

- Obtain a plot of $V(x)$ for $0\leq x \leq 20$
- Obtain a plot of the function $f(x) = x ^ 2$
- Obtain a plot of the function $f(x) = 1 / x$

## Finding stationary points

These stationary points correspond to places where the derivative of the function is 0:

$$
\frac{dV}{dx}=0
$$

Let us find the $\frac{dV}{dx}$ using `sympy`:

```python
first_derivative = V().diff(x)
first_derivative
```

Let us simplify our output:

```python
first_derivative = first_derivative.simplify()
first_derivative
```

Now to find the solutions to the equation:

$$\frac{dV}{dx}=0$$

```python
stationary_points = sym.solveset(first_derivative, x)
stationary_points
```

### Exercises

- Find the stationary points of $f(x)=x^2$
- Find the stationary points of $f(x)=mx^2-w$

## Qualifying stationary points

As we can see in our graph, one of our stationary points is a maximum and the other a minumum. These can be quantified by looking at the second derivative:

- If the second derivative at a stationary point is **positive** then the stationary point is a **local minima**;
- If the second derivative at a stationary point is **negative** then the stationary point is a **local maxima**.

Let us compute the second derivative using `sympy`:

```python
second_derivative = V().diff(x, 2)
second_derivative
```

```python
stationary_points
```

```python
second_derivative_values = [(sol, second_derivative.subs({x: sol})) for sol in stationary_points]
second_derivative_values
```

We can see that the first solution gives a negative second derivative thus it's a **local** maximum (as we saw in our plot).

```python
optimal_x = second_derivative_values[0][0]
optimal_x
```

We can compute the actual value for the running example:

```python
particular_values = {"l": 20, "L": 38}
particular_optimal_x = optimal_x.subs(particular_values)
float(particular_optimal_x), float(V(particular_optimal_x, **particular_values))
```

### Exercises

- Qualify the stationary points of $f(x)=x^2$
- Qualify the stationary points of $f(x)=mx^2-w$


# Local Minima for a function of Two Variables Using Optimality Conditions

Find a local minimum point for the funtion:
$$ f(x) = x_1 + \frac{(4\times 10^6)}{x_1 x_2} + 250x_2$$

See reference for details.

## Solution

```python
import sympy as sp
import numpy as np
from scipy import linalg
from symopt import DiffNotation
```

```python
x1, x2 = sp.symbols(['x1', 'x2'])
```

```python
fx = x1 + (4*10**6) / (x1*x2) + 250*x2
fx
```

$\displaystyle x_{1} + 250 x_{2} + \frac{4000000}{x_{1} x_{2}}$

```python
# Necessary conditions
d1 = DiffNotation(fx, x1, x2)
grad_f = d1.gradient()
grad_f
```

$\displaystyle \left[\begin{matrix}1 - \frac{4000000}{x_{1}^{2} x_{2}}\\250 - \frac{4000000}{x_{1} x_{2}^{2}}\end{matrix}\right]$

```python
# The solution of the system gives the stationary points of the function
sp.nonlinsolve(grad_f, (x1, x2))
```

$\displaystyle \left\{\left( 1000, \  4\right), \left( -500 - 500 \sqrt{3} i, \  -2 - 2 \sqrt{3} i\right), \left( -500 + 500 \sqrt{3} i, \  -2 + 2 \sqrt{3} i\right)\right\}$

```python
# The Hessian of fx is a suficient condition of minimum if it is defined positive
hess_f = d1.hessian()
hess_f
```

$\displaystyle \left[\begin{matrix}\frac{8000000}{x_{1}^{3} x_{2}} & \frac{4000000}{x_{1}^{2} x_{2}^{2}}\\\frac{4000000}{x_{1}^{2} x_{2}^{2}} & \frac{8000000}{x_{1} x_{2}^{3}}\end{matrix}\right]$

```python
# we check the eigenvalues of the hessian at the stationary point (1000, 4)
hess_f_values = np.array(
    hess_f.subs(x1, 1000).subs(x2, 4),
    dtype=float
)
linalg.eig(hess_f_values)
```

    (array([1.499994e-03+0.j, 1.250005e+02+0.j]),
     array([[-0.999998  , -0.00200002],
            [ 0.00200002, -0.999998  ]]))

Eigenvalues positive implies that (1000, 4) is a point of minimum

## Contour plot

```python
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib.mlab as mlab

%matplotlib notebook
```

```python
# Contornos
x1_arr = np.linspace(500, 3000, 1000)
x2_arr = np.linspace(2, 14, 1000)
x1_mesh, x2_mesh = np.meshgrid(x1_arr, x2_arr)
fx_mesh = [[x1_ + (4*10**6) / (x1_*x2_) + 250*x2_
           for x1_ in x1_arr]
          for x2_ in x2_arr]
fx_mesh = np.array(fx_mesh)
```

```python
plt.figure()
CS1 = plt.contour(x1_mesh, x2_mesh, fx_mesh, 10)
plt.clabel(CS1, fontsize=9, inline=1)
origin = (1000, 4)
CS2 = plt.plot(*origin, 'ro',label='optimum')

plt.xlabel('x1')
plt.ylabel('x2')
plt.legend()
plt.grid()
plt.show()
#https://matplotlib.org/2.0.2/examples/pylab_examples/contour_demo.html
#https://www.tutorialspoint.com/how-to-plot-vectors-in-python-using-matplotlib
```

    <IPython.core.display.Javascript object>

```python
# The minimum is
fx.subs(x1, 1000).subs(x2, 4)
```

$\displaystyle 3000$

Reference
========
[1] ARORA, Jasbir S. Introduction to Optimum Design. Elservie. 2nd ed. Example 4.23 from page 115.


# Calculating the Mass of The Sun

One of the most important relationships in astronomy is Kepler's laws of planetary motion, because they allow us to use orbits to calculate the masses of astronomical objects. Of particular use is Kepler's third law, which relates the period, separation, and masses of two objects:

\begin{equation}
P^2 = \frac{4 \pi^2}{G (m_1 + m_2)} a^3
\end{equation}

Here $G$ is the gravitational constant, $m_1$ and $m_2$ are the masses of the two bodies orbiting eachother, $P$ is the period of their orbit, and $a$ is the distance between them. In the case of our Solar System, because the mass of the Sun is significantly greater than the mass of any of the planets (i.e. $m_1 + m_2 \approx m_1$), this can be reduced to:

\begin{equation}
P^2 = \frac{4 \pi^2}{G \, M} a^3
\end{equation}

Because we know the periods and distances for all of the planets, we can use those values to calculate the mass of the body that they are orbiting, i.e the Sun.

First, pick your favorite planet. Look up their orbital period and the distance from the Sun (i.e. their "semi-major axis"). You will also need to look up the value of the gravitational constant, $G$. Make sure to check your units! Define those as variables here:

```python

```

Now, using the above equation, calculate the mass of the Sun:

```python

```

Use what you know about string formatting to print out the mass you calculated with nice formatting, including units:

```python

```

Just to make sure we did that correctly, pick your second favorite planet, and repeat. Do you get the same answer?

```python

```

Kepler's laws are universal, so they don't just work for the Sun, but can be used to measure the mass of anything, if there's something orbiting it. Let's try measuring the mass of Jupiter. Look up the separation and period of Jupiter's moon, Callisto, and use Kepler's third law to calculate the mass of Jupiter in the same way:

```python

```

Finally, astronomers think there is a *supermassive black hole* at the center of our Milky Way galaxy. Why is that? Well, we've been able to watch stars orbit something at the center of our Galaxy:

How massive is the object that they are orbiting? Look-up the period and distance (i.e. semi-major axis) of the star "S2", and calculate the mass of that object (*Hint1: Wikipedia knows about this star. Hint 2: The units '"', or arcseconds, that are listed there for the semi-major axis can be converted into a more useful number by multiplying by the distance to the source. What is a "pc"?*):

```python

```

How does the mass you measured compare with the mass of our Sun:

```python

```

Often in astronomy we will list measured masses in terms of the mass of the Sun, because units of grams or kilograms are not easy to interpret when the numbers are so large. Use string formatting to display the mass of the object at the center of our Solar System in terms of the number of "Solar masses" that it is, with the proper units:

```python

```


# 2.2 矩阵乘法

两个矩阵相乘得到第三个矩阵：

$$
\bf C=AB
$$

为了使得定义合法，我们需要 $\mathbf A$ 的形状为 $m\times n$, $\mathbf B$ 的形状为 $n\times p$，得到的矩阵为 $\mathbf C$ 的形状为 ${m\times p}$。

定义为

$$
C_{i,j} = \sum_{k} A_{i,k}B_{k,j}
$$

注意矩阵乘法不是逐元素相乘，逐元素相乘又叫 `Hadamard` 乘积，记作 $\bf A\odot B$。

向量可以看出是列为 $1$ 的矩阵，两个相同大小的向量 $\bf x, y$ 的点乘（`dot product`）或者内积，可以使用矩阵乘法表示为 $\bf x^\top y$。

我们也可以把矩阵乘法理解为： $C_{i,j}$ 表示 $\bf A$ 的第 $i$ 行与 $\bf B$ 的第 $j$ 列的点乘。

### 矩阵乘法的性质

矩阵乘法满足结合律（`associative`）和分配律（`distributive`）：

$$
\begin{align}
\bf A(B+C)&=\bf AB+AC \\
\bf A(BC)&=\bf (AB)C
\end{align}
$$

矩阵乘法通常是不可交换的：

$$
\bf AB \neq BA
$$

但是向量内积是可交换的：

$$
\bf x^\top y = y^\top x
$$

矩阵乘法的转置形式如下：

$$
\bf (AB)^\top = B^\top A^\top
$$

利用这个式子和标量转置等于其本身，我们马上得到内积是可交换的结论：

$$
\bf x^\top y = (x^\top y)^\top = y^\top x
$$

### 线性方程组

线性方程组可以表示为矩阵和向量乘法的形式：

$$
\bf Ax = b
$$

其中 $\mathbf A\in\mathbb R^{m\times n}, \mathbf b\in\mathbb R^{m}$ 是已知的，$\mathbf x\in\mathbb R^{n}$ 是我们要求的未知量。

它是线性方程组的一种紧凑表示：

$$
\begin{align}
\mathbf A_{1,:} \mathbf x&=b_1 \\
\mathbf A_{2,:} \mathbf x&=b_2 \\
\dots &\\
\mathbf A_{m,:} \mathbf x&=b_m \\
\end{align}
$$


## Exercise:

This example is sourced from the [Scipy website](https://scipython.com/book/chapter-8-scipy/additional-examples/the-sir-epidemic-model/) where you can find more details. For more on these models and their variations, see [Compartmental models in epidimiology, Wikipedia](https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology).

Numerically solve the SIR epidemic model:

1. In pure Python using for loops to integrate in time.
1. Using Scipy's odeint package.

The differential equations that describe the model are:

\begin{equation}
\frac{dS}{dt} = \frac{-\beta I S}{N}
\end{equation}

\begin{equation}
\frac{dI}{dt} = \frac{\beta I S}{N} - \gamma I
\end{equation}

\begin{equation}
\frac{dR}{dt} = \gamma I
\end{equation}

Where S are the susceptible numbers in the population, I is the number of infected, R is the number of recovered persons in the population. $\beta$ is the *effective contact rate*, that is, an infected individual comes into contact with $\beta N$ individuals. $\gamma$ is the mean recovery rate: $1/\gamma$ is the mean period of time that an individual can pass on the infection.

A vitally important number to keep track of is the ratio: $R_0 = \beta/\gamma$; when $R_0 \gt 1$, the disease spreads through the population, when $R_0 \lt 1$, the disease quickly dies out.

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
N = 1000
Rec0 = 0
Inf0 = 1
Sus0 = N - Inf0
```

```python
beta = 0.2
gamma = 0.1
R0 = beta/gamma
```

```python
R0
```

    2.0

```python
Sus = [Sus0]
Rec = [Rec0]
Inf = [Inf0]
for t in range(150):
    delta_S = -beta * Inf[-1] * Sus[-1] / N
    Sus.append(Sus[-1]+delta_S)

    delta_I = beta * Inf[-1] * Sus[-1] / N - gamma * Inf[-1]
    Inf.append(Inf[-1] + delta_I)

    delta_R = gamma * Inf[-1]
    Rec.append(Rec[-1] + delta_R)
    #print(delta_S, delta_I, delta_R)
```

```python
Sus = np.array(Sus)
Rec = np.array(Rec)
Inf = np.array(Inf)
```

```python
plt.plot(np.arange(151), Sus, label="Susceptible")
plt.plot(np.arange(151), Rec, label="Recovered")
plt.plot(np.arange(151), Inf, label="Infected")
plt.legend()
```

### Exercise 07: SIR model in scipy

Use scipy.integrate.odeint to solve the SIR model. For more information, refer to the [documentation page](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html).

```python
# %load ./solutions/sol_scipy_SIR.py
```


```python
import numpy as np
import matplotlib.pyplot as plt
import math
from numpy.linalg import inv
from numpy import matmul as mm
```

- Dataset $\mathcal{D}=\{x_i,y_i\}^N_{i=1}$ of N pairs of inputs $x_i$ and targets $y_i$. This data can be measurements in an experiment.
- Goal: predict target $y_*$ associated to any arbitrary input $x_*$. This is known as a **regression** task in machine learning.

### Generate Dataset
$$y_i = (x_i+1)^3+\epsilon_i$$
where $\epsilon_i \sim \mathcal{N}(0,1)$

```python
x = np.linspace(-4,2,20)
y = (x+1)**3+10*np.random.normal(0,1,20)
```

```python
plt.plot(x,y,'b.')
```

### Model of the data
Assume that the dataset can be fit with a $M^{th}$ order polynomial (**polynomial regression**),

$$f_w(x) = w_0+w_1x+w_2x^2+w_3x^3+...+w_Mx^M=\sum^M_{j=1}w_j\phi_j(x)$$
The $w_j$ are the weights of the polynomial, the parameters of the model, and $\phi_j(x)$ is a basis function of our linear in the parameters model.

### Fitting model parameters via the least squares approach
- Measure the quality of the fit to the training data.
- For each train point, measure the squared error $e^2_i = (y_i-f(x_i))^2$.
- Find the parameters that minimize the sum of squared errors:

$$E(\mathbf{w})=\sum^N_{i=1}e^2_i=\|\mathbf{e}\|^2 = \mathbf{e}^\top\mathbf{e}=(\mathbf{y}-\mathbf{f})^\top(\mathbf{y}-\mathbf{f})$$
where $\mathbf{y} = [y_1,...y_N]^\top$ is a vector that stacks the N training targets, $\mathbf{f}=[f_\mathbf{W}(x_1),...,f_\mathbf{w}(x_N)]^\top$ stacks the prediction evaluated at the N training inputs.

Therefore,

\begin{align}
\mathbf{y}&=\mathbf{f}+\mathbf{e}=\mathbf{\Phi w}+\mathbf{e} \\
\begin{pmatrix} y_1\\y_2\\\vdots\\y_N\end{pmatrix}&=\begin{pmatrix}1&x_1&x_1^2&...&x_1^M\\1&x_2&x_2^2&...&x_2^M\\\vdots&\vdots&\vdots&\cdots&\vdots\\1&x_N&x_N^2&...&x_N^M\end{pmatrix}\begin{pmatrix}w_0\\w_1\\\vdots\\w_M\end{pmatrix} + \mathbf{e}
\end{align}

The sum of squared errors is a convex function of $\mathbf{w}$:

$$E(\mathbf{w})=(\mathbf{y}-\mathbf{\Phi w})^\top(\mathbf{y}-\mathbf{\Phi w})$$
To minimize the errors, find the weight vector $\mathbf{\hat{w}}$ that sets the gradient with respect to the weights to zero,

$$\frac{\partial E(\mathbf{w})}{\partial \mathbf{w}}=-2\mathbf{\Phi}^\top(\mathbf{y}-\mathbf{\Phi w})=2\mathbf{\Phi^\top\Phi w}-2\mathbf{\Phi}^\top \mathbf{y}=0$$

The weight vector is

$$\mathbf{\hat{w}}=(\mathbf{\Phi^\top\Phi})^{-1}\mathbf{\Phi^\top y}$$

```python
def polynomialFit(x,y,order=3):
    for i in range(order+1):
        if i == 0:
            Phi = x**i
        else:
            Phi = np.vstack((Phi,x**i))
    Phi = Phi.T
    if order == 0:
        Phi = Phi.reshape(-1,1)
    w = mm(mm(inv(mm(Phi.T,Phi)),Phi.T),y)
    f = mm(Phi,w)
    dif = y-f
    err = mm(dif.T,dif)
    return f,err,w,Phi
```

```python
f,err,w,Phi = polynomialFit(x,y)
```

```python
plt.plot(x,y,'b.')
plt.plot(x,f,'r-')
print(w) # ideal: 1,3,3,1
print(err)
```

### M-th order Polynomial

```python
errlist = []
plt.figure(figsize=(20,20))
for i in range(21):
    plt.subplot(7,3,i+1)
    f,err,w,Phi = polynomialFit(x,y,i)
    errlist.append(err)
    plt.plot(x,y,'b.')
    plt.plot(x,f,'r-')
    plt.title('Order '+str(i)+': '+str(err))
```

```python
plt.plot(np.arange(16),errlist[:16])
```

#### The fitting becomes very unstable after the order of 15. This may be due to the inverse instability. This can be resolved via LU decomposition, Cholesky decomposition or QR decomposition.

### LU decomposition

```python
import scipy
from scipy.linalg import lu_factor,lu_solve
```

```python
def polynomialFitLU(x,y,order=3):
    for i in range(order+1):
        if i == 0:
            Phi = x**i
        else:
            Phi = np.vstack((Phi,x**i))
    Phi = Phi.T
    if order == 0:
        Phi = Phi.reshape(-1,1)
    lu,piv = lu_factor(mm(Phi.T,Phi))
    tmp = lu_solve((lu,piv),Phi.T)
    w = mm(tmp,y)
    f = mm(Phi,w)
    dif = y-f
    err = mm(dif.T,dif)
    return f,err,w,Phi
```

```python
errlistLU = []
plt.figure(figsize=(20,20))
for i in range(21):
    plt.subplot(7,3,i+1)
    f,err,w,Phi = polynomialFitLU(x,y,i)
    errlistLU.append(err)
    plt.plot(x,y,'b.')
    plt.plot(x,f,'r-')
    plt.title('Order '+str(i)+': '+str(err))
```

```python
plt.plot(np.arange(21),errlistLU)
```

### Cholesky decomposition

```python
from scipy.linalg import cho_factor,cho_solve
```

```python
def polynomialFitChol(x,y,order=3):
    for i in range(order+1):
        if i == 0:
            Phi = x**i
        else:
            Phi = np.vstack((Phi,x**i))
    Phi = Phi.T
    if order == 0:
        Phi = Phi.reshape(-1,1)
    c,low = cho_factor(mm(Phi.T,Phi))
    tmp = cho_solve((c,low),Phi.T)
    w = mm(tmp,y)
    f = mm(Phi,w)
    dif = y-f
    err = mm(dif.T,dif)
    return f,err,w,Phi
```

```python
errlistChol = []
plt.figure(figsize=(20,20))
for i in range(21):
    plt.subplot(7,3,i+1)
    f,err,w,Phi = polynomialFitLU(x,y,i)
    errlistChol.append(err)
    plt.plot(x,y,'b.')
    plt.plot(x,f,'r-')
    plt.title('Order '+str(i)+': '+str(err))
```

```python
plt.plot(np.arange(21),errlistChol)
```

### Comparison between inverse, LU decomposition and Cholesky decomposition

```python
plt.plot(np.arange(21),errlist)
plt.plot(np.arange(21),errlistLU)
plt.plot(np.arange(21),errlistChol)
```

```python
plt.plot(np.arange(21),errlistLU)
plt.plot(np.arange(21),errlistChol)
```

```python

```

Probabilistic Machine Learning/Introduction to Probabilistic Machine Learning/.ipynb_checkpoints/Linear in the parameters regression-checkpoint.ipynb


```python
from sympy import *
init_printing(use_latex='mathjax')
n = symbols('n', integer=True)
x, y, z = symbols('x,y,z')
```

## Integrals

In the first section we learned symbolic differentiation with `diff`.  Here we'll cover symbolic integration with `integrate`.

Here is how we write the indefinite integral

$$ \int x^2 dx = \frac{x^3}{3}$$

```python
# Indefinite integral
integrate(x**2, x)
```

And the definite integral

$$ \int_0^3 x^2 dx = \left.\frac{x^3}{3} \right|_0^3 = \frac{3^3}{3} - \frac{0^3}{3} = 9 $$

```python
# Definite integral
integrate(x**2, (x, 0, 3))
```

As always, because we're using symbolics, we could use a symbol wherever we previously used a number

$$ \int_y^z x^n dx $$

```python
integrate(x**n, (x, y, z))
```

### Exercise

Compute the following integrals:

$$ \int \sin(x) dx $$
$$ \int_0^{\pi} \sin(x) dx $$
$$ \int_0^y x^5 + 12x^3 - 2x + 1  $$
$$ \int e^{\frac{(x - \mu)^2}{\sigma^2}} $$

Feel free to play with various parameters and settings and see how the results change.

```python
# Use `integrate` to solve the integrals above

```

Are there some integrals that SymPy can't do?  Find some.

```python
# Use `integrate` on other equations. Symbolic integration has it limits, find them.
```


# 2. Linear Algebra_2. Systems of Linear Equations

Linear Algebra는 Linear Equations problem을 Vector를 이용해 해결하는 것에 관한 학문으로 이해할 수 있습니다.

따라서, Linear Equation과 이의 조합인 Systems of Linear Equations(선형 연립방정식) 또한 Vector, Matrix로 표현이 가능합니다.

예를 들어, 다음과 같은 선형 연립방정식을 행렬 $A$, 벡터 $x$, $b$ 를 이용해 간단히 표현할 수 있습니다.

$$ x_{1} + x_{2} + x_{3} = 3 $$
$$ x_{1} - x_{2} + 2x_{3} = 2 $$
$$         x_{2} + x_{3} = 2 $$

$$
\begin{align}
A=
\begin{bmatrix}
1&1&1\\
1&-1&2\\
0&1&1
\end{bmatrix}
,\;\;
x=
\begin{bmatrix}
x1\\
x2\\
x3
\end{bmatrix}
,\;\;
b=
\begin{bmatrix}
3\\
2\\
2
\end{bmatrix}
\end{align}
$$

$$Ax = b$$
$$x = A^{-1}b$$

그리고, 이러한 Systems of Linear equations의 solution ( 벡터 $x$ ) 을 쉽게 찾을 수 있다.
이때, 역행렬을 구하기 위해 Numpy패키지의 np.linalg.inv() method를 활용한다

**참고 : @ 기호는 dotproduct(내적)연산을 수행한다. 기본 행렬곱인 element-wise 연산을 수행하기 위해선, 아래와 같이 @를 통해 행렬과 벡터의 연산을 수행한다.**

```python
A = np.array([[1,1,1],[1,-1,2],[0,1,1]])
b = np.array([[3],[2],[2]])
x = np.linalg.inv(A)@b
x
```

    array([[1.],
           [1.],
           [1.]])

이와 같은 선형연립방정식을 풀 때, 위와 같이 역행렬을 직접 구해 solution을 구해줄 수 있지만, Numpy의 'lstsq()' 명령을 활용해 구할 수도 있다.

`lstsq()` 명령은 향후 자세히 보게 될 solution, RSS, rank, Singular vlaue 를 각각 반환한다. `lstsq()` 명령은 이후 보게될 'Ordinary Least Square Problem'을 풀기위한 연산 명령이다.

위와 같이 미지수의 갯수 $=$ 방정식의 갯수 (A가 Square Matrix)인 경우, OLS와 Inverse Matrix의 solution이 같기 때문에*, `lstsq()` 명령을  사용해도 동일한 답을 얻을 수 있다.

*미지수의 갯수 $=$ 방정식의 갯수인 경우, unique solution을 갖기 때문에

```python
x, resid, rank, s = np.linalg.lstsq(A, b)
x
```

    array([[1.],
           [1.],
           [1.]])

1.Study/2. with computer/1.Math_code/5. Mathematics for Machine Learning/2. Linear Algebra_2. Systems of Linear Equations.ipynb


```python
import numpy as np
from scipy import linalg as la
import sympy as sp
```

Starting from L9

Vectors x1 and x2 are independent if c1x1+c2x2 <>0

Vectors v1,v2,...,vn are columns of A. They are independant if nullspace of A is only zero vector(r=n, no free variables). They are dependant if Ac=0 for some nonzero c(r<n, there are free variables).

Vectors v1,...,vl span a space means: the space consists of all combs of those vectors.

Basis for a vector space is a sequence of vectors v1,v2,...,vd with 2 properties: 1. They are independant; 2. They span the space.

I3 is a basis for R^3

```python
I3 = np.identity(3)
I3
```

    array([[ 1.,  0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.]])

```python
Z = np.zeros(3)
Z
```

    array([ 0.,  0.,  0.])

The only vector  that gives zeros:

```python
np.dot(I3,Z)
```

    array([ 0.,  0.,  0.])

Another basis:

```python
A = np.array([[1,1,4],
             [1,2,3],
             [2,7,11]])
A
```

    array([[ 1,  1,  4],
           [ 1,  2,  3],
           [ 2,  7, 11]])

n x n matrix that is invertible.

```python
np.linalg.inv(A)
```

    array([[ 0.125,  2.125, -0.625],
           [-0.625,  0.375,  0.125],
           [ 0.375, -0.625,  0.125]])

```python
np.linalg.det(A)
```

    8.0000000000000018

Every basis for the space has the same number of vectors and this number is the dimension of this space.

```python
np.linalg.matrix_rank(A)
```

    3

Rank is a number of pivot columns and it is a dimension of the columnspace. C(A)

The dimension of a Null Space is the number of free variables, total - pivot variables.

4 Fundamental subspaces:

    Columnspace C(A) in R^m
    nullspace N(A) in R^n
    rowspace C(A^T) in R^n
    nullspace of A^T = N(A^T) (Left NullSpace) in R^m

Just a lyrical digression. One can do a PLU transformation using scipy.linalg.lu.

```python
P,L,U = la.lu(A)

P,L,U
```

    (array([[ 0.,  1.,  0.],
            [ 0.,  0.,  1.],
            [ 1.,  0.,  0.]]), array([[ 1. ,  0. ,  0. ],
            [ 0.5,  1. ,  0. ],
            [ 0.5,  0.6,  1. ]]), array([[  2. ,   7. ,  11. ],
            [  0. ,  -2.5,  -1.5],
            [  0. ,   0. ,  -1.6]]))

```python
A2 = np.array([[1,3,1,4],
              [2,7,3,9],
              [1,5,3,1],
              [1,2,0,8]])
```

Using sympy it is possible to calculate Reduced Row Echelon Form of the Matrix and thus find the basis of the matrix. And understand which rows are linearly independant.

```python
sp.Matrix(A2).rref()
```

    (Matrix([
     [1, 0, -2, 0],
     [0, 1,  1, 0],
     [0, 0,  0, 1],
     [0, 0,  0, 0]]), (0, 1, 3))

In a matrix A2 columns 1, 2 and 4 are linearly independant and form a basis. Thus the rank of this matrix is 3.

```python
np.linalg.matrix_rank(A2)
```

    3

```python
sp.Matrix(A).rref()
```

    (Matrix([
     [1, 0, 0],
     [0, 1, 0],
     [0, 0, 1]]), (0, 1, 2))

And here all the columns are linearly independant and form the basis.

```python
sp.Matrix(np.transpose(A2)).rref()
```

    (Matrix([
     [1, 0, 0,  0],
     [0, 1, 0,  1],
     [0, 0, 1, -1],
     [0, 0, 0,  0]]), (0, 1, 2))

```python
sp.Matrix(A2).nullspace()
```

    [Matrix([
     [ 2],
     [-1],
     [ 1],
     [ 0]])]

```python
sp.Matrix(A2).rowspace()
```

    [Matrix([[1, 3, 1, 4]]), Matrix([[0, 1, 1, 1]]), Matrix([[0, 0, 0, -5]])]

```python
sp.Matrix(A2).columnspace()
```

    [Matrix([
     [1],
     [2],
     [1],
     [1]]), Matrix([
     [3],
     [7],
     [5],
     [2]]), Matrix([
     [4],
     [9],
     [1],
     [8]])]

The dim of both column space and row space are rank of matrix - r.

```python
A3 = np.array ([[1,2,3],
               [1,2,3],
               [2,5,8]])
```

```python
np.linalg.matrix_rank(A3)
```

    2

```python
R = np.array(sp.Matrix(A3).rref()[0])
R
```

    array([[1, 0, -1],
           [0, 1, 2],
           [0, 0, 0]], dtype=object)

```python
A4 = np.array([[1,2,3,1],
              [1,1,2,1],
              [1,2,3,1]])
```

```python
R4 = np.array(sp.Matrix(A4).rref()[0])
R4
```

    array([[1, 0, 1, 1],
           [0, 1, 1, 0],
           [0, 0, 0, 0]], dtype=object)

```python
sp.Matrix(A4).rowspace()
```

    [Matrix([[1, 2, 3, 1]]), Matrix([[0, -1, -1, 0]])]

```python
sp.Matrix(A4).columnspace()
```

    [Matrix([
     [1],
     [1],
     [1]]), Matrix([
     [2],
     [1],
     [2]])]

```python
N41,N42 = np.array(sp.Matrix(A4).nullspace())
np.dot(A4,N41), np.dot(A4,N42)
```

    (array([0, 0, 0], dtype=object), array([0, 0, 0], dtype=object))

```python
lN4 = np.array(sp.Matrix(np.transpose(A4)).nullspace())
np.dot(lN4,A4)
```

    array([[0, 0, 0, 0]], dtype=object)

E matrix, such as EA = R

```python
preE4 = np.rint(np.array(sp.Matrix(np.c_[A4,np.eye(3)]).rref()[0]).astype(np.double))
```

```python
preE4
```

    array([[ 1.,  0.,  1.,  1.,  0.,  2., -1.],
           [ 0.,  1.,  1.,  0.,  0., -1.,  1.],
           [ 0.,  0.,  0.,  0.,  1.,  0., -1.]])

```python
A4
```

    array([[1, 2, 3, 1],
           [1, 1, 2, 1],
           [1, 2, 3, 1]])

```python
E4 = preE4[:,4:7]
E4
```

    array([[ 0.,  2., -1.],
           [ 0., -1.,  1.],
           [ 1.,  0., -1.]])

```python
np.dot(E4,A4)
```

    array([[ 1.,  0.,  1.,  1.],
           [ 0.,  1.,  1.,  0.],
           [ 0.,  0.,  0.,  0.]])

Ending Lecture 10 here.


## Barycentric Lagrange Interpolation

This rewriting of the Lagrange interpolation goes back at least seventy-five years but was popularized by [Berrut and Trefethen](https://epubs.siam.org/doi/pdf/10.1137/S0036144502417715) in the early 21st century.  The idea here is basically to rewrite the Lagrange form to avoid the problems mentioned above.

First define

$$\mathcal{l}(x)\equiv \prod_{i=0}^n (x-x_i).$$

Then note that

$$L_j^n(x) = \frac{\mathcal{l}(x)}{(x-x_j)} w_j,$$

where the baricentric weight $w_j$ is

$$ w_j \equiv \frac{1}{\prod_{i=0,i\neq j}^n (x_j-x_i)}.$$

Our interpolating polynomial now becomes

$$p_n(x) = \sum_{i=0}^n L_i^n(x) f(x_i) =  \mathcal{l}(x)\sum_{i=0}^n \frac{w_i}{x-x_i} f(x_i).$$

This is called the *first form of barycentric interpolation*.  Once we have computed the weights $w_j$, evaluation of $p_n(x)$ now requires $\mathcal{O}(n)$ flops thus removing the first objection to the Lagrange form of the interpolating polynomial.

If we add a new node $x_{n+1}$, we can construct new weights $w_j$ by dividing the old weights by $(x_j-x_{n+1})$ to get the new $w_j$ (plus computing $w_{n+1}$ using the original formula).  This effectively eliminates the second objection to the Lagrange form of the intepolating polynomial.

There is another similification we can use before constructing an algorithm for interpolation.  Note that for the function $g(x)\equiv 1$,

$$
\begin{align}
g(x) &=  \mathcal{l}(x)\sum_{i=0}^n \frac{w_i}{x-x_i} g(x_i),\\
 &= \mathcal{l}(x)\sum_{i=0}^n \frac{w_i}{x-x_i}.
\end{align}
$$
As we shall see when we discuss interpolation errors, this formula is actually exact.  As a result,

$$
\begin{align}
p_n(x) &= \frac{p_n(x)}{1} = \frac{p_n(x)}{g(x)},\\
 &=\frac{\mathcal{l}(x)\sum_{i=0}^n \frac{w_i}{x-x_i} f(x_i)}{\mathcal{l}(x)\sum_{i=0}^n \frac{w_i}{x-x_i}},
\end{align}
$$

which simiplifies to the second, or *true form of barycentric interpolation*

$$p_n(x)=\frac{\sum_{i=0}^n \frac{w_i}{x-x_i} f(x_i)}{\sum_{i=0}^n \frac{w_i}{x-x_i}}.$$

In this form, we avoid evaluating $\mathcal{l}(x)$ entirely and the sum in the numerator and denominator can easily be constructed simultaneously.

Note that there are two stages to implementation of barycentric Lagrange interpolation: i) computing the weights and ii) computing the interpolation at the desired points.  The first should be done *only once* whereas the second needs to be done at each point where the interpolation is computed.  A python code to do this is below

```python
import numpy as np
from matplotlib import pyplot as plt

def baryweight(x):
    # build the weights recursively by effectively adding new points one at a time
    w = [1.]*len(x)
    for i in range(0,len(x)):
        # in iteration i we compute w[i] from scratch while
        # including the contribution of x[i] to the other weights
        for j in range(0,i):
            w[j]=(x[j]-x[i])*w[j]
            w[i]=(x[i]-x[j])*w[i]
    # do the divisions at the end to minimize the total number of divisions
    return [1/x for x in w]

def baryinterp(w,x,y,xp):
    # Compute the barycentric interpolation at xp given the observed data in lists
    # x and y and the precomputed weights in a list w
    numer=0.
    denom=0.
    for i in range(0,len(x)):
        xdiff=xp-x[i]
        # We first check to ensure we are not actually evaluated at a given data point.
        # (This would lead to division by zero which would cause problems).
        if (xdiff == 0):
            return y[i]
        else:
            tmp=w[i]/xdiff
            numer += tmp*y[i]
            denom += tmp
    return numer/denom

# Main program showing functions defined above used to interpolate data taken from
# a simple sine function

# This creates a list of x values for our observed data
dx = 0.5
x_data = np.arange(0, 6.5+dx, dx)

# Compute the weights for the interpolating polynomial.
weights=baryweight(x_data)

# Create a list of y values for our observed data and plot
# our observations
y_data = np.sin(x_data)
plt.plot(x_data, y_data, "o", label="data")

# Now we want to fill in between our observed data points using
# a barycentric Lagrange interpolation and plot the result
dx = 0.01
x_all = np.arange(0,6.5+dx,dx)
y_all = [0]*len(x_all)
for i in range(0,len(x_all)):
    y_all[i] = baryinterp(weights,x_data,y_data,x_all[i])

plt.plot(x_all, y_all, label="barycentric interpolation")
plt.legend()
plt.show()
```

There are a few things to note here:

- the weights depend only on the $x_i$ and not on $f(x_i)$.  As such, the same weights could be used in the interpolation of any function whose values are known at the same set of $x_i$  (i.e. we can resuse the weights for other functions).
- the order of the nodes does not matter.  This means that if we want to add a new data point, it can just be appended to the end of the list, it does not need to be inserted in order.
- if you want to add a new data point, you could easily write another function (say "addpoint") to update the weights.  It would essentially consist of the "extra" iteration of the loop in the above `baryweight` function, accounting for the fact that we have already done the `1/x` operation.  It should be clear that the update is $\mathcal{O}(n)$ operation as only the inner loop is computed.
- you may have noticed that the `baryinterp` function dealt with the case where $x=x_i$ as a special case to avoid division by zero.  You might be concerned about overflow in the situation where $x$ is close to $x_i$ but not exactly the same.  There are ways of dealing with this when it arises, but it turns out it is not a serious concern for most practical applications of interpolation.
- as with any algorithm, one should ask if this is numerically stable.  As we emphasized when we discussed errors in scientific computing, it is only reasonable to ask if the algorithm is stable when applied to a well-conditioned problem.  As we will discuss shortly, polynomial interpolation for large numbers of points is only stable if these points are chosen appropriately.  In this case, [Higham](https://www.maths.manchester.ac.uk/~higham/narep/narep440.pdf) has shown that barycentric Lagrange interpolation is unconditionally stable (the first form is slightly better than the "true" form in this regards, but the difference for practical purposes is not usually significant).
- for some node spacings, it is possible to give an explicit formula for the barycentric weights $w_j$.  As these weights appear in the true form of the barycentric interpolation in both the numerator and denominator, any common factor in all the weights can be cancelled without affecting the result.

There are also built in versions of [barycentric interpolation in SciPy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.barycentric_interpolate.html#scipy.interpolate.barycentric_interpolate). The SciPy version also has functions to add new nodes and new, or different, values of the function to be interpolated (making use of the previously computed weights in both cases).


## Homework Problem: Alice Hunts Dragons

When she is not calculating marginal distributions, Alice spends her time hunting dragons. For every dragon she encounters, Alice measures its fire power $X$ (measured on a scale from $1$ to $4$) and its roar volume $Y$ (measured on a scale from $1$ to $3$). She notices that the proportion of dragons with certain fire power and roar volume in the population behaves as the following function:

$$\begin{eqnarray}
f(x,y) = \begin{cases} x^2+y^2 &\text{if } x \in \{1,2,4\} \text{ and } y \in \{1,3\} \\
0 &\text{otherwise}. \end{cases}
\end{eqnarray}$$

In other words, the joint probability table $p_{X,Y}$ is of the form

$$p_{X,Y}(x,y) = c f(x, y) \qquad \text {for }x \in \{ 1, 2, 3, 4\} , y \in \{ 1, 2, 3\} ,$$

for some constant $c>0$ that you will determine.

**Question:** Determine the constant $c$, which ensures that $p_{X,Y}$ is a valid probability distribution. (Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

```python
Ω = {(i, j) for i in range(1, 5) for j in range(1, 4)}
from fractions import Fraction
joint_X_Y = {(i, j): (i**2 + j**2) for i in {1, 2, 4} for j in {1, 3}}
c = 1/sum(joint_X_Y.values())
print(Fraction(c).limit_denominator())
```

    1/72

**Question:** Determine $P(Y<X)$. (Note that $\{Y<X\}$ is an event. Think about what outcomes are in it.)

(Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

```python
Y_lt_X = {x for x in joint_X_Y if x[1] < x[0]}
prob_Y_lt_X = sum([joint_X_Y[x] for x in Y_lt_X]) * c
print(Fraction(prob_Y_lt_X).limit_denominator())
```

    47/72

**Question:** Determine $P(X<Y)$. (Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

```python
X_lt_Y = {x for x in joint_X_Y if x[1] > x[0]}
prob_X_lt_Y = sum([joint_X_Y[x] for x in X_lt_Y]) * c
print(Fraction(prob_X_lt_Y).limit_denominator())
```

    23/72

**Question:** Determine $P(Y=X)$. (Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

```python
X_eq_Y = {x for x in joint_X_Y if x[1] == x[0]}
prob_X_eq_Y = sum([joint_X_Y[x] for x in X_eq_Y]) * c
print(Fraction(prob_X_eq_Y).limit_denominator())
```

    1/36

**Question:** Determine $P(Y=3)$. (Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

```python
Y_eq_3 = {x for x in joint_X_Y if x[1] == 3}
prob_Y_eq_3 = sum([joint_X_Y[x] for x in Y_eq_3]) * c
print(Fraction(prob_Y_eq_3).limit_denominator())
```

    2/3

**Question:** Find the probability tables for $p_X$ and $p_Y$. Express your answers as Python dictionaries. (Your answer should be the Python dictionary itself, and not the dictionary assigned to a variable, so please do not include, for instance, “prob_table =" before specifying your answer. You can use fractions. If you use decimals instead, please be accurate and use at least 5 decimal places.)

$p_X$ probability table (the dictionary keys should be the Python integers 1, 2, 3, 4):

```python
prob_X = {i: 0 for i in range(1, 5)} # initialize the dictionary with 0
for key, values in joint_X_Y.items():
    if key[0] in prob_X:
        prob_X[key[0]] += values * c

prob_X
```

    {1: 0.16666666666666669, 2: 0.25, 3: 0, 4: 0.5833333333333333}

$p_Y$ probability table (the dictionary keys should be the Python integers 1, 2, 3):

```python
prob_Y = {i: 0 for i in range(1, 4)} # initialize the dictionary with 0
for key, values in joint_X_Y.items():
    if key[1] in prob_Y:
        prob_Y[key[1]] += values * c

prob_Y
```

    {1: 0.33333333333333337, 2: 0, 3: 0.6666666666666666}

## Homework Problem: Alice's Coins

Alice has five coins in a bag: two coins are normal, two are double-headed, and the last one is double-tailed. She reaches into the bag and randomly pulls out a coin. Without looking at the coin she drew, she tosses it.

**Question:** What is the probability that once the coin lands, the side of the coin that is face-down is heads? (Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

**Answer:** Let $X$ is the random variable for selecting the coin. Let $X = \{F, H, T\}$ takes values $F$ for fair coin, $H$ for double-headed coin and $T$ for double-tailed coin. Also $Y = \{h, t\}$ is the random variable for getting $h$ for face down head and $t$ for face down tail. Then the joint probability distribution is given by

|                      | Y=h | Y=t | Xmarginal |
|----------------------|-----|-----|----------------------|
| X=F                  | 1/5 | 1/5 | 2/5                  |
| X=H                  | 2/5 | 0   | 2/5                  |
| X=T                  | 0   | 1/5 | 1/5                  |
| Ymarginal | 3/5 | 2/5 |                      |

The probability of getting face down head is given by marginal probibility $\mathbb{P}(Y=h) = 3/5$.

**Question:** The coin lands and shows heads face-up. What is the probability that the face-down side is heads? (Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

**Answer:** This question asking about conditional probability
$$\mathbb{P}(X=H|Y=h) = \frac{\mathbb{P}(X=H, Y=h)}{\mathbb{P}(Y=h) } = \frac{2/5}{3/5} = \frac{2}{3} $$

Alice discards the first coin (the one from part (b) that landed and showed heads face-up), reaches again into the bag and draws out a random coin. Again, without looking at it, she tosses it.

**Question:** What is the probability that the coin shows heads face-up? (Please be precise with at least 3 decimal places, unless of course the answer doesn't need that many decimal places. You could also put a fraction.)

**Answer:** Let $Z$ is the random varable for getting second coin face-up heads or tails. Also note that the first coin with face-up heads is discarded. There are two possibiliy of getting heads on face-up if $X=F$ or $X=H$. Let $W$ is the random variable for withdrawing second coin.

The joint probability distribution of $W$ and $Z$ given $X=F$, $i.e.$, $P(W,Z|X=F)$

|                      | Z=h | Z=t | Wmarginal |
|----------------------|-----|-----|:--------------------:|
| W=F                  | 1/8 | 1/8 | 1/4                  |
| W=H                  | 1/2 | 0   | 1/2                  |
| W=T                  | 0   | 1/4 | 1/4                  |
| Zmarginal | 5/8 | 3/8 |                      |

The joint probability distribution of $W$ and $Z$ given $X=H$, $i.e.$, $P(W,Z|X=H)$

|                      | Z=h | Z=t | Wmarginal |
|----------------------|-----|-----|:--------------------:|
| W=F                  | 1/4 | 1/4 | 1/2                  |
| W=H                  | 1/4 | 0   | 1/4                  |
| W=T                  | 0   | 1/4 | 1/4                  |
| Zmarginal | 1/2 | 1/2 |                      |

Hence the probability of getting heads in face-up is given by

$$
\begin{align}
\mathbb{P}(Z=h|Y=h)
&= \mathbb{P}(Z=h|X=F) \times \mathbb{P}(X=F|Y=h) + \mathbb{P}(Z=h|X=H) \times \mathbb{P}(X=H|Y=h) \\
&= \mathbb{P}(Z=h|X=F) \times \frac{\mathbb{P}(X=F, Y=h)}{ \mathbb{P}(Y=h)} + \mathbb{P}(Z=h|X=H) \times \frac{\mathbb{P}(X=H, Y=h)}{ \mathbb{P}(Y=h)} \\
&= \frac{5}{8} \times \frac{1/5}{3/5} + \frac{1}{2} \times \frac{2/5}{3/5}\\
&= \frac{13}{24}
\end{align}
$$

```python

```


Sympy variables are created using unique string identifiers.

```python
import sympy as sp
import numpy as np
x = sp.Symbol("x")
y = sp.Symbol("y")
z = sp.Symbol("z")
```

One can form expression from symbols.
Sympy expressions are made up of numbers, symbols, and sympy functions.

```python
expression = x**2. + y**2. + z ** 2.
expression
```

    x**2.0 + y**2.0 + z**2.0

Two expressions may be added together to form a new one.

```python
other_expression = x**2.
expression += other_expression
expression
```

    2*x**2.0 + y**2.0 + z**2.0

One can form sympy `Matrix` objects.

```python
sp.Matrix([[1,2],[3,4]])
```

    Matrix([
    [1, 2],
    [3, 4]])

An important `Matrix` function is `eye(n)`, which forms a $n \times n$ identity matrix.

```python
sp.eye(3)
```

    Matrix([
    [1, 0, 0],
    [0, 1, 0],
    [0, 0, 1]])

One can stuff expressions into matrices, too.

```python
# One can stuff expressions into matrices
f1 = x**2.+y**2-z**2.
f2 = 2*x + y + z
function_matrix = sp.Matrix([f1,f2])
function_matrix
```

    Matrix([
    [x**2.0 + y**2 - z**2.0],
    [           2*x + y + z]])

One may compute the Jacobian of vector valued functions, too.

```python
function_matrix.jacobian([x,y,z]) # pass in a list of Sympy Symbols to take the Jacobian
```

    Matrix([
    [2.0*x**1.0, 2*y, -2.0*z**1.0],
    [         2,   1,           1]])

Sympy expressions can be evaluated by passing in a Python dictionary mapping Symbol `Symbol`s to specific values.

```python
x_val = 1.0
y_val = 2.0
z_val = 3.0
values={"x":x_val,"y":y_val,"z":z_val}
f1.subs(values)
```

    -4.00000000000000

One can even valuate the Jacobian of functions.

```python
J_mat = function_matrix.jacobian([x,y,z]).subs(values)
J_mat
```

    Matrix([
    [2.0, 4.0, -6.0],
    [  2,   1,    1]])

To convert a Sympy `Matrix` into a Numpy array, one may use the following:

```python
np.array(J_mat)
```

    array([[2.00000000000000, 4.00000000000000, -6.00000000000000],
           [2, 1, 1]], dtype=object)

After evaluating an expression in Sympy, the return type is a `sympy.Float`.
However, this is not readily usable by Numpy. Therefore, consider casting `sympy.Float` to a `numpy.float64`.

```python
J=np.array(J_mat).astype(np.float64)
J
```

    array([[ 2.,  4., -6.],
           [ 2.,  1.,  1.]])

At this point, one cna do all the usual stuff one would in Numpy.

```python
J.T@J
```

    array([[  8.,  10., -10.],
           [ 10.,  17., -23.],
           [-10., -23.,  37.]])

Symp's `Lambdify` can help increase the speed of Sympy's numerical computations.

```python
function_matrix.subs(values)
```

    Matrix([
    [-4.0],
    [ 7.0]])

```python
from sympy.utilities.lambdify import lambdify
array2mat = [{'ImmutableDenseMatrix': np.array}, 'numpy']
lam_f_mat = lambdify((x,y,z), function_matrix, modules=array2mat)
lam_f_mat(1,2,3)
```

    array([[-4.],
           [ 7.]])

Or if it is more convenient to have the function evaluation occur from a list of some sort, the Python `*` operator on lists can help.

```python
lam_f_mat(*[1,2,3])
```

    array([[-4.],
           [ 7.]])


## Homework-4
### CSC-722: Machine Learning Fundamentals
### Md Hafizur Rahman

###  1) Write a code to find norm 0, norm 1, norm 2, and norm infinity of the vectore x

### Theory:
In mathematics, a norm is a function from a vector space over the real or complex numbers to the nonnegative real numbers, that satisfies certain properties pertaining to scalability and additivity and takes the value zero only if the input vector is zero [1].

On $R^n$ and $p≥1$, where R is real number and n=0,1,2... the p-norm is defined as
\begin{align}
l_p=(\sum_{j=1}^n|x_j|^p)^{1/p}
\end{align}

In our case, we have to calculate $l_0$, $l_1$, $l_2$ and $l_\inf$. X is a list of 10 random numbers between -10 and 10. So, j= 1,2,3...,10

```python
#importing the necessary libraries
import numpy as np
from scipy.sparse import csr_matrix
```

```python
#generating 10 random numbers of a list between -10 and 10.
x = np.random.randint(-10, 10, 10)
print(x)
```

    [  2   9  -9 -10   6  -5  -8  -9   5  -8]

#### Norm-0:

The $l_0$ norm corresponds to the total number of nonzero elements in a given vector.

We have used count_nonzero method of numpy to find the norm-0.

```python
norm_0=np.count_nonzero(x)
print("Norm-0 of the X is", norm_0)
```

    Norm-0 of the X is 10

#### Norm-1:
Norm-1 is also known as Manhattan Distance or Taxicab norm. L1 norm is the sum of the magnitudes of the vectors in a space.

Mathematically,

\begin{align}
l_1=(\sum_{j=1}^n|x_j|)
\end{align}

We used sum method of numpy to calculate the norm-1. abs() has to use for getting the absolute values of x.

```python
norm_1=sum(abs(x))
print("Norm-1 of the X is", norm_1)
```

    Norm-1 of the X is 71

#### Norm-2:
Norm-2 is also known as the Euclidean norm. It is the shortest distance to go from one point to another.

Mathematically, we can write the $l_2$ norm as

\begin{align}
l_2=(\sum_{j=1}^n|x_j|^2)^{1/2}
\end{align}

We used sum method of numpy to calculate the norm-1. abs() has to use for getting the absolute values of x.

```python
norm_2=(sum(x**2))**(1/2)
print("Norm-2 of the X is",norm_2)
```

    Norm-2 of the X is 23.68543856465402

#### Norm-inf:
Norm-inf gives the largest magnitude among each element of a vector.

Mathematically, we can write $l_\inf$ norm as

\begin{align}
l_\inf=max|x_j|
\end{align}

We used max method of numpy to calculate the norm-inf. abs() has to use for getting the absolute values of x.

```python
norm_inf=max(abs(x))
print("Norm-inf of the X is",norm_inf)
```

    Norm-inf of the X is 10

###  2) Explain the following cells
###  3) Explain what they do and why

If most of the elements in the matrix are zero then the matrix is called a sparse matrix. It is wasteful to store the zero elements in the matrix since they do not affect the results of our computation. This is why we implement these matrices in more efficient representations than the standard 2D Array. Using more efficient representations we can cut down space and time complexities of operations significantly without changing the main matrix[2]. We are disscusing the following example step by step.

At first, we generate a 3 $\times$ 6 matrix and store in A. We are seeing that most of the elements of A are zero. We can create CSR matrix for A.

```python
A = np.array([[1, 0, 0, 1, 0, 0], [0, 0, 2, 0, 0, 1], [0, 0, 0, 2, 0, 0]])
print(A)
```

    [[1 0 0 1 0 0]
     [0 0 2 0 0 1]
     [0 0 0 2 0 0]]

To create CSR matrix, we used csr_matrix function from numpy.

```python
# convert to sparse matrix (CSR method)
S = csr_matrix(A)
print(S)
```

      (0, 0)	1
      (0, 3)	1
      (1, 2)	2
      (1, 5)	1
      (2, 3)	2

From the result we can see that there are 5 items [consider in row] with value.
The row and position start from 0.

The 1. item is in row 0 position 0 and has the value 1.

The 2. item is in row 0 position 3 and has the value 1.

The 3. item is in row 1 position 2 and has the value 2.

The 4. item is in row 1 position 5 and has the value 1.

The 5. item is in row 2 position 3 and has the value 2.

Therefore, this matrix S will take less space and reduce the time of computation.

We can reconstruct the orginal matrix A from S. For that we have used todense() function.

```python
# reconstruct dense matrix
B = S.todense()
print(B)
```

    [[1 0 0 1 0 0]
     [0 0 2 0 0 1]
     [0 0 0 2 0 0]]

#### Refs:
[1] Bourbaki, Nicolas (1987) [1981]. Topological Vector Spaces: Chapters 1–5 [Sur certains espaces vectoriels topologiques]. Annales de l'Institut Fourier. Elements of mathematics. 2. Translated by Eggleston, H.G.; Madan, S. Berlin New York: Springer-Verlag. ISBN 978-3-540-42338-6.

[2] Pissanetzky, Sergio (1984). Sparse Matrix Technology. Academic Press.

For the code please click the following link:

https://github.com/hafizurr/Machine_Learnig_course/blob/master/home_work/Homework_4.ipynb