# Coordinate systems

## Introduction

This notebooks provides examples of curvilinear coordinates. Particularly,
it presents the coordinate surfaces for three common coordinate systems.

### Use of PyVista in this notebook

We use `PyVista` to represent the coordinate surfaces and `ìtkwidgets`
to render them interactively in the notebook.

If installing things manually, the following would be required

    conda install -c conda-forge pyvista
    conda install -c conda-forge itkwidgets

**More information:** https://docs.pyvista.org/examples/index.html

```python
import numpy as np
import pyvista as pv
from itkwidgets import view
```

```python
red = (0.9, 0.1, 0.1)
blue = (0.2, 0.5, 0.7)
green = (0.3, 0.7, 0.3)
```

## Cylindrical coordinates

The [ISO standard 80000-2](https://en.wikipedia.org/wiki/ISO_80000-2) recommends the use of $(ρ, φ, z)$, where $ρ$ is the radial coordinate, $\varphi$ the azimuth, and $z$ the height.

For the conversion between cylindrical and Cartesian coordinates, it is convenient to assume that the reference plane of the former is the Cartesian $xy$-plane (with equation $z=0$, and the cylindrical axis is the Cartesian $z$-axis. Then the $z$-coordinate is the same in both systems, and the correspondence between cylindrical $(\rho, \varphi)$ and Cartesian $(x, y)$ are the same as for polar coordinates, namely

\begin{align}
x &= \rho \cos \varphi \\
y &= \rho \sin \varphi
\end{align}

in one direction, and

\begin{align}
    \rho &= \sqrt{x^2+y^2} \\
    \varphi &= \begin{cases}
        0 & \mbox{if } x = 0 \mbox{ and } y = 0\\
        \arcsin\left(\frac{y}{\rho}\right) & \mbox{if } x \geq 0 \\
        \arctan\left(\frac{y}{x}\right) & \mbox{if } x > 0 \\
        -\arcsin\left(\frac{y}{\rho}\right) + \pi & \mbox{if } x < 0
    \end{cases}
\end{align}

in the other. It is also common to use [$\varphi = \mathrm{atan2}(y, x)$](https://en.wikipedia.org/wiki/Atan2).

```python
# Cylinder
phi, z = np.mgrid[0:2*np.pi:31j, -2:2*np.pi:31j]
x = 2*np.cos(phi)
y = 2*np.sin(phi)
cylinder = pv.StructuredGrid(x, y, z)
```

```python
# Vertical plane
rho, z = np.mgrid[0:3:31j, -2:2*np.pi:31j]
x = rho*np.cos(np.pi/4)
y = rho*np.sin(np.pi/4)
vert_plane = pv.StructuredGrid(x, y, z)
```

```python
# Horizontal plane
rho, phi = np.mgrid[0:3:31j, 0:2*np.pi:31j]
x = rho*np.cos(phi)
y = rho*np.sin(phi)
z = np.ones_like(x)
hor_plane = pv.StructuredGrid(x, y, z)
```

```python
view(geometries=[cylinder, vert_plane, hor_plane],
     geometry_colors=[blue, red, green])
```

    Viewer(geometries=[{'vtkClass': 'vtkPolyData', 'points': {'vtkClass': 'vtkPoints', 'name': '_points', 'numberO…

## Spherical coordinates

The use of $(r, θ, φ)$ to denote radial distance, inclination (or elevation), and azimuth, respectively, is common practice in physics, and is specified by [ISO standard 80000-2](https://en.wikipedia.org/wiki/ISO_80000-2).

The spherical coordinates of a point can be obtained from its [Cartesian coordinate system](https://en.wikipedia.org/wiki/ISO_80000-2) $(x, y, z)$

\begin{align}
r&=\sqrt{x^2 + y^2 + z^2} \\
\theta &= \arccos\frac{z}{\sqrt{x^2 + y^2 + z^2}} = \arccos\frac{z}{r} \\
\varphi &= \arctan \frac{y}{x}
\end{align}

The inverse tangent denoted in $\varphi = \arctan\left(\frac{y}{x}\right)$ must be suitably defined , taking into account the correct quadrant of $(x, y)$ (using the function ``atan2``).

Conversely, the Cartesian coordinates may be retrieved from the spherical coordinates (_radius_ $r$, _inclination_ $\theta$, _azimuth_ $\varphi$), where $r \in [0, \infty)$, $\theta \in [0, \pi]$ and $\varphi \in [0, 2\pi)$, by:

\begin{align}
x&=r \, \sin\theta \, \cos\varphi \\
y&=r \, \sin\theta \, \sin\varphi \\
z&=r \, \cos\theta
\end{align}

```python
theta, phi = np.mgrid[0:np.pi:21j, 0:np.pi:21j]
```

```python
# Sphere
x = np.sin(phi) * np.cos(theta)
y = np.sin(phi) * np.sin(theta)
z = np.cos(phi)
sphere = pv.StructuredGrid(x, y, z)
sphere2 =  pv.StructuredGrid(-x, -y, z)
```

```python
# Cone
x = theta/3 * np.cos(phi)
y = theta/3 * np.sin(phi)
z = theta/3
cone = pv.StructuredGrid(x, y, z)
cone2 = pv.StructuredGrid(-x, -y, z)
```

```python
# Plane
x = theta/np.pi
y = theta/np.pi
z = phi - np.pi/2
plane = pv.StructuredGrid(x, y, z)
```

```python
view(geometries=[sphere, sphere2, cone, cone2, plane],
     geometry_colors=[blue, blue, red, red, green],
     geometry_opacities=[1.0, 0.5, 1.0, 0.5, 1.0])
```

    Viewer(geometries=[{'vtkClass': 'vtkPolyData', 'points': {'vtkClass': 'vtkPoints', 'name': '_points', 'numberO…

## Ellipsoidal coordinates

```python
v, theta = np.mgrid[0:np.pi/2:21j, 0:np.pi:21j]
a = 3
b = 2
c = 1
```

```python
# Ellipsoid
lam = 3
x =  np.sqrt(a**2  + lam) * np.cos(v) * np.cos(theta)
y = np.sqrt(b**2  + lam)* np.cos(v) * np.sin(theta)
z = np.sqrt(c**2 + lam) * np.sin(v)
ellipsoid = pv.StructuredGrid(x, y, z)
ellipsoid2 = pv.StructuredGrid(x, y, -z)
ellipsoid3 = pv.StructuredGrid(x, -y, z)
ellipsoid4 = pv.StructuredGrid(x, -y, -z)
```

```python
# Hyperboloid of one sheet
mu = 2
x = np.sqrt(a**2 + mu) * np.cosh(v) * np.cos(theta)
y = np.sqrt(b**2 + mu) * np.cosh(v) * np.sin(theta)
z = np.sqrt(c**2 + mu) * np.sinh(v)
z = np.sqrt(c**2 + mu) * np.sinh(v)
hyper = pv.StructuredGrid(x, y, z)
hyper2 = pv.StructuredGrid(x, y, -z)
hyper3 = pv.StructuredGrid(x, -y, z)
hyper4 = pv.StructuredGrid(x, -y, -z)
```

```python
# Hyperboloid of two sheets
nu = 1
x = np.sqrt(a**2 + nu) * np.cosh(v)
y = np.sqrt(c**2 + nu) * np.sinh(v) * np.sin(theta)
z = np.sqrt(b**2 + nu) * np.sinh(v) * np.cos(theta)
hyper_up = pv.StructuredGrid(x, y, z)
hyper_down = pv.StructuredGrid(-x, y, z)
hyper_up2 = pv.StructuredGrid(x, -y, z)
hyper_down2 = pv.StructuredGrid(-x, -y, z)
```

```python
view(geometries=[ellipsoid, ellipsoid2, ellipsoid3, ellipsoid4,
                 hyper, hyper2, hyper3, hyper4,
                 hyper_up, hyper_down, hyper_up2, hyper_down2],
     geometry_colors=[blue]*4 + [red]*4 + [green]*4)
```

    Viewer(geometries=[{'vtkClass': 'vtkPolyData', 'points': {'vtkClass': 'vtkPoints', 'name': '_points', 'numberO…

## References

1. Wikipedia contributors. ["Cylindrical coordinate system."](https://en.wikipedia.org/wiki/Cylindrical_coordinate_system) Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 12 Dec. 2016. Web. 9 Feb. 2017.

2. Wikipedia contributors. ["Spherical coordinate system."](https://en.wikipedia.org/wiki/Spherical_coordinate_system) Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 29 Jan. 2017. Web. 9 Feb. 2017.

3. Sullivan et al., (2019). [PyVista: 3D plotting and mesh analysis through a streamlined interface for the Visualization Toolkit (VTK)](https://joss.theoj.org/papers/10.21105/joss.01450). Journal of Open Source Software, 4(37), 1450, https://doi.org/10.21105/joss.01450

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>
<style>
/* Based on Lorena Barba template available at: https://github.com/barbagroup/AeroPython/blob/master/styles/custom.css*/
@font-face {
font-family: "Computer Modern";
src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}
div.cell{
width:800px;
margin-left:16% !important;
margin-right:auto;
}
h1 {
font-family: 'Alegreya Sans', sans-serif;
}
h2 {
font-family: 'Fenix', serif;
}
h3{
font-family: 'Fenix', serif;
margin-top:12px;
margin-bottom: 3px;
}
h4{
font-family: 'Fenix', serif;
}
h5 {
font-family: 'Alegreya Sans', sans-serif;
}
div.text_cell_render{
font-family: 'Alegreya Sans',Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
line-height: 135%;
font-size: 120%;
width:600px;
margin-left:auto;
margin-right:auto;
}
.CodeMirror{
font-family: "Source Code Pro";
font-size: 90%;
}
/* .prompt{
display: None;
}*/
.text_cell_render h1 {
font-weight: 200;
font-size: 50pt;
line-height: 100%;
color:#CD2305;
margin-bottom: 0.5em;
margin-top: 0.5em;
display: block;
}
.text_cell_render h5 {
font-weight: 300;
font-size: 16pt;
color: #CD2305;
font-style: italic;
margin-bottom: .5em;
margin-top: 0.5em;
display: block;
}
.warning{
color: rgb( 240, 20, 20 )
}
</style>

```python

```


Solution to: [Day 2: More Dice](https://www.hackerrank.com/challenges/s10-mcq-2/problem)

Table of Contents

- Table of Contents
- Sample question
- Math Solution
- Monte Carlo Solution
    - Imports
    - Probability of unique dice rolls at target
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Sample question
Find the probability of getting 1 head and 1 tail when 2 fair coins are tossed.

Given the above question, we can extract the following:
- Experiment: tossing 2 coins.
- Sample space (S): The possible outcomes for the toss of 1 coin are {H, T}.

As our experiment tosses 2 coins, we have to consider all possible toss outcomes by finding the Cartesian Product of the possible outcomes for each coin:

\begin{equation}
\normalsize
P(S) = \{H, T\} * \{H * T\}
\end{equation}

\begin{equation}
\normalsize
P(S) = \text{{(H, H), (H, T), (T, H), (T, T)}} = 4
\end{equation}

- Event (A n B): that the outcome of 1 toss will be H, and the outcome of the other toss will be T (i.e., P(A) = {(H, T), (T, H)}}

\begin{equation}
\normalsize
P(A) = \frac
{\text{number of favorable events}}
{\text{probabiliy space}}
\end{equation}

\begin{equation}
\normalsize
= \frac{P(A)}{P(S)} = \frac{2}{4} = \frac{1}{2}
\end{equation}

# Math Solution
- Experiment: Roll 2 die and find the probability that sum == 6 with different die faces
- Sample Space: possible outcomes == 36
- Event: P( (A != B) n (A + B == 6))
- Formula:

\begin{equation}
\large
\frac{P(\text{number favorable events})}{P(\text{total number of events})} = \frac{P(A)}{P(S)}
\end{equation}

\begin{equation}
\large
P(A) =
\frac{P(1,5) + P(2, 4) + P(4, 2) + P(1, 5)}
{P(S)}
\end{equation}

\begin{equation}
\large
= \frac{1 + 1 + 1 + 1}{36}
\end{equation}

\begin{equation}
\large
= \frac{4}{36}
\end{equation}

\begin{equation}
\large
= \frac{1}{9}
\end{equation}

\begin{equation}
\large
= 0.111
\end{equation}

# Monte Carlo Solution

## Imports

```python
from typing import List
```

## Probability of unique dice rolls at target

```python
def get_all_possible_combos(num_dice: int, dice_face: int, target: int) -> List[List[int]]:
	"""Returns Cartesian product of all possible dice rolls.

	Uses two helper functions:
		- get_possible_rolls, generator func to yield all die rolls.
		- recurse_roll_combos, helper recursion to create combinations for `num_dice`
	"""
	def get_possible_rolls(dice_face: int) -> int:
		"""Generator funct to yield possible rolls."""
		for i in range(1, dice_face + 1): yield i

	def recurse_roll_combos(processed_dice: int, all_roll_combos: list) -> List[List[int]]:
		"""Helper func to recurse through all dice."""
		if processed_dice == num_dice:		## Base case
			return all_roll_combos

		new_roll_combos = []
		for roll in all_roll_combos:
			roll_combo = [roll + [val] for val in get_possible_rolls(dice_face)]
			new_roll_combos += (roll_combo)

		processed_dice += 1
		return recurse_roll_combos(processed_dice= processed_dice, all_roll_combos = new_roll_combos)

	first_rolls = [[val] for val in get_possible_rolls(dice_face)]
	roll_combinations = recurse_roll_combos( 1, first_rolls)

	assert (len(roll_combinations) == dice_face ** num_dice)
	return roll_combinations
```

## Main

```python
def main():
	num_dice = 2
	dice_face = 6
	target = 6

	## Get Combinations
	roll_combos = get_all_possible_combos(num_dice, dice_face, target)

	## Count unique combos in target
	num_at_target = 0
	for combo in roll_combos:

		flag_unique = True
		for c in combo:
			if combo.count(c) > 1:
				flag_unique = False
				break

		if flag_unique and sum(combo) == target:
			num_at_target += 1

	## Return P(A) / P(S)
	prob = num_at_target / len(roll_combos)
	print( round(prob, 3))
```

```python
if __name__ == "__main__":
	main()
```

    0.111


### Example 4: Burgers' equation

Now that we have seen how to construct the non-linear convection and diffusion examples, we can combine them to form Burgers' equations. We again create a set of coupled equations which are actually starting to form quite complicated stencil expressions, even if we are only using a low-order discretisations.

Let's start with the definition fo the governing equations:
$$ \frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} + v \frac{\partial u}{\partial y} = \nu \; \left(\frac{\partial ^2 u}{\partial x^2} + \frac{\partial ^2 u}{\partial y^2}\right)$$

$$ \frac{\partial v}{\partial t} + u \frac{\partial v}{\partial x} + v \frac{\partial v}{\partial y} = \nu \; \left(\frac{\partial ^2 v}{\partial x^2} + \frac{\partial ^2 v}{\partial y^2}\right)$$

The discretized and rearranged form then looks like this:

\begin{aligned}
u_{i,j}^{n+1} &= u_{i,j}^n - \frac{\Delta t}{\Delta x} u_{i,j}^n (u_{i,j}^n - u_{i-1,j}^n) - \frac{\Delta t}{\Delta y} v_{i,j}^n (u_{i,j}^n - u_{i,j-1}^n) \\
&+ \frac{\nu \Delta t}{\Delta x^2}(u_{i+1,j}^n-2u_{i,j}^n+u_{i-1,j}^n) + \frac{\nu \Delta t}{\Delta y^2} (u_{i,j+1}^n - 2u_{i,j}^n + u_{i,j+1}^n)
\end{aligned}

\begin{aligned}
v_{i,j}^{n+1} &=  v_{i,j}^n - \frac{\Delta t}{\Delta x} u_{i,j}^n (v_{i,j}^n - v_{i-1,j}^n) - \frac{\Delta t}{\Delta y} v_{i,j}^n (v_{i,j}^n - v_{i,j-1}^n) \\
&+ \frac{\nu \Delta t}{\Delta x^2}(v_{i+1,j}^n-2v_{i,j}^n+v_{i-1,j}^n) + \frac{\nu \Delta t}{\Delta y^2} (v_{i,j+1}^n - 2v_{i,j}^n + v_{i,j+1}^n)
\end{aligned}

Great. Now before we look at the Devito implementation, let's re-create the NumPy-based implementation form the original.

```python
from examples.cfd import plot_field, init_hat
import numpy as np
%matplotlib inline

# Some variable declarations
nx = 41
ny = 41
nt = 120
c = 1
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .0009
nu = 0.01
dt = sigma * dx * dy / nu
```

```python
#NBVAL_IGNORE_OUTPUT

# Assign initial conditions
u = np.empty((nx, ny))
v = np.empty((nx, ny))

init_hat(field=u, dx=dx, dy=dy, value=2.)
init_hat(field=v, dx=dx, dy=dy, value=2.)

plot_field(u)
```

```python
#NBVAL_IGNORE_OUTPUT

for n in range(nt + 1): ##loop across number of time steps
    un = u.copy()
    vn = v.copy()

    u[1:-1, 1:-1] = (un[1:-1, 1:-1] -
                     dt / dx * un[1:-1, 1:-1] *
                     (un[1:-1, 1:-1] - un[1:-1, 0:-2]) -
                     dt / dy * vn[1:-1, 1:-1] *
                     (un[1:-1, 1:-1] - un[0:-2, 1:-1]) +
                     nu * dt / dx**2 *
                     (un[1:-1,2:] - 2 * un[1:-1, 1:-1] + un[1:-1, 0:-2]) +
                     nu * dt / dy**2 *
                     (un[2:, 1:-1] - 2 * un[1:-1, 1:-1] + un[0:-2, 1:-1]))

    v[1:-1, 1:-1] = (vn[1:-1, 1:-1] -
                     dt / dx * un[1:-1, 1:-1] *
                     (vn[1:-1, 1:-1] - vn[1:-1, 0:-2]) -
                     dt / dy * vn[1:-1, 1:-1] *
                    (vn[1:-1, 1:-1] - vn[0:-2, 1:-1]) +
                     nu * dt / dx**2 *
                     (vn[1:-1, 2:] - 2 * vn[1:-1, 1:-1] + vn[1:-1, 0:-2]) +
                     nu * dt / dy**2 *
                     (vn[2:, 1:-1] - 2 * vn[1:-1, 1:-1] + vn[0:-2, 1:-1]))

    u[0, :] = 1
    u[-1, :] = 1
    u[:, 0] = 1
    u[:, -1] = 1

    v[0, :] = 1
    v[-1, :] = 1
    v[:, 0] = 1
    v[:, -1] = 1

plot_field(u)
```

Nice, our wave looks just like the original. Now we shall attempt to write our entire Burgers' equation operator in a single cell - but before we can demonstrate this, there is one slight problem.

The diffusion term in our equation requires a second-order space discretisation on our velocity fields, which we set through the `TimeFunction` constructor for $u$ and $v$. The `TimeFunction` objects will store this dicretisation information and use it as default whenever we use the shorthand notations for derivative, like `u.dxl` or `u.dyl`. For the advection term, however, we want to use a first-order discretisation, which we now have to create by hand when combining terms with different stencil discretisations. To illustrate let's consider the following example:

```python
from devito import Grid, TimeFunction, first_derivative, left

grid = Grid(shape=(nx, ny), extent=(2., 2.))
x, y = grid.dimensions
t = grid.stepping_dim

u1 = TimeFunction(name='u1', grid=grid, space_order=1)
print("Space order 1:\n%s\n" % u1.dxl)

u2 = TimeFunction(name='u2', grid=grid, space_order=2)
print("Space order 2:\n%s\n" % u2.dxl)

# We use u2 to create the explicit first-order derivative
u1_dx = first_derivative(u2, dim=x, side=left, order=1)
print("Explicit space order 1:\n%s\n" % u1_dx)
```

    Space order 1:
    u1(t, x, y)/h_x - u1(t, x - h_x, y)/h_x

    Space order 2:
    1.5*u2(t, x, y)/h_x + 0.5*u2(t, x - 2*h_x, y)/h_x - 2.0*u2(t, x - h_x, y)/h_x

    Explicit space order 1:
    u2(t, x, y)/h_x - u2(t, x - h_x, y)/h_x

Ok, so by constructing derivative terms explicitly we again have full control of the spatial discretisation - the power of symbolic computation. Armed with that trick, we can now build and execute our advection-diffusion operator from scratch in one cell.

```python
#NBVAL_IGNORE_OUTPUT
from sympy import solve
from devito import Operator, Constant, Eq, INTERIOR

# Define our velocity fields and initialise with hat function
u = TimeFunction(name='u', grid=grid, space_order=2)
v = TimeFunction(name='v', grid=grid, space_order=2)
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=v.data[0], dx=dx, dy=dy, value=2.)

# Write down the equations with explicit backward differences
a = Constant(name='a')
u_dx = first_derivative(u, dim=x, side=left, order=1)
u_dy = first_derivative(u, dim=y, side=left, order=1)
v_dx = first_derivative(v, dim=x, side=left, order=1)
v_dy = first_derivative(v, dim=y, side=left, order=1)
eq_u = Eq(u.dt + u*u_dx + v*u_dy, a*u.laplace, region=INTERIOR)
eq_v = Eq(v.dt + u*v_dx + v*v_dy, a*v.laplace, region=INTERIOR)

# Let SymPy rearrange our stencils to form the update expressions
stencil_u = solve(eq_u, u.forward)[0]
stencil_v = solve(eq_v, v.forward)[0]
update_u = Eq(u.forward, stencil_u)
update_v = Eq(v.forward, stencil_v)

# Create Dirichlet BC expressions using the low-level API
bc_u = [Eq(u.indexed[t+1, 0, y], 1.)]  # left
bc_u += [Eq(u.indexed[t+1, nx-1, y], 1.)]  # right
bc_u += [Eq(u.indexed[t+1, x, ny-1], 1.)]  # top
bc_u += [Eq(u.indexed[t+1, x, 0], 1.)]  # bottom
bc_v = [Eq(v.indexed[t+1, 0, y], 1.)]  # left
bc_v += [Eq(v.indexed[t+1, nx-1, y], 1.)]  # right
bc_v += [Eq(v.indexed[t+1, x, ny-1], 1.)]  # top
bc_v += [Eq(v.indexed[t+1, x, 0], 1.)]  # bottom

# Create the operator
op = Operator([update_u, update_v] + bc_u + bc_v)

# Execute the operator for a number of timesteps
op(time=nt, dt=dt, a=nu)

plot_field(u.data[0])
```


# Think Bayes

This notebook presents example code and exercise solutions for Think Bayes.

Copyright 2018 Allen B. Downey

MIT License: https://opensource.org/licenses/MIT

```python
# Configure Jupyter so figures appear in the notebook
%matplotlib inline

# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import classes from thinkbayes2
from thinkbayes2 import Hist, Pmf, Suite
```

**Exercise:** Let's consider [a more general version of the Monty Hall problem](https://en.wikipedia.org/wiki/Monty_Hall_problem#Other_host_behaviors) where Monty is more unpredictable.  As before, Monty never opens the door you chose (let's call it A) and never opens the door with the prize.  So if you choose the door with the prize, Monty has to decide which door to open.  Suppose he opens B with probability `p` and C with probability `1-p`.

1.  If you choose A and Monty opens B, what is the probability that the car is behind A, in terms of `p`?

2.  What if Monty opens C?

Hint: you might want to use SymPy to do the algebra for you.

```python
from sympy import symbols
p = symbols('p')
```

    p

```python
# Solution

# Here's the solution if Monty opens B.

pmf = Pmf('ABC')
pmf['A'] *= p
pmf['B'] *= 0
pmf['C'] *= 1
pmf.Normalize()
pmf['A'].simplify()
```

    1.0*p/(p + 1)

```python
# Solution

# When p=0.5, the result is what we saw before

pmf['A'].evalf(subs={p:0.5})
```

    0.333333333333333

```python
# Solution

# When p=0.0, we know for sure that the prize is behind C

pmf['C'].evalf(subs={p:0.0})
```

    1.00000000000000

```python
# Solution

# And here's the solution if Monty opens C.

pmf = Pmf('ABC')
pmf['A'] *= 1-p
pmf['B'] *= 1
pmf['C'] *= 0
pmf.Normalize()
pmf['A'].simplify()
```

    0.333333333333333*(p - 1)/(0.333333333333333*p - 0.666666666666667)


# Week10
## Random Number Generators

### Exercise 1) Linear Congrugential Generators

In the lecture notes of L17, we showed several examples of LCGs. You will now be tasked with creating a more general framework for creating and using LCGs in an object-oriented manner.

Recall that an LCG generates a new pseudorandom number according to the formula

$$ X_{n+1} = a\cdot X_n + c \mod m.$$

#### 1a) Defining the class

(In Python) Define a class `LinearCongrugentialGenerator`. The class should have a constructor that takes the *seed* as input, and sets the state of the pRNG to be this seed.

The constructor should also take the parameters $a$, $c$ and $m$ as keyword arguments and store these in the class. As the default values, choose one of the LCG commonly used parameters found in this [table](https://en.wikipedia.org/wiki/Linear_congruential_generator#Parameters_in_common_use) on wikipedia. You can for example choose the one used by glibc, i.e., the gcc compiler.

#### 1b) Adding a call method

Create a `__call__` special method that advances the state of the pRNG by one number, and returns this new number.

Test your method by implementing an instance of your pRNG and producing ten numbers.

#### 1c) Random floats

Add a new method called `rand()` that should be called with no input. The function should return a random floating point number on the interval $[0, 1)$.

Hint: To do this, the method should call on the `__call__` special method to advance the state and produce a random integer. This integer then needs to be scaled to a proper float by dividing it by some value.

#### 1d) Uniform

Add a new method called `uniform(a, b)` that takes two numbers in: $a$ and $b$ and returns a uniformly distributed floating point number on the interval $[a, b)$.

Hint, you can first call `rand()` to get a number on the range $[0, 1)$, and then scale this number by multiplying and shifting it.

#### 1e) Randint

Now add a method called `randint(a, b)` (short for random integer). That should return a uniformly distributed integer on the interval $[a, b]$.

Test your function by throwing 1000 dice, and computing the average result, which should be close to 3.5.

#### 1f) Normally Distributed Numbers

Now add a method `normal()` that should return a normally distributed number with standard deviation 1 and a mean of 0.

To compute a normally distributed number, use the Box-Muller metho, as described in the lecture notes. To do this, you need two uniformly distributed numbers on the interval $[0, 1)$: $U_1$ and $U_2$. Then you can compute the two independent normally distributed numbers based on the formulas:

$$\begin{align}
Z_1 = \sqrt{-2 \ln U_1}\cos (2\pi U_2), \\
Z_2 = \sqrt{-2 \ln U_1}\sin (2\pi U_2).
\end{align}$$

Note that to compute and return *one* normally distributed number, you can simply draw two numbers with `rand()`, and compute $Z_1$, and ignore $Z_2$.

However, slightly more efficient, is to compute both values, and then use the other number the next time `normal()` is called.  See if you can find a nice way to do this, and if not, simply implement the simpler solution of only computing $Z_1$.

### Exercise 2

#### 2a) Implementing RANDU

Use your `LinearCongrugentialGenerator` class to implement a RANDU pRNG. RANDU is a LCG with $a = 65539$, $c=0$ and $m=2^{31}.$$

#### 2b) - Checking the mean of the generated numbers

As we have discussed in the lectures. For a pRNG to produce numbers that "look" random, they have to reproduce certain statistical properties. One of these is the *mean*. If we are drawing numbers on the interval $[0, 1)$, then the average over a large number of random numbers, i.e., the *sample mean*, should tend to exactly 0.5.

Generate $n=10^6$ samples from your RANDU class and find the sample mean. Does RANDU reproduce the expected mean?

#### 2c) Checking the sample variance

The random numbers should, in addition to respecting the mean, reproduce the variance of the distribution. For random numbers drawn from a uniform distribution between 0 and 1, this variance should be $1/12$.

Using your $n=10^6$ generated samples, check that the sample variance is reasonable.

#### 2d) Moments of the pdf (Only for those who have taken or are taking STK1100)

We are attempting to draw random numbers from the probability density function

$$f(x) = \begin{cases}
1 & \mbox{ if } 0 \leq x < 1, \\
0 & \mbox{else}.
\end{cases}$$

For any pdf, the *moments* of the sample are defined as:
$$E(x^k) = \int_{-\infty}^\infty x^k \cdot f(x) \ {\rm d}x.$$

Show that for our given pdf, that the $k$-th moment can be written as
$$E(X^k) = \frac{1}{k+1}.$$

From this, use that fact that the variance can be written as
$${\rm Var}(X) = E(X^2) - E(X)^2,$$
to show that the variance of the pdf is 1/12.

### Plotting the correlation found in RANDU

We have seen that RANDU reproduces the mean and the variance of its distribution, which is good. However, there are other statistical properties it also needs to follow. One of these is that different samples from the distribution, i.e., different generated numbers, should be *uncorrelated*. It turns out that RANDU breaks this requirement, horribly. This is very briefly shown in L17.

A common way to show how bad RANDU actually is, is to plot random points within a three dimensional unit cube. A unit cube is a cube with sides that goes from 0 to 1 in all three dimensions. We can place a randomly located point inside the cube by drawing three random numbers in the $[0, 1)$ range, one for each of the three coordinates $x$, $y$ and $z$. If all three coordinates are chosen randomly *and* independent of each other, the random point will have an equal likelihood of landing anywhere within the cube. For RANDU however, if we plot this out, we see that these "random" points aren't distributed uniformly throughout the cube at all, but rather all land in specific planes. Let us produce such a plot.

#### 2e) Drawing random points

Use your RANDU-generator to generate $n=1000$ such points. Note that for this to work you need to draw the three coordinates for each point consecutively, you cannot first draw all the x-positions, then all the y-positions for example.

#### 2f) A three dimensional scatter plot

Now, plot your $n=1000$ points in a 3D scatter plot. If you do not know how to draw a 3D scatter plot. Take a look at this [matplotlib example script](https://matplotlib.org/2.1.1/gallery/mplot3d/scatter3d.html).

Once you have your scatter plot, the points might *look* uniformly distributed. To properly see that they lie in planes, we need to look at it from the right angle. If you are plotting outside Jupyter, the plot window should be interactive and so you can simply drag the view around untill you find a good angle. If you are plotting inside Jupyter you can use `ax.view_init(elevation, method)` before you show the plot. For example `view_init(30, 60)` should be a good angle. At least for me.

Increase the number of points to $n=10000$ to really make the planes apparent.

#### 2g) A proper uniformly unit cube

Repeat the plot, but this time replace your RANDU generater with `numpy.random.rand()` which produces floats in the range $[0, 1)$ based on the Mersenne Twister algorithm. Verify that no planes are visible in the cube in this case.

### Exercise 3) Randomness in C++

Before you do this exercise, be sure to read about generating random numbers in C++ in the lecture notes, or watch this video:
* https://channel9.msdn.com/Events/GoingNative/2013/rand-Considered-Harmful

For this exercise, the C++ reference is a good resource.

#### Exercise 3a) Uniform numbers
Create a C++ script that uses the Mersenne Twister algorithm to produce 10 uniformly distributed numbers in the range 0 and 1

#### Exercise 3b) Normally distributed numbers

Use a different distribution to produce normally distributed numbers.

### Exercise 4) The Birthday Problem

The Birthday Problem concerns the probabilty that two or more people share a birthday in a room with $n$ people. This is also a good test of an RNG, because an RNG of truley uncorrelated numbers should produce duplicates with a given frequency.

#### 4a) Drawing a random birthday

Use [`numpy.random.randint`](https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.random.randint.html) (click for the offical reference) to draw a birthday as a random integer in the range $[1, 366]$. We have 366 possibilities because we include leap years.

#### 4b) Drawing $n$ random birthdays

Now use the `size` keyword to draw a whole set of random birthdays, to simulate the birthdays of $n$ randomly people located in the same room. As an example, let's say we are $n=20$ people in the same room.

#### 4c) Checking for duplicates

If the array we get from randint contains any *duplicate* values. Write a function for checking if any duplicates are contained in the array.

Hint: There are many ways to check for duplicates. You can for example do `len(np.unique(birthdays)` to check how many *unique* birthdays there are. Or you can use the fact that a Python *set* cannot contain any duplicates, and so on. Give it a go on your own, and if you cannot figure it out, google is sure to be helpful :)

#### 4d) Repeated Simulation

Now use a loop to repeat the experiment of drawing $n=23$ birthdays and checking wether there is a duplicate 1000 times. Count the number of times there is a duplicate, and the number of times there is not.

The probabilty should be close to 50/50 for 23 people, if the RNG returns the expected number of duplicates. Did you get a value close to 50/50?

#### 4e) Edge cases

For $n=1$, the probability of a duplicate is "obivously" 0. And for $n=366$ it is "obviously" 1. Why is this?

### Drawing out the full curve

The probability of a duplicate birthday can be found analytically, and produces the curve shown in this Figure:

Let's try to produce a similar function ourselves.

#### 4f) Drawing the Curve

Generalize your answer to 4D so that it instead finds the probability of $n$ people having a shared birthday.

#### 4g) Plotting the curve

For $n$ in the range $[0, 100]$, simulate 1000 cases and count the number of shared birthdays for each case. Divide by the number of simulations, $1000$, to find the probability of a shared birthday for each $n$

If you have managed to find the probability for each $n$, use `plt.step` to plot it. The function `plt.step` works just like `plt.plot`, but plots the data as a discontinous stepwise function, just like in the analytical function above.

#### 4h) Comparing the curves

Does your program look like the analytical function? Should it?

### More exercises

For more exercises, turn to Langtangen Chapter 8. We suggest the following:
* Exercise 8.8
* Exercise 8.9
* Exercise 8.19
* Exercise 8.21


```python
import sympy as sp
import numpy as np

print(f"SymPy Version: {sp.__version__}")

# 数式をキレイに表示する
sp.init_printing()

# 乱数の種の固定しておく
np.random.seed(123)
```

    SymPy Version: 1.8

### C言語のコードを生成する。

- C言語のコードは、`sympy.ccode()`で出力することができる。

```python
x, y, z = sp.symbols('x y z')
```

```python
sp.ccode(sp.exp(-x))
```

    'exp(-x)'

```python
distance = (x ** 2 + y ** 2 + z ** 2) ** 0.5
distance
```

```python
sp.ccode(distance)
```

    'sqrt(pow(x, 2) + pow(y, 2) + pow(z, 2))'

```python
x = sp.MatrixSymbol('x', 3, 1)

A = sp.Matrix(np.random.randint(15, size=(3, 3)))
A
```

$\displaystyle \left[\begin{matrix}14 & 13 & 14\\2 & 12 & 2\\6 & 1 & 3\end{matrix}\right]$

```python
Ax = A * sp.Matrix(x)
Ax
```

$\displaystyle \left[\begin{array}{c}14 x_{0, 0} + 13 x_{1, 0} + 14 x_{2, 0}\\2 x_{0, 0} + 12 x_{1, 0} + 2 x_{2, 0}\\6 x_{0, 0} + x_{1, 0} + 3 x_{2, 0}\end{array}\right]$

```python
n_rows = 3
for i in range(n_rows):
    code = sp.ccode(Ax[i], assign_to=f"y[{i}]")
    print(code)
```

    y[0] = 14*x[0] + 13*x[1] + 14*x[2];
    y[1] = 2*x[0] + 12*x[1] + 2*x[2];
    y[2] = 6*x[0] + x[1] + 3*x[2];

```python

```


# Basis for grayscale images

## Introduction

Consider the set of real-valued matrices of size $M\times N$; we can turn this into a vector space by defining addition and scalar multiplication in the usual way:

\begin{align}
\mathbf{A} + \mathbf{B} &=
    \left[
        \begin{array}{ccc}
            a_{0,0} & \dots & a_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    +
    \left[
        \begin{array}{ccc}
            b_{0,0} & \dots & b_{0,N-1} \\
            \vdots & & \vdots \\
            b_{M-1,0} & \dots & b_{M-1,N-1}
        \end{array}
    \right]
    \\
    &=
    \left[
        \begin{array}{ccc}
            a_{0,0}+b_{0,0} & \dots & a_{0,N-1}+b_{0,N-1} \\
            \vdots & & \vdots \\
            a_{M-1,0}+b_{M-1,0} & \dots & a_{M-1,N-1}+b_{M-1,N-1}
        \end{array}
    \right]
    \\ \\ \\
\beta\mathbf{A} &=
    \left[
        \begin{array}{ccc}
            \beta a_{0,0} & \dots & \beta a_{0,N-1} \\
            \vdots & & \vdots \\
            \beta a_{M-1,0} & \dots & \beta a_{M-1,N-1}
        \end{array}
    \right]
\end{align}

As a matter of fact, the space of real-valued $M\times N$ matrices is completely equivalent to $\mathbb{R}^{MN}$ and we can always "unroll" a matrix into a vector. Assume we proceed column by column; then the matrix becomes

$$
    \mathbf{a} = \mathbf{A}[:] = [
        \begin{array}{ccccccc}
            a_{0,0} & \dots & a_{M-1,0} & a_{0,1} & \dots & a_{M-1,1} & \ldots & a_{0, N-1} & \dots & a_{M-1,N-1}
        \end{array}]^T
$$

Although the matrix and vector forms represent exactly the same data, the matrix form allows us to display the data in the form of an image. Assume each value in the matrix is a grayscale intensity, where zero is black and 255 is white; for example we can create a checkerboard pattern of any size with the following function:

```python
# usual pyton bookkeeping...
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
import IPython
from IPython.display import Image
import math
```

```python
# ensure all images will be grayscale
plt.gray();
```

    <matplotlib.figure.Figure at 0x7f04b466a208>

```python
# let's create a checkerboard pattern
SIZE = 4
img = np.zeros((SIZE, SIZE))
for n in range(0, SIZE):
    for m in range(0, SIZE):
        if (n & 0x1) ^ (m & 0x1):
            img[n, m] = 255

# now display the matrix as an image
plt.matshow(img);
```

Given the equivalence between the space of $M\times N$ matrices and $\mathbb{R}^{MN}$ we can easily define the inner product between two matrices in the usual way:

$$
\langle \mathbf{A}, \mathbf{B} \rangle = \sum_{m=0}^{M-1} \sum_{n=0}^{N-1} a_{m,n} b_{m, n}
$$

(where we have neglected the conjugation since we'll only deal with real-valued matrices); in other words, we can take the inner product between two matrices as the standard inner product of their unrolled versions. The inner product allows us to define orthogonality between images and this is rather useful since we're going to explore a couple of bases for this space.

## Actual images

Conveniently, using IPython, we can read images from disk in any given format and convert them to numpy arrays; let's load and display for instance a JPEG image:

```python
img = np.array(plt.imread('cameraman.jpg'), dtype=int)
plt.matshow(img);
```

The image is a $64\times 64$ low-resolution version of the famous "cameraman" test picture. Out of curiosity, we can look at the first column of this image, which is is a $64×1$ vector:

```python
img[:,0]
```

    array([156, 157, 157, 152, 154, 155, 151, 157, 152, 155, 158, 159, 159,
           160, 160, 161, 155, 160, 161, 161, 164, 162, 160, 162, 158, 160,
           158, 157, 160, 160, 159, 158, 163, 162, 162, 157, 160, 114, 114,
           103,  88,  62, 109,  82, 108, 128, 138, 140, 136, 128, 122, 137,
           147, 114, 114, 144, 112, 115, 117, 131, 112, 141,  99,  97])

The values are integers between zero and 255, meaning that each pixel is encoded over 8 bits (or 256 gray levels).

## The canonical basis

The canonical basis for any matrix space $\mathbb{R}^{M\times N}$ is the set of "delta" matrices where only one element equals to one while all the others are 0. Let's call them $\mathbf{E}_n$ with $0 \leq n < MN$. Here is a function to create the canonical basis vector given its index:

```python
def canonical(n, M=5, N=10):
    e = np.zeros((M, N))
    e[(n % M), int(n / M)] = 1
    return e
```

Here are some basis vectors: look for the position of white pixel, which differentiates them and note that we enumerate pixels column-wise:

```python
plt.matshow(canonical(0));
plt.matshow(canonical(1));
plt.matshow(canonical(49));
```

## Transmitting images

Suppose we want to transmit the "cameraman" image over a communication channel. The intuitive way to do so is to send the  pixel values one by one, which corresponds to sending the coefficients of the decomposition of the image over the canonical basis. So far, nothing complicated: to send the cameraman image, for instance, we will send $64\times 64 = 4096$ coefficients in a row.

Now suppose that a communication failure takes place after the first half of the pixels have been sent. The received data will allow us to display an approximation of the original image only. If we replace the missing data with zeros, here is what we would see, which is not very pretty:

```python
# unrolling of the image for transmission (we go column by column, hence "F")
tx_img = np.ravel(img, "F")

# oops, we lose half the data
tx_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.reshape(tx_img, (64, 64), "F")
plt.matshow(rx_img);
```

Can we come up with a trasmission scheme that is more robust in the face of channel loss? Interestingly, the answer is yes, and it involves a different, more versatile basis for the space of images. What we will do is the following:

* describe the Haar basis, a new basis for the image space
* project the image in the new basis
* transmit the projection coefficients
* rebuild the image using the basis vectors

We know a few things: if we choose an orthonormal basis, the analysis and synthesis formulas will be super easy (a simple inner product and a scalar multiplication respectively). The trick is to find a basis that will be robust to the loss of some coefficients.

One such basis is the **Haar basis**. We cannot go into too many details in this notebook but, for the curious, a good starting point is [here](https://chengtsolin.wordpress.com/2015/04/15/real-time-2d-discrete-wavelet-transform-using-opengl-compute-shader/). Mathematical formulas aside, the Haar basis works by encoding the information in a *hierarchical* way: the first basis vectors encode the broad information and the higher coefficients encode the detail. Let's have a look.

First of all, to keep things simple, we will remain in the space of square matrices whose size is a power of two. The code to generate the Haar basis matrices is the following: first we generate a 1D Haar vector and then we obtain the basis matrices by taking the outer product of all possible 1D vectors (don't worry if it's not clear, the results are what's important):

```python
def haar1D(n, SIZE):
    # check power of two
    if math.floor(math.log(SIZE) / math.log(2)) != math.log(SIZE) / math.log(2):
        print("Haar defined only for lengths that are a power of two")
        return None
    if n >= SIZE or n < 0:
        print("invalid Haar index")
        return None

    # zero basis vector
    if n == 0:
        return np.ones(SIZE)

    # express n > 1 as 2^p + q with p as large as possible;
    # then k = SIZE/2^p is the length of the support
    # and s = qk is the shift
    p = math.floor(math.log(n) / math.log(2))
    pp = int(pow(2, p))
    k = SIZE / pp
    s = (n - pp) * k

    h = np.zeros(SIZE)
    h[int(s):int(s+k/2)] = 1
    h[int(s+k/2):int(s+k)] = -1
    # these are not normalized
    return h

def haar2D(n, SIZE=8):
    # get horizontal and vertical indices
    hr = haar1D(n % SIZE, SIZE)
    hv = haar1D(int(n / SIZE), SIZE)
    # 2D Haar basis matrix is separable, so we can
    #  just take the column-row product
    H = np.outer(hr, hv)
    H = H / math.sqrt(np.sum(H * H))
    return H
```

First of all, let's look at a few basis matrices; note that the matrices have positive and negative values, so that the value of zero will be represented as gray:

```python
plt.matshow(haar2D(0));
plt.matshow(haar2D(1));
plt.matshow(haar2D(10));
plt.matshow(haar2D(63));
```

We can notice two key properties

* each basis matrix has positive and negative values in some symmetric patter: this means that the basis matrix will implicitly compute the difference between image areas
* low-index basis matrices take differences between large areas, while high-index ones take differences in smaller **localized** areas of the image

We can immediately verify that the Haar matrices are orthogonal:

```python
# let's use an 8x8 space; there will be 64 basis vectors
# compute all possible inner product and only print the nonzero results
for m in range(0,64):
    for n in range(0,64):
        r = np.sum(haar2D(m, 8) * haar2D(n, 8))
        if r != 0:
            print("[%dx%d -> %f] " % (m, n, r), end="")
```

    [0x0 -> 1.000000] [1x1 -> 1.000000] [2x2 -> 1.000000] [3x3 -> 1.000000] [4x4 -> 1.000000] [5x5 -> 1.000000] [6x6 -> 1.000000] [7x7 -> 1.000000] [8x8 -> 1.000000] [9x9 -> 1.000000] [10x10 -> 1.000000] [11x11 -> 1.000000] [12x12 -> 1.000000] [13x13 -> 1.000000] [14x14 -> 1.000000] [15x15 -> 1.000000] [16x16 -> 1.000000] [16x17 -> -0.000000] [17x16 -> -0.000000] [17x17 -> 1.000000] [18x18 -> 1.000000] [19x19 -> 1.000000] [20x20 -> 1.000000] [21x21 -> 1.000000] [22x22 -> 1.000000] [23x23 -> 1.000000] [24x24 -> 1.000000] [24x25 -> -0.000000] [25x24 -> -0.000000] [25x25 -> 1.000000] [26x26 -> 1.000000] [27x27 -> 1.000000] [28x28 -> 1.000000] [29x29 -> 1.000000] [30x30 -> 1.000000] [31x31 -> 1.000000] [32x32 -> 1.000000] [33x33 -> 1.000000] [34x34 -> 1.000000] [35x35 -> 1.000000] [36x36 -> 1.000000] [37x37 -> 1.000000] [38x38 -> 1.000000] [39x39 -> 1.000000] [40x40 -> 1.000000] [41x41 -> 1.000000] [42x42 -> 1.000000] [43x43 -> 1.000000] [44x44 -> 1.000000] [45x45 -> 1.000000] [46x46 -> 1.000000] [47x47 -> 1.000000] [48x48 -> 1.000000] [49x49 -> 1.000000] [50x50 -> 1.000000] [51x51 -> 1.000000] [52x52 -> 1.000000] [53x53 -> 1.000000] [54x54 -> 1.000000] [55x55 -> 1.000000] [56x56 -> 1.000000] [57x57 -> 1.000000] [58x58 -> 1.000000] [59x59 -> 1.000000] [60x60 -> 1.000000] [61x61 -> 1.000000] [62x62 -> 1.000000] [63x63 -> 1.000000]

OK! Everything's fine. Now let's transmit the "cameraman" image: first, let's verify that it works

```python
# project the image onto the Haar basis, obtaining a vector of 4096 coefficients
# this is simply the analysis formula for the vector space with an orthogonal basis
tx_img = np.zeros(64*64)
for k in range(0, (64*64)):
    tx_img[k] = np.sum(img * haar2D(k, 64))

# now rebuild the image with the synthesis formula; since the basis is orthonormal
#  we just need to scale the basis matrices by the projection coefficients
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += tx_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

Cool, it works! Now let's see what happens if we lose the second half of the coefficients:

```python
# oops, we lose half the data
lossy_img = np.copy(tx_img);
lossy_img[int(len(tx_img)/2):] = 0

# rebuild matrix
rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

That's quite remarkable, no? We've lost the same amount of information as before but the image is still acceptable. This is because we lost the coefficients associated to the fine details of the image but we retained the "broad strokes" encoded by the first half.

Note that if we lose the first half of the coefficients the result is markedly different:

```python
lossy_img = np.copy(tx_img);
lossy_img[0:int(len(tx_img)/2)] = 0

rx_img = np.zeros((64, 64))
for k in range(0, (64*64)):
    rx_img += lossy_img[k] * haar2D(k, 64)

plt.matshow(rx_img);
```

In fact, schemes like this one are used in *progressive encoding*: send the most important information first and add details if the channel permits it. You may have experienced this while browsing the interned over a slow connection.

All in all, a great application of a change of basis!


# Penalised Regression

## YouTube Videos
1. **Scikit Learn Linear Regression:** https://www.youtube.com/watch?v=EvnpoUTXA0E
2. **Scikit Learn Linear Penalise Regression:** https://www.youtube.com/watch?v=RhsEAyDBkTQ

## Introduction
We often do not want the coefficients/ weights to be too large. Hence we append the loss function with a penalty function to discourage large values of $w$.

\begin{align}
\mathcal{L} & = \sum_{i=1}^N (y_i-f(x_i|w,b))^2 + \alpha \sum_{j=1}^D w_j^2 + \beta \sum_{j=1}^D |w_j|
\end{align}
where, $f(x_i|w,b) = wx_i+b$. The values of $\alpha$ and $\beta$ are positive (or zero), with higher values enforcing the weights to be closer to zero.

## Lesson Structure
1. The task of this lesson is to infer the weights given the data (observations, $y$ and inputs $x$).
2. We will be using the module `sklearn.linear_model`.

```python
from IPython.display import YouTubeVideo
YouTubeVideo("EvnpoUTXA0E")
```

```python
YouTubeVideo("RhsEAyDBkTQ")
```

```python
import numpy as np
from sklearn.linear_model import LinearRegression

import matplotlib.pyplot as plt
%matplotlib inline

# In order to reproduce the exact same number we need to set the seed for random number generators:
np.random.seed(1)
```

A normally distributed random looks as follows:

```python
e = np.random.randn(10000,1)
plt.hist(e,100) #histogram with 100 bins
plt.ylabel('y')
plt.xlabel('x')
plt.title('Histogram of Normally Distributed Numbers')
plt.show()
```

Generate observations $y$ given feature (design) matrix $X$ according to:
$$
y = Xw + \xi\\
\xi_i \sim \mathcal{N}(0,\sigma^2)
$$

In this particular case, $w$ is a 100 dimensional vector where 90% of the numbers are zero. i.e. only 10 of the numbers are non-zero.

```python
# Generate the data
N = 40 # Number of observations
D = 100 # Dimensionality

x = np.random.randn(N,D) # get random observations of x
w_true = np.zeros((D,1)) # create a weight vector of zeros
idx = np.random.choice(100,10,replace=False) # randomly choose 10 of those weights
w_true[idx] = np.random.randn(10,1) # populate then with 10 random weights

e = np.random.randn(N,1) # have a noise vector
y = np.matmul(x,w_true) + e # generate observations

# create validation set:
N_test = 50
x_test = np.random.randn(50,D)
y_test_true = np.matmul(x_test,w_true)
```

```python
model = LinearRegression()
model.fit(x,y)

# plot the true vs estimated coeffiecients
plt.plot(np.arange(100),np.squeeze(model.coef_))
plt.plot(np.arange(100),w_true)
plt.legend(["Estimated","True"])
plt.title('Estimated Weights')
plt.show()
```

One way of testing how good your model is to look at metrics. In the case of regression Mean Squared Error (MSE) is a common metric which is defined as:
$$ \frac{1}{N}\sum_{i=1}^N \xi_i^2$$ where, $\xi_i = y_i-f(x_i|w,b)$. Furthermore it is best to look at the MSE on a validation set, rather than on the training dataset that we used to train the model.

```python
y_est = model.predict(x_test)
mse = np.mean(np.square(y_test_true-y_est))
print(mse)
```

    6.808967330386871

Ridge regression is where you penalise the weights by setting the $\alpha$ parameter right at the top. It penalises it so that the higher **the square of the weights** the higher the loss.

```python
from sklearn.linear_model import Ridge

model = Ridge(alpha=5.0,fit_intercept = False)
model.fit(x,y)

# plot the true vs estimated coeffiecients
plt.plot(np.arange(100),np.squeeze(model.coef_))
plt.plot(np.arange(100),w_true)
plt.legend(["Estimated","True"])
plt.show()
```

This model is slightly better than without any penalty on the weights.

```python
y_est = model.predict(x_test)
mse = np.mean(np.square(y_test_true-y_est))
print(mse)
```

    6.422880725012181

Lasso is a model that encourages weights to go to zero exactly, as opposed to Ridge regression which encourages small weights.

```python
from sklearn.linear_model import Lasso

model = Lasso(alpha=0.1,fit_intercept = False)
model.fit(x,y)

# plot the true vs estimated coeffiecients
plt.plot(np.arange(100),np.squeeze(model.coef_))
plt.plot(np.arange(100),w_true)
plt.legend(["Estimated","True"])
plt.title('Lasso regression weight inference')
plt.show()
```

The MSE is significantly better than both the above models.

```python
y_est = model.predict(x_test)[:,None]
mse = np.mean(np.square(y_test_true-y_est))
print(mse)
```

    2.306001340842457

Automated Relevance Determination (ARD) regression is similar to lasso in that it encourages zero weights. However, the advantage is that you do not need to set a penalisation parameter, $\alpha$, $\beta$ in this model.

```python
from sklearn.linear_model import ARDRegression

model = ARDRegression(fit_intercept = False)
model.fit(x,y)

# plot the true vs estimated coeffiecients
plt.plot(np.arange(100),np.squeeze(model.coef_))
plt.plot(np.arange(100),w_true)
plt.legend(["Estimated","True"])
plt.show()
```

```python
y_est = model.predict(x_test)[:,None]
mse = np.mean(np.square(y_test_true-y_est))
print(mse)
```

    2.870211080521882

### Note:
Rerun the above with setting N=400

## Inverse Problems
The following section is optional and you may skip it. It is not necessary for understanding Deep Learning.

Inverse problems are where given the outputs you are required to infer the inputs. A typical example is X-rays. Given the x-ray sensor readings, the algorithm needs to build an image of an individuals bone structure.

See [here](http://scikit-learn.org/stable/auto_examples/applications/plot_tomography_l1_reconstruction.html#sphx-glr-auto-examples-applications-plot-tomography-l1-reconstruction-py) for an example of l1 reguralisation applied to a compressed sensing problem (has a resemblance to the x-ray problem).

```python

```

jupyter/Keras_TensorFlow_Course/Lesson 01 - PenalisedRegression - Solutions.ipynb


# SciPy
SciPy is a collection of mathematical algorithms and convenience functions. In this this notebook there are just a few examples of the features that are most important to us. But if you want to see all that SciPy has to offer, have a look at the [official documentation](https://docs.scipy.org/doc/scipy/reference/).

Since SciPy has several sublibraries, it is commom practice to import just the one we are going to use, as you'll in the following examples.

```python
import numpy as np
import matplotlib as mpl  # ignore this for now
import matplotlib.pyplot as plt  # ignore this for now
```

# Interpolation
There are several general interpolation facilities available in SciPy, for data in 1, 2, and higher dimensions. First, let's generate some sample data.

```python
x = np.linspace(0, 10, num=11, endpoint=True)
y = np.cos(-x**2/9.0)

plt.scatter(x,y)
```

The `interp1d` funtions grabs data points and **returns a *function***. The default interpolation method is the linear interpolation, but there are several to choose from.

```python
from scipy.interpolate import interp1d

f1 = interp1d(x, y)  # linear is the default
f2 = interp1d(x, y, kind='cubic')  # cubic splines
f3 = interp1d(x, y, kind='nearest')  # grab the nearest value
```

Now that we have the interpolated function, lets generate a tighter grid in the x axis and plot the resulto of the different interpolation methods.

```python
xnew = np.linspace(0, 10, num=101, endpoint=True)
xnew
```

    array([ 0. ,  0.1,  0.2,  0.3,  0.4,  0.5,  0.6,  0.7,  0.8,  0.9,  1. ,
            1.1,  1.2,  1.3,  1.4,  1.5,  1.6,  1.7,  1.8,  1.9,  2. ,  2.1,
            2.2,  2.3,  2.4,  2.5,  2.6,  2.7,  2.8,  2.9,  3. ,  3.1,  3.2,
            3.3,  3.4,  3.5,  3.6,  3.7,  3.8,  3.9,  4. ,  4.1,  4.2,  4.3,
            4.4,  4.5,  4.6,  4.7,  4.8,  4.9,  5. ,  5.1,  5.2,  5.3,  5.4,
            5.5,  5.6,  5.7,  5.8,  5.9,  6. ,  6.1,  6.2,  6.3,  6.4,  6.5,
            6.6,  6.7,  6.8,  6.9,  7. ,  7.1,  7.2,  7.3,  7.4,  7.5,  7.6,
            7.7,  7.8,  7.9,  8. ,  8.1,  8.2,  8.3,  8.4,  8.5,  8.6,  8.7,
            8.8,  8.9,  9. ,  9.1,  9.2,  9.3,  9.4,  9.5,  9.6,  9.7,  9.8,
            9.9, 10. ])

```python
plt.plot(x, y, 'o', xnew, f1(xnew), '-', xnew, f2(xnew), '--', xnew, f3(xnew), '-.')
plt.legend(['data', 'linear', 'cubic', 'nearest'], loc='best')
plt.show()
```

The `interpolate` sublibrary also has interpolation methods for multivariate data and has **integration with pandas**. Have a look at the documentation.

---
# Definite Integrals
The function `quad` is provided to integrate a function of one variable between two points. This functions has 2 outputs, the first one is the computed integral value and the second is an estimate of the absolute error.

As an example consider the following integral:

$$
\int_{0}^{2} x^{2}dx
$$

```python
import scipy.integrate as integrate

def my_func(x):
    return x**2

integrate.quad(my_func, 0, 2)
```

    (2.666666666666667, 2.960594732333751e-14)

The `quad` functions also allows for infinite limits.

$$
\int_{-\infty}^{\infty} e^{-x^{2}}dx
$$

```python
def my_func(x):
    return np.exp(-x**2)

integrate.quad(my_func, -np.inf, np.inf)
```

    (1.7724538509055159, 1.4202636781830878e-08)

SciPy's `integrate` library also has functions for double and triple integrals. Check them out in the documentations.

# Optimization
The `scipy.optimize` package provides several commonly used optimization algorithms. Here we are going to use just one to illustrate.

Consider that you have 3 assets available. Their expected returns, risks (standard-deviations) and betas are on the table bellow and $\rho$ is the correlation matrix of the returns.

| Asset | Return | Risk | Beta |
|-------|--------|------|------|
|A      |3%      | 10%  | 0.5  |
|B      |3.5%    | 11%  | 1.2  |
|C      |5%      | 15%  | 1.8  |

$$
\rho =
\begin{bmatrix}
1 & 0.3 & -0.6 \\
0.3 & 1 & 0 \\
-0.6 & 0 & 1
\end{bmatrix}
$$

Use the `minimize` function to find the weights of each asset that maximizes it's Sharpe index.

```python
retu = np.array([0.03, 0.035, 0.05])
risk = np.array([0.10, 0.11, 0.15])
beta = np.array([0.5, 1.2, 1.8])

corr = np.array([[1, 0.3, -0.6],
                 [0.3, 1, 0],
                 [-0.6, 0, 1]])

def port_return(w):
    return retu.dot(w)

def port_risk(w):
    covar = np.diag(risk).dot(corr).dot(np.diag(risk))
    return (w.dot(covar).dot(w))**0.5

def port_sharpe(w):
    return -1*(port_return(w) / port_risk(w))   # The -1 is because we want to MINIMIZE the negative of the Sharpe

def port_weight(w):
    return w.sum()
```

```python
w_teste = np.array([0.5, 0.30, 0.2])
print(port_return(w_teste))
print(port_risk(w_teste))
print(port_sharpe(w_teste))
print(port_weight(w_teste))
```

    0.035500000000000004
    0.06065476073648301
    -0.5852796972397789
    1.0

When declaring an optimization problem with inequality restrictions, they have the form of:

$$
\begin{align*}
\min_{w} & f\left(w\right)\\
s.t. & \quad g\left(w\right)\geq0
\end{align*}
$$

```python
from scipy.optimize import minimize

eq_cons = {'type': 'eq',
           'fun' : lambda w: port_weight(w) - 1}

w0 = np.array([1, 0, 0])

res = minimize(port_sharpe, w0, method='SLSQP', constraints=eq_cons, options={'ftol': 1e-9, 'disp': True})
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -0.7140791324512301
                Iterations: 7
                Function evaluations: 37
                Gradient evaluations: 7

```python
res.x
```

    array([0.54864871, 0.06613309, 0.3852182 ])

```python
res.x.sum()
```

    1.0

```python
res.fun
```

    -0.7140791324512301

# Linear Algebra (again)
`scipy.linalg` contains all the functions in `numpy.linalg` plus some more advanced ones.

```python
from scipy import linalg as la

A = np.array([[1,3,5],[2,5,1],[2,3,8]])
la.inv(A)
```

    array([[-1.48,  0.36,  0.88],
           [ 0.56,  0.08, -0.36],
           [ 0.16, -0.12,  0.04]])

Matrix and vector **norms** can also be computed with SciPy. A wide range of norm definitions are available using different parameters to the order argument of `linalg.norm`.

```python
A = np.array([[1, 2], [3, 4]])
print(la.norm(A))  # frobenius norm is the default.
print(la.norm(A, 1)) # L1 norm (max column sum)
print(la.norm(A, np.inf)) # L inf norm (max row sum)
```

    5.477225575051661
    6.0
    7.0

Some more advanced matrix decompositions are also available, like the **Schur Decomposition**

```python
la.schur(A)
```

    (array([[-0.37228132, -1.        ],
            [ 0.        ,  5.37228132]]), array([[-0.82456484, -0.56576746],
            [ 0.56576746, -0.82456484]]))

Some notable matrices can also be created, like block **diagonal matrices**.

```python
A = np.array([[1, 0],
              [0, 1]])

B = np.array([[3, 4, 5],
              [6, 7, 8]])

C = np.array([[7]])

la.block_diag(A, B, C)
```

    array([[1, 0, 0, 0, 0, 0],
           [0, 1, 0, 0, 0, 0],
           [0, 0, 3, 4, 5, 0],
           [0, 0, 6, 7, 8, 0],
           [0, 0, 0, 0, 0, 7]], dtype=int32)

# Solving Linear Systems

$$
\begin{align}
x+3y+5 & =10\\
2x+5y+z & =8\\
2x+3y+8z & =3
\end{align}
$$

The system above can be written with matrix notation as $AX=B$ and we know we can find the solution by doing $X=A^{-1}B$, but inverting a matrix is computationally expensive. When solving big linear system it is advised to use the `solve` method.

```python
A = np.array([[1, 3, 5], [2, 5, 1], [2, 3, 8]])
B = np.array([[10], [8], [3]])
```

Lets check the time that it takes to solve the system in both ways...

```python
la.inv(A).dot(B)
```

    array([[-9.28],
           [ 5.16],
           [ 0.76]])

```python
la.solve(A, B)
```

    array([[-9.28],
           [ 5.16],
           [ 0.76]])

let's try with a bigger matrix

```python
import numpy.random as rnd
A = rnd.random((2000, 2000))
B = rnd.random((2000, 1))
```

```python
%%timeit
la.inv(A).dot(B)
```

    154 ms ± 1.58 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

```python
%%timeit
la.solve(A, B)
```

    123 ms ± 539 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Python Lectures/Section 03 - SciPy.ipynb


Hello world.

###A. Polarization of the liquid crystal

If the electric field $\mathbf{E}$ is applied at some angle with respect to the director $\hat{n}$, there will be a component of the electric field $\mathbf{E}_\parallel$ parallel to $\hat{n}$ and a component $\mathbf{E}_\perp$ perpendicular to $\hat{n}$. Show that the polarization $\mathbf{P} = \epsilon_0 ( \chi_\parallel \mathbf{E}_\parallel + \chi_\perp \mathbf{E}_\perp)$ induced in the liquid crystal material can be written as

\begin{equation}
\mathbf{P} = \epsilon_0[\chi_\perp\mathbf{E} + \chi_a ( \hat{n} \cdot \mathbf{E})\hat{n}].
\end{equation}

To start, we have

$$\mathbf{P} = \epsilon_0 ( \chi_\parallel \mathbf{E}_\parallel + \chi_\perp \mathbf{E}_\perp).$$

Since $\chi_a = \chi_\parallel - \chi_\perp$, we get $$\chi_\parallel = \chi_a + \chi_\perp.$$

Now $\mathbf{P} = \epsilon_0 ( \chi_\parallel \mathbf{E}_\parallel + \chi_\perp \mathbf{E}_\perp)$ becomes

$$\mathbf{P} = \epsilon_0 ( (\chi_a + \chi_\perp) \mathbf{E}_\parallel + \chi_\perp \mathbf{E}_\perp).$$

Distributing and since $\mathbf{E} = \mathbf{E}_\parallel + \mathbf{E}_\perp$, we get

$$\mathbf{P} = \epsilon_0 ( \chi_a \mathbf{E}_\parallel + \chi_\perp \mathbf{E}).$$

By dot product rules, $$\mathbf{E}_\parallel = \hat{n} \cdot \mathbf{E}.$$

Now we come to the final equation $$\mathbf{P} = \epsilon_0[\chi_\perp\mathbf{E} + \chi_a ( \hat{n} \cdot \mathbf{E})\hat{n}]. \: (1)$$

###B. Electric potential energy of liquid crystals

Use the expression in Eq. (1) to find the potential energy per unit volume of the system. This can be done by calculating the work done by the electric field in turning the director from perpendicular to the field (taking that the orientation to be $\theta = 0$) to some angle $\theta$. This is the integral with respect to $\theta$ of the torque per unit volume, $\mathbf{P} \times \mathbf{E}$. The electric potential energy density $U_e$, taken to be zero at $\theta = 0$, will then be the negative of the work done by the field. Show that
$$U_e = -\frac{1}{2} \epsilon_0 \chi_a E^2 \sin^2\theta. \: (2)$$

To start

$$\mathbf{P} \times \mathbf{E} = \epsilon_0[\chi_\perp\mathbf{E} + \chi_a ( \hat{n} \cdot \mathbf{E})\hat{n}] \times \mathbf{E}.$$

Some definitions:

$$ \hat{n} \cdot \mathbf{E} = \lVert\hat{n}\rVert \lVert\mathbf{E}\rVert \cos\theta = E\cos\theta$$

$$ \hat{n} \times \mathbf{E} = \lVert\hat{n}\rVert \lVert\mathbf{E}\rVert \sin\theta = E\sin\theta$$

since $\lVert\hat{n}\rVert = 1.$

$$ \mathbf{E}\times\mathbf{E} = 0 $$

With these defined above, $\mathbf{P} \times \mathbf{E}$ becomes

$$\mathbf{P} \times \mathbf{E} = \epsilon_0\chi_aE^2\cos\theta\sin\theta $$

The integral looks like this:

$$ U_e = \int_0^\theta \epsilon_0\chi_aE^2\cos\theta\sin\theta d\theta$$

With a $u$ sub of $\sin$ and knowing the antiderivative of $\sin$ is $-\cos$, we get:

$$U_e = -\frac{1}{2} \epsilon_0 \chi_a E^2 \sin^2\theta.$$

###C. Distortion energy produced by surfaces

The effect of the surfaces is to make the $x$ and $y$ components of the director, i.e. $n_x$ and $n_y$, vary as functions of $y$. In other words, $\frac{dn_x}{dy}$ and $\frac{dn_y}{dy}$ become nonzero; there is no variation of $\hat{n}$ in the $x$ direction. These derivatives are measures of the amount of distortion introduced by applying an electric field to a liquid crystal that has the directions of its molecules anchored at the surfaces. If the distortion is not too large, the analogy with a simple harmonic oscillator, the energy density increases with the square of the distortion, and we can write the energy density due to distortion as $$U_d = \frac{1}{2}k\left(\frac{dn_x}{dy}\right)^2 + \frac{1}{2}k\left(\frac{dn_y}{dy}\right)^2. \: (3)$$

Since the two terms represent different types of distortion, there is no reason to expect that the coefficient $k$ is the same for both. Nevertheless, the simplifying assumption is that they are the same is a fairly good approximation in many cases.

Use the fact that $\hat{n}$ is a unit vector to show that the total energy density is, $U_d+U_e$, can be expressed as

$$U = \frac{1}{2}k\left(\frac{d\theta}{dy}\right)^2 - \frac{1}{2}\epsilon_0\chi_aE^2\sin^2\theta. \: (4)$$

To start

$$ U = \frac{1}{2}k\left(\frac{dn_x}{dy}\right)^2 + \frac{1}{2}k\left(\frac{dn_y}{dy}\right)^2 - \frac{1}{2}\epsilon_0\chi_aE^2\sin^2\theta$$

The components, $n_x$ and $n_y$, are

$$n_x = \cos\theta$$ and $$n_y = \sin\theta$$ since $\lVert\hat{n}\rVert = 1$.

Then the derivatives of $\theta$ with respect to $y$ are
$$\frac{dn_x}{dy} = -\sin\theta\frac{d\theta}{dy}$$ and $$\frac{dn_y}{dy} = \cos\theta\frac{d\theta}{dy}.$$

Squaring and adding both of them yields the familiar trig identity

$$\left(-\sin\theta\frac{d\theta}{dy}\right)^2 + \left(\cos\theta\frac{d\theta}{dy}\right)^2 = \left(\frac{d\theta}{dy}\right)^2(\sin^2\theta + \cos^2\theta) = \left(\frac{d\theta}{dy}\right)^2.$$

Now

$$ U = \frac{1}{2}k\left(\frac{d\theta}{dy}\right)^2 - \frac{1}{2}\epsilon_0\chi_aE^2\sin^2\theta.$$

###D. Distribution of the director

(a). Solving the extrema of director involves using calculus of variations and the Euler-Lagrange equation. The function $\theta(y)$ minimizes the total energy.

We must minimize $U$

$$ U = \frac{1}{2}k\left(\frac{d\theta}{dy}\right)^2 - \frac{1}{2}\epsilon_0\chi_aE^2\sin^2\theta.$$

The E-L equation is, for this problem, defined as

$$\frac{\partial U}{\partial \theta} - \frac{d}{dy}\frac{\partial U}{\partial \left(d\theta/dy\right)} = 0 $$

The first part is

$$\frac{\partial U}{\partial \theta} = -\epsilon_0\chi_aE^2\sin^2\theta$$

The second is

$$\frac{d}{dy}\frac{\partial U}{\partial \left(d\theta/dy\right)} = \frac{d}{dy}\left(k\left(\dfrac{d\theta}{dy}\right)\right) = k\left(\dfrac{d^2\theta}{dy^2}\right)$$

Putting the parts together

$$-\epsilon_0\chi_aE^2\sin\theta\cos\theta - k\left(\dfrac{d^2\theta}{dy^2}\right) = 0$$

Which can be written as $$k\left(\dfrac{d^2\theta}{dy^2}\right)+\epsilon_0\chi_aE^2\sin\theta\cos\theta=0$$

Since $\xi^2 = \frac{k}{\epsilon_0\chi_aE^2}$, we can divide by the denominator and change the $y$ variable to $y/d$ to get $$\xi_d\left(\dfrac{d^2\theta}{dy^2}\right)+\sin\theta\cos\theta=0. \: (5)$$

HUNTER


### Using Numpy & Sympy: Case Study
(created by Chenkuan Liu; this notebook can also be found at https://github.com/ck44liu/scientific-computing-python-notes/tree/main/Note2)

In previous homework, there was a problem asking the expression of the plane given three points in $\mathbb{R}^3$. The key to this problem is to set up the expression $ax+by+cz+d=0$ (some places write it as $ax+by+cz=d$, but having $d$ on the left side will make it more straightforward in this problem), and then plug in the points to solve the linear system.

At here, we are going to extend and look at this problem from both mathematical and scientific programming perspectives. Without further ado, let's import the libraries:

```python
import numpy as np
import sympy
```

#### Example Demo

Let's say we want to determine the plane containing the points $(1,-2,4)$, $(-2,5,-3)$, and $(2,-3,7)$. After plugging them into $ax+by+cz+d=0$, we get the following system:  (note that we are plugging into $x,y,z$ and trying to solve for $a,b,c,d$ )

$$a-2b+4c+d=0$$
$$-2a+5b-3c+d=0$$
$$2a-3b-7c+d=0$$

To solve this linear system, it's better to look at its reduced row echelon form. At here, we can use numpy and sympy to do this:

```python
# initializing coefficient matrix
M = np.array([[1,-2,4,1],[-2,5,-3,1],[2,-3,7,1]])
M
```

    array([[ 1, -2,  4,  1],
           [-2,  5, -3,  1],
           [ 2, -3,  7,  1]])

```python
# using sympy to compute its rref
M = sympy.Matrix(M)
M.rref()
```

    (Matrix([
     [1, 0, 0, -7/3],
     [0, 1, 0, -1/3],
     [0, 0, 1,  2/3]]),
     (0, 1, 2))

Note that .rref() command returns two things: the first one is the rref matrix, and the second one lists the pivot columns. At here the pivot columns are the first three columns. Python starts counting from zero, so it returns (0,1,2). Also, since all the values on the right side of the linear system are zeros, we only need to look at the coefficient matrix here.

#### "Meaningless" Derivation

Now let's use the rref to derive some equations by hand and let the numbers tell the tale:

the rref tells us $a-\frac{7}{3}d=0$, $b-\frac{1}{3}d=0$, and $c+\frac{2}{3}d=0$, so we have $a=\frac{7}{3}d$, $b=\frac{1}{3}d$, $c=-\frac{2}{3}d$, and the plane becomes:

$$\frac{7}{3}dx + \frac{1}{3}dy - \frac{2}{3}dz + d = 0.$$

To have a meaningful plane, we need to set $d$ be non-zero. In this way, we can divide $d$ on both sides and get:

$$\frac{7}{3}x + \frac{1}{3}y - \frac{2}{3}z + 1 = 0,$$

which is the expression of our plane.

And now, let's do something "meaningless": multiply both sides by $-1$:

$$-\frac{7}{3}x - \frac{1}{3}y + \frac{2}{3}z - 1 = 0$$

This is still the same plane. However, it actually makes it easier for our program: the last column entries of rref are exactly the $a,b,c$ we seek given that the expression of the plane is $ax+by+cz-1=0$. Our hand derivation above shows that this is true as long as the pivot columns are the first three columns.

In this way, our program becomes more straightforward: we can just extract the last column of rref and give the expression of the plane:

```python
# extract last column from rref
last_col = M.rref()[0].col(-1)    # M.rref()[0] is the actual rref, while M.rref()[1] gives the pivot columns as shown above
last_col
```

$\displaystyle \left[\begin{matrix}- \frac{7}{3}\\- \frac{1}{3}\\\frac{2}{3}\end{matrix}\right]$

```python
# assign the values from the last column to a, b and c
a, b, c = last_col
print(a,b,c)
```

    -7/3 -1/3 2/3

```python
# print the expression of the plane
print("The plane is: ({})x + ({})y + ({})z - 1 = 0".format(a, b, c))
```

    The plane is: (-7/3)x + (-1/3)y + (2/3)z - 1 = 0

#### Plot Twist

Just like nothing is perfect, not all combination of three points can be solved in the same steps. Take a look at the next example: say we want to determine the plane containing $(1,1,2)$, $(-2,-2,-3)$, and $(3,3,5)$:

```python
N = np.array([[1,1,2,1],[-2,-2,-3,1],[3,3,5,1]])
N = sympy.Matrix(N)
N.rref()
```

    (Matrix([
     [1, 1, 0, 0],
     [0, 0, 1, 0],
     [0, 0, 0, 1]]),
     (0, 2, 3))

At here, the pivot columns are no longer the first three columns, and we get zero entries in the last column. However, if you observe the points closely, you can find that for each of the three points, we have $x$ equals to $y$. Actually the plane is just $x=y$, namely $a=1,b=-1,c=0,d=0$ if we express it in the form of $ax+by+cz+d=0$. The picture below is generated from Geogebra, which visualizes the plane $x=y$ and our three points.

```python
from PIL import Image
im = Image.open("img\plane_visualization.png")
im.resize((700,600))
```

Hence, in such case, it's better to take a look and solve by ourselves. Though the rref becomes "irregular", the plane actually gets simpler and easier to visualize.

#### Bring it Together

Now we can combine what we've got so far into a single Python function.

```python
def get_plane(p1, p2, p3):
    """
    Arguments:
    p1, p2, p3 -- the three points in 3d plane, expressed in numpy array

    Returns:
    a, b, c -- the coefficients of the plane ax + by + cz + 1 = 0;
               returns -1 if the pivot columns are not the first three columns
    """
    # concatenate the numpy arrays to form a 3 by 4 coefficient matrix
    M = np.concatenate((np.array([p1,p2,p3]), np.ones((3,1), dtype=int)), axis=1)
    # convert to sympy matrix
    M = sympy.Matrix(M)

    # compute and print rref
    rref = M.rref()
    print("The rref matrix is:\n", rref)

    # compute the plane or suggest further manual analysis
    if rref[1] == (0,1,2):
        a, b, c = rref[0].col(-1)
        print("\nThe plane is: ({})x + ({})y + ({})z - 1 = 0".format(a, b, c))
    else:
        a, b, c = -1, -1, -1
        print("\nSpecial case, manual analysis needed.")

    return a, b, c
```

We can check the program by creating and running different set of points:

```python
p1 = np.array([1,-2,4])
p2 = np.array([-2,5,-3])
p3 = np.array([2,-3,7])
get_plane(p1,p2,p3)
print("\n")

p4 = np.array([3,3,3])
p5 = np.array([1,-1,1])
p6 = np.array([-2,-2,-2])
get_plane(p4,p5,p6)
print("\n")
```

    The rref matrix is:
     (Matrix([
    [1, 0, 0, -7/3],
    [0, 1, 0, -1/3],
    [0, 0, 1,  2/3]]), (0, 1, 2))

    The plane is: (-7/3)x + (-1/3)y + (2/3)z - 1 = 0

    The rref matrix is:
     (Matrix([
    [1, 0, 1, 0],
    [0, 1, 0, 0],
    [0, 0, 0, 1]]), (0, 1, 3))

    Special case, manual analysis needed.

#### Extensions

There are some further extensions can be done to this program:

- If given four points $A,B,C,D$ in $\mathbb{R}^3$, we can determine whether they reside on the same plane. Namely, we first compute the plane determined by $A,B$ and $C$, and then plug in vertices of $D$ to see whether the equation holds true.
- We can also extend our program into higher dimensions: we can modify it so that, given $n$ points in n-dimensional space, it can compute the $(n-1)$ dimensional hyperplane determined by these $n$ points.


# Multidimentional data - Matrices and Images

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

from scipy import linalg
```

```python
plt.style.use('ggplot')
plt.rc('axes', grid=False)   # turn off the background grid for images
```

Let us work with the matrix:
$
\left[
\begin{array}{cc}
1 & 2 \\
1 & 1
\end{array}
\right]
$

```python
my_matrix = np.array([[1,2],[1,1]])

print(my_matrix.shape)
```

```python
print(my_matrix)
```

```python
my_matrix_transposed = np.transpose(my_matrix)

print(my_matrix_transposed)
```

```python
my_matrix_inverse = linalg.inv(my_matrix)

print(my_matrix_inverse)
```

### numpy matrix multiply uses the `dot()` function:

```python
my_matrix_inverse.dot(my_matrix)
```

### Caution the `*` will just multiply the matricies on an element-by-element basis:

```python
my_matrix_inverse * my_matrix_inverse
```

### Solving system of linear equations

$$
\begin{array}{c}
x + 2y = 4 \\
x + y = 3 \\
\end{array}
\hspace{2cm}
\left[
\begin{array}{cc}
1 & 2 \\
1 & 1 \\
\end{array}
\right]
\left[
\begin{array}{c}
x\\
y
\end{array}
\right]
=
\left[
\begin{array}{c}
4\\
3\\
\end{array}
\right]
\hspace{2cm}
{\bf A}x = {\bf b}
\hspace{2cm}
\left[
\begin{array}{c}
x\\
y
\end{array}
\right]
=
\left[
\begin{array}{cc}
1 & 2 \\
1 & 1 \\
\end{array}
\right]^{-1}
\left[
\begin{array}{c}
4\\
3\\
\end{array}
\right]
=
\left[
\begin{array}{c}
2\\
1
\end{array}
\right]
$$

```python
A = np.array([[1,2],[1,1]])

print(A)
```

```python
b = np.array([[4],[3]])

print(b)
```

```python
# Solve by inverting A and then mulitply by b

linalg.inv(A).dot(b)
```

```python
# Cleaner looking

linalg.solve(A,b)
```

### System of 3 equations example (Numpy):

$$
\begin{array}{c}
x + 3y + 5z = 10 \\
2x + 5y + z = 8 \\
2x + 3y + 8z = 3 \\
\end{array}
\hspace{3cm}
\left[
\begin{array}{ccc}
1 & 3 & 5 \\
2 & 5 & 1 \\
2 & 3 & 8
\end{array}
\right]
\left[
\begin{array}{c}
x\\
y\\
z
\end{array}
\right]
=
\left[
\begin{array}{c}
10\\
8\\
3
\end{array}
\right]
$$

```python
A = np.array([[1,3,5],[2,5,1],[2,3,8]])
b = np.array([[10],[8],[3]])

print(linalg.inv(A))

print(linalg.solve(A,b))
```

### System of 3 equations example (SymPy) - Python's Symbolic Math Package

```python
import sympy as sym

AA = sym.Matrix([[1,3,5],[2,5,1],[2,3,8]])
bb = sym.Matrix([[10],[8],[3]])

print(AA**-1)

print(AA**-1 * bb)
```

### SymPy is slower than NumPy

```python
%timeit AA**-1 * bb
%timeit linalg.solve(A,b)
```

# Images are just 2-d arrays - `imshow` will display 2-d arrays as images

```python
print(A)

plt.imshow(A, interpolation='nearest', cmap=plt.cm.Blues);
```

### Read in some data

```python
I = np.load("test_data.npy")    # load in a saved numpy array
```

```python
I.ndim, I.shape, I.dtype
```

```python
print("The minimum value of the array I is {0:.2f}".format(I.min()))
print("The maximum value of the array I is {0:.2f}".format(I.max()))
print("The mean value of the array I is {0:.2f}".format(I.mean()))
print("The standard deviation of the array I is {0:.2f}".format(I.std()))
```

```python
#flatten() collapses n-dimentional data into 1-d

plt.hist(I.flatten(),bins=30);
```

### Math on images applies to every value (pixel)

```python
II = I + 8

print("The minimum value of the array II is {0:.2f}".format(II.min()))
print("The maximum value of the array II is {0:.2f}".format(II.max()))
print("The mean value of the array II is {0:.2f}".format(II.mean()))
print("The standard deviation of the array II is {0:.2f}".format(II.std()))
```

### Show the image represenation of `I` with a colorbar

```python
plt.imshow(I, cmap=plt.cm.gray)
plt.colorbar();
```

### Colormap reference: http://matplotlib.org/examples/color/colormaps_reference.html

```python
fig, ax = plt.subplots(1,5,sharey=True)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(I, cmap=plt.cm.viridis)
ax[0].set_xlabel('viridis')

ax[1].imshow(I, cmap=plt.cm.hot)
ax[1].set_xlabel('hot')

ax[2].imshow(I, cmap=plt.cm.magma)
ax[2].set_xlabel('magma')

ax[3].imshow(I, cmap=plt.cm.spectral)
ax[3].set_xlabel('spectral')

ax[4].imshow(I, cmap=plt.cm.gray)
ax[4].set_xlabel('gray')
```

## WARNING! Common image formats DO NOT preserve dynamic range of original data!!
- Common image formats: jpg, gif, png, tiff
- Common image formats will re-scale your data values to [0:1]
- Common image formats are **NOT** suitable for scientific data!

```python
plt.imsave('Splash.png', I, cmap=plt.cm.gray)     # Write the array I to a PNG file

Ipng = plt.imread('Splash.png')                   # Read in the PNG file

print("The original data has a min = {0:.2f} and a max = {1:.2f}".format(I.min(), I.max()))

print("The PNG file has a min = {0:.2f} and a max = {1:.2f}".format(Ipng.min(), Ipng.max()))
```

## Creating images from math

```python
X = np.linspace(-5, 5, 500)
Y = np.linspace(-5, 5, 500)

X, Y = np.meshgrid(X, Y)     # turns two 1-d arrays (X, Y) into one 2-d grid

Z = np.sqrt(X**2+Y**2)+np.sin(X**2+Y**2)

Z.min(), Z.max(), Z.mean()
```

### Fancy Image Display

```python
from matplotlib.colors import LightSource

ls = LightSource(azdeg=0,altdeg=40)
shadedfig = ls.shade(Z,plt.cm.copper)

fig, ax = plt.subplots(1,3)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(shadedfig)

contlevels = [1,2,Z.mean()]

ax[1].axis('equal')
ax[1].contour(Z,contlevels)

ax[2].imshow(shadedfig)
ax[2].contour(Z,contlevels);
```

### Reading in images (`imread`) - Common Formats

```python
I2 = plt.imread('doctor5.png')

print("The image I2 has a shape [height,width] of {0}".format(I2.shape))
print("The image I2 is made up of data of type {0}".format(I2.dtype))
print("The image I2 has a maximum value of {0}".format(I2.max()))
print("The image I2 has a minimum value of {0}".format(I2.min()))
```

```python
plt.imshow(I2,cmap=plt.cm.gray);
```

## Images are just arrays that can be sliced.

- ### For common image formats the origin is the upper left hand corner

```python
fig, ax = plt.subplots(1,4)
fig.set_size_inches(12,6)

fig.tight_layout()

# You can show just slices of the image - Rememeber: The origin is the upper left corner

ax[0].imshow(I2, cmap=plt.cm.gray)
ax[0].set_xlabel('Original')

ax[1].imshow(I2[0:300,0:100], cmap=plt.cm.gray)
ax[1].set_xlabel('[0:300,0:100]')                 # 300 rows, 100 columns

ax[2].imshow(I2[:,0:100], cmap=plt.cm.gray)       # ":" = whole range
ax[2].set_xlabel('[:,0:100]')                     # all rows, 100 columns

ax[3].imshow(I2[:,::-1], cmap=plt.cm.gray);
ax[3].set_xlabel('[:,::-1]')                      # reverse the columns
```

```python
fig, ax = plt.subplots(1,2)
fig.set_size_inches(12,6)

fig.tight_layout()

CutLine = 300

ax[0].imshow(I2, cmap=plt.cm.gray)
ax[0].hlines(CutLine, 0, 194, color='b', linewidth=3)

ax[1].plot(I2[CutLine,:], color='b', linewidth=3)
ax[1].set_xlabel("X Value")
ax[1].set_ylabel("Pixel Value")
```

## Simple image manipulation

```python
from scipy import ndimage
```

```python
fig, ax = plt.subplots(1,5)
fig.set_size_inches(14,6)

fig.tight_layout()

ax[0].imshow(I2, cmap=plt.cm.gray)

I3 = ndimage.rotate(I2,45,cval=0.75)               # cval is the value to set pixels outside of image
ax[1].imshow(I3, cmap=plt.cm.gray)                 # Rotate and reshape

I4 = ndimage.rotate(I2,45,reshape=False,cval=0.75) # Rotate and do not reshape
ax[2].imshow(I4, cmap=plt.cm.gray)

I5 = ndimage.shift(I2,(10,30),cval=0.75)           # Shift image
ax[3].imshow(I5, cmap=plt.cm.gray)

I6 = ndimage.gaussian_filter(I2,5)                # Blur image
ax[4].imshow(I6, cmap=plt.cm.gray);
```

### `ndimage` can do much more: http://scipy-lectures.github.io/advanced/image_processing/

---

## FITS file (Flexible Image Transport System) - Standard Astro File Format
- **FITS format preserves dynamic range of data**
- FITS format can include lists, tables, images, and combunations of different types of data

```python
import astropy.io.fits as fits
```

```python
x = fits.open('bsg01.fits')

x.info()
```

```python
x[0].header
```

```python
xd = x[0].data

print("The image x has a shape [height,width] of {0}".format(xd.shape))
print("The image x is made up of data of type {0}".format(xd.dtype))
print("The image x has a maximum value of {0}".format(xd.max()))
print("The image x has a minimum value of {0}".format(xd.min()))
```

```python
fig, ax = plt.subplots(1,2)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(xd,cmap=plt.cm.gray)

ax[1].hist(xd.flatten(),bins=20);
```

## You can use masks on images

```python
CopyData = np.copy(xd)

CutOff = 40

mask = np.where(CopyData > CutOff)
CopyData[mask] = 50                   # You can not just throw data away, you have to set it to something.

fig, ax = plt.subplots(1,2)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(CopyData,cmap=plt.cm.gray)

ax[1].hist(CopyData.flatten(),bins=20);
```

## You can add and subtract images

```python
fig, ax = plt.subplots(1,2)
fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(xd, cmap=plt.cm.gray)

# Open another file 'bsg02.fits'

y = fits.open('bsg02.fits')
yd = y[0].data

ax[1].imshow(yd, cmap=plt.cm.gray);
```

### The two images above may look the same but they are not! Subtracting the two images reveals the truth.

```python
fig, ax = plt.subplots(1,3)
fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(xd, cmap=plt.cm.gray)
ax[1].imshow(yd, cmap=plt.cm.gray)

z = xd - yd                          # Subtract the images pixel by pixel

ax[2].imshow(z, cmap=plt.cm.gray);
```

## FITS Tables - An astronomical example

* Stellar spectra data from the [ESO Library of Stellar Spectra](http://www.eso.org/sci/facilities/paranal/decommissioned/isaac/tools/lib.html)

```python
S = fits.open('SolarSpectra.fits')

S.info()
```

```python
Data = S[0].data
```

```python
Head = S[0].header
Head
```

```python
# The FITS header has the information to make an array of wavelengths

Start = Head['CRVAL1']
Number = Head['NAXIS1']
Delta  = Head['CDELT1']

End = Start + (Number * Delta)

Wavelength = np.arange(Start,End,Delta)
```

```python
fig, ax = plt.subplots(2,1)
fig.set_size_inches(11,8.5)

fig.tight_layout()

# Full spectra

ax[0].plot(Wavelength, Data, color='b')
ax[0].set_ylabel("Flux")
ax[0].set_xlabel("Wavelength [angstroms]")

# Just the visible range with the hydrogen Balmer lines

ax[1].set_xlim(4000,7000)
ax[1].set_ylim(0.6,1.2)
ax[1].plot(Wavelength, Data, color='b')
ax[1].set_ylabel("Flux")
ax[1].set_xlabel("Wavelength [angstroms]")

H_Balmer = [6563,4861,4341,4102,3970,3889,3835,3646]

ax[1].vlines(H_Balmer,0,2, color='r', linewidth=3, alpha = 0.25)
```

# Pseudocolor - All color astronomy images are fake.

### Color images are composed of three 2-d images:

### JPG images are 3-d, even grayscale images

```python
redfilter = plt.imread('sphereR.jpg')

redfilter.shape,redfilter.dtype
```

### We just want to read in one of the three channels

```python
redfilter = plt.imread('sphereR.jpg')[:,:,0]

redfilter.shape,redfilter.dtype
```

```python
plt.imshow(redfilter,cmap=plt.cm.gray);
```

```python
greenfilter = plt.imread('sphereG.jpg')[:,:,0]
bluefilter = plt.imread('sphereB.jpg')[:,:,0]
```

```python
fig, ax = plt.subplots(1,3)
fig.set_size_inches(12,3)

fig.tight_layout()

ax[0].set_title("Red Filter")
ax[1].set_title("Green Filter")
ax[2].set_title("Blue Filter")

ax[0].imshow(redfilter,cmap=plt.cm.gray)
ax[1].imshow(greenfilter,cmap=plt.cm.gray)
ax[2].imshow(bluefilter,cmap=plt.cm.gray);
```

### Need to create a blank 3-d array to hold all of the images

```python
rgb = np.zeros((480,640,3),dtype='uint8')

print(rgb.shape, rgb.dtype)

plt.imshow(rgb,cmap=plt.cm.gray);
```

## Fill the array with the filtered images

```python
rgb[:,:,0] = redfilter
rgb[:,:,1] = greenfilter
rgb[:,:,2] = bluefilter
```

```python
fig, ax = plt.subplots(1,4)
fig.set_size_inches(14,3)

fig.tight_layout()

ax[0].set_title("Red Filter")
ax[1].set_title("Green Filter")
ax[2].set_title("Blue Filter")
ax[3].set_title("All Filters Stacked")

ax[0].imshow(redfilter,cmap=plt.cm.gray)
ax[1].imshow(greenfilter,cmap=plt.cm.gray)
ax[2].imshow(bluefilter,cmap=plt.cm.gray)
ax[3].imshow(rgb,cmap=plt.cm.gray);
```

```python
print("The image rgb has a shape [height,width] of {0}".format(rgb.shape))
print("The image rgb is made up of data of type {0}".format(rgb.dtype))
print("The image rgb has a maximum value of {0}".format(rgb.max()))
print("The image rgb has a minimum value of {0}".format(rgb.min()))
```

```python
rgb[:,:,0] = redfilter * 1.5

plt.imshow(rgb)
```

```python

```


# Case Study III: (1) Solving Laplace's equation with Python

In the first part of the case study we will solve a simple electrostatics problem with Python. In the second part, we will use NumPy.

## Solving Laplace's or Poisson's equation

**Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

Solutions depend on the **boundary conditions**:

* the *value of the potential* on the *boundary* or
* the *electric field* (i.e. the derivative of the potential, $\mathbf{E} = -\nabla\Phi$ *normal to the surface* ($\mathbf{n}\cdot\mathbf{E}$), which directly follows from the charge distribution).

### Example: 2D Laplace equation for the "Wire on the grounded box" problem

As a problem. consider a grounded box where the left side is a wire held at a potential of 100 V. There are no charges _inside_ the box.

Boundary conditions:
* square area surrounded by wires
* three wires at ground (0 V), one wire at 100 V (the line with $x=0$ is at 100 V)

_Inside the box_ the **Laplace equation** applies:
$$
\frac{\partial^2 \Phi(x,y)}{\partial x^2} + \frac{\partial^2 \Phi(x,y)}{\partial y^2} = 0
$$

i.e., the Poisson equation with charges set to 0.

## Finite difference algorithm for Poisson's equation
Discretize space on a lattice (2D) and solve for $\Phi$ on each lattice site.

Taylor-expansion of the four neighbors of $\Phi(x, y)$:

\begin{align}
\Phi(x \pm \Delta x, y) &= \Phi(x, y) \pm \Phi_x \Delta x + \frac{1}{2} \Phi_{xx} \Delta x^2 + \dots\\
\Phi(x, y \pm \Delta y) &= \Phi(x, y) \pm \Phi_y \Delta x + \frac{1}{2} \Phi_{yy} \Delta x^2 + \dots\\
\end{align}

Add equations in pairs: odd terms cancel, and **central difference approximation** for 2nd order partial derivatives (to $\mathcal{O}(\Delta^4)$):

\begin{align}
\Phi_{xx}(x,y) = \frac{\partial^2 \Phi}{\partial x^2} & \approx
  \frac{\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) - 2\Phi(x,y)}{\Delta x^2} \\
\Phi_{yy}(x,y) = \frac{\partial^2 \Phi}{\partial y^2} &\approx
  \frac{\Phi(x,y+\Delta y) + \Phi(x,y-\Delta y) - 2\Phi(x,y)}{\Delta y^2}
\end{align}

Take $x$ and $y$ grids of equal spacing $\Delta$: Discretized Poisson equation

$$
\begin{split}
\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\Phi(x,y+\Delta y) &+ \\
   +\, \Phi(x,y-\Delta y) - 4\Phi(x,y) &= -4\pi\rho(x,y)\,\Delta^2
   \end{split}
$$

Defines a system of $N_x \times N_y$ simultaneous algebraic equations for $\Phi_{ij}$ to be solved.

Can be solved directly via matrix approaches (and then is the best solution) but can be unwieldy for large grids.

Alternatively: **iterative solution**:

$$
\begin{split}
4\Phi(x,y) &= \Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\\
 &+ \Phi(x,y+\Delta y) + \Phi(x,y-\Delta y) + 4\pi\rho(x,y)\,\Delta^2
\end{split}
$$

Compute a new value for $\Phi(x,y)$ (left hand site) from a guessed potential (right hand side).

Or written for lattice sites $(i, j)$ where

$$
x = x_0 + i\Delta\quad\text{and}\quad y = y_0 + j\Delta, \quad 0 \leq i,j < N_\text{max}
$$

$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

* Converged solution at $(i, j)$ will be the average potential from the four neighbor sites + charge density contribution.
* *Not a direct solution*: iterate and hope for convergence.

#### Jacobi method
Do not change $\Phi_{i,j}$ until a complete sweep has been completed.

The Jacobi algorithm is the simplest iterative approach and much better solutions exist (which we explore in the PHY432 _Computational Methods_ class). Jacobi converges slower than better algorithms but it will eventually give the right answer. It has the advantage of being easier to understand and easier to speed up with NumPy.

## Solution via relaxation (Jacobi method) in Python

Solve the box-wire problem on a lattice: The wire at $x=0$ (the $y$-axis) is at 100 V, the other three sides of the box are grounded (0 V).

Note: $\rho=0$ inside the box.

Note for Jupyter notebook use:
* For interactive 3D plots, select
  ```
  %matplotlib notebook
  ```
* For standard inline figures (e.g. for exporting the notebook to LaTeX/PDF or html) use
  ```
  %matplotlib inline
  ```

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
# %matplotlib inline
%matplotlib notebook
```

```python
%matplotlib inline
```

#### Wire on a box: Solution of Laplace's equation with the Jacobi algorithm

The potential is represented as a numpy array `Phi[x, y]`. We will consider axis 0 to correspond to x and axis 1 to y. (Technically speaking, row indices are "x" and column indices are "y" but this becomes confusing when translating the equations into code. We just need to be careful when we plot...)

We only iterate to `Max_iter = 70`: this will *not* produce a converged solution. We will address this problem in the second part.

Note how the actual update of the array does not update the boundaries of the array because they are set by the problem. It also means that the indices loop over `Nmax-2` values via `range(1, Nx-1)` and `range(1, Ny-1)`. This is important to remember for the numpyfication in Part 2.

```python
Nmax = 100
Max_iter = 70
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

# Jacobi: do not change the potential during one update, so we need to work on a copy
Phi_new = Phi.copy()

Nx, Ny = Phi.shape
for n_iter in range(Max_iter):
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi_new[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                 + Phi[xi, yj+1] + Phi[xi, yj-1])
    # update the potential for the next iteration
    Phi[:, :] = Phi_new
```

#### Visualization of the potential

We use 2D and 3D plotting in matplotlib. The meshgrid function provides a convenient way to produce arrays that can be easily plotted with these functions. The notebook [meshgrid.ipynb](meshgrid.ipynb) provides additional information.

```python
# plot Phi(x,y)
x = np.arange(Phi.shape[0])
y = np.arange(Phi.shape[1])
X, Y = np.meshgrid(x, y)

Z = Phi[X, Y]
```

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel(r'potential $\Phi$ (V)')

ax.view_init(elev=40, azim=-65)
```

Nicer plot (you can use this code for other projects):

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2, linewidth=0.5, color="gray")
surf = ax.plot_surface(X, Y, Z, cmap=plt.cm.coolwarm, alpha=0.6)
cset = ax.contourf(X, Y, Z, zdir='z', offset=-50, cmap=plt.cm.coolwarm)

ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel(r'potential $\Phi$ (V)')
ax.set_zlim(-50, 100)
ax.view_init(elev=40, azim=-65)

cb = fig.colorbar(surf, shrink=0.5, aspect=5)
cb.set_label(r"potential $\Phi$ (V)")
```

(Note that the calculation above is is *not converged* ... see next lecture.)