```python
%%html
<link href="https://fonts.googleapis.com/css?family=Open+Sans" rel="stylesheet">
<style>#notebook-container{font-size: 13pt;font-family:'Open Sans', sans-serif;} div.text_cell{max-width: 104ex;}</style>
```

<link href="https://fonts.googleapis.com/css?family=Open+Sans" rel="stylesheet">
<style>#notebook-container{font-size: 13pt;font-family:'Open Sans', sans-serif;} div.text_cell{max-width: 104ex;}</style>

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

# 2D Rendering

To render in 2D we will be using vectors with $(x, y)$ coordinates. We will then explore matrices and linear transformations.

## Python Image Library

```python
from PIL import *
```

We have a pixel grid with $(x,y)$ coordinates that is initiated with a width $w$ and a height $h$. The coordinate $(0,0)$ is in the top-left corner. We want to correct this and have our coordinate system start in the bottom-left corner.

We want additional functions to help with drawing shapes:

 * Drawing a point
 * Drawing a rectangle between two points
 * Drawing a line between two points
     * This is first done with a simple line $y=mx+b$ and then calculate the $y$ values for that particular range.
     * There is a better algorithm, named the Bresenham algorithm (https://en.wikipedia.org/wiki/Bresenham%27s_line_algorithm) that should be implemented later. Hey, it's later, and implemented.
 * Drawing a circle
 * Filling a triangle

```python
class Drawing():
    def __init__(self, w, h, bg_color = 'black'):
        self.width = w
        self.height = h
        self.img = Image.new('RGB', (w, h), bg_color)
        self.color = (255, 255, 255)
        self.pixels = self.img.load()

    def SetColor(self, r, g, b):
        self.color = (r, g, b)

    def SetPixel(self, x, y):
        # Clip any attempts at drawings outside of the range.
        if x < 0 or x > self.width - 1 or y < 0 or y > self.height - 1: return
        # Swap the y-values because the top left is (0,0).
        self.pixels[x, self.height-y-1] = self.color

    def DrawPoint(self, x, y, size = 1):
        if size < 1: raise ValueError('Parameter \'size\' must be positive.'); return;
        if size == 1: self.SetPixel(x, y); return;
        half = size / 2
        for i in range(size):
            for j in range(size):
                self.SetPixel(x-half+i+1, y-half+j+1)

    def DrawRectangle(self, x1, y1, x2, y2):
        if x1 > x2: raise ValueError('x1 > x2'); return;
        if y1 > y2: raise ValueError('y1 > y2'); return;
        for i in range(x2 - x1):
            for j in range(y2 - y1):
                self.SetPixel(x1 + i, y1 + j)

    """ Simple line drawing algorithm """
    def DrawLine(self, x1, y1, x2, y2):
        # Is x1 the closest point on the x-axis?
        if x1 > x2:
            # No, so we swap the points
            tx = x1; x1 = x2; x2 = tx;
            ty = y1; y1 = y2; y2 = ty;
        # Check if the slope is not zero
        if x2 - x1 == 0:
            # If it is zero, draw a vertical line
            ymin = min(y1,y2)
            ymax = max(y1,y2)
            for i in (range(math.ceil(ymax-ymin))):
                self.SetPixel(x1, i+ymin)
            return
        # Draw line
        m = (y2 - y1) / (x2 - x1)
        r = x2 - x1
        for i in range(math.ceil(r)):
            x = x1 + i
            y = m * (x - x1) + y1
            self.SetPixel(round(x), round(y))

    """ Drawing a line with the Bresenham algorithm """
    def DrawBLine(self, x1, y1, x2, y2):
        if abs(y2 - y1) < abs(x2 - x1):
            if x1 > x2:
                self.DrawBLineLow(int(x2), int(y2), int(x1), int(y1))
            else:
                self.DrawBLineLow(int(x1), int(y1), int(x2), int(y2))
        else:
            if y1 > y2:
                self.DrawBLineHigh(int(x2), int(y2), int(x1), int(y1))
            else:
                self.DrawBLineHigh(int(x1), int(y1), int(x2), int(y2))

    """ Drawing the BLine segment on a negative gradient """
    def DrawBLineLow(self, x1, y1, x2, y2):
        dx = x2 - x1
        dy = y2 - y1
        yi = 1
        if dy < 0:
            yi = -1
            dy = -dy
        D = 2*dy - dx
        y = y1
        for x in range(math.ceil(x2-x1)):
            self.SetPixel(round(x1 + x), round(y))
            if D > 0:
                y = y + yi
                D = D - 2*dx
            D = D + 2*dy

    """ Drawing the BLine segment on a positive gradient """
    def DrawBLineHigh(self, x1, y1, x2, y2):
        dx = x2 - x1
        dy = y2 - y1
        xi = 1
        if dx < 0:
            xi = -1
            dx = -dx
        D = 2*dx - dy
        x = x1
        for y in range(math.ceil(y2-y1)):
            self.SetPixel(x, y1 + y)
            if D > 0:
                x = x + xi
                D = D - 2*dy
            D = D + 2*dx

    def DrawCircle(self, px, py, r):
        n = 50
        theta = 2*pi/n
        coords = []
        # Figure out some the points on the circle.
        for i in range(n):
            x = r * cos(i * theta) + px
            y = r * sin(i * theta) + py
            coords.append((x, y))
        # Connect all the points.
        for i in range(1, len(coords)):
            self.DrawBLine(coords[i-1][0], coords[i-1][1], coords[i][0], coords[i][1])
        # Connect the first and last point.
        self.DrawBLine(coords[0][0], coords[0][1], coords[n-1][0], coords[n-1][1])
```

Let's test the drawing class.

```python
drawing01 = Drawing(300, 200)
drawing01.DrawPoint(25, 25, 3)
drawing01.SetColor(255, 0, 0)
drawing01.DrawPoint(25, 25)
drawing01.DrawPoint(100, 100, 20)
drawing01.SetColor(0, 255, 0)
drawing01.DrawRectangle(50, 50, 60, 80)
drawing01.SetColor(0, 0, 255)
drawing01.DrawLine(10, 10, 290, 190)
drawing01.SetColor(0, 255, 255)
drawing01.DrawBLine(10, 190, 290, 10)
drawing01.SetColor(0, 255, 0)
drawing01.DrawLine(40, 10, 240, 30)
drawing01.DrawBLine(40, 15, 240, 35)
drawing01.DrawBLine(265, 160, 275, 30)
drawing01.SetColor(255, 0, 0)
drawing01.DrawLine(260, 160, 270, 30)
drawing01.DrawCircle(150, 100, 30)
drawing01.img
```

Now we can create simple drawings. Let's continue to the math.

## Transformations

We will have a list of vertices $V$ which gives all the points for the object. Those are then scaled, rotated, and translated. We create a homogeneous coordinate that we need for the $3\times3$ matrix multiplication. This simply means there is a $w$ element, which is used for translations. So each vertex is represented as:

$$\vec{v} = \begin{bmatrix}x\\y\\w=1\end{bmatrix}$$

### Cube

We create a normalized unit cube, which we can scale, then rotate, and finally translate. Note the order, this is important.

```python
cube_vertices = []

for x in [-1, 1]:
    for y in [-1, 1]:
        cube_vertices.append(np.array([x, y, 1]))

cube_vertices
```

    [array([-1, -1,  1]),
     array([-1,  1,  1]),
     array([ 1, -1,  1]),
     array([1, 1, 1])]

### Scaling

To scale the homogenous coordinate of the vertex $v$ we multiply it with a scaling matrix. This will set the width $w$ and the height $h$.

$$ M_{s} = \begin{bmatrix}w & 0 & 0 \\ 0 & h & 0 \\ 0 & 0 & 1\end{bmatrix} $$

```python
scaleMatrix = np.identity(3)
scaleMatrix[0,0] = 20
scaleMatrix[1,1] = 20
scaleMatrix
```

    array([[ 20.,   0.,   0.],
           [  0.,  20.,   0.],
           [  0.,   0.,   1.]])

```python
def create_scaling_matrix(w, h):
    scaleMatrix[0,0] = w
    scaleMatrix[1,1] = h
    return scaleMatrix
```

### Rotation

To rotate the homogenous coordinate of the vector we use a rotation matrix.

$$ M_{r} = \begin{bmatrix}\cos\theta & -\sin\theta & 0 \\ \sin\theta & \cos\theta & 0 \\ 0 & 0 & 1\end{bmatrix} $$

```python
def create_rotation_matrix(angle):
    m = np.zeros([3,3])
    m[0, 0] = cos(angle)
    m[0, 1] = -sin(angle)
    m[1, 0] = sin(angle)
    m[1, 1] = cos(angle)
    m[2, 2] = 1
    return m

rotationMatrix = create_rotation_matrix(0)
```

```python
rotationMatrix
```

    array([[ 1., -0.,  0.],
           [ 0.,  1.,  0.],
           [ 0.,  0.,  1.]])

### Translation

To move the homogeneous coordinate of the vertex we use the extra column in our matrix.

$$ M_{t} = \begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ x & y & 1\end{bmatrix} $$

Because $w=1$ in the coordinate $[x,y,w=1]$, this matrix will add the $x$ and $y$ positions.

```python
translationMatrix = np.identity(3)
translationMatrix[2,0] = 100
translationMatrix[2,1] = 100
translationMatrix
```

    array([[   1.,    0.,    0.],
           [   0.,    1.,    0.],
           [ 100.,  100.,    1.]])

```python
def create_translation_matrix(x, y):
    translationMatrix = np.identity(3)
    translationMatrix[2,0] = x
    translationMatrix[2,1] = y
    return translationMatrix
```

### Applying all the transformations

```python
M = np.dot(scaleMatrix, rotationMatrix)
np.dot(M, translationMatrix)
```

    array([[  20.,    0.,    0.],
           [   0.,   20.,    0.],
           [ 100.,  100.,    1.]])

To get the transformed coordinates we multiply it with all the matrices.

$$\vec{v_t} = \left(M_s \cdot M_r \cdot M_t\right) \cdot \vec{v}$$

```python
scene = Drawing(400, 300)

transformed_vertices = []

rotationMatrix = create_rotation_matrix(pi/4)

# Multiply all the matrices to get all the transformations.
M = np.dot(scaleMatrix, rotationMatrix)
transform = np.dot(M, translationMatrix)

for vertex in cube_vertices:
    # Apply transformations to vertex
    translatedVertex = np.dot(transform.T, vertex)
    transformed_vertices.append(translatedVertex)

for i in range(len(transformed_vertices)):
    n = len(transformed_vertices)
    x = transformed_vertices[i][0]
    y = transformed_vertices[i][1]
    scene.SetColor(255,0,0)
    scene.DrawPoint(x, y, 3)

scene.img
```

## Meshes

An object consists of multiple vertices. To move them all around at once, we represent them as a mesh. The mesh will have a position, scale, and rotation. This will be applied to all the vertices with the transformation matrices.

```python
class Mesh():
    def __init__(self, name):
        self.name = name
        self.vertices = []
        self.edges = []
        self.position = np.zeros([1,2])
        self.scale = 1
        self.rotation = 0

def create_cube(name, x, y, scale, rotation):
    mesh = Mesh(name)
    mesh.vertices.append(np.array([-1,  1]))
    mesh.vertices.append(np.array([ 1,  1]))
    mesh.vertices.append(np.array([ 1, -1]))
    mesh.vertices.append(np.array([-1, -1]))
    mesh.edges.append([3,0])
    mesh.edges.append([0,1])
    mesh.edges.append([1,2])
    mesh.edges.append([2,3])
    mesh.position = np.array([x,y])
    mesh.scale = scale
    mesh.rotation = rotation
    return mesh
```

```python
def draw_meshes(meshes, w, h):
    scene = Drawing(w, h)
    scene.SetColor(255, 0, 0)

    for mesh in meshes:
        scaleMatrix = create_scaling_matrix(mesh.scale, mesh.scale)
        rotationMatrix = create_rotation_matrix(mesh.rotation)
        translationMatrix = create_translation_matrix(mesh.position[0], mesh.position[1])
        translatedVertices = []
        scene.SetColor(255, 0, 0)
        for vertex in mesh.vertices:
            scaledVertex = np.dot(scaleMatrix.T, np.array([vertex[0], vertex[1], 1]))
            rotatedVertex = np.dot(rotationMatrix.T, scaledVertex)
            translatedVertex = np.dot(translationMatrix.T, rotatedVertex)
            translatedVertices.append(translatedVertex)
            scene.DrawPoint(translatedVertex[0], translatedVertex[1], 2)
        scene.SetColor(100, 100, 100)
        for (a,b) in mesh.edges:
            scene.DrawBLine(translatedVertices[a][0], translatedVertices[a][1], translatedVertices[b][0], translatedVertices[b][1])

    return scene.img
```

### Scaling & translation

```python
meshes = []

for i in range(4):
    meshes.append(create_cube('sqr_{}'.format(i), 100 * (i + 1) + 20 + (2*i), 100, 20 + (5*i), 0))

draw_meshes(meshes, 565, 200)
```

### Rotation

```python
meshes = []

for i in range(6):
    meshes.append(create_cube('sqr_{}'.format(i), 100 * (i + 1), 100, 20, i*pi/10))

draw_meshes(meshes, 700, 200)
```

## Geometry

Now let's go onto another route. With geometry we can trace out rays from a point and see if they hit anything in the scene. We will first create a point with coordinates $(p_x, p_y)$.

```python
# Coordinates of our point (px, py)
px = py = 100
```

We also create a circle with coordinates $(c_x, c_y)$ and a radius $c_r$.

```python
# Coordinates and radius of our circle (cx, cy, cr)
cx = 500
cy = 100
cr = 50
```

```python
scene = Drawing(600, 200)
scene.DrawPoint(px, py, 5)
scene.DrawCircle(cx, cy, cr)
scene.DrawPoint(cx, cy, 2)
scene.img
```

Now we place a line with two points in front of the point, obstructing the view of the circle. The coordinates for the two points that represent the lines are: $A=(a_x, a_y), B=(b_x, b_y)$.

```python
scene.SetColor(100, 100, 100)
ax = 150
ay = 125
bx = 150
by = 75
scene.DrawBLine(ax, ay, bx, by)
scene.img
```

Now we grab a point $Q$ on our line, and draw a line from $P$ to $Q$.

```python
qx = ax
qy = by + 3 * (ay - by) / 5
```

```python
scene.SetColor(255, 0, 0)
scene.DrawPoint(qx, qy, 3)
scene.DrawBLine(px, py, qx, qy)
scene.img
```

Now we can get the formula for the line $l$, and find the $y$ on the right end of the screen.

$$y=\frac{\Delta y}{\Delta x}(x-p_x)+p_y$$

```python
lm = (qy - py) / (qx - px)
ly = lm * (600 - px) + py
```

```python
scene.DrawPoint(600-2, ly, 4)
scene.SetColor(255, 150, 150)
scene.DrawLine(qx, qy, 600, ly)
scene.img
```

We can see that we intersect the circle twice, but we need to actually check this mathematically. To do this we will draw a line $k$ that is perpedicular to $l$ and through the center of the circle. Then we check if the distance between the line and the point is smaller than the radius of the circle, if so we have hit it.

To get the slope of line $k$ we take the normal of $l$ which means that $k\perp l$. This is done with with:

$$k = \dfrac{\Delta y}{\Delta x} \xrightarrow{\text{normal}} k_n = -\dfrac{\Delta x}{\Delta y}$$

```python
km = -(qx - px) / (qy - py)
km
```

    -10.0

If we use the slope $k$ and simply add our point $c_x, c_y$ we have a line that is perpendicular to $l$ and goes through the circle. Now to find the top point $R$ in the image (where $y=200$, because the height is $200$) we simply solve for $x$ with a gives $y$:

$$ \begin{align} y &= k_m(x - c_x) + c_y \\ k_m\cdot x &= k_m\cdot c_x - c_y + y \\ x &= \dfrac{k_m\cdot c_x - c_y + y}{k_m}\end{align}$$

To draw a line in the image we need to find the $x$ value that corresponds to the $y$ values that is the top of our screen. We can figure this out with the equation above and setting $y=200$.

```python
ry = 200
rx = (km * cx - cy + ry) / km
rx, ry
```

    (490.0, 200)

```python
scene.DrawPoint(rx, ry-1, 2)
scene.img
```

For the bottom point we use the same formula and set $y=0$.

```python
qy = 0
qx = (km * cx - cy + qy) / km
qx, qy
```

    (510.0, 0)

And finally draw a line between them.

```python
scene.SetColor(255, 200, 200)
scene.DrawPoint(qx, qy, 2)
scene.DrawBLine(rx, ry, qx, qy)
scene.img
```

We can see that the line $k$ we drawn is perpendicular to $l$ and goes through $C$. Now we need to figure out point $I$ where the line $l$ and $k$ intersect. We simply put $l=k$ and solve for $x$.

$$\begin{align} l &= k \\ l_m(x-p_x)+p_y &= k_m(x-q_x)+q_y \\ l_m\cdot x-l_m \cdot p_x + p_y &= k_m\cdot x - k_m - k_m\cdot q_x + q_y \\ l_m\cdot x - k_m \cdot x &= l_m\cdot p_x - p_y - k_m \cdot q_x + q_y \end{align}$$

Now we factor out $x$ on the left side, and divide by $l_m - k_m$ to isolate $x$.

$$x = \dfrac{l_m \cdot p_x - p_y - k_m \cdot q_x + q_y}{l_m - k_m}$$

We can now use this formula to get the $x$ value where the two lines intersect.

```python
ix = (lm * px - py - km * qx + qy) / (lm - km)
ix
```

    496.03960396039605

Fill in the $x$ value into any of those lines to get the $y$ value.

```python
iy = lm * (ix - px) + py
iy
```

    139.6039603960396

Finally we connect them with a blue line.

```python
scene.SetColor(0, 100, 255)
scene.DrawPoint(ix, iy, 4)
scene.DrawPoint(cx, cy, 4)
scene.DrawBLine(ix, iy, cx, cy)
scene.img
```

Now we can calculate the length of the blue line with $x=\sqrt{(c_x-i_x)^2+(c_y-i_y)^2}$.

```python
length = sqrt((cx - ix)**2 + (cy - iy)**2)
length
```

    39.801487608399569

To finally conclude if we hit the circle, we check if this distance is smaller than the radius of the circle.

```python
length < cr
```

    True

We have hit the circle! Here is an image with annotations of all the points and lines we have so far.

## Vector Geometry

To make all the calculations a lot easier we are going to use vector geometry. This will also help us to find out where our line intersect the circle.

We are going to recreate the scene but this time we are going to use vectors. Also, we are going to add a few classes that will help us keep track of all our points and lines.

### Points

A point is a row vector $[x, y]$.

```python
def create_point(x, y):
    return np.array([x, y])
```

```python
def distance(p, q):
    return sqrt((q[0] - p[0])**2 + (q[1] - p[1])**2)
```

```python
def normal(v):
    return np.array([v[1], -v[0]])
```

### Lines

A line has a position which is a point, and a direction which is a row vector. Lines are created between two points.

```python
class Line():
    def __init__(self, pos, direction):
        self.position = np.array([pos[0], pos[1]])
        self.direction = np.array([direction[0], direction[1]])

    def __repr__(self):
        return 'Line (position: {}, direction: {})'.format(self.position, self.direction)

    # Find the slope of the line.
    def Slope(self):
        if self.direction[0] == 0: return 0;
        return self.direction[1] / self.direction[0];

    # Find the y value of a point on the line for a given value of x.
    def PointY(self, x):
        return self.Slope() * (x - self.position[0]) + self.position[1]

    # Find the x value of a point on the line for a given value of y.
    def PointX(self, y):
        m = self.Slope()
        return (y + m * self.position[0] - self.position[1]) / m
```

```python
def create_line(p1, p2):
    return Line(p1, p2 - p1)
```

#### Intersection between lines

To find where two lines intersect we calculate the intersection and return them as a tuple $(x,y)$.

```python
def line_intersect(l, k):
    lm = l.Slope()
    km = k.Slope()
    x = (lm * l.position[0] - l.position[1] - km * k.position[0] + k.position[1]) / (lm - km)
    return create_point(x, l.PointY(x))
```

### Circles

A circle has a position which is a point and a radius $r$ which is a float.

```python
class Circle():
    def __init__(self, pos, radius, color = (255, 255, 255)):
        self.position = pos
        self.radius = radius
        self.color = color
        self.diffusion = 1
        self.reflection = 1
```

```python
def create_circle(p, r):
    return Circle(p, r)
```

### Our scene revisited

We will now reconstruct the scene with the classes from above.

```python
P = create_point(100, 100)
A = create_point(150, 125)
B = create_point(150, 75)
Q = create_point(A[0], (A[1] + B[1] + 10) / 2)
C = create_point(500, 100)
circle = create_circle(C, 75)

P, Q, C
```

    (array([100, 100]), array([ 150.,  105.]), array([500, 100]))

```python
pointWidth = 4
scene = Drawing(600, 200)
scene.SetColor(0, 0, 255)
scene.DrawBLine(P[0], P[1], Q[0], Q[1])
scene.SetColor(255,255,255)
scene.DrawPoint(P[0], P[1], pointWidth)
scene.SetColor(100, 100, 100)
scene.DrawBLine(A[0], A[1], B[0], B[1])
scene.SetColor(255,255,255)
scene.DrawPoint(Q[0], Q[1], pointWidth)
scene.DrawPoint(C[0], C[1], pointWidth)
scene.DrawCircle(C[0], C[1], circle.radius)
```

```python
scene.img
```

```python
PQ = create_line(P, Q)
PQ
```

    Line (position: [100 100], direction: [ 50.   5.])

```python
scene.SetColor(255, 0, 0)
scene.DrawBLine(Q[0], Q[1], 600, PQ.PointY(600))
```

```python
scene.img
```

```python
PQ
```

    Line (position: [100 100], direction: [ 50.   5.])

```python
normal(PQ.direction)
```

    array([  5., -50.])

```python
k = create_line(C, 0)
k.direction = normal(PQ.direction)
k
```

    Line (position: [500 100], direction: [  5. -50.])

```python
scene.SetColor(100, 100, 100)
scene.DrawBLine(k.PointX(200), 200, k.PointX(0), 0)
scene.img
```

```python
I = line_intersect(PQ, k)
I
```

    array([ 496.03960396,  139.6039604 ])

```python
scene.SetColor(255, 255, 255)
scene.DrawPoint(I[0], I[1], pointWidth)
scene.img
```

```python
d = distance(C, I)
d
```

    39.801487608399569

```python
thc = sqrt(circle.radius**2 - d**2)
thc
```

    63.567614271407223

```python
tca = create_line(P, I)
tca
```

    Line (position: [100 100], direction: [ 396.03960396   39.6039604 ])

```python
kL = Line(np.array([k.position[0], k.position[1]]), k.direction)
kL.position[0] -= thc
kR = Line(k.position, k.direction)
kR.position[0] += thc
k, kL, kR
```

    (Line (position: [500 100], direction: [  5. -50.]),
     Line (position: [436 100], direction: [  5. -50.]),
     Line (position: [563 100], direction: [  5. -50.]))

```python
scene.SetColor(50, 50, 50)
scene.DrawBLine(kL.PointX(200), 200, kL.PointX(0), 0)
scene.DrawBLine(kR.PointX(200), 200, kR.PointX(0), 0)
P1 = line_intersect(PQ, kL)
P2 = line_intersect(PQ, kR)
scene.SetColor(0, 255, 0)
scene.DrawBLine(C[0], C[1], P1[0], P1[1])
scene.DrawBLine(P2[0], P2[1], C[0], C[1])
scene.DrawBLine(P1[0], P1[1], P2[0], P2[1])
scene.DrawBLine(C[0], C[1], I[0], I[1])
scene.SetColor(255, 255 ,255)
scene.DrawPoint(P1[0], P1[1], pointWidth)
scene.DrawPoint(P2[0], P2[1], pointWidth)
scene.img
```

## 2D Raytracing

### Vec2

This class implements vector properties.

 * Position $x, y$
 * Normal vector
 * Length of the vector
 * Vector addition/subtraction
 * Vector multiplication
 * Vector dot product

```python
class Vec2():
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def normal(self):
        return Vec2(-self.y, self.x)

    def length(self):
        return sqrt(self.x**2 + self.y**2)

    def __add__(self, v):
        if type(v) is Vec2: return Vec2(self.x + v.x, self.y + v.y)
        elif type(v) in [int, float]: return Vec2(self.x + v, self.y + v)
        else: raise ValueError('Type mismatch \'{}\' for vector addition.'.format(type(v)))

    def __sub__(self, v):
        if type(v) is Vec2: return Vec2(self.x - v.x, self.y - v.y)
        elif type(v) in [int, float]: return Vec2(self.x - v, self.y - v)
        else: raise ValueError('Type mismatch \'{}\' for vector subtraction.'.format(type(v)))

    def __mul__(self, v):
        if type(v) in [int, float, Vec2]: return Vec2(self.x * v, self.y * v)
        else: raise ValueError('Type mismatch \'{}\' for vector-scalar multiplication.'.format(type(v)))

    __rmul__ = __mul__

    def __repr__(self):
        return '({}, {})'.format(self.x, self.y)

    def dot(self, v):
        return self.x * v.x + self.y * v.y

    def transform(self, m):
        v = np.array([self.x, self.y, 1])
        tv = np.dot(m, v)
        return Vec2(tv[0], tv[1])
```

### Camera

The camera has all the properties for finding rays from our view port.

```python
class Camera():
    def __init__(self, x, y, filmSize, focalLength):
        self.position = Vec2(x, y)
        self.filmSize = filmSize
        self.focalLength = focalLength
        self.focalPoint = Vec2(x - focalLength, y)
        self.filmTopPoint = Vec2(x, y + filmSize / 2)
        self.filmBottomPoint = Vec2(x, y - filmSize / 2)
```

### Ray

A ray has a position $x, y$ and a direction which is also a vector. To get this point add the position and direction.

```python
class Ray():
    def __init__(self, position, direction):
        self.position = Vec2(position.x, position.y)
        self.direction = Vec2(direction.x, direction.y)

    def __repr__(self):
        return 'Ray - P: {}, D: {}'.format(self.position, self.direction)
```

### Drawing helpers

To help us draw...

```python
red   = (255, 0  , 0  )
green = (0  , 255, 0  )
blue  = (0  , 0  , 255)
gray  = (100, 100, 100)
white = (255, 255, 255)
```

```python
def draw_vec(scene, v, color):
    originalColor = scene.color
    scene.SetColor(color[0], color[1], color[2])
    scene.DrawPoint(v.x, v.y, 2)
    scene.SetColor(originalColor[0], originalColor[1], originalColor[2])
```

```python
def draw_line(v, u, color):
    originalColor = scene.color
    scene.SetColor(color[0], color[1], color[2])
    scene.DrawBLine(v.x, v.y, u.x, u.y)
    scene.SetColor(originalColor[0], originalColor[1], originalColor[2])
```

```python
def draw_circle(position, r, color):
    originalColor = scene.color
    scene.SetColor(color[0], color[1], color[2])
    scene.DrawCircle(position.x, position.y, r)
    scene.DrawPoint(position.x, position.y, 5)
    scene.SetColor(color[0], color[1], color[2])
```

### Trace

Main function of the algorithm. Here we trace the ray $\vec{PQ}$ that is given as input. We will test it against all the circles to see if there are intersections, and returning the correct color.

* $P$ is `ray.position`
* $Q$ is `ray.position + ray.direction`
* $I$ is `intersectionPoint`
* $C$ is `circle.position`
* $r$ is `circle.radius`

We want to find the closest point $S$ and the color of the circle.

* First we want to create a line $k \perp l$ through $C$.
* Find point $I$, this is were the lines $k$ and $l$ intersect.
* With points $C$ and $I$ we can find length $d$ which is $||\ \vec{CI}\ ||$ (or `(I - C).length()`).
  * $d > r$: ray does not intersect with the circle, we are done.
* $d < r$: ray goes through the circle, find the points $S$ and $S'$.
  * Find the color of the circle and save it if the previous color is found further away.
  * Find $h$ which is $\sqrt{r^2-d^2}$ by Pythagorean theorem.
  * Create a new line $k_l$ from $k$ which is also perpendicular to $l$ and goes through $(k_{lx} - h, k_{ly})$.
  * Find $S$ which is the intersect of $l$ and $k_l$.
  * Do the same for $S'$ by finding $k_r$.
  * Calculate the distance for $||\ \vec{PS}\ ||$ and $||\ \vec{PS'}\ ||$.
  * Return the closest point $S$.

```python
def trace(scene, ray, showTraces = False):
    resultColor = (0, 0, 0)
    resultDistance = scene.width
    for circle in circles:
        #if showTraces:
            #screenEdge = Vec2(800, solve_line_y(ray, 800))
        # Creates a grey triangle for the camera
        draw_line(ray.position, ray.position + ray.direction, gray)

        # Calculate the line k that is perpendicular to our ray l and goes
        # through the circles center C.
        k = Ray(circle.position, ray.direction.normal())
        intersectionPoint = intersect(ray, k)
        if intersectionPoint == False: continue

        # Find the distance d between the intersection point of our ray and the normal
        # and the center of the circle. If this is smaller than the circles radius, we have hit it.
        d = (intersectionPoint - circle.position).length()

        # Calculate a color for this current circle (see gradiant colored circles section)
        cvalue = int(255 - 255 * 0.7 / circle.radius * d)
        color = (0, 0, 0)
        if circle.color == red  : color = (cvalue, 0, 0)
        if circle.color == green: color = (0, cvalue, 0)
        if circle.color == blue : color = (0, 0, cvalue)

        # If the distance is smaller than the radius it is a hit.
        if d < circle.radius:
            # Find h which is the distance from the circle edge S to the intersection point I.
            h = sqrt(circle.radius**2 - d**2)
            # Create a new line that is normal to the ray and goes through I-h.
            kL = Ray(k.position, k.direction)
            kL.position.x -= h
            # Find where they intersect to get the left point on the circle where the
            # line intersected.
            SL = intersect(ray, kL)
            # Same trick, but now for the right point; this is I+h.
            kR = Ray(k.position, k.direction)
            kR.position.x += h
            # Find where they intersect to get the right intersection point.
            SR = intersect(ray, kR)
            # Find the closest point.
            S = SL if (ray.position - SL).length() < (ray.position - SR).length() else SR
            if showTraces:
                #draw_vec(scene, intersectionPoint, color)
                #draw_line(circle.position, intersectionPoint, color)
                draw_vec(scene, S, color)
                draw_line(intersectionPoint, ray.position + ray.direction, color)

        # Save the color of the closest point to our origin which we have hit.
        if d < circle.radius and d < resultDistance:
            resultColor = color
            resultDistance = d

    return resultColor
```

### Intersect

Used to find the point where the two lines $k$ and $l$ intersect. We simply put $l=k$ and solve for $x$.

$$\begin{align} l &= k \\ l_m(x-p_x)+p_y &= k_m(x-q_x)+q_y \\ l_m\cdot x-l_m \cdot p_x + p_y &= k_m\cdot x - k_m - k_m\cdot q_x + q_y \\ l_m\cdot x - k_m \cdot x &= l_m\cdot p_x - p_y - k_m \cdot q_x + q_y \end{align}$$

Now we factor out $x$ on the left side, and divide by $l_m - k_m$ to isolate $x$.

$$x = \dfrac{l_m \cdot p_x - p_y - k_m \cdot q_x + q_y}{l_m - k_m}$$

We can now use this formula to get the $x$ value where the two lines intersect. Finally we use the value to find the $y$ coordinate by plugging in the found $x$ into any of the two lines.

```python
def intersect(k, l):
    m = k.direction.y / k.direction.x
    # assuming that l is always the normal of k (IF IT IS A HORIZONTAL LINE).
    if l.direction.x == 0: return Vec2(l.position.x, k.position.y)
    n = l.direction.y / l.direction.x
    if m - n == 0: return False;
    x = (m * k.position.x - n * l.position.x + l.position.y - k.position.y) / (m - n)
    y = solve_line_y(k, x) if k.direction.y != 0 else k.position.y
    return Vec2(x, y)
```

### Finding y for a given x (ray)

Solving a simple line equation for $y$.

$$ y = m(x-x_0)+y_0$$

```python
def solve_line_y(ray, x):
    m = ray.direction.y / ray.direction.x
    return m * x - m * ray.position.x + ray.position.y
```

### Finding x for a given y (ray)

Solving with the inverse function for $x$.

$$ \begin{align} y &= m(x-x_0)+y_0 \\ y &= mx-mx_0+y_0 \\ mx &= y + mx_0-y_0 \\ x &= \dfrac{y+mx_0-y_0}{m} \end{align}$$

```python
def solve_line_x(ray, y):
    m = ray.direction.y / ray.direction.x
    return (y + m * ray.position.x + ray.position.y) / m
```

### Gradient colored circles

To get nicer colors we can use the light brightness based on distance. I started with the inverse square law. Adjusted the interval to $[0, 255]$. Finally, compensated for circles with different diameters by multiplying by $r$. To alter the brightness `reduced` can be used. The higher the value of `reduced`, the more dim it appears.

$$ \text{color brightness} = 255 - \dfrac{255 \cdot \text{reduced}}{r \sqrt{(q_x-p_x)^2+(q_y-p_y)^2}}$$

```python
def draw_gradient_circle(scene, x, y, r, reduced, color = red):
    P = Vec2(x, y)
    S = Vec2(x - r, y - r)
    M = Vec2(x + r, x + r)
    for i in range(r*2):
        for j in range(r*2):
            Q = Vec2(S.x + i, S.y + j)
            PQ = Q - P
            d = PQ.length()
            if d > r: continue
            if color == red:   scene.SetColor(int(255 - 255 * reduced / r * d), 0, 0);
            if color == green: scene.SetColor(0, int(255 - 255 * reduced / r * d), 0);
            if color == blue:  scene.SetColor(0, 0, int(255 - 255 * reduced / r * d));
            scene.DrawPoint(Q.x, Q.y)

scene = Drawing(600, 200)
reduced = 1
draw_gradient_circle(scene, 100, 100, 100, reduced, red)
draw_gradient_circle(scene, 300, 100, 100, reduced, green)
draw_gradient_circle(scene, 500, 100, 100, reduced, blue)
scene.img
```

To give an idea of how `reduced` behaves for different values.

```python
scene = Drawing(800, 100)
for i in range(10):
    reduced = 0.1 * (i + 5)
    draw_gradient_circle(scene, 50 + (75 * i), 50, 25, reduced, green)
    print('Light {}: reduced = {}'.format(i, round(reduced * 10) / 10))
scene.img
```

### Main loop

Here we loop over all the points in our film and our focal point to find a ray. This ray is then passed to `trace`. This will test the ray against all the circles.

```python
# Viewport (this will be our coordinate system with x: [0, w-1] and y: [0, h-1] starting in the bottom left corner)
width = 800
height = 400

# Settings for the camera. This should be moved into the camera class sometime. The
# camera is centered from the midpoint between the film top and bottom. The focal point
# is moved into the -x direction. The found colors of the objects that are found with
# the ray tracing should be printed onto the film. The resolution on the film 'focalLength' will
# determine how many rays we are going to trace.
filmSize = 200
focalLength = 150
camera = Camera(200, 200, filmSize, focalLength)

# Remember the matrices? Let's rotate the camera with it.
#
# (Bug: the film is drawn from the bottom film point up into the y direction until y+focalLength,
#       instead, it should interpolate between the two points and return 'focalLength' amount of points.
#
#       Also, the rotation goes horribly wrong because we have already scaled and translated by drawing it
#       directly. To fix this it must scale while it is still centered on the origin, then
#       we rotate it, and finally translate to the correct position.
#rotationMatrix = create_rotation_matrix(0)
#focalPoint = focalPoint.transform(rotationMatrix)
#filmTopPoint = filmTopPoint.transform(rotationMatrix)
#filmBottomPoint = filmBottomPoint.transform(rotationMatrix)

# Drawing two scenes. The first is our ray tracing scene, and the second is our film.
scene = Drawing(width, height)
scene2 = Drawing(camera.filmSize, 20)

# Add three different colored and placed circles into the scene.
circles = [Circle(Vec2(600, 200), 100, red), Circle(Vec2(350, 100), 25, green), Circle(Vec2(400, 325), 50, blue)]

# Draw our camera into the scene.
draw_vec(scene, camera.focalPoint, white)
draw_vec(scene, camera.filmTopPoint, white)
draw_vec(scene, camera.filmBottomPoint, white)
draw_line(camera.filmTopPoint, camera.filmBottomPoint, gray)

# Draw all the circles into the scene.
for circle in circles: draw_gradient_circle(scene, circle.position.x, circle.position.y, circle.radius, 0.7, circle.color); # draw_circle(circle.position, circle.radius, circle.color);

# For every point on the film size we want to trace a ray.
for i in range(camera.filmSize):
    # Create a point from the current pixel on our film.
    filmPoint = Vec2(camera.filmBottomPoint.x, camera.filmBottomPoint.y + i)
    # Find the ray between the focal point and the point on the film.
    ray = Ray(camera.focalPoint, filmPoint - camera.focalPoint)
    # Trace the ray to find the color for this pixel on the film.
    color = trace(scene, ray, i % 1 == 0)
    # If there is no object found.
    if color == False: continue
    # Set the color that is found by the ray.
    scene.SetColor(color[0], color[1], color[2])
    scene2.SetColor(color[0], color[1], color[2])
    # Draw the color on the camera film in the first scene for visual purposes.
    for j in range(3):
        scene.DrawPoint(camera.filmBottomPoint.x - j, camera.filmBottomPoint.y + i)
    # Draw the color onto the film in scene2.
    for j in range(20):
        scene.DrawPoint(camera.filmBottomPoint.x - camera.focalLength + i, camera.filmBottomPoint.y - j - 50)
        scene2.DrawPoint(i, j)

scene.img
```

And without the tracing lines.

#### Projected film

This is what gets registered onto the film.

```python
scene2.img
```

## Reflections

We will again start with a simple scene and trace a ray manually. This time we are going to find the angle of reflection for our ray. Finally, we can trace again with this new ray. Find another color. Blend it into the colors we have already found. This is a recursive algorithm.

To find the angle between two lines we use the fact that:

$$ \theta = \tan^{-1}\left(\dfrac{|\ \vec{l_d}\cdot\vec{m_d}|\ }{||\ \vec{l_d}\ ||\cdot||\ \vec{m_d}\ ||}\right) $$

```python
def find_angle(l, m):
    return arctan(abs(m.direction.dot(l.direction)) / (m.direction.length() * l.direction.length()))
```

```python
scene = Drawing(600, 300)

# Circle
C = Vec2(500, 100)
Cr = 50
circle = Circle(C, Cr, red)

draw_gradient_circle(scene, circle.position.x, circle.position.y, circle.radius, 0.6, red)
#draw_circle(circle.position, circle.radius, red)

# Points P and Q
P = Vec2(50, 50)
Q = Vec2(30, 5)

draw_vec(scene, P, white)

l = Ray(P, Q)
draw_line(P, P + Q, red)

# Trace ray PQ
k = Ray(C, l.direction.normal())
I = intersect(l, k)

# Find intersection point S for the blue line with the circle
d = (C-I).length()
if d > Cr: raise ValueError('d > Cr: the line does not intersect the circle.')

h = sqrt(circle.radius**2 - d**2)
kL = Ray(k.position, k.direction)
kL.position.x -= h
S = intersect(l, kL)
draw_vec(scene, S, white)

# Draw right triangle with d, h, S
draw_line(P + Q, S, red)
draw_line(S, I, blue)
draw_line(I, C, blue)

CS = (S-C)
draw_vec(scene, CS + C, gray)
draw_line(CS + C, C, gray)

m = Ray(C, CS)
my = solve_line_y(m, 0)
draw_line(CS + C, Vec2(0, my), gray)

angle = find_angle(l, m)
print('angleLM Θ: {} rad / {} deg'.format(angle, angle * 180 / pi))

# Draw inner triangle construction
p = Ray(S + m.direction, m.direction.normal())

B1 = intersect(m, p)
B2 = intersect(p, l)

draw_vec(scene, B1, white)
draw_vec(scene, B2, white)

draw_line(B2, S + m.direction, white)

# Find B3 (green dot)
Bx = (B1 - B2).x + B1.x
By = solve_line_y(p, Bx)
B3 = Vec2(Bx, By)
draw_vec(scene, B3, green)
draw_line(B1, B3, green)

# The reflection line of l starts at S and goes through B3.
SB3 = Ray(S, B3 - S)
draw_line(SB3.position, SB3.position + SB3.direction, red)
draw_line(SB3.position + SB3.direction, Vec2(0, solve_line_y(SB3, 0)), red)

# Show that both angles are equal.
angleSB3 = find_angle(SB3, m)
print('angleSB3 Θ: {} rad / {} deg'.format(angleSB3, angleSB3 * 180 / pi))

# Draw normal line
normal = Ray(S, p.direction)
nL = Vec2(0, solve_line_y(normal, 0))
nR = Vec2(600, solve_line_y(normal, 600))
draw_line(nL, nR, (50, 50, 50))

scene.img
```

```python
for angle in [i * pi / 2 for i in range(4)]:
    print('angle Θ = {} => x = {}, y = {}'.format(round(angle*100)/100, round(cos(angle)), round(sin(angle))))
```

    angle Θ = 0.0 => x = 1.0, y = 0.0
    angle Θ = 1.57 => x = 0.0, y = 1.0
    angle Θ = 3.14 => x = -1.0, y = 0.0
    angle Θ = 4.71 => x = -0.0, y = -1.0

```python

```


# Linear Models

- [Problem Definition](#problem-definition)
    - [Linear Model](#linear-model)
    - [Simplified CAPM](#simplified-capm)
- [Target Function](#target-function)
- [Raw Data](#raw-data)
- [Preprocessing](#preprocessing)
- [Machine Learning](#machine-learning)
    - [Training](#training)
    - [Model](#model)
    - [Error Function](#error-function)
- [Stochasic Gradient Descent](#stochastic-gradient-descent)
    - [`NumPy`](#sgd-numpy)
    - [`scikit-learn`](#sgd-sklearn)
    - [`TensorFlow`](#sgd-tf)
    - [`Keras`](#sgd-keras)
- [Maximum Likelihood Estimator](#maximum-likelihood-estimator)
    - [`NumPy`](#mle-numpy)
    - [`scikit-learn`](#mle-sklearn)
- [Challenge](#challenge)

## Problem Definition

Provided a set $\mathcal{S}$ of $(x_{i}, y_{i})$ pairs:
    $$\mathcal{S} = \{ (x_{1}, y_{1}), (x_{2}, y_{2}), \text{...}, (x_{k}, y_{k}) \}$$
Find a function $f$ that maps $x_{i} \rightarrow y_{i}$:
    $$f: \mathcal{X} \rightarrow \mathcal{Y}$$

$f$ is a:
* **(Discriminate) Model**
* **Map Function**
* **Function Approximation**
for $\mathcal{S}$, such that:
    $$f(x_{i}) = \hat{y_{i}} \approx y_{i}, \quad \forall i$$

## Linear Model

$f$ is a **Linear Model**, iff:
\begin{equation}
    f(x_{i}) = w * x_{i}, \quad x_{i} \in \mathbb{R}^{n}; y_{i} \in \mathbb{R}^{m}; w \in \mathbb{R}^{m \times n}
\end{equation}

An example of the $k^{th}$ datapoint for $n=2$ and $m=1$ is:
$$ x_{k} = \begin{bmatrix}
               x_{k}^{(1)} & x_{k}^{(2)}
           \end{bmatrix}^{T}
         = \begin{bmatrix}
               1 & 2
           \end{bmatrix}^{T}
\text{,} \quad
   y_{k} = \begin{bmatrix}
               y_{k}^{(1)}
           \end{bmatrix}
         = \begin{bmatrix}
               3
           \end{bmatrix}
$$

For a finite $\mathcal{S}$ where $|\mathcal{S}| = k$, we express the above equation in matrix format as:
    $$f(\mathbf{X}) = \mathbf{\hat{y}} = \mathbf{X} * \mathbf{w},
    \quad \mathbf{X} \in \mathbb{R}^{k \times n}; \mathbf{y} \in \mathbb{R}^{k \times m}; \mathbf{w} \in \mathbb{R}^{n \times m}$$

Matrices $\mathbf{X}$ and $\mathbf{y}$ are constructed by stacking the $k$ pairs as row vectors.

### Simplified CAPM (Capital Asset Pricing Model )

Factor models are a way of explaining the returns of one asset via a linear combination of the returns of other assets.
The general form of a factor model is:
\begin{equation}
    y = \alpha + \beta_{1} x_{1} + \beta_{2} x_{2} + \text{...} + \beta_{n} x_{n}
    \label{eq::beta}
\end{equation}

An asset's $\beta$ to another is just the $\beta$ factor from the equation above.

Look for:
    $$y_{stock} = \beta x_{benchmark} + \alpha =
        \begin{bmatrix}
           x_{benchmark} & 1
       \end{bmatrix}
       \begin{bmatrix}
           \beta \\
           \alpha
       \end{bmatrix}$$

A sample of the dataset $S$ is visualized below:

| $x_{benchmark}$  | $y_{stock}$ |
| ---------------- | ----------- |
| 5.0              | 1.6         |
| 1.0              | 0.6         |
| 3.7              | 1.3         |
| 9.8              | 2.3         |
| 2.8              | 0.8         |

```python
# scientific computing library
import numpy as np
# machine learning library
import sklearn

# visualization tools
import matplotlib.pyplot as plt
import seaborn as sns

# show plots without need of calling `.show()`
%matplotlib inline

# prettify plots
plt.rcParams['figure.figsize'] = [20.0, 5.0]
sns.set_palette(sns.color_palette("muted"))
sns.set_style("ticks")

# supress warnings
import warnings
warnings.filterwarnings('ignore')
```

## Helper Functions

```python
def visualize(x, y=None, y_noise=None, y_hat=None):
    """Visualization helper function.

    Parameters
    ----------
    x: array-like
        x-axis values
    y: array-like
        y-axis target values
    y_noise: array-like
        y-axis observations
    y_hat: array-like
        y-axis model predictions
    """
    if y is not None:
        plt.plot(x, y, label='Target Function')
    if y_noise is not None:
        plt.plot(x, y_noise, 'o', label='Noisy Observations')
    if y_hat is not None:
        plt.plot(x, y_hat, '--o', label='Model Predictions')
    plt.legend();
```

## Target Function

```python
N = 10
np.random.seed(0)

x = np.linspace(0, N, N)
y = 0.5 + 0.2 * x
visualize(x, y);
```

## Raw Data

```python
y_noise = y + np.random.normal(0,0.1, N)

visualize(x, y, y_noise)
```

## Preprocessing

```python
X = np.ones((N, 2))
X[:, 1 ] = x
```

## Machine Learning

Machine learning is the science of getting computers to act without being explicitly programmed.
Their course of action highly depends on the data that has been used in order to **train** them, while the generated program (a.k.a **model**) is a function of the data and the algorithm used to consume this data.

### Training

Training is nothing more than the process of optimizating:
$$\mathcal{J}(w) = \min_{w}\bigg[ \mathbb{E} \big[ \mathcal{L}(\mathbf{y}, f(\mathbf{X} | w)) \big] + \lambda \mathcal{R}(w) \bigg]$$

where,
* $\mathcal{J}$: Objective Function
* $\mathcal{L}$: Loss Function
* $\mathcal{R}$: Regularization Term
* $f$: Mapping Function (`model`)
* $w$: `model` Parameters
* $S = \{ (x_{i}, y_{i}): i \in \mathcal{D} \}$

Assumming that $\mathcal{J}$ has a (global) minimum, such that:

$\DeclareMathOperator*{\argmin}{arg\,min}$
$$w^{*} = \argmin_{w}\bigg[ \mathbb{E} \big[ \mathcal{L}(\mathbf{y}, f(\mathbf{X} | w)) \big] + \lambda \mathcal{R}(w) \bigg]$$

where,
* $w^{*}$: Objective Function (Global) Minimizer

### Model

Having found $w^{*}$ the `model` $f$ can be deployed and predictions can be made according to:

$$\mathbf{\hat{y}} = f(\mathbf{X} | w^{*}) \approx \mathbf{y}$$

In the case of Linear Models, $f$ gets the form of a linear function, with $w^{*}$ coefficients:

$$\mathbf{\hat{y}} = f(\mathbf{X} | w^{*}) = \mathbf{X} * w^{*} \approx \mathbf{y}$$

### Error Function

Use **Mean Squared Error (MSE)** as the metric function in order to evaluate how good/bad the model $f$ is, such that:
    $$MSE_{f} = \frac{1}{k} \sum_{i=1}^{k} (y_{i} - \hat{y}_{i})^{2} = \frac{1}{2k} \sum_{i=1}^{k} (y_{i} - w_{i} * x_{i})^{2}$$
In matrix form:
    $$MSE_{f} = \frac{1}{k} (\mathbf{y} - \mathbf{X} * \mathbf{w})^{T}(\mathbf{y} - \mathbf{X} * \mathbf{w})$$

We can address the optimization problem of $MSE_{f}$:
* computationally (Stochastic Gradient Descent)
* analytically (Maximum Likelihood Estimator)

## Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is the simplest and most used optimizer.
It is based on making incremental updates, towards the correctsolution,
using the derivative of the error at each step.

$$w_{t+1} = w_{t} - \eta (y_{i} - \hat{y}) x_{i}$$

### `NumPy`

```python
# hyperparameters
n_epochs = 50
eta = 0.01

# parameters initialization
w = np.random.random(X.shape[1])

# training error over time
loss = []

# model fitting
for _ in range(n_epochs):
    y_hat = np.dot(X, w)
    l = 0.5 * np.sum((y_noise - y_hat)**2)
    i = np.random.randint(len(y_noise))
    error = y_hat[i] - y_noise[i]
    w = w - eta * error * X[i]
    loss.append(l)

# plot results
visualize(x, y, y_noise, y_hat)

# plot training error
plt.figure()
plt.plot(loss)
plt.title('Learning Curve')
plt.legend(['Training Error']);
```

### `scikit-learn`

```python
from sklearn.linear_model import SGDRegressor

# hyperparameters
n_epochs = 100
eta = 0.01

# model initialization
model = SGDRegressor(loss='squared_loss', penalty='none', max_iter=n_epochs, learning_rate='constant', eta0=eta)

# model fitting & predictions
model.fit(X, y_noise)
y_hat = model.predict(X).ravel()

# plot results
visualize(x, y, y_noise, y_hat)
```

## Maximum Likelihood Estimator

Differentiate $MSE_{f}(w)$ and find its minimum. There is a global minimiser, since $MSE_{f}$ is convex (quadratic).

$$\mathbf{w_{MLE}} = (\mathbf{X}^{T} \mathbf{X})^{-1} * \mathbf{X}^{T} * \mathbf{y}$$

### `NumPy`

```python
w = np.dot(np.dot(np.linalg.inv(np.dot(X.T, X)), X.T), y_noise)
y_hat = np.dot(X, w)

visualize(x, y, y_noise, y_hat)
```

### `scikit-learn`

```python
from sklearn.linear_model import LinearRegression

# model initialization
model = LinearRegression()

# model fitting & predictions
model.fit(X, y_noise)
y_hat = model.predict(X).ravel()

# plot results
visualize(x, y, y_noise, y_hat)
```

## Challenges

### Beta Hedging

Repeat the above experiment using the implementation you prefer most, on real world market data.

Using the data provided in `{proj_dir}/notebooks/data/`:
* **S&P500 (SPY)**: as the market index (benchmark)
* **Apple (AAPL)**: as stock A
* **Google (GOOG)** as stocks B

come up with a Beta Hedging Strategy to minimize risk.

Load the data to a numpy.ndarray using the commands below:

```python
# helper loader function
loader = lambda asset: np.loadtxt('./data/%s.txt' % asset, delimiter=',')

# raw data
spy = loader('SPY')
aapl = loader('AAPL')
goog = loader('GOOG')

# normalised data
spy_ = spy / spy[0] - 1
aapl_ = aapl / aapl[0] - 1
goog_ = goog / goog[0] - 1

# visualization
plt.plot(spy_, label='S&P500')
plt.plot(aapl_, label='Apple Inc.')
plt.plot(goog_, label='Alphabet Inc.')
plt.title('Normalised Prices')
plt.ylabel('Normalised Change')
plt.xlabel('Time Index')
plt.legend();
```

```python
# regression plots
sns.regplot(spy_, aapl_, label='APPL')
sns.regplot(spy_, goog_, label='GOOG')
plt.title('Beta Exposure to S&P500')
plt.xlabel('SPY Percentage Change')
plt.ylabel('Stock Price Percentage Change')
plt.legend();
```

```python
# helper percentage return
pct_change = lambda series: np.diff(series) / (series[:-1] + 1e-6) # WARNING: prone to division by zero

aapl_pct = pct_change(aapl)
goog_pct = pct_change(goog)
spy_pct = pct_change(spy)

sns.distplot(aapl_pct, label='AAPL')
sns.distplot(goog_pct, label='GOOG')
sns.distplot(spy_pct, label='SPY')
plt.legend();
```


```python
# import pandas as pd
# import numpy as np
import matplotlib.pyplot as plt
# import math
# from networkx import minimum_cut, DiGraph
from mip import *
# from itertools import product

from src.read_instance import MDOVRP
# rnd = np.random
# rnd.seed(454)
```

```python
# filepath = "Instancias/ES-n78-m2-Q10138.txt"
filepath = "Instancias/Vrp-Set-A/A-n80-m2-Q60.vrp"
```

```python
N, D, V, Q, q, c, coord_x, coord_y = MDOVRP(filepath)
```

```python
# fazendo c ser simétrica
# c = {(i,j) : (c[i,j] + c[j, i])/2  for (i,j) in c}
```

Usando a formulação $MDOVRP_{2i− flv}$ de Lalla-Ruiz e Mes (2019)

    Lalla-Ruiz, Eduardo, and Martijn Mes. "Mathematical formulations and improvements for the multi-depot open vehicle routing problem." Optimization Letters 15 (2021): 271-286.

```python
#cria o modelo
model = Model('PRVMD', solver_name = GUROBI)

# Variaveis de decisao
x = [[model.add_var(var_type=BINARY) if i!=j else model.add_var(lb=0, ub=0) for i in V] for j in V]
u = [[model.add_var(var_type=CONTINUOUS) if i!=j else model.add_var(lb=0, ub=0) for i in V] for j in V]
y = [model.add_var(var_type=BINARY) for i in N]
```

    Set parameter WLSAccessID
    Set parameter WLSSecret
    Set parameter LicenseID
    Academic license - for non-commercial use only - registered to thiago.giachetto@aluno.ufop.edu.br

\begin{equation}
\label{eq:9}
    x_{ij} \in \{0, 1\}, \quad \forall i, j \in V
\end{equation}

\begin{equation}
\label{eq:10}
    u_{ij} \geq 0, \quad \forall i, j \in V
\end{equation}

```python
# Funcao objetivo
model.objective = minimize(xsum(x[i][j] * c[i,j] for i in V for j in V if i != j))
```

\begin{equation}
\label{eq:1}
 Minimizar \quad \sum_{i \in V }\sum_{j \in V} c_{ij}x_{ij}
\end{equation}

```python
# Restricao 2
for j in N:
#     model += xsum(x[i][j] for i in V if i != j and c[i][j] <= 180) == 1
    model += xsum(x[i][j] for i in V if i != j) == 1
```

\begin{equation}
    \sum_{i \in V, i \neq j}x_{ij} = 1, \quad \forall j \in N
    \tag{2}
\end{equation}

```python
# Restricao 18

for j in N:
    model += (xsum(x[i][j] for i in V if i != j) - xsum(x[j][i] for i in N if i != j)) >= 0
```

\begin{equation}
    \sum_{i \in V, i \neq j}x_{ij} - \sum_{i \in N, i \neq j}x_{ji} \geq 0, \quad \forall j \in N
    \tag {18}
\end{equation}

```python
# Restricao 19
for i in V:
    for j in V:
        if i != j:
            model += x[i][j] + x[j][i] <= 1
```

\begin{equation}
    x_{ij} + x_{ji} \leq 1, \quad \forall i, j \in V, i \neq j
    \tag {19}
\end{equation}

```python
# Restricao 20
# Ninguem volta para o depósito
model += xsum(x[j][k] for k in D for j in V) == 0
```

\begin{equation}
    \sum_{k \in D}\sum_{j \in V}x_{jk} = 0
    \tag {20}
\end{equation}

```python
# Restricao 21
for j in N:
    model += xsum(u[i][j] for i in V if i != j) - xsum(u[j][i] for i in V if i != j) >= q[j]
```

\begin{equation}
    \sum_{i \in V, i \neq j}u_{ij} - \sum_{i \in V, i \neq j}u_{ji} \geq q_j, \quad \forall j \in N
    \tag {21}
\end{equation}

```python
#Restricao 22
for i in N:
    for j in N:
        model += (Q - q[i])*x[i][j] >= u[i][j]
```

\begin{equation}
    (Q - q_i) \cdot x_{ij} \geq u_{ij}, \quad \forall i, j \in N
    \tag {22}
\end{equation}

```python
# Restricao 23
for k in D:
    for j in N:
        model += Q * x[k][j] >= u[k][j]
```

\begin{equation}
    Q \cdot x_{kj} \geq u_{kj}, \quad \forall k \in D, j \in N
    \tag {23}
\end{equation}

```python
# Restricao 8
d = [min([c[j, i] for j in D]) for i in N]
r = [min([c[j, i] for j in N if i != j]) for i in N]
M = max(r)

for i in N:
    model += d[i] + M * y[i] >= r[i]
```

\begin{equation}
    d_{i} + M y_{i} \geq r_{i}, \quad \forall k \in N
    \tag {8}
\end{equation}

```python
# Restricao 9

for i in N:
    k_l = D[np.argmin([c[j, i] for j in D], axis=0)]
    model += x[k_l][i] >= y[i]
```

\begin{equation}
    x_{k'i} \geq y_{i}, \quad \forall i \in N, k' = argmin(c_{ki})_{k \in D}
    \tag {9}
\end{equation}

```python
# Restricao 10

model += xsum(x[k][i] for i in N for k in D) >= np.ceil(sum(q)/Q)
```

\begin{equation}
    \sum_{i \in N}\sum_{k \in D}x_{ki} \geq \lceil \frac{\sum_{i \in N} q_{i}}{Q} \rceil
    \tag {10}
\end{equation}

```python
# Restricao de distancia maxima entre duas cidades

for i in V:
    for j in V:
        if i != j:
            model += c[i,j]*x[i][j] <= 180
```

```python
# Resolve o modelo
# model.optimize(max_seconds=7200)
model.optimize(max_seconds=30)
print("status: {} objective value : {} best possible: {}".format(model.status, model.objective_value, model.objective_bound))

#                   Objective                Residual
# Iter       Primal          Dual         Primal    Dual     Compl     Time
# usando c simétrica
# 421367 260494 2594.76360   95  102 2608.15000 2560.04596  1.84%   117 7200s
# usando c original
# 351473 211907 2581.12423   69  161 2608.75395 2552.97499  2.14%   138 7200s
```

    Set parameter TimeLimit to value 30
    Set parameter NodeLimit to value 1073741824
    Set parameter SolutionLimit to value 1073741824
    Set parameter IntFeasTol to value 1e-06
    Set parameter Method to value 3
    Gurobi Optimizer version 9.5.0 build v9.5.0rc5 (linux64)
    Thread count: 2 physical cores, 4 logical processors, using up to 4 threads
    Academic license - for non-commercial use only - registered to thiago.giachetto@aluno.ufop.edu.br
    Optimize a model with 20246 rows, 13528 columns and 65850 nonzeros
    Model fingerprint: 0xc772627a
    Variable types: 6806 continuous, 6722 integer (6722 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 2e+02]
      Objective range  [1e+01, 2e+02]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 2e+02]
    Found heuristic solution: objective 5766.8171932
    Presolve removed 10837 rows and 1224 columns
    Presolve time: 0.27s
    Presolved: 9409 rows, 12304 columns, 43024 nonzeros
    Found heuristic solution: objective 3445.4188966
    Variable types: 6112 continuous, 6192 integer (6192 binary)

    Concurrent LP optimizer: dual simplex and barrier
    Showing barrier log only...

    Root barrier log...

    Ordering time: 0.01s

    Barrier statistics:
     AA' NZ     : 1.198e+04
     Factor NZ  : 2.179e+04 (roughly 2 MB of memory)
     Factor Ops : 3.017e+05 (less than 1 second per iteration)
     Threads    : 1

                      Objective                Residual
    Iter       Primal          Dual         Primal    Dual     Compl     Time
       0   1.70769949e+06 -1.55562149e+06  2.13e+03 2.41e+01  2.45e+03     0s
       1   6.72411615e+05 -1.34933120e+06  8.45e+02 5.33e+01  1.02e+03     0s
       2   1.30258449e+05 -5.20668748e+05  1.63e+02 2.56e-13  2.04e+02     0s
       3   1.62040735e+04 -6.00647008e+04  1.38e+01 1.99e-13  1.82e+01     0s
       4   5.52603825e+03 -1.72476820e+04  1.24e-14 6.71e-14  2.60e+00     0s
       5   4.72022242e+03 -3.11747986e+03  4.26e-14 4.26e-14  8.95e-01     0s
       6   3.65069547e+03 -2.63388943e+02  6.39e-14 2.84e-14  4.47e-01     0s
       7   2.54614550e+03  2.62204181e+02  8.17e-14 2.19e-14  2.61e-01     0s
       8   2.30141863e+03  6.69097745e+02  6.93e-14 1.67e-14  1.86e-01     0s
       9   2.08740671e+03  7.41358063e+02  1.21e-13 1.42e-14  1.54e-01     0s
      10   1.66164855e+03  8.17920132e+02  9.24e-14 1.51e-14  9.64e-02     0s
      11   1.33113494e+03  9.04432826e+02  1.67e-13 2.84e-14  4.87e-02     0s
      12   1.19859720e+03  9.27366258e+02  1.10e-13 1.49e-14  3.10e-02     0s

    Barrier performed 12 iterations in 0.45 seconds (0.23 work units)
    Barrier solve interrupted - model solved by another algorithm

    Solved with dual simplex

    Root relaxation: objective 9.532590e+02, 1204 iterations, 0.11 seconds (0.04 work units)

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

    *    0     0               0     953.2590181  953.25902  0.00%     -    0s

    Explored 1 nodes (1936 simplex iterations) in 0.49 seconds (0.24 work units)
    Thread count was 4 (of 4 available processors)

    Solution count 3: 953.259 3445.42 5766.82

    Optimal solution found (tolerance 1.00e-04)
    status: OptimizationStatus.OPTIMAL objective value : 953.2590180780452 best possible: 953.2590180780452Best objective 9.532590180780e+02, best bound 9.532590180780e+02, gap 0.0000%

```python
# --- PLOTA O GRAFICO ---
plt.figure(figsize=(10, 6))
plt.scatter(coord_x[0:], coord_y[0:])
for i in N:
    plt.annotate(f"{i}", (coord_x[i], coord_y[i]))
for i in D:
    plt.plot(coord_x[i], coord_y[i], c = 'r', marker = 's')

for i in V:
    for j in V:
        if x[i][j].x > 0:
            plt.plot([coord_x[i], coord_x[j]], [coord_y[i], coord_y[j]], c='g', zorder=0)

#escala dos eixos
plt.yticks([i for i in range(1,80,20)]);
plt.xticks([i for i in range(1,80,20)]);
```


## Data Analytics

### Interactive Distribution Transformations in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Data Analytics: Distribution Transformations

Here's a demonstration of making and general use of distribution transformations in Python. This demonstration is part of the resources that I include for my courses in Spatial / Subsurface Data Analytics at the Cockrell School of Engineering at the University of Texas at Austin.

#### Distribution Transformations

Why do we do this?

* **Inference**: variable has expected shape

* **Data Preparation / Cleaning**: correcting for too few data and outliers

* **Theory**: a specific distribution assumption required for a method

How do we do it?

We apply this to all sample data, $x_{\alpha}$ $\forall$ $\alpha = 1,\ldots,n$.

\begin{equation}
y_{\alpha} = G^{-1}_Y\left(F_X(x_{\alpha})\right)
\end{equation}

were $X$ is the original feature with a $F_X$ original cumulative distribution function and $Y$ is transformed feature with a $G_Y$ transformed cumulative distribution function.

* Mapping from one distribution to another through percentiles

* This may be applied to any parametric or nonparametric distributions

* This is a rank preserving transform, e.g. P50 of 𝑋 is P50 of 𝑌

I have a lecture on distribution transformations available on [YouTube](https://www.youtube.com/watch?v=ZDIpE3OkAIU&list=PLG19vXLQHvSB-D4XKYieEku9GQMQyAzjJ&index=14).

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - [sample_data.csv](https://github.com/GeostatsGuy/GeoDataSets/blob/master/sample_data.csv).

#### Importing Packages

We will need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # plotting
from scipy import stats                   # summary statistics
import math                               # trigonometry etc.
import scipy.signal as signal             # kernel for moving window calculation
import random                             # randon numbers
import seaborn as sns                     # matrix scatter plots
from scipy.stats import norm              # Gaussian parametric distribution
from sklearn import preprocessing
import geostatspy.GSLIB as GSLIB
from ipywidgets import interactive                      # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
```

#### Set the Random Number Seed

Set the random number seed so that we have a repeatable workflow

```python
seed = 73073
```

#### Set the Working Directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.  For fun try misspelling the name. You will get an ugly, long error.

```python
df = pd.read_csv('sample_data.csv')     # load our data table (wrong name!)
```

It worked, we loaded our file into our DataFrame called 'df'. But how do you really know that it worked? Visualizing the DataFrame would be useful and we already leard about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head(n=6)                           # we could also use this command for a table preview
```

      X
      Y
      Facies
      Porosity
      Perm
      AI

      0
      100.0
      900.0
      1.0
      0.100187
      1.363890
      5110.699751

      1
      100.0
      800.0
      0.0
      0.107947
      12.576845
      4671.458560

      2
      100.0
      700.0
      0.0
      0.085357
      5.984520
      6127.548006

      3
      100.0
      600.0
      0.0
      0.108460
      2.446678
      5201.637996

      4
      100.0
      500.0
      0.0
      0.102468
      1.952264
      3835.270322

      5
      100.0
      400.0
      0.0
      0.110579
      3.691908
      5295.267191

#### Calculating and Plotting a CDF by Hand

Let's demonstrate the calculation and plotting of a non-parametric CDF by hand

1. make a copy of the feature as a 1D array (ndarray from NumPy)
2. sort the data in ascending order
3. assign cumulative probabilities based on the tail assumptions
4. plot cumuative probability vs. value

```python
por = df['Porosity'].copy(deep = True).values # make a deepcopy of the feature from the DataFrame
print('The ndarray has a shape of ' + str(por.shape) + '.')

por = np.sort(por)                           # sort the data in ascending order
n = por.shape[0]                             # get the number of data samples

cprob = np.zeros(n)
for i in range(0,n):
    index = i + 1
    cprob[i] = index / n                     # known upper tail
    # cprob[i] = (index - 1)/n               # known lower tail
    # cprob[i] = (index - 1)/(n - 1)         # known upper and lower tails
    # cprob[i] = index/(n+1)                 # unknown tails

plt.subplot(111)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.2, wspace=0.1, hspace=0.2)
plt.show()
```

#### Transformation to a Parametric Distribution

We can transform our data feature distribution to any parametric distribution with this workflow.

1. Calculate the cumulative probability value of each of our data values, $p_{\alpha} = F_x(x_\alpha)$, $\forall$ $\alpha = 1,\ldots, n$.

2. Apply the inverse of the target parametric cumulative distribution function (CDF) to calculate the transformed values. $y_{\alpha} = G_y^{-1}\left(F_x(x_\alpha)\right)$, $\forall$ $\alpha = 1,\ldots, n$.

```python
y = np.zeros(n)

for i in range(0,n):
    y[i] = norm.ppf(cprob[i],loc=0.0,scale=1.0)

plt.subplot(121)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplot(122)
plt.plot(y,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(y,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([-3.0,3.0]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("After Distribution Transformation to Gaussian")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

```

Let's make an interactive version of this plot to visualize the transformation.

```python
# widgets and dashboard
l = widgets.Text(value='                                                  Data Analytics, Distribution Transformation, Prof. Michael Pyrcz, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

data_index = widgets.IntSlider(min=1, max = n-1, value=1.0, step = 10.0, description = 'Data Index, $\\alpha$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)

ui = widgets.VBox([l,data_index],)

def run_plot(data_index):                       # make data, fit models and plot
    plt.subplot(131)
    plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(por,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
    plt.xlabel("Original Feature, $x$"); plt.ylabel("Cumulative Probability"); plt.title("Original Cumulative Distribution Function")
    plt.plot([por[data_index-1],por[data_index-1]],[0.0,cprob[data_index-1]],color = 'red',linestyle='dashed')
    plt.plot([por[data_index-1],3.0],[cprob[data_index-1],cprob[data_index-1]],color = 'red',linestyle='dashed')
    plt.annotate('x = ' + str(round(por[data_index-1],2)), xy=(por[data_index-1]+0.003, 0.01))
    plt.annotate('p = ' + str(round(cprob[data_index-1],2)), xy=(0.225, cprob[data_index-1]+0.02))

    plt.subplot(132)
    plt.plot(y,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(y,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
    plt.grid(); plt.xlim([-3.0,3.0]); plt.ylim([0.0,1.0])
    plt.xlabel("Gaussian Transformed Feature, $y$"); plt.ylabel("Cumulative Probability"); plt.title("After Distribution Transformation to Gaussian")
    plt.plot([-3.0,y[data_index-1]],[cprob[data_index-1],cprob[data_index-1]],color = 'red',linestyle='dashed')
    plt.plot([y[data_index-1],y[data_index-1]],[0.0,cprob[data_index-1]],color = 'red',linestyle='dashed')
    #plt.arrow(y[data_index-1],cprob[data_index-1],0.0,-1.0*(cprob[data_index-1]-0.01),color = 'red',width = 0.02, head_width = 0.1, linestyle='dashed', head_length = 0.01)
    plt.annotate('p = ' + str(round(cprob[data_index-1],2)), xy=(-2.90, cprob[data_index-1]+0.02))
    plt.annotate('y = ' + str(round(y[data_index-1],2)), xy=(y[data_index-1]+0.1, 0.01))

    plt.subplot(133)
    plt.plot(por,y, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([-3.0,3.0])
    plt.xlabel("Original Porosity (fraction)"); plt.ylabel("Gaussian Transformed Porosity (N[fraction])"); plt.title("Parametric Distribution Transformation, Q-Q Plot")
    #plt.plot([0.05,0.25],[0.05,0.25],color = 'red',linestyle='dashed', alpha = 0.4)
    plt.scatter(por[data_index-1],y[data_index-1],s = 50, c = 'red', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points
    plt.scatter(por,y,s = 20, c = 'red', edgecolor = 'black', alpha = 0.1, zorder=100) # plot the CDF points

    plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(run_plot, {'data_index':data_index})
interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Data Analytics Distribution Transformation Demonstration

#### Michael Pyrcz, Associate Professor, The University of Texas at Austin

Select any data value and observe the distribution transform by mapping through cumulative probability.

### The Inputs

* **data_index** - the data index from 1 to n in the sorted ascending order

```python
display(ui, interactive_plot)                           # display the interactive plot
```

    VBox(children=(Text(value='                                                  Data Analytics, Distribution Tran…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '
', 'i…

#### Distribution Transform to a Non-Parametric Distribution

We can apply the mapping through cumulative probabilities to transform from any distribution to any other distribution.

* let's make a new data set by randomly sampling from the previous one and adding error

Then we can demonstrate transforming this dataset to match the original distribution

* this is mimicking the situation where we transform a dataset to match the distribution of a better sampled analog distribution

```python
n_sample = 30
df_sample = df.sample(n_sample,random_state = seed)

df_sample = df_sample.copy(deep = True) # make a deepcopy of the feature from the DataFrame

df_sample['Porosity'] = df_sample['Porosity'].values + np.random.normal(loc = 0.0, scale = 0.01, size = n_sample)

df_sample = df_sample.sort_values(by = 'Porosity')                # sort the DataFrame
por_sample = df_sample['Porosity'].values
print('The sample ndarray has a shape of ' + str(por_sample.shape) + '.')

cprob_sample = np.zeros(n_sample)
for i in range(0,n_sample):
    index = i + 1
    cprob_sample[i] = index / n_sample       # known upper tail
    # cprob[i] = (index - 1)/n               # known lower tail
    # cprob[i] = (index - 1)/(n - 1)         # known upper and lower tails
    # cprob[i] = index/(n+1)                 # unknown tails

plt.subplot(121)
plt.plot(por_sample,cprob_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por_sample,cprob_sample,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Sparse Sample with Noise Cumulative Distribution Function")

plt.subplot(122)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 10, alpha = 1.0, c = 'red', edgecolor = 'black') # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Let's transform the values and show them on the target distribution.

```python
y_sample = np.zeros(n_sample)

for i in range(0,n_sample):
    y_sample[i] = np.percentile(por,cprob_sample[i]*100, interpolation = 'linear') # piecewise linear interpolation of inverse of target CDF

plt.subplot(121)
plt.plot(por_sample,cprob_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por_sample,cprob_sample,s = 30, alpha = 1.0, c = 'green', edgecolor = 'black', zorder = 100) # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Sparse Sample with Noise Cumulative Distribution Function")

plt.subplot(122)
plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
plt.scatter(por,cprob,s = 10, c = 'red', edgecolor = 'black', alpha = 0.3) # plot the CDF points
plt.scatter(y_sample,cprob_sample,s = 30, c = 'green', edgecolor = 'black', alpha = 1.0, zorder = 100) # plot the CDF points
plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Porosity Cumulative Distribution Function")

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Let's make an interactive version of this plot to visualize the transformation.

```python
# widgets and dashboard
l_sample = widgets.Text(value='                                                  Data Analytics, Distribution Transformation, Prof. Michael Pyrcz, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

data_index_sample = widgets.IntSlider(min=1, max = n_sample, value=1.0, step = 1.0, description = 'Data Sample Index, $\\beta$',orientation='horizontal', style = {'description_width': 'initial'}, continuous_update=False)

ui_sample = widgets.VBox([l_sample,data_index_sample],)

def run_plot_sample(data_index_sample):                       # make data, fit models and plot
    plt.subplot(131)
    plt.plot(por_sample,cprob_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(por_sample,cprob_sample,s = 30, alpha = 1.0, c = 'green', edgecolor = 'black',zorder = 100) # plot the CDF points
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
    plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Original Sparse Sample with Noise, Cumulative Distribution Function")
    plt.plot([por_sample[data_index_sample-1],por_sample[data_index_sample-1]],[0.0,cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.plot([por_sample[data_index_sample-1],3.0],[cprob_sample[data_index_sample-1],cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.annotate('x = ' + str(round(por_sample[data_index_sample-1],2)), xy=(por_sample[data_index_sample-1]+0.003, 0.01))
    plt.annotate('p = ' + str(round(cprob_sample[data_index_sample-1],2)), xy=(0.225, cprob_sample[data_index_sample-1]+0.02))

    plt.subplot(132)
    plt.plot(por,cprob, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.scatter(por,cprob,s = 10, c = 'red', edgecolor = 'black', alpha = 1.0) # plot the CDF points
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.0,1.0])
    plt.xlabel("Porosity (fraction)"); plt.ylabel("Cumulative Probability"); plt.title("Non-parametric Target Porosity Cumulative Distribution Function")
    plt.plot([0.0,y_sample[data_index_sample-1]],[cprob_sample[data_index_sample-1],cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.plot([y_sample[data_index_sample-1],y_sample[data_index_sample-1]],[0.0,cprob_sample[data_index_sample-1]],color = 'red',linestyle='dashed')
    plt.annotate('p = ' + str(round(cprob_sample[data_index_sample-1],2)), xy=(0.053, cprob_sample[data_index_sample-1]+0.02))
    plt.annotate('y = ' + str(round(y_sample[data_index_sample-1],2)), xy=(y_sample[data_index_sample-1]+0.003, 0.01))
    plt.scatter(y_sample[data_index_sample-1],cprob_sample[data_index_sample-1],s = 50, c = 'green', edgecolor = 'black', alpha = 1.0, zorder=100) # plot the CDF points

    plt.subplot(133)
    plt.plot(por_sample,y_sample, alpha = 0.2, c = 'black') # plot piecewise linear interpolation
    plt.grid(); plt.xlim([0.05,0.25]); plt.ylim([0.05,0.25])
    plt.xlabel("Original Porosity (fraction)"); plt.ylabel("Transformed Porosity (fraction)"); plt.title("Non-parametric Distribution Transformation, Q-Q Plot")
    plt.plot([0.05,0.25],[0.05,0.25],color = 'red',linestyle='dashed', alpha = 0.4)
    plt.scatter(por_sample[data_index_sample-1],y_sample[data_index_sample-1],s = 50, c = 'green', edgecolor = 'black', alpha = 1.0, zorder=200) # plot the CDF points
    plt.scatter(por_sample,y_sample,s = 20, c = 'green', edgecolor = 'black', alpha = 0.3, zorder=100) # plot the CDF points

    plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot_s = widgets.interactive_output(run_plot_sample, {'data_index_sample':data_index_sample})
#interactive_plot_sample.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Data Analytics Distribution Transformation Demonstration

#### Michael Pyrcz, Associate Professor, The University of Texas at Austin

Select any data value and observe the distribution transform by mapping through cumulative probability.

#### The Inputs

* **data_index** - the data index from 1 to n in the sorted ascending order

```python
display(ui_sample, interactive_plot_s)                           # display the interactive plot
```

    VBox(children=(Text(value='                                                  Data Analytics, Distribution Tran…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '
', 'i…

To summarize let's look at a DataFrame with the original noisey sample and the transformed to match the original distribution.

* we're making and showing a table of original values, $x_{\beta}$ $\forall$ $\beta = 1, \ldots, n_{sample}$, and the transformed values, $y_{\beta}$ $\forall$ $\beta = 1, \ldots, n_{sample}$.

```python
df_sample['Transformed_Por'] = y_sample
df_sample.head(n=n_sample)
```

      X
      Y
      Facies
      Porosity
      Perm
      AI
      Transformed_Por

      80
      900.0
      100.0
      0.0
      0.085512
      1.280257
      4573.656072
      0.081044

      207
      201.0
      426.0
      0.0
      0.087920
      0.400658
      5263.542112
      0.085867

      210
      231.0
      426.0
      0.0
      0.091760
      5.584040
      4919.074871
      0.091834

      189
      201.0
      456.0
      0.0
      0.096367
      0.546396
      5018.355476
      0.095628

      72
      900.0
      900.0
      0.0
      0.105738
      12.433996
      6242.704810
      0.099378

      3
      100.0
      600.0
      0.0
      0.107195
      2.446678
      5201.637996
      0.101987

      41
      500.0
      400.0
      0.0
      0.114506
      6.312198
      5515.918646
      0.103970

      5
      100.0
      400.0
      0.0
      0.115988
      3.691908
      5295.267191
      0.106704

      218
      251.0
      416.0
      0.0
      0.118467
      1.003374
      5822.467914
      0.108460

      47
      600.0
      700.0
      0.0
      0.123547
      12.384496
      3595.586977
      0.111491

      226
      211.0
      396.0
      0.0
      0.124984
      6.368529
      5725.334803
      0.113887

      71
      800.0
      100.0
      0.0
      0.131719
      7.739105
      5274.532660
      0.117984

      53
      600.0
      100.0
      1.0
      0.154893
      42.396044
      4204.150893
      0.121592

      93
      955.0
      549.0
      1.0
      0.175362
      374.298925
      3181.557281
      0.127131

      151
      995.0
      489.0
      1.0
      0.186489
      460.494986
      2792.804322
      0.137062

      150
      985.0
      489.0
      1.0
      0.187811
      73.133040
      2672.294567
      0.153759

      245
      690.0
      529.0
      1.0
      0.188944
      316.905689
      4271.013148
      0.176536

      158
      975.0
      479.0
      1.0
      0.195490
      69.336576
      2493.128177
      0.185270

      174
      955.0
      459.0
      1.0
      0.197341
      45.002088
      3394.563038
      0.188136

      111
      955.0
      529.0
      1.0
      0.198112
      1113.971076
      3177.635737
      0.191655

      165
      955.0
      469.0
      1.0
      0.198773
      26.197239
      2889.196647
      0.195998

      113
      975.0
      529.0
      1.0
      0.201255
      1548.094062
      3167.185377
      0.198182

      99
      925.0
      539.0
      1.0
      0.203368
      211.163296
      3442.885245
      0.199315

      84
      955.0
      559.0
      1.0
      0.205265
      74.215058
      3386.182722
      0.201943

      136
      935.0
      499.0
      1.0
      0.207755
      523.287810
      2579.032897
      0.206934

      141
      985.0
      499.0
      1.0
      0.210925
      68.276148
      2547.526113
      0.209051

      181
      935.0
      449.0
      1.0
      0.216698
      368.507601
      4249.477923
      0.211638

      125
      1005.0
      519.0
      1.0
      0.226149
      54.667195
      2577.714678
      0.215686

      149
      975.0
      489.0
      1.0
      0.232487
      21.109085
      2412.875330
      0.224214

      129
      955.0
      509.0
      1.0
      0.237183
      1525.247066
      2512.061434
      0.242298

It would be straitforward to modify the code above to perform distribution transformations:

* to a parametric distribution like Gaussian

* to a non-parametric distribution from actual data (build a CDF and interpolate between the data samples)

#### Comments

This was a basic demonstration of distribution transformations.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at [Python Demos](https://github.com/GeostatsGuy/PythonNumericalDemos) and a Python package for data analytics and geostatistics at [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


## Surfinpy

#### Tutorial 1 - Bulk Phase diagrams

In this tutorial we learn how to generate a basic bulk phase diagram from DFT energies.  This enables the comparison of the thermodynamic stability of various different bulk phases under different chemical potentials giving valuable insight in to the syntheis of solid phases.  This example will consider a series of bulk phases which can be defined through a reaction scheme across all phases, thus for this example including MgO, H2O and CO2 as reactions and A as a generic product.

\begin{align}
x\text{MgO} + y\text{H}_2\text{O} + z\text{CO}_2 \rightarrow \text{A}
\end{align}

##### Methodology

The system is in equilibrium when the chemical potentials of the reactants and product are equal; i.e. the change in Gibbs free energy is $\delta G_{T,p} = 0$.

\begin{align}
	\delta G_{T,p} = \mu_A - x\mu_{\text{MgO}} - y\mu_{\text{H}_2\text{O}} - z\mu_{\text{CO}_2} = 0
\end{align}

Assuming that H2O and CO2 are gaseous species, $\mu_{CO_2}$ and $\mu_{H_2O}$ can be written as

\begin{align}
	\mu_{\text{H}_2\text{O}} = \mu^0_{\text{H}_2\text{O}} + \Delta\mu_{\text{H}_2\text{O}}
\end{align}
and
\begin{align}
	\mu_{\text{CO}_2} = \mu^0_{\text{CO}_2} + \Delta\mu_{\text{CO}_2}
\end{align}

The chemical potential $\mu^0_x$ is the partial molar free energy of any reactants or products (x) in their standard states, in this example we assume all solid components can be expressed as

\begin{align}
\mu_{\text{component}} = \mu^0_{\text{component}}
\end{align}

Hence, we can now rearrange the equations to produce;

\begin{align}
	\mu^0_A - x\mu^0_{\text{MgO}} - y\mu^0_{\text{H}_2\text{O}} - z\mu^0_{\text{CO}_2} = y\Delta\mu_{\text{H}_2\text{O}} + z\Delta\mu_{\text{CO}_2}
\end{align}

As $\mu^0_A$  corresponds to the partial molar free energy of product A, we can replace the left side with the Gibbs free energy ($\Delta G_{\text{f}}^0$).

\begin{align}
	\delta G_{T,p} = \Delta G_{\text{f}}^0 - y\Delta\mu_{\text{H}_2\text{O}} - z\Delta\mu_{\text{CO}_2}
\end{align}

At equilibrium $\delta G_{T,p} = 0$, and hence

\begin{align}
	\Delta G_{\text{f}}^0 = y\Delta\mu_{\text{H}_2\text{O}} + z\Delta\mu_{\text{CO}_2}
\end{align}

Thus, we can find the values of $\Delta\mu_{\text{H}_2\text{O}}$ and $\Delta\mu_{\text{CO}_2}$ (or $(p_{\text{H}_2\text{O}})^y$ and $(p_{\text{CO}_2})^z$) when Mg-rich phases are in thermodynamic equilibrium; i.e. they are more or less stable than MgO.  This procedure can then be applied to all phases to identify which is the most stable, provided that the free energy $\Delta G_f^0$ is known for each Mg-rich phase.

The free energy can be calculated using

\begin{align}
\Delta G^{0}_{f} = \sum\Delta G_{f}^{0,\text{products}} - \sum\Delta G_{f}^{0,\text{reactants}}
\end{align}

Where for this tutorial the free energy (G) is equal to the calculated DFT energy (U0).

```python
import matplotlib.pyplot as plt
from surfinpy import bulk_mu_vs_mu as bmvm
from surfinpy import utils as ut
from surfinpy import data
```

The first thing to do is input the data that we have generated from our DFT simulations. The input data needs to be contained within a `surfinpy.data` object. First we have created a `surfinpy.data.ReferenceDataSet` object for the bulk data (reference data), where `cation` is the number of cations, `anion` is the number of anions, `energy` is the DFT energy and `funits` is the number of formula units.

```python
bulk = data.ReferenceDataSet(cation = 1, anion = 1, energy = -92.0, funits = 10)
```

Next we create the bulk `surfinpy.data.DataSet` objects - one for each surface or "phase". `cation` is the number of cations, `x` is in this case the number of oxygen species (corresponding to the X axis of the phase diagram), `y` is the number of in this case water molecules (corresponding to the Y axis of our phase diagram), `energy` is the DFT energy, `label` is the label for the phase (appears on the phase diagram) and finally `nSpecies` is the number of adsorbin species.

```python
MgO = data.DataSet(cation = 10, x = 0, y = 0, energy = -92.0, label = "Periclase")
Art = data.DataSet(cation = 10, x = 5, y = 20, energy = -468.0, label = "Artinite")
Bru = data.DataSet(cation = 10, x = 0, y = 10, energy = -228.0, label = "Brucite")
Nes = data.DataSet(cation = 10, x = 10, y = 30, energy = -706.0, label = "Nesquehonite")
Mag = data.DataSet(cation = 10, x = 10, y = 0, energy = -310.0, label = "Magnesite")
Lan = data.DataSet(cation = 10, x = 10, y = 50, energy = -972.0, label = "Lansfordite")
Hyd = data.DataSet(cation = 10, x = 8, y = 10, energy = -398.0, label = "Hydromagnesite")
```

Next we need to create a list of our data. Don't worry about the order, surfinpy will sort that out for you.

```python
data = [MgO, Art, Bru, Nes,  Mag, Lan, Hyd]
```

We now need to generate our X and Y axis, or more appropriately, our chemical potential values. These exist in a dictionary. 'Range' corresponds to the range of chemcial potential values to be considered and 'Label' is the axis label.  Additionally, the x and y energy need to be specified.

```
    deltaX = {'Range': Range of Chemical Potential,
              'Label': Species Label}

```

```python
deltaX = {'Range': [ -3, 2],  'Label': 'CO_2'}
deltaY = {'Range': [ -3, 2], 'Label': 'H_2O'}
x_energy=-20.53412969
y_energy=-12.83725889
```

And finally we can generate our plot using these 6 variables of data.

```python
system = bmvm.calculate(data, bulk, deltaX, deltaY, x_energy, y_energy)

ax = system.plot_phase(figsize=(6, 4))
plt.show()
```