# Elliptic curve cryptography

What is an Elliptic curve (EC)? An elliptic curve is a plane algebraic curve over a [finite field](https://en.wikipedia.org/wiki/Finite_field) which is defined by an equation of the form:

\begin{equation}
y^2 = x^3+ax+b \quad \textrm{where} \quad 4a^3+27b^2 ≠ 0
\label{eq:ecurve}
\tag{1}
\end{equation}

The $4a^3+27b^2 ≠ 0$ restrained is required to avoid singular points.

A finite field is a set where operations of multiplication, addition, subtraction and division are defined according to basic rules. Examples of finite fields are [integers mod p](https://en.wikipedia.org/wiki/Modular_arithmetic#Integers_modulo_n) when p is a prime number.

```python
# Import the necessary libraries
# to remove code in browser, press f12 and in console type: document.querySelectorAll("div.input").forEach(function(a){a.remove()})
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import seaborn as sns
import numpy as np
import pandas as pd

from typing import Callable, Tuple
from scipy import optimize

def ecurve_power2(a: float, b: float, x: float) -> float:
    # y²=x³+ax+b and 4a³+27b²≠0
    # secp256k1 curve is y² = x³+7
    return x**3 + x*a + b

def ecurve(domain: pd.array, ecurve_power2_func: Callable[[float], float]) -> pd.DataFrame:
    # y = sqrt(x³+ax+b)
    # Only return domain where y>0
    y2 = ecurve_power2_func(domain)
    x_ = domain[y2>0]
    y2 = y2[y2>0]
    y = np.sqrt(y2)
    dataset = pd.DataFrame({'x': x_, 'y': y, 'y_neg': (-1)*y})
    return dataset

def domain(x1: float, x2: float, step: float = 0.1) -> np.ndarray:
    return np.arange(x1, x2, step).astype(np.float64)

def straight_line(m: float, c: float, x: float) -> float:
    # y = xm + c
    return m*x + c

def calc_straight_line_params(point1: Tuple[float, float], point2: Tuple[float, float]) -> Tuple[float, float]:
    # Calculate the gradient(m) and y intercept(c) in: y = xm + c
    x1, y1 = point1
    x2, y2 = point2
    m = (y2 - y1)/(x2 - x1)
    c = -1*x2*m + y2
    return m, c

def plot_elliptic_curve(axs: plt.axes, domain: pd.array, ecurve_power2_partial: Callable[[float], float], title="") -> None:
    # data must have x and y coloms
    data = ecurve(domain, ecurve_power2_partial)
    # to display as a continues function, the grid needs to go past the cut of values for the ec, hence the -1's
    X, Y = np.mgrid[min(data.x)-1:max(data.x):100j, min(data.y_neg)-1:max(data.y):100j]
    axs.contour(X, Y, Y**2 - ecurve_power2_partial(X), levels=[0]) # pos graph
    axs.contour(X, Y*-1, Y**2 - ecurve_power2_partial(X), levels=[0]) # pos graph
    axs.set_title(title)
    axs.set_xlim(min(data.x)-1, max(data.x)+1)
    axs.set_ylim(min(data.y_neg)-1, max(data.y)+1)

def plot_straight_line(axs: plt.axes, domain: pd.array, straight_line_partial_func: Callable[[float], float], title="") -> None:
    axs.plot(domain, straight_line_partial_func(domain))
    if title != "":
        axs.set_title(title)

def roots(f: Callable[[float], float], g: Callable[[float], float], domain: pd.array) -> np.array:
    d = lambda x: f(x) - g(x)
    roots_index = np.argwhere(np.diff(np.sign(d(domain)))).flatten()
    return domain[roots_index].to_numpy()

def calc_intersection(domain: pd.array, ecurve_power2_partial_func: Callable[[float], float], straight_line_partial_func: Callable[[float], float]) -> Tuple[float, float]:
    data = ecurve(domain, ecurve_power2_partial_func)

    ecurve_pos_partial = lambda x: np.sqrt(ecurve_power2_partial_func(x))
    ecurve_neg_partial = lambda x: np.sqrt(ecurve_power2_partial_func(x))*-1

    roots_pos = roots(ecurve_pos_partial, straight_line_partial_func, data.x)
    roots_neg = roots(ecurve_neg_partial, straight_line_partial_func, data.x)
    intersections = pd.DataFrame({'x': roots_pos, 'y': ecurve_pos_partial(roots_pos)})
    intersections2 = pd.DataFrame({'x': roots_neg, 'y': ecurve_neg_partial(roots_neg)})
    return intersections.append(intersections2).reset_index()
```

Example of elliptic curves with different a and b values:

```python
# Setup domain and Elliptic Curve function
dom = domain(-5,5) # Domain
secp256k1_pow2 = lambda x: ecurve_power2(0, 7, x) # EllipticCurve function y^2 with secp256k1 parameters
```

```python
# calc_point_on_ec = ecurve(dom, secp256k1_pow2) #  EllipticCurve function sqrrt(y^2)
fig_example, (ax1_example, ax2_example, ax3_example, ax4_example) = plt.subplots(1,4, sharex='col', sharey='row', gridspec_kw={'hspace': 0.2, 'wspace': 0.1},figsize=(20,5))
plot_elliptic_curve(ax1_example, dom, lambda x: ecurve_power2(1, -1, x), 'y² = x³+x-1')
plot_elliptic_curve(ax2_example, dom, lambda x: ecurve_power2(1, 1, x), 'y² = x³+x+1')
plot_elliptic_curve(ax3_example, dom, lambda x: ecurve_power2(-3, 3, x), 'y² = x³-3x+3')
plot_elliptic_curve(ax4_example, dom, lambda x: ecurve_power2(-4, 0, x), 'y² = x³-4x')
```

The elliptic curve used by most cryptocurrencies is called the secp256k1 and takes the form
\begin{equation}
y^2 = x^3+x+7
\label{eq:secp256k1}
\tag{2}
\end{equation}

```python
fig_secp256k1, (ax_secp256k1) = plt.subplots(1,1, sharex='col', sharey='row', gridspec_kw={'hspace': 0.2, 'wspace': 0.1},figsize=(4,5))
plot_elliptic_curve(ax_secp256k1, dom, secp256k1_pow2, 'y² = x³+7')
```

## Finite field
The elliptic curve operation for point addition are different than normal addition. With normal addition you would expect that point1 (x1, y1) + point2 (x2, y2) would equal (x1+1x2, y1+y2). This is not so with elliptic curve where the add operation is defined differently: When you add two points on a elliptic curve together, you get a third point on the curve.

The process is can described as when you have 2 points on a elliptic curve, you draw a line bewteen the points, determine where it intersects the curve. This intersection point is then reflected across the x-axis (i.e multiply the y-coordinate by -1 (x, y*-1)).

A example of addition would be:

```python
fig_intersec, (ax1_intersec, ax2_intersec, ax3_intersec, ax4_intersec) = plt.subplots(1,4, sharex='col', sharey='row', gridspec_kw={'hspace': 0.2, 'wspace': 0.1},figsize=(20,5))
plot_elliptic_curve(ax1_intersec, dom, secp256k1_pow2, 'To add point P and Q')
plot_elliptic_curve(ax2_intersec, dom, secp256k1_pow2, 'Draw a line between the points')
plot_elliptic_curve(ax3_intersec, dom, secp256k1_pow2, 'Reflect that point across the x-axis')
plot_elliptic_curve(ax4_intersec, dom, secp256k1_pow2, ' P+Q=R')

# Arbitrary points on elliptic curve
points = ecurve(pd.array([-1, 4]), secp256k1_pow2)
point1 = (points.x[0], points.y[0])
point2 = (points.x[1], points.y_neg[1])

m, c = calc_straight_line_params(point1=point1, point2=point2) # Calculate straight line paramaters giving the points

straight_line_partial = lambda x: straight_line(m, c, x) # Straight line function with paramaters

# Calculate intersections between the Straight line function and the EllipticCurve function
intersections = calc_intersection(domain=dom, ecurve_power2_partial_func=secp256k1_pow2, straight_line_partial_func=straight_line_partial)

# First plot
ax1_intersec.plot(intersections.x[0], intersections.y[0], "o", label="P", c='b')
ax1_intersec.plot(intersections.x[2], intersections.y[2], "o", label="Q", c='k')
ax1_intersec.legend()
ax1_intersec.set_xlabel("Fig 1")
ax1_intersec.axhline(linewidth=1, color='k')

# Second plot
plot_straight_line(axs=ax2_intersec, domain=dom, straight_line_partial_func=straight_line_partial, title="")
ax2_intersec.plot(intersections.x[0], intersections.y[0], "o", label="P", c='b')
ax2_intersec.plot(intersections.x[2], intersections.y[2], "o", label="Q", c='k')
ax2_intersec.plot(intersections.x[1], intersections.y[1], "o", label="Intersection", c='g')
ax2_intersec.legend()
ax2_intersec.set_xlabel("Fig 2")
ax2_intersec.axhline(linewidth=1, color='k')

# Third plot
plot_straight_line(axs=ax3_intersec, domain=dom, straight_line_partial_func=straight_line_partial, title="")
ax3_intersec.plot(intersections.x[0], intersections.y[0], "o", label="P", c='b')
ax3_intersec.plot(intersections.x[2], intersections.y[2], "o", label="Q", c='k')
ax3_intersec.plot(intersections.x[1], intersections.y[1], "o", label="Intersection", c='g')
ax3_intersec.plot(intersections.x[1], intersections.y[1]*-1, "o", label="R", c='r')
ax3_intersec.legend()
ax3_intersec.set_xlabel("Fig 3")
ax3_intersec.axhline(linewidth=1, color='k')

ax3_intersec.vlines(intersections.x[1], ymin=intersections.y[1], ymax=intersections.y[1]*-1, colors='r', linestyles='dashed')

# Fourth plot
ax4_intersec.plot(intersections.x[0], intersections.y[0], "o", label="P", c='b')
ax4_intersec.plot(intersections.x[2], intersections.y[2], "o", label="Q", c='k')
ax4_intersec.plot(intersections.x[1], intersections.y[1]*-1, "o", label="R", c='r')
ax4_intersec.legend()
ax4_intersec.set_xlabel("Fig 4")
ax4_intersec.axhline(linewidth=1, color='k')

print("")
```

Steps to find $P+Q$
- Fig1: If you have point $P$ (-1, 2.5) and $Q$ (4.0, -8.4) on the elliptic curve
- Fig2: Draw a line between the points, find the intersect point at (1.7, -3.5)
- Fig3: Reflect the intersect point across the x-axis to found the new point, $R$ (1.7, 3.5)
- Fig4: $P+Q=R$

With elliptic curve cryptography, you do not just add two arbitrary points together, but rather you start with a base point on the curve and add that point to it self. If we start with a base point $P$ than we have to find a line that goes through $P$ and $P$. Unfortunately there are infinite such lines. With elliptic curve cryptography the tangent line is used in this special case. The same process is followed now to calculate $P+P=2P$:

```python
fig_ecurve, (ax1_ecurve, ax2_ecurve, ax3_ecurve, ax4_ecurve) = plt.subplots(1,4, sharex='col', sharey='row', gridspec_kw={'hspace': 0.2, 'wspace': 0.1},figsize=(20,5))
plot_elliptic_curve(ax1_ecurve, dom, secp256k1_pow2, 'Initial point P')
plot_elliptic_curve(ax2_ecurve, dom, secp256k1_pow2, 'Find tangent line that goes through P and P')
plot_elliptic_curve(ax3_ecurve, dom, secp256k1_pow2, 'Reflect the intersection point across the x-axis')
plot_elliptic_curve(ax4_ecurve, dom, secp256k1_pow2, 'P+P=2P')

# Choose a arbitrary point P on the elliptic curve
p_points = ecurve(pd.array([-1.3, -1.31]), secp256k1_pow2)
p_point1 = (p_points.x[0], p_points.y[0])
p_point2 = (p_points.x[1], p_points.y[1])

m, c = calc_straight_line_params(point1=p_point1, point2=p_point2) # Calculate straight line paramaters giving the points

straight_line_partial = lambda x: straight_line(m, c, x) # Straight line function with paramaters

# Calculate intersections between the Straight line function and the EllipticCurve function
intersections_ecurve = calc_intersection(domain=dom, ecurve_power2_partial_func=secp256k1_pow2, straight_line_partial_func=straight_line_partial)

# First plot
ax1_ecurve.plot(p_points.x[0], p_points.y[0], "o", label="P", c='b')
ax1_ecurve.legend()
ax1_ecurve.set_xlabel("Fig 1")
ax1_ecurve.axhline(linewidth=1, color='k')

# Second plot
plot_straight_line(axs=ax2_ecurve, domain=dom, straight_line_partial_func=straight_line_partial, title="")
ax2_ecurve.plot(p_points.x[0], p_points.y[0], "o", label="P", c='b')
ax2_ecurve.plot(intersections_ecurve.x[0], intersections_ecurve.y[0], "o", label="Intersection", c='g')
ax2_ecurve.legend()
ax2_ecurve.set_xlabel("Fig 2")
ax2_ecurve.axhline(linewidth=1, color='k')

# Third plot
plot_straight_line(axs=ax3_ecurve, domain=dom, straight_line_partial_func=straight_line_partial, title="")
# ax3_ecurve.plot(intersections_ecurve.x[0], intersections_ecurve.y[0], "o", label="P", c='b')
ax3_ecurve.plot(p_points.x[0], p_points.y[0], "o", label="P", c='b')
ax3_ecurve.plot(intersections_ecurve.x[0], intersections_ecurve.y[0], "o", label="Intersection", c='g')
ax3_ecurve.plot(intersections_ecurve.x[0], intersections_ecurve.y[0]*-1, "o", label="P+P=2P", c='r')
ax3_ecurve.legend()
ax3_ecurve.set_xlabel("Fig 3")
ax3_ecurve.axhline(linewidth=1, color='k')

ax3_ecurve.vlines(intersections_ecurve.x[0], ymin=intersections_ecurve.y[0], ymax=intersections_ecurve.y[0]*-1, colors='r', linestyles='dashed')

# Fourth plot
ax4_ecurve.plot(p_points.x[0], p_points.y[0], "o", label="P", c='b')
# ax4_ecurve.plot(intersections_ecurve.x[0], intersections_ecurve.y[0], "o", label="P", c='b')
ax4_ecurve.plot(intersections_ecurve.x[0], intersections_ecurve.y[0]*-1, "o", label="P+P=2P", c='r')

ax4_ecurve.legend()
ax4_ecurve.set_xlabel("Fig 4")
ax4_ecurve.axhline(linewidth=1, color='k')

print("")
```

Now that we have $2P$, we can add $P$ again to get $3P$, see the example below which follows the same process as before. Draw a line between $P$ and $2P$, find the intersect and reflect this intersect value across the x-axis to find $3P$.

```python
fig_ecurve3P, (ax1_ecurve3P, ax2_ecurve3P, ax3_ecurve3P, ax4_ecurve3P) = plt.subplots(1,4, sharex='col', sharey='row', gridspec_kw={'hspace': 0.2, 'wspace': 0.1},figsize=(20,5))
plot_elliptic_curve(ax1_ecurve3P, dom, secp256k1_pow2, 'P + 2P')
plot_elliptic_curve(ax2_ecurve3P, dom, secp256k1_pow2, 'Draw a line that goes through P and 2P')
plot_elliptic_curve(ax3_ecurve3P, dom, secp256k1_pow2, 'Reflect the intersection point across the x-axis')
plot_elliptic_curve(ax4_ecurve3P, dom, secp256k1_pow2, '2P+P=3P')

# Use P and 2P from previous run
p_point1 = (p_points.x[0], p_points.y[0])
p_point2 = (intersections_ecurve.x[0], intersections_ecurve.y[0]*-1)

m, c = calc_straight_line_params(point1=p_point1, point2=p_point2) # Calculate straight line paramaters giving the points

straight_line_partial = lambda x: straight_line(m, c, x) # Straight line function with paramaters

# Calculate intersections between the Straight line function and the EllipticCurve function
intersections = calc_intersection(domain=dom, ecurve_power2_partial_func=secp256k1_pow2, straight_line_partial_func=straight_line_partial)

# First plot
ax1_ecurve3P.plot(p_points.x[0], p_points.y[0], "o", label="P", c='b')
ax1_ecurve3P.plot(intersections_ecurve.x[0], intersections_ecurve.y[0]*-1, "o", label="2P", c='r')
ax1_ecurve3P.legend()
ax1_ecurve3P.set_xlabel("Fig 1")
ax1_ecurve3P.axhline(linewidth=1, color='k')

# Second plot
plot_straight_line(axs=ax2_ecurve3P, domain=dom, straight_line_partial_func=straight_line_partial, title="")
ax2_ecurve3P.plot(p_points.x[0], p_points.y[0], "o", label="P", c='b')
ax2_ecurve3P.plot(intersections_ecurve.x[0], intersections_ecurve.y[0]*-1, "o", label="2P", c='r')
ax2_ecurve3P.plot(intersections.x[1], intersections.y[1], "o", label="Intersection", c='g')

ax2_ecurve3P.legend()
ax2_ecurve3P.set_xlabel("Fig 2")
ax2_ecurve3P.axhline(linewidth=1, color='k')

# Third plot
plot_straight_line(axs=ax3_ecurve3P, domain=dom, straight_line_partial_func=straight_line_partial, title="")
ax3_ecurve3P.plot(p_points.x[0], p_points.y[0], "o", label="P", c='b')
ax3_ecurve3P.plot(intersections_ecurve.x[0], intersections_ecurve.y[0]*-1, "o", label="2P", c='r')
ax3_ecurve3P.plot(intersections.x[1], intersections.y[1], "o", label="Intersection", c='g')
ax3_ecurve3P.plot(intersections.x[1], intersections.y[1]*-1, "o", label="2P+P=3P", c='m')
ax3_ecurve3P.legend()
ax3_ecurve3P.set_xlabel("Fig 3")
ax3_ecurve3P.axhline(linewidth=1, color='k')

ax3_ecurve3P.vlines(intersections.x[1], ymin=intersections.y[1], ymax=intersections.y[1]*-1, colors='r', linestyles='dashed')

# Fourth plot
ax4_ecurve3P.plot(p_points.x[0], p_points.y[0], "o", label="P", c='b')
ax4_ecurve3P.plot(intersections_ecurve.x[0], intersections_ecurve.y[0]*-1, "o", label="2P", c='r')
ax4_ecurve3P.plot(intersections.x[1], intersections.y[1]*-1, "o", label="3P", c='m')
ax4_ecurve3P.legend()
ax4_ecurve3P.set_xlabel("Fig 4")
ax4_ecurve3P.axhline(linewidth=1, color='k')

print("")
```

The same process now can be used to calculate $4P, 5P ... nP$.

The base point used in secp256k1 curve has the following ($x, y$) coordinates:

$x:$ 55066263022277343669578718895168534326250603453777594175500187360389116729240

$y:$ 32670510020758816978083085130507043184471273380659243275938904335757337482424

In the examples above a base point was choosen to view all the calculated points in small graph.

## Addition properties

In this finite field, addition also has the property of

\begin{equation}
nP+rP = (n+r)P
\label{eq:addition}
\tag{3}
\end{equation}

A example are $4P+6P = (4+6)P = 10P$. The easiest method to calculate for example $10P$ would require 4 calculations:

$$
\begin{align}
P+P &= 2P \\
2P+2P &= 4P \\
4P+4P &= 8P \\
8P+2P &= 10P \\
\end{align}
$$

## Diffie–Hellman key exchange

In this section we will exploring the Diffie–Hellman key exchange (DH). This will serve as a basis to understand Elliptic curve cryptography. DH is one of the earliest practical examples of public key exchange implemented within the field of cryptography.

Encryption is the process of converting information or data into a code to allow only the intended precipitant to decode and read the message. Often times this encryption/decryption is done with a shared secret key. In the following example we will show how two parties can get a shared key (in the past, they physically shared the key on a piece of paper).

Let's start with the example of Nick and Connie, they want to sent messages to each other without being eavesdropped on. They will share a arbitary number: $g$, this is sent over the internet and could have intercepted, but that does not matter. Nick and Connie also create their own secret key (a big number) and do not share it with anybody:
$$
\begin{align}
Nick&: n  \\
Connie&: c
\end{align}
$$

Then they will raise the arbitary number $g$ to the power of there secret key:

$$
\begin{align}
Nick: g^n &= H_n \\
Connie: g^c &= H_c
\tag{4}
\end{align}
$$

Onces they have the $H$ term, they exchange that to each other. $g, H_n, H_c$ are publicly sent and anybody can view these values. Once they have that $H$ term, they raise it to their secret key.

$$
\begin{align}
Nick: H_c^n &= S \\
Connie: H_n^c &= S
\tag{5}
\end{align}
$$

By doing this, they end up with the same number: $S$, this is the shared key, neither of them had to send it to each other explicitly. Now, for example to encrypt a message using a Caesar cipher (a simple cipher) you can shift all the letters in your message by $S$ number of letters, and shift it back by $S$ number of letters to decrypt (decipher) it. You now have method to encrypt your communication with each other.

To prove equation 5, you can subsitute equation 4 into 5:

$$
Nick: H_c^n = (g^c)^n = g^{cn} = S
$$

$$
Connie: H_n^c = (g^n)^c = g^{nc} = S
$$

Unfortunately we are not done yet, remeber that publicly sent values are $g, H_n, H_c$ are publicly sent. To calculate for example Nick's private $n$ will be trivial since the equation is

$$
Nick: g^n = H_n
$$

Calculating $n$ is as easy as solving this log problem $2^n=16$

What about this discrete log problem: $2^n mod 17 = 16$. This becomes difficult because of the [modulus](https://en.wikipedia.org/wiki/Modular_arithmetic), you do not know how many times over 17 we have gone. Another example of modulus is a clock. If I told you that the start time is 12 o'clock and the end time is 1 o'clock and I ask you how many hours has passed you would not know because you do not know how many times the clock went round. It could be 1 hour, 13 hours or 25 hours and so on. It is because of this fact that you have to start guessing, the discrete log problem is the basis for the DH key exchange. The calculations to create the shared key is simple, but it very difficult to solve for the private key.

Now you can just use the modulus operator in equations 4 and 5

$$
\begin{align}
Nick: g^n \, mod(p) &= H_n \\
Connie: g^c \, mod(p) &= H_c
\tag{6}
\end{align}
$$

You will end up with a shared key again, but this time is very difficult, almost impossible to figure out what the private keys are if the private keys are very big.

$$
\begin{align}
Nick: H_c^n \, mod(p) &= S \\
Connie: H_n^c \, mod(p) &= S
\tag{7}
\end{align}
$$

A more practical example: Nick and Connie both decide publicly on a generator, $G=3$, and a prime modulus, $P=17$. Then Connie decides on a random private key, $c=15$ and Nick does the same $n=13$.

$$
\begin{align}
Nick: G^n \, mod(p) &= H_n \\
3^{13} mod 17 &= 12\\
Connie: G^c \, mod(p) &= H_c \\
3^{15} mod 17 &= 6
\tag{6}
\end{align}
$$

Nick send $H_n=12$ publicly to Connie, and Connie sends $H_c=6$ to Nick publicly. Now the heart of the trick, Nick takes Connies publicly sent value and raises it to the power of his private number, and vice versa to obtain the same shared secret of 10.

$$
\begin{align}
Nick: H_c^n \, mod(p) &= S \\
 6^{13} mod 17 &= 10 \\
Connie: H_n^c \, mod(p) &= S \\
12^{15}mod 17 &= 10
\tag{7}
\end{align}
$$

For the DH, the reason that we choose a prime number for the modulus, is that this guarantees the group is cyclic. It also has a other property, because it is a modulus of a prime ($p$), a generator exists ($g$). A generator is smaller than the prime ($<p$), it will produce all the numbers from $1$ to $p-1$ exactly once with $p^x mod \, g$ where $x = 1, 2 ... p-1$. A example with a prime of 7, then the generator is 5:
$$
\begin{align}
5^1 mod \, 7 &= 5 \\
5^2 mod \, 7 &= 4 \\
5^3 mod \, 7 &= 6 \\
5^4 mod \, 7 &= 2 \\
5^5 mod \, 7 &= 3 \\
5^6 mod \, 7 &= 1 \\
\end{align}
$$

## Elliptic curve discrete log problem

If you look back now at Elliptic curve's addition rule in equation 3, where $nP$ are represented by $nP=P+P+P+...$

We can use the same form of equation 4 in the DH and apply it to EC:

$$
\begin{align}
nG &= H_n
\tag{8}
\end{align}
$$

where $G$ is the starting point and $H_n$ is the end point. $n$ is amount of times that $g$ is added to each self. Even if you know what $G$ and $H_n$ is, it very difficult to figure out what $n$ is.

Knwowing this, we can just use the DH procedure with these elliptic curve equations and it end's up working the same.

$$
\begin{align}
Nick: nG &= H_n \\
Connie: cG &= H_c
\tag{9}
\end{align}
$$

where $G, H_n, H_c$ are publicly sent. The shared public key can also be calculated in the same way as DH:

$$
\begin{align}
Nick: nH_c &= S \\
Connie: cH_n &= S
\tag{10}
\end{align}
$$

With the DF, the modulus allows to take the possible answers to the exponent problem and reduce the possible set of numbers. Using this $2^n mod 17 = 16$ example again, because of the mod 17, you limit the possible answers to 16. Now in EC, you also can take the modulus of the curve from being a function with infinite values to a finite set of values.

\begin{align}
y^2 &= x^3+ax+b\\
y^2 mod p &= (x^3+ax+b)\, mod p
\label{eq:ecurve}
\tag{11}
\end{align}

where $p$ is a prime number, it is a prime number to ensure that addition and multiplication operations can always be undone.

In secp256k1, $p$ is the largest prime that is smaller than $2^{256}$, this would be $2^{256}–2^{32}–977 = $

```python
p = 2**256 - 2**32 - 977
p
```

    115792089237316195423570985008687907853269984665640564039457584007908834671663

This means that x and y coordinates of the elliptic curve can be any number up to this prime.

```python
# http://en.wikibooks.org/wiki/Algorithm_Implementation/Mathematics/Extended_Euclidean_algorithm
def egcd(a, b):
    x,y, u,v = 0,1, 1,0
    while a != 0:
        q, r = b//a, b%a
        m, n = x-u*q, y-v*q
        b,a, x,y, u,v = a,r, u,v, m,n
    return b, x, y

# calculate modular inverse
def modinv(a, m):
    g, x, y = egcd(a, m)
    if g != 1:
        return None  # modular inverse does not exist
    else:
        return x % m

# ecurve(dom, secp256k1_pow2).y % 7777
# secp256k1_pow2(dom) % 5

```

```python
# https://andrea.corbellini.name/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/
def extended_euclidean_algorithm(a, b):
    """
    Returns a three-tuple (gcd, x, y) such that
    a * x + b * y == gcd, where gcd is the greatest
    common divisor of a and b.

    This function implements the extended Euclidean
    algorithm and runs in O(log b) in the worst case.
    """
    s, old_s = 0, 1
    t, old_t = 1, 0
    r, old_r = b, a

    while r != 0:
        quotient = old_r // r
        old_r, r = r, old_r - quotient * r
        old_s, s = s, old_s - quotient * s
        old_t, t = t, old_t - quotient * t

    return old_r, old_s, old_t

def inverse_of(n, p):
    """
    Returns the multiplicative inverse of
    n modulo p.

    This function returns an integer m such that
    (n * m) % p == 1.
    """
    gcd, x, y = extended_euclidean_algorithm(n, p)
    assert (n * x + p * y) % p == gcd

    if gcd != 1:
        # Either n is 0, or p is not a prime number.
        raise ValueError(
            '{} has no multiplicative inverse '
            'modulo {}'.format(n, p))
    else:
        return x % p
```

```python
# https://andrea.corbellini.name/2015/05/23/elliptic-curve-cryptography-finite-fields-and-discrete-logarithms/
# https://www.youtube.com/watch?v=NnyZZw8d1wI
p = 19
ecurve_pow2_mod = lambda x: ecurve_power2(-7, 10, x) % p

dom = domain(0,p, 1) # Domain
dom = pd.array(dom)
dom = dom[dom>0]

y2_mod = pd.DataFrame({'x': dom, 'y_1': ecurve_pow2_mod(dom), 'y_2': ecurve_pow2_mod(dom)})

# plt.plot(y2_mod.x, y, 'ro')
# plt.show()
y2_mod
```

      x
      y_1
      y_2

      0
      1.0
      4.0
      4.0

      1
      2.0
      4.0
      4.0

      2
      3.0
      16.0
      16.0

      3
      4.0
      8.0
      8.0

      4
      5.0
      5.0
      5.0

      5
      6.0
      13.0
      13.0

      6
      7.0
      0.0
      0.0

      7
      8.0
      10.0
      10.0

      8
      9.0
      11.0
      11.0

      9
      10.0
      9.0
      9.0

      10
      11.0
      10.0
      10.0

      11
      12.0
      1.0
      1.0

      12
      13.0
      7.0
      7.0

      13
      14.0
      15.0
      15.0

      14
      15.0
      12.0
      12.0

      15
      16.0
      4.0
      4.0

      16
      17.0
      16.0
      16.0

      17
      18.0
      16.0
      16.0

```python
ecurve_pow2_mod(5)
```

    5

```python
inverse_of(16, 19)
```

    6

```python
inverse_of(5, 19)
```

    4

```python
x = 6
g = [5,2,3]
gp = pd.array(g)
df = pd.DataFrame({'x': gp, 'diff': gp-4})
df = df.sort_values(by ='diff' )
f = lambda x: x + 1
f(a for a in df.x)
```

```python
# https://medium.com/asecuritysite-when-bob-met-alice/nothing-up-my-sleeve-creating-a-more-trust-world-with-the-elliptic-curve-pedersen-commitment-7b363d136579
2 ^= 1
```

```python
a = 1
b = 2
a ^= b
a
```

    3

```python
b
```

    2

```python

```


# Coordinates, Projections, and Grids

## Synopisis

- Review of coordinate systems and projections of the sphere onto the 2d placne
- Discussion about lengths and areas in finite volume grids used by ESMs

```python
%run -i figure-scripts/init.py
```

## Coordinate systems

A coordinate system allows us to (uniquely) specify points in space. You should be familiar with the Cartesian 2d coordinate system where, by convention, (x,y) are the normal distances, measured in the same units, from two perpendicular lines through the origin.

We live in a 3d world (referring to spatial dimensions) and so require 3 numerical values to label a point in space. In Cartesian coordinates they might be (x,y,z) referenced to the center of the Earth, with z being height above the equatorial plane (positive in the direction of the North pole), y being the distance from a plane through the poles and a reference meridian, and x the distance from the plane perpendicular to both other planes. The equations of motion used by many models are often derived starting in this Cartesian coordinate system. However, these Cartesian coordinates are inconvenient to use in practice because we live on the surface of a sphere and "up", as defined by gravity, is sometimes increasing z (at the North Pole), and sometimes changing x or y (at the Equator).

### Spherical coordinates

In ESMs we typically use spherical coordinates, $\lambda$, $\phi$ and $r$, where $\lambda$ is "longitude", a rotation angle eastward around the poles starting at a reference meridian; $\phi$ is "latitude", an elevation angle from the Equatorial plane (positive in Northern hemisphere), and $r$ is the radial distance from the center of the Earth. $\lambda,\phi,r$ are related to Cartesian $x,y,z$ by some simple relations:

$$
\begin{pmatrix}
x \\ y \\ z
\end{pmatrix} =
\begin{pmatrix}
r \cos \phi \cos \lambda \\ r \cos \phi \sin \lambda \\ r \sin \phi
\end{pmatrix}
$$

Note that $r, x, y, z$ are all in the same units (eg. kilometets or meters) and $\lambda,\phi$ are angles usually given in degrees or radians.

__Coordinate singularities:__ At the North and South poles of the coordinate system, $\phi = \pm \pi/2 = \pm 90^\circ$, all values of longitude refer to the same point. There is no "east" when you are positioned at the pole. This has many consequences, but one of the more fundamental is that spherical coordinates are not a good coordinate to use to design a discretization of the spherical domain.

__Periodic coordinates:__ While a tuple of longitude, latitude and radius unambiguously define a point in space, given a point in space you there are multiple valid longitudes that refer to the same point. Longitude is cyclic ($\pm360^\circ$ is equivalent to $0^\circ$). This can cause problems in practice, particularly for plotting spherical data for which effort is sometimes needed to handle the periodicity.

### Geographic coordinates

We live on the surface of the Earth and to precisely refer to points near the Earth's surface requires a properly defined geographic coordinate system. A common choice of coordinates is latitude, longitude and altitude, where altitude is height above a particular surface. Unfortunately the Earth is not spherical and that reference surface is better approximated as an ellipsoidal.

In order to be unambiguous about the definition of coordinates, map-makers choose a reference ellipse with a agreed upon scale and orientation. They then choose the most appropriate mapping of the spherical coordinate system onto that ellipsoid, called a _geodetic datum_. A widely used global datum includes the [World Geodetic System](https://en.wikipedia.org/wiki/World_Geodetic_System) (WGS 84), the default datum used for the Global Positioning System. When you are given a latitude-longitude pair of values, strictly speaking without the geodetic datum, the is some ambiguity about the actual physical point being referred to. For ESMs, the datum is rarely provided and this is because ESMs almost universally approximate the Earth as a sphere and use spherical coordinates for referencing locations. This means some approximation is required when comparing real-world positions and model positions.

The latitude and longitude using these horizontal datums are the spherical coordinates of the point on an ellipse. It you draw a straight line from the point on the ellipsoidal to the center, if passes through all spheres co-centered with the same latitude and longitude.

Different datum have different reference points and scales, and so longitude and latitude can differ between geodetic datum.

## Projections

To view data covering the surface of a sphere, or the Earth, we have to project that 3d surface into 2d. Imagine peeling the rind off an orange in one piece and then trying to flatten it onto a table top; the curvature in the peel requires you to distort the rind or make cuts, in order to flatten it fully. This is the function of the map projections and distortion is inevitable. Some projection preserve properties such as relative angles between lines, or relative area, but there is no projection of the surface of the sphere that can avoid distortion of some form.

A projection maps the longitude and latitude of spherical coordinates into a new coordinate system. Very confusingly, sometimes the projection coordinates will be called longitude and latitude too! The projection coordinates are meaningless unless you know what the projection is so you often find a reference to the projection in the meta-data of coordinates; it means the longitude and latitude are not spherical coordinate but projection coordinates.

```python
%run -i figure-scripts/some-projections.py
```

Figure 1: The colored circles in these plots are circles in the tangent plane to the sphere and projected onto the surface. The various projections can distort the circles. The circles are separated zonally or meridionally by 60$^\circ$. In 1a, a perspective image of the sphere, the circles appear non-circular because of the viewing angle. The blue circle appears circular because we are viewing it from directly overhead. The projection in 1b is the easy to use Plate-Carrée projection, a "lat-lon" plot, in which circles are stretched zonally with distance from the equator. 1d shows the Mercator projection in which circles remain circles but are expanded in size away from th equator. 1c shows the Robinson projection which compromises between the two. The purple dashed lines is a straight line in latitude-longitude coordinates, and the yellow dashed line is a straight line in the Mercator coordinates. The cyan dashed line is a great arc, and is straight in the perspective view because we are viewing it from directly overhead.

The two most useful projections are the equirectangular and Mercator projections.

### Equirectangular projection

This is the simplest projection, sometimes thought of a non-projection which is incorrect. In general it takes the form
$$
\begin{align}
x & = R \left( \lambda - \lambda_0 \right) \cos \phi_0 \\
y & = R \left( \phi - \phi_0 \right)
\end{align}
$$

The origin of the plot, $(x,y)=(0,0)$ corresponds to $(\lambda,\phi)=(\lambda_0,\phi_0)$. The $\cos \phi_0$ term is a constant and the most common choice of $\phi_0=0$ gives the plate carrée projection, which means "flat square" in French. In this case, the projection is simply
$$
\begin{align}
x & = R \left( \lambda - \lambda_0 \right) \\
y & = R \phi
\end{align}
$$

Distances in the y-direction are proportional to the meridional direction on the sphere, but the x-direction are stretched zonally, more so further from the equator. This is apparent in the orange and green circles in figure 1b, where the heights or the loops are the same as circles on the equator but the width is markedly increased.

In the cartopy package, this projection is called "Plate-Carrée" which is French for flat square. Other names for this projection are equidistant cylindrical projection and geographic projection. See https://en.wikipedia.org/wiki/Equirectangular_projection.

### Mercator projection

The Mercator projection has the same stretching in the x-direction as the equirectangular projection but, in order to preserve shape, it also stretches the y direction so that infinitesimal elements are stretched isotropically (the y-stretching is equal to the x-stretching).

$$
\begin{align}
x & = R \left( \lambda - \lambda_0 \right) \\
y & = R \tanh^{-1} \left( \sin \phi \right)
\end{align}
$$

At the polar singularities, the x-stretching is infinite so y becomes infinite and the Mercator projection can never show the poles. See https://en.wikipedia.org/wiki/Mercator_projection.

### Lines

A length of a line between two points is a function of the path taken. On the surface of sphere, the shortest path between two given points is a great arc. A great arc does not appear straight in many projections. Unfortunately, many grid calculations use a great arc for the length of a line between nodes on a model grid, which can be inconsistent with the constraints or assumptions about the grid.

The dashed curves in figure 1 are "straight" lines between two points in various projections. The cyan dashed curve is a great arc. The purple dashed curve is a straight line in the Plate-Carree projection (latitude-longitude space) and the yellow dashed curve is a straight line in the Mercator projection. All are curved in most other projections. To describe a straight line in some projection then _the projection must be known_, irrespective of the coordinate system defining the end points. That is, we can define the end points of the line in latitude-longitude coordinates but say a line is straight in the Mercator projection, and by so doing unambiguously define that line.

In the Mercator projection, the length of a line is $\frac{R}{\cos \alpha} \Delta \phi$ where $\tan \alpha = \frac{\Delta y}{\Delta x}$

## ESM grids

Many ESMs use quadrilateral grids to discretize the surface of the sphere. The following discussion also applies to fully unstructured grids built from polygons but here we use quadrilateral grids for simplicity. There are also grids that have cuts and joins but here we'll stick to space-filling grids that are logically rectangular, meaning they can be stored in rectangular arrays in computer memory and referenced with a pair of indices ($i,j$ by convention).

A quadrilateral grid is a rectangular mesh of adjacent quadrilateral cells that share edges and vertexes. Although the mesh and the cell are logically rectangular they might be physically curvilinear. From the grid we require positions of nodes, distances along edges, and areas of cells.

If we choose a coordinate system with which to record the locations of mesh nodes, say spherical latitude-longitude with appropriate definitions, then we can unambiguously define those node locations. We could describe the exact same grid using a different coordinate system, say 3D Cartesian coordinates. The physical positions of the nodes of the grids are part of what define the grid, but the choice of coordinates with which we describe those positions does not change the grid.

The edges of each cell are a curve between two adjacent nodes but the particular path of the curve has to be defined. Different paths will have different lengths. Similarly, the particular paths of the cell edges will determine the cell area. Thus the path of the cell edges is a fundamental component of a model grid needed for calculating the lengths and areas on a grid.

### Simple spherical coordinate grid

Before we discuss the best choice for defining a curve between points, let's briefly define a simple spherical-coordinate grid.
The mesh is formed of lines of constant longitude and lines of constant latitude.

Let $i \in 0,1,\ldots, n_i$ and $j \in 0,1,\ldots, n_j$, then node $i,j$ is at longitude $\lambda_i$ and latitude $\phi_j$ where $\lambda_i=\lambda_0 + i \Delta \lambda$, $\phi_j=\phi_0 + j \Delta \phi$.

Here, $\Delta \lambda$ and $\Delta \phi$ are grid spacings. In practice, these can be smooth functions of $i$ and $j$ respectively but here we treat them as constant.

An example simple spherical grid is shown below. The red dots are the nodes of the mesh with positions $\lambda_i,\phi_j$. The dashed lines are the cell edges that for a regular net. Notice that in the Plate-Carrée projection the grid is regular because the grid-spacing in constant in longitude-latitude coordinates.

The lengths and areas of the grid are measured on the surface of sphere. We defined the edges to be either lines of constant longitude or latitude. Using spherical geometry, the length of a meridionally-oriented (constant longitude) cell edge is $R \Delta \phi$. For a zonally-oriented edge at constant latitude $\phi_j$, the length is $R \Delta \lambda \cos \phi_j$. The area of a cell labelled $i+\frac{1}{2},j+\frac{1}{2}$ bounded by four edges is $R^2 \Delta \lambda \left( \sin \phi_{j+1} - \sin \phi_j \right)$.

The metric factors for this grid are the same as for a Plate-Carrée projection because we are defining the paths of the cell edges to be straight in the Plate-Carrée projection. The use of the Plate-Carrée coordinates for position, namely longitude and latitude, is a happy coincidence which means everything, positions and metrics, are defined by one projection.

```python
%run -i figure-scripts/simple-spherical-grid.py
```

Figure 2: A simple spherical grid view in different projections. The red dots are the nodes of the mesh. The dashed lines are the grid of cell edges. Here the grid spacing is constant in latitude-longitude coordinates which is why the grid looks regular in the Plate-Carrée projection.

In the above example of a simple spherical grid we used spherical geometry combined with the specification of the paths of cell edges to calculate everything. If we were given just the positions of nodes a spherical grid, we could re-calculate the grid lengths and areas given the knowledge that the cell edges are straight in the Plate-Carrée projection.

### Not to use great arcs

If is quite common for the projection for the cell edges to be omitted from a grid file. Sometimes, you will encounter some software that was written to handle arbitrary grids that does not know about the projections. In this instance, developers often make a choice for how to calculate a length between two points. There are two immediately simple choices we can make: i) the shortest curve, or ii) choose a projection in which to draw a straight line. There are other ways to choose and define the curve forming edges but they are not as useful for out purposes.

As discussed above, the shortest line is a great arc but it turns out that choosing great arcs does not result in orthogonal meshes without a particular distributions of nodes. For example, if you consider the example spherical grid defined above, using a great arc to approximate the distance along a latitude circle will underestimate the distance. If you were to then calculate the actual paths implied by the great arcs, the grid is not orthogonal at the nodes.

If no projection is given, then straight lines in the Plate-Carrée projection make a reasonable approximation that at least works for Plate-Carrée and Mercator. Many grids a composed of patches of grids that used different projections but have to match at the joins and it seems the Plate-Carrée is often the common denominator.

## Contents of grid files

As mentioned earlier, because most ESM treat the Earth as spherical, they rarely bother to choose a datum for the geographic coordinate system. Further, the longitudes and latitudes _may not be spherical coordinates_ but instead may be the result of a projection.

The habit of not specifying the projection for cell edges is somewhat alleviated by the general pattern of providing the numerical values for lengths and areas as part of the grid specification. For example, see https://www.researchgate.net/publication/228641121_Gridspec_A_standard_for_the_description_of_grids_used_in_Earth_System_models. For many calculations, including integrals and gradients of scalars, having the numerical values of lengths and areas is sufficient. For vector operations, the angle of grid-lines is needed.

The [CF convention](http://cfconventions.org/cf-conventions/cf-conventions.html) requires the grid to be provided in all files. That convention supports specification of mappings (projections) but does not distinguish the projection for paths from the projections for coordinates. by default, longitude and latitude are the true geographic coordinates. And the required grid information is limited to the node positions. Further the nodes correspond to the data locations which is a finite-difference perspective of a mesh, and quite different from the finite volume perspective that most ESMs assume.

## Summary


# Modelling Change - Project

## Topic: Gradient Descent

> `numerical` `optimisation`

Gradient descent is a general method for finding minima of functions taught is widely used in many fields.
Write out the equations for a simple gradient descent method and code an implementation in Python.
Locate the minima of an example function who's min you can find using the analytically (that is, as we did
in lectures and tutorials). Investigate how the convergence is affected by:

1. step size or other parameters in the algorith
2. the initial starting point

### Author:

**Name:** Jonathon Belotti

**Student ID:** 13791607

## Overview

This report details the mathematics and computer code implementation of the _Gradient Descent_ optimisation method, and investigates the behaviour of the optimisation process when subject to variance in algorithm paramaters ('step size', iteration number) and variance in the initial starting point.

### Report Structure

1. [**Introduction to the Pure-Python3 Implementation.**](#1.-Introduction-to-the-Pure-Python3-Implementation)
2. [**Introduction to the Gradient Descent method.**](#2.-Introduction-to-the-Gradient-Descent-Method)
3. [**Convergence.**](#3.-Convergence)
5. [**References**](#References)
6. [**Appendix**](#Appendix)

### 1. Introduction to the Pure-Python3 Implementation

To assist in the exploration and communication of the Gradient Descent optimisation technique, a 'from scratch' [pure-Python](https://stackoverflow.com/a/52461357/4885590) implementation of the gradient descent algorithm has been written and is used throughout. It is a 'standalone module', and imports no third-party code. The implementation targets usefulness in learning, not performance, but is fast enough for practical example.

To reduce implementation complexity and length, not all differentiable functions are supported by the implementation. Supported functions include:

- [Polynomial functions](https://en.wikipedia.org/wiki/Polynomial)

The implementation is wholly contained in one Python3 module, `gradient_descent`, and it is available at: https://github.com/thundergolfer/modelling_change

```python
from typing import Mapping

import gradient_descent  # Assumes gradient_descent.py is on PYTHONPATH. For module implementation see appendix.
```

**Variables** become components in expressions and are used in the differentiation functions. They don't do much else.

```python
x = Variable("x")
```

**Expressions** can be evaluated at a point in function space, eg $(x=1, y=2, z=4)$ and can be differentiated with respect to a reference variable.

`Expression` is a base class for `ConstantExpression`, `PolynomialExpression`, and `Multiply`.

```python
class Expression:
    def diff(self, ref_var: Optional[Variable] = None) -> Optional["Expression"]:
        raise NotImplementedError

    def evaluate(self, point: Point) -> float:
        raise NotImplementedError
```

**MultiVariableFunctions** are created by specifying their inputs, `variables`, and composing `Expression` objects by
addition (subtraction is handled by negative coefficients on expressions).

We can get the gradient of a function and evaluated it at a point in function space, just like `Expression` objects.

```python
class MultiVariableFunction:
    """
    MultiVariableFunction support the composition of expressions by addition into a
    function of multiple real-valued variables.

    Partial differentiation with respect to a single variable is supported, as is
    evaluation at a Point, and gradient finding.
    """

    def __init__(self, variables: Set[Variable], expressions: List[Expression]): ...

    def gradient(self) -> GradientVector: ...

    def diff(self, ref_var: Variable) -> "MultiVariableFunction": ...

    def evaluate(self, point: Point) -> float: ...
```

### 2. Introduction to the Gradient Descent Method

Gradient Descent is an iterative optimization process that can be used effectively to find local mimina of differentiable functions, particulary when those functions are convex. When the output of a differentiable function under some set of inputs can be framed as a _cost_, the minimization of this _cost function_ becomes an optimization problem to which the Gradient Descent process can be applied.

The "Deep Neural Networks" revolution that swept through the 2010s has its foundation in the simple single-layer neural networks first published in the 1980s, and those simple networks were optimized through gradient descent. Thus, a first lesson in understanding today's hottest technological field, Deep Neural Networks, involves going right back to the start and understanding the basic Gradient Descent optimization process.

#### 2.1 A Function's Gradient

In order to minimise a function's value, we need to ascertain which way we should nudge its inputs to decrease the output value, and we have to be sure than a series of decreases will eventually lead to a minimum (local or global). For differentiable functions, the first-derivative of a function can be the way.

For a function of a single variable, $f(x)$, the rate of change at some value $x=a$ is given by the first-derivative $f'(x)$. In the case of $f(x) = x^2 + x$, we know that:

$$f'(x) = 2x + 1$$

and thus at $f'(1) = 2(1) + 1 = 3$ the function is increasing in output value 'to the right' and decreasing 'to the left'. At $f'(-1) = 2(-1) + 1 = -1$ the function is decreasing in output value 'to the right' and increasing 'to the left;.

In either case, we know from the first-derivative which direction to nudge $x$, until we reach $f'(1/2) = 2*(1/2) + 1 = 0$ and we've reached the critical point.

But for a multi-variable function there are multiple ways in which to influence the output value and thus multiple dimensions along which a we could change inputs. How can we extend our understanding of the direction of function decrease beyond 'left and right' and into 3-dimensions and more? We use partial derivatives.

If $z = f(x, y)$, then we have a multi-variable function with partial derivatives:

$$f_x(x_0, y_0) = \lim_{h \to 0}\frac{f(x_0+h, y_0) - f(x_0, y_0)}{h}$$

$$f_y(x_0, y_0) = \lim_{h \to 0}\frac{f(x_0, y_0+h) - f(x_0, y_0)}{h}$$

with each capturing the rate-of-change with respect to a single variable in our multi-variable function.

In the $x,y$ plane, we can imagine being at some point $(x_0,y_0)$ and nudging away from that point in the plane by the vector $\mathbf{u} = \langle a, b \rangle $.

Now not restricted to moving 'left and right' in the x-axis or 'up and down' the y-axis, we have a **Directional Derivative** of $f$ at $(x_0,y_0)$.

$$D_uf(x_0, y_0) = \lim_{h \to 0}\frac{f(x_0 + ha, y_0+hb) - f(x_0, y_0)}{h}$$

More intuitively, we can consider that nudge as being of length $h$ at some angle $\theta$ (capturing direction). Thus our $a$ and $b$ are $\cos{\theta}$ and $\sin{\theta}$ respectively, and $\mathbf{u} = \langle a, b \rangle $ is a vector of length 1.

In fact, any differentiable function of $x$ and $y$ has a directional derivative in a direction of $\mathbf{u}$ and this relationship can be expressed as:

$$D_uf(x, y) = f_x(x, y)a + f_y(x, y)b$$

**To prove this.**

Define a function $g$ of the single variable $h$ as

$$
\begin{align}
g(h) & = f(x_0 + ha, y_0+hb) \\
\end{align}
$$

By definition of the derivative:

$$
\begin{align}
g'(0) & = \lim_{h \to 0}\frac{g(h) - g(0)}{h} \\
& = \lim_{h \to 0}\frac{f(x_0+ha, y_0+hb) - f(x_0, y_0)}{h}\\
& = D_uf(x_0, y_0)
\end{align}
$$

Writing $x = x_0 + ha$ and $y = y_0 + hb$ we get $g(h) = f(x, y)$ and

$$
\begin{align}
g'(h) & = \frac{\partial f}{\partial x}\frac{\partial x}{\partial h} + \frac{\partial f}{\partial y}\frac{\partial y}{\partial h} \\
& = f_x(x, y)a + f_y(x, y)b \\
& = D_uf(x_0, y_0)
\end{align}
$$

Substituting in $h=0$ then $x$ and $y$ become $x = x_0$, $y = y_0$, so:

$$
\begin{align}
g'(0) & = f_x(x, y)a + f_y(x, y)b \\
 & = f_x(x_0, y_0)a + f_y(x_0, y_0)b \\
& = D_uf(x_0, y_0)
\end{align}
$$

Thus:

$$D_uf(x, y) = f_x(x, y)a + f_y(x, y)b$$

This relationship between partial derivatives and a directional nudges in each input dimension generalises beyond 2 dimensions, and can be compactly represented by _vectorising_ the combination of the partial derivatives and an input.

$$
\begin{align}
D_uf(x,y) & = f_x(x, y)a + f_y(x, y)b \\
 & = \langle f_x(x,y), f_y(x, y) \rangle \cdot \langle a, b \rangle \\
 & = \langle f_x(x,y), f_y(x, y) \rangle \cdot \mathbf{u} \\
\end{align}
$$

**In the `gradient_descent` library, we can calculate the gradient vector from a `MultiVariableFunction`. For the function:**

$$f(x,y) = x^2 + y^2 - 2x - 6y + 14$$

```python
x = gradient_descent.Variable("x")
y = gradient_descent.Variable("y")
f = gradient_descent.MultiVariableFunction(
    variables={x, y},
    expressions=[
        gradient_descent.PolynomialExpression(variable=x, coefficient=1, exponent=2),
        gradient_descent.PolynomialExpression(variable=y, coefficient=1, exponent=2),
        gradient_descent.PolynomialExpression(variable=x, coefficient=-2, exponent=1),
        gradient_descent.PolynomialExpression(variable=y, coefficient=-6, exponent=1),
        gradient_descent.ConstantExpression(real=14.0),
    ],
)

f.gradient()
```

#### 2.2 Steepest Descent

Now able to determine the gradient vector of a function, capturing the rate of change along each dimension of a function, the question becomes in which 'direction' to go to 'descend' or decrease the function's value?.

$$\nabla_{u} f(x_0,y_0) = \mathbf{u} \cdot f(x_0, y_0)$$

Firstly, if the gradient is zero

$$\nabla_{u} f(x_0,y_0) = 0$$

then the directional gradient is zero in every direction. This would be the end of our descent. But for a non-zero gradient, then the gradient itself is the direction that maximises the dot product.

$$\max \nabla f(x_0, y_0) \cdot \mathbf{u} = \frac{\nabla f(x_0, y_0)}{\lvert \nabla f(x_0, y_0) \rvert}$$

This is actually great, because in order to descend fastest from some point $f(x_0, y_0)$ we don't need to calculate which direction is best, the direction of the gradient is the best direction.

#### 2.3 Gradient Descent - Iterating Towards the Bottom

Now with a method to calculate the direction of maximum descent from a point $\mathbf{a}$ in a function's input space, we are very close to creating the _Gradient Descent_ optimisation process.

Given a differentiable multi-variable function $f(\mathbf{x})$, with $\mathbf{x}$ being a vector of inputs $\langle x, y, z, ... \rangle$, then we know:

**At some point $\mathbf{a} \in \mathbf{x}$, $f(\mathbf{x})$ decreases _fastest_ in the direction of the negative gradient:** $-\nabla \mathbf{f(a)}$

In the Python library `gradient_descent`, we can calculate this:

```python
f_grad = f.gradient()

print(f"Gradient: {f_grad}")

a: gradient_descent.Point = {
    x: -1,
    y: 1,
}

f_grad_a: Mapping[gradient_descent.Variable, float] = {
    var: grad_elem.evaluate(a)
    for var, grad_elem
    in f_grad.items()
}

print("Gradient of f(x, y) @ point 'a'")
print(f_grad_a)
```

#### 2.4  Analytical vs. Iterative

Now, understanding the process, we are ready to run Gradient Descent in Python. The optimisation problem we'll solve is minimising:

$$cost= f(x,y) = x^2 + y^2 - 2x - 6y + 14$$

We can solve this analytically, which will be useful in validating the Python implementation:

$$
\begin{align}
f_x(x, y) & = 2x + 0 - 2 - 0 + 0 \\
f_x(x, y) & = 2x - 2\\
\\
f_y(x, y) & = 0 + 2y - 0 - 6 + 0 \\
f_y(x, y) & = 2y - 6
\end{align}
$$

Solving...

$$
\begin{align}
f_x(x, y) & = 2x - 2 = 0 \\
2x - 2 & = 0 \\
2x & = 2 \\
x & = 1 \\
\\
f_y(x, y) & = 2y - 6 = 0 \\
2y - 6 & = 0 \\
2y & = 6 \\
y & = 3 \\
\end{align}
$$

So $f(x, y)$ has a critical point at $(1, 3)$ and we can show graphically that this critical point is a minimum:

```python
import sympy
import numpy as np
import matplotlib.pyplot as plt
```

```python
fig = plt.figure()
ax = fig.gca(projection='3d')

# Create boundaries of f(x,y)
x = np.linspace(-1,15)
y = np.linspace(-1,15)

# Create 2D domain of f(x,y)
xf, yf = np.meshgrid(x,y)

# Discrete version of f(x,y) over 2D domain
fxy = (xf**2) + (yf**2) - (2*xf) - (6*yf) + 14

# Plot our function to be optimised f(x,y)
ax.plot_surface(xf, yf, fxy, alpha=0.1)
ax.view_init(15, 20)

# Plot extrema
x_extrema = np.array([1])
y_extrema = np.array([3])
z_extrema = (x_extrema**2) + (y_extrema**2) - (2*x_extrema) - (6*y_extrema) + 14

ax.scatter(x_extrema, y_extrema, z_extrema,  color='r', label='maxima')

ax.grid(False)
ax.legend()
# plt.locator_params(nbins=6) # Amount of numbers per axis
# Label our axes
ax.set_xlabel("x")
ax.set_ylabel("y")
ax.set_zlabel("z")
```

**Now let's solve the same problem iteratively using the Python implementation of the Gradient Descent algorithm:**

The core of the iterative gradient descent process is nice and simple:

$$ \mathbf{a}_{n+1} = \mathbf {a}_{n}-\gamma \nabla f(\mathbf {a} _{n})$$

$$f(\mathbf{a_0}) \ge f(\mathbf{a_1}) \ge f(\mathbf{a_1}) \ge f(\mathbf{a_2}) \ge f(\mathbf{a_3}) ...$$

This process has 4 clear inputs:

1. The function
2. The initial starting point, $\mathbf{a_0}$
3. The step size, $\gamma$ (gamma)
4. The number of iterations

```python
minimum_val, minimum_point = gradient_descent.gradient_descent(
        gamma=0.1,
        max_iterations=5000,
        f=f,
    )
print("\nResu")
print(f"Min Value: {minimum_val}")
print(f"Min Location: {minimum_point}")
```

Success! The answers are not exact because of [floating-point arithmetic error](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html), but $(1, 3)$ is correct.

We can re-run the function and no matter which values are randomly assigned to the initial starting point, the process converges to the correct result.

### 3. Convergence Behaviour

Having demonstrated gradient descent convergence of a simple convex function, let's investigate convergence behaviour
on different functions when the parameters of the convergence process are manipulated.

#### 3.1 Changing Max Iterations

Clearly we have have a relationship between the function's gradient, the step size, and the number of iterations that can cause nonconvergence when max iterations is limited.

We want the step size to be small enough such that the monotonic series is stable in its convergence to a minima,
but 'small enough' assumes a sufficient series length. By severely limiting iterations, we can prevent convergence even on the simple function we've observed earlier.

```python
minimum_val, minimum_point = gradient_descent.gradient_descent(
        gamma=0.01,
        max_iterations=5,
        f=f,
    )
print("\nResults:")
print(f"Min Value: {minimum_val}")
print(f"Min Location: {minimum_point}")
```

#### 3.2 Changing Step Size

The _Rosenbrock function_ function below was specifically designed to test optimization functions like gradient descent. The global minima of the function is $(1, 1)$ but is difficult to reach this minima because it is in a _very_
shallow valley.

$$f(x,y)=(1-x)^2+100(y-x^2)^2$$

alternate form

$$100 x^4 - 200 x^2 y + x^2 - 2 x + 100 y^2 + 1$$

We can use `matplotlib` to visualise the shallowness of the area in which the $(1, 1)$ global minima resides.
Due to the sensitivity of our contour bands, the function's value area to be unchanging in a large parabolic region
containing the minima. In fact, the area contains a _very slight_ decline.

```python
delta = 0.001
x = np.arange(-3.0, 3.0, delta)
y = np.arange(-2.0, 2.0, delta)

X, Y = np.meshgrid(x, y)

Z1 = np.exp(-X**2 - Y**2)
Z2 = np.exp(-(X - 1)**2 - (Y - 1)**2)
Z = (Z1 - Z2) * 2

x = np.linspace(-2,2)
y = np.linspace(-2,2)

# Create our grid of points
xv, yv = np.meshgrid(x,y)
ax = plt.subplot(1,2,1)

# Make a contour plot that is filled with color.
ax.contourf(xv,yv, (1 - xv)**2+ 100*(yv - (xv**2))**2)
ax.set_title('Contours of f(x,y)')
```

Attempting gradient descent on this function, we do not the global minimum even after a large number of iterations,
_but we get close_.

```python
x = gradient_descent.Variable("x")
y = gradient_descent.Variable("y")
f = gradient_descent.MultiVariableFunction(                                                    # f(x, y) =
    variables={x, y},
    expressions=[
        gradient_descent.PolynomialExpression(variable=x, coefficient=100, exponent=4),        # 100x^4 -
        gradient_descent.Multiply(                                                             # 200x^2y +
            a=gradient_descent.PolynomialExpression(variable=x, coefficient=-200, exponent=2),
            b=gradient_descent.PolynomialExpression(variable=y, coefficient=1, exponent=1),
        ),
        gradient_descent.PolynomialExpression(variable=x, coefficient=1, exponent=2),          # x^2 -
        gradient_descent.PolynomialExpression(variable=x, coefficient=-2, exponent=1),         # 2x +
        gradient_descent.PolynomialExpression(variable=y, coefficient=100, exponent=2),        # 100y^2 +
        gradient_descent.ConstantExpression(real=1.0),                                         # 1
    ],
)

f.gradient()

# The grad descent process on this function is quite sensitive to initial point.
# If started too far away from minima, the gradients 'explode' and convergence does not occur.
initial_points = [
    {x: 1, y: 1},
    {x: 1.1, y: 1.1},
]

MAX_ITERATIONS = 5000  # How big does this need to be to achieve convergence???

for i_p in initial_points:
    minimum_val, minimum_point = gradient_descent.gradient_descent(
        gamma=0.00001,
        max_iterations=MAX_ITERATIONS,
        initial_point=i_p,
        f=f,
    )
    print("\nResults:")
    print(f"--- Min Value: {minimum_val}")
    print(f"--- Min Location: {minimum_point}\n\n")
```

#### 3.3 Changing initial starting points

For functions with _multiple_ local minima, it matters which initial starting point we use, as gradient descent has
no mathematical mechanism of exploring multiple local minima.

The function below has two minima, at $(1, 1)$ and $(-1, -1)$. Gradient descent can find either, depending on initial point of descent.

$$f(x, y) = x^4 + y^4 - 4xy + 1$$

```python
x = gradient_descent.Variable("x")
y = gradient_descent.Variable("y")
f = gradient_descent.MultiVariableFunction(                                                    # f(x, y) =
    variables={x, y},
    expressions=[
        gradient_descent.PolynomialExpression(variable=x, coefficient=1, exponent=4),          # x^4 +
        gradient_descent.PolynomialExpression(variable=y, coefficient=1, exponent=4),          # y^4 -
        gradient_descent.Multiply(                                                             # 4xy +
            a=gradient_descent.PolynomialExpression(variable=x, coefficient=-4, exponent=1),
            b=gradient_descent.PolynomialExpression(variable=y, coefficient=1, exponent=1),
        ),
        gradient_descent.ConstantExpression(real=1.0),                                         # 1
    ],
)

# The grad descent process on this function is quite sensitive to initial point.
# If started too far away from minima, the gradients 'explode' and convergence does not occur.
initial_points = [
    {x: 1, y: 1},
    {x: -1, y: -1},
    {x: -1.5, y: -1.5},
    {x: 1.1, y: 1.2},
]

for i_p in initial_points:
    minimum_val, minimum_point = gradient_descent.gradient_descent(
        gamma=0.1,
        max_iterations=50000,
        initial_point=i_p,
        f=f,
    )
    print("\nResults:")
    print(f"--- Min Value: {minimum_val}")
    print(f"--- Min Location: {minimum_point}")
```

### Conclusion

In this report we've demonstrated the core components of the gradient descent optimization process, in both Python and equations. Convergence has been demonstrated on simple polynomials and more complicated polynomials that can demonstrate how careful parameter tuning of the gradient descent process can be necessary to ensure convergence, even on convex functions.

## References

*  RUMELHART, David E.; HINTON, Geoffrey E.; WILLIAMS, Ronald J. (1986). "Learning representations by back-propagating errors". Nature. 323 (6088): 533–536. doi:10.1038/323533a0. S2CID 205001834 -http://www.cs.utoronto.ca/~hinton/absps/naturebp.pdf
* STEWART, J. (2019). "Calculus: concepts and contexts". Boston, MA, USA, Cengage.
* SANDERSON, G. (2018). "Why the gradient is the direction of steepest ascent". https://www.youtube.com/watch?v=TEB2z7ZlRAw

## Appendix

The full `gradient_descent` implementation is available online at [github.com/thundergolfer/modelling_change/](https://github.com/thundergolfer/modelling_change/), but it has also been copied in below:

```python
"""
Pure-Python3 implementation of Gradient Descent (https://en.wikipedia.org/wiki/Gradient_descent).
Written for educational/learning purposes and not performance.

Completed as part of the UTS course '35512 - Modelling Change' (https://handbook.uts.edu.au/subjects/35512.html).
"""
import random
from typing import List, Mapping, Optional, Dict, Set, Tuple

# Used to make chars like 'x' resemble typical mathematical symbols.
def _italic_str(text: str) -> str:
    return f"\x1B[3m{text}\x1B[23m"

def _superscript_exp(n: str) -> str:
    return "".join(["⁰¹²³⁴⁵⁶⁷⁸⁹"[ord(c) - ord('0')] for c in str(n)])

class Variable:
    """
    A object representing a mathematical variable, for use in building expressions.

    Usage: `x = Variable("x")`
    """
    def __init__(self, var: str):
        if len(var) != 1 or (not var.isalpha()):
            raise ValueError("Variable must be single alphabetical character. eg. 'x'")
        self.var = var

    def __repr__(self):
        return _italic_str(self.var)

    def __eq__(self, other):
        """Overrides the default implementation"""
        if isinstance(other, Variable):
            return self.var == other.var
        return False

    def __key(self):
        return self.var

    def __hash__(self):
        return hash(self.__key())

# An element of some set called a space. Here, that 'space' will be the domain of a multi-variable function.
Point = Dict[Variable, float]

class Expression:
    def diff(self, ref_var: Optional[Variable] = None) -> Optional["Expression"]:
        raise NotImplementedError

    def evaluate(self, point: Point) -> float:
        raise NotImplementedError

class ConstantExpression(Expression):
    """
    ConstantExpression is a single real-valued number.
    It cannot be parameterised and its first-derivative is always 0 (None).
    """

    def __init__(self, real: float):
        super().__init__()
        self.real = real

    def diff(self, ref_var: Optional[Variable] = None) -> Optional[Expression]:
        return None

    def evaluate(self, point: Point) -> float:
        return self.real

    def __repr__(self):
        return str(self.real)

class PolynomialExpression(Expression):
    """
    An expression object that support evaluation and differentiation of single-variable polynomials.
    """
    def __init__(
            self,
            variable: Variable,
            coefficient: float,
            exponent: int
    ):
        super().__init__()
        self.var = variable
        self.coefficient = coefficient
        self.exp = exponent

    def diff(self, ref_var: Optional[Variable] = None) -> Optional[Expression]:
        if ref_var and ref_var != self.var:
            return None
        if self.exp == 1:
            return ConstantExpression(real=self.coefficient)
        return PolynomialExpression(
            variable=self.var,
            coefficient=self.coefficient * self.exp,
            exponent=self.exp - 1,
        )

    def evaluate(self, point: Point) -> float:
        return (
                self.coefficient *
                point[self.var] ** self.exp
        )

    def __repr__(self):
        return f"{self.coefficient}{self.var}{_superscript_exp(str(self.exp))}"

class Multiply(Expression):
    def __init__(self, a: PolynomialExpression, b: PolynomialExpression):
        self.a = a
        self.b = b

    def diff(self, ref_var: Optional[Variable] = None) -> Optional["Expression"]:
        if not ref_var:
            raise RuntimeError("Must pass ref_var when differentiating Multiply expression")
        if self.a.var == ref_var:
            diff_a = self.a.diff(ref_var=ref_var)
            if not diff_a:
                return None
            else:
                return Multiply(a=diff_a, b=self.b)
        elif self.b.var == ref_var:
            diff_b = self.b.diff(ref_var=ref_var)
            if not diff_b:
                return None
            else:
                return Multiply(a=self.a, b=diff_b)
        else:
            return None  # diff with respect to some non-involved variable is 0

    def evaluate(self, point: Point) -> float:
        return self.a.evaluate(point) * self.b.evaluate(point)

    def __repr__(self):
        return f"({self.a})({self.b})"

GradientVector = Dict[Variable, "MultiVariableFunction"]

class MultiVariableFunction:
    """
    MultiVariableFunction support the composition of expressions by addition into a
    function of multiple real-valued variables.

    Partial differentiation with respect to a single variable is supported, as is
    evaluation at a Point, and gradient finding.
    """

    def __init__(self, variables: Set[Variable], expressions: List[Expression]):
        self.vars = variables
        self.expressions = expressions

    def gradient(self) -> GradientVector:
        grad_v: GradientVector = {}
        for v in self.vars:
            grad_v[v] = self.diff(ref_var=v)
        return grad_v

    def diff(self, ref_var: Variable) -> "MultiVariableFunction":
        first_partial_derivatives: List[Expression] = []
        for expression in self.expressions:
            first_partial_diff = expression.diff(ref_var=ref_var)
            if first_partial_diff:
                first_partial_derivatives.append(first_partial_diff)
        return MultiVariableFunction(
            variables=self.vars,
            expressions=first_partial_derivatives,
        )

    def evaluate(self, point: Point) -> float:
        return sum(
            expression.evaluate(point)
            for expression
            in self.expressions
        )

    def __repr__(self):
        return " + ".join([str(e) for e in self.expressions])

def gradient_descent(
    gamma: float,
    max_iterations: int,
    f: MultiVariableFunction,
    initial_point: Optional[Point] = None,
) -> Tuple[float, Point]:
    """
    Implements Gradient Descent (https://en.wikipedia.org/wiki/Gradient_descent) in pure-Python3.6+ with
    no external dependencies.

    :param gamma: 'step size', or 'learning rate'
    :param max_iterations: Maximum number of steps in descent process.
    :param f: A differentiable function off multiple real-valued variables.
    :param initial_point: Optionally, a place to start the descent process
    :return: A tuple of first a local minimum and second the point at which minimum is found.
    """
    if gamma <= 0:
        raise ValueError("gamma value must be a positive real number, γ∈ℝ+")

    iterations_per_logline = 100
    a: Point = {}
    f_grad = f.gradient()

    if not initial_point:
        for v in f.vars:
            a[v] = random.randrange(4)
    else:
        a = initial_point
    for i in range(max_iterations):
        # Calculate function's gradient @ point `a`
        grad_a: Mapping[Variable, float] = {
            var: grad_elem.evaluate(a)
            for var, grad_elem
            in f_grad.items()
        }
        # update estimate of minimum point
        a_next = {
            var: current - (gamma * grad_a[var])
            for var, current
            in a.items()
        }
        a_prev = a
        a = a_next

        if a_prev == a:
            print("Iteration as not changed value. Stopping early.")
            break
        if i % iterations_per_logline == 0:
            print(f"Iteration {i}. Current min estimate: {a}")
    return f.evaluate(a), a

def main() -> None:
    x = Variable("x")
    y = Variable("y")

    # Test variable comparisons
    ##########################
    assert Variable("x") == Variable("x")
    assert Variable("x") != Variable("y")
    assert Variable("y") != Variable("x")
    assert Variable("y") != Variable("z")

    # Test gradient evaluations of Expressions
    ##########################################
    # ConstantExpressions
    assert ConstantExpression(real=0.0).diff() is None
    assert ConstantExpression(real=4.5).diff() is None
    # PolynomialExpression
    poly1 = PolynomialExpression(
        variable=Variable("x"),
        coefficient=2,
        exponent=4,
    )
    poly1_grad1 = poly1.diff()
    assert poly1_grad1.var == Variable("x")
    assert poly1_grad1.coefficient == 8
    assert poly1_grad1.exp == 3
    poly1_grad2 = poly1.diff(ref_var=Variable("y"))
    assert poly1_grad2 is None

    # Test function evaluation
    ##########################
    x = Variable("x")
    y = Variable("y")
    # f = 3x + y^2
    f1 = MultiVariableFunction(
        variables={x, y},
        expressions=[
            PolynomialExpression(variable=x, coefficient=3, exponent=1),
            PolynomialExpression(variable=y, coefficient=1, exponent=2),
        ],
    )
    assert f1.evaluate(point={x: 1.0, y: 1.0}) == 4
    assert f1.evaluate(point={x: 1.0, y: 2.0}) == 7
    # Test function gradient
    g = f1.gradient()
    assert str(g[x]) == "3"

    # Test Multiply
    ##########################
    a = PolynomialExpression(variable=x, coefficient=3, exponent=1)
    b = PolynomialExpression(variable=y, coefficient=1, exponent=2)
    a_times_b = Multiply(a=a, b=b)
    result = a_times_b.evaluate(point={x: 2.0, y: 4.0})
    assert result == (6 * 16)
    result = a_times_b.evaluate(point={x: 3.0, y: 5.0})
    assert result == 225
    # Test diff on multiplication expression
    a_times_b_diff = a_times_b.diff(ref_var=x)
    assert a_times_b_diff.evaluate(point={x: 1.0, y: 5.0}) == 75

if __name__ == "__main__":
    main()
```


# Factoring Polynomials with SymPy

Here is an example that uses [SymPy](http://sympy.org/en/index.html) to factor polynomials.

```python
from ipywidgets import interact
```

```python
from sympy import Symbol, Eq, factor
```

```python
x = Symbol('x')
```

```python
def factorit(n):
    return Eq(x**n-1, factor(x**n-1))
```

```python
factorit(12)
```

    Eq(x**12 - 1, (x - 1)*(x + 1)*(x**2 + 1)*(x**2 - x + 1)*(x**2 + x + 1)*(x**4 - x**2 + 1))

```python
interact(factorit, n=(2,40));
```

    interactive(children=(IntSlider(value=21, description='n', max=40, min=2), Output()), _dom_classes=('widget-in…

```python

```


# Initialize Enviroment

```python
from IPython.display import display, Math
from sympy import *
init_printing()
```

# Plotting
## Single variable function
### Basic plotting

```python
from sympy import symbols
from sympy.plotting import plot
x = symbols('x')

expr = x*x

plot(expr);
```

    <Figure size 640x480 with 1 Axes>

## Suppresss Str representation

If you don't like the '<sympy.plotting.plot.Plot at 0x7f6d26fb8780> string, add a ';' at the end of the plot command.

```python
plot(expr);
```

### Plot with range

To specify the plotting range, pass a tuple of the form ```(variable, lower_limit, upper_limit)```

```python
from sympy import symbols
from sympy.plotting import plot
x = symbols('x')

expr = x**2
expr_range = (x,-2,3)

plot(expr, expr_range);
```

### Title and Label

Pass in keyword arguments ```Title```,```xlabel```,```ylabel```. Latex is supported by these arguments.

```python
from sympy import symbols
from sympy.plotting import plot
x = symbols('x')

expr = x**2
expr_range = (x,-2,3)

title = '$y = {}$'.format(latex(expr))

plot(expr, expr_range, title = title, xlabel = 'x', ylabel = 'y');
```

### Line color

Pass keyword argument ```line_color```.

```python
from sympy import symbols
from sympy.plotting import plot
x = symbols('x')

expr = x**2

plot(expr, expr_range, line_color = 'r');
```

### Multiple plot with same range

Use the syntax of ```plot(expr_1, expr_2, expr_3, range)```

```python
expr_1 = x
expr_2 = x**2
expr_3 = x**3

plot(expr_1, expr_2, expr_3, (x, -1, 1));
```

### Multiple plots with different range

Use the syntax of
```
plot(
    (expr_1,range_1),
    (expr_2,range_2),
    ...
)
```

```python
expr_1 = x**2
range_1 = (x,-2,2)

expr_2 = x
range_2 = (x,-1,1)

plot(
    (expr_1,range_1),
    (expr_2,range_2)
);
```

### Multiple plots with differnt colors

Plotting multiple plots with different colors is not straight forward.

1. Pass ```show = False``` to suppress displaying the plot, save the returned object to a variable.
2. Get the indivisual data series by indexing the plotting object and set line_color seperately.
3. Call ```show()``` method of the plotting object to display the image.

```python
expr_1 = x**2
range_1 = (x,-2,2)

expr_2 = x
range_2 = (x,-1,1)

p = plot(
    (expr_1,range_1),
    (expr_2,range_2),
    show = False
);

p[0].line_color = 'r'
p[1].line_color = 'b'

p.show()
```

To add a legend to the plot, take additional two steps.

1. Pass ```Legend = True``` when construct the plotting object.
2. Set label for each data series seperately.

```python
expr_1 = x**2
range_1 = (x,-2,2)

expr_2 = x
range_2 = (x,-1,1)

p = plot(
    (expr_1,range_1),
    (expr_2,range_2),
    show = False,
    legend = True
);

p[0].line_color = 'r'
p[1].line_color = 'b'

p[0].label = 'Line 1'
p[1].label = 'Line 2'

p.show()
```

## 3D surface plot
### Single plot

Use ```plot3d(expr, x_range , y_range)``` to draw a 3D surface plot.

```python
from sympy import symbols
from sympy.plotting import plot3d

x, y = symbols('x y')

expr = x*y
x_range = (x, -5, 5)
y_range = (y, -5, 5)

plot3d(expr, x_range, y_range);
```

### Multiple plot

Use the following syntax to draw multiple 3D surface plot.

```
plot3d(
    （expr_1, x_range_1 , y_range_1）,
    （expr_2, x_range_2 , y_range_2）
)
```

```python
plot3d(
    (x**2 + y**2, (x, -5, 5), (y, -5, 5)),
    (x*y, (x, -3, 3), (y, -3, 3))
);
```

## Single variable parametric function

Use ```plot_parametric(expr_x, expr_y, range_u)``` to draw a single variable parametric function.

```python
from sympy import symbols, cos, sin
from sympy.plotting import plot_parametric

u = symbols('u')
expr_x = cos(u)
expr_y = sin(u)

p = plot_parametric(expr_x, expr_y, (u, -5, 5));
```

The result is not a perfect circle. Sympy offers the ```aspect_ratio``` argment to adjust the ratio, but it doesn't work in sympy 1.0 yet. The community is working on the problem and it may work in the next release.

## 3D parametric line

Use ```plot3d_parametric_line(expr_x, expr_y, expr_z, range_u)``` to draw a 3D parametric function.

```python
from sympy import symbols, cos, sin
from sympy.plotting import plot3d_parametric_line
u = symbols('u')
expr_x = cos(u)
expr_y = sin(u)
expr_z = u

plot3d_parametric_line(expr_x, expr_y, expr_z, (u, -5, 5));
```

## 3D parametric surface
Use ```plot3d_parametric_surface(expr_x, expr_y, expr_z, u_range, v_range)``` to draw a 3d parametric surface.

```python
from sympy import symbols, cos, sin
from sympy.plotting import plot3d_parametric_surface
u, v = symbols('u v')

expr_x = cos(u + v)
expr_y = sin(u-v)
expr_z = u-v
u_range = (u, -5, 5)
v_range = (v, -5, 5)

plot3d_parametric_surface(expr_x, expr_y, expr_z, u_range, v_range);
```

## Implicit function
### Single variable implicit function
Use ```plot_implicit(equition)``` to draw implicit function plotting.

```python
p1 = plot_implicit(Eq(x**2 + y**2-5))
```

### Multiple variable implicit function

```python
p2 = plot_implicit(
    Eq(x**2 + y**2, 3),
    (x, -3, 3),
    (y, -3, 3)
)
```

## Implicit inequilities

Pass an inequility to ```plot_implicit```

```python
plot_implicit(y > x**2);
```

To combine several conditions to define the region, and ```And,Or``` logic conjunctions.

```python
plot_implicit(And(y > x, y > -x));
```

```python
plot_implicit(Or(y > x, y > -x));
```

Sometimes Sympy doesn't choose the variable for horizontal axis as you expect.

```python
plot_implicit(Eq(y - 1));
```

In this case, use ```x_var``` to choose the variable for x_axis.

```python
plot_implicit(Eq(y - 1),x_var=x);
```

# Reference
[Sympy Documentation](http://docs.sympy.org/latest/index.html)

# Related Articles
* [Sympy Notes I]({filename}0026_sympy_intro_1_en.ipynb)
* [Sympy Notes II]({filename}0027_sympy_intro_2_en.ipynb)
* [Sympy Notes III]({filename}0028_sympy_intro_3_en.ipynb)
* [Sympy Notes IV]({filename}0029_sympy_intro_4_en.ipynb)


# Advanced Expression Manipulation

```python
from sympy import *
x, y, z = symbols('x y z')
```

For each exercise, fill in the function according to its docstring.

## Creating expressions from classes

Create the following objects without using any mathematical operators like `+`, `-`, `*`, `/`, or `**` by explicitly using the classes `Add`, `Mul`, and `Pow`.  You may use `x` instead of `Symbol('x')` and `4` instead of `Integer(4)`.

$$x^2 + 4xyz$$
$$x^{(x^y)}$$
$$x - \frac{y}{z}$$

```python
def explicit_classes1():
    """
    Returns the expression x**2 + 4*x*y*z, built using SymPy classes explicitly.

    >>> explicit_classes1()
    x**2 + 4*x*y*z
    """

```

```python
explicit_classes1()
```

```python
def explicit_classes2():
    """
    Returns the expression x**(x**y), built using SymPy classes explicitly.

    >>> explicit_classes2()
    x**(x**y)
    """

```

```python
explicit_classes2()
```

```python
def explicit_classes3():
    """
    Returns the expression x - y/z, built using SymPy classes explicitly.

    >>> explicit_classes3()
    x - y/z
    """

```

```python
explicit_classes3()
```

## Nested args

```python
expr = x**2 - y*(2**(x + 3) + z)
```

Use nested `.args` calls to get the 3 in expr.

```python
def nested_args():
    """
    Get the 3 in the above expression.

    >>> nested_args()
    3
    """

```

```python
nested_args()
```

## Traversal

Write a post-order traversal function that prints each node.

```python
def post(expr):
    """
    Post-order traversal

    >>> expr = x**2 - y*(2**(x + 3) + z)
    >>> post(expr)
    -1
    y
    2
    3
    x
    x + 3
    2**(x + 3)
    z
    2**(x + 3) + z
    -y*(2**(x + 3) + z)
    x
    2
    x**2
    x**2 - y*(2**(x + 3) + z)
    """

```

```python
post(expr)
```

```python
for i in postorder_traversal(expr):
    print(i)
```


## Solution to exercise 1

### Dynamic Programming with John Stachurski

The question was: Does there always exists a $x \in [0, \infty)$ that solves the equation
$$
    x
    = c (1-\beta) + \beta
    \sum_{k=1}^K \max \left\{
        w_k ,\, x
    \right\}
    \, p_k
$$
Is it unique?  Suggest a strategy for computing it.

We are assuming here that $\beta \in (0, 1)$.

There were hints, as follows:

* Use the metric space $(\mathbb R, \rho)$ where $\rho(x, y) = |x-y|$

*  If $x_1, \ldots, x_K$ are any $K$ numbers, then

$$ \left| \sum_{k=1}^K x_k \right| \leq \sum_{k=1}^K |x_k| $$

* For any $a, x, y$ in $\mathbb R$,

$$
    \left|
            \max \left\{ a,\, x \right\} - \max \left\{ a,\, y \right\}
        \right|
        \leq | x - y |
$$

You can convince yourself of the second inequality by sketching and checking different cases...

### Solution

We're going to use the contraction mapping theorem.  Let

$$
    f(x)
    = c (1-\beta) + \beta
    \sum_{k=1}^K \max \left\{
        w_k ,\, x
    \right\}
    \, p_k
$$

We're looking for a fixed point of $f$ on $\mathbb R_+ = [0, \infty)$.

Using the hints above, we see that, for any $x, y$ in $\mathbb R_+$, we have

\begin{align}
    |f(x) - f(y)|
    & = \left|
          \beta \sum_{k=1}^K \max \left\{
                w_k ,\, x
            \right\} \, p_k
           -
            \beta \sum_{k=1}^K \max \left\{
                w_k ,\, y
            \right\} \, p_k
         \right|
            \\
    & = \beta\, \left|
               \sum_{k=1}^K [\max \left\{
                w_k ,\, x
            \right\} - \max \left\{
                w_k ,\, y
            \right\} ]\, p_k
         \right|
            \\
    & \leq \beta\,\sum_{k=1}^K
            \left|
                \max \left\{
                w_k ,\, x
            \right\} - \max \left\{
                w_k ,\, y
            \right\}
         \right|  p_k
            \\
                & \leq \beta\,\sum_{k=1}^K
            \left|
                x - y
         \right|  p_k
            \\
\end{align}

Since $\sum_k p_k = 1$, this yields

$$ |f(x) - f(y)| \leq \beta |x - y| $$

Hence $f$ is a contraction map on $\mathbb R_+$, and therefore has a unique fixed point $x^*$ such that $f^n(x) \to x^*$ as $n \to \infty$ from any $x \in \mathbb R_+$.

Let's plot $f$ when

* $K = 2$
* $w_1 = 1$ and $w_2 = 2$
* $p_1 = 0.3$ and $p_3 = 0.7$
* $c=1$ and $\beta = 0.9$

```python
import numpy as np
import matplotlib.pyplot as plt

c, w1, w2, p1, p2, β = 1, 1, 2, 0.3, 0.7, 0.9

def f(x):
    return c * (1 - β) + β * (max(x, w1)*p1 + max(x, w2)*p2)
```

```python
xvec = np.linspace(0, 4, 100)
yvec = [f(x) for x in xvec]

fig, ax = plt.subplots()
ax.plot(xvec, yvec, label='$f$')
ax.plot(xvec, xvec, 'k-', label='$45$')
ax.legend()
plt.show()
```

Now let's compute that fixed point by iteration:

```python
x = 1.0
x_vals = []
for i in range(50):
    x_vals.append(x)
    x = f(x)

fig, ax = plt.subplots()
ax.plot(x_vals)
ax.set(xlabel="$n$", ylabel="$f^n(x)$")
ax.grid()
plt.show()
```

```python

```


### floor, ceil

$\lfloor \sqrt{\lfloor x\rfloor}\rfloor = \lfloor \sqrt{ x}\rfloor$

$\lceil \sqrt{\lceil x\rceil}\rceil = \lceil \sqrt{x}\rceil$

continuous, monotonically increasing $f$ with property $f(x)\in Z \Rightarrow x \in Z$. Then below holds
$$
\begin{align}
    \lfloor f(\lfloor x\rfloor)\rfloor = \lfloor f(x)\rfloor,\ \lceil f(\lceil x\rceil)\rceil = \lceil f(x)\rceil
\end{align}
$$

$\lceil \sqrt{\lfloor x\rfloor}\rceil = \lceil \sqrt{ x}\rceil$, only when $m^2 < x < m^2+1$ not hold.

$$
\begin{align}
    \sum_{k=0}^{m-1} \lfloor \frac{n+k}{m} \rfloor =n= \sum_{k=0}^{m-1} \lceil \frac{n-k}{m} \rceil
\end{align}
$$

$$
\begin{align}
    \sum_{k=0}^{n-1} \lfloor \sqrt{k} \rfloor = na - \frac{1}{3} a^3 - \frac{1}{2}a^2 - \frac{1}{6}a,\ a = \lfloor \sqrt{n} \rfloor
\end{align}
$$

$$
\begin{align}
    \sum_{k=0}^{m-1} \lfloor \frac{nk+x}{m} \rfloor = d  \lfloor \frac{x}{d} \rfloor + \frac{(m-1)(n-1)}{2} + \frac{d-1}{2} = \sum_{k=0}^{n-1} \lfloor \frac{mk+x}{n} \rfloor,\ d = \text{gcd}(m,n)
\end{align}
$$

```python

```


# Gleichungssystem
Lösen Sie das folgende Gleichungssystem einmal ausschließlich mit `sympy` Routinen und einmal mit `numpy`:

$$
5x+4y+3z = 2 \\
2x+3y+z = 0 \\
3x+3y+z = 2 \\
$$

Geben Sie die Ergebnisse für $x, y, z$ sinnvoll strukturiert am Bildschirm aus.

```python
from sympy import Eq, solve
from sympy.abc import x, y, z

system = [
    Eq(5*x + 4*y + 3*z, 2),
    Eq(2*x + 3*y + z, 0),
    Eq(3*x + 3*y + z, 2)
]
solution = solve(system, [x, y, z])
for k, v in solution.items():
    print(f'{k} = {float(v)}')
```

    x = 2.0
    y = -0.8
    z = -1.6

```python
import numpy as np
system = np.array(
    [
        [5, 4, 3],
        [2, 3, 1],
        [3, 3, 1]
    ]
)
solution = np.linalg.solve(system, np.array([2, 0, 2]))
for k, v in zip('xyz', solution):
    print(f'{k} = {round(v, 1)}')
```

    x = 2.0
    y = -0.8
    z = -1.6


```python
from sympy import *
```

```python
e1, p1, e2, p2, m = symbols("E_1 p_1 E_2 p_2 m",real=True, positive=True)
```

We work in 1-dimensional coordinates with $p = p_z$. We generally use $m = m_p = m_n$, in other words we neglect the mass difference between protons and neutrons. We consider a proton projectile and a nuclear target with $A$ nucleons.

We first compute the cms energy $\sqrt{s_{nn}}$ in the nucleon-nucleon system from the fixed target system.

In this system, we have these 4-vectors:
* projectile $(E, p)$
* target $(m, 0)$

```python
s2 = (e1 + m)**2 - (p1 + 0)**2; s2
```

$\displaystyle - p_{1}^{2} + \left(E_{1} + m\right)^{2}$

```python
s2 = s2.subs(e1, sqrt(p1 ** 2 + m ** 2)); s2
```

$\displaystyle - p_{1}^{2} + \left(m + \sqrt{m^{2} + p_{1}^{2}}\right)^{2}$

```python
s2.series(m, 0, 2)
```

$\displaystyle 2 m p_{1} + O\left(m^{2}\right)$

```python
s = sqrt((e1 + m)**2 - (p1 + 0)**2); s
```

$\displaystyle \sqrt{- p_{1}^{2} + \left(E_{1} + m\right)^{2}}$

```python
s = s.subs(e1, sqrt(p1 ** 2 + m ** 2)); s
```

$\displaystyle \sqrt{- p_{1}^{2} + \left(m + \sqrt{m^{2} + p_{1}^{2}}\right)^{2}}$

```python
s.series(m, 0, 2)
```

$\displaystyle \sqrt{2} \sqrt{m} \sqrt{p_{1}} + \frac{\sqrt{2} m^{\frac{3}{2}}}{2 \sqrt{p_{1}}} + O\left(m^{2}\right)$

```python
def sqrt_snn(energy):
    mass = (938.27 + 939.57) * 0.5e-9 # nucleon mass in PeV
    return sqrt(2 * mass * energy) * 1e3 # sqrt(sNN) in TeV
```

```python
0.5 * (sqrt_snn(320000+90000) - sqrt_snn(320000-90000))
```

$\displaystyle 110.127117815583$

```python
def beam_energy(sqrt_snn):
    mass = (938.27 + 939.57) * 0.5e-3 # nucleon mass in GeV
    return (sqrt_snn * 1e3) ** 2 / (2 * mass) / 1e6 # beam in PeV
```

```python
beam_energy(14)
```

    104.37523963702976

From fixed target to sqrt_s_nn and back.ipynb


Initialize printing:

```
%pylab inline
from sympy.interactive.printing import init_printing
init_printing()
```

    Welcome to pylab, a matplotlib-based Python environment [backend: module://IPython.zmq.pylab.backend_inline].
    For more information, type 'help(pylab)'.

# Poisson Equation

Radial Poisson equation is
$$
\phi''(r)+{2\over r} \phi'(r) = -4\pi\rho(r).
$$
Alternatively, this can also be written as:
$$
{1\over r}(r \phi(r))'' = -4\pi\rho(r).
$$
### Example I

Positive Gaussian charge density:

```
from sympy import var, exp, pi, sqrt, integrate, refine, Q, oo, Symbol, DiracDelta
var("r alpha Z")
alpha = Symbol("alpha", positive=True)
rho = Z * (alpha / sqrt(pi))**3 * exp(-alpha**2 * r**2)
rho
```

The total charge is $Z$:

```
integrate(4*pi*rho*r**2, (r, 0, oo))
```

Solve for $\phi(r)$ from the Poisson equation:

```
phi = integrate(-4*pi*rho*r, r, r)/r
phi
```

Tell SymPy that $\alpha$ is positive:

```
phi = refine(phi, Q.positive(alpha))
phi
```

Plot the charge:

```
from sympy import plot
```

```
plot(rho.subs({Z: 1, alpha: 0.1}), (r, 0, 100));
```

Plot the potential

```
plot(phi.subs({Z: 1, alpha: 0.1}), 1/r, (r, 0, 100), ylim=[0, 0.12]);
```

```

```


---
author: Nathan Carter (ncarter@bentley.edu)
---

This answer assumes you have imported SymPy as follows.

```python
from sympy import *                   # load all math functions
init_printing( use_latex='mathjax' )  # use pretty math output
```

In SymPy, we tend to work with formulas (that is, mathematical expressions)
rather than functions (like $f(x)$).  So if we wish to compute the
derivative of $f(x)=10x^2-16x+1$, we will focus on just the $10x^2-16x+1$ portion.

```python
var( 'x' )
formula = 10*x**2 - 16*x + 1
formula
```

$\displaystyle 10 x^{2} - 16 x + 1$

We can compute its derivative by using the `diff` function.

```python
diff( formula )
```

$\displaystyle 20 x - 16$

If it had been a multi-variable function, we would need to specify the
variable with respect to which we wanted to compute a derivative.

```python
var( 'y' )               # introduce a new variable
formula2 = x**2 - y**2   # consider the formula x^2 + y^2
diff( formula2, y )      # differentiate with respect to y
```

$\displaystyle - 2 y$

We can compute second or third derivatives by repeating the variable
with respect to which we're differentiating.  To do partial derivatives,
use multiple variables.

```python
diff( formula, x, x )    # second derivative with respect to x
```

$\displaystyle 20$

```python
diff( formula2, x, y )   # mixed partial derivative
```

$\displaystyle 0$

database/tasks/How to compute the derivative of a function/Python, using SymPy.ipynb