# Neutron scattering and Monte Carlo methods

Please indicate your name below, since you will need to submit this notebook completed latest the day after the datalab.

Don't forget to save your progress during the datalab to avoid any loss due to crashes.

```python
name=''
```

In this datalab we are going to get acquainted with the basics of Monte Carlo particle transport methods, and we will learn how to sample random events and random values from various distributions. These are going to be our bricks which later we will put together into an actual Monte Carlo simulation.

Since neutron reactions, espescially scattering, provide an excellent ground to familiarize ourselves with Monte Carlo particle transport methods, we will also use this lab to review some of the features of elastic neutron scattering.

**Prerequisites**: Before the lab you should have reviewed the lecture on neutron scattering and the short introduction on Monte Carlo methods and pseudorandom numbers.

The new python knowledge from the lab is going to be
- histograms with `plt.hist`
- random number generators from `numpy.random`

Let's get started and have some fun!

## Experiment 1: Relation of angle and energy in elastic scattering

We have discussed the elastic potential scattering in the CM frame, and showed that for the LAB energy

$$E_l'=\frac{1}{2}E_l[(1+\alpha)+(1-\alpha)\cos\theta_C]$$

where

$$\alpha=\big(\frac{A-1}{A+1}\big)^2$$

and $A=M/m$

Let's investigate how the ratio of the incoming and the outgoing neutron energy depends on the scattering angle.

Plot the above formula for several nuclides (eg. A=1, A=12, A=23, etc) and for angles between $0^\circ-360^\circ$. Do not repeat the plotting command, use a loop instead to iterate through all mass numbers. After the plot write a sentence on your conclusion!

**Note #1**: Remember, `np.cos` can perform the operation on a numpy array or list.

**Note #2**: Trigonometric functions in numpy take values in radians.

**Note #3**: $\pi$ can be accessed as `np.pi`.

**Note #4**: If you wish to use specific colors for the curves, you can define your own list of colors, and call a color according to the indices in the plot (eg. `colors=['blue','green',...]`

```python
import numpy as np
import matplotlib.pyplot as plt
theta=#np.linspace(0,360,361)*np.pi/180 #Remove the first comment!
As=[1,12,23] #Feel free to add more
plt.figure()
# Your loop and plotting comes here
plt.xlabel('angle (deg)')
plt.ylabel(r"$\frac{E_{n,after}}{E_{n,before}}$")
plt.show()
```

Change this cell to your conclusion!

## Experiment 2: Isotropic directions

When sampling *isotropic* directions, one is often tempted to think that the colatitude or polar angle $\theta$ is uniformly distributed over $[0,\pi]$ and the azimuth $\phi$ is uniformly distrubted over $[0,2\pi]$. However this is not the case. It is $\cos\theta$ which is uniformly distributed over $[-1,1]$. Further reading: http://corysimon.github.io/articles/uniformdistn-on-sphere/ (note the angles are named opposite).  Remember the conversion between Cartesian coordinates and polar coordinates:

$$x=r\sin\theta\cos\phi$$
$$y=r\sin\theta\sin\phi$$
$$z=r\cos\theta$$

Read and run the two code cells below. The code creates 1000 unit length ($r=1$) vectors' coordinates, and visualizes them. The two code blocks contain the same code besides the way how `theta` is being created. The first code block samples `theta` uniformly between $[0,\pi]$ (incorrect), and the second samples the cosine of `theta` uniformly between $[-1,1]$. Observe that in the first **incorrect** case the poles are oversampled.

**Note #1**. We are using `np.random.uniform` to generate uniformly generated random numbers. The first input of this function is the lower boundary of the distribution, the second is the higher boundary, and the third is the number of random numbers to be sampled. Note that `np. random` has several built-in functions to sample random numbers from various distributions, you can review them with `?np.random`.

```python
from mpl_toolkits.mplot3d import Axes3D

N=1000
r=np.ones(N)
theta=np.random.uniform(0,np.pi,N) ### INCORRECT
mu=np.cos(theta)
phi=np.random.uniform(0,2*np.pi,N)

x=r*np.sin(theta)*np.cos(phi)
y=r*np.sin(theta)*np.sin(phi)
z=r*np.cos(theta)

plt.figure()
plt.scatter(phi,theta)
plt.xlabel(r'$\phi$')
plt.ylabel(r'$\theta$')
plt.title('Incorrect solution')
plt.show()

fig = plt.figure(figsize=plt.figaspect(1.0)*1.5) #Adjusts the aspect ratio and enlarges the figure (text does not enlarge)
ax = fig.gca(projection='3d')
ax.scatter(x,y,z)
plt.title('Incorrect solution')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
ax.azim = 113
ax.elev = 28
plt.show()
```

```python
N=1000
r=np.ones(N)
mu=np.random.uniform(-1,1,N)        ### CORRECT
theta=np.arccos(mu)
phi=np.random.uniform(0,2*np.pi,N)

x=r*np.sin(theta)*np.cos(phi)
y=r*np.sin(theta)*np.sin(phi)
z=r*np.cos(theta)

plt.figure()
plt.scatter(phi,theta)
plt.xlabel(r'$\phi$')
plt.ylabel(r'$\theta$')
plt.title('Correct solution')
plt.show()

fig = plt.figure(figsize=plt.figaspect(1.0)*1.5) #Adjusts the aspect ratio and enlarges the figure (text does not enlarge)
ax = fig.gca(projection='3d')
ax.scatter(x,y,z)
plt.title('Correct solution')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
ax.azim = 113
ax.elev = 28
plt.show()
```

## Experiment 3: Distribution of outgoing energy

We just showed that isotropic scattering means that the CM cosine of the angle is uniformly distributed. So let us combine exercise 1 and 2, and investigate the distribution of the outgoing neutron energy for isotropic scattering.

Generate 1 million uniformly distributed angle cosines in CM (`muC`), and  calculate the final energy distribution of 1 MeV neutrons after scattering isotropically. Then use `plt.hist` to visualize the distribution of the energy. What is your  expectation? Conclude what you have found.

```python
A=10 #You can change this to study other target nuclides
Ei=1 #MeV
Nsample=1e6 #Number of angles to sample

alpha=# Finish the line
muC=np.random.uniform()#Complete the line
Ef=#Final energy from muC, Ei, alpha. Note: muC is already the cosine!

#Here we create a histogram with 100 bins
Nbin=100
plt.figure()
plt.hist(Ef,Nbin)
plt.axvline(Ei,color='r') #adds vertical line at Ei
plt.axvline(alpha*Ei,color='r')
plt.show()

```

Change this cell to your conclusion!

## Experiment 4: Scattering angle in LAB

We looked into how the energy change depends on the CM scattering angle. We saw what isotropy in CM means, and we also saw how the outgoing energy is distributed for isotropic CM angles. There is one last thing, which can  sometimes be confusing: we intuitively prefer the LAB system! So how does the cosine of the scattering angle look in the LAB? That's what we will try to find out now!

Sample 1 million angle cosines in the CM (`muC`), and then convert the angle to the LAB (`thetaL`). Use the formula below, and finally calculate the cosine of the LAB angle (`muL`). The formula to convert from CM to LAB:

$$\theta_L=\tan^{-1}\Big(\frac{\sin \theta_C}{\frac{1}{A}+\mu_C}\Big)$$

Read and execute the code block below to evaluate this for several mass numbers and to calculate the mean (with `np.mean`) of the LAB cosines. Compare the empirical mean with the value from the lecture ($\bar\mu_L=\frac{2}{3A}$)

What is your conclusion: is the scattering isotropic in the LAB? Write down your conclusion!

```python
As=[1,12,50,238]

muC=np.random.uniform(-1,1,1000000)
thetaC=np.arccos(muC)

for A in As:

    thetaL=np.arctan2(np.sin(thetaC),((1/A)+muC))

    muL=np.cos(thetaL)

    plt.figure()
    plt.hist(muL,100)
    plt.xlabel(r'$\mu_L$')
    plt.ylabel('number of occurrences')
    plt.title(str(A))
    plt.show()
    print(str(A),str(2/(3*A)),str(np.mean(muL)))
```

Change this cell to your conclusion!

## Experiment 5: Neutron slowing down in elastic scattering

Let's slow down a neutron! From the previous exercises we could conclude that in our "billiard ball" scattering the energy and the angle is in a direct relationship. Therefore, we can sample one and figure out the other. But for the moment let's just neglect the angle completely and care only about the energy loss (for which the distribution we have played with in Experiment 3). We will try to figure out how many scattering events it takes to slow down a neutron from 2 MeV to 1eV (we consider that there are no temperature effects).

Investigate some target nuclide. Which nuclides are effective in slowing neutrons? Note down some values.

```python
def neutronLetargizer(Ei,Ef,A):
    """Function calculate how many scattering events are needed to slow a neutron
    from an initial energy (Ei) to a final energy (Ef)
    Parameters
    ----------
    Ei : float
        Initial energy of the neutron
    Ef : float
        Final energy to be reached
    A : float
        Mass number of the scatterer

    Returns
    -------
    N : int
        Number of scattering events
    """
    alpha=#finish the line
    N=0
    E=Ei
    while E>=Ef:
        E=#sample a random outgoing energy based on alpha and E.
        N=N+1
    return N
```

Let's run this function for various nuclides, both light and heavy!

```python
A=1
Ei=2.0 #MeV
Ef=1e-6 #MeV
print(neutronLetargizer(Ei,Ef,A))
```

That's a pretty cool function we have now. We can use it to look at the statistics of the number of scattering needed! Let's run this function for 10k neutrons and calculate the mean and visualize the distribution of the number of scattering events needed. (Notice that running this many neutrons for heavy nuclide might take some time)

```python
A=12
E0=2.0 #MeV
Ef=1e-6 #MeV

#######
Ns=[]

for _ in range(10000): # _ means here that we just don't care about the iteration index
    Ns.append(neutronLetargizer(E0,Ef,A))

print('Mean \t Standard deviation')
print()#complete the line by calculating the mean of the list of number of scattering events with np.mean and
       #the standard deviation with np.std

plt.figure()
# use plt.hist to investigate the distribution of the number of scattering events.
plt.show()
```

Change this cell to your conclusion! And fill out the table

|A  | $\bar N$ | std |
|---|----------|-----|
|1  |    ?     |  ?  |
|12 |    ?     |  ?  |
|238|    ?     |  ?  |

## Experiment 6: Sampling from distributions

In the video recording and in the Appendix of the lecture notes we have reviewed how to sample numbers from distributions, in the following we are going to implement these methods for neutron transport related tasks.

### Discrete distribution: which event happens?

The probability of reaction $i$ happening at energy $E$ is

\begin{equation}
\frac{\Sigma_i(E)}{\Sigma_t(E)}
\end{equation}

Let us consider that in our material only two reactions might happen: scattering or capture, thus a simple condition can be used to decide which happens.

Complete the `reactionType` function to return a random event type. Assume that at the energy the neutron is travelling with $\Sigma_s=0.64 \: \text{cm}^{-1}$ and $\Sigma_c=0.39 \: \text{cm}^{-1}$. Call the function with these values.

```python
def reactionType(SigS,SigC):
    """Function to sample a random event type

    Parameters
    ----------
    SigS : float
        Macroscopic scattering cross section
    SigC : float
        Macroscopic capture cross section
    """
    SigT=#complete the line
    x=#sample random number between 0 and 1
    if x < SigS/SigT:
        return 'scatter'
    # else return 'capture'
    #

ss=0.64
sc=0.39
print()#complete the line with the function call
```

Numpy actually has a built in function `np.random.choice()`, which does the same for us. As an input it takes a list of choices to sample from, and optionally one can also pass a list of probabilities.

```python
np.random.choice(['scatter','capture'],p=[ss/(ss+sc),sc/(ss+sc)])
```

### Continous distribution I: path to next collision

Let's consider that we have some probability density function $p(x)$, and the related cumulative distribution function is $F(x)=\int_{-\infty}^xp(t)dt$. This function is going to take values between 0 and 1. So if we can sample random numbers uniformly between 0 and 1, we could just convert them by evaluating the inverse of the cumulative distribution function to obtain a random value $x$ sampled from the distribution:

$x=F^{-1}(r)$

This of course is only useful, when it is possible to easily integrate the probability density function.

Let's see how can we use this to sample random distances travelled by a neutron between collision events. We learnt that

$\exp(-\Sigma_t x)$ is the probability that a neutron moves a distance dx without any interaction.

and

$\Sigma_t \exp(-\Sigma_t x)dx$ is the probability that the neutron has its interaction at dx.

So

$p(x)=\Sigma_t \exp(-\Sigma_t x)$

Thus

$F(x)=1-\exp(\Sigma_tx)$

If we take the inverse, to sample a random path

$x=-\frac{\ln(1-r)}{\Sigma_t}$

but if r is uniform over $[0,1]$, than $1-r$ is also uniform over $[0,1]$, so this simplifies to

$x=-\frac{\ln r}{\Sigma_t}$

Complete the `distanceToCollision` function below.

**Note #1** computational speed is everything in MC calculations. Although in this course we don't try to avoid every unnecessary operation, this example is just to highlight that sometimes operations can be avoided with some reasoning.

**Note #2** the natural logarithm can be computed with `np.log`.

**Note #3** `numpy.random` has a built-in function to sample the exponential distribution, nevertheless here we will convert the uniformly distributed random numbers between $[0,1]$ to exponentially distributed random numbers.

```python
def distanceToCollision(SigT,N=1):
    """Function to sample the distance between collisions

    Parameters
    ----------
    SigT : float
        Total Macroscopic cross section in 1/cm
    N : int
        Number of events to be sampled (default=1)

    Returns
    -------
    x : float or array-like
        Random distance between collisions
    """
    r = np.random.uniform(0,1,N)
    x = # Complete the line
    return x
```

We can now try this function. Let's consider that 1 MeV neutrons enter a material which has a total cross section of $\Sigma_t=0.18 \: \text{cm}^{-1}$ at this energy. Or well, let's consider that 10k neutrons enter the material, and let's see how the distribution of the random distances looks like, and what is the mean.

```python
SigT=0.18
N=10000
ds=#call distanceToCollision() here

plt.figure()
plt.hist(ds,100)
plt.show()

print('Empirical Mean (cm) \t Theoretical mean (cm)')
print() #print the empirical mean free path. and the mean free path expected from theory
```

### Continous distribution II: Watt distribution

When the probability density function is less well behaving, and we cannot obtain the cumulative distribution function easily, we can use for example the rejection method. In this case, we draw a random number (r1), convert it to be between $a$ and $b$ (the bounds of the random value), then we draw an other random number (r2) to create a $y$ value based on the maximum of the probaility density function (M). If the $(x,y)$ pair is under the curve (ie. $y<p(x)$) we accept the value.

**Note** This might be very inefficient if the probability density function is peaked. There are several other methods to more efficient sampling.

Let's try to use this method for sampling the Watt-spectrum which is the probability density function of the energy of neutrons emerging from fission.

$$\chi (E)=C_1\cdot \exp(-\frac{E}{C_2})\cdot \sinh(\sqrt{C_3\cdot E})$$

For now, we will just visualize how the function works (later in a Home Assignment you will extend this method to generate random numbers sampled from this distribution)

Draw 100 numbers $x$ between 0-10 MeV and draw 100 numbers $y$ between 0 and the maximum of $\chi(E)$. If the sampled energy is accepted, plot the $(x,y)$ coordinate with green, else with red.

Does this method seem to be efficient to sample the Watt-spectrum? Count the number of accepted random samples to estimate the efficiency!

```python
def watt(x):
    C1 = 0.453
    C2 = 0.965
    C3 = 2.29
    return #complete the line

E=np.linspace(0,10,10000)
plt.figure()
plt.plot(E,watt(E))
maxW=np.max(watt(E))

for _ in range(100):
    xi=np.random.uniform(0,10)
    yi=#complete the line
    if yi<watt(xi):
        plt.plot(xi,yi,'gx')
    #complete the if/else statements
    #count how often a number is accepted!

plt.xlabel('Energy (MeV)')
plt.ylabel(r'$\chi (MeV^{-1})$')
plt.show()

print()#print the estimated efficiency
```

Change this cell to your conclusion!

# Experiment 7: scattering of thermal neutrons (optional)

Do you feel brave enough to look at the scattering kernel for thermal neutrons? If yes, you are at the right place.

Implement the following scattering kernel, then read and run the code block below for plotting the kernel. We are going to plot $\sigma_s(E'\rightarrow E)E'(1-\alpha)$

\begin{equation}
\begin{aligned}
\sigma_s(E'\rightarrow E)=\frac{\sigma_s}{2E'}\eta^2\Bigg[\text{erf}\Bigg(\eta\sqrt{\frac{E}{kT}}-\rho\sqrt{\frac{E'}{kT}}\Bigg)\pm \text{erf}\Bigg(\eta\sqrt{\frac{E}{kT}}+\rho\sqrt{\frac{E'}{kT}}\Bigg)\Bigg]+ \\ \frac{\sigma_s}{2E'}\eta^2\exp\Bigg(-\frac{E-E'}{kT}\Bigg)\Bigg[\text{erf}\Bigg(\eta\sqrt{\frac{E'}{kT}}-\rho\sqrt{\frac{E}{kT}}\Bigg)\mp \text{erf}\Bigg(\eta\sqrt{\frac{E'}{kT}}+\rho\sqrt{\frac{E}{kT}}\Bigg)\Bigg]
\end{aligned}
\end{equation}

where

$$\eta=\frac{A+1}{2\sqrt{A}} \quad \text{and} \quad \rho=\frac{A-1}{2\sqrt{A}}$$

and the upper sign is for $E\leq E'$, and the lower sign is for $E\geq E'$.

If you are unsure about $kT$ units, convert the values to eV. What is your conclusion for scattering at thermal energies? How do heavy and light nuclides act differently as scatterers? Where does the kernel converge with increasing incoming neutron energy (at $100 kT$)?

**Note #1**: you can use the `scipy.special.erf` function.

**Note #2**: The function will probably break if you use too high ingoing neutron energies.

```python
from scipy.special import erf
def scatteringKernel(Ep,Es,A=1,T=300):
    """Function to calculate the scattering probability for thermal neutrons

    Parameters
    ----------
    Ep : float
        In-going neutron energy
    Es : array-like
        Out-going neutron energies
    A : int
        mass number (default=1)
    T : float
        temperature (default=300)

    """
    k=8.617333262145E-5 #Boltzmann constant eV/K
    eta=(A+1)/(2*np.sqrt(A))
    rho=(A-1)/(2*np.sqrt(A))
    alpha=((A-1)/(A+1))**2
    ps=[]
    for E in Es:
        if E<=Ep:
            p=#complete the line
            ps.append(p)
        else:
            p=#complete the line
            ps.append(p)
    return np.array(ps)*Ep*(1-alpha)
```

```python
%matplotlib inline
k=8.617333262145E-5 #eV/K
T=1200 #K

for Ai in [1,2,12]:
    plt.figure()
    for i in [1,4,10,25]:
        Ef=np.linspace(0,3*i*k*T,1000)
        Ei=i*k*T
        plt.plot(Ef/Ei,scatteringKernel(Ei,Ef,A=Ai,T=T),label='{}kT'.format(i))
    plt.title('A = '+str(Ai))
    plt.legend()
    plt.xlabel(r'$E_{final}/E_{initial}$')
    plt.ylabel(r'$\sigma(E_{final}\rightarrow E_{initial})E_{initial}(1-\alpha)/\sigma_s$')
    plt.show()
```


# Machine Learning and Statistics Assignment, 2020
***

## CONTENTS
1. [Task 1](#task1): calculate `sqrt2` to 100 decimal spaces without using imported modules
2. [Task 2](#task2): calculate Chi-square statistic using `scipy.stats` module
3. [Task 3](#task3): explain the difference between `STDDEV.S` and `STDDEV.P` in Excel, and show why `STDDEV.S` is a better estimator of population standard deviation when working with a sample
4. [Task 4](#task4): Apply `k`-means clustering to Fisher's Iris dataset

***

## Task 1
The first task is to write a Python function called `sqrt2` that returns the square root of 2 to 100 decimal spaces without making use of any imported modules.

There are several known methods [[1]](#refs). Here, we will look at two methods. Note that the methods described below only calculate the square root of whole numbers, not real numbers.

### Method 1: Newton's Method
This method is thought to date back to the Babylonians circa 1000 BC [[2]](#refs). It determines the square root of a number with increasing accuracy through each iteration. This is achieved in the following way [[3]](#refs):

1. **Make a reasonable guess for the square root**

   A reasonable way would be to identify the nearest whole integer that, when squared, comes closest to the value  under investigation. The following piece of `code` demonstrates:

```python
#Find largest integer n whose square is less than the number under investigation, in this case, 10

# Value of number for which square root is sought
num = 20

# Variable to contain answer
nearestValue = 0

# Loop through all options until exact integer found, or i*i > num
for i in range(num):
    if i*i < num:
        continue
    if i*i == num:
        nearestValue = i
        break
    if i*i > num:
        nearestValue = i-1
        break

# Prevent instances where nearestValue = 0
if nearestValue == 0:
    nearestValue = 1
```

```python
# Print largest integer
print("The nearest value which when squared is less than the number under investigation is: ", nearestValue)
```

    The nearest value which when squared is less than the number under investigation is:  4

2. **Increase the precision of the approximate guess**

   Add the above guessed root value to the original number under investigation divided by the guessed root value, and divide this final figure by 2:

$$x^{next} = \frac{(x^{current} + \frac{n}{x^{current}})}{2}$$

```python
# Increase the precision of the answer from the previous step
nextNum = (nearestValue + (num/nearestValue))/2

nextNum
```

    4.5

3. **Repeat until the desired precision is achieved**

   The `nextNum` value from step 2 gets reused in the calculation as the next best approximate guess until the desired precision of answer is achieved

```python
# Use the previous answer as the next value in the equation
nearestValue = nextNum

# Repeat the equation
nextNum = (nearestValue + (num/nearestValue))/2

nextNum
```

    4.472222222222222

These steps can be placed into a single Python function, here called `sqrtNewton` [[3]](#refs). In this example, the user can adapt the precision as needed.

```python
# Define variables
precision = 10 ** (-10) # Specify number of decimal spaces
num = 2 # Specify the number of which square root is sought

# Create function
def sqrtNewton(num):
    # Find nearest value as starting guesstimate
    nearestValue = 0

    for i in range(num):
        if i*i < num:
            continue
        if i*i == num:
            nearestValue = i
            break
        if i*i > num:
            nearestValue = i-1
            break

    # Prevent instances where nearestValue = 0
    if nearestValue == 0:
        nearestValue = 1

    # Loop through the equation until value with desired precision is identified
    while abs(num - (nearestValue * nearestValue)) > precision:
        nearestValue = (nearestValue + (num / nearestValue)) / 2

    # Return the value to the specified precision
    return nearestValue

#Print result
print("The square root of", num, "to the precision of", precision, "decimal spaces is", sqrtNewton(num))

```

    The square root of 2 to the precision of 1e-10 decimal spaces is 1.4142135623746899

The **problem** here is that the display does not show beyond the 16th decimal space. The Python `decimal` module allows specifying the number of decimal spaces to display [[4]](#refs); however, the instructions for this task clearly stated that no imported modules are to be used. This is where the second method comes in.

## Method 2: Long Division Algorithm
This method makes use of long division calculations by hand to determine each successive number in a square root answer [[5]](#refs). Because it allows determining each number in turn, it can be used to create a string of undetermined length without the limitation of decimal spaces display that is inherent in method 1. There are several steps involved:

1. **Seperate the digits into pairs**

This step is not needed, given that the issue here is the number of digits *after* the decimal. For whole numbers, the computer will be able to determine the square root without needing to break down into pairs.

2. **Find the largest integer**

As before, this can be done with a `for` loop.

```python
#Find largest integer n whose square is less than the number under investigation

# Value of number for which square root is sought
num = 1234

# Variable to contain answer
nearestValue = 0

# Loop through all options until exact integer found, or i*i > num
for i in range(num):
    if i*i < num:
        continue
    if i*i == num:
        nearestValue = i
        break
    if i*i > num:
        nearestValue = i-1
        break

# Prevent instances where nearestValue = 0
if nearestValue == 0:
    nearestValue = 1

# Print nearest integer
print("The nearest value which when squared is less than the number under investigation is: ", nearestValue)
```

    The nearest value which when squared is less than the number under investigation is:  35

This value now becomes the next number in the answer, and is added to the answer string

```python
#Create final answer string variable
ans = ''

# Add number to string
ans += str(nearestValue)
print(ans)
```

    35

3. **Subtract square of largest integer from current number**

The square of the value from the previous step is subtracted from the current number. The answer of this difference will now be added to the front of the next digit pair to create a new value to investigate.

```python
#Calculate square of nearestValue
sqrNearestValue = nearestValue * nearestValue
# print(sqrNearestValue)

#Subtract sqrNearestValue from num
nextNumber = num - sqrNearestValue
print(nextNumber)
```

    9

4. **Move to the next pair of values in the number under investigation**

Place the next two values next to the answer calculated in the step above. Where no values exist, use zeros instead. To achieve this, one can merely multiple the answer above by 100.

```python
#Multiply nextNumber by 100
nextNumber = nextNumber * 100
print('nextNumber: ', nextNumber)
```

    nextNumber:  900

Multiply the current answer by 2. This will be used in the next step

```python
#Multiply current ans by 2
currentAnswer = int(ans) * 2
print('currentAnswer: ', currentAnswer)
```

    currentAnswer:  70

5. **Find the right value match**

The next step is to use the value from the previous step, and identify what single integer, when placed at the end of that value and the whole value is multiplied by the same integer, is less than or equal to the nextNumber.

```python
#Identify the integer that completes the following equation and is less than the nextNumber: currentAnswery * y <= nextNumber
for i in range(1,11): #up to 11 to include the possibility of 9 being the integer
    testNumber = int(str(currentAnswer) + str(i)) * i
    if testNumber < nextNumber:
        continue
    if testNumber == nextNumber:
        nextAnswer = i
        testNumber = int(str(currentAnswer) + str(i)) * i
        # print('integer i is: ', i)
        break
    if testNumber > nextNumber:
        nextAnswer = i - 1
        testNumber = int(str(currentAnswer) + str(i-1)) * (i-1)
        #print('integer i is: ', i-1)
        break
print('testNumber: ', testNumber)
print('nextAnswer: ',nextAnswer)
```

    testNumber:  701
    nextAnswer:  1

This `nextAnswer` value then becomes the next value in the root answer.

```python
#Add nextAnswer to answer
ans += str(nextAnswer)
print('ans: ',ans)
```

    ans:  351

6. **Identify the next number**

Once again, subtract the total of the value identified in step 5 from the value on the left

```python
#Get the nextNumber
lastNumber = nextNumber
nextNumber = lastNumber - testNumber
print('lastNumber:', lastNumber)
print('testNumber:', testNumber)
print('nextNumber:', nextNumber)
```

    lastNumber: 900
    testNumber: 701
    nextNumber: 199

7. **Repeat steps 4 to 6 for the total number of decimal spaces required**

Repeat the steps until you have total number of decimal spaces required in the answer. This can be achieved by placing the above into a `while` loop as below.

```python
#FROM HERE TO END LOOPS UNTIL 100 DECIMAL SPACES#
while len(ans) < (100 + nearestValue):
#Multiply nextNumber by 100
    nextNumber = nextNumber * 100
    #print('nextNumber: ', nextNumber)

    #Multiply current ans by 2
    currentAnswer = int(ans) * 2
    #print('currentAnswer: ', currentAnswer)

    #Identify the integer that completes the following equation and is less than the nextNumber: currentAnswery * y <= nextNumber
    for i in range(1,11): #up to 11 to include the possibility of 9 being the integer
        testNumber = int(str(currentAnswer) + str(i)) * i
        if testNumber < nextNumber:
            continue
        if testNumber == nextNumber:
            nextAnswer = i
            testNumber = int(str(currentAnswer) + str(i)) * i
            #print('integer i is: ', i)
            break
        if testNumber > nextNumber:
            nextAnswer = i - 1
            testNumber = int(str(currentAnswer) + str(i-1)) * (i-1)
            #print('integer i is: ', i-1)
            break
    #print('testNumber: ', testNumber)
    #print('nextAnswer: ',nextAnswer)

    #Add nextAnswer to answer
    ans += str(nextAnswer)
    #print('ans: ',ans)

    #Get the nextNumber
    lastNumber = nextNumber
    nextNumber = lastNumber - testNumber
    #print('nextNumber: ', nextNumber)
    #print('line break - next iteration starts after this')
    #print()

print("ans: ", ans)
```

    ans:  351283361405005916058703116253563067645404854787765405690202683926394175654576700713118864847740795362187533615970865896869519707927067

The issue, however, is that the answer does not contain a decimal point. Inserting the decimal point can be done by determining the length of the string created by the `nearestValue`, and inserting a decimal point into the string after that value. For example, the square root of 16 is 4. The length of 4 as a string is 1, therefore the decimal will be placed after the first digit in the string. Likewise, the square root of 120 is 10.95. The `nearestValue` for calculating the square root of 120 is 10 ($10^2 = 100$). The length of 10 as a string is 2. Therefore, the decimal point will be placed after the second digit in the string. And so on.

```python
#Place decimal point at appropriate place
ans = ans[0:len(str(nearestValue))] + '.' + ans[len(str(nearestValue)):]
print("ans: ", ans)
```

    ans:  35.1283361405005916058703116253563067645404854787765405690202683926394175654576700713118864847740795362187533615970865896869519707927067

The final `function` for the long division algorithm which incorporates all of the steps is below. The commented out `print` statements are for debugging purposes.

```python
def sqrtLongDiv(num, decSpaces):
    largestInteger = 0
    ans = ''
    nextAnswer = 0

    #Find largest integer n whose square is less than num
    for i in range(num+1):
        if i*i < num:
            continue
        if i*i == num:
            largestInteger = i
            break
        if i*i > num:
            largestInteger = i-1
            break
    #print(largestInteger)

    # Prevent instances where largestInteger = 0
    if largestInteger == 0:
        largestInteger = 1

    #Add integer to answer
    ans += str(largestInteger)
    #print(ans)

    #Calculate square of largestInteger
    sqrLargestInteger = largestInteger*largestInteger
    #print(sqrLargestInteger)

    #Subtract sqrLargestInteger from num
    nextNumber = num - sqrLargestInteger
    #print(nextNumber)

    #FROM HERE TO END LOOPS UNTIL 100 DECIMAL SPACES#
    while len(ans) < (decSpaces + len(str(largestInteger))):
    #Multiply nextNumber by 100
        nextNumber = nextNumber * 100
        #print('nextNumber: ', nextNumber)

        #Multiply current ans by 2
        currentAnswer = int(ans) * 2
        #print('currentAnswer: ', currentAnswer)

        #Identify the integer that completes the following equation and is less than the nextNumber: currentAnswery * y <= nextNumber
        for i in range(1,11): #up to 11 to include the possibility of 9 being the integer
            testNumber = int(str(currentAnswer) + str(i)) * i
            if testNumber < nextNumber:
                continue
            if testNumber == nextNumber:
                nextAnswer = i
                testNumber = int(str(currentAnswer) + str(i)) * i
                #print('integer i is: ', i)
                break
            if testNumber > nextNumber:
                nextAnswer = i - 1
                testNumber = int(str(currentAnswer) + str(i-1)) * (i-1)
                #print('integer i is: ', i-1)
                break
        #print('testNumber: ', testNumber)
        #print('nextAnswer: ',nextAnswer)

        #Add nextAnswer to answer
        ans += str(nextAnswer)
        #print('ans: ',ans)

        #Get the nextNumber
        lastNumber = nextNumber
        nextNumber = lastNumber - testNumber
        #print('nextNumber: ', nextNumber)
        #print('line break - next iteration starts after this')

    #Place decimal point at appropriate place
    ans = ans[0:len(str(largestInteger))] + '.' + ans[len(str(largestInteger)):]

    return ans

#Run function with designated number
num = 123
decSpaces = 20
print("The long division square root of", num,"to", decSpaces,"decimal spaces is", sqrtLongDiv(num, decSpaces))
```

    The long division square root of 123 to 20 decimal spaces is 11.09053650640941716205

## Final output as required

The brief requested a single function `sqrt2` to 100 decimal spaces. Below is that function and output.

```python
def sqrt2():
    largestInteger = 0
    ans = ''
    nextAnswer = 0
    num = 2

    #Find largest integer n whose square is less than num
    for i in range(num+1):
        if i*i < num:
            continue
        if i*i == num:
            largestInteger = i
            break
        if i*i > num:
            largestInteger = i-1
            break

    #print(largestInteger)

    #Add integer to answer
    ans += str(largestInteger)
    #print(ans)

    #Calculate square of largestInteger
    sqrLargestInteger = largestInteger*largestInteger
    #print(sqrLargestInteger)

    #Subtract sqrLargestInteger from num
    nextNumber = num - sqrLargestInteger
    #print(nextNumber)

    #FROM HERE TO END LOOPS UNTIL 100 DECIMAL SPACES#
    while len(ans) < (100 + len(str(largestInteger))):
    #Multiply nextNumber by 100
        nextNumber = nextNumber * 100
        #print('nextNumber: ', nextNumber)

        #Multiply current ans by 2
        currentAnswer = int(ans) * 2
        #print('currentAnswer: ', currentAnswer)

        #Identify the integer that completes the following equation and is less than the nextNumber: currentAnswery * y <= nextNumber
        for i in range(1,11): #up to 11 to include the possibility of 9 being the integer
            testNumber = int(str(currentAnswer) + str(i)) * i
            if testNumber < nextNumber:
                continue
            if testNumber == nextNumber:
                nextAnswer = i
                testNumber = int(str(currentAnswer) + str(i)) * i
                #print('integer i is: ', i)
                break
            if testNumber > nextNumber:
                nextAnswer = i - 1
                testNumber = int(str(currentAnswer) + str(i-1)) * (i-1)
                #print('integer i is: ', i-1)
                break
        #print('testNumber: ', testNumber)
        #print('nextAnswer: ',nextAnswer)

        #Add nextAnswer to answer
        ans += str(nextAnswer)
        #print('ans: ',ans)

        #Get the nextNumber
        lastNumber = nextNumber
        nextNumber = lastNumber - testNumber
        #print('nextNumber: ', nextNumber)
        #print('line break - next iteration starts after this')

    #Place decimal point at appropriate place
    ans = ans[0:len(str(largestInteger))] + '.' + ans[len(str(largestInteger)):]

    return ans

# Print out answer
print("The square root of 2 to 100 decimal spaces is:")
print(sqrt2())
```

    The square root of 2 to 100 decimal spaces is:
    1.4142135623730950488016887242096980785696718753769480731766797379907324784621070388503875343276415727

## Compare answer

The algorithm veracity is tested against a string of the square root of 2 calculated to 1 million digits [[6]](#refs). Note how there is no multi-line wrapping of the imported string - this will give you an error, as it reads in a `\n` character.

```python
#Get string from website extract
nasaImport = '1.41421356237309504880168872420969807856967187537694807317667973799073247846210703885038753432764157273501384623091229702492483605585073721264412149709993583141322266592750559275579995050115278206057147010955997160597027453459686201472851'

#Create string of 100 decimal spaces (up to 102 values to include first digit and decimal point)
sqrt2Nasa = nasaImport[0:102]

#print(len(sqrt2()))
#print(len(sqrt2Nasa))

#Compare Nasa string with algorithm
print("The function sqrt2 is equal to the first 100 decimal spaces of the square root of 2 as calculated by Nasa:", sqrt2Nasa == sqrt2())

```

    The function sqrt2 is equal to the first 100 decimal spaces of the square root of 2 as calculated by Nasa: True

***

## Task 2
The second task is to calculate the Chi-square test statistic of a table in Wikipedia [[7]](#refs) using `scipy.stats`, and confirm that the value is 24.6. In addition, the task asks to calculate the associated *p* value.

### Chi-square test
The Chi square test is a statistical hypothesis test used to test for a relationship between categorical variables. It assumes that the variables being investigated are categorical and independent [[8]](#refs). It determines whether there is a relationship by comparing the *expected* values to the *observed* values for each category [[9]](#refs). As per any statistical test, it assumes a null hypothesis of no relationship; if the Chi square statistic is significantly different from a  critical value (determined by the degrees of freedom), then the null hypothesis is rejected.

### Chi-square calculation
The Chi-square statistic is the sum of the observed value minus the expected value squared, divided by the expected value, for each category. This is presented as follows [[10]](#refs):

\begin{equation}
\chi^2=\Sigma\frac{(O-E)^2}{E} \\
\text{where O is the actual value and E is the expected value.}
\end{equation}

The expected value is determined by multiplying the row total by the column total, and dividing the answer by the grand total, as follows:

$$expected\ value\ = \frac{row\ total\ x\ column\ total}{grand\ total}$$

### Table of observed values
The table from Wikipedia is presented below:

```python
#Import necessary modules
import pandas as pd

#Generate table - adapted from: https://towardsdatascience.com/gentle-introduction-to-chi-square-test-for-independence-7182a7414a95
cells = pd.DataFrame([[90, 60, 104, 95], #Row 1, White collar
         [30, 50, 51, 20],  #Row 2, Blue collar
         [30, 40, 45, 35]   #Row 3, No collar
        ],
index = ["White Collar", "Blue Collar", "No Collar"],
columns = ["A", "B", "C", "D"])

#Add in columns totals to a new copy of cells, as chi2_contingency needs internal cell values only
table = cells.copy()
table.loc['Total',:] = table.sum(axis=0)

#Add in row totals
table.loc[:,'Total'] = table.sum(axis=1)

#Tidy up display by converting all floating points to integers
table = table.astype(int)

#Display table
table
```

      A
      B
      C
      D
      Total

      White Collar
      90
      60
      104
      95
      349

      Blue Collar
      30
      50
      51
      20
      151

      No Collar
      30
      40
      45
      35
      150

      Total
      150
      150
      200
      150
      650

### Determination of Chi square statistic and associated values
The method to use is the `stats.scipy.chi2_contingency`, as the table is larger than a 2x2, and this statistical test is the hypothesis test of independence of the observed frequencies in the contingency table.

It takes an array-like structure, and the method returns the Chi square statistic, the *p* value, Degrees of freedom, and the expected frequencies, in that order, based on marginal values calculated by the method from the observed frequencies.

```python
#Import necessary modules
from scipy.stats import chi2_contingency
import numpy as np

#Turn table into a numpy array
obs = np.array(cells)
#print(obs)

#Run chi2 contingency method
chiStat, pVal, dof, expF = chi2_contingency(obs)

#Print output
print("The Chi square statistic is:", round(chiStat,1))
print("The p value is:", round(pVal,4))
print("The Degrees of Freedom are:", dof)
print("The expected frequencies are:\n", (pd.DataFrame(expF,index = ["White Collar", "Blue Collar", "No Collar"],
columns = ["A", "B", "C", "D"])).round(2))

```

    The Chi square statistic is: 24.6
    The p value is: 0.0004
    The Degrees of Freedom are: 6
    The expected frequencies are:
                       A      B       C      D
    White Collar  80.54  80.54  107.38  80.54
    Blue Collar   34.85  34.85   46.46  34.85
    No Collar     34.62  34.62   46.15  34.62

## Discussion
The Chi square statistic of 24.6 calculated by the `scipy.stats.ch2_contingency` method aligns with that in the Wikipedia article. When one compares the *expected* values in the above output to the *observed* values in the original table, it is evident that in some categories, there are large differences between these two values. This is reflected in the significant *p* value of 0.0004, reported as p < 0.05.

We therefore reject the null hypothesis that there is no difference between a person's neighbourhood of residence based on their occupational class, and state that there is a statistically significant difference between the distribution of persons across each neighbourhood based on their occupational class.

***

## Task 3
The third task relates to one measure of dispersion around the mean, namely standard deviation. The task is to examine the difference between `STDDEV.S` and `STDDEV.P` in Excel, and then use `numpy` to explain why `STDDEV.S` is a better estimate for the standard deviation of a population when performed on a sample.

### What is a standard deviation?
The standard deviation (SD) gives a measure of dispersement or spread of one's data around the mean [[11]](#refs). A small SD indicates that the data is gathered close to the mean (small spread), while a large SD indicates that data is spread out quite widely around the mean [[12]](#refs). The below image compares two populations with the same mean of 0, but with a SD of 1 and 3 respectively. Notice how the plot with SD of 3 is more dispersed around the mean of 0.

```python
# Import necessary modules
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

# Plot of 1,000 data points from a normal distribution with mean of 0 and SD of 1 - adapted from: https://stackoverflow.com/a/51758496
value1 = np.around(np.random.normal(loc=0,scale=1,size=1000), 2)
sns.distplot(value1, label="Mean 0, SD 1")

# Plot of 1,000 data points from a normal distribution with a mean of 0 and SD of 3
value2 = np.around(np.random.normal(loc=0, scale = 3, size = 1000),2)
sns.distplot(value2, label="Mean 0, SD 3")

# Show legend
plt.legend()

```

There are some interesting conclusions that can be gleaned from a SD when the data is distributed normally around the mean as in the above example. Roughly 68% of all data points will lie within ±1 SD of the mean, ~95% within ±2 SD of the mean, and ~99% within ±3 SD of the mean [[13]](#refs). One can evaluate that as follows:

```python
'''Proportion of values within 1SD of value1 (from -1 to +1 around mean of 0)'''
# Sort array
value1.sort()

# Extract all values between -1 and +1
prop_sd1_value1 = []
for value in value1:
    if value < 1.0 and value > -1.0:
        prop_sd1_value1.append(value)

# Determine proportion of values
prop = len(prop_sd1_value1)/1000 * 100

# Print result
print(f"The proportion of values between ±1 SD in a normal distribution with a mean of 0 and SD of 1 is {prop:.2f}%")
```

    The proportion of values between ±1 SD in a normal distribution with a mean of 0 and SD of 1 is 69.10%

```python
'''Proportion of values within 2SD of value1 (from -2 to +2 around mean of 0)'''
# Extract all values between -1 and +1
prop_sd2_value1 = []
for value in value1:
    if value < 2.0 and value > -2.0:
        prop_sd2_value1.append(value)

# Determine proportion of values
prop = len(prop_sd2_value1)/1000 * 100

# Print result
print(f"The proportion of values between ±2 SD in a normal distribution with a mean of 0 and SD of 1 is {prop:.2f}%")
```

    The proportion of values between ±2 SD in a normal distribution with a mean of 0 and SD of 1 is 95.20%

```python
'''Proportion of values within 3SD of value1 (from -3 to +3 around mean of 0)'''
# Extract all values between -1 and +1
prop_sd3_value1 = []
for value in value1:
    if value < 3.0 and value > -3.0:
        prop_sd3_value1.append(value)

# Determine proportion of values
prop = len(prop_sd3_value1)/1000 * 100

# Print result
print(f"The proportion of values between ±3 SD in a normal distribution with a mean of 0 and SD of 1 is {prop:.2f}%")
```

    The proportion of values between ±3 SD in a normal distribution with a mean of 0 and SD of 1 is 99.70%

Finally, the SD is used in calculating the confidence interval (CI). A CI represents a range in which some value that we are interested in determining can be found [[14]](#refs). Oftentimes, a 95% CI interval is presented. As has been shown above, 2SD around a mean usually contain ~95% of all values in the sample or population (1.96SD is exactly 95%). It is this characteristic of the SD that is useful for determining a range in which a value can be found.

### Calculating standard deviation
The equation for calculating the SD of a **population** is:
$$\sigma = \sqrt{\frac{\Sigma(x_i - \mu)}{N}}$$ where $\sigma$ represents the population SD, $x_i$ is each value in the population, $\mu$ is the population mean, and $N$ is the size of the population.

The SD represents the spread of values around the mean, or put another way, how far a value is from the mean. This 'distance' can be represented by the difference between the value and a mean, represented as $x - \mu$, where $x$ is the value, and $\mu$ is the mean of all the data points. The mean is simply calculated by adding all the values in the data set together, and dividing by the number of values present. In other words, $\frac{\Sigma{x_i}}{N}$, where $x_i$ are each value in the data set, and $N$ is the total number of observations in the data set.

If we merely added the difference for each $x - \mu$, we would get a result of zero, because a value can be smaller or larger than the mean. For example, in an array of `[3,4,5,6,7]`, the mean is `5`. The difference between each value and the mean ($x - \mu$) in turn is `[-2, -1, 0, 1, 2]`, which when added together equals zero. To prevent this, the value for each difference is squared: $\Sigma(x - \mu)^2$. This gives a positive array for each data point: `[4, 1, 0, 1, 4]`. The sum of this is now 10. This value is divided by the number of observations in the population, $N$, before the square root is applied to arrive at the population SD, $\sigma$.

More frequently, a sample of a population is examined. The equation for calculating the SD of a **sample** is:

$$s = \sqrt{\frac{\Sigma(x_i - \overline x)}{n - 1}}$$

where $s$ represents the sample SD, $x_i$ is each value in the sample, $\overline x$ is the sample mean, and $n$ is the sample size.

The main difference between these two equations is in the denominator: population SD makes use of the *whole* population size, while sample SD makes use of the sample size minus one. It is this difference that is conveyed in the different Excel functions: `STDDEV.P` is used to calculate the SD of a *whole* population (e.g. all children in a class of interest), while `STDDEV.S` is used to calculate the SD of a *sample* of the population (e.g. a handful of children from the class of interest) [[15]](#refs). Because it is often difficult to measure a complete population (either because the population is too big to measure in it's entirety, or the true size of the population is difficult to define), a sample is used. In the next section, we will use `numpy` to look at why using the sample equation (denominator $n-1$) is a better estimator of the SD than the population equation (denominator $N$).

### Sample Standard Deviation Calculation example
Let's create a dataset of random numbers on a normal distribution between 0 and 85 representing the age of a population. Let's assume that this data set represents the WHOLE population of interest.

```python
#Create array
pop = np.random.randint(0, 86, 1000)
pop.sort()
#print(pop)
print(f"Values range from a minimum of {pop[0]} to {pop[-1]}, with a mean of {sum(pop)/len(pop):.1f}, and a population standard deviation of {np.std(pop):.1f}.")
```

    Values range from a minimum of 0 to 85, with a mean of 41.4, and a population standard deviation of 24.9.

Let's now assume that we take a sample from this population, because we cannot access the whole population to measure age. Ideally, we want to get as close to the actual SD of the population as possible using the sample. We randomly sample 10 observations from the population. We repeat this sampling 50 times, and determine the average *population* SD of all these samples.

```python
#Create list to store each result of population SD
avg_pop_sd = []

#Create sample and run 50 times
for i in range(50):
    sample = np.random.choice(pop, 10, replace=False)
    avg_pop_sd.append(round(np.std(sample),1))

#print(avg_sample_sd)
print(f"The average SD using a population calculation for a random sample of size 10 repeated 50 times from the same population is {sum(avg_pop_sd)/len(avg_pop_sd):.1f}.")
print(f"This differs from the true population SD by {abs(np.std(pop) - (sum(avg_pop_sd)/len(avg_pop_sd))):.1f}.")
```

    The average SD using a population calculation for a random sample of size 10 repeated 50 times from the same population is 23.7.
    This differs from the true population SD by 1.2.

In the next step, we repeat this process, but apply the *sample* SD calculation of $n-1$.

```python
#Create list to store each result of population SD
avg_sample_sd = []

#Create sample and run 50 times
for i in range(50):
    sample = np.random.choice(pop, 10, replace=False)
    avg_sample_sd.append(round(np.std(sample, ddof=1),1))

#print(avg_sample_sd)
print(f"The average SD using a sample calculation for a random sample of size 10 repeated 50 times from the same population is {sum(avg_sample_sd)/len(avg_sample_sd):.1f}.")
print(f"This differs from the true population SD by {abs(np.std(pop) - (sum(avg_sample_sd)/len(avg_sample_sd))):.1f}.")
```

    The average SD using a sample calculation for a random sample of size 10 repeated 50 times from the same population is 25.2.
    This differs from the true population SD by 0.3.

### Discussion
The above indicates that the use of `STDDEV.S` offers a better approximation of the population SD than `STDDEV.P` when studying a sample of a population. When the whole population is known and measured, `STDDEV.P` should be used, as this calculates the true SD.

***

## Task 4
The final task requires using `scikit-learn` to apply k-means clustering to Fisher's Iris dataset.

### K-means clustering
K-means clustering is a method of unsupervised machine learning that groups together similar points in a dataset by identifying clusters of data objects in a dataset [[16]](#refs). This is achieved by first telling the algorithm how many clusters (k) the data should be seperated into. The algorithm then selects k number of random points in the dataset as the centre of the starting clusters. These centres are referred to as the centroid. Each datapoint is then assigned to the nearest centroid. The algorithm then calculates the mean of all the datapoints, and repositions the centroid to this calculated value. After the new centroid is calculated, all the datapoints are again reassigned to the new nearest centroid and hence, new cluster. The process repeats until no datapoints change to new clusters, i.e. the centroid does not change [[17]](#refs).

Becuase the initial starting centroids are chosen at random, k-means will provide different results each time [[16]](#refs). For this reason, the algorithm is usually run several times, with cluster assignments leading to the lowest sum of the squared error (SSE: the sum of the squared Euclidean distance of each point to its closest centroid) been selected as the final result [[17]](#refs).

### Determining starting cluster size *k*
Because k-means is a method of unsupervised machine learning, we don't usually know how many clusters, or in this case Iris species, to expect (even though we know it's three). There are two ways of determining an appropriate cluster size [[17]](#refs). Here we'll look at one, the **elbow method**.

#### The elbow method
The elbow method runs through repeated k-means calculations with increasing number of k, recording the SSE for each iteration. The k is plotted against the SEE for each iteration. Because the 'best' cluster distribution is usually that with the smallest SSE, the the number k with the smallest SSE would be preferred. However, there is a trade-off between k and SSE - the smaller the cluster, the less 'meaningful' the cluster becomes. So a 'sweet spot' is usually sought, right where the SSE curve starts to bend like an elbow. Hence, the elbow method.

In the next section, we'll look at the iris dataset and perform a k-means clustering.

### Import the Iris dataset
`scikit-learn` contains a copy of the Iris dataset. This can be imported from the `scikit-learn` package.

```python
#Import necessary modules
import numpy as np
import sklearn.cluster as skcl
import matplotlib.pyplot as plt

#Import datasets (https://scikit-learn.org/stable/tutorial/basic/tutorial.html#loading-example-dataset)
from sklearn import datasets

#Load data
dataset = datasets.load_iris()
irisMeasurements = dataset.data
irisMeasurements
```

    array([[5.1, 3.5, 1.4, 0.2],
           [4.9, 3. , 1.4, 0.2],
           [4.7, 3.2, 1.3, 0.2],
           [4.6, 3.1, 1.5, 0.2],
           [5. , 3.6, 1.4, 0.2],
           [5.4, 3.9, 1.7, 0.4],
           [4.6, 3.4, 1.4, 0.3],
           [5. , 3.4, 1.5, 0.2],
           [4.4, 2.9, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.1],
           [5.4, 3.7, 1.5, 0.2],
           [4.8, 3.4, 1.6, 0.2],
           [4.8, 3. , 1.4, 0.1],
           [4.3, 3. , 1.1, 0.1],
           [5.8, 4. , 1.2, 0.2],
           [5.7, 4.4, 1.5, 0.4],
           [5.4, 3.9, 1.3, 0.4],
           [5.1, 3.5, 1.4, 0.3],
           [5.7, 3.8, 1.7, 0.3],
           [5.1, 3.8, 1.5, 0.3],
           [5.4, 3.4, 1.7, 0.2],
           [5.1, 3.7, 1.5, 0.4],
           [4.6, 3.6, 1. , 0.2],
           [5.1, 3.3, 1.7, 0.5],
           [4.8, 3.4, 1.9, 0.2],
           [5. , 3. , 1.6, 0.2],
           [5. , 3.4, 1.6, 0.4],
           [5.2, 3.5, 1.5, 0.2],
           [5.2, 3.4, 1.4, 0.2],
           [4.7, 3.2, 1.6, 0.2],
           [4.8, 3.1, 1.6, 0.2],
           [5.4, 3.4, 1.5, 0.4],
           [5.2, 4.1, 1.5, 0.1],
           [5.5, 4.2, 1.4, 0.2],
           [4.9, 3.1, 1.5, 0.2],
           [5. , 3.2, 1.2, 0.2],
           [5.5, 3.5, 1.3, 0.2],
           [4.9, 3.6, 1.4, 0.1],
           [4.4, 3. , 1.3, 0.2],
           [5.1, 3.4, 1.5, 0.2],
           [5. , 3.5, 1.3, 0.3],
           [4.5, 2.3, 1.3, 0.3],
           [4.4, 3.2, 1.3, 0.2],
           [5. , 3.5, 1.6, 0.6],
           [5.1, 3.8, 1.9, 0.4],
           [4.8, 3. , 1.4, 0.3],
           [5.1, 3.8, 1.6, 0.2],
           [4.6, 3.2, 1.4, 0.2],
           [5.3, 3.7, 1.5, 0.2],
           [5. , 3.3, 1.4, 0.2],
           [7. , 3.2, 4.7, 1.4],
           [6.4, 3.2, 4.5, 1.5],
           [6.9, 3.1, 4.9, 1.5],
           [5.5, 2.3, 4. , 1.3],
           [6.5, 2.8, 4.6, 1.5],
           [5.7, 2.8, 4.5, 1.3],
           [6.3, 3.3, 4.7, 1.6],
           [4.9, 2.4, 3.3, 1. ],
           [6.6, 2.9, 4.6, 1.3],
           [5.2, 2.7, 3.9, 1.4],
           [5. , 2. , 3.5, 1. ],
           [5.9, 3. , 4.2, 1.5],
           [6. , 2.2, 4. , 1. ],
           [6.1, 2.9, 4.7, 1.4],
           [5.6, 2.9, 3.6, 1.3],
           [6.7, 3.1, 4.4, 1.4],
           [5.6, 3. , 4.5, 1.5],
           [5.8, 2.7, 4.1, 1. ],
           [6.2, 2.2, 4.5, 1.5],
           [5.6, 2.5, 3.9, 1.1],
           [5.9, 3.2, 4.8, 1.8],
           [6.1, 2.8, 4. , 1.3],
           [6.3, 2.5, 4.9, 1.5],
           [6.1, 2.8, 4.7, 1.2],
           [6.4, 2.9, 4.3, 1.3],
           [6.6, 3. , 4.4, 1.4],
           [6.8, 2.8, 4.8, 1.4],
           [6.7, 3. , 5. , 1.7],
           [6. , 2.9, 4.5, 1.5],
           [5.7, 2.6, 3.5, 1. ],
           [5.5, 2.4, 3.8, 1.1],
           [5.5, 2.4, 3.7, 1. ],
           [5.8, 2.7, 3.9, 1.2],
           [6. , 2.7, 5.1, 1.6],
           [5.4, 3. , 4.5, 1.5],
           [6. , 3.4, 4.5, 1.6],
           [6.7, 3.1, 4.7, 1.5],
           [6.3, 2.3, 4.4, 1.3],
           [5.6, 3. , 4.1, 1.3],
           [5.5, 2.5, 4. , 1.3],
           [5.5, 2.6, 4.4, 1.2],
           [6.1, 3. , 4.6, 1.4],
           [5.8, 2.6, 4. , 1.2],
           [5. , 2.3, 3.3, 1. ],
           [5.6, 2.7, 4.2, 1.3],
           [5.7, 3. , 4.2, 1.2],
           [5.7, 2.9, 4.2, 1.3],
           [6.2, 2.9, 4.3, 1.3],
           [5.1, 2.5, 3. , 1.1],
           [5.7, 2.8, 4.1, 1.3],
           [6.3, 3.3, 6. , 2.5],
           [5.8, 2.7, 5.1, 1.9],
           [7.1, 3. , 5.9, 2.1],
           [6.3, 2.9, 5.6, 1.8],
           [6.5, 3. , 5.8, 2.2],
           [7.6, 3. , 6.6, 2.1],
           [4.9, 2.5, 4.5, 1.7],
           [7.3, 2.9, 6.3, 1.8],
           [6.7, 2.5, 5.8, 1.8],
           [7.2, 3.6, 6.1, 2.5],
           [6.5, 3.2, 5.1, 2. ],
           [6.4, 2.7, 5.3, 1.9],
           [6.8, 3. , 5.5, 2.1],
           [5.7, 2.5, 5. , 2. ],
           [5.8, 2.8, 5.1, 2.4],
           [6.4, 3.2, 5.3, 2.3],
           [6.5, 3. , 5.5, 1.8],
           [7.7, 3.8, 6.7, 2.2],
           [7.7, 2.6, 6.9, 2.3],
           [6. , 2.2, 5. , 1.5],
           [6.9, 3.2, 5.7, 2.3],
           [5.6, 2.8, 4.9, 2. ],
           [7.7, 2.8, 6.7, 2. ],
           [6.3, 2.7, 4.9, 1.8],
           [6.7, 3.3, 5.7, 2.1],
           [7.2, 3.2, 6. , 1.8],
           [6.2, 2.8, 4.8, 1.8],
           [6.1, 3. , 4.9, 1.8],
           [6.4, 2.8, 5.6, 2.1],
           [7.2, 3. , 5.8, 1.6],
           [7.4, 2.8, 6.1, 1.9],
           [7.9, 3.8, 6.4, 2. ],
           [6.4, 2.8, 5.6, 2.2],
           [6.3, 2.8, 5.1, 1.5],
           [6.1, 2.6, 5.6, 1.4],
           [7.7, 3. , 6.1, 2.3],
           [6.3, 3.4, 5.6, 2.4],
           [6.4, 3.1, 5.5, 1.8],
           [6. , 3. , 4.8, 1.8],
           [6.9, 3.1, 5.4, 2.1],
           [6.7, 3.1, 5.6, 2.4],
           [6.9, 3.1, 5.1, 2.3],
           [5.8, 2.7, 5.1, 1.9],
           [6.8, 3.2, 5.9, 2.3],
           [6.7, 3.3, 5.7, 2.5],
           [6.7, 3. , 5.2, 2.3],
           [6.3, 2.5, 5. , 1.9],
           [6.5, 3. , 5.2, 2. ],
           [6.2, 3.4, 5.4, 2.3],
           [5.9, 3. , 5.1, 1.8]])

There are 150 entries in the dataset. Each column represents, in turn: Sepal Length (sl), Sepal Width (sw), Petal Length (pl), Petal Width (pw) [[18]](#refs). In addition, the dataset has classified each observation according to the type of Iris: Iris-Setosa, Iris-Versicolour, Iris-Virginica. These are recorded as 0, 1 or 2 respectively.

```python
#Display classification of each datapoint
dataset['target']
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])

### Determine an appropriate number for k
We are assuming that we don't know there are 3 iris species in our data. So let's apply the elbow method and see how many clusters are 'advised'.

```python
#Applying the elbow method [adapted from https://predictivehacks.com/k-means-elbow-method-code-for-python/]
#Create blank list to store SSE
sse = []

#Run through 7 clusters determining SSE for each
for k in range (1,8):
    kmeans = skcl.KMeans(n_clusters = k)
    kmeans.fit(irisMeasurements)
    sse.append(kmeans.inertia_)

#Display the results of SSE against k
plt.plot(range(1,8), sse)
plt.xlabel('Number of clusters, k')
plt.ylabel('SSE')
plt.title('The Elbow Method showing the optimal k')
plt.show()
```

As mentioned previously, we are looking for that sweet spot trade-off between cluster size and smallest SSE. From the above diagram, it would appear that there isn't much gain to be had above 3 clusters. So we will define 3 clusters for our kmeans clustering.

### Perform kmeans clustering on dataset

```python
#Instantiate algorithm for dataset, using a deterministic initialization of centroids
kmIris = skcl.KMeans(n_clusters=3, random_state=0)

#Fit data
kmIris.fit(irisMeasurements)

#Check labels
kmIris.labels_
```

    array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 2, 2, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 0, 2, 0, 0, 0, 0, 2, 0, 0, 0,
           0, 0, 0, 2, 2, 0, 0, 0, 0, 2, 0, 2, 0, 2, 0, 0, 2, 2, 0, 0, 0, 0,
           0, 2, 0, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 2], dtype=int32)

We can see that there is some break in the continuity of clustering of the datapoints; numbers should run consecutively, i.e. 1,1,1,...2,2,2,...0,0,0. There should be no mixing of values. We can determine the size of each  cluster identified through kmeans to see the accuracy of the clustering, as we know that the target dataset is clustered contiguously in groups of 50 for each Iris species.

```python
#Count number of items in each cluster [taken from https://stackoverflow.com/a/5829377]
from collections import Counter

#Divide cluster result into clusters of 50 contiguous values
#Convert to list to allow slicing
calculatedLabels = list(kmIris.labels_)

#Create clusters of 50
cluster0 = calculatedLabels[0:50]
cluster1 = calculatedLabels[50:100]
cluster2 = calculatedLabels[100:]

#Count values in each cluster and assign clusters to variables
dict0 = Counter(cluster0)
dict1 = Counter(cluster1)
dict2 = Counter(cluster2)

#Determine accuracy in each cluster
print(f"In the first cluster of 50, there are {dict0[1]} datapoints grouped to Iris Setosa (accuracy of {dict0[1]/50 * 100}%)")
print(f"In the second cluster of 50, there are {dict1[2]} datapoints grouped to Iris Versicolor (accuracy of {dict1[2]/50 * 100}%)")
print(f"In the third cluster of 50, there are {dict2[0]} datapoints grouped to Iris Virginica (accuracy of {dict2[0]/50 * 100}%)")

```

    In the first cluster of 50, there are 50 datapoints grouped to Iris Setosa (accuracy of 100.0%)
    In the second cluster of 50, there are 48 datapoints grouped to Iris Versicolor (accuracy of 96.0%)
    In the third cluster of 50, there are 36 datapoints grouped to Iris Virginica (accuracy of 72.0%)

The kmeans clustering was only able to identify all objects from the Iris Setosa species correctly with 100% accuracy. It was only able to identify Iris Versicolor with 96% accuracy, and Iris Virginica with 72% accuracy. We can represent this visually in the following plots.

```python
#Tidy up calculated array so that same colors are used between both plots
cleanArray = []
dirty = kmIris.labels_
for i in dirty:
    if i == 1:
        cleanArray.append(0)
    if i == 2:
        cleanArray.append(1)
    if i == 0:
        cleanArray.append(2)
```

```python
#Visualise actual vs calculated clusters [adapted from https://medium.com/@belen.sanchez27/predicting-iris-flower-species-with-k-means-clustering-in-python-f6e46806aaee]

#Plot side by side
fig, axes = plt.subplots(1, 2, figsize=(16,8))

#First plot of actual clusters
axes[0].scatter(irisMeasurements[:, 0], irisMeasurements[:, 1], c=dataset['target'])
axes[0].set_xlabel('Sepal length')
axes[0].set_ylabel('Sepal width')
axes[0].set_title('Actual clustering')

#Second plot of calculated clusters
axes[1].scatter(irisMeasurements[:, 0], irisMeasurements[:, 1], c=cleanArray)
axes[1].set_xlabel('Sepal length')
axes[1].set_ylabel('Sepal width')
axes[1].set_title('Calculated clustering')
```

### Discussion
Kmeans clustering is a form of unsupervised machine learning. While it is useful in identifying common groupings in datasets of various shapes and sizes, it is limited by the need to identify starting number of clusters, and the fact that outliers can be excluded from the correct grouping, as seen above [[20]](#refs).

***

### References
[1] Wikipedia. Methods of computing square roots \[Internet\]. Wikimedia Foundation; 2020 \[Last updated 2020 October 30\]. Available from https://en.wikipedia.org/wiki/Methods_of_computing_square_roots

[2] Johnson, SG. Square Roots via Newton's Method \[published lecture notes\] MIT; 2015 February 4 \[Cited 2020 October 14\]. Available at: https://math.mit.edu/~stevenj/18.335/newton-sqrt.pdf

[3] Regmi, S. Calculating the Square Root of a Number using the  Newton-Raphson Method \[A How To Guide\]. \[Internet\] Hackernoon; 2020 \[Last updated 2020 January 18; cited 2020 October 14\] Available at https://hackernoon.com/calculating-the-square-root-of-a-number-using-the-newton-raphson-method-a-how-to-guide-yr4e32zo

[4] Python Software Foundation. decimal — Decimal fixed point and floating point arithmetic \[Internet\]. Python Software Foundation; 2020. \[Last updated 2020 October 31; cited 2020 October 10\]. Available at https://docs.python.org/3.8/library/decimal.html

[5] Arobelidze, A. How to find the square root of a number and calculate it by hand \[Internet\]. USA: FreeCodeCamp; 2020 \[Last updated 2020 February 6; cited 2020 October 16\] Available at https://www.freecodecamp.org/news/find-square-root-of-number-calculate-by-hand/

[6] Nemirof, R and Bonnell, J. The Square Root of Two to 1 Million Digits \[Internet\].  Available from https://apod.nasa.gov/htmltest/gifcity/sqrt2.1mil

[7] Wikipedia. Chi-squared test \[Internet\]. Wikimedia Foundation; 2020 \[Last updated 2020 October 11; cited 2020 November 14\]. Available from https://en.wikipedia.org/wiki/Chi-squared_test

[8] Laerd Statistics. Chi-Square Test for Association using SPSS Statistics \[Internet\]. UK: Lund Lund Research Ltd. \[Last updated 2018; cited 2020 November 14\] Available from https://statistics.laerd.com/spss-tutorials/chi-square-test-for-association-using-spss-statistics.php

[9] Glen, S. Chi-Square Statistic: How to Calculate It / Distribution \[Internet\]. StatisticsHowTo.com; 2020 \[Last updated 2020; cited 2020 November 14\] Available from https://www.statisticshowto.com/probability-and-statistics/chi-square/

[10] Okada, S. Gentle Introduction to Chi-Square Test for Independence: Beginners guide to Chi-square using Jupyter Notebook \[Internet\]. Medium - towards data science; 2020. \[Last updated 2020 January; cited 2020 November 14\] Available from https://towardsdatascience.com/gentle-introduction-to-chi-square-test-for-independence-7182a7414a95#92ef

[11] Glen, S. Standard Deviation: Simple Definition, Step by Step Video \[Internet\]. StatisticsHowTo.com; 2020 \[Last updated 2020; cited 2020 November 24\] Available from https://www.statisticshowto.com/probability-and-statistics/standard-deviation/#SDD

[12] Wikipedia. Standard deviation \[Internet\]. Wikimedia Foundation; 2020 \[Last updated 10 December 2020; cited 2020 November 24\] Available from https://en.wikipedia.org/wiki/Standard_deviation

[13] Hayes, A. Empirical Rule. \[Internet\] Investopedia; nd. \[Last updated 2020 July 27; cited 2020 November 24\] Available from https://www.investopedia.com/terms/e/empirical-rule.asp

[14] Maths is Fun. Confidence Intervals \[Internet\]. MathsisFun.com; 2017 \[Last updated 2017; cited 2020 November 24\] Available from https://www.mathsisfun.com/data/confidence-interval.html

[15] Microsoft. STDEV.S function \[Internet\]. USA: Microsoft; 2020 \[Last updated 2020; cited 2020 November 24\] Available from https://support.microsoft.com/en-us/office/stdev-s-function-7d69cf97-0c1f-4acf-be27-f3e83904cc23

[16] Munnelly, G. K-Means \[Internet\]. Trinity College Dublin; 2017 \[Last updated 2017; cited 2020 December 10\] Available from https://www.scss.tcd.ie/~munnellg/projects/kmeans.html

[17] Arvai, K. K-Means Clustering in Python: A Practical Guide \[Internet\]. Real Python; 2020 \[Last updated 2020 July 20; cited 2020 December 10\] Available from https://realpython.com/k-means-clustering-python/

[18] scikit-learn developers. The Iris Dataset \[Internet\]. scikit-learn; 2020 \[Last updated 2020; cited 2020 December 10\] Available from https://scikit-learn.org/stable/auto_examples/datasets/plot_iris_dataset.html

[19] Garbade, MJ. Understanding K-means Clustering in Machine Learning \[Internet\]. Medium - towards data science; 2020 \[Last updated 2018 September 12; cited 2020 December 10\] Available from https://towardsdatascience.com/understanding-k-means-clustering-in-machine-learning-6a6e67336aa1

[20] Google. k-Means Advantages and Disadvantages \[Internet\]. Google Developers; 2020 \[Last updated 2020 February 10; cited 2020 December 11\] Available from https://developers.google.com/machine-learning/clustering/algorithm/advantages-disadvantages


```python
import numpy as np
from sympy import *
init_printing(use_latex='mathjax')
```

```python
x1, x2, t = symbols('x1 x2 t')

F = Matrix([
    x1 ** 2 * x2 ** 2 + x1 * x2,
    1 - t ** 2,
    1 + t ** 2
])
F.jacobian([x1,x2, t])
```

$$\left[\begin{matrix}2 x_{1} x_{2}^{2} + x_{2} & 2 x_{1}^{2} x_{2} + x_{1} & 0\\0 & 0 & - 2 t\\0 & 0 & 2 t\end{matrix}\right]$$

```python
x1, x2, x3, t = symbols('x1 x2 x3 t')

F = Matrix([
       x1 ** 3 * cos(x2) * exp(x3)
])
T = Matrix([
    2 * t,
    1 - t ** 2,
    exp(t)
])
jf = F.jacobian([x1,x2, x3])
jt = T.jacobian([t])
display(jf)
display(jt)
```

$$\left[\begin{matrix}3 x_{1}^{2} e^{x_{3}} \cos{\left (x_{2} \right )} & - x_{1}^{3} e^{x_{3}} \sin{\left (x_{2} \right )} & x_{1}^{3} e^{x_{3}} \cos{\left (x_{2} \right )}\end{matrix}\right]$$

$$\left[\begin{matrix}2\\- 2 t\\e^{t}\end{matrix}\right]$$

```python
x1, x2, u1, u2, t = symbols('x1 x2 u1, u2, t')

F = Matrix([
    x1 ** 2 - x2 ** 2
])
U = Matrix([
    2 * u1 + 3 * u2,
    2 * u1 - 3 * u2
])
T = Matrix([
    cos(t / 2),
    sin(2 * t)
])
jf = F.jacobian([x1,x2, x3])
ju = U.jacobian([u1, u2])
jt = T.jacobian([t])
display(jf)
display(ju)
display(jt)
```

$$\left[\begin{matrix}2 x_{1} & - 2 x_{2} & 0\end{matrix}\right]$$

$$\left[\begin{matrix}2 & 3\\2 & -3\end{matrix}\right]$$

$$\left[\begin{matrix}- \frac{1}{2} \sin{\left (\frac{t}{2} \right )}\\2 \cos{\left (2 t \right )}\end{matrix}\right]$$

```python
x1, x2, u1, u2, t = symbols('x1 x2 u1, u2, t')

F = Matrix([
    cos(x1)*sin(x2)
])
U = Matrix([
    2 * u1 ** 2 + 3 * u2 ** 2 - u2,
    2 * u1 - 5 * u2 ** 3
])
T = Matrix([
    exp(t / 2),
    exp(-2 * t)
])
jf = F.jacobian([x1,x2, x3])
ju = U.jacobian([u1, u2])
jt = T.jacobian([t])
display(jf)
display(ju)
display(jt)
```

$$\left[\begin{matrix}- \sin{\left (x_{1} \right )} \sin{\left (x_{2} \right )} & \cos{\left (x_{1} \right )} \cos{\left (x_{2} \right )} & 0\end{matrix}\right]$$

$$\left[\begin{matrix}4 u_{1} & 6 u_{2} - 1\\2 & - 15 u_{2}^{2}\end{matrix}\right]$$

$$\left[\begin{matrix}\frac{e^{\frac{t}{2}}}{2}\\- 2 e^{- 2 t}\end{matrix}\right]$$

```python
x1, x2, x3, u1, u2, t = symbols('x1 x2 x3 u1 u2 t')

F = Matrix([
    sin(x1) * cos(x2) * exp(x3)
])
U = Matrix([
    sin(u1) + cos(u2),
    cos(u1) - sin(u2),
    exp(u1 + u2)
])
T = Matrix([
    1 + t / 2,
    1 - t / 2
])
jf = F.jacobian([x1,x2, x3])
ju = U.jacobian([u1, u2])
jt = T.jacobian([t])
display(jf)
display(ju)
display(jt)
```

$$\left[\begin{matrix}e^{x_{3}} \cos{\left (x_{1} \right )} \cos{\left (x_{2} \right )} & - e^{x_{3}} \sin{\left (x_{1} \right )} \sin{\left (x_{2} \right )} & e^{x_{3}} \sin{\left (x_{1} \right )} \cos{\left (x_{2} \right )}\end{matrix}\right]$$

$$\left[\begin{matrix}\cos{\left (u_{1} \right )} & - \sin{\left (u_{2} \right )}\\- \sin{\left (u_{1} \right )} & - \cos{\left (u_{2} \right )}\\e^{u_{1} + u_{2}} & e^{u_{1} + u_{2}}\end{matrix}\right]$$

$$\left[\begin{matrix}\frac{1}{2}\\- \frac{1}{2}\end{matrix}\right]$$

```python

```

Certification 2/Week3.1 - Multivariate Chain rule.ipynb


# Neural Network Fundamentals

## Gradient Descent Introduction:
https://www.youtube.com/watch?v=IxBYhjS295w

```python
from IPython.display import YouTubeVideo
YouTubeVideo("IxBYhjS295w")
```

```python
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import LinearRegression

np.random.seed(1)

%matplotlib inline
np.random.seed(1)
```

```python
N = 100
x = np.random.rand(N,1)*5
# Let the following command be the true function
y = 2.3 + 5.1*x
# Get some noisy observations
y_obs = y + 2*np.random.randn(N,1)
```

```python
plt.scatter(x,y_obs,label='Observations')
plt.plot(x,y,c='r',label='True function')
plt.legend()
plt.show()
```

## Gradient Descent

We are trying to minimise $\sum \xi_i^2$.

\begin{align}
\mathcal{L} & = \frac{1}{N}\sum_{i=1}^N (y_i-f(x_i,w,b))^2 \\
\frac{\delta\mathcal{L}}{\delta w} & = -\frac{1}{N}\sum_{i=1}^N 2(y_i-f(x_i,w,b))\frac{\delta f(x_i,w,b)}{\delta w} \\
& = -\frac{1}{N}\sum_{i=1}^N 2\xi_i\frac{\delta f(x_i,w,b)}{\delta w}
\end{align}
where $\xi_i$ is the error term $y_i-f(x_i,w,b)$ and
$$
\frac{\delta f(x_i,w,b)}{\delta w} = x_i
$$

Similar expression can be found for $\frac{\delta\mathcal{L}}{\delta b}$ (exercise).

Finally the weights can be updated as $w_{new} = w_{current} - \gamma \frac{\delta\mathcal{L}}{\delta w}$ where $\gamma$ is a learning rate between 0 and 1.

```python
# Helper functions
def f(w,b):
    return w*x+b

def loss_function(e):
    L = np.sum(np.square(e))/N
    return L

def dL_dw(e,w,b):
    return -2*np.sum(e*df_dw(w,b))/N

def df_dw(w,b):
    return x

def dL_db(e,w,b):
    return -2*np.sum(e*df_db(w,b))/N

def df_db(w,b):
    return np.ones(x.shape)
```

```python
# The Actual Gradient Descent
def gradient_descent(iter=100,gamma=0.1):
    # get starting conditions
    w = 10*np.random.randn()
    b = 10*np.random.randn()

    params = []
    loss = np.zeros((iter,1))
    for i in range(iter):
#         from IPython.core.debugger import Tracer; Tracer()()
        params.append([w,b])
        e = y_obs - f(w,b) # Really important that you use y_obs and not y (you do not have access to true y)
        loss[i] = loss_function(e)

        #update parameters
        w_new = w - gamma*dL_dw(e,w,b)
        b_new = b - gamma*dL_db(e,w,b)
        w = w_new
        b = b_new

    return params, loss

params, loss = gradient_descent()
```

```python
iter=100
gamma = 0.1
w = 10*np.random.randn()
b = 10*np.random.randn()

params = []
loss = np.zeros((iter,1))
for i in range(iter):
#         from IPython.core.debugger import Tracer; Tracer()()
    params.append([w,b])
    e = y_obs - f(w,b) # Really important that you use y_obs and not y (you do not have access to true y)
    loss[i] = loss_function(e)

    #update parameters
    w_new = w - gamma*dL_dw(e,w,b)
    b_new = b - gamma*dL_db(e,w,b)
    w = w_new
    b = b_new
```

```python
dL_dw(e,w,b)
```

    0.007829640537794828

```python
plt.plot(loss)
```

```python
params = np.array(params)
plt.plot(params[:,0],params[:,1])
plt.title('Gradient descent')
plt.xlabel('w')
plt.ylabel('b')
plt.show()
```

```python
params[-1]
```

    array([4.98991104, 2.72258102])

## Multivariate case

We are trying to minimise $\sum \xi_i^2$. This time $ f = Xw$ where $w$ is Dx1 and $X$ is NxD.

\begin{align}
\mathcal{L} & = \frac{1}{N} (y-Xw)^T(y-Xw) \\
\frac{\delta\mathcal{L}}{\delta w} & = -\frac{1}{N} 2\left(\frac{\delta f(X,w)}{\delta w}\right)^T(y-Xw) \\
& = -\frac{2}{N} \left(\frac{\delta f(X,w)}{\delta w}\right)^T\xi
\end{align}
where $\xi_i$ is the error term $y_i-f(X,w)$ and
$$
\frac{\delta f(X,w)}{\delta w} = X
$$

Finally the weights can be updated as $w_{new} = w_{current} - \gamma \frac{\delta\mathcal{L}}{\delta w}$ where $\gamma$ is a learning rate between 0 and 1.

```python
N = 1000
D = 5
X = 5*np.random.randn(N,D)
w = np.random.randn(D,1)
y = X.dot(w)
y_obs = y + np.random.randn(N,1)
```

```python
w
```

    array([[ 0.93774813],
           [-2.62540124],
           [ 0.74616483],
           [ 0.67411002],
           [ 1.0142675 ]])

```python
X.shape
```

    (1000, 5)

```python
w.shape
```

    (5, 1)

```python
(X*w.T).shape
```

    (1000, 5)

```python
# Helper functions
def f(w):
    return X.dot(w)

def loss_function(e):
    L = e.T.dot(e)/N
    return L

def dL_dw(e,w):
    return -2*X.T.dot(e)/N
```

```python
def gradient_descent(iter=100,gamma=1e-3):
    # get starting conditions
    w = np.random.randn(D,1)
    params = []
    loss = np.zeros((iter,1))
    for i in range(iter):
        params.append(w)
        e = y_obs - f(w) # Really important that you use y_obs and not y (you do not have access to true y)
        loss[i] = loss_function(e)

        #update parameters
        w = w - gamma*dL_dw(e,w)

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
params[-1]
```

    array([[ 0.94792987],
           [-2.60989696],
           [ 0.72929842],
           [ 0.65272494],
           [ 1.01038855]])

```python
model = LinearRegression(fit_intercept=False)
model.fit(X,y)
model.coef_.T
```

    array([[ 0.93774813],
           [-2.62540124],
           [ 0.74616483],
           [ 0.67411002],
           [ 1.0142675 ]])

```python
# compare parameters side by side
np.hstack([params[-1],model.coef_.T])
```

    array([[ 0.94792987,  0.93774813],
           [-2.60989696, -2.62540124],
           [ 0.72929842,  0.74616483],
           [ 0.65272494,  0.67411002],
           [ 1.01038855,  1.0142675 ]])

## Stochastic Gradient Descent

```python
def dL_dw(X,e,w):
    return -2*X.T.dot(e)/len(X)

def gradient_descent(gamma=1e-3, n_epochs=100, batch_size=20, decay=0.9):
    epoch_run = int(len(X)/batch_size)

    # get starting conditions
    w = np.random.randn(D,1)
    params = []
    loss = np.zeros((n_epochs,1))
    for i in range(n_epochs):
        params.append(w)

        for j in range(epoch_run):
            idx = np.random.choice(len(X),batch_size,replace=False)
            e = y_obs[idx] - X[idx].dot(w) # Really important that you use y_obs and not y (you do not have access to true y)
            #update parameters
            w = w - gamma*dL_dw(X[idx],e,w)
        loss[i] = e.T.dot(e)/len(e)
        gamma = gamma*decay #decay the learning parameter

    return params, loss

params, loss = gradient_descent()
```

```python
plt.plot(loss)
```

```python
np.hstack([params[-1],model.coef_.T])
```

    array([[ 0.94494132,  0.93774813],
           [-2.6276984 , -2.62540124],
           [ 0.74654537,  0.74616483],
           [ 0.66766209,  0.67411002],
           [ 1.00760747,  1.0142675 ]])

```python

```

jupyter/Keras_TensorFlow_Course/Lesson 02 - GradientDescent.ipynb


# ★ Numerical Differentiaion and Integration ★

```python
# Import modules
import math
import sympy as sym
import numpy as np
import scipy
import matplotlib.pyplot as plt
import plotly
import plotly.plotly as ply
import plotly.figure_factory as ply_ff
from IPython.display import Math
from IPython.display import display

# Startup plotly
plotly.offline.init_notebook_mode(connected=True)

''' Fix MathJax issue '''
# The polling here is to ensure that plotly.js has already been loaded before
# setting display alignment in order to avoid a race condition.
from IPython.core.display import display, HTML
display(HTML(
    ''
))
```

# 5.1 Numerical Differentiation

## Two-point forward-difference formula

## $f'(x) = \frac{f(x+h) - f(x)}{h} - \frac{h}{2}f''(c)$

where $c$ is between $x$ and $x+h$

### Example

Use the two-point forward-difference formula with $h = 0.1$ to approximate the derivative of $f(x) = 1/x$ at $x = 2$

```python
# Parameters
x = 2
h = 0.1

# Symbolic computation
sym_x = sym.Symbol('x')
sym_deri_x1 = sym.diff(1 / sym_x, sym_x)
sym_deri_x1_num = sym_deri_x1.subs(sym_x, x).evalf()

# Approximation
f = lambda x : 1 / x
deri_x1 = (f(x + h) - f(x)) / h

# Comparison
print('approximate = %f, real value = %f, backward error = %f' %(deri_x1, sym_deri_x1_num, abs(deri_x1 - sym_deri_x1_num)) )
```

    approximate = -0.238095, real value = -0.250000, backward error = 0.011905

## Three-point centered-difference formula

## $f'(x) =  \frac{f(x+h) - f(x-h)}{2h} - \frac{h^2}{6}f'''(c)$

where $x-h < c < x+h$

### Example

Use the three-point centered-difference formula with $h = 0.1$ to approximate the derivative of $f(x) = 1 / x$ at $x = 2$

```python
# Parameters
x = 2
h = 0.1
f = lambda x : 1 / x

# Symbolic computation
sym_x = sym.Symbol('x')
sym_deri_x1 = sym.diff(1 / sym_x, sym_x)
sym_deri_x1_num = sym_deri_x1.subs(sym_x, x).evalf()

# Approximation
deri_x1 = (f(x + h) - f(x - h)) / (2 * h)

# Comparison
print('approximate = %f, real value = %f, backward error = %f' %(deri_x1, sym_deri_x1_num, abs(deri_x1 - sym_deri_x1_num)) )
```

    approximate = -0.250627, real value = -0.250000, backward error = 0.000627

## Three-point centered-difference formula for second derivative

## $f''(x) = \frac{f(x - h) - 2f(x) + f(x + h)}{h^2} - \frac{h^2}{12}f^{(iv)}(c)$

for some $c$ between $x - h$ and $x + h$

## Rounding error

### Example

Approximate the derivative of $f(x) = e^x$ at $x = 0$

```python
# Parameters
f = lambda x : math.exp(x)
real_value = 1
h_msg = "$10^{-%d}$"
twp_deri_x1 = lambda x, h : ( f(x + h) - f(x) ) / h
thp_deri_x1 = lambda x, h : ( f(x + h) - f(x - h) ) / (2 * h)

data = [
    ["h",
     "$f'(x) \\approx \\frac{e^{x+h} - e^x}{h}$",
     "error",
     "$f'(x) \\approx \\frac{e^{x+h} - e^{x-h}}{2h}$",
     "error"],
]

for i in range(1,10):
    h = pow(10, -i)
    twp_deri_x1_value = twp_deri_x1(0, h)
    thp_deri_x1_value = thp_deri_x1(0, h)
    row = ["", "", "", "", ""]
    row[0] = h_msg %i
    row[1] = '%.14f' %twp_deri_x1_value
    row[2] = '%.14f' %abs(twp_deri_x1_value - real_value)
    row[3] = '%.14f' %thp_deri_x1_value
    row[4] = '%.14f' %abs(thp_deri_x1_value - real_value)
    data.append(row)

table = ply_ff.create_table(data)
plotly.offline.iplot(table, show_link=False)
```

## Extrapolation for order n formula

## $ Q \approx \frac{2^nF(h/2) - F(h)}{2^n - 1} $

```python
sym.init_printing(use_latex=True)

x = sym.Symbol('x')
dx = sym.diff(sym.exp(sym.sin(x)), x)
Math('Derivative : %s' %sym.latex(dx) )
```

$$Derivative : e^{\sin{\left (x \right )}} \cos{\left (x \right )}$$

# 5.2 Newton-Cotes Formulas For Numerical Integration

## Trapezoid Rule

## $\int_{x_0}^{x_1} f(x) dx = \frac{h}{2}(y_0 + y_1) - \frac{h^3}{12}f''(c)$

where $h = x_1 - x_0$ and $c$ is between $x_0$ and $x_1$

## Simpson's Rule

## $\int_{x_0}^{x_2} f(x) dx = \frac{h}{3}(y_0 + 4y_1 + y_2) - \frac{h^5}{90}f^{(iv)}(c)$

where $h = x_2 - x_1 = x_1 - x_0$ and $c$ is between $x_0$ and $x_2$

### Example

Apply the Trapezoid Rule and Simpson's Rule to approximate $\int_{1}^{2} \ln(x) dx$ and find an upper bound for the error in your approximations

```python
# Apply Trapezoid Rule
trapz = scipy.integrate.trapz([np.log(1), np.log(2)], [1, 2])

# Evaluate the error term of Trapezoid Rule
sym_x = sym.Symbol('x')
expr = sym.diff(sym.log(sym_x), sym_x, 2)
trapz_err = abs(expr.subs(sym_x, 1).evalf() / 12)

# Print out results
print('Trapezoid rule : %f and upper bound error : %f' %(trapz, trapz_err) )
```

    Trapezoid rule : 0.346574 and upper bound error : 0.083333

```python
# Apply Simpson's Rule
area = scipy.integrate.simps([np.log(1), np.log(1.5), np.log(2)], [1, 1.5, 2])

# Evaluate the error term
sym_x = sym.Symbol('x')
expr = sym.diff(sym.log(sym_x), sym_x, 4)
simps_err = abs( pow(0.5, 5) / 90 * expr.subs(sym_x, 1).evalf() )

# Print out results
print('Simpson\'s rule : %f and upper bound error : %f' %(area, simps_err) )
```

    Simpson's rule : 0.385835 and upper bound error : 0.002083

## Composite Trapezoid Rule

## $\int_{a}^{b} f(x) dx = \frac{h}{2} \left ( y_0 + y_m + 2\sum_{i=1}^{m-1}y_i \right ) - \frac{(b-a)h^2}{12}f''(c)$

where $h = (b - a) / m $ and $c$ is between $a$ and $b$

## Composite Simpson's Rule

## $ \int_{a}^{b}f(x)dx = \frac{h}{3}\left [ y_0 + y_{2m} + 4\sum_{i=1}^{m}y_{2i-1} + 2\sum_{i=1}^{m - 1}y_{2i} \right ] - \frac{(b-a)h^4}{180}f^{(iv)}(c) $

where $c$ is between $a$ and $b$

### Example

Carry out four-panel approximations of $\int_{1}^{2} \ln{x} dx$ using the composite Trapezoid Rule and composite Simpson's Rule

```python
# Apply composite Trapezoid Rule
x = np.linspace(1, 2, 5)
y = np.log(x)
trapz = scipy.integrate.trapz(y, x)

# Error term
sym_x = sym.Symbol('x')
expr = sym.diff(sym.log(sym_x), sym_x, 2)
trapz_err = abs( (2 - 1) * pow(0.25, 2) / 12 * expr.subs(sym_x, 1).evalf() )

print('Trapezoid Rule : %f, error = %f' %(trapz, trapz_err) )
```

    Trapezoid Rule : 0.383700, error = 0.005208

```python
# Apply composite Trapezoid Rule
x = np.linspace(1, 2, 9)
y = np.log(x)
area = scipy.integrate.simps(y, x)

# Error term
sym_x = sym.Symbol('x')
expr = sym.diff(sym.log(sym_x), sym_x, 4)
simps_err = abs( (2 - 1) * pow(0.125, 4) / 180 * expr.subs(sym_x, 1).evalf() )

print('Simpson\'s Rule : %f, error = %f' %(area, simps_err) )
```

    Simpson's Rule : 0.386292, error = 0.000008

## Midpoint Rule

## $ \int_{x_0}^{x_1} f(x)dx = hf(\omega) + \frac{h^3}{24}f''(c) $

where $ h = (x_1 - x_0) $, $\omega$ is the midpoint $ x_0 + h / 2 $, and $c$ is between $x_0$ and $x_1$

## Composite Midpoint Rule

## $ \int_{a}^{b} f(x) dx = h \sum_{i=1}^{m}f(\omega_{i}) + \frac{(b - a)h^2}{24} f''(c) $

where $h = (b - a) / m$ and $c$ is between $a$ and $b$. The $\omega_{i}$ are the midpoints of the $m$ equal subintervals of $[a,b]$

### Example
Approximate $\int_{0}^{1} \frac{\sin x}{x} dx$ by using the composite Midpoint Rule with $m = 10$ panels

```python
# Parameters
m = 10
h = (1 - 0) / m
f = lambda x : np.sin(x) / x
mids = np.arange(0 + h/2, 1, h)

# Apply composite midpoint rule
area = h * np.sum(f(mids))

# Error term
sym_x = sym.Symbol('x')
expr = sym.diff(sym.sin(sym_x) / sym_x, sym_x, 2)
mid_err = abs( (1 - 0) * pow(h, 2) / 24 * expr.subs(sym_x, 1).evalf() )

# Print out
print('Composite Midpoint Rule : %.8f, error = %.8f' %(area, mid_err) )
```

    Composite Midpoint Rule : 0.94620858, error = 0.00009964

# 5.3 Romberg Integration

```python
def romberg(f, a, b, step):
    R = np.zeros(step * step).reshape(step, step)
    R[0][0] = (b - a) * (f(a) + f(b)) / 2
    for j in range(1, step):
        h = (b - a) / pow(2, j)
        summ = 0
        for i in range(1, pow(2, j - 1) + 1):
            summ += h * f(a + (2 * i - 1) * h)
        R[j][0] = 0.5 * R[j - 1][0] + summ

        for k in range(1, j + 1):
            R[j][k] = ( pow(4, k) * R[j][k - 1] - R[j - 1][k - 1] ) / ( pow(4, k) - 1 )

    return R[step - 1][step - 1]
```

### Example

Apply Romberg Integration to approximate $\int_{1}^{2} \ln{x}dx$

```python
f = lambda x : np.log(x)
result = romberg(f, 1, 2, 4)
print('Romberg Integration : %f' %(result) )
```

    Romberg Integration : 0.386294

```python
f = lambda x : np.log(x)
result = scipy.integrate.romberg(f, 1, 2, show=True)
print('Romberg Integration : %f' %(result) )
```

    Romberg integration of <function vectorize1.<locals>.vfunc at 0x7fa72aad8510> from [1, 2]

     Steps  StepSize   Results
         1  1.000000  0.346574
         2  0.500000  0.376019  0.385835
         4  0.250000  0.383700  0.386260  0.386288
         8  0.125000  0.385644  0.386292  0.386294  0.386294
        16  0.062500  0.386132  0.386294  0.386294  0.386294  0.386294
        32  0.031250  0.386254  0.386294  0.386294  0.386294  0.386294  0.386294

    The final result is 0.38629436112 after 33 function evaluations.
    Romberg Integration : 0.386294

# 5.4 Adaptive Quadrature

```python
''' Use Trapezoid Rule '''

def adaptive_quadrature(f, a, b, tol):
    return adaptive_quadrature_recursively(f, a, b, tol, a, b, 0)

def adaptive_quadrature_recursively(f, a, b, tol, orig_a, orig_b, deep):
    c = (a + b) / 2
    S = lambda x, y : (y - x) * (f(x) + f(y)) / 2
    if abs( S(a, b) - S(a, c) - S(c, b) ) < 3 * tol * (b - a) / (orig_b - orig_a) or deep > 20 :
        return S(a, c) + S(c, b)
    else:
        return adaptive_quadrature_recursively(f, a, c, tol / 2, orig_a, orig_b, deep + 1) + adaptive_quadrature_recursively(f, c, b, tol / 2, orig_a, orig_b, deep + 1)
```

```python
''' Use Simpon's Rule '''

def adaptive_quadrature(f, a, b, tol):
    return adaptive_quadrature_recursively(f, a, b, tol, a, b, 0)

def adaptive_quadrature_recursively(f, a, b, tol, orig_a, orig_b, deep):
    c = (a + b) / 2
    S = lambda x, y : (y - x) * ( f(x) + 4 * f((x + y) / 2) + f(y) ) / 6
    if abs( S(a, b) - S(a, c) - S(c, b) ) < 15 * tol  or deep > 20 :
        return S(a, c) + S(c, b)
    else:
        return adaptive_quadrature_recursively(f, a, c, tol / 2, orig_a, orig_b, deep + 1) + adaptive_quadrature_recursively(f, c, b, tol / 2, orig_a, orig_b, deep + 1)
```

### Example

Use Adaptive Quadrature to approximate the integral $ \int_{-1}^{1} (1 + \sin{e^{3x}}) dx $

```python
f = lambda x : 1 + np.sin(np.exp(3 * x))
val = adaptive_quadrature(f, -1, 1, tol=1e-12)
print(val)
```

    2.50080911034

# 5.5 Gaussian Quadrature

```python
poly = scipy.special.legendre(2)
# Find roots of polynomials
comp = scipy.linalg.companion(poly)
roots = scipy.linalg.eig(comp)[0]
```

### Example

Approximate $\int_{-1}^{1} e^{-\frac{x^2}{2}}dx$ using Gaussian Quadrature

```python
f = lambda x : np.exp(-np.power(x, 2) / 2)
quad = scipy.integrate.quadrature(f, -1, 1)
print(quad[0])
```

    1.71124878401

```python
# Parametes
a = -1
b = 1
deg = 3
f = lambda x : np.exp( -np.power(x, 2) / 2 )

x, w = scipy.special.p_roots(deg) # Or use numpy.polynomial.legendre.leggauss
quad = np.sum(w * f(x))

print(quad)
```

    1.7120202452

### Example

Approximate the integral $\int_{1}^{2} \ln{x} dx$ using Gaussian Quadrature

```python
# Parametes
a = 1
b = 2
deg = 4
f = lambda t : np.log( ((b - a) * t + b + a) / 2) * (b - a) / 2

x, w = scipy.special.p_roots(deg)
np.sum(w * f(x))
```

    0.38629449693871409


# Evaluating numerical errors and accuracy

- toc: false
- branch: master
- badges: true
- comments: false
- categories: [mathematics, numerical recipes]
- hide: true

-----
Questions:

- Which numerical errors are unavoidable in a Python programme?
- How do I choose the optimum step size $h$ when using the finite difference method?
- What do the terms first-order accurate and second-order accurate mean?
- How can I measure the speed of my code?

Objectives:

- Understand that there are unavoidable rounding errors when working with floats
- Write code for testing if two floats are equivalent (to within machine accuracy)
- Calculate the optimum step size $h$ for finite difference methods
- Measure the length of time a Notebook cell takes to run using the `%time` magic.
-----

### Computers have inherent limitations that lead to rounding errors

- We have seen how computer programming can be used to model physical systems. However computers have inherent limitations - they cannot store real numbers with an infinite number of decimal places.
- In many cases this is not a problem, but it is something to be aware of. For example, take the following piece of code:

```python
def add_numbers(a,b):
    return a+b

def test_add_numbers():
    assert add_numbers(0.1,0.2) == 0.3
```

- `add_numbers` is a function for adding two Python objects `a` and `b`.
- `test_add_numbers` is a function for testing is the `add_numbers` function is working as expected (we will see more on testing later in the course). This function will raise an error if $0.1 + 0.2$ does not equal 0.3.

```python
add_numbers(1,2)
```

    3

The `add_numbers` function works as expected if we pass it two integers. However when we run the test function we raise an assertion error:

```python
test_add_numbers()
```

This <mark> rounding error </mark> is given because $0.1+0.2$ does not equal 0.3 exactly:

```python
0.1+0.2
```

    0.30000000000000004

This is because floating point numbers (floats) are represented on the computer to a certain precision. In Python the standard level of precision is 16 significant digits.

> Note: The largest value you can give a floating point variable is about $10^{308}$. The smallest is -$10^{308}$. If you exceed these values (which is unlikely) then the computer will return an Overflow error. In contrast, PYthon can represent integers to any precision - limited only by the memory of the machine.

### Do not test for the equality of two floats

As we have seen in the previous example, we should not test for the equality of two floats. Instead we should ask if they are equal up to a given precision:

```python
def add_numbers(a,b):
    return a+b

epsilon = 1e-12

def test_add_numbers():
    assert abs(add_numbers(0.1,0.2) - 0.3) < epsilon
```

```python
test_add_numbers()
```

###  Finite difference methods have two sources of error

- There are two sources of errors for finite difference methods: one is from the approximation that the step size $h$ is small but not zero. The second is from the rounding errors introduced at the start of this tutorial.
- One way of improving the finite-$h$ approximation is to decrease the step size in space (use a higher number of points on our real space grid). However when the step size is decreased the programme will run more slowly.
- We also need to think about the rounding errors associated with finite differences. Counter-intuitively, these errors can increase as we decrease the step size $h$.

To demonstrate this, consider the Taylor expansion of $f(x)$ about $x$:

\begin{equation}
f(x+h) = f(x) + hf'(x) +\frac{1}{2}h^2f''(x) + \ldots
\end{equation}

Re-arrange the expression to get the expression for the forward difference method:

\begin{equation}
f'(x) = \frac{f(x+h)}{h} - \frac{1}{2}hf''(x)+\ldots
\end{equation}

A computer can typically store a number $f(x)$ to an accuracy of 16 significant figures, or $Cf(x)$ where $C=10^{-16}$. In the worst case, this makes the error $\epsilon$ on our derivative:

\begin{equation}
\epsilon = \frac{2C|f(x)|}{h} + \frac{1}{2}h|f''(x)|.
\end{equation}

We want to find the value of $h$ which minimises this error so we differentiate with respect to $h$ and set the result equal to zero.

\begin{equation}
-\frac{2C|f(x)|}{h^2} + h|f''(x)|
\end{equation}

\begin{equation}
h = \sqrt{4C\lvert\frac{f(x)}{f''(x)}\rvert}
\end{equation}

If $f(x)$ and $f''(x)$ are order 1, then $h$ should be order $\sqrt{C}$, or $10^{-8}$.

Similar reasoning applied to the central difference formula suggests that the optimum step size for this method is $10^{-5}$.

### The relaxation method for PDEs is limited by the accuracy of the finite difference method.

- For solving PDEs we use the relaxation method combined with a finite difference method.
- Even if we use a very small target accuracy for convergence of the relaxation method, our accuracy will still be limited by the finite differences. Higher-order finite difference methods (such as the 5-point or 7-point methods) can be used here to improve the overrall accuracy of the calculation.

### Euler's method is a first-order method accurate to order $h$.

- As we have seen in previous tutorials, numerical methods (such as Euler's method or the Runge-Kutta method) give approximate solutions.
- Euler's method neglected the term in $h^2$ and higher:
\begin{equation}
x(t+h) = x(t)+hf(x,t)+\mathcal{O}(h^2)
\end{equation}
- This tells us the error introduced on a single step of the method is proportional to $h^2$ - this makes Euler's method  a <mark> first-order </mark> method, accurate to order $h$.
- However the cumulative error over several steps is proportional to $h$
- So to make our error half as large we need to double the number of steps (halve the step size) and double the length of the calculation.

### The second-order Runge-Kutta method is accurate to order $h^2$.

- The error term for one step of the Runge-Kutta method is ${O}(h^3)$ - this makes the Runge-Kutta method accurate to order $h^2$ which is why this is called the <mark>second-order</mark> Runge Kutta method (RK2).
- With the RK2 can use a fewer number of steps whilst getting the same accuracy as Euler's method.
- There are higher order Runge-Kutta methods which increase the accuracy further.

### Use the %time magic to measure the length of time a Jupyter Notebook cell takes to run

```python
def sum_integers(max_integer):
    count = 0
    for i in range(max_integer):
        count += max_integer + 1

    return count

```

```python
%time sum = sum_integers(1000000)
```

    CPU times: user 100 ms, sys: 3.79 ms, total: 104 ms
    Wall time: 110 ms

---
Keypoints:

- Computers have inherent limitations that lead to rounding errors
- Do not test for the equality of two floats
- Finite difference methods have two sources of error
- The relaxation method for PDEs is limited by the accuracy of the finite difference method.
- Euler's method is a first-order method accurate to order $h$.
- The second-order Runge-Kutta method is accurate to order $h^2$.

---

Do [the quick-test](https://nu-cem.github.io/CompPhys/2021/08/02/Evaluating-Accuracy-Qs.html).

Back to [Modelling with Partial Differential Equations](https://nu-cem.github.io/CompPhys/2021/08/02/PDEs.html).

---


```python
%matplotlib inline
from IPython.display import display,Math
from sympy import *
init_session()
```

```python
%%time
# 最大公約数 gcd(a, b) を求めるアルゴリズム その1
a, b = 20210608, 80601202
i = 1
while i <= b:
    if a%i == 0:
        if b%i == 0:
            M  = i
    i = i+1
print(M)
```

```python
%%time
# 最大公約数 gcd(a, b) を求めるアルゴリズム その2
a, b = 20210608, 80601202
r = a%b
while r != 0:
    a,b = b,r
    r = a%b
print(b)
```

```python
from ipywidgets import interact
import time
def mygcd2(a,b):
    r = a%b
    count = 1
    while r != 0:
        count += 1
        a,b = b,r
        r = a%b
    return b,count
@interact
def _(a="314159265",b="35"):
    digits = len(b)
    a,b = int(a),int(b)
    d,count = mygcd2(a,b)
    return display(Math("gcd({0:d},{1:d})={2:d}, (digits,count)=({3:d},{4:d})".format(a,b,d,digits,count)))
```

```python

```

```python

```