# Statistics Fundamentals
Statistics is primarily about analyzing data samples, and that starts with udnerstanding the distribution of data in a sample.

## Analyzing Data Distribution
A great deal of statistical analysis is based on the way that data values are distributed within the dataset. In this section, we'll explore some statistics that you can use to tell you about the values in a dataset.

### Measures of Central Tendency
The term *measures of central tendency* sounds a bit grand, but really it's just a fancy way of saying that we're interested in knowing where the middle value in our data is. For example, suppose decide to conduct a study into the comparative salaries of people who graduated from the same school. You might record the results like this:

| Name     | Salary      |
|----------|-------------|
| Dan      | 50,000      |
| Joann    | 54,000      |
| Pedro    | 50,000      |
| Rosie    | 189,000     |
| Ethan    | 55,000      |
| Vicky    | 40,000      |
| Frederic | 59,000      |

Now, some of the former-students may earn a lot, and others may earn less; but what's the salary in the middle of the range of all salaries?

#### Mean
A common way to define the central value is to use the *mean*, often called the *average*. This is calculated as the sum of the values in the dataset, divided by the number of observations in the dataset. When the dataset consists of the full population, the mean is represented by the Greek symbol ***μ*** (*mu*), and the formula is written like this:

\begin{equation}\mu = \frac{\displaystyle\sum_{i=1}^{N}X_{i}}{N}\end{equation}

More commonly, when working with a sample, the mean is represented by ***x̄*** (*x-bar*), and the formula is written like this (note the lower case letters used to indicate values from a sample):

\begin{equation}\bar{x} = \frac{\displaystyle\sum_{i=1}^{n}x_{i}}{n}\end{equation}

In the case of our list of heights, this can be calculated as:

\begin{equation}\bar{x} = \frac{50000+54000+50000+189000+55000+40000+59000}{7}\end{equation}

Which is **71,000**.

>In technical terminology, ***x̄*** is a *statistic* (an estimate based on a sample of data) and ***μ*** is a *parameter* (a true value based on the entire population). A lot of the time, the parameters for the full population will be impossible (or at the very least, impractical) to measure; so we use statistics obtained from a representative sample to approximate them. In this case, we can use the sample mean of salary for our selection of surveyed students to try to estimate the actual average salary of all students who graduate from our school.

In Python, when working with data in a *pandas.dataframe*, you can use the ***mean*** function, like this:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].mean())
```

    71000.0

So, is **71,000** really the central value? Or put another way, would it be reasonable for a graduate of this school to expect to earn $71,000? After all, that's the average salary of a graduate from this school.

If you look closely at the salaries, you can see that out of the seven former students, six earn less than the mean salary. The data is *skewed* by the fact that Rosie has clearly managed to find a much higher-paid job than her classmates.

#### Median
OK, let's see if we can find another definition for the central value that more closely reflects the expected earning potential of students attending our school. Another measure of central tendancy we can use is the *median*. To calculate the median, we need to sort the values into ascending order and then find the middle-most value. When there are an odd number of observations, you can find the position of the median value using this formula (where *n* is the number of observations):

\begin{equation}\frac{n+1}{2}\end{equation}

Remember that this formula returns the *position* of the median value in the sorted list; not the value itself.

If the number of observations is even, then things are a little (but not much) more complicated. In this case you calculate the median as the average of the two middle-most values, which are found like this:

\begin{equation}\frac{n}{2} \;\;\;\;and \;\;\;\; \frac{n}{2} + 1\end{equation}

So, for our graduate salaries; first lets sort the dataset:

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

There's an odd number of observation (7), so the median value is at position (7 + 1) ÷ 2; in other words, position 4:

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
|***>54,000*** |
| 55,000      |
| 59,000      |
| 189,000     |

So the median salary is **54,000**.

The *pandas.dataframe* class in Python has a ***median*** function to find the median:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].median())
```

    54000.0

#### Mode
Another related statistic is the *mode*, which indicates the most frequently occurring value. If you think about it, this is potentially a good indicator of how much a student might expect to earn when they graduate from the school; out of all the salaries that are being earned by former students, the mode is earned by more than any other.

Looking at our list of salaries, there are two instances of former students earning **50,000**, but only one instance each for all other salaries:

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

The mode is therefore **50,000**.

As you might expect, the *pandas.dataframe* class has a ***mode*** function to return the mode:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].mode())
```

    0    50000
    dtype: int64

##### Multimodal Data
It's not uncommon for a set of data to have more than one value as the mode. For example, suppose Ethan receives a raise that takes his salary to **59,000**:

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
|***>59,000***|
|***>59,000***|
| 189,000     |

Now there are two values with the highest frequency. This dataset is *bimodal*. More generally, when there is more than one mode value, the data is considered *multimodal*.

The *pandas.dataframe.**mode*** function returns all of the modes:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,59000,40000,59000]})

print (df['Salary'].mode())
```

    0    50000
    1    59000
    dtype: int64

### Distribution and Density
Now we know something about finding the center, we can start to explore how the data is distributed around it. What we're interested in here is understanding the general "shape" of the data distribution so that we can begin to get a feel for what a 'typical' value might be expected to be.

We can start by finding the extremes - the minimum and maximum. In the case of our salary data, the lowest paid graduate from our school is Vicky, with a salary of **40,000**; and the highest-paid graduate is Rosie, with **189,000**.

The *pandas.dataframe* class has ***min*** and ***max*** functions to return these values.

Run the following code to compare the minimum and maximum salaries to the central measures we calculated previously:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print ('Min: ' + str(df['Salary'].min()))
print ('Mode: ' + str(df['Salary'].mode()[0]))
print ('Median: ' + str(df['Salary'].median()))
print ('Mean: ' + str(df['Salary'].mean()))
print ('Max: ' + str(df['Salary'].max()))
```

    Min: 40000
    Mode: 50000
    Median: 54000.0
    Mean: 71000.0
    Max: 189000

We can examine these values, and get a sense for how the data is distributed - for example, we can see that the *mean* is closer to the max than the *median*, and that both are closer to the *min* than to the *max*.

However, it's generally easier to get a sense of the distribution by visualizing the data. Let's start by creating a histogram of the salaries, highlighting the *mean* and *median* salaries (the *min*, *max* are fairly self-evident, and the *mode* is wherever the highest bar is):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
salary.plot.hist(title='Salary Distribution', color='lightblue', bins=25)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

The ***mean*** and ***median*** are shown as dashed lines. Note the following:
- *Salary* is a continuous data value - graduates could potentially earn any value along the scale, even down to a fraction of cent.
- The number of bins in the histogram determines the size of each salary band for which we're counting frequencies. Fewer bins means merging more individual salaries together to be counted as a group.
- The majority of the data is on the left side of the histogram, reflecting the fact that most graduates earn between 40,000 and 55,000
- The mean is a higher value than the median and mode.
- There are gaps in the histogram for salary bands that nobody earns.

The histogram shows the relative frequency of each salary band, based on the number of bins. It also gives us a sense of the *density* of the data for each point on the salary scale. With enough data points, and small enough bins, we could view this density as a line that shows the shape of the data distribution.

Run the following cell to show the density of the salary data as a line on top of the histogram:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
density = stats.gaussian_kde(salary)
n, x, _ = plt.hist(salary, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*5)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()

```

Note that the density line takes the form of an asymmetric curve that has a "peak" on the left and a long tail on the right. We describe this sort of data distribution as being *skewed*; that is, the data is not distributed symmetrically but "bunched together" on one side. In this case, the data is bunched together on the left, creating a long tail on the right; and is described as being *right-skewed* because some infrequently occurring high values are pulling the *mean* to the right.

Let's take a look at another set of data. We know how much money our graduates make, but how many hours per week do they need to work to earn their salaries? Here's the data:

| Name     | Hours |
|----------|-------|
| Dan      | 41    |
| Joann    | 40    |
| Pedro    | 36    |
| Rosie    | 30    |
| Ethan    | 35    |
| Vicky    | 39    |
| Frederic | 40    |

Run the following code to show the distribution of the hours worked:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Hours':[41,40,36,30,35,39,40]})

hours = df['Hours']
density = stats.gaussian_kde(hours)
n, x, _ = plt.hist(hours, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7)
plt.axvline(hours.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(hours.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

Once again, the distribution is skewed, but this time it's **left-skewed**. Note that the curve is asymmetric with the ***mean*** to the left of the ***median*** and the *mode*; and the average weekly working hours skewed to the lower end.

Once again, Rosie seems to be getting the better of the deal. She earns more than her former classmates for working fewer hours. Maybe a look at the test scores the students achieved on their final grade at school might help explain her success:

| Name     | Grade |
|----------|-------|
| Dan      | 50    |
| Joann    | 50    |
| Pedro    | 46    |
| Rosie    | 95    |
| Ethan    | 50    |
| Vicky    | 5     |
| Frederic | 57    |

Let's take a look at the distribution of these grades:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Grade':[50,50,46,95,50,5,57]})

grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7.5)
plt.axvline(grade.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(grade.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

This time, the distribution is symmetric, forming a "bell-shaped" curve. The ***mean***, ***median***, and mode are at the same location, and the data tails off evenly on both sides from a central peak.

Statisticians call this a *normal* distribution (or sometimes a *Gaussian* distribution), and it occurs quite commonly in many scenarios due to something called the *Central Limit Theorem*, which reflects the way continuous probability works - more about that later.

#### Skewness and Kurtosis
You can measure *skewness* (in which direction the data is skewed and to what degree) and kurtosis (how "peaked" the data is) to get an idea of the shape of the data distribution. In Python, you can use the ***skew*** and ***kurt*** functions to find this:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' skewness: ' + str(df[col].skew()))
    print(df[col].name + ' kurtosis: ' + str(df[col].kurt()))
    density = stats.gaussian_kde(df[col])
    n, x, _ = plt.hist(df[col], histtype='step', normed=True, bins=25)
    plt.plot(x, density(x)*6)
    plt.show()
    print('\n')
```

Now let's look at the distribution of a real dataset - let's see how the heights of the father's measured in Galton's study of parent and child heights are distributed:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

import statsmodels.api as sm

df = sm.datasets.get_rdataset('GaltonFamilies', package='HistData').data

fathers = df['father']
density = stats.gaussian_kde(fathers)
n, x, _ = plt.hist(fathers, histtype='step', normed=True, bins=50)
plt.plot(x, density(x)*2.5)
plt.axvline(fathers.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(fathers.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()

```

As you can see, the father's height measurements are approximately normally distributed - in other words, they form a more or less *normal* distribution that is symmetric around the mean.

### Measures of Variance
We can see from the distribution plots of our data that the values in our dataset can vary quite widely. We can use various measures to quantify this variance.

#### Range
A simple way to quantify the variance in a dataset is to identify the difference between the lowest and highest values. This is called the *range*, and is calculated by subtracting the minimim value from the maximum value.

The following Python code creates a single Pandas dataframe for our school graduate data, and calculates the *range* for each of the numeric features:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' range: ' + str(df[col].max() - df[col].min()))
```

#### Percentiles and Quartiles
The range is easy to calculate, but it's not a particularly useful statistic. For example, a range of 149,000 between the lowest and highest salary does not tell us which value within that range a graduate is most likely to earn -  it doesn't tell us nothing about how the salaries are distributed around the mean within that range.  The range tells us very little about the comparative position of an individual value within the distribution - for example, Frederic scored 57 in his final grade at school; which is a pretty good score (it's more than all but one of his classmates); but this isn't immediately apparent from a score of 57 and range of 90.

##### Percentiles
A percentile tells us where a given value is ranked in the overall distribution. For example, 25% of the data in a distribution has a value lower than the 25th percentile; 75% of the data has a value lower than the 75th percentile, and so on. Note that half of the data has a value lower than the 50th percentile - so the 50th percentile is also the median!

Let's examine Frederic's grade using this approach. We know he scored 57, but how does he rank compared to his fellow students?

Well, there are seven students in total, and five of them scored less than Frederic; so we can calculate the percentile for Frederic's grade like this:

\begin{equation}\frac{5}{7} \times 100 \approx 71.4\end{equation}

So Frederic's score puts him at the 71.4th percentile in his class.

In Python, you can use the ***percentileofscore*** function in the *scipy.stats* package to calculate the percentile for a given value in a set of values:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'strict'))
```

We've used the strict definition of percentile; but sometimes it's calculated as being the percentage of values that are less than *or equal to* the value you're comparing. In this case, the calculation for Frederic's percentile would include his own score:

\begin{equation}\frac{6}{7} \times 100 \approx 85.7\end{equation}

You can calculate this way in Python by using the ***weak*** mode of the ***percentileofscore*** function:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'weak'))
```

We've considered the percentile of Frederic's grade, and used it to rank him compared to his fellow students. So what about Dan, Joann, and Ethan? How do they compare to the rest of the class? They scored the same grade (50), so in a sense they share a percentile.

To deal with this *grouped* scenario, we can average the percentage rankings for the matching scores. We treat half of the scores matching the one we're ranking as if they are below it, and half as if they are above it. In this case, there were three matching scores of 50, and for each of these we calculate the percentile as if 1 was below and 1 was above. So the calculation for a percentile for Joann based on scores being less than or equal to 50 is:

\begin{equation}(\frac{4}{7}) \times 100 \approx 57.14\end{equation}

The value of **4** consists of the two scores that are below Joann's score of 50, Joann's own score, and half of the scores that are the same as Joann's (of which there are two, so we count one).

In Python, the ***percentileofscore*** function has a ***rank*** function that calculates grouped percentiles like this:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 50, 'rank'))
```

##### Quartiles
Rather than using individual percentiles to compare data, we can consider the overall spread of the data by dividing those percentiles into four *quartiles*. The first quartile contains the values from the minimum to the 25th percentile, the second from the 25th percentile to the 50th percentile (which is the median), the third from the 50th percentile to the 75th percentile, and the fourth from the 75th percentile to the maximum.

In Python, you can use the ***quantile*** function of the *pandas.dataframe* class to find the threshold values at the 25th, 50th, and 75th percentiles (*quantile* is a generic term for a ranked position, such as a percentile or quartile).

Run the following code to find the quartile thresholds for the weekly hours worked by our former students:

```python
# Quartiles
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
print(df['Hours'].quantile([0.25, 0.5, 0.75]))
```

Its usually easier to understand how data is distributed across the quartiles by visualizing it. You can use a histogram, but many data scientists use a kind of visualization called a *box plot* (or a *box and whiskers* plot).

Let's create a box plot for the weekly hours:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Hours'].plot(kind='box', title='Weekly Hours Distribution', figsize=(10,8))
plt.show()
```

The box plot consists of:
- A rectangular *box* that shows where the data between the 25th and 75th percentile (the second and third quartile) lie. This part of the distribution is often referred to as the *interquartile range* - it contains the middle 50 data values.
- *Whiskers* that extend from the box to the bottom of the first quartile and the top of the fourth quartile to show the full range of the data.
- A line in the box that shows that location of the median (the 50th percentile, which is also the threshold between the second and third quartile)

In this case, you can see that the interquartile range is between 35 and 40, with the median nearer the top of that range. The range of the first quartile is from around 30 to 35, and the fourth quartile is from 40 to 41.

#### Outliers
Let's take a look at another box plot - this time showing the distribution of the salaries earned by our former classmates:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8))
plt.show()
```

So what's going on here?

Well, as we've already noticed, Rosie earns significantly more than her former classmates. So much more in fact, that her salary has been identifed as an *outlier*. An outlier is a value that is so far from the center of the distribution compared to other values that it skews the distribution by affecting the mean. There are all sorts of reasons that you might have outliers in your data, including data entry errors, failures in sensors or data-generating equipment, or genuinely anomalous values.

So what should we do about it?

This really depends on the data, and what you're trying to use it for. In this case, let's assume we're trying to figure out what's a reasonable expectation of salary for a graduate of our school to earn. Ignoring for the moment that we have an extremly small dataset on which to base our judgement, it looks as if Rosie's salary could be either an error (maybe she mis-typed it in the form used to collect data) or a genuine anomaly (maybe she became a professional athelete or some other extremely highly paid job). Either way, it doesn't seem to represent a salary that a typical graduate might earn.

Let's see what the distribution of the data looks like without the outlier:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8), showfliers=False)
plt.show()
```

Now it looks like there's a more even distribution of salaries. It's still not quite symmetrical, but there's much less overall variance. There's potentially some cause here to disregard Rosie's salary data when we compare the salaries, as it is tending to skew the analysis.

So is that OK? Can we really just ignore a data value we don't like?

Again, it depends on what you're analyzing. Let's take a look at the distribution of final grades:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

Once again there are outliers, this time at both ends of the distribution. However, think about what this data represents. If we assume that the grade for the final test is based on a score out of 100, it seems reasonable to expect that some students will score very low (maybe even 0) and some will score very well (maybe even 100); but most will get a score somewhere in the middle.  The reason that the low and high scores here look like outliers might just be because we have so few data points. Let's see what happens if we include a few more students in our data:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Plot a box-whisker chart
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

With more data, there are some more high and low scores; so we no longer consider the isolated cases to be outliers.

The key point to take away here is that you need to really understand the data and what you're trying to do with it, and you need to ensure that you have a reasonable sample size, before determining what to do with outlier values.

#### Variance and Standard Deviation
We've seen how to understand the *spread* of our data distribution using the range, percentiles, and quartiles; and we've seen the effect of outliers on the distribution. Now it's time to look at how to measure the amount of variance in the data.

##### Variance
Variance is measured as the average of the squared difference from the mean. For a full population, it's indicated by a squared Greek letter *sigma* (***σ2***) and calculated like this:

\begin{equation}\sigma^{2} = \frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}\end{equation}

For a sample, it's indicated as ***s2*** calculated like this:

\begin{equation}s^{2} = \frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}}{n-1}\end{equation}

In both cases, we sum the difference between the individual data values and the mean and square the result. Then, for a full population we just divide by the number of data items to get the average. When using a sample, we divide by the total number of items **minus 1** to correct for sample bias.

Let's work this out for our student grades (assuming our data is a sample from the larger student population).

First, we need to calculate the mean grade:

\begin{equation}\bar{x} = \frac{50+50+46+95+50+5+57}{7}\approx 50.43\end{equation}

Then we can plug that into our formula for the variance:

\begin{equation}s^{2} = \frac{(50-50.43)^{2}+(50-50.43)^{2}+(46-50.43)^{2}+(95-50.43)^{2}+(50-50.43)^{2}+(5-50.43)^{2}+(57-50.43)^{2}}{7-1}\end{equation}

So:

\begin{equation}s^{2} = \frac{0.185+0.185+19.625+1986.485+0.185+2063.885+43.165}{6}\end{equation}

Which simplifies to:

\begin{equation}s^{2} = \frac{4113.715}{6}\end{equation}

Giving the result:

\begin{equation}s^{2} \approx 685.619\end{equation}

The higher the variance, the more spread your data is around the mean.

In Python, you can use the ***var*** function of the *pandas.dataframe* class to calculate the variance of a column in a dataframe:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(df['Grade'].var())
```

##### Standard Deviation
To calculate the variance, we squared the difference of each value from the mean. If we hadn't done this, the numerator of our fraction would always end up being zero (because the mean is at the center of our values). However, this means that the variance is not in the same unit of measurement as our data - in our case, since we're calculating the variance for grade points, it's in grade points squared; which is not very helpful.

To get the measure of variance back into the same unit of measurement, we need to find its square root:

\begin{equation}s = \sqrt{685.619} \approx 26.184\end{equation}

So what does this value represent?

It's the *standard deviation* for our grades data. More formally, it's calculated like this for a full population:

\begin{equation}\sigma = \sqrt{\frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}}\end{equation}

Or like this for a sample:

\begin{equation}s = \sqrt{\frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}}{n-1}}\end{equation}

Note that in both cases, it's just the square root of the corresponding variance forumla!

In Python, you can calculate it using the ***std*** function:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(df['Grade'].std())
```

#### Standard Deviation in a Normal Distribution

In statistics and data science, we spend a lot of time considering *normal* distributions; because they occur so frequently. The standard deviation has an important relationship to play in a normal distribution.

Run the following cell to show a histogram of a *standard normal* distribution (which is a distribution with a mean of 0 and a standard deviation of 1):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

# Create a random standard normal distribution
df = pd.DataFrame(np.random.randn(100000, 1), columns=['Grade'])

# Plot the distribution as a histogram with a density curve
grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, color='lightgrey', normed=True, bins=100)
plt.plot(x, density(x))

# Get the mean and standard deviation
s = df['Grade'].std()
m = df['Grade'].mean()

# Annotate 1 stdev
x1 = [m-s, m+s]
y1 = [0.25, 0.25]
plt.plot(x1,y1, color='magenta')
plt.annotate('1s (68.26%)', (x1[1],y1[1]))

# Annotate 2 stdevs
x2 = [m-(s*2), m+(s*2)]
y2 = [0.05, 0.05]
plt.plot(x2,y2, color='green')
plt.annotate('2s (95.45%)', (x2[1],y2[1]))

# Annotate 3 stdevs
x3 = [m-(s*3), m+(s*3)]
y3 = [0.005, 0.005]
plt.plot(x3,y3, color='orange')
plt.annotate('3s (99.73%)', (x3[1],y3[1]))

# Show the location of the mean
plt.axvline(grade.mean(), color='grey', linestyle='dashed', linewidth=1)

plt.show()
```

The horizontal colored lines show the percentage of data within 1, 2, and 3 standard deviations of the mean (plus or minus).

In any normal distribution:
- Approximately 68.26% of values fall within one standard deviation from the mean.
- Approximately 95.45% of values fall within two standard deviations from the mean.
- Approximately 99.73% of values fall within three standard deviations from the mean.

#### Z Score
So in a normal (or close to normal) distribution, standard deviation provides a way to evaluate how far from a mean a given range of values falls, allowing us to compare where a particular value lies within the distribution. For example, suppose Rosie tells you she was the highest scoring student among her friends - that doesn't really help us assess how well she scored. She may have scored only a fraction of a point above the second-highest scoring student. Even if we know she was in the top quartile; if we don't know how the rest of the grades are distributed it's still not clear how well she performed compared to her friends.

However, if she tells you how many standard deviations higher than the mean her score was, this will help you compare her score to that of her classmates.

So how do we know how many standard deviations above or below the mean a particular value is? We call this a *Z Score*, and it's calculated like this for a full population:

\begin{equation}Z = \frac{x - \mu}{\sigma}\end{equation}

or like this for a sample:

\begin{equation}Z = \frac{x - \bar{x}}{s}\end{equation}

So, let's examine Rosie's grade of 95. Now that we know the *mean* grade is 50.43 and the *standard deviation* is 26.184, we can calculate the Z Score for this grade like this:

\begin{equation}Z = \frac{95 - 50.43}{26.184} = 1.702\end{equation}.

So Rosie's grade is 1.702 standard deviations above the mean.

### Summarizing Data Distribution in Python
We've seen how to obtain individual statistics in Python, but you can also use the ***describe*** function to retrieve summary statistics for all numeric columns in a dataframe. These summary statistics include many of the statistics we've examined so far (though it's worth noting that the *median* is not included):

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
print(df.describe())
```


# Homework 01

Congratulations! You've managed to open this Juypter notebook on either Github or on your local machine.

Help for Jupyter Notebooks can be found in the Jupyter Lab by going to `Help > Notebook Reference`. You can also go to the [Notebook basics](https://jupyter-notebook.readthedocs.io/en/latest/examples/Notebook/Notebook%20Basics.html) documentation.

The basics are that you can write *markdown* cells that have math, words, and other markdown (like Rmarkdown!)... and *code* cells that, well, have code and display the output below (as in Rmarkdown). Switch *mode* to change between the two (sidebar will change colors).

If you want to export your document to .pdf (you don't have to!), you an go to `File > Export Notebook As... > PDF`. To do this, I had to install the *Inkscape* application. (I did this with the [Chocolatey](https://chocolatey.org/) package manager on Windows. You can probably do this with [Homewbrew](https://brew.sh/) on a Mac)

# Instructions

Your homework is to generate 3d plots of a quadratic function in $\mathbb R^2$ and to examine the relationship between eigenvalues of the Hessian matrices, shapes of the functions, and the (possible) existence of minima and maxima.

You can find the documentation for `Plots.jl` at <http://docs.juliaplots.org/latest/>

For the following functions

\begin{align}
    f^a(x,y) &= -x^2       - y^2 \\
    f^b(x,y) &= -x^2 +  xy - y^2 \\
    f^c(x,y) &= -x^2 + 2xy - y^2 \\
    f^d(x,y) &= -x^2 + 3xy - y^2
\end{align}

1. Write the Hessian matrix in \LaTeX

2. Compute the determinants by hand. Are the Hessians PD, PSD, NSD, or ND? What does this imply about convexity / concavity of the function? What about the existence of a minimum or maximum over the domain $\mathbb R^2$?

3. `@assert` statements are wonderful to include in your functions because they make sure that the inputs meet certain assumptions... such as that the length of two vectors is the same. Using them regularly can help you avoid errors

    Use an `@assert` statement to check that your determinants computed by hand are correct. See what Julia does when you put the wrong determinanmt in. See [`LinearAlgebra.det`](https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/index.html#LinearAlgebra.det) docs

    ```julia
    @assert det(Ha) == ???
    ```

4. Compute the eigenvalues of your matrix using [`LinearAlgebra.eigvals`](https://docs.julialang.org/en/v1/stdlib/LinearAlgebra/index.html#LinearAlgebra.eigvals)

5. Create a function in Julia to compute $f^a, f^b, f^c, f^d$ as above. Plot them!

To submit your homework, commit this notebook to your personal homework repo, push it, and issue a pull request to turn it in to me.

# Top of your Julia Code

```julia
# Load libraries first of all
using LinearAlgebra  # load LinearAlgebra standard library
using Plots          # loads the Plots module
```

```julia
# tells Plots to use the GR() backend.
#    Note: for interactive 3d plots, you can also install
#    PyPlot or PlotlyJS and try using those. You might
#    need to use Atom or the REPL to get the interactivity
gr()
```

    Plots.GRBackend()

```julia
# define a range we can iterate over
xrange = -3.0 : 0.1 : 3.0
```

    -3.0:0.1:3.0

# Question 1

$$
f^a(x,y) = -x^2 - y^2
$$

## Part 1a

Hessian is $H^a = \begin{bmatrix} ? & ? \\ ? & ? \end{bmatrix}$

## Part 1b

Determinant is $|H^a| = ?$, so the matrix is ???, and it has a global minimum / maximum at ????

```julia
# define the Hessian for H^a.
# Note:
#     Julia in Jupyter Notebooks and Atom can handle latexy characters
#     I got fancy by typing H\^a [tab] and getting a superscript
#     We could have also gotten greek letters with \beta [tab]
#     or (very important) approximately equals with \approx [tab]

Hᵃ = [-2 4 ; 2 3]
```

    2×2 Array{Int64,2}:
     -2  4
      2  3

## Part 1c

```julia
@assert det(Hᵃ) == 3
```

## Part 1d

```julia
eigvals(Hᵃ)
```

    2-element Array{Float64,1}:
     -3.274917217635375
      4.274917217635375

```julia
# functions to plot
fa(x,y) = -x^2 - y^2
```

    fa (generic function with 1 method)

```julia
plot(xrange, xrange, fa, st = :surface, title = "\$a=0\$")
```

# Question 2

# Question 3

# Question 4


## Probability and Computing - Mitzenmacher, Upfal

```python
import numpy as np
from sympy import poly
from sympy.abc import x
```

Define two polynomials:
- $F$ is a product of monomials
- $G$ is in canonical form

```python
F = poly((x + 1) * (x - 2) * (x + 3) * (x - 4) * (x + 5) * (x - 6))
G = poly(x**6 - 7 * x**3 + 25)
```

Want to check if $F \equiv G$ without converting $F$ to canonical form.

```python
def polycheck(F: 'poly', G: 'poly', δ: int, k: int, replacement: bool) -> bool:
    """Randomized algorithm for verifying whether F and G are equivalent.

    If F ≡ G, then the algo always computes the correct answer.
    If F ≢ G, then the algo can compute the wrong answer by
        finding r s.t. r is the root of F(x) - G(x) = 0,
        which, by FTA, can happen at most in d / (δ * d) cases,
        meaning that the prob. of error (in one iter) is <= 1/δ.

    If sampling is performed WITH replacement, then iterations are independent,
        therefore, the probability of error becomes <= (1/δ)**k
        i.e. exponentially small in the number of trials.
    If sampling is performed WITHOUT replacement, we get a tighter bound <= (1/δ)**k,
        since the error now consists of the event "finding k distinct roots",
        which is much stronger.

    Args:
        F, G: sympy polynomials.
        δ: upper bound for the sample space.
        k: number of iterations (trials).
        replacement: whether to perform sampling with or without replacement.

    Returns:
        True if F,G are found to be equivalent, otherwise False.
    """
    d = max(F.degree(), G.degree())
    space = np.arange(1, δ * d + 1)  # {1, ..., δ * d}

    # choose values uniformly at random from the sample space
    rs = np.random.choice(space, replace=replacement, size=k)

    for r in rs:
        if F(r) != G(r):
            return False

    return True
```

```python
polycheck(F, G, δ=100, k=10, replacement=False)
```


Data Science Notebooks without math are lame.

Thankfully Jupyter Notebooks support LatEx

This cheatsheet is neither comprehensive nor complete. For now, it is only a placeholder. My plans are to grow it over time to be a most useful stuff in one place with cut and paste snapshots, but without explicitly spending much time in its development

Here are some references:

http://bebi103.caltech.edu.s3-website-us-east-1.amazonaws.com/2015/tutorials/t0c_intro_to_latex.html

https://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Typesetting%20Equations.html

https://oeis.org/wiki/List_of_LaTeX_mathematical_symbols

Inline as follows
Joint Probability is the probability ofntersection of two or more events. Written as $ P(A \cap B) , P(A,B), P(A\ and\ B)$

Block equation as follows

\begin{align*}
\text{good: } P(A \cap B) =  P(B)\,P(A \mid B)
\end{align*}

\begin{align*}
\text{not good: } P(A \cap B) =  P(B) P(A | B)
\end{align*}

Multiline as follows. Presence of align* messes up *sometimes*. Just align

\begin{align}
\text{good: } P(A \cap B) =  P(B)\,P(A \mid B) \\
\text{not good: } P(A \cap B) =  P(B) P(A | B)
\end{align}

Another example of multi line equations
\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

Multi functions
\begin{align*}
\min_{x \in R^{n}}\ f(x)\ subject\ to\  \left\{\begin{array}{lll}
                c_{i}(x)\ = \ 0,\  i \ \in \varepsilon, \\
                c_{i}(x)\ \geq \ 0,\  i\in \iota
            \end{array}\right.
\end{align*}

Bolded equation
\begin{equation*},
\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right) \end{equation*}

Providing equation number on right

\begin{equation}
\begin{aligned}
&x = 1 \\
&y = 2 + 2x^2
\end{aligned}
\tag{fun system}
\end{equation}

```python

```


```python
##Import and define symbols
import sympy as sp
import numpy as np

Ti = sp.Symbol('T_i'); k = sp.Symbol('k'); To = sp.Symbol('T_o'); Ti0 = sp.Symbol('T_i_0'); To0 = sp.Symbol('T_o_0'); ho = sp.Symbol('h_o'); hi = sp.Symbol('h_i'); r = sp.Symbol('r'); ro = sp.Symbol('r_o'); ri = sp.Symbol('r_i'); Cp = sp.Symbol('C_p'); rho = sp.Symbol('rho');
T = sp.Function('T')(r)
U = sp.Function('U')(r)

C1 = sp.Symbol('C1'); C2 = sp.Symbol('C2')
```

# Problem 2.1.1 in the problems manual

## Nomenclature table
| Nomenclature                         | Variable | Expression              |
|--------------------------------------|----------|-------------------------|
| Temperature                          | T        |                         |
| Radius                               | r        |                         |
| Convective heat transfer coefficient | h        |                         |
| Conductive heat transfer coefficient | k        |                         |
| Biot number                          | Bi       | $\frac{hR}{k}$          |
| Temperature fraction                 | $\phi$   | $\frac{T-T_o}{T_i-T_o}$ |
| Quantity "X" of internal fluid       | $X_i$    |                         |
| Quantity "X" of external fluid       | $X_o$    |                         |

## Simplifying assumptions:
1. Steady state; $\frac{dT}{dt} = 0$
2. Infinite symetric cylinder; $\frac{dT}{dz} = \frac{dT}{d\theta} = 0$; $T(r)$
3. No heat generation within the clinder; $q''' = 0$

## Differential conservation equation solution
The consititutive equation for cylindrical coordinates:
$$\rho c \frac{dT}{dt}= \frac{1}{r}\frac{d}{dr}(r\cdot k\frac{dT}{dr})+\frac{1}{r}\frac{d}{d\theta}(k\frac{dT}{d\theta})+\frac{d}{dz}(k\frac{dT}{dz})+q'''$$

When assumptions are applied:

$$0 =\frac{d^2T}{dr^2}+\frac{1}{r}\frac{dT}{dr}$$

The boundary conditions for convective heat transfer at the walls:

$$\frac{dT}{dr}(r = r_o) = \frac{h_o}{k}[T_o - T(r = r_o)]$$

$$\frac{dT}{dr}(r = r_i) = \frac{-h_i}{k}[T_i - T(r = r_i)]$$

Substituting the derivative of temperature $\frac{dT}{dr} = U(r)$ into the constitutive equation:

$$0 = \frac{dU(r)}{dr} + \frac{1}{r}\cdot U(r)$$

Seperating and integrating:

$$U(r) = \frac{dT}{dr} = \frac{c_1}{r}$$

And again:

$$T(r) = c_1\ln{r} + c_2$$

Substituting in the temperature equations into the boundary conditions yields a system of two equations and unkowns $c_1, c_2$:

$$\frac{c_1}{r_o} = \frac{h_o}{k}[T_o - (c_1\ln{r_o} + c_2)]$$

$$\frac{c_1}{r_i} = \frac{-h_i}{k}[T_i - (c_1\ln{r_i} + c_2)]$$

```python
## Solve DE
#Define equation with U
eqn = (sp.Derivative(U,r)+1/r*U)
print('System differential equation with substitution for derivative of temperature:')
display(eqn)

#Solve DE for derivative of temperature (U)
Diff_U = sp.dsolve(eqn, U)
print('Expression for differential in temperature with respect to r:')
display(Diff_U)

#Redefine Temperature
Diff_T = Diff_U.subs(U, sp.Derivative(T,r))
print('Differential equation for temperature:')
display(Diff_T)

#Solve for temperature
Temp = sp.dsolve(Diff_T, T)
print('Solved expression for temperature with integration constants:')
display(Temp)
```

    System differential equation with substitution for derivative of temperature:

$\displaystyle \frac{d}{d r} U{\left(r \right)} + \frac{U{\left(r \right)}}{r}$

    Expression for differential in temperature with respect to r:

$\displaystyle U{\left(r \right)} = \frac{C_{1}}{r}$

    Differential equation for temperature:

$\displaystyle \frac{d}{d r} T{\left(r \right)} = \frac{C_{1}}{r}$

    Solved expression for temperature with integration constants:

$\displaystyle T{\left(r \right)} = C_{1} \log{\left(r \right)} + C_{2}$

```python
#Define the two boundary conditions
eqn1= ho/k*(To-(Temp.rhs.subs(r, ro)))-Diff_U.rhs.subs(r, ro)
eqn2= -hi/k*(Ti-(Temp.rhs.subs(r, ri)))-Diff_U.rhs.subs(r, ri)

print('First Equation')
display(eqn1)

print('Second Equation')
display(eqn2)

#Solve for c1 and c2
C1_ = sp.solve(eqn1,C1)[0]
C2_ = sp.solve(eqn2,C2)[0]
C1eq = C1_.subs(C2,C2_)-C1
C1_ = sp.simplify(sp.solve(C1eq,C1)[0])
C2_ = sp.simplify(C2_.subs(C1,C1_))

#Define biot numbers
Bi_i = sp.Symbol('Bi_i')
Bi_o = sp.Symbol('Bi_o')

#substitute biot numbers into the equation
C1_ = sp.simplify((C1_.subs(hi*ri, Bi_i*k)).subs(ho*ro, Bi_o*k))
C2_ = sp.simplify((C2_.subs(hi*ri, Bi_i*k)).subs(ho*ro, Bi_o*k))

print('C1 solved')
display(C1_)
print('C2 solved')
display(C2_)
```

    First Equation

$\displaystyle - \frac{C_{1}}{r_{o}} + \frac{h_{o} \left(- C_{1} \log{\left(r_{o} \right)} - C_{2} + T_{o}\right)}{k}$

    Second Equation

$\displaystyle - \frac{C_{1}}{r_{i}} - \frac{h_{i} \left(- C_{1} \log{\left(r_{i} \right)} - C_{2} + T_{i}\right)}{k}$

    C1 solved

$\displaystyle \frac{Bi_{i} Bi_{o} \left(T_{i} - T_{o}\right)}{Bi_{i} Bi_{o} \log{\left(r_{i} \right)} - Bi_{i} Bi_{o} \log{\left(r_{o} \right)} - Bi_{i} - Bi_{o}}$

    C2 solved

$\displaystyle \frac{Bi_{i} Bi_{o} T_{i} \log{\left(r_{o} \right)} - Bi_{i} Bi_{o} T_{o} \log{\left(r_{i} \right)} + Bi_{i} T_{i} + Bi_{o} T_{o}}{- Bi_{i} Bi_{o} \log{\left(r_{i} \right)} + Bi_{i} Bi_{o} \log{\left(r_{o} \right)} + Bi_{i} + Bi_{o}}$

With $Bi = \frac{hR}{k}$

Defining dimensionless parameter $\phi (r) = \frac{T(r)-T_o}{T_i-T_o}$ and solving for $\phi$

$$\phi(r) = \frac{c_1\ln{r}+c_2-T_o}{T_i-T_o}$$

## Investigating this behavior:

```python
##Set some constants for r for a few cases
#Thick wall vs thin wall
import numpy as np
import matplotlib.pyplot as plt

r_i = 1
r_o = np.array([10, 5, 1.1])

#Investigate outside biot for constant inside biot
Bi_i = 1
Bi_o = np.array([0.01, 1, 10])
T_i = 100
T_o = 200

for j, R_o in enumerate(r_o):
    rs = np.linspace(r_i, R_o, 100)
    phis = np.zeros((len(Bi_o), len(rs)))
    for k, Bi_out in enumerate(Bi_o):
        c1 = Bi_i*Bi_out*(T_i-T_o)/(Bi_i*Bi_out*np.log(r_i)-Bi_i*Bi_out*np.log(R_o)-Bi_i-Bi_out)
        c2 = (Bi_i*Bi_out*T_i*np.log(R_o)-Bi_i*Bi_out*T_o*np.log(r_i)+Bi_i*T_i+Bi_out*T_o)/(-Bi_i*Bi_out*np.log(r_i)+Bi_i*Bi_out*np.log(R_o)+Bi_i+Bi_out)
        #phis[k][:] = (c1*np.log(rs)+c2 - T_o)/(T_i-T_o)
        phis[k][:] = (np.log(rs/R_o))/(np.log(r_i/R_o)+1/Bi_out + 1/Bi_i)
        plt.figure(j)
        plt.plot(rs, phis[k][:],label = 'Bi_o/Bi_i ='+str(Bi_out))
    plt.legend()
    plt.xlabel('r')
    plt.ylabel('phi')
    plt.title('R = '+str(R_o))

```

In interpereting the graphs, it us useful to remember that $\phi = 1$ corresponds to the temperature being equal to the internal air temperature, and $\phi = 0$ corresponds to temoerature being equal to the external air temperature.

## Points to note:
1. For a thin wall (Thickness << cylinder diameter), the internal temperature is nearly constant and determined by the convective coefficients. If convective transfer is much more prominent on the external surface than the internal surface, then the cylinder temperature is equal to the external temperature and vice versa. For comparible convective forces, the cylinder temperature is somewhere in between the two air temperatures.
2. For thin walls, the slight temperature distribution that is exhibited is nearly linear, approcimating this case to a slab wall instead of a cylinder wall.
3. For thick walls (Thickness ~ cylinder diameter), a distribution of temperatures is much more prominent, and the curviture of the cylinder is noted as it is non linear. This is intuitive, as for a cyliner the area of flux increases as radius increases, so temperature change should slow down as radius increases, which we do see.
4. What we note is that the greater a Biot number is compared to the other side of the cylinder, the closer the wall temperature on that side comes to the air temperature. Alternatively, if the Biot numbers are of similar magnitude, the wall temperature on both sides of the cylinder walls do not approach the air temperatures but are instead in between the two.

```python

```


```python
import sympy as sp
from sympy import init_printing
import numpy as np
import matplotlib.pyplot as plt

init_printing()
```

```python
# Define the system
x = (sp.Matrix([sp.symbols('q p', real = True)])).T
m, l , g = sp.symbols('m l g', real = True)
H = 1/2 * x[1]**2 / (l**2 * m) + m*g*l*sp.sin(x[0])
dH = H.diff(x)
dx = sp.Matrix([[0, 1],[-1, 0]]) * dH
dx
```

```python
# Taylor series
dx.taylor_term(1, x)
```

```python
dq = np.arange(0, 2*np.pi, np.pi/20)
dp = np.arange(-10, 10, 0.1)
DQ, DP = np.meshgrid(dq, dp)
# Make a func
dx_f = sp.lambdify((x[0], x[1]),dx.subs([(m, 1), (g, -9.81), (l, 1)]) )
```

```python
dt = sp.symbols('t')
J = (sp.eye(2) + dt*dx.jacobian(x))
J_E = J.subs(([(m, 1), (g, -9.81), (l, 1), (dt, 0.1)]) )
e1, e2 = J_E.eigenvals().items()
eigenvals_n = sp.lambdify(x[0], np.array([e1[0], e2[0]]))
```

```python
e1, e2 = eigenvals_n(dq)
plt.plot(dq, e1)
plt.plot(dq, e2)
```

```python

```


The error in a cross product calculation with (3,) tuples

```
from sympy import Symbol, symarray, Matrix, matrices
u = Matrix(symarray('u', (3, 1)))
v = Matrix(symarray('v', (3, 1)))
```

```
c = u.cross(v)
c
```

    [u_1_0*v_2_0 - u_2_0*v_1_0, -u_0_0*v_2_0 + u_2_0*v_0_0, u_0_0*v_1_0 - u_1_0*v_0_0]

Assuming same error $\delta$ for both vectors:

```
d = Symbol('d')
e = matrices.ones((3,1)) * d
```

Same calculation as above, with error

```
ue = u + e
ve = v + e
ce = ue.cross(ve)
```

Calculate absolute error by subtracting true result from result with error

```
cce = ce - c
cce.simplify()
cce
```

    [d*(u_1_0 - u_2_0 - v_1_0 + v_2_0), d*(-u_0_0 + u_2_0 + v_0_0 - v_2_0), d*(u_0_0 - u_1_0 - v_0_0 + v_1_0)]

Floating point calculation error given by operations on elements:

```
c
```

    [u_1_0*v_2_0 - u_2_0*v_1_0, -u_0_0*v_2_0 + u_2_0*v_0_0, u_0_0*v_1_0 - u_1_0*v_0_0]

Each element has two products and one subtraction; The input values are $\le 1$.  Calculation error per element then $3 \epsilon / 2$


```python
import sympy as sp
from sympy.physics.quantum import TensorProduct
import numpy as np

print(f"SymPy Version: {sp.__version__}")

# 数式をキレイに表示する
sp.init_printing()
```

    SymPy Version: 1.8

### ベクトルを生成する

- ベクトルの生成には、`sympy.Matrix`を使用する。
  + 生成されたベクトルは列ベクトルになる。
  + `ベクトル.T`とすることで、行ベクトルを作ることができる(いわゆる、行列の転置に相当)

```python
vec1 = sp.Matrix([0, 1, 2, 3])
vec1
```

$\displaystyle \left[\begin{matrix}0\\1\\2\\3\end{matrix}\right]$

```python
vec1.T
```

$\displaystyle \left[\begin{matrix}0 & 1 & 2 & 3\end{matrix}\right]$

```python
# numpyからベクトルを作ることも出来る
sp.Matrix(np.arange(5))
```

$\displaystyle \left[\begin{matrix}0\\1\\2\\3\\4\end{matrix}\right]$

### 行列を生成する

- 行列の生成においても、`sympy.Matrix`を使用する。
- 単位行列は、`sympy.eye()`で生成できる。
- ゼロ行列は、`sympy.zeros()`で生成できる。
- 対角行列は、`sympy.diag()`で生成できる。
- 行列のシンボルを使用する場合、`sympy.MatrixSymbol`を使用する。

```python
mat = sp.Matrix([[1, 2], [3, 4]])
mat
```

$\displaystyle \left[\begin{matrix}1 & 2\\3 & 4\end{matrix}\right]$

```python
sp.eye(3)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$

```python
sp.zeros(2, 5)
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0\end{matrix}\right]$

```python
sp.diag(1, 0, 2, 4)
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & 0\\0 & 0 & 0 & 0\\0 & 0 & 2 & 0\\0 & 0 & 0 & 4\end{matrix}\right]$

```python
mat2 = sp.MatrixSymbol('X', 3, 4)
mat2 = sp.Matrix(mat2)
mat2
```

$\displaystyle \left[\begin{array}{cccc}X_{0, 0} & X_{0, 1} & X_{0, 2} & X_{0, 3}\\X_{1, 0} & X_{1, 1} & X_{1, 2} & X_{1, 3}\\X_{2, 0} & X_{2, 1} & X_{2, 2} & X_{2, 3}\end{array}\right]$

```python
# 行列の要素にアクセスもできる
mat2[0, 0] = 1
mat2
```

$\displaystyle \left[\begin{array}{cccc}1 & X_{0, 1} & X_{0, 2} & X_{0, 3}\\X_{1, 0} & X_{1, 1} & X_{1, 2} & X_{1, 3}\\X_{2, 0} & X_{2, 1} & X_{2, 2} & X_{2, 3}\end{array}\right]$

### ベクトルの演算

- 加算、乗算は`+`, `-`演算子で実行できる。
- 定数を掛けることで、ベクトルのスカラー倍も可能
- 内積には、`v1.dot(v2)`を用いる。
- 要素ごとの乗算(アダマール積)には、`v1.multiply_elementwise(v2)`を用いる。
- ベクトルの大きさは、`v1.norm()`で計算できる。

```python
vec1 = sp.Matrix(np.arange(4))
vec2 = sp.Matrix(np.arange(4))
```

```python
vec1 + vec2
```

$\displaystyle \left[\begin{matrix}0\\2\\4\\6\end{matrix}\right]$

```python
2 * vec1 - 0.5 * vec2
```

$\displaystyle \left[\begin{matrix}0\\1.5\\3.0\\4.5\end{matrix}\right]$

```python
# 内積
vec1.dot(vec2)
```

```python
# 要素ごとの乗算 (アダマール積)
vec1.multiply_elementwise(vec2)
```

$\displaystyle \left[\begin{matrix}0\\1\\4\\9\end{matrix}\right]$

```python
# ベクトルの大きさ
vec1.norm()
```

### 行列の演算

- 加算、乗算は`+`, `-`演算子で実行できる。
- `*`演算子は、行列積となる。
- `mat.T`で、転置行列を計算できる。
- `mat.inv()`で、逆行列を計算できる。
- `mat.det()`で、行列式を計算できる。

```python
mat1 = sp.Matrix(np.arange(0, 4).reshape(2, 2))
mat1
```

$\displaystyle \left[\begin{matrix}0 & 1\\2 & 3\end{matrix}\right]$

```python
mat2 = sp.Matrix(np.arange(5, 9).reshape(2, 2))
mat2
```

$\displaystyle \left[\begin{matrix}5 & 6\\7 & 8\end{matrix}\right]$

```python
mat1 + mat2
```

$\displaystyle \left[\begin{matrix}5 & 7\\9 & 11\end{matrix}\right]$

```python
2.0 * mat1 - 0.5 * mat2
```

$\displaystyle \left[\begin{matrix}-2.5 & -1.0\\0.5 & 2.0\end{matrix}\right]$

```python
# 行列式
mat1 * mat2
```

$\displaystyle \left[\begin{matrix}7 & 8\\31 & 36\end{matrix}\right]$

```python
# 転置行列
mat1.T
```

$\displaystyle \left[\begin{matrix}0 & 2\\1 & 3\end{matrix}\right]$

```python
# 逆行列
mat1.inv()
```

$\displaystyle \left[\begin{matrix}- \frac{3}{2} & \frac{1}{2}\\1 & 0\end{matrix}\right]$

```python
mat1.inv() * mat1
```

$\displaystyle \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$

```python
# 行列式
mat1.det()
```

### 関数の適用

- `行列.applyfunc()`を使用することで、要素ごとの計算が可能になる。
  + 引数には、関数オブジェク、またはラムダ式を使用する。

```python
x = sp.Matrix(sp.MatrixSymbol('X', 3, 3))
x
```

$\displaystyle \left[\begin{array}{ccc}X_{0, 0} & X_{0, 1} & X_{0, 2}\\X_{1, 0} & X_{1, 1} & X_{1, 2}\\X_{2, 0} & X_{2, 1} & X_{2, 2}\end{array}\right]$

```python
# 単純に2乗すると、行列積となる。
x ** 2
```

$\displaystyle \left[\begin{matrix}X_{0, 0}^{2} + X_{0, 1} X_{1, 0} + X_{0, 2} X_{2, 0} & X_{0, 0} X_{0, 1} + X_{0, 1} X_{1, 1} + X_{0, 2} X_{2, 1} & X_{0, 0} X_{0, 2} + X_{0, 1} X_{1, 2} + X_{0, 2} X_{2, 2}\\X_{0, 0} X_{1, 0} + X_{1, 0} X_{1, 1} + X_{1, 2} X_{2, 0} & X_{0, 1} X_{1, 0} + X_{1, 1}^{2} + X_{1, 2} X_{2, 1} & X_{0, 2} X_{1, 0} + X_{1, 1} X_{1, 2} + X_{1, 2} X_{2, 2}\\X_{0, 0} X_{2, 0} + X_{1, 0} X_{2, 1} + X_{2, 0} X_{2, 2} & X_{0, 1} X_{2, 0} + X_{1, 1} X_{2, 1} + X_{2, 1} X_{2, 2} & X_{0, 2} X_{2, 0} + X_{1, 2} X_{2, 1} + X_{2, 2}^{2}\end{matrix}\right]$

```python
# applyfunc を用いることで、要素ごとに2乗することも可能
x.applyfunc(lambda y: y ** 2)
```

$\displaystyle \left[\begin{matrix}X_{0, 0}^{2} & X_{0, 1}^{2} & X_{0, 2}^{2}\\X_{1, 0}^{2} & X_{1, 1}^{2} & X_{1, 2}^{2}\\X_{2, 0}^{2} & X_{2, 1}^{2} & X_{2, 2}^{2}\end{matrix}\right]$

### ベクトルの直交化

- `symply.GramSchmidt()`を使用することで、グラム・シュミットの正規直交化を行うことができる。
- `symply.Matrix.orthogonalize()`でも、同様の結果が得られる。

```python
vec1 = sp.Matrix([1, 1, 0])
vec2 = sp.Matrix([0, -1, 1])
vec3 = sp.Matrix([1, 1, 1])

vec1, vec2, vec3
```

$\displaystyle \left( \left[\begin{matrix}1\\1\\0\end{matrix}\right], \  \left[\begin{matrix}0\\-1\\1\end{matrix}\right], \  \left[\begin{matrix}1\\1\\1\end{matrix}\right]\right)$

```python
# 内積の結果から、定義したベクトルは直交化していない
print(f"vec1 ・ vec2 = {vec1.dot(vec2)}")
print(f"vec1 ・ vec3 = {vec1.dot(vec3)}")
```

    vec1 ・ vec2 = -1
    vec1 ・ vec3 = 2

```python
vec_list = [vec1, vec2, vec3]
out1, out2, out3 = sp.GramSchmidt(vec_list)
out1, out2, out3
```

$\displaystyle \left( \left[\begin{matrix}1\\1\\0\end{matrix}\right], \  \left[\begin{matrix}\frac{1}{2}\\- \frac{1}{2}\\1\end{matrix}\right], \  \left[\begin{matrix}- \frac{1}{3}\\\frac{1}{3}\\\frac{1}{3}\end{matrix}\right]\right)$

```python
# 内積の結果から、互いに直交しているのがわかる
print(f"out1 ・ out2 = {out1.dot(out2)}")
print(f"out1 ・ out3 = {out1.dot(out3)}")
```

    out1 ・ out2 = 0
    out1 ・ out3 = 0

```python
sp.Matrix.orthogonalize(*vec_list)
```

$\displaystyle \left[ \left[\begin{matrix}1\\1\\0\end{matrix}\right], \  \left[\begin{matrix}\frac{1}{2}\\- \frac{1}{2}\\1\end{matrix}\right], \  \left[\begin{matrix}- \frac{1}{3}\\\frac{1}{3}\\\frac{1}{3}\end{matrix}\right]\right]$

### ベクトルの外積・直積

- 外積は、`vec1.corss(vec2)` で計算できる。
- 直積は、`sympy.physics.quantum.TensorProduct(vec1, vec.T)` で計算できる。

#### [参考] 直積とは？

$$
\boldsymbol{u} \otimes \boldsymbol{u} = \boldsymbol{u} \cdot \boldsymbol{u}^{T}
$$

```python
ux, uy, uz = sp.symbols('ux uy uz')
vx, vy, vz = sp.symbols('vx vy vz')

u = sp.Matrix([ux, uy, uz])
v = sp.Matrix([vx, vy, vz])

u, v
```

$\displaystyle \left( \left[\begin{matrix}ux\\uy\\uz\end{matrix}\right], \  \left[\begin{matrix}vx\\vy\\vz\end{matrix}\right]\right)$

```python
# これは内積
u.dot(v)
```

```python
# 外積の計算
u.cross(v)
```

$\displaystyle \left[\begin{matrix}uy vz - uz vy\\- ux vz + uz vx\\ux vy - uy vx\end{matrix}\right]$

```python
# 直積の計算
TensorProduct(u, u.T)
```

$\displaystyle \left[\begin{matrix}ux^{2} & ux uy & ux uz\\ux uy & uy^{2} & uy uz\\ux uz & uy uz & uz^{2}\end{matrix}\right]$

### 行列の固有値と固有ベクトル

- 行列の固有値は、`行列.eigenvals()`で計算できる。
- また、固有ベクトルは、`行列.eigenvects()`で計算できる。

```python
A = sp.Matrix([[5, 3], [4, 9]])
A
```

$\displaystyle \left[\begin{matrix}5 & 3\\4 & 9\end{matrix}\right]$

```python
A.eigenvals()
```

```python
A.eigenvects()
```

$\displaystyle \left[ \left( 3, \  1, \  \left[ \left[\begin{matrix}- \frac{3}{2}\\1\end{matrix}\right]\right]\right), \  \left( 11, \  1, \  \left[ \left[\begin{matrix}\frac{1}{2}\\1\end{matrix}\right]\right]\right)\right]$

```python

```


# Classical Physics Models
### Yingbo Ma, Chris Rackauckas

If you're getting some cold feet to jump in to DiffEq land, here are some handcrafted differential equations mini problems to hold your hand along the beginning of your journey.

## Radioactive Decay of Carbon-14

#### First order linear ODE

$$f(t,u) = \frac{du}{dt}$$

The Radioactive decay problem is the first order linear ODE problem of an exponential with a negative coefficient, which represents the half-life of the process in question. Should the coefficient be positive, this would represent a population growth equation.

```julia
using OrdinaryDiffEq, Plots
gr()

#Half-life of Carbon-14 is 5,730 years.
C₁ = 5.730

#Setup
u₀ = 1.0
tspan = (0.0, 1.0)

#Define the problem
radioactivedecay(u,p,t) = -C₁*u

#Pass to solver
prob = ODEProblem(radioactivedecay,u₀,tspan)
sol = solve(prob,Tsit5())

#Plot
plot(sol,linewidth=2,title ="Carbon-14 half-life", xaxis = "Time in thousands of years", yaxis = "Percentage left", label = "Numerical Solution")
plot!(sol.t, t->exp(-C₁*t),lw=3,ls=:dash,label="Analytical Solution")
```

## Simple Pendulum

#### Second Order Linear ODE

We will start by solving the pendulum problem. In the physics class, we often solve this problem by small angle approximation, i.e. $ sin(\theta) \approx \theta$, because otherwise, we get an elliptic integral which doesn't have an analytic solution. The linearized form is

$$\ddot{\theta} + \frac{g}{L}{\theta} = 0$$

But we have numerical ODE solvers! Why not solve the *real* pendulum?

$$\ddot{\theta} + \frac{g}{L}{\sin(\theta)} = 0$$

```julia
# Simple Pendulum Problem
using OrdinaryDiffEq, Plots

#Constants
const g = 9.81
L = 1.0

#Initial Conditions
u₀ = [0,π/2]
tspan = (0.0,6.3)

#Define the problem
function simplependulum(du,u,p,t)
    θ  = u[1]
    dθ = u[2]
    du[1] = dθ
    du[2] = -(g/L)*sin(θ)
end

#Pass to solvers
prob = ODEProblem(simplependulum,u₀, tspan)
sol = solve(prob,Tsit5())

#Plot
plot(sol,linewidth=2,title ="Simple Pendulum Problem", xaxis = "Time", yaxis = "Height", label = ["Theta","dTheta"])
```

So now we know that behaviour of the position versus time. However, it will be useful to us to look at the phase space of the pendulum, i.e., and representation of all possible states of the system in question (the pendulum) by looking at its velocity and position. Phase space analysis is ubiquitous in the analysis of dynamical systems, and thus we will provide a few facilities for it.

```julia
p = plot(sol,vars = (1,2), xlims = (-9,9), title = "Phase Space Plot", xaxis = "Velocity", yaxis = "Position", leg=false)
function phase_plot(prob, u0, p, tspan=2pi)
    _prob = ODEProblem(prob.f,u0,(0.0,tspan))
    sol = solve(_prob,Vern9()) # Use Vern9 solver for higher accuracy
    plot!(p,sol,vars = (1,2), xlims = nothing, ylims = nothing)
end
for i in -4pi:pi/2:4π
    for j in -4pi:pi/2:4π
        phase_plot(prob, [j,i], p)
    end
end
plot(p,xlims = (-9,9))
```

## Simple Harmonic Oscillator

### Double Pendulum

```julia
#Double Pendulum Problem
using OrdinaryDiffEq, Plots

#Constants and setup
const m₁, m₂, L₁, L₂ = 1, 2, 1, 2
initial = [0, π/3, 0, 3pi/5]
tspan = (0.,50.)

#Convenience function for transforming from polar to Cartesian coordinates
function polar2cart(sol;dt=0.02,l1=L₁,l2=L₂,vars=(2,4))
    u = sol.t[1]:dt:sol.t[end]

    p1 = l1*map(x->x[vars[1]], sol.(u))
    p2 = l2*map(y->y[vars[2]], sol.(u))

    x1 = l1*sin.(p1)
    y1 = l1*-cos.(p1)
    (u, (x1 + l2*sin.(p2),
     y1 - l2*cos.(p2)))
end

#Define the Problem
function double_pendulum(xdot,x,p,t)
    xdot[1]=x[2]
    xdot[2]=-((g*(2*m₁+m₂)*sin(x[1])+m₂*(g*sin(x[1]-2*x[3])+2*(L₂*x[4]^2+L₁*x[2]^2*cos(x[1]-x[3]))*sin(x[1]-x[3])))/(2*L₁*(m₁+m₂-m₂*cos(x[1]-x[3])^2)))
    xdot[3]=x[4]
    xdot[4]=(((m₁+m₂)*(L₁*x[2]^2+g*cos(x[1]))+L₂*m₂*x[4]^2*cos(x[1]-x[3]))*sin(x[1]-x[3]))/(L₂*(m₁+m₂-m₂*cos(x[1]-x[3])^2))
end

#Pass to Solvers
double_pendulum_problem = ODEProblem(double_pendulum, initial, tspan)
sol = solve(double_pendulum_problem, Vern7(), abs_tol=1e-10, dt=0.05);
```

```julia
#Obtain coordinates in Cartesian Geometry
ts, ps = polar2cart(sol, l1=L₁, l2=L₂, dt=0.01)
plot(ps...)
```

### Poincaré section

The Poincaré section is a contour plot of a higher-dimensional phase space diagram. It helps to understand the dynamic interactions and is wonderfully pretty.

The following equation came from [StackOverflow question](https://mathematica.stackexchange.com/questions/40122/help-to-plot-poincar%C3%A9-section-for-double-pendulum)

$$\frac{d}{dt}
 \begin{pmatrix}
 \alpha \\ l_\alpha \\ \beta \\ l_\beta
 \end{pmatrix}=
 \begin{pmatrix}
  2\frac{l_\alpha - (1+\cos\beta)l_\beta}{3-\cos 2\beta} \\
  -2\sin\alpha - \sin(\alpha + \beta) \\
  2\frac{-(1+\cos\beta)l_\alpha + (3+2\cos\beta)l_\beta}{3-\cos2\beta}\\
  -\sin(\alpha+\beta) - 2\sin(\beta)\frac{(l_\alpha-l_\beta)l_\beta}{3-\cos2\beta} + 2\sin(2\beta)\frac{l_\alpha^2-2(1+\cos\beta)l_\alpha l_\beta + (3+2\cos\beta)l_\beta^2}{(3-\cos2\beta)^2}
 \end{pmatrix}$$

The Poincaré section here is the collection of $(β,l_β)$ when $α=0$ and $\frac{dα}{dt}>0$.

#### Hamiltonian of a double pendulum
Now we will plot the Hamiltonian of a double pendulum

```julia
#Constants and setup
using OrdinaryDiffEq
initial2 = [0.01, 0.005, 0.01, 0.01]
tspan2 = (0.,200.)

#Define the problem
function double_pendulum_hamiltonian(udot,u,p,t)
    α  = u[1]
    lα = u[2]
    β  = u[3]
    lβ = u[4]
    udot .=
    [2(lα-(1+cos(β))lβ)/(3-cos(2β)),
    -2sin(α) - sin(α+β),
    2(-(1+cos(β))lα + (3+2cos(β))lβ)/(3-cos(2β)),
    -sin(α+β) - 2sin(β)*(((lα-lβ)lβ)/(3-cos(2β))) + 2sin(2β)*((lα^2 - 2(1+cos(β))lα*lβ + (3+2cos(β))lβ^2)/(3-cos(2β))^2)]
end

# Construct a ContiunousCallback
condition(u,t,integrator) = u[1]
affect!(integrator) = nothing
cb = ContinuousCallback(condition,affect!,nothing,
                        save_positions = (true,false))

# Construct Problem
poincare = ODEProblem(double_pendulum_hamiltonian, initial2, tspan2)
sol2 = solve(poincare, Vern9(), save_everystep = false, callback=cb, abstol=1e-9)

function poincare_map(prob, u₀, p; callback=cb)
    _prob = ODEProblem(prob.f,[0.01, 0.01, 0.01, u₀],prob.tspan)
    sol = solve(_prob, Vern9(), save_everystep = false, callback=cb, abstol=1e-9)
    scatter!(p, sol, vars=(3,4), markersize = 2)
end
```

```julia
p = scatter(sol2, vars=(3,4), leg=false, markersize = 2, ylims=(-0.01,0.03))
for i in -0.01:0.00125:0.01
    poincare_map(poincare, i, p)
end
plot(p,ylims=(-0.01,0.03))
```

## Hénon-Heiles System

The Hénon-Heiles potential occurs when non-linear motion of a star around a galactic center with the motion restricted to a plane.

$$
\begin{align}
\frac{d^2x}{dt^2}&=-\frac{\partial V}{\partial x}\\
\frac{d^2y}{dt^2}&=-\frac{\partial V}{\partial y}
\end{align}
$$

where

$$V(x,y)={\frac {1}{2}}(x^{2}+y^{2})+\lambda \left(x^{2}y-{\frac {y^{3}}{3}}\right).$$

We pick $\lambda=1$ in this case, so

$$V(x,y) = \frac{1}{2}(x^2+y^2+2x^2y-\frac{2}{3}y^3).$$

Then the total energy of the system can be expressed by

$$E = T+V = V(x,y)+\frac{1}{2}(\dot{x}^2+\dot{y}^2).$$

The total energy should conserve as this system evolves.

```julia
using OrdinaryDiffEq, Plots

#Setup
initial = [0.,0.1,0.5,0]
tspan = (0,100.)

#Remember, V is the potential of the system and T is the Total Kinetic Energy, thus E will
#the total energy of the system.
V(x,y) = 1//2 * (x^2 + y^2 + 2x^2*y - 2//3 * y^3)
E(x,y,dx,dy) = V(x,y) + 1//2 * (dx^2 + dy^2);

#Define the function
function Hénon_Heiles(du,u,p,t)
    x  = u[1]
    y  = u[2]
    dx = u[3]
    dy = u[4]
    du[1] = dx
    du[2] = dy
    du[3] = -x - 2x*y
    du[4] = y^2 - y -x^2
end

#Pass to solvers
prob = ODEProblem(Hénon_Heiles, initial, tspan)
sol = solve(prob, Vern9(), abs_tol=1e-16, rel_tol=1e-16);
```

```julia
# Plot the orbit
plot(sol, vars=(1,2), title = "The orbit of the Hénon-Heiles system", xaxis = "x", yaxis = "y", leg=false)
```

```julia
#Optional Sanity check - what do you think this returns and why?
@show sol.retcode

#Plot -
plot(sol, vars=(1,3), title = "Phase space for the Hénon-Heiles system", xaxis = "Position", yaxis = "Velocity")
plot!(sol, vars=(2,4), leg = false)
```

```julia
#We map the Total energies during the time intervals of the solution (sol.u here) to a new vector
#pass it to the plotter a bit more conveniently
energy = map(x->E(x...), sol.u)

#We use @show here to easily spot erratic behaviour in our system by seeing if the loss in energy was too great.
@show ΔE = energy[1]-energy[end]

#Plot
plot(sol.t, energy, title = "Change in Energy over Time", xaxis = "Time in iterations", yaxis = "Change in Energy")
```

### Symplectic Integration

To prevent energy drift, we can instead use a symplectic integrator. We can directly define and solve the `SecondOrderODEProblem`:

```julia
function HH_acceleration!(dv,v,u,p,t)
    x,y  = u
    dx,dy = dv
    dv[1] = -x - 2x*y
    dv[2] = y^2 - y -x^2
end
initial_positions = [0.0,0.1]
initial_velocities = [0.5,0.0]
prob = SecondOrderODEProblem(HH_acceleration!,initial_velocities,initial_positions,tspan)
sol2 = solve(prob, KahanLi8(), dt=1/10);
```

Notice that we get the same results:

```julia
# Plot the orbit
plot(sol2, vars=(3,4), title = "The orbit of the Hénon-Heiles system", xaxis = "x", yaxis = "y", leg=false)
```

```julia
plot(sol2, vars=(3,1), title = "Phase space for the Hénon-Heiles system", xaxis = "Position", yaxis = "Velocity")
plot!(sol2, vars=(4,2), leg = false)
```

but now the energy change is essentially zero:

```julia
energy = map(x->E(x[3], x[4], x[1], x[2]), sol2.u)
#We use @show here to easily spot erratic behaviour in our system by seeing if the loss in energy was too great.
@show ΔE = energy[1]-energy[end]

#Plot
plot(sol2.t, energy, title = "Change in Energy over Time", xaxis = "Time in iterations", yaxis = "Change in Energy")
```

It's so close to zero it breaks GR! And let's try to use a Runge-Kutta-Nyström solver to solve this. Note that Runge-Kutta-Nyström isn't symplectic.

```julia
sol3 = solve(prob, DPRKN6());
energy = map(x->E(x[3], x[4], x[1], x[2]), sol3.u)
@show ΔE = energy[1]-energy[end]
gr()
plot(sol3.t, energy, title = "Change in Energy over Time", xaxis = "Time in iterations", yaxis = "Change in Energy")
```

Note that we are using the `DPRKN6` sovler at `reltol=1e-3` (the default), yet it has a smaller energy variation than `Vern9` at `abs_tol=1e-16, rel_tol=1e-16`. Therefore, using specialized solvers to solve its particular problem is very efficient.


# Bracketing Methods (Bisection example)

Consider the refrigeration tank example from Belegundu and Chandrupatla [1].
We want to minimize the cost of a cylindrical refrigeration tank that must have
a volume of 50 m$^3$.
The costs of the tank are
- Circular ends cost \$10 per m$^2$
- Cylindrical walls cost \$6 per m$^2$
- Refrigerator costs \$80 per m$^2$ over its life

Let $d$ be the tank diameter, and $L$ the height.

\begin{align}
  f &= 10  \left(\frac{2 \pi d^2}{4}\right) + 6 (\pi d L) + 80 \left( \frac{2
\pi d^2}{4} + \pi d L \right)\\
  &= 45 \pi d^2 + 86 \pi d L
\end{align}

However, $L$ is a function of $d$ because the volume is constrained.  We could
add a constraint to the problem
\begin{align}
  \frac{\pi d^2}{4} L = V
\end{align}
but it is easier to express $V$ as a function of $d$ and make the problem
unconstrained.
\begin{align}
  L &= \frac{4 V}{\pi d^2}\\
  &= \frac{200}{\pi d^2}
\end{align}
Thus the optimization can be expressed as
\begin{align*}
\textrm{minimize} &\quad 45 \pi d^2 + \frac{17200}{d}\\
\textrm{with respect to} &\quad d \\
\textrm{subject to} &\quad d \ge 0
\end{align*}

One-dimensional optimization problems are silly of course, we can just find the
minimum by looking at a plot.  However, we use a one-dimensional example to
illustrate line searches.  A line search seeks an approximate minimum to a one-dimensional optimization problem within a N-dimensional space.

We will use bisection to find the minimum of this function.  This is a recursive
function.

```python
from math import fabs

def bisection(x1, x2, f1, f2, fh, sizevec):
    """
    This function finds the root of a function using bisection.

    Parameters
    ----------
    x1 : float
        lower bound
    x2 : float
        upper bound
    f1 : float
        function value at lower bound
    f2 : float
        function value at upper bound
        f1 * f2 must be < 0 in order to contain a root.
        Currently this is left up to the user to check.
    fh : function handle
        should be of form f = fh(x)
        where f is the function value
    sizevec : list
        input an empty array and the interval size
        will be appended at each iteration

    Returns
    -------
    xroot : float
        root of function fh

    """

    # divide interval in half
    x = 0.5*(x1 + x2)

    # save in iteration history
    sizevec.append(x2-x1)

    # if interval is small, then we have converged
    if (fabs(x2 - x1) < 1e-6):
        return x

    # evaluate function at the new point (midpoint of interval)
    f = fh(x)

    # determine which side of the interval are root is in
    if (f*f1 < 0):  # left brack applies
        x2 = x
        f2 = f
    else:  # right bracket applies
        x1 = x
        f1 = f

    # recursively call bisection with our new interval
    return bisection(x1, x2, f1, f2, fh, sizevec)
```

We are interseted in optimization, so we don't want to find the root of our
function, but rather the "root" of the derivative as a potential minimum point.
Let's define our objectve function, its derivative, and solve for the minium.

```python
%matplotlib inline

import numpy as np
from math import pi
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')

def func(d):
    return 45*pi*d**2 + 17200.0/d

def deriv(d):
    return 90*pi*d - 17200.0/d**2

# choose starting interval
d1 = 1.0
d2 = 10.0

# evalaute function
g1 = deriv(d1)
g2 = deriv(d2)

# check that our bracket is ok
assert(g1*g2 < 0)

# find optimal point
size = []
dopt = bisection(d1, d2, g1, g2, deriv, size)

# plot function
dvec = np.linspace(d1, d2, 200)
plt.figure()
plt.plot(dvec, func(dvec)/1e3)
plt.plot(dopt, func(dopt)/1e3, 'r*', markersize=12)
plt.xlabel('diameter (m)')
plt.ylabel('cost (thousands of dollars)')

# plot convergence history (interval size)
plt.figure()
plt.semilogy(size)
plt.xlabel('iteration')
plt.ylabel('interval size')
```

Note the linear convergence behavior.

[1] Belegundu, A. D. and Chandrupatla, T. R., Optimization Concepts and Applications in Engineering, Cambridge University Press, Mar 2011.