# Statistics Fundamentals
Statistics is primarily about analyzing data samples, and that starts with udnerstanding the distribution of data in a sample.

## Analyzing Data Distribution
A great deal of statistical analysis is based on the way that data values are distributed within the dataset. In this section, we'll explore some statistics that you can use to tell you about the values in a dataset.

### Measures of Central Tendency
The term *measures of central tendency* sounds a bit grand, but really it's just a fancy way of saying that we're interested in knowing where the middle value in our data is. For example, suppose decide to conduct a study into the comparative salaries of people who graduated from the same school. You might record the results like this:

| Name     | Salary      |
|----------|-------------|
| Dan      | 50,000      |
| Joann    | 54,000      |
| Pedro    | 50,000      |
| Rosie    | 189,000     |
| Ethan    | 55,000      |
| Vicky    | 40,000      |
| Frederic | 59,000      |

Now, some of the former-students may earn a lot, and others may earn less; but what's the salary in the middle of the range of all salaries?

#### Mean
A common way to define the central value is to use the *mean*, often called the *average*. This is calculated as the sum of the values in the dataset, divided by the number of observations in the dataset. When the dataset consists of the full population, the mean is represented by the Greek symbol ***μ*** (*mu*), and the formula is written like this:

\begin{equation}\mu = \frac{\displaystyle\sum_{i=1}^{N}X_{i}}{N}\end{equation}

More commonly, when working with a sample, the mean is represented by ***x̄*** (*x-bar*), and the formula is written like this (note the lower case letters used to indicate values from a sample):

\begin{equation}\bar{x} = \frac{\displaystyle\sum_{i=1}^{n}x_{i}}{n}\end{equation}

In the case of our list of heights, this can be calculated as:

\begin{equation}\bar{x} = \frac{50000+54000+50000+189000+55000+40000+59000}{7}\end{equation}

Which is **71,000**.

>In technical terminology, ***x̄*** is a *statistic* (an estimate based on a sample of data) and ***μ*** is a *parameter* (a true value based on the entire population). A lot of the time, the parameters for the full population will be impossible (or at the very least, impractical) to measure; so we use statistics obtained from a representative sample to approximate them. In this case, we can use the sample mean of salary for our selection of surveyed students to try to estimate the actual average salary of all students who graduate from our school.

In Python, when working with data in a *pandas.dataframe*, you can use the ***mean*** function, like this:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].mean())
```

So, is **71,000** really the central value? Or put another way, would it be reasonable for a graduate of this school to expect to earn $71,000? After all, that's the average salary of a graduate from this school.

If you look closely at the salaries, you can see that out of the seven former students, six earn less than the mean salary. The data is *skewed* by the fact that Rosie has clearly managed to find a much higher-paid job than her classmates.

#### Median
OK, let's see if we can find another definition for the central value that more closely reflects the expected earning potential of students attending our school. Another measure of central tendancy we can use is the *median*. To calculate the median, we need to sort the values into ascending order and then find the middle-most value. When there are an odd number of observations, you can find the position of the median value using this formula (where *n* is the number of observations):

\begin{equation}\frac{n+1}{2}\end{equation}

Remember that this formula returns the *position* of the median value in the sorted list; not the value itself.

If the number of observations is even, then things are a little (but not much) more complicated. In this case you calculate the median as the average of the two middle-most values, which are found like this:

\begin{equation}\frac{n}{2} \;\;\;\;and \;\;\;\; \frac{n}{2} + 1\end{equation}

So, for our graduate salaries; first lets sort the dataset:

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

There's an odd number of observation (7), so the median value is at position (7 + 1) ÷ 2; in other words, position 4:

| Salary      |
|-------------|
| 40,000      |
| 50,000      |
| 50,000      |
|***>54,000*** |
| 55,000      |
| 59,000      |
| 189,000     |

So the median salary is **54,000**.

The *pandas.dataframe* class in Python has a ***median*** function to find the median:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].median())
```

#### Mode
Another related statistic is the *mode*, which indicates the most frequently occurring value. If you think about it, this is potentially a good indicator of how much a student might expect to earn when they graduate from the school; out of all the salaries that are being earned by former students, the mode is earned by more than any other.

Looking at our list of salaries, there are two instances of former students earning **50,000**, but only one instance each for all other salaries:

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
| 55,000      |
| 59,000      |
| 189,000     |

The mode is therefore **50,000**.

As you might expect, the *pandas.dataframe* class has a ***mode*** function to return the mode:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print (df['Salary'].mode())
```

##### Multimodal Data
It's not uncommon for a set of data to have more than one value as the mode. For example, suppose Ethan receives a raise that takes his salary to **59,000**:

| Salary      |
|-------------|
| 40,000      |
|***>50,000***|
|***>50,000***|
| 54,000      |
|***>59,000***|
|***>59,000***|
| 189,000     |

Now there are two values with the highest frequency. This dataset is *bimodal*. More generally, when there is more than one mode value, the data is considered *multimodal*.

The *pandas.dataframe.**mode*** function returns all of the modes:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,59000,40000,59000]})

print (df['Salary'].mode())
```

    0    50000
    1    59000
    dtype: int64

### Distribution and Density
Now we know something about finding the center, we can start to explore how the data is distributed around it. What we're interested in here is understanding the general "shape" of the data distribution so that we can begin to get a feel for what a 'typical' value might be expected to be.

We can start by finding the extremes - the minimum and maximum. In the case of our salary data, the lowest paid graduate from our school is Vicky, with a salary of **40,000**; and the highest-paid graduate is Rosie, with **189,000**.

The *pandas.dataframe* class has ***min*** and ***max*** functions to return these values.

Run the following code to compare the minimum and maximum salaries to the central measures we calculated previously:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

print ('Min: ' + str(df['Salary'].min()))
print ('Mode: ' + str(df['Salary'].mode()[0]))
print ('Median: ' + str(df['Salary'].median()))
print ('Mean: ' + str(df['Salary'].mean()))
print ('Max: ' + str(df['Salary'].max()))
```

We can examine these values, and get a sense for how the data is distributed - for example, we can see that the *mean* is closer to the max than the *median*, and that both are closer to the *min* than to the *max*.

However, it's generally easier to get a sense of the distribution by visualizing the data. Let's start by creating a histogram of the salaries, highlighting the *mean* and *median* salaries (the *min*, *max* are fairly self-evident, and the *mode* is wherever the highest bar is):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
salary.plot.hist(title='Salary Distribution', color='lightblue', bins=25)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

The ***mean*** and ***median*** are shown as dashed lines. Note the following:
- *Salary* is a continuous data value - graduates could potentially earn any value along the scale, even down to a fraction of cent.
- The number of bins in the histogram determines the size of each salary band for which we're counting frequencies. Fewer bins means merging more individual salaries together to be counted as a group.
- The majority of the data is on the left side of the histogram, reflecting the fact that most graduates earn between 40,000 and 55,000
- The mean is a higher value than the median and mode.
- There are gaps in the histogram for salary bands that nobody earns.

The histogram shows the relative frequency of each salary band, based on the number of bins. It also gives us a sense of the *density* of the data for each point on the salary scale. With enough data points, and small enough bins, we could view this density as a line that shows the shape of the data distribution.

Run the following cell to show the density of the salary data as a line on top of the histogram:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000]})

salary = df['Salary']
density = stats.gaussian_kde(salary)
n, x, _ = plt.hist(salary, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*5)
plt.axvline(salary.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(salary.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()

```

Note that the density line takes the form of an asymmetric curve that has a "peak" on the left and a long tail on the right. We describe this sort of data distribution as being *skewed*; that is, the data is not distributed symmetrically but "bunched together" on one side. In this case, the data is bunched together on the left, creating a long tail on the right; and is described as being *right-skewed* because some infrequently occurring high values are pulling the *mean* to the right.

Let's take a look at another set of data. We know how much money our graduates make, but how many hours per week do they need to work to earn their salaries? Here's the data:

| Name     | Hours |
|----------|-------|
| Dan      | 41    |
| Joann    | 40    |
| Pedro    | 36    |
| Rosie    | 30    |
| Ethan    | 35    |
| Vicky    | 39    |
| Frederic | 40    |

Run the following code to show the distribution of the hours worked:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Hours':[41,40,36,30,35,39,40]})

hours = df['Hours']
density = stats.gaussian_kde(hours)
n, x, _ = plt.hist(hours, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7)
plt.axvline(hours.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(hours.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

Once again, the distribution is skewed, but this time it's **left-skewed**. Note that the curve is asymmetric with the ***mean*** to the left of the ***median*** and the *mode*; and the average weekly working hours skewed to the lower end.

Once again, Rosie seems to be getting the better of the deal. She earns more than her former classmates for working fewer hours. Maybe a look at the test scores the students achieved on their final grade at school might help explain her success:

| Name     | Grade |
|----------|-------|
| Dan      | 50    |
| Joann    | 50    |
| Pedro    | 46    |
| Rosie    | 95    |
| Ethan    | 50    |
| Vicky    | 5     |
| Frederic | 57    |

Let's take a look at the distribution of these grades:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Grade':[50,50,46,95,50,5,57]})

grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, histtype='step', normed=True, bins=25)
plt.plot(x, density(x)*7.5)
plt.axvline(grade.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(grade.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()
```

This time, the distribution is symmetric, forming a "bell-shaped" curve. The ***mean***, ***median***, and mode are at the same location, and the data tails off evenly on both sides from a central peak.

Statisticians call this a *normal* distribution (or sometimes a *Gaussian* distribution), and it occurs quite commonly in many scenarios due to something called the *Central Limit Theorem*, which reflects the way continuous probability works - more about that later.

#### Skewness and Kurtosis
You can measure *skewness* (in which direction the data is skewed and to what degree) and kurtosis (how "peaked" the data is) to get an idea of the shape of the data distribution. In Python, you can use the ***skew*** and ***kurt*** functions to find this:

```python
%matplotlib inline
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
import scipy.stats as stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' skewness: ' + str(df[col].skew()))
    print(df[col].name + ' kurtosis: ' + str(df[col].kurt()))
    density = stats.gaussian_kde(df[col])
    n, x, _ = plt.hist(df[col], histtype='step', normed=True, bins=25)
    plt.plot(x, density(x)*6)
    plt.show()
    print('\n')
```

Now let's look at the distribution of a real dataset - let's see how the heights of the father's measured in Galton's study of parent and child heights are distributed:

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

import statsmodels.api as sm

df = sm.datasets.get_rdataset('GaltonFamilies', package='HistData').data

fathers = df['father']
density = stats.gaussian_kde(fathers)
n, x, _ = plt.hist(fathers, histtype='step', normed=True, bins=50)
plt.plot(x, density(x)*2.5)
plt.axvline(fathers.mean(), color='magenta', linestyle='dashed', linewidth=2)
plt.axvline(fathers.median(), color='green', linestyle='dashed', linewidth=2)
plt.show()

```

As you can see, the father's height measurements are approximately normally distributed - in other words, they form a more or less *normal* distribution that is symmetric around the mean.

### Measures of Variance
We can see from the distribution plots of our data that the values in our dataset can vary quite widely. We can use various measures to quantify this variance.

#### Range
A simple way to quantify the variance in a dataset is to identify the difference between the lowest and highest values. This is called the *range*, and is calculated by subtracting the minimim value from the maximum value.

The following Python code creates a single Pandas dataframe for our school graduate data, and calculates the *range* for each of the numeric features:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

numcols = ['Salary', 'Hours', 'Grade']
for col in numcols:
    print(df[col].name + ' range: ' + str(df[col].max() - df[col].min()))
```

#### Percentiles and Quartiles
The range is easy to calculate, but it's not a particularly useful statistic. For example, a range of 149,000 between the lowest and highest salary does not tell us which value within that range a graduate is most likely to earn -  it doesn't tell us nothing about how the salaries are distributed around the mean within that range.  The range tells us very little about the comparative position of an individual value within the distribution - for example, Frederic scored 57 in his final grade at school; which is a pretty good score (it's more than all but one of his classmates); but this isn't immediately apparent from a score of 57 and range of 90.

##### Percentiles
A percentile tells us where a given value is ranked in the overall distribution. For example, 25% of the data in a distribution has a value lower than the 25th percentile; 75% of the data has a value lower than the 75th percentile, and so on. Note that half of the data has a value lower than the 50th percentile - so the 50th percentile is also the median!

Let's examine Frederic's grade using this approach. We know he scored 57, but how does he rank compared to his fellow students?

Well, there are seven students in total, and five of them scored less than Frederic; so we can calculate the percentile for Frederic's grade like this:

\begin{equation}\frac{5}{7} \times 100 \approx 71.4\end{equation}

So Frederic's score puts him at the 71.4th percentile in his class.

In Python, you can use the ***percentileofscore*** function in the *scipy.stats* package to calculate the percentile for a given value in a set of values:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'strict'))
```

We've used the strict definition of percentile; but sometimes it's calculated as being the percentage of values that are less than *or equal to* the value you're comparing. In this case, the calculation for Frederic's percentile would include his own score:

\begin{equation}\frac{6}{7} \times 100 \approx 85.7\end{equation}

You can calculate this way in Python by using the ***weak*** mode of the ***percentileofscore*** function:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 57, 'weak'))
```

We've considered the percentile of Frederic's grade, and used it to rank him compared to his fellow students. So what about Dan, Joann, and Ethan? How do they compare to the rest of the class? They scored the same grade (50), so in a sense they share a percentile.

To deal with this *grouped* scenario, we can average the percentage rankings for the matching scores. We treat half of the scores matching the one we're ranking as if they are below it, and half as if they are above it. In this case, there were three matching scores of 50, and for each of these we calculate the percentile as if 1 was below and 1 was above. So the calculation for a percentile for Joann based on scores being less than or equal to 50 is:

\begin{equation}(\frac{4}{7}) \times 100 \approx 57.14\end{equation}

The value of **4** consists of the two scores that are below Joann's score of 50, Joann's own score, and half of the scores that are the same as Joann's (of which there are two, so we count one).

In Python, the ***percentileofscore*** function has a ***rank*** function that calculates grouped percentiles like this:

```python
import pandas as pd
from scipy import stats

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(stats.percentileofscore(df['Grade'], 50, 'rank'))
```

##### Quartiles
Rather than using individual percentiles to compare data, we can consider the overall spread of the data by dividing those percentiles into four *quartiles*. The first quartile contains the values from the minimum to the 25th percentile, the second from the 25th percentile to the 50th percentile (which is the median), the third from the 50th percentile to the 75th percentile, and the fourth from the 75th percentile to the maximum.

In Python, you can use the ***quantile*** function of the *pandas.dataframe* class to find the threshold values at the 25th, 50th, and 75th percentiles (*quantile* is a generic term for a ranked position, such as a percentile or quartile).

Run the following code to find the quartile thresholds for the weekly hours worked by our former students:

```python
# Quartiles
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
print(df['Hours'].quantile([0.25, 0.5, 0.75]))
```

    0.25    35.5
    0.50    39.0
    0.75    40.0
    Name: Hours, dtype: float64

Its usually easier to understand how data is distributed across the quartiles by visualizing it. You can use a histogram, but many data scientists use a kind of visualization called a *box plot* (or a *box and whiskers* plot).

Let's create a box plot for the weekly hours:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Hours'].plot(kind='box', title='Weekly Hours Distribution', figsize=(10,8))
plt.show()
```

The box plot consists of:
- A rectangular *box* that shows where the data between the 25th and 75th percentile (the second and third quartile) lie. This part of the distribution is often referred to as the *interquartile range* - it contains the middle 50 data values.
- *Whiskers* that extend from the box to the bottom of the first quartile and the top of the fourth quartile to show the full range of the data.
- A line in the box that shows that location of the median (the 50th percentile, which is also the threshold between the second and third quartile)

In this case, you can see that the interquartile range is between 35 and 40, with the median nearer the top of that range. The range of the first quartile is from around 30 to 35, and the fourth quartile is from 40 to 41.

#### Outliers
Let's take a look at another box plot - this time showing the distribution of the salaries earned by our former classmates:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,30,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8))
plt.show()
```

So what's going on here?

Well, as we've already noticed, Rosie earns significantly more than her former classmates. So much more in fact, that her salary has been identifed as an *outlier*. An outlier is a value that is so far from the center of the distribution compared to other values that it skews the distribution by affecting the mean. There are all sorts of reasons that you might have outliers in your data, including data entry errors, failures in sensors or data-generating equipment, or genuinely anomalous values.

So what should we do about it?

This really depends on the data, and what you're trying to use it for. In this case, let's assume we're trying to figure out what's a reasonable expectation of salary for a graduate of our school to earn. Ignoring for the moment that we have an extremly small dataset on which to base our judgement, it looks as if Rosie's salary could be either an error (maybe she mis-typed it in the form used to collect data) or a genuine anomaly (maybe she became a professional athelete or some other extremely highly paid job). Either way, it doesn't seem to represent a salary that a typical graduate might earn.

Let's see what the distribution of the data looks like without the outlier:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Salary'].plot(kind='box', title='Salary Distribution', figsize=(10,8), showfliers=False)
plt.show()
```

Now it looks like there's a more even distribution of salaries. It's still not quite symmetrical, but there's much less overall variance. There's potentially some cause here to disregard Rosie's salary data when we compare the salaries, as it is tending to skew the analysis.

So is that OK? Can we really just ignore a data value we don't like?

Again, it depends on what you're analyzing. Let's take a look at the distribution of final grades:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

# Plot a box-whisker chart
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

Once again there are outliers, this time at both ends of the distribution. However, think about what this data represents. If we assume that the grade for the final test is based on a score out of 100, it seems reasonable to expect that some students will score very low (maybe even 0) and some will score very well (maybe even 100); but most will get a score somewhere in the middle.  The reason that the low and high scores here look like outliers might just be because we have so few data points. Let's see what happens if we include a few more students in our data:

```python
%matplotlib inline
import pandas as pd
from matplotlib import pyplot as plt

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic', 'Jimmie', 'Rhonda', 'Giovanni', 'Francesca', 'Rajab', 'Naiyana', 'Kian', 'Jenny'],
                   'Grade':[50,50,46,95,50,5,57,42,26,72,78,60,40,17,85]})

# Plot a box-whisker chart
df['Grade'].plot(kind='box', title='Grade Distribution', figsize=(10,8))
plt.show()
```

With more data, there are some more high and low scores; so we no longer consider the isolated cases to be outliers.

The key point to take away here is that you need to really understand the data and what you're trying to do with it, and you need to ensure that you have a reasonable sample size, before determining what to do with outlier values.

#### Variance and Standard Deviation
We've seen how to understand the *spread* of our data distribution using the range, percentiles, and quartiles; and we've seen the effect of outliers on the distribution. Now it's time to look at how to measure the amount of variance in the data.

##### Variance
Variance is measured as the average of the squared difference from the mean. For a full population, it's indicated by a squared Greek letter *sigma* (***σ2***) and calculated like this:

\begin{equation}\sigma^{2} = \frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}\end{equation}

For a sample, it's indicated as ***s2*** calculated like this:

\begin{equation}s^{2} = \frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}}{n-1}\end{equation}

In both cases, we sum the difference between the individual data values and the mean and square the result. Then, for a full population we just divide by the number of data items to get the average. When using a sample, we divide by the total number of items **minus 1** to correct for sample bias.

Let's work this out for our student grades (assuming our data is a sample from the larger student population).

First, we need to calculate the mean grade:

\begin{equation}\bar{x} = \frac{50+50+46+95+50+5+57}{7}\approx 50.43\end{equation}

Then we can plug that into our formula for the variance:

\begin{equation}s^{2} = \frac{(50-50.43)^{2}+(50-50.43)^{2}+(46-50.43)^{2}+(95-50.43)^{2}+(50-50.43)^{2}+(5-50.43)^{2}+(57-50.43)^{2}}{7-1}\end{equation}

So:

\begin{equation}s^{2} = \frac{0.185+0.185+19.625+1986.485+0.185+2063.885+43.165}{6}\end{equation}

Which simplifies to:

\begin{equation}s^{2} = \frac{4113.715}{6}\end{equation}

Giving the result:

\begin{equation}s^{2} \approx 685.619\end{equation}

The higher the variance, the more spread your data is around the mean.

In Python, you can use the ***var*** function of the *pandas.dataframe* class to calculate the variance of a column in a dataframe:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(df['Grade'].var())
```

##### Standard Deviation
To calculate the variance, we squared the difference of each value from the mean. If we hadn't done this, the numerator of our fraction would always end up being zero (because the mean is at the center of our values). However, this means that the variance is not in the same unit of measurement as our data - in our case, since we're calculating the variance for grade points, it's in grade points squared; which is not very helpful.

To get the measure of variance back into the same unit of measurement, we need to find its square root:

\begin{equation}s = \sqrt{685.619} \approx 26.184\end{equation}

So what does this value represent?

It's the *standard deviation* for our grades data. More formally, it's calculated like this for a full population:

\begin{equation}\sigma = \sqrt{\frac{\displaystyle\sum_{i=1}^{N} (X_{i} -\mu)^{2}}{N}}\end{equation}

Or like this for a sample:

\begin{equation}s = \sqrt{\frac{\displaystyle\sum_{i=1}^{n} (x_{i} -\bar{x})^{2}}{n-1}}\end{equation}

Note that in both cases, it's just the square root of the corresponding variance forumla!

In Python, you can calculate it using the ***std*** function:

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})

print(df['Grade'].std())
```

#### Standard Deviation in a Normal Distribution

In statistics and data science, we spend a lot of time considering *normal* distributions; because they occur so frequently. The standard deviation has an important relationship to play in a normal distribution.

Run the following cell to show a histogram of a *standard normal* distribution (which is a distribution with a mean of 0 and a standard deviation of 1):

```python
%matplotlib inline
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import scipy.stats as stats

# Create a random standard normal distribution
df = pd.DataFrame(np.random.randn(100000, 1), columns=['Grade'])

# Plot the distribution as a histogram with a density curve
grade = df['Grade']
density = stats.gaussian_kde(grade)
n, x, _ = plt.hist(grade, color='lightgrey', density=True, bins=100)
plt.plot(x, density(x))

# Get the mean and standard deviation
s = df['Grade'].std()
m = df['Grade'].mean()

# Annotate 1 stdev
x1 = [m-s, m+s]
y1 = [0.25, 0.25]
plt.plot(x1,y1, color='magenta')
plt.annotate('1s (68.26%)', (x1[1],y1[1]))

# Annotate 2 stdevs
x2 = [m-(s*2), m+(s*2)]
y2 = [0.05, 0.05]
plt.plot(x2,y2, color='green')
plt.annotate('2s (95.45%)', (x2[1],y2[1]))

# Annotate 3 stdevs
x3 = [m-(s*3), m+(s*3)]
y3 = [0.005, 0.005]
plt.plot(x3,y3, color='orange')
plt.annotate('3s (99.73%)', (x3[1],y3[1]))

# Show the location of the mean
plt.axvline(grade.mean(), color='grey', linestyle='dashed', linewidth=1)

plt.show()
```

The horizontal colored lines show the percentage of data within 1, 2, and 3 standard deviations of the mean (plus or minus).

In any normal distribution:
- Approximately 68.26% of values fall within one standard deviation from the mean.
- Approximately 95.45% of values fall within two standard deviations from the mean.
- Approximately 99.73% of values fall within three standard deviations from the mean.

#### Z Score
So in a normal (or close to normal) distribution, standard deviation provides a way to evaluate how far from a mean a given range of values falls, allowing us to compare where a particular value lies within the distribution. For example, suppose Rosie tells you she was the highest scoring student among her friends - that doesn't really help us assess how well she scored. She may have scored only a fraction of a point above the second-highest scoring student. Even if we know she was in the top quartile; if we don't know how the rest of the grades are distributed it's still not clear how well she performed compared to her friends.

However, if she tells you how many standard deviations higher than the mean her score was, this will help you compare her score to that of her classmates.

So how do we know how many standard deviations above or below the mean a particular value is? We call this a *Z Score*, and it's calculated like this for a full population:

\begin{equation}Z = \frac{x - \mu}{\sigma}\end{equation}

or like this for a sample:

\begin{equation}Z = \frac{x - \bar{x}}{s}\end{equation}

So, let's examine Rosie's grade of 95. Now that we know the *mean* grade is 50.43 and the *standard deviation* is 26.184, we can calculate the Z Score for this grade like this:

\begin{equation}Z = \frac{95 - 50.43}{26.184} = 1.702\end{equation}.

So Rosie's grade is 1.702 standard deviations above the mean.

### Summarizing Data Distribution in Python
We've seen how to obtain individual statistics in Python, but you can also use the ***describe*** function to retrieve summary statistics for all numeric columns in a dataframe. These summary statistics include many of the statistics we've examined so far (though it's worth noting that the *median* is not included):

```python
import pandas as pd

df = pd.DataFrame({'Name': ['Dan', 'Joann', 'Pedro', 'Rosie', 'Ethan', 'Vicky', 'Frederic'],
                   'Salary':[50000,54000,50000,189000,55000,40000,59000],
                   'Hours':[41,40,36,17,35,39,40],
                   'Grade':[50,50,46,95,50,5,57]})
print(df.describe())
```


---

Implementation of Finite Element Analysis for One Dimensional Bar Elements in Python

---

  Pramod Kumar Yadav

@iampramodyadav

Roll No. 20132015

M.Tech- Machine Design (3rd Sem)

Indian Institute of Technology (BHU) Varanasi

---

Table of Contents

1  Overview

2  Goals

3  Specification

4  Detail Code Overview with Example

4.1  Shape function and it's derivative

4.1.1  Function for shape function in Python (Shape)

4.1.2  Function for derivative of shape function (dShape)

4.1.3  Function for Visualization of shape function (PlotShape)

4.2  Gauss-Legendre Quadrature implementation in Python

4.2.1  Function for Legendre Polynomial in symbolic form (Legendre)

4.2.2  Function for Derivative of Legendre Polynomial in symbolic form (dLegendre)

4.2.3  Function to find roots of Legendre Polynomial or Gauss points (LegendreRoots)

4.2.4  Function to find Weight Coefficient (GaussWeights)

4.2.5  Function to Plot Legendre Polynomial (PlotLegendre)

4.2.6  Testing Final Result

4.3  Function for Values of Shape Function and derivative at Gauss Points (ShapeLegRoot)

4.4  Element Calculation

4.4.1  Function for Global (All elements of same size) $K_{ij}$ & $f_i$ if 1-D Bar (ElemKfi)

4.4.1.1  Example:1

4.4.1.2  Example:2

4.4.2  Function for Global (All elements of same size) $K_{ij}$ & $f_i$ if 1-D Bar (GlobalKfi)

4.4.2.1  Example:1

4.4.3  Function for Global (Elements of different Size or same size) $K_{ij}$ & $f_i$ if 1-D Bar (GlobalKfiU)

4.4.3.1  Example:2

4.4.3.2  Example:1 (Method-2)

5  Future scope

---

---

## Overview
For this project I have implemented Finite Element Analysis(FEA) in python for a bar element. Purpose of this project is to automate FEA algorithm or steps so it will become easy to solve problems based on the 1-Dimensional (1-D) bar of any number of elements and any order of approximation function.
In this project statnderd python library used are [SymPy](https://docs.sympy.org/latest/tutorial/basic_operations.html) and [NumPy](https://numpy.org/devdocs/user/whatisnumpy.html) for Mathemetical calculation and [MatPlotLib](https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.plot.html) for plotting graphs. Integration method used here is Gauss-Legendre Quadrature method and also implemented it from scratch.

## Goals

1. Write python code to implement Gauss-Legendre Quadrature for numerical integration
2. Generate Local or Global stiffness matrix and force vector for a given order of approximation (shape function), applied external distributed force and x-Coordinate of nodes

## Specification
This project consist of 4 parts-
1. Function (or Subroutine) for generating shape functions of given order and visualization of its plot in natural coordinate system
2. Function to find Gauss roots of Legendre polynomial, weights and plot Legendre polynomial for required order of polynomial (for minimum integration error.)
3. Function for calculation of shape functions at Gauss roots
4. Functions to generate Local and Global, Stiffness matrices and force vector for given problem
---

## **Detail Code Overview with Example**

### Shape function and it's derivative

We know that shape function in natural coordinate for $P_{th}$ order of approximation can be written as

Where physical coordinate $x$ and natural coordinate $\xi$ (here, z in program)[Nptel Reference](https://nptel.ac.in/content/storage2/courses/112104116/lecture-11/11_10.htm)

#### Function for shape function in Python (Shape)

```python
from sympy import *  #importing libraries
import numpy as np
```

```python
z=Symbol('z') #for symbolic represenation
x=Symbol('x')
```

```python
def Shape(p):
  '''
  =====================================================
  Shape(p)
  p:  order (p) of approximation
  z:  natural coordinate
  This function return Shape funtions values at given p
  =====================================================
  '''
  z=Symbol('z')
  n=[]

  for i in range(0, p+1):
    point=-1
    point=point+2*i/p
    n.append(point)

  shape=[1]*(p+1)

  for i in range(0,p+1):
    for j in range(0,p+1):
      if i!=j:
        shape[i]=shape[i]*((z-n[j])/(n[i]-n[j]))

  return shape
```

```python
print(Shape.__doc__)
```

      =====================================================
      Shape(p)
      p:  order (p) of approximation
      z:  natural coordinate
      This function return Shape funtions values at given p
      =====================================================

---

**Example:** For shape function of 2nd order

```python
Shape(2)
```

    [-1.0*z*(0.5 - 0.5*z), (1.0 - 1.0*z)*(1.0*z + 1.0), 1.0*z*(0.5*z + 0.5)]

---

#### Function for derivative of shape function (dShape)

```python
def dShape(p):
  '''
  =====================================================
  dShape(p)
  p:  order (p) of approximation

  z:  natural coordinate variable
  This function return derivative of Shape funtions values at given p
  =====================================================
  '''
  z=Symbol('z')
  f=Shape(p)

  return [simplify(diff(f1)) for f1 in f]
```

```python
print(dShape.__doc__)
```

      =====================================================
      dShape(p)
      p:  order (p) of approximation

      z:  natural coordinate variable
      This function return derivative of Shape funtions values at given p
      =====================================================

---

**Example:** For derivative of shape function of 2nd order

```python
dShape(2)
```

    [1.0*z - 0.5, -2.0*z, 1.0*z + 0.5]

---

#### Function for Visualization of shape function (PlotShape)

```python
import seaborn as sns
from matplotlib import style
import matplotlib.pyplot as plt
```

```python
def plotShape(p):
  '''
  =====================================================
  plotShape(p)

  p: order (p) of approximation
  z:value of natural coordinate
  This function return plot of Shape funtions (Natural Coordinate)
  =====================================================
  '''
  sns.set()
  sns.set_style("whitegrid", {'grid.linestyle': '--'})

  z=Symbol('z')
  f=Shape(p)
  p1=plot(0,(z,-1,1),show=False)
  for i in f:
    p2=plot(i,(z,-1,1),show=False)
    p1.append(p2[0])

  return p1.show()
```

```python
print(plotShape.__doc__)
```

      =====================================================
      plotShape(p)

      p: order (p) of approximation
      z:value of natural coordinate
      This function return plot of Shape funtions (Natural Coordinate)
      =====================================================

---

**Example:** Plot shape function of 3rd order

```python
plotShape(3)
```

---

### Gauss-Legendre Quadrature implementation in Python
According to Gauss-Legendre Quadrature rule in any numerical integration scheme, the integral is expressed as a sum over certain points of the domain of integration called as **sampling points** . A typical term in the sum consists of the value of the function at that point multiplied by a suitable number called as the **weight** . Thus,

$${\displaystyle\int_{-1}^{1}f(x)\,dx\approx\sum _{i=1}^{n}w_{i}f(x_{i})}$$

Here, $w_i$ are the weights. In case of Gauss quadrature rule, the sampling points $i= 1,2,3....n$ are called as the Gauss points and $x_i$,$i= 1,2,3....n$ are called as Gauss points coordinates (Roots of [Legendre polynomial](https://en.wikipedia.org/wiki/Legendre_polynomials#Recurrence_relations).)

#### Function for Legendre Polynomial in symbolic form (Legendre)

To write program for Legendre polynomial we will Bonnet’s recursion formula for Legendre polynomial, which can be written as:
 * $$\boxed{\displaystyle(n+1)P_{n+1}(x)=(2n+1)xP_{n}(x)-nP_{n-1}(x)}$$

 * $${P_{0}(x)=1\,}$$

 * $${P_{1}(x)=x}$$

```python
from sympy import *
import numpy as np
```

```python
z=Symbol('z') #for symbolic represenation
x=Symbol('x')
```

```python
def Legendre(n):
  """
  =====================================================
  n: Order of polynomial
  x: Variable
  This function print Legendre polynomial of order n (Symbolic)
  =====================================================
  """

  x=symbols('x')
  if (n==0):
    return x*0+1.0
  elif (n==1):
    return x
  else:
    return ((2.0*n-1.0)*x*Legendre(n-1)-(n-1)*Legendre(n-2))/n
```

```python
print(Legendre.__doc__)
```

      =====================================================
      n: Order of polynomial
      x: Variable
      This function print Legendre polynomial of order n (Symbolic)
      =====================================================

---

**Example:** Find Legendre Polynomial for 2nd and 3rd order

```python
Legendre(2) #2nd Order
```

$\displaystyle 1.5 x^{2} - 0.5$

```python
Legendre(3) #3rd order
```

$\displaystyle 1.66666666666667 x \left(1.5 x^{2} - 0.5\right) - \frac{2 x}{3}$

```python
simplify(Legendre(3)) #simplified form
```

$\displaystyle x \left(2.5 x^{2} - 1.5\right)$

---

#### Function for Derivative of Legendre Polynomial in symbolic form (dLegendre)
Here also Bonnet's recursion formula is used. it can be writen as ([Source](https://en.wikipedia.org/wiki/Legendre_polynomials#Recurrence_relations).)

$$\boxed{\displaystyle{\frac{x^{2}-1}{n}}{\frac{d}{dx}}P_{n}(x)=xP_{n}(x)-P_{n-1}(x)\,}$$
$$or$$
$$\boxed{\displaystyle{\frac{d}{dx}}P_{n}(x)={\frac{n}{x^{2}-1}}(xP_{n}(x)-P_{n-1}(x))\,}$$

```python
def dLegendre(n):
  """
  =====================================================
  n: Order of polynomial
  x: Variable
  This function print Derivative of Legendre polynomial of order n  (Symbolic)
  =====================================================
  """
  x=symbols('x')
  if (n==0):
    return x*0
  elif (n==1):
    return x*0+1.0
  else:
    return (n/(x**2-1.0))*(x*Legendre(n)-Legendre(n-1))
```

```python
print(dLegendre.__doc__)
```

      =====================================================
      n: Order of polynomial
      x: Variable
      This function print Derivative of Legendre polynomial of order n  (Symbolic)
      =====================================================

---

**Example:** Find derivative of Legendre Polynomial for 2nd and 3rd order

```python
dLegendre(2)
```

$\displaystyle \frac{2 \left(x \left(1.5 x^{2} - 0.5\right) - x\right)}{x^{2} - 1.0}$

```python
simplify(dLegendre(2)) #2nd order simpified form
```

$\displaystyle 3.0 x$

```python
dLegendre(3)
```

$\displaystyle \frac{3 \left(- 1.5 x^{2} + x \left(1.66666666666667 x \left(1.5 x^{2} - 0.5\right) - \frac{2 x}{3}\right) + 0.5\right)}{x^{2} - 1.0}$

```python
simplify(dLegendre(3)) #3rd order simpified form
```

$\displaystyle 7.5 x^{2} - 1.5$

---

#### Function to find roots of Legendre Polynomial or Gauss points (LegendreRoots)
Here to find legendre roots Newton–Raphson method is used, which can be written as

$${x_{1}=x_{0}-{\frac{f(x_{0})}{f'(x_{0})}}}$$

The first guess ${x_{0}}$ for the ${i}$-th root of a ${n}$ order polynomial ${P_{n}}$ can be given by

$${x_{0}=\cos\left({\frac{\pi(i-1/4)}{n+1/2}}\right)}$$

```python
def LegendreRoots(polyorder,tolerance=1e-20):
  """
  =====================================================
  LegendreRoots(polyorder,tolerance=1e-20)

  polyorder: Order of polynomial(>2)
  tolerance: tolerance of error (default: 1e-20)
  Output: [roots, err]
  This function return Roots of Legendre polynomial
  =====================================================
  """
  if polyorder<2:
    err=1

  else:
    roots=[]

    for i in range(1,int((polyorder)/2) +1):
      x1=cos(pi*(i-0.25)/(polyorder+0.5))
      error=10*tolerance
      iters=0

      while (error>tolerance) and (iters<1000):
        dx=-Legendre(polyorder)/dLegendre(polyorder)
        dx=N(dx.subs(x,x1))
        x1=N(x1+dx)
        iters=iters+1
        error=abs(dx)

      #print(roots)
      roots.append(x1)
      #print(roots)

    roots=np.array(roots)
    if polyorder%2==0:
      roots=np.concatenate( (-1.0*roots, roots[::-1]) )

    else:
      roots=np.concatenate( (-1.0*roots, [0.0], roots[::-1]) )

    err=0

  return [roots, err]
```

```python
print(LegendreRoots.__doc__)
```

      =====================================================
      LegendreRoots(polyorder,tolerance=1e-20)

      polyorder: Order of polynomial(>2)
      tolerance: tolerance of error (default: 1e-20)
      Output: [roots, err]
      This function return Roots of Legendre polynomial
      =====================================================

---

**Examples** Find Gauss roots or Legendre Roots for 3rd,4th and 7th order legendre polynomial

```python
[Xis, err]=LegendreRoots(3)
print("Gauss points for 3rd order:\n",Xis)
```

    Gauss points for 3rd order:
     [-0.774596669241483 0.0 0.774596669241483]

```python
[Xis, err]=LegendreRoots(4)
print("Gauss points for 4th order:\n",Xis)
```

    Gauss points for 4th order:
     [-0.861136311594053 -0.339981043584856 0.339981043584856 0.861136311594053]

```python
[Xis, err]=LegendreRoots(7)
print("Gauss points for 7th order:\n",Xis)
```

    Gauss points for 7th order:
     [-0.949107912342759 -0.741531185599394 -0.405845151377397 0.0
     0.405845151377397 0.741531185599394 0.949107912342759]

---

#### Function to find Weight Coefficient (GaussWeights)
To find weight coefficient the $i-th$ Gauss node, $x_i$, is the $i-th$ root of $P_n$ and the weights are given by the formula ([Abramowitz & Stegun 1972](https://en.wikipedia.org/wiki/Gaussian_quadrature#Gauss%E2%80%93Legendre_quadrature))

$${w_{i}={\frac{2}{\left(1-x_{i}^{2}\right)\left[P'_{n}(x_{i})\right]^{2}}}.}$$

```python
def GaussWeights(polyorder):
  """
  =====================================================
  GaussWeights(polyorder)

  polyorder: Order of polynomial
  Output: [W, xis, err]
  This function return Weights(W) and Roots(xis) of Legendre polynomial
  =====================================================
  """
  W=[]
  [xis,err]=LegendreRoots(polyorder)
  xis=list(xis)
  if err==0:
    for x1 in xis:
      w=2.0/( (1.0-x**2)*(dLegendre(polyorder)**2) )
      w=w.subs(x,x1)
      W.append(w)
      err=0
  else:
    err=1 # could not determine roots - so no weights
  return [W, xis, err]
```

```python
print(GaussWeights.__doc__)
```

      =====================================================
      GaussWeights(polyorder)

      polyorder: Order of polynomial
      Output: [W, xis, err]
      This function return Weights(W) and Roots(xis) of Legendre polynomial
      =====================================================

---

**Example:** Find weights for 3rd,4th and 7th order legendre polynomial

```python
[W, xis, err]=GaussWeights(3)
print("Gauss weights for 3rd order:\n",W)
```

    Gauss weights for 3rd order:
     [0.555555555555556, 0.888888888888889, 0.555555555555556]

```python
[W, xis, err]=GaussWeights(4)
print("Gauss weights for 4th order:\n",W)
```

    Gauss weights for 4th order:
     [0.347854845137454, 0.652145154862546, 0.652145154862546, 0.347854845137454]

```python
[W, xis, err]=GaussWeights(7)
print("Gauss weights for 7th order:\n",W)
```

    Gauss weights for 7th order:
     [0.129484966168870, 0.279705391489277, 0.381830050505119, 0.417959183673469, 0.381830050505119, 0.279705391489277, 0.129484966168870]

---

#### Function to Plot Legendre Polynomial (PlotLegendre)
Function can be written as,

```python
def PlotLegendre(polyorder):
  """
  =====================================================
  PlotLegendre(polyorder)
  polyorder: Order of polynomial
  This function plot legendre plynomial
  =====================================================
  """
  sns.set()
  sns.set_style("whitegrid", {'grid.linestyle': '--'})
  plot(Legendre(polyorder),(x,-1,1))

```

```python
print(PlotLegendre.__doc__)
```

      =====================================================
      PlotLegendre(polyorder)
      polyorder: Order of polynomial
      This function plot legendre plynomial
      =====================================================

---

**Example:**plot 3rd,4th order legendre polynomial

```python
PlotLegendre(3)
```

```python
PlotLegendre(4)
```

**Plot of legendre polynomial from 1 to 5th order**

```python
p=plot(Legendre(1),Legendre(2),Legendre(3),Legendre(4),Legendre(5),(x,-1,1),
       title='Legendre PolyNomial',show=False);
p[0].line_color = 'red'
p[1].line_color = 'green'
p[2].line_color = 'yellow'
p[3].line_color = ['b','r']

p.show()
```

---

#### Testing Final Result
Plot and find Gauss roots and weights for polynomial order 2 to 6

```python
from matplotlib import style
import matplotlib.pyplot as plt
```

```python
for Polyorder in range(2,7):
  plt.rcParams['figure.figsize'] = 6.4, 4.8
  plot(Legendre(Polyorder),(x,-1,1))

  [W, xis, err]=GaussWeights(Polyorder)
  print("order:", Polyorder)
  print("Weights:", W)
  print ("Roots    : ",xis)

  print('='*90)
  print('\n\n')
  print('='*90)
```

---

### Function for Values of Shape Function and derivative at Gauss Points (ShapeLegRoot)

 This function calculates the values of shape functions and their derivatives for given order of approximation $p$
Note: Here I assumed,
- Area is maximum of 2nd order i.e. $Area=a_0+a_1x+a_2x^2$
- Distributed Force also is maximum of 2nd order i.e.
 $f=f_0+f_1x+f_2x^2$

 We can increase it by just one line code change.

```python
def ShapeLegRoot(p):
  """
  =====================================================
  ShapeLegRoot(p)

  p : order of approximation (integer)

  Output: [n, W, xis, ShapeValueXi, dShapeValueXi]
  Here,
  n-    Order of gauss legendre polynomial
  W-    Legendre Weights
  xis-  Gauss point coordinates (real)
  ShapeValueXi-   Value of Shape function at  at all the  Xi (real)(Legendre Roots)
  ShapeValueXi-   Value of derivative of Shape function at Gauss points Xi(Legendre Roots)
  =====================================================
  """

  p=int(p)

  pbar=max(2*p, p+2)
  if pbar%2!=0:                                   #if odd
    n=(pbar+1)/2
  else:
    n=(pbar+2)/2

  n=int(n)

  #print("Order of gauss legendre polynomial:",n)

  [W, xis, err]=GaussWeights(n)           #order: Polyorder(n), Weights:  W, Roots: xis

  ShapeValueXi=np.zeros((n, p+1))
  dShapeValueXi=np.zeros((n, p+1))

  #print('ndofel(p+1):',p+1)
  #print('Shape of Shape value matrix:',ShapeValueXi.Shape)

  s=Shape(p)
  ds=dShape(p)

  for i in range(0,n):
    for j in range(0,p+1):

      ShapeValueXi[i][j]=s[j].subs(z,xis[i])          #w.subs(x,x1)
      dShapeValueXi[i][j]=ds[j].subs(z,xis[i])

  return [n, W, xis, ShapeValueXi, dShapeValueXi]
```

```python
print(ShapeLegRoot.__doc__)
```

      =====================================================
      ShapeLegRoot(p)

      p : order of approximation (integer)

      Output: [n, W, xis, ShapeValueXi, dShapeValueXi]
      Here,
      n-    Order of gauss legendre polynomial
      W-    Legendre Weights
      xis-  Gauss point coordinates (real)
      ShapeValueXi-   Value of Shape function at  at all the  Xi (real)(Legendre Roots)
      ShapeValueXi-   Value of derivative of Shape function at Gauss points Xi(Legendre Roots)
      =====================================================

---

**Example:** find shape function and derivative of shape function at legendre roots (Gauss Points) for 1st and 2nd order approximation

```python
[n, W,xis,ShapeValueXi,dShapeValueXi]=ShapeLegRoot(1)
print('Shape Function values at Legendre roots:\n',ShapeValueXi)
print('Derivative of Shape Function values at Legendre roots\n:',dShapeValueXi)
```

    Shape Function values at Legendre roots:
     [[0.78867513 0.21132487]
     [0.21132487 0.78867513]]
    Derivative of Shape Function values at Legendre roots
    : [[-0.5  0.5]
     [-0.5  0.5]]

```python
[n, W,xis,ShapeValueXi,dShapeValueXi]=ShapeLegRoot(3)
print('Shape Function values at Legendre roots:\n',ShapeValueXi)
print('Derivative of Shape Function values at Legendre roots\n:',dShapeValueXi)
```

    Shape Function values at Legendre roots:
     [[ 0.66000567  0.52093769 -0.2301879   0.04924455]
     [ 0.00337374  1.00488585 -0.00992135  0.00166176]
     [ 0.00166176 -0.00992135  1.00488585  0.00337374]
     [ 0.04924455 -0.2301879   0.52093769  0.66000567]]
    Derivative of Shape Function values at Legendre roots
    : [[-2.15765367  3.03540432 -1.09784762  0.22009697]
     [-0.51503192 -0.71986158  1.48481893 -0.24992543]
     [ 0.24992543 -1.48481893  0.71986158  0.51503192]
     [-0.22009697  1.09784762 -3.03540432  2.15765367]]

---

### Element Calculation
 Here I will implement function to calculate Stiffness matrix and Force vector for different cases:

* Here linear mapping function used between the physical coordinate (x) and the natural coordinate $\xi$ (here, z) :

  *

  * Hence, $x_1^k$ (xL) and $x_{p+1}^k$ (xR) are the coordinates of the end nodes of the element $k$ and $p$ denotes the order of approximation.

* Expressions for Element Stiffness Matrix and Force Vector
  * $$\boxed{\displaystyle k_{i j}^{k}=\int_{-1}^{1} \hat{E}(\xi) \hat{A}(\xi) \frac{d \hat{N}_{i}(\xi)}{d \xi} \frac{d \hat{N}_{j}(\xi)}{d \xi} \frac{2}{h_{k}} d \xi \quad \text { for } i, j=1,2, \ldots, p+1}$$

  * $$\boxed{\displaystyle f_{i}^{k}=\int_{-1}^{1} \hat{f}(\xi) \hat{N}_{i}(\xi) \frac{h_{k}}{2} d \xi \quad i=1,2, \ldots, p+1}$$

#### Function for Global (All elements of same size) $K_{ij}$ & $f_i$ if 1-D Bar (ElemKfi)

```python
def ElemKfi(k,p,xL,xR,E,a0,a1=0,a2=0,f0=0,f1=0,f2=0):
  """
  =====================================================
  ElemKfi(k,p,xL,xR,E,a0,a1=0,a2=0,f0=0,f1=0,f2=0)

  k :   element number (integer)
  p :   order of approximation (integer)
  xL :  x -coordinate of the left end of the element (real)
  xR :  x -coordinate of the right end of the element (real)
  E :   Young's modules ( E ) of the bar material which is taken as constant (real)
  a0,a1,a2 : coefficients in (A=a0 + a1*x + a2*(x**2)) for the area of cross-section ( A ) of the bar (real)
  f0,f1,f2 : coefficients in (f = f0 + f1*x + f2*x*x) for the distributed force ( f ) acting on the bar (real)

  Output: [Ke, fe]
  Ke : element stiffness matrix (real)
  fe : element force vector (real)
  =====================================================
  """

  Ke=np.zeros((p+1,p+1))  # size of element matrix
  h=xR - xL               # h : element length (real)

  [n, W,xis,ShapeValueXi,dShapeValueXi]=ShapeLegRoot(p)
  fe=np.zeros(p+1)

  for m in range(0,n):
    xi=xis[m]
    x = xL*(1 - z)/2 + xR*(1 + z)/2

    EA = E*(a0 + a1*x+a2*(x**2))
    EA = EA.subs(z,xi)

    f = f0 + f1*x + f2*x*x
    f=f.subs(z,xi)

    for i in range(0, p+1):
      fe[i] = fe[i] + f*ShapeValueXi[m][i]*(h/2)*W[m]

      for j in range(0,p+1):
        Ke[i][j] = Ke[i][j]+EA*dShapeValueXi[m][i]*dShapeValueXi[m][j]*(2/h)*W[m]

  return [Ke, fe]
```

```python
print(ElemKfi.__doc__)
```

      =====================================================
      ElemKfi(k,p,xL,xR,E,a0,a1=0,a2=0,f0=0,f1=0,f2=0)

      k :   element number (integer)
      p :   order of approximation (integer)
      xL :  x -coordinate of the left end of the element (real)
      xR :  x -coordinate of the right end of the element (real)
      E :   Young's modules ( E ) of the bar material which is taken as constant (real)
      a0,a1,a2 : coefficients in (A=a0 + a1*x + a2*(x**2)) for the area of cross-section ( A ) of the bar (real)
      f0,f1,f2 : coefficients in (f = f0 + f1*x + f2*x*x) for the distributed force ( f ) acting on the bar (real)

      Output: [Ke, fe]
      Ke : element stiffness matrix (real)
      fe : element force vector (real)
      =====================================================

---

##### Example:1
**Find Local stiffness matrix for each element and also find force vector due to distributed force.**

```python
[ke,fe]=ElemKfi(k=1,p=1,xL=0,xR=10,a0=6,a1=-0.1,a2=0,E=30,f0=0.1296,f1=-0.00216)
print("Local Stiffness Matrix(1): \n\n", ke,"\n\n")
print("Local Force Vector (due to distributed force):\n\n ", fe)
```

    Local Stiffness Matrix(1):

     [[ 16.5 -16.5]
     [-16.5  16.5]]

    Local Force Vector (due to distributed force):

      [0.612 0.576]

```python
[ke,fe]=ElemKfi(k=2,p=1,xL=10,xR=20,a0=6,a1=-0.1,E=30,f0=0.1296,f1=-0.00216)
print("Local Stiffness Matrix(2): \n\n", ke,"\n\n")
print("Lobal Force Vector (due to distributed force):\n\n ", fe)
```

    Local Stiffness Matrix(2):

     [[ 13.5 -13.5]
     [-13.5  13.5]]

    Lobal Force Vector (due to distributed force):

      [0.504 0.468]

```python
[ke,fe]=ElemKfi(k=3,p=1,xL=20,xR=30,a0=6,a1=-0.1,E=30,f0=0.1296,f1=-0.00216)
print("Local Stiffness Matrix(3): \n\n", ke,"\n\n")
print("Lobal Force Vector (due to distributed force):\n\n ", fe)
```

    Local Stiffness Matrix(3):

     [[ 10.5 -10.5]
     [-10.5  10.5]]

    Lobal Force Vector (due to distributed force):

      [0.396 0.36 ]

---

##### Example:2
**Find Local stiffness matrix for each element and also find force vector due to distributed force.**
- book:
Fundamentals of Finite Element Analysis
Linear Finite Element Analysis-Ioannis Koutromanos
- example-33.2
- page-87/729

**Stiffeness and Force**

```python
[ke,fe]=ElemKfi(k=1,p=1,xL=0,xR=5,E=5000,a0=1,a1=0,a2=1/400,f0=2) #for element-1
print("Local Stiffness Matrix(1): \n\n", ke,"\n\n")
print("Lobal Force Vector (due to distributed force):\n\n ", fe)
```

    Local Stiffness Matrix(1):

     [[ 1020.83333333 -1020.83333333]
     [-1020.83333333  1020.83333333]]

    Lobal Force Vector (due to distributed force):

      [5. 5.]

```python
[ke,fe]=ElemKfi(k=1,p=1,xL=5,xR=20,E=5000,a0=1,a1=0,a2=1/400,f0=2) #for element-2
print("Local Stiffness Matrix(2): \n\n", ke,"\n\n")
print("Lobal Force Vector (due to distributed force):\n\n ", fe)
```

    Local Stiffness Matrix(2):

     [[ 479.16666667 -479.16666667]
     [-479.16666667  479.16666667]]

    Lobal Force Vector (due to distributed force):

      [15. 15.]

---

 #### Function for Global (All elements of same size) $K_{ij}$ & $f_i$ if 1-D Bar (GlobalKfi)

```python
def GlobalKfi(k,p,xL,xR,E,a0,a1=0,a2=0,f0=0,f1=0,f2=0):

  """
  =====================================================
    Note: For elements of equal length

    GlobalKfi(k,p,xL,xR,E,a0,a1=0,a2=0,f0=0,f1=0,f2=0)

    k :   number of elements of same size (integer)
    p :   order of approximation (integer)
    xL :  x -coordinate of the left end of the element (real)
    xR :  x -coordinate of the right end of the element (real)
    E :   Young's modules ( E ) of the bar material which is taken as constant (real)
    a0,a1,a2 : coefficients in (A=a0 + a1*x + a2*(x**2)) for the area of cross-section ( A ) of the bar (real)
    f0,f1,f2 : coefficients in (f = f0 + f1*x + f2*x*x) for the distributed force ( f ) acting on the bar (real)

    Output: [Kg, fg]
    Kg : Global stiffness matrix (real)
    fg : Global force vector (real)
  =====================================================
  """

  Kg=np.zeros((k*p+1,k*p+1))   #size of element matrix
  h=xR - xL

  [n, W,xis,ShapeValueXi,dShapeValueXi]=ShapeLegRoot(p)
  fg=np.zeros(k*p+1)

  for e in range(0,k*p,p):
    #xL=N[e]
    #xR=N[e+1]
    for m in range(0,n):
      xi=xis[m]
      x = (xL+h*int(e/p))*(1 - z)/2 + (xR+h*int(e/p))*(1 + z)/2

      EA = E*(a0 + a1*x+a2*(x**2))
      EA = EA.subs(z,xi)

      f = f0 + f1*x + f2*x*x
      f=f.subs(z,xi)

      for i in range(0, p+1):
        fg[i+e] = fg[i+e] + f*ShapeValueXi[m][i]*(h/2)*W[m]

        for j in range(0,p+1):
          Kg[i+e][j+e] = Kg[i+e][j+e]+EA*dShapeValueXi[m][i]*dShapeValueXi[m][j]*(2/h)*W[m]

  return [Kg, fg]
```

```python
print(GlobalKfi.__doc__)
```

      =====================================================
        Note: For elements of equal length

        GlobalKfi(k,p,xL,xR,E,a0,a1=0,a2=0,f0=0,f1=0,f2=0)

        k :   number of elements of same size (integer)
        p :   order of approximation (integer)
        xL :  x -coordinate of the left end of the element (real)
        xR :  x -coordinate of the right end of the element (real)
        E :   Young's modules ( E ) of the bar material which is taken as constant (real)
        a0,a1,a2 : coefficients in (A=a0 + a1*x + a2*(x**2)) for the area of cross-section ( A ) of the bar (real)
        f0,f1,f2 : coefficients in (f = f0 + f1*x + f2*x*x) for the distributed force ( f ) acting on the bar (real)

        Output: [Kg, fg]
        Kg : Global stiffness matrix (real)
        fg : Global force vector (real)
      =====================================================

---

##### **Example:1**
**Find Global stiffness matrix for each element and also find force vector due to distributed force.**
Take shape function as 1st order of approximation(p=1)

**Solution:**
Local Stiffeness Matrices (alreday evaluated above)

Global Stiffeness Matrix
1. Equal Elements Length (Methode-1)

```python
[Kij, fi]=GlobalKfi(k=3,p=1,xL=0,xR=10,a0=6,a1=-0.1,a2=0,E=30,f0=0.1296,f1=-0.00216)
print("Global Stiffness Matrix: \n\n", Kij,"\n\n")
print("Global Force Vector (due to distributed force):\n\n ", fi)
```

    Global Stiffness Matrix:

     [[ 16.5 -16.5   0.    0. ]
     [-16.5  30.  -13.5   0. ]
     [  0.  -13.5  24.  -10.5]
     [  0.    0.  -10.5  10.5]]

    Global Force Vector (due to distributed force):

      [0.612 1.08  0.864 0.36 ]

---

#### Function for Global (Elements of different Size or same size) $K_{ij}$ & $f_i$ if 1-D Bar (GlobalKfiU)

*   NodesList=Nodes X- Cordinate as list
*   k=Number of element
*   here,K+1=length(NodesList)

```python
def GlobalKfiU(k,p,E,NodesList,a0,a1=0,a2=0,f0=0,f1=0,f2=0):

  """
  =====================================================
Note: For elements of unequal (or equal) length

GlobalKfiU(k,p,E,NodesList,a0,a1=0,a2=0,f0=0,f1=0,f2=0)

k :   number of elements (integer)
p :   order of approximation (integer)
NodesList:  [x0,x1,x2....xk] (X coordinates of nodes)
E :   Young's modules ( E ) of the bar material which is taken as constant (real)
a0,a1,a2 : coefficients in (A=a0 + a1*x + a2*(x**2)) for the area of cross-section ( A ) of the bar (real)
f0,f1,f2 : coefficients in (f = f0 + f1*x + f2*x*x) for the distributed force ( f ) acting on the bar (real)

Output: [Kg, fg]
Kg : Globel stiffness matrix (real)
fg : Globel force vector (real)
  =====================================================
  """

  Kg=np.zeros((k*p+1,k*p+1))                        #size of element matrix

  [n, W,xis,ShapeValueXi,dShapeValueXi]=ShapeLegRoot(p)
  fg=np.zeros(k*p+1)

  for e in range(0,k*p,p):
    xL=NodesList[int(e/p)]
    xR=NodesList[int(e/p)+1]
    h=xR - xL

    for m in range(0,n):
      xi=xis[m]
      x = (xL)*(1 - z)/2 + (xR)*(1 + z)/2

      EA = E*(a0 + a1*x+a2*(x**2))
      EA = EA.subs(z,xi)

      f = f0 + f1*x + f2*x*x
      f=f.subs(z,xi)

      for i in range(0, p+1):
        fg[i+e] = fg[i+e] + f*ShapeValueXi[m][i]*(h/2)*W[m]

        for j in range(0,p+1):
          Kg[i+e][j+e] = Kg[i+e][j+e]+EA*dShapeValueXi[m][i]*dShapeValueXi[m][j]*(2/h)*W[m]

  return [Kg, fg]
```

```python
print(GlobalKfiU.__doc__)
```

      =====================================================
    Note: For elements of unequal (or equal) length

    GlobalKfiU(k,p,E,NodesList,a0,a1=0,a2=0,f0=0,f1=0,f2=0)

    k :   number of elements (integer)
    p :   order of approximation (integer)
    NodesList:  [x0,x1,x2....xk] (X coordinates of nodes)
    E :   Young's modules ( E ) of the bar material which is taken as constant (real)
    a0,a1,a2 : coefficients in (A=a0 + a1*x + a2*(x**2)) for the area of cross-section ( A ) of the bar (real)
    f0,f1,f2 : coefficients in (f = f0 + f1*x + f2*x*x) for the distributed force ( f ) acting on the bar (real)

    Output: [Kg, fg]
    Kg : Globel stiffness matrix (real)
    fg : Globel force vector (real)
      =====================================================

---

##### **Example:2**
**Find Global stiffness matrix for each element and also find force vector due to distributed force.**
(take p=1)
**

**Solution**

```python
[Kij,fi]=GlobalKfiU(k=2,p=1,NodesList=[0,5,20],E=5000,a0=1,a1=0,a2=1/400,f0=2)
print("Global Stiffness Matrix: \n\n", Kij,"\n\n")
print("Global Force Vector (due to distributed force):\n\n ", fi)
```

    Global Stiffness Matrix:

     [[ 1020.83333333 -1020.83333333     0.        ]
     [-1020.83333333  1500.          -479.16666667]
     [    0.          -479.16666667   479.16666667]]

    Global Force Vector (due to distributed force):

      [ 5. 20. 15.]

---

##### Example:1 (Method-2)

```python
[Kij,fi]=GlobalKfiU(k=3,p=1,NodesList=[0,10,20,30],a0=6,a1=-0.1,a2=0,E=30,f0=0.1296,f1=-0.00216) #global K,f
print("Global Stiffness Matrix: \n\n", Kij,"\n\n")
print("Global Force Vector (due to distributed force):\n\n ", fi)
```

    Global Stiffness Matrix:

     [[ 16.5 -16.5   0.    0. ]
     [-16.5  30.  -13.5   0. ]
     [  0.  -13.5  24.  -10.5]
     [  0.    0.  -10.5  10.5]]

    Global Force Vector (due to distributed force):

      [0.612 1.08  0.864 0.36 ]

---

## Future scope
- Find unknown reaction force and displacement vector for any bar problem.
- Code can be optimized for low latency.

---


# Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## White Noise

### Definition

[White noise](https://en.wikipedia.org/wiki/White_noise) is a wide-sense stationary (WSS) random signal with constant power spectral density (PSD)

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = N_0
\end{equation}

where $N_0$ denotes the power per frequency. White noise draws its name from the analogy to white light. It refers typically to an idealized model of a random signal, e.g. emerging from measurement noise. The auto-correlation function (ACF) of white noise can be derived by inverse discrete-time Fourier transformation (DTFT) of the PSD

\begin{equation}
\varphi_{xx}[\kappa] = \mathcal{F}_* \{ N_0 \} = N_0 \cdot \delta[\kappa]
\end{equation}

This result implies that white noise has to be a zero-mean random process. It can be concluded from the ACF that two neighboring samples $k$ and $k+1$ are uncorrelated. Hence they show no dependencies in the statistical sense. Although this is often assumed, the probability density function (PDF) of white noise is not necessarily given by the normal distribution. In general, it is required to additionally state the amplitude distribution when denoting a signal as white noise.

### Example

Toolboxes for numerical mathematics like `Numpy` or `scipy.stats` provide functions to draw uncorrelated random samples with a given amplitude distribution. In order to evaluate this, a function is defined to estimate and plot the PDF and CDF for a given random signal $x[k]$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

def estimate_plot_pdf_acf(x, nbins=50, acf_range=30):

    # compute and truncate ACF
    acf = 1/len(x) * np.correlate(x, x, mode='full')
    acf = acf[len(x)-acf_range-1:len(x)+acf_range-1]
    kappa = np.arange(-acf_range, acf_range)

    # plot PSD
    plt.figure(figsize = (10, 6))
    plt.subplot(121)
    plt.hist(x, nbins, normed=True)
    plt.title('Estimated PDF')
    plt.xlabel(r'$\theta$')
    plt.ylabel(r'$\hat{p}_x(\theta)$')

    # plot ACF
    plt.subplot(122)
    plt.stem(kappa, acf)
    plt.title('Estimated ACF')
    plt.ylabel(r'$\hat{\varphi}_{xx}[\kappa]$')
    plt.xlabel(r'$\kappa$')
    plt.axis([-acf_range, acf_range, 0, 1.1*max(acf)]);
    plt.grid()
```

**Uniformly distributed white noise**

For samples drawn from a zero-mean random process with uniform amplitude distribution, the PDF and ACF are estimated as

```python
np.random.seed(3)
estimate_plot_pdf_acf(np.random.uniform(size=10000)-1/2)
```

Lets listen to uniformly distributed white noise

```python
from scipy.io import wavfile
fs = 44100

x = np.random.uniform(size=5*fs)-1/2
wavfile.write('uniform_white_noise.wav', fs, np.int16(x*32768))
```

<audio src="./uniform_white_noise.wav" controls>Your browser does not support the audio element.</audio>[./uniform_white_noise.wav](./uniform_white_noise.wav)

**Laplace distributed white noise**

For samples drawn from a zero-mean random process with with Laplace amplitude distribution, the PDF and ACF are estimated as

```python
estimate_plot_pdf_acf(np.random.laplace(size=10000, loc=0, scale=1/np.sqrt(2)))
```

**Exercise**

* Do both random processes represent white noise?
* Estimate the power spectral density $N_0$ of both examples.
* How does the ACF change if you lower the length `size` of the random signal. Why?

Both processes represent white noise since the ACF can be approximated reasonably well as Dirac impulse $\delta[\kappa]$. The weight of the Dirac impulse is equal to $N_0$. In case of the uniformly distributed white noise $N_0 \approx \frac{1}{12}$, in case of the Laplace distributed white noise $N_0 \approx 1$. Decreasing the length `size` of the signal increases the statistical uncertainties in the estimate of the ACF.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


# Inverted Pendulum Control
## Summary
This notebook outlines state space controller design for an inverted pendulum on a cart.

This is a classic engineering school design problem. This notebook is light on theory and borrows heavily from the excellent outline by [University of Michigan Engineering Control Tutorials](http://ctms.engin.umich.edu/CTMS/index.php?example=InvertedPendulum&section=SystemModeling).

## Goals
The ultimate goal is to apply the control system to a simulated environment - for this I have chosen Python and specifially [pymunk](http://www.pymunk.org) as the 2D physics simulator and [pyglet](http://www.pyglet.org) as the game engine to render the results.

### Control System Requirements
The design criteria for this system with the cart receiving a 0.2 m step input are as follows:
* Settling time for x and Θ of less than 5 seconds
* Rise time for x of less than 1 second
* Overshoot of Θ less than 20 degrees (0.35 radians)

## System Description
We will use the following diagram to derive the equations of motion (image from Wikipedia):

* $\theta$ is 0 radians when vertical and is positive in the CCW direction
* $x, y$ are the position of the cart, $y$ will be constant and considered 0
* $x1, y1$ are the pendulum position
* $v1, v2$ are cart and pendulum linear velocities
* $b$ is friction, and is not shown in the image, but is a force that acts against $F$ and depends on the velocity of the cart, $v1$
* $I$ is the pendulum moment of inertia
* $M, m$ are mass, in kg, of cart and pendulum respectively
* $l$ is the distance to the pendulum center of mass
* $g$ is gravity
* $\omega$ is the angular velocity of the pendulum, and equivalent to $\frac{d}{dt}\theta$

### Equations of Motion
The equations of motion for this system are derived using the Lagrangian and Euler equations. This derivation references the following sources:
* [University of Michigan: Control Tutorials](http://ctms.engin.umich.edu/CTMS/index.php?example=InvertedPendulum&section=SystemModeling)
* [Wikipedia: Inverted Pendulum](https://en.wikipedia.org/wiki/Inverted_pendulum)
* [Brilliant: Lagrangian Mechanics](https://brilliant.org/wiki/lagrangian-formulation-of-mechanics/)

#### Lagrangian
The Lagrangian, L, is the difference between kinetic energy, T and potential energy V.
$L = T - V$

In our case, we need to consider kinetic energy of the cart and pendulum (including rotational kinetic energy of the pendulum), and potential energy of the pendulum (cart is assumed to be on ground so it has no potential energy).
$L = \frac{1}{2}Mv_{1}^2 + \frac{1}{2}mv_{2}^2 + \frac{1}{2}I\omega^2 - mgl\cos\theta$

#### Euler Equations for this system
Both x and Θ vary over time, so the Euler equations are as follows:

$\dfrac{d}{dt}\dfrac{\partial L}{\partial \dot{x}} - \dfrac{\partial L}{\partial x} = F - b\dot{x}$

$\dfrac{d}{dt}\dfrac{\partial L}{\partial \dot{\theta}} - \dfrac{\partial L}{\partial \theta} = 0$

### derivation of equations of motion

```python
import sympy
sympy.init_printing()
```

```python
# create our functions for x and Θ against time, t
t = sympy.symbols('t')
x = sympy.Function('x')(t)
Θ = sympy.Function('Θ')(t)

# constants: pendulum length, mass, and inertia, then cart mass, gravity, and friction
l, m, I, M, g, b = sympy.symbols('l m I M g b')

# cart position is x, y, but we can ignore y since it will be constant and zero
# cart velocity is then simply the first derivate of it's x position
v1 = x.diff(t)

# pendulum position, WRT to the cart
x2 = x - l*sympy.sin(Θ)
y2 = l*sympy.cos(Θ)

# pendulum velocity is made up of x and y velocities, found from first derivates of position
v2 = sympy.sqrt(sympy.simplify(x2.diff(t)**2 + y2.diff(t)**2))

# kinetic energy of the cart, plus kinetic energy of the pendulum, subtract potential energy of pendulum and cart
L = sympy.Rational(1, 2)*(M*v1**2 + m*v2**2 + I*Θ.diff(t)**2) - m*g*l*sympy.cos(Θ)

# cleanup a bit, expand and collect on linear and angular acceleration
L = L.expand().collect((Θ.diff(t, t), x.diff(t, t))).simplify()

# print out the Lagrangian
L
```

```python
# euler equations
F = sympy.symbols('F')
euler1 = L.diff(x.diff(t)).diff(t) - L.diff(x)
euler2 = L.diff(Θ.diff(t)).diff(t) - L.diff(Θ)

# create the equations, first arg is LHS, second is RHS
eq1 = sympy.Eq(euler1.simplify().expand().collect(x.diff(t, t)), F - b*x.diff(t))
eq2 = sympy.Eq(euler2.simplify().expand().collect(Θ.diff(t, t)), 0)

# print out the 2 equations
eq1, eq2
```

```python
# linearize our system
# we want the pendulum vertical, so linearize around Θ = 0
# sin(Θ) = Θ, cos(Θ) = 1, (d/dt(Θ))**2 = 0
linearize = [(sympy.sin(Θ), Θ), (sympy.cos(Θ), 1), (Θ.diff(t)**2, 0)]
eq1_lin = eq1.subs(linearize)
eq2_lin = eq2.subs(linearize)

# print out the linearized equations
eq1_lin, eq2_lin
```

```python
# solve our linear system for linear and angular acceleration
eq_of_mot = sympy.linsolve([eq1_lin, eq2_lin], [x.diff(t, t), Θ.diff(t, t)])

# get our linear and angular velocity equations
lin_accel = eq_of_mot.args[0][0].expand().collect((Θ, Θ.diff(t), x, x.diff(t), F)).simplify()
ang_accel = eq_of_mot.args[0][1].expand().collect((Θ, Θ.diff(t), x, x.diff(t), F)).simplify()

# print the solutions
lin_accel, ang_accel
```

#### State Space Representation
$\dot{x(t)} = Ax(t) + Bu(t)$

$y(t) = Cx(t) + Du(t)$

$\dot{x(t)} =
\begin{bmatrix}
\dot{x} \\
\ddot{x} \\
\dot{\theta} \\
\ddot{\theta} \\
\end{bmatrix}
$      $x(t) =
\begin{bmatrix}
x \\
\dot{x} \\
\theta \\
\dot{\theta} \\
\end{bmatrix}
$

$A =
\begin{bmatrix}
  0 & 1 & 0 & 0 \\
  0 & \dfrac{-b(I + l^2m)}{I(M + m) + Ml^2m} & \dfrac{gl^2m^2}{I(M + m) + Ml^2m} & 0 \\
  0 & 0 & 0 & 1 \\
  0 & \dfrac{-blm}{I(M + m) + Ml^2m} & \dfrac{lmg(M + m)}{I(M + m) + Ml^2m} & 0
\end{bmatrix}
$      $B =
 \begin{bmatrix}
  0 \\
  \dfrac{I + l^2m}{I(M + m) + Ml^2m} \\
  0 \\
  \dfrac{lm}{I(M + m) + Ml^2m}
 \end{bmatrix}
$      $C =
 \begin{bmatrix}
  1 & 0 & 0 & 0 \\
  0 & 0 & 1 & 0
 \end{bmatrix}
$      $D =
 \begin{bmatrix}
  0 \\
  0
 \end{bmatrix}
$

### controller design

```python
# start of controller design
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal
import scipy.linalg
import math
```

```python
# constants for model
constants = {
    M: 0.5,   # mass of cart
    m: 0.2,   # mass of pendulum
    b: 0.1,   # friction of cart
    l: 0.6,   # length to pendulum center of mass
    I: 0.001, # inertia of pendulum
    g: 9.81,  # gravity
}
```

```python
# state space model
p = (I*(M+m)+M*l**2*m).evalf(subs=constants)  # denominator for the A and B matrices
A = np.matrix([
    [0,              1,             0, 0],
    [0, (-b*(I+l**2*m)/p).evalf(subs=constants), ((g*l**2*m**2)/p).evalf(subs=constants), 0],
    [0,              0,             0, 1],
    [0,     (-(b*l*m)/p).evalf(subs=constants), (l*m*g*(M+m)/p).evalf(subs=constants), 0]
], dtype='float64')
B = np.matrix([
    [0],
    [((I+l**2*m)/p).evalf(subs=constants)],
    [0],
    [(l*m/p).evalf(subs=constants)]
], dtype='float64')
C = np.matrix([
    [1, 0, 0, 0],
    [0, 0, 1, 0]
], dtype='float64')
D = np.matrix([
    [0],
    [0]
], dtype='float64')
```

```python
# plot the poles to look at open loop system stability
poles = scipy.linalg.eigvals(A)
plt.scatter([p.real for p in poles], [p.imag for p in poles], color='red', marker='x')
plt.axvline(0, color='k', linestyle='dashed')
plt.axhline(0, color='k', linestyle='dashed')
plt.grid(True)
plt.xlabel('Real')
plt.ylabel('Imaginary')
plt.title('Poles')
plt.show()
```

```python
# system controllability - if rank[B, AB, A^2B, A^3B] == 4, it is controllable
RM = np.concatenate((B, A * B, np.linalg.matrix_power(A, 2) * B, np.linalg.matrix_power(A, 3) * B), axis=1)
if np.linalg.matrix_rank(RM) == A.shape[0]:
    print('System is controllable')
else:
    print('System is NOT controllable!')
```

    System is controllable

```python
# system observability
OM = np.concatenate((C, C * A, C * np.linalg.matrix_power(A, 2), C * np.linalg.matrix_power(A, 3)), axis=0)
if np.linalg.matrix_rank(OM) == A.shape[0]:
    print('System is observable')
else:
    print('System is NOT observable!')
```

    System is observable

### System Overview
System is controllable and observable, and because it has a pole in the RHS, it is unstable

```python
# system has a pole in the right hand plane and looks unstable, simulate the step response with 0 initial conditions
system = scipy.signal.lti(A, B, C, D)
t = np.linspace(0, 2)
u = np.ones_like(t) * 0.2
tout, yout, xout = scipy.signal.lsim(system, u, t)

fig, ax1 = plt.subplots()
ax1.plot(tout, yout[:, :1], 'b-')
ax1.set_xlabel('time (s)')
ax1.set_ylabel('Cart Position (m)', color='b')
ax1.tick_params('y', colors='b')

ax2 = ax1.twinx()
ax2.plot(tout, yout[:, 1:], 'r-')
ax2.set_ylabel('Pendulum Angle (Radians)', color='r')
ax2.tick_params('y', colors='r')

plt.grid(True)
plt.title('Open Loop Step Response')
plt.show()
```

```python
# define an LQR function, adapted from http://www.mwm.im/lqr-controllers-with-python/
def lqr(A, B, Q, R):
    X = np.matrix(scipy.linalg.solve_continuous_are(A, B, Q, R))
    K = np.matrix(scipy.linalg.inv(R) * (B.T * X))
    return K
```

```python
# find structure of Q
C.T * C
```

    matrix([[1., 0., 0., 0.],
            [0., 0., 0., 0.],
            [0., 0., 1., 0.],
            [0., 0., 0., 0.]])

```python
# element in 0,0 position used to weight cart's position, element in 2,2 position to weight pendulum angle
x = 5000
y = 100

Q = np.matrix([
    [x, 0, 0, 0],
    [0, 0, 0, 0],
    [0, 0, y, 0],
    [0, 0, 0, 0]
])
R = np.ones((1, 1));
K = lqr(A, B, Q, R)

# simulate the system with the calculated gain
Ac = A - B * K
system = scipy.signal.lti(Ac, B, C, D)
t = np.linspace(0, 3)
u = np.ones_like(t) * 0.2
tout, yout, xout = scipy.signal.lsim(system, u, t)

fig, ax1 = plt.subplots()
ax1.plot(tout, yout[:, :1], 'b-')
ax1.set_xlabel('time (s)')
ax1.set_ylabel('Cart Position (m)', color='b')
ax1.tick_params('y', colors='b')

ax2 = ax1.twinx()
ax2.plot(tout, yout[:, 1:], 'r-')
ax2.set_ylabel('Pendulum Angle (Radians)', color='r')
ax2.tick_params('y', colors='r')

plt.grid(True)
plt.title('Step Response with LQR Control')
plt.show()
```

```python
# adapted from: http://ctms.engin.umich.edu/CTMS/index.php?aux=Extras_rscale
def rscale(A, B, C, D, K):
    s = A.shape[0]
    Z = np.append(np.zeros((1, s)), 1)
    ABcat = np.concatenate((A, B), axis=1)
    CDcat = np.concatenate((C, D), axis=1)
    ABCDinv = np.linalg.inv(np.concatenate((ABcat, CDcat), axis=0))
    N = ABCDinv.dot(Z.T)
    Nx = N.A[0][0:s]
    Nu = N.A[0][s]
    if hasattr(K, 'gain_matrix'):
        Nbar = Nu + K.gain_matrix.dot(Nx)
    else:
        Nbar = Nu + K.dot(Nx)
    return Nbar
```

### What is 'rscale'?
We want to be able to provide the cart position (x-direction) as an input - the rscale function let's us map this input to something the system understands - in this case the x-direction force applied to the cart.

If you look at the above graph, we have applied a step input of 0.2 and this has resulted in a stable system (pendulum angle is at 0) but the x-position of the cart is not at 0.2.

```python
# let's remove steady state error - step input is 0.2m so we want the cart to reach this position
# "A different C had to be used because the rscale function will not work for multiple outputs"
Cn = C[:1]
Dn = D[:1,0]
Nbar = rscale(A, B, Cn, Dn, K)

# simulate the system with Nbar - we should see the cart position reach steady state of 0.2m
Bcn = B.dot(Nbar)
system = scipy.signal.lti(Ac, Bcn, C, D)
t = np.linspace(0, 3)
u = np.ones_like(t) * 0.2
tout, yout, xout = scipy.signal.lsim(system, u, t)

fig, ax1 = plt.subplots()
ax1.plot(tout, yout[:, :1], 'b-')
ax1.set_xlabel('time (s)')
ax1.set_ylabel('Cart Position (m)', color='b')
ax1.tick_params('y', colors='b')
ax1.grid(True)

ax2 = ax1.twinx()
ax2.plot(tout, yout[:, 1:], 'r-')
ax2.set_ylabel('Pendulum Angle (Radians)', color='r')
ax2.tick_params('y', colors='r')

plt.title('Step Response with LQR Control and Nbar')
plt.show()
```

### System Design Results
We have now met our design requirements:
* Settling time for x and Θ of less than 5 seconds
* Rise time for x of less than 1 second
* Overshoot of Θ less than 20 degrees (0.35 radians)

### Digital Controller Design
Because our physics simulation is in discrete time, we need to convert our system to discrete time and perform similar steps. We will be sampling the states 60 times per second in the simulation.

```python
# let's convert the system into discrete time, so we can simulate with pymunk physics engine and pyglet game engine
# we'll sample state 60 times per second
ts = 1.0/60.0
F, G, H, J, dt = scipy.signal.cont2discrete((A, B, C, D), ts)
dsystem = (F, G, H, J, dt)
```

```python
# test the open loop system against a step input
system = scipy.signal.dlti(F, G, H, J)
tsim = 2
samples = math.ceil(tsim / ts)
t = np.linspace(0, tsim, samples)
u = np.ones_like(t) * 0.2
tout, yout, xout = scipy.signal.dlsim(system, u)

fig, ax1 = plt.subplots()
ax1.step(tout * dt, yout[:, :1], 'b-')
ax1.set_xlabel('time (s)')
ax1.set_ylabel('Cart Position (m)', color='b')
ax1.tick_params('y', colors='b')

ax2 = ax1.twinx()
ax2.step(tout * dt, yout[:, 1:], 'r-')
ax2.set_ylabel('Pendulum Angle (Radians)', color='r')
ax2.tick_params('y', colors='r')

plt.grid(True)
plt.title('Digital Open Loop Step Response')
plt.show()
```

```python
# define an discrete time LQR function - created using the following reference:
# https://en.wikipedia.org/wiki/Linear%E2%80%93quadratic_regulator
def dlqr(A, B, Q, R):
    P = np.matrix(scipy.linalg.solve_discrete_are(A, B, Q, R))
    K = np.matrix(scipy.linalg.inv(R + B.T * P * B) * (B.T * P * A))
    return K
```

```python
# design digital LQR controller - using same gains as before
x = 5000
y = 100
Qd = np.matrix([
    [x, 0, 0, 0],
    [0, 0, 0, 0],
    [0, 0, y, 0],
    [0, 0, 0, 0]
])
R = np.ones((1, 1));
Kd = dlqr(F, G, Qd, R)

# simulate the system with the calculated gain
Fc = F - G * Kd
system = scipy.signal.dlti(Fc, G, H, J)
tsim = 3
samples = math.ceil(tsim / ts)
t = np.linspace(0, tsim, samples)
u = np.ones_like(t) * 0.2
tout, yout, xout = scipy.signal.dlsim(system, u)

fig, ax1 = plt.subplots()
ax1.step(tout * dt, yout[:, :1], 'b-')
ax1.set_xlabel('time (s)')
ax1.set_ylabel('Cart Position (m)', color='b')
ax1.tick_params('y', colors='b')

ax2 = ax1.twinx()
ax2.step(tout * dt, yout[:, 1:], 'r-')
ax2.set_ylabel('Pendulum Angle (Radians)', color='r')
ax2.tick_params('y', colors='r')

plt.grid(True)
plt.title('Step Response with Digital LQR Control')
plt.show()
```

```python
Nbar
```

    matrix([[-70.71067812]])

### Digital Nbar
We can't use the original rscale function because it is for a continuous time (CT) system, but we can use the CT Nbar value as a starting point and manually tweak it to get our steady state to the desired 0.2 m output.

```python
# let's remove steady state error - step input is 0.2 m so we want the cart to reach this position
# we can't use rscale since it was for continuous time (CT) systems, but we can use CT Nbar as a starting point
# and tweak it
# using trial and error - started with Nbar value
Nbard = np.matrix([[-57.25]])

# simulate the system with Nbar - we should see the cart position reach steady state of 0.2 m if Nbard is set
# appropriately
Gcn = G * Nbard
system = scipy.signal.dlti(Fc, Gcn, H, J)
tsim = 3
samples = math.ceil(tsim / ts)
t = np.linspace(0, tsim, samples)
u = np.ones_like(t) * 0.2
tout, yout, xout = scipy.signal.dlsim(system, u)

fig, ax1 = plt.subplots()
ax1.step(tout * dt, yout[:, :1], 'b-')
ax1.set_xlabel('time (s)')
ax1.set_ylabel('Cart Position (m)', color='b')
ax1.tick_params('y', colors='b')
ax1.grid(True)

ax2 = ax1.twinx()
ax2.step(tout * dt, yout[:, 1:], 'r-')
ax2.set_ylabel('Pendulum Angle (Radians)', color='r')
ax2.tick_params('y', colors='r')

plt.title('Step Response with Digital LQR Control and Nbar')
plt.show()
```

```python
# dump out our gain matrix so we can use it in our simulation
Kd
```

    matrix([[-57.38901804, -36.24133932, 118.51380879,  28.97241832]])

### Applying this to Code
We can derive the function for our input (u) using the following diagram of the system:

$\vec{u} = \vec{r}N - K\vec{x}$

Where $\vec{r}$ is our desired results (just the x-position of the cart in this case), N is our digital Nbar matrix (a 1x1 matrix in this case), K is our digital gain matrix, and $\vec{x}$ is the state vector.

There are 4 values in the gain matrix K that correspond to the 4 states we are measuring:
* cart x-position
* cart x-velocity
* pendulum angle from vertical
* pendulum angular velocity.

The input vector $\vec{u}$ in this case only has the x-direciton force to apply to the cart.

In practice, the code will look something like this:

```python
# this code would sit in some periodic loop (at our sampling rate, which in this case is 60 times per second)

# constants - gain matrix and Nbar
K = [-57.38901804, -36.24133932, 118.51380879,  28.97241832]
Nbar = -57.25

# the following states would be sampled
# posx is cart position (x-direction)
# velx is cart velocity (x-direction)
# ang is pendulum angle
# angv is pendulum angular velocity

# calculate the gain based on the current state
gain = K[0] * posx + K[1] * velx + K[2] * ang + K[3] * angv

# calculate the required force (u)
# ref is our desired x-position of the cart - in our examples this was 0.2 m
force = ref * NBAR - gain

# we can now apply this force to the cart center of mass (in the x-direction)
```

```python
# plot results from simulation run in pyglet/pymunk
import csv

t, x, theta = [], [], []
with open('../data/invpend.csv', 'r') as f:
    for row in csv.reader(f):
        t.append(row[0])
        x.append(row[1])
        theta.append(row[2])

# ignore first row (labels) and convert data to floats
t = [float(a) for a in t[1:]]
x = [float(a) for a in x[1:]]
theta = [float(a) for a in theta[1:]]

fig, ax1 = plt.subplots()
ax1.step(t, x, 'b-')
ax1.set_xlabel('time (s)')
ax1.set_ylabel('Cart Position (m)', color='b')
ax1.tick_params('y', colors='b')
ax1.grid(True)

ax2 = ax1.twinx()
ax2.step(t, theta, 'r-')
ax2.set_ylabel('Pendulum Angle (Radians)', color='r')
ax2.tick_params('y', colors='r')

plt.title('Response from Simulation')
plt.show()
```

```python

```