# Exploring data with Python - real world data

Last time, we looked at grades for our student data, and investigated this visually with histograms and box plots. Now we will look into more complex cases, describe the data more fully, and discuss how to make basic comparisons between data.

### Real world data distributions

Last time, we looked at grades for our student data, and estimated from this sample what the full population of grades might look like. Just to refresh, lets take a look at this data again.

Run the code below to print out the data and make a histogram + boxplot that show the grades for our sample of students.

```python
import pandas as pd
from matplotlib import pyplot as plt

# Load data from a text file
df_students = pd.read_csv('data/grades.csv',delimiter=',',header='infer')

# Remove any rows with missing data
df_students = df_students.dropna(axis=0, how='any')

# Calculate who passed, assuming '60' is the grade needed to pass
passes  = pd.Series(df_students['Grade'] >= 60)

# Save who passed to the Pandas dataframe
df_students = pd.concat([df_students, passes.rename("Pass")], axis=1)

# Print the result out into this notebook
print(df_students)

# Create a function that we can re-use
def show_distribution(var_data):
    '''
    This function will make a distribution (graph) and display it
    '''

    # Get statistics
    min_val = var_data.min()
    max_val = var_data.max()
    mean_val = var_data.mean()
    med_val = var_data.median()
    mod_val = var_data.mode()[0]

    print('Minimum:{:.2f}\nMean:{:.2f}\nMedian:{:.2f}\nMode:{:.2f}\nMaximum:{:.2f}\n'.format(min_val,
                                                                                mean_val,
                                                                                med_val,
                                                                                mod_val,
                                                                                max_val))

    # Create a figure for 2 subplots (2 rows, 1 column)
    fig, ax = plt.subplots(2, 1, figsize = (10,4))

    # Plot the histogram
    ax[0].hist(var_data)
    ax[0].set_ylabel('Frequency')

    # Add lines for the mean, median, and mode
    ax[0].axvline(x=min_val, color = 'gray', linestyle='dashed', linewidth = 2)
    ax[0].axvline(x=mean_val, color = 'cyan', linestyle='dashed', linewidth = 2)
    ax[0].axvline(x=med_val, color = 'red', linestyle='dashed', linewidth = 2)
    ax[0].axvline(x=mod_val, color = 'yellow', linestyle='dashed', linewidth = 2)
    ax[0].axvline(x=max_val, color = 'gray', linestyle='dashed', linewidth = 2)

    # Plot the boxplot
    ax[1].boxplot(var_data, vert=False)
    ax[1].set_xlabel('Value')

    # Add a title to the Figure
    fig.suptitle('Data Distribution')

    # Show the figure
    fig.show()

show_distribution(df_students['Grade'])
```

As you might recall, our data had the mean and mode at the center, with data spread symmetrically from there.

Now let's take a look at the distribution of the study hours data.

```python
# Get the variable to examine
col = df_students['StudyHours']
# Call the function
show_distribution(col)
```

The distribution of the study time data is significantly different from that of the grades.

Note that the whiskers of the box plot only begin at around 6.0, indicating that the vast majority of the first quarter of the data is above this value. The minimum is marked with an **o**, indicating that it is statistically an *outlier* - a value that lies significantly outside the range of the rest of the distribution.

Outliers can occur for many reasons. Maybe a student meant to record "10" hours of study time, but entered "1" and missed the "0". Or maybe the student was abnormally lazy when it comes to studying! Either way, it's a statistical anomaly that doesn't represent a typical student. Let's see what the distribution looks like without it.

```python
# Get the variable to examine
# We will only get students who have studied more than one hour
col = df_students[df_students.StudyHours>1]['StudyHours']

# Call the function
show_distribution(col)
```

For learning purposes we have just treated the value **1** is a true outlier here and excluded it. In the real world, though, it would be unusual to exclude data at the extremes without more justification when our sample size is so small. This is because the smaller our sample size, the more likely it is that our sampling is a bad representation of the whole population (here, the population means grades for all students, not just our 22). For example, if we sampled study time for another 1000 students, we might find that it's actually quite common to not study much!

When we have more data available, our sample becomes more reliable. This makes it easier to consider outliers as being values that fall below or above percentiles within which most of the data lie. For example, the following code uses the Pandas **quantile** function to exclude observations below the 0.01th percentile (the value above which 99% of the data reside).

```python
# calculate the 0.01th percentile
q01 = df_students.StudyHours.quantile(0.01)
# Get the variable to examine
col = df_students[df_students.StudyHours>q01]['StudyHours']
# Call the function
show_distribution(col)
```

> **Tip**: You can also eliminate outliers at the upper end of the distribution by defining a threshold at a high percentile value - for example, you could use the **quantile** function to find the 0.99 percentile below which 99% of the data reside.

With the outliers removed, the box plot shows all data within the four quartiles. Note that the distribution is not symmetric like it is for the grade data though - there are some students with very high study times of around 16 hours, but the bulk of the data is between 7 and 13 hours; The few extremely high values pull the mean towards the higher end of the scale.

Let's look at the density for this distribution.

```python
def show_density(var_data):
    fig = plt.figure(figsize=(10,4))

    # Plot density
    var_data.plot.density()

    # Add titles and labels
    plt.title('Data Density')

    # Show the mean, median, and mode
    plt.axvline(x=var_data.mean(), color = 'cyan', linestyle='dashed', linewidth = 2)
    plt.axvline(x=var_data.median(), color = 'red', linestyle='dashed', linewidth = 2)
    plt.axvline(x=var_data.mode()[0], color = 'yellow', linestyle='dashed', linewidth = 2)

    # Show the figure
    plt.show()

# Get the density of StudyHours
show_density(col)
```

This kind of distribution is called *right skewed*. The mass of the data is on the left side of the distribution, creating a long tail to the right because of the values at the extreme high end; which pull the mean to the right.

#### Measures of variance

So now we have a good idea where the middle of the grade and study hours data distributions are. However, there's another aspect of the distributions we should examine: how much variability is there in the data?

Typical statistics that measure variability in the data include:

- **Range**: The difference between the maximum and minimum. There's no built-in function for this, but it's easy to calculate using the **min** and **max** functions.
- **Variance**: The average of the squared difference from the mean. You can use the built-in **var** function to find this.
- **Standard Deviation**: The square root of the variance. You can use the built-in **std** function to find this.

```python
for col_name in ['Grade','StudyHours']:
    col = df_students[col_name]
    rng = col.max() - col.min()
    var = col.var()
    std = col.std()
    print('\n{}:\n - Range: {:.2f}\n - Variance: {:.2f}\n - Std.Dev: {:.2f}'.format(col_name, rng, var, std))
```

Of these statistics, the standard deviation is generally the most useful. It provides a measure of variance in the data on the same scale as the data itself (so grade points for the Grade distribution and hours for the StudyHours distribution). The higher the standard deviation, the more variance there is when comparing values in the distribution to the distribution mean - in other words, the data is more spread out.

When working with a *normal* distribution, the standard deviation works with the particular characteristics of a normal distribution to provide even greater insight. Run the cell below to see the relationship between standard deviations and the data in the normal distribution.

```python
import scipy.stats as stats

# Get the Grade column
col = df_students['Grade']

# get the density
density = stats.gaussian_kde(col)

# Plot the density
col.plot.density()

# Get the mean and standard deviation
s = col.std()
m = col.mean()

# Annotate 1 stdev
x1 = [m-s, m+s]
y1 = density(x1)
plt.plot(x1,y1, color='magenta')
plt.annotate('1 std (68.26%)', (x1[1],y1[1]))

# Annotate 2 stdevs
x2 = [m-(s*2), m+(s*2)]
y2 = density(x2)
plt.plot(x2,y2, color='green')
plt.annotate('2 std (95.45%)', (x2[1],y2[1]))

# Annotate 3 stdevs
x3 = [m-(s*3), m+(s*3)]
y3 = density(x3)
plt.plot(x3,y3, color='orange')
plt.annotate('3 std (99.73%)', (x3[1],y3[1]))

# Show the location of the mean
plt.axvline(col.mean(), color='cyan', linestyle='dashed', linewidth=1)

plt.axis('off')

plt.show()
```

The horizontal lines show the percentage of data within 1, 2, and 3 standard deviations of the mean (plus or minus).

In any normal distribution:
- Approximately 68.26% of values fall within one standard deviation from the mean.
- Approximately 95.45% of values fall within two standard deviations from the mean.
- Approximately 99.73% of values fall within three standard deviations from the mean.

So, since we know that the mean grade is 49.18, the standard deviation is 21.74, and distribution of grades is approximately normal; we can calculate that 68.26% of students should achieve a grade between 27.44 and 70.92.

The descriptive statistics we've used to understand the distribution of the student data variables are the basis of statistical analysis; and because they're such an important part of exploring your data, there's a built-in **Describe** method of the DataFrame object that returns the main descriptive statistics for all numeric columns.

```python
df_students.describe()
```

## Comparing data

Now that you know something about the statistical distribution of the data in your dataset, you're ready to examine your data to identify any apparent relationships between variables.

First of all, let's get rid of any rows that contain outliers so that we have a sample that is representative of a typical class of students. We identified that the StudyHours column contains some outliers with extremely low values, so we'll remove those rows.

```python
df_sample = df_students[df_students['StudyHours']>1]
df_sample
```

### Comparing numeric and categorical variables

The data includes two *numeric* variables (**StudyHours** and **Grade**) and two *categorical* variables (**Name** and **Pass**). Let's start by comparing the numeric **StudyHours** column to the categorical **Pass** column to see if there's an apparent relationship between the number of hours studied and a passing grade.

To make this comparison, let's create box plots showing the distribution of StudyHours for each possible Pass value (true and false).

```python
df_sample.boxplot(column='StudyHours', by='Pass', figsize=(8,5))
```

Comparing the StudyHours distributions, it's immediately apparent (if not particularly surprising) that students who passed the course tended to study for more hours than students who didn't. So if you wanted to predict whether or not a student is likely to pass the course, the amount of time they spend studying may be a good predictive feature.

### Comparing numeric variables

Now let's compare two numeric variables. We'll start by creating a bar chart that shows both grade and study hours.

```python
# Create a bar plot of name vs grade and study hours
df_sample.plot(x='Name', y=['Grade','StudyHours'], kind='bar', figsize=(8,5))
```

The chart shows bars for both grade and study hours for each student; but it's not easy to compare because the values are on different scales. Grades are measured in grade points, and range from 3 to 97; while study time is measured in hours and ranges from 1 to 16.

A common technique when dealing with numeric data in different scales is to *normalize* the data so that the values retain their proportional distribution, but are measured on the same scale. To accomplish this, we'll use a technique called *MinMax* scaling that distributes the values proportionally on a scale of 0 to 1. You could write the code to apply this transformation; but the **Scikit-Learn** library provides a scaler to do it for you.

```python
from sklearn.preprocessing import MinMaxScaler

# Get a scaler object
scaler = MinMaxScaler()

# Create a new dataframe for the scaled values
df_normalized = df_sample[['Name', 'Grade', 'StudyHours']].copy()

# Normalize the numeric columns
df_normalized[['Grade','StudyHours']] = scaler.fit_transform(df_normalized[['Grade','StudyHours']])

# Plot the normalized values
df_normalized.plot(x='Name', y=['Grade','StudyHours'], kind='bar', figsize=(8,5))
```

With the data normalized, it's easier to see an apparent relationship between grade and study time. It's not an exact match, but it definitely seems like students with higher grades tend to have studied more.

So there seems to be a correlation between study time and grade; and in fact, there's a statistical *correlation* measurement we can use to quantify the relationship between these columns.

```python
df_normalized.Grade.corr(df_normalized.StudyHours)
```

The correlation statistic is a value between -1 and 1 that indicates the strength of a relationship. Values above 0 indicate a *positive* correlation (high values of one variable tend to coincide with high values of the other), while values below 0 indicate a *negative* correlation (high values of one variable tend to coincide with low values of the other). In this case, the correlation value is close to 1; showing a strongly positive correlation between study time and grade.

> **Note**: Data scientists often quote the maxim "*correlation* is not *causation*". In other words, as tempting as it might be, you shouldn't interpret the statistical correlation as explaining *why* one of the values is high. In the case of the student data, the statistics demonstrates that students with high grades tend to also have high amounts of study time; but this is not the same as proving that they achieved high grades *because* they studied a lot. The statistic could equally be used as evidence to support the nonsensical conclusion that the students studied a lot *because* their grades were going to be high.

Another way to visualise the apparent correlation between two numeric columns is to use a *scatter* plot.

```python
# Create a scatter plot
df_sample.plot.scatter(title='Study Time vs Grade', x='StudyHours', y='Grade')
```

Again, it looks like there's a discernible pattern in which the students who studied the most hours are also the students who got the highest grades.

We can see this more clearly by adding a *regression* line (or a *line of best fit*) to the plot that shows the general trend in the data. To do this, we'll use a statistical technique called *least squares regression*.

> **Warning - Math Ahead!**
>
> Cast your mind back to when you were learning how to solve linear equations in school, and recall that the *slope-intercept* form of a linear equation looks like this:
>
> \begin{equation}y = mx + b\end{equation}
>
> In this equation, *y* and *x* are the coordinate variables, *m* is the slope of the line, and *b* is the y-intercept (where the line goes through the Y-axis).
>
> In the case of our scatter plot for our student data, we already have our values for *x* (*StudyHours*) and *y* (*Grade*), so we just need to calculate the intercept and slope of the straight line that lies closest to those points. Then we can form a linear equation that calculates a new *y* value on that line for each of our *x* (*StudyHours*) values - to avoid confusion, we'll call this new *y* value *f(x)* (because it's the output from a linear equation ***f***unction based on *x*). The difference between the original *y* (*Grade*) value and the *f(x)* value is the *error* between our regression line and the actual *Grade* achieved by the student. Our goal is to calculate the slope and intercept for a line with the lowest overall error.
>
> Specifically, we define the overall error by taking the error for each point, squaring it, and adding all the squared errors together. The line of best fit is the line that gives us the lowest value for the sum of the squared errors - hence the name *least squares regression*.

Fortunately, you don't need to code the regression calculation yourself - the **SciPy** package includes a **stats** class that provides a **linregress** method to do the hard work for you. This returns (among other things) the coefficients you need for the slope equation - slope (*m*) and intercept (*b*) based on a given pair of variable samples you want to compare.

```python
from scipy import stats

#
df_regression = df_sample[['Grade', 'StudyHours']].copy()

# Get the regression slope and intercept
m, b, r, p, se = stats.linregress(df_regression['StudyHours'], df_regression['Grade'])
print('slope: {:.4f}\ny-intercept: {:.4f}'.format(m,b))
print('so...\n f(x) = {:.4f}x + {:.4f}'.format(m,b))

# Use the function (mx + b) to calculate f(x) for each x (StudyHours) value
df_regression['fx'] = (m * df_regression['StudyHours']) + b

# Calculate the error between f(x) and the actual y (Grade) value
df_regression['error'] = df_regression['fx'] - df_regression['Grade']

# Create a scatter plot of Grade vs Salary
df_regression.plot.scatter(x='StudyHours', y='Grade')

# Plot the regression line
plt.plot(df_regression['StudyHours'],df_regression['fx'], color='cyan')

# Display the plot
plt.show()
```

Note that this time, the code plotted two distinct things - the scatter plot of the sample study hours and grades is plotted as before, and then a line of best fit based on the least squares regression coefficients is plotted.

The slope and intercept coefficients calculated for the regression line are shown above the plot.

The line is based on the ***f*(x)** values calculated for each **StudyHours** value. Run the following cell to see a table that includes the following values:

- The **StudyHours** for each student.
- The **Grade** achieved by each student.
- The ***f(x)*** value calculated using the regression line coefficients.
- The *error* between the calculated ***f(x)*** value and the actual **Grade** value.

Some of the errors, particularly at the extreme ends, and quite large (up to over 17.5 grade points); but in general, the line is pretty close to the actual grades.

```python
# Show the original x,y values, the f(x) value, and the error
df_regression[['StudyHours', 'Grade', 'fx', 'error']]
```

### Using the regression coefficients for prediction

Now that you have the regression coefficients for the study time and grade relationship, you can use them in a function to estimate the expected grade for a given amount of study.

```python
# Define a function based on our regression coefficients
def f(x):
    m = 6.3134
    b = -17.9164
    return m*x + b

study_time = 14

# Get f(x) for study time
prediction = f(study_time)

# Grade can't be less than 0 or more than 100
expected_grade = max(0,min(100,prediction))

#Print the estimated grade
print ('Studying for {} hours per week may result in a grade of {:.0f}'.format(study_time, expected_grade))
```

So by applying statistics to sample data, you've determined a relationship between study time and grade; and encapsulated that relationship in a general function that can be used to predict a grade for a given amount of study time.

This technique is in fact the basic premise of machine learning. You can take a set of sample data that includes one or more *features* (in this case, the number of hours studied) and a known *label* value (in this case, the grade achieved) and use the sample data to derive a function that calculates predicted label values for any given set of features.

## Summary

Here we've looked at:

1. What an outlier is and how to remove them
2. How data can be skewed
3. How to look at the spread of data
4. Basic ways to compare variables, such as grades and study time

## Further Reading

To learn more about the Python packages you explored in this notebook, see the following documentation:

- [NumPy](https://numpy.org/doc/stable/)
- [Pandas](https://pandas.pydata.org/pandas-docs/stable/)
- [Matplotlib](https://matplotlib.org/contents.html)

## Challenge: Analyze Flight Data

If this notebook has inspired you to try exploring data for yourself, why not take on the challenge of a real world dataset containing flight records from the US Department of Transportation? You'll find the challenge in the [/challenges/01 - Flights Challenge.ipynb](./challenges/01%20-%20Flights%20Challenge.ipynb) notebook!

> **Note**: The time to complete this optional challenge is not included in the estimated time for this exercise - you can spend as little or as much time on it as you like!

## Further Reading

To learn more about the Python packages you explored in this notebook, see the following documentation:

- [NumPy](https://numpy.org/doc/stable/)
- [Pandas](https://pandas.pydata.org/pandas-docs/stable/)
- [Matplotlib](https://matplotlib.org/contents.html)

## Challenge: Analyze Flight Data

If this notebook has inspired you to try exploring data for yourself, why not take on the challenge of a real world dataset containing flight records from the US Department of Transportation? You'll find the challenge in the [/challenges/01 - Flights Challenge.ipynb](./challenges/01%20-%20Flights%20Challenge.ipynb) notebook!

> **Note**: The time to complete this optional challenge is not included in the estimated time for this exercise - you can spend as little or as much time on it as you like!

```python

```

01 - Data exploration 3 - real world data.ipynb


```python
from scipy import signal
from bokeh.plotting import figure, output_file, show
from bokeh.io import output_notebook
from bokeh.layouts import row
import numpy as np
import math
import sympy as sy
output_notebook()
```

    Loading BokehJS ...

```python
def lti_to_sympy(lsys, symplify=True):
    """ Convert Scipy's LTI instance to Sympy expression """
    s = sy.Symbol('s')
    G = sy.Poly(lsys.num, s) / sy.Poly(lsys.den, s)
    return sy.simplify(G) if symplify else G

def sympy_to_lti(xpr, s=sy.Symbol('s')):
    """ Convert Sympy transfer function polynomial to Scipy LTI """
    num, den = sy.simplify(xpr).as_numer_denom()  # expressions
    p_num_den = sy.poly(num, s), sy.poly(den, s)  # polynomials
    c_num_den = [sy.expand(p).all_coeffs() for p in p_num_den]  # coefficients
    l_num, l_den = [sy.lambdify((), c)() for c in c_num_den]  # convert to floats
    return signal.lti(l_num, l_den)

def polyphase_iir_to_ba_coeffs(directCoeffs,delayedCoeffs):
    #For z^-1:
    z_1 = sy.Symbol('z_1')
    directPath = 1
    delayedPath = 1
    for c in directCoeffs:
        directPath = directPath * (sy.Poly([1,0,c], z_1) / sy.Poly([c,0,1], z_1))
    for c in delayedCoeffs:
        delayedPath = delayedPath * (sy.Poly([1,0,c], z_1) / sy.Poly([c,0,1], z_1))

    tranferFunction = 0.5 *( directPath + sy.Poly([1,0], z_1) * delayedPath)
    n,d = sy.fraction(sy.simplify(tranferFunction))
    b = [float(c) for c in sy.Poly(n).all_coeffs()]
    a = [float(c) for c in sy.Poly(d).all_coeffs()]
    return b,a

""" Adapted from:
    Digital signal processing schemes for efficient interpolation and decimation
    R.A. Velenzuela, A.G. Constantinides
    IEEE 1983

    transitionBandwitdth between 0 and 0.5 ( 0.5 = half nyquist)
    stopbandAttenuation in dB
"""
def design_polyphase_halfband_iir(transitionBandwidth, stopbandAttenuation):
    k = np.tan((np.pi-2*np.pi*transitionBandwidth)/4)**2
    kp = np.sqrt(1-k**2)
    e = 0.5 * (1 -np.sqrt(kp))/(1+np.sqrt(kp))
    q = e + 2*e**5 + 15*e**9 + 150*e**13
    ds = 10**(-stopbandAttenuation/20)
    k1 = ds**2 / (1 - ds**2)
    n = int(math.ceil(math.log(k1**2/16)/math.log(q)))
    if n % 2 ==0:
        n += 1
    if n ==1 :
        n = 3
    print("Order: %d" % n)
    q1 = q**n
    k1 = 4 * math.sqrt(q1)
    ds = math.sqrt(k1 / (1+k1))
    dp = 1 - math.sqrt(1-ds**2)

    def ellipticSum1(q,n,i):
        s = 0
        for m in range(5):
            s += (-1)**m *q**(m*(m+1)) * math.sin((2*m+1)*(math.pi*i)/n)
        return s
    def ellipticSum2(q,n,i):
        s = 0
        for m in range(1,5):
            s += (-1)**m *q**(m*m) * math.cos(2*m*(math.pi*i)/n)
        return s
    wi = [ 2*math.pow(q,0.25) * ellipticSum1(q,n,i)/(1+2*ellipticSum2(q,n,i)) for i in range(1,int((n-1)/2)+1) ]
    ai = [ math.sqrt((1-(w**2)*k)*(1-(w**2)/k))/(1+w**2) for w in wi ]
    ai = [ (1-a)/(1+a) for a in ai ]
    #print(ai)
    return ai[0::2], ai[1::2]

```

```python
def plot_filter(b,a,name):
    p1 = figure(plot_width = 500, plot_height=300, title = "%s Gain" % name, y_range = (-80,6))
    p1.xaxis.axis_label = 'Frequency [rad/sample]'
    p1.yaxis.axis_label = 'Amplitude [dB]'
    w, h = signal.freqz(b,a)
    p1.line(w, 20 * np.log10(abs(h)))

    w, gd = signal.group_delay((b, a))
    p2 = figure(plot_width = 500, plot_height=300, title = "%s Group delay" % name)
    p2.xaxis.axis_label = 'Frequency [rad/sample]'
    p2.yaxis.axis_label = 'Group delay [samples]'
    p2.line(w,gd)
    show(row(p1,p2))
```

```python
#b, a = signal.iirdesign(0., 0.3, 5, 50, ftype='cheby1')
b,a  = signal.cheby1(5,1,0.5)
np.set_printoptions(precision=16)
#print(a)
#print(b)
plot_filter(b,a,'Cheby1')
```

```python
b,a  = signal.butter(7,0.5)
plot_filter(b,a,'Butterworth')
```

    C:\Users\jmarin\AppData\Local\Continuum\Anaconda3\lib\site-packages\scipy\signal\filter_design.py:536: UserWarning: The group delay is singular at frequencies [3.129, 3.135], setting to 0
      format(", ".join("{0:.3f}".format(ws) for ws in w[singular]))

```python
b,a  = signal.cheby2(5,40,0.5)
plot_filter(b,a,'Cheby2')
```

```python
b,a = polyphase_iir_to_ba_coeffs([1.0 / (5 + 2 * np.sqrt(5))],[5-2*np.sqrt(5)])
plot_filter(b,a,'ButterPolyphase')
```

```python
int(math.ceil(0.9))
```

    1

```python
directCoeffs,delayedCoeffs = design_polyphase_halfband_iir(0.1, 30)
print(directCoeffs)
print(delayedCoeffs)
b,a = polyphase_iir_to_ba_coeffs(directCoeffs,delayedCoeffs)
plot_filter(b,a,'Cheby-ish polyphase')
```

    Order: 5
    [0.23647102099689202]
    [0.71454214971259888]

```python
[i for i in range(1,int((7-1)/2)+1)]
```

    [1, 2, 3]

```python
[1.0 / (5 + 2 * np.sqrt(5))],[5-2*np.sqrt(5)]
```

    ([0.10557280900008412], [0.52786404500042039])

```python

```


O. Pannekoucke$^{1,2,3}$

$^1$ INPT-ENM, Toulouse, France

$^2$ CNRM, Université de Toulouse, Météo-France, CNRS, Toulouse, France

$^3$ CERFACS, Toulouse, France

(olivier.pannekoucke@meteo.fr)

**Abstract**

This notebook illustrates the use of sympkf to build and handle the PKF dynamics associated with the harmonic oscillator as 2D fields given by
$$\left\{\begin{array}{c}
\partial_t u =v\\
\partial_t v =-u
\end{array}\right.$$
where $u$ and $v$ are functions of $t,x$.
For this dynamics, the resulting PKF system is not closed because of the cros-correlation.

# Definition of the 1D multivariate dynamics

```python
import sympy
sympy.init_printing()
```

**Definition of the dynamics from sympy tools**

```python
from sympy import Function, Derivative, Eq, symbols
from sympkf import SymbolicPKF, t
```

```python
x  = symbols('x')
u = Function('u')(t,x)
v = Function('v')(t,x)
dynamics = [Eq(Derivative(u,t), v), Eq(Derivative(v,t), -u)]
dynamics
```

# Computation of the PKF dynamics by using SymPKF

```python
pkf_dynamics = SymbolicPKF(dynamics)
```

```python
for equation in pkf_dynamics.in_metric:    display(equation)
```

```python
for equation in pkf_dynamics.in_aspect:    display(equation)
```

```python
pkf_dynamics.internal_closure
```

example/PKF on 1D multivariate oscillator.ipynb


## Case Study

New alternator on an old engine.

The tachometer display reflects the ratio of drive wheels on a serpentine belt. There's often a peculiar-looking ratio of engine wheel size to tach wheel size, for example, 2.3:1 or 4.65:1. When the engine is rotating at 1000 RPM, the alternator may be spinning at 2300 RPM.

The new alternator has a smaller drive wheel. This means the tachometer display is now wrong.

Preliminary measurements made it look like the new wheel is $\tfrac{5}{7}$ the original wheel size. In principle it means scaling tachomter readings by $\tfrac{5}{7} = 0.7142$.

Later, with an optical device in the engine room, detailed samples were gathered. Recorded in a spreadsheet, and saved as a CSV file, `engine.csv`.

```python
from csv import DictReader
from pathlib import Path
```

```python
source_path = Path("engine.csv")
with source_path.open() as source_file:
    reader = DictReader(source_file)
    text_tach_readings = list(reader)
text_tach_readings
```

    [{'Sample': '1', 'Tach': '1000', 'Engine': '883'},
     {'Sample': '2', 'Tach': '1500', 'Engine': '1242'},
     {'Sample': '3', 'Tach': '1500', 'Engine': '1217'},
     {'Sample': '4', 'Tach': '1600', 'Engine': '1306'},
     {'Sample': '5', 'Tach': '1750', 'Engine': '1534'},
     {'Sample': '6', 'Tach': '2000', 'Engine': '1805'},
     {'Sample': '7', 'Tach': '2000', 'Engine': '1720'}]

So far, so good. We have the data.

It's all text, however.

Since we need to convert it, we should convert it to something we can work with.

A named tuple.  This is an extension to the tuple class, but it has names for each item. We'll start with the `typing.NamedTuple` definition as a base class, and extend it.

```python
from typing import NamedTuple

class Sample(NamedTuple):
    sample: int
    tach: float
    engine: float
```

We can build these objects from the input CSV file.

```python
def make_sample(text_row: dict[str, str]) -> Sample:
    return Sample(
        sample=int(text_row['Sample']),
        tach=float(text_row['Tach']),
        engine=float(text_row['Engine'])
    )
```

```python
source_path = Path("engine.csv")
with source_path.open() as source_file:
    reader = DictReader(source_file)
    sample_iter = (
        make_sample(text_row) for text_row in reader
    )
    tach_readings = list(sample_iter)
```

```python
tach_readings
```

    [Sample(sample=1, tach=1000.0, engine=883.0),
     Sample(sample=2, tach=1500.0, engine=1242.0),
     Sample(sample=3, tach=1500.0, engine=1217.0),
     Sample(sample=4, tach=1600.0, engine=1306.0),
     Sample(sample=5, tach=1750.0, engine=1534.0),
     Sample(sample=6, tach=2000.0, engine=1805.0),
     Sample(sample=7, tach=2000.0, engine=1720.0)]

Let's work through this carefully.

We create a `DictReader`. This is an iterable of `dict[str, str]` -- each dictionary is a row from the CSV file. It's assinged to the `reader` variable.

`sample_iter` is a generator expression. It consumes the rows in the `reader` object.  It performs a computation on each row, creating a `Sample()` object from the input row.

This expression's input is `Iterable[dict[str, str]]`. Output is `Iterator[float]`. We could make this into a generator function, if the conversion processing become more complex.

The final result `tach_readings` is a list, built from the generated sequence of `Sample` objects.

## The Questions:

- How well does our initial model correlate with the actual engine?

- How well does tach reading correlate with the actual engine?

Clearly, we need to compute correlation coefficients.

## Correlation Coefficient

Here's the formula for computing the coefficient of correlation between two
sequences of values, $x$ and $y$.

\\[ r_{xy} = \frac{\sum x y - n \bar x \bar y}{(n-1)\sigma_x\sigma_y} \\]

This means this requires the following intermediate results:

- $\sum x y$
- $\bar x$, the mean of x
- $\bar y$, the mean of y
- $\sigma_x$, the standard deviation of x
- $\sigma_y$, the standard deviation of y
- $n$, the number of samples

Here are two more formulae for $\bar x$, $\bar y$, $\sigma_x$, and $\sigma_y$:

\\[
\begin{align}
\bar x& = \frac{\sum x}{n}
\\
\sigma_x& = \sqrt{\frac{\frac{\sum x^2}{n-1}-\left( \frac{\sum x}{n} \right) ^ 2}{n-1}}
\end{align}
\\]

These require

- $\sum x$
- $\sum x^2$

In and among the various symbols like $\bar x$ and $\sigma_y$, it appears
that this computation requires a bunch of intermediate sums.

We'll need some functions for simple sum. sum of squares, sum of products.

As a test case, we'll like some data with known values for means and standard deviations.
That way we can check the ourselves. There are four small data sets called *Anscombe's Quartet* that's handy for this kind of testing.

- $\bar x = 9$
- $\bar y \approx 7.50$
- $\sigma_x \approx \sqrt {11} = 3.316$
- $\sigma_y \approx \sqrt {4.125} = 2.031$
- $r_{xy} \approx 0.816$
- Regression: $y = 3 + 0.5 x$

## Some Statistical Tools

See https://www.kaggle.com/carlmcbrideellis/data-anscombes-quartet/version/1 for test data.

This CSV file has six columns:
- x123
- y1
- y2
- y3
- x4
- y4

To create any of the four series, use one of these combinations of columns:

1. (x123, y1)
2. (x123, y2)
3. (x123, y3)
4. (x4, y4)

We want to create a similar structure to our engine samples, so we'll have similar-looking code.

We also need to define this series to column-pair mapping.

It helps to parallel the `make_sample()` function with a `make_anscome()` function that uses the series number to decide which two columns to use.

```python
class Anscombe(NamedTuple):
    x: float
    y: float

Series_Column_Pair = {
    1: ("x123", "y1"),
    2: ("x123", "y2"),
    3: ("x123", "y3"),
    4: ("x4", "y4"),
}

def make_anscombe(
        text_row: dict[str, str],
        series: int = 1
) -> Anscombe:
    x_col, y_col = Series_Column_Pair[series]
    return Anscombe(
        float(text_row[x_col]),
        float(text_row[y_col])
    )

series_source = Path("Anscombe_quartet_data.csv")
with series_source.open() as source:
    reader = DictReader(source)
    anscome_iter = (
        make_anscombe(text_row)
        for text_row in reader
    )
    test_series = list(anscome_iter)
test_series
```

    [Anscombe(x=10.0, y=8.04),
     Anscombe(x=8.0, y=6.95),
     Anscombe(x=13.0, y=7.58),
     Anscombe(x=9.0, y=8.81),
     Anscombe(x=11.0, y=8.33),
     Anscombe(x=14.0, y=9.96),
     Anscombe(x=6.0, y=7.24),
     Anscombe(x=4.0, y=4.26),
     Anscombe(x=12.0, y=10.84),
     Anscombe(x=7.0, y=4.82),
     Anscombe(x=5.0, y=5.68)]

```python
from typing import Iterable
```

```python
def sum_2(data: Iterable[float]) -> float:
    return sum(v**2 for v in data)
```

```python
def sum_prod(s_1: Iterable[float], s_2: Iterable[float]) -> float:
    pairs = zip(s_1, s_2)
    return sum(
        x*y for x, y in pairs
    )
```

```python
def mean(data: list[float]) -> float:
    return sum(data) / len(data)
```

```python
from math import sqrt

def stdev(data: list[float]) -> float:
    n = len(data)
    mean_2 = (sum(data) / n) ** 2
    squares = sum_2(data) / (n-1)
    return sqrt(squares - mean_2 * n / (n - 1))
```

```python
mean([s.x for s in test_series])
```

    9.0

```python
mean([s.y for s in test_series])
```

    7.500909090909093

```python
stdev([s.x for s in test_series])
```

    3.3166247903554

```python
stdev([s.y for s in test_series])
```

    2.031568135925802

```python
from math import isclose
assert isclose(mean([s.x for s in test_series]), 9.0)
assert isclose(mean([s.y for s in test_series]), 7.5, abs_tol=1E-3)
assert isclose(stdev([s.x for s in test_series]), 3.316, abs_tol=1E-3)
assert isclose(stdev([s.y for s in test_series]), 2.031, abs_tol=1E-3)
```

```python
def corr(s_1: list[float], s_2: list[float]) -> float:
    n = len(s_1)
    num = sum_prod(s_1, s_2) - n * mean(s_1) * mean(s_2)
    den = (n-1) * stdev(s_1) * stdev(s_2)
    return num / den
```

```python
corr([s.x for s in test_series], [s.y for s in test_series])
```

    0.8164205163448433

```python
assert isclose(corr([s.x for s in test_series], [s.y for s in test_series]), 0.816, abs_tol=1E-3)
```

## Now That We Have Stats

First, tach vs. engine correlation? Are these consistent?

```python
corr([s.tach for s in tach_readings], [s.engine for s in tach_readings])
```

    0.9847275588081925

That's wonderful to see. It means the optical device and the tachometer are likely to agree.

The question now, is how far wrong is this model? What's the range of gaps between model engine RPM's and actual engine RPM's.

```python
def engine_1(s: Sample):
    return s.tach * 0.7142
```

```python
diffs_1 = [
    engine_1(s) - s.engine for s in tach_readings
]
```

```python
min(diffs_1), max(diffs_1), mean(diffs_1)
```

    (-376.60000000000014, -145.70000000000005, -228.69000000000008)

The predictions of the `engine_1()` model are wrong by anywhere from 146 to 376 RPM's. On average, it's wrong by 228 RPMS.  And, since all the numbers are negative, the model is consistently too low.

This is, perhaps, less than ideal.

## Visual Aids

```python
%matplotlib inline
import matplotlib.pyplot as plt
```

```python
x = [s.tach for s in tach_readings]
y = [s.engine for s in tach_readings]
est_1 = [engine_1(s) for s in tach_readings]
```

```python
plt.plot(x, y, 'o', color='blue')
plt.plot(x, est_1, '+', color='black')
```

The estimations are consistently below the actual values. And they seem to have a wrong slope.

## Linear Regression

Given the correlation value, $r_{xy}$, we can compute the linear regression $y = a + bx$ line as follows:

\\[
\begin{align}
b& = \frac{\sum {x}{y} - n  \bar x \bar y}{(n-1){\sigma_x} ^ 2} = r_{xy} \frac {\sigma_y}{\sigma_x}
\\
a& = \bar y - b \bar x
\end{align}
\\]

This doesn't seem to require any new computations.

```python
def regression(s_1: list[float], s_2: list[float]) -> tuple[float, float]:
    b = corr(s_1, s_2) * stdev(s_2)/stdev(s_1)
    a = mean(s_2) - b * mean(s_1)
    return a, b
```

Test Case using the `test_series`.

```python
regression([s.x for s in test_series], [s.y for s in test_series])
```

    (3.000090909090921, 0.500090909090908)

The above means $y = a + bx = 3 + 0.5x$. This was the expected answer from Anscombe's Quartet.

```python
a, b = regression([s.x for s in test_series], [s.y for s in test_series])
for s in test_series:
    print(f"{s.x:5.2f} {s.y:5.2f} model={a+b*s.x:5.2f}")
```

    10.00  8.04 model= 8.00
     8.00  6.95 model= 7.00
    13.00  7.58 model= 9.50
     9.00  8.81 model= 7.50
    11.00  8.33 model= 8.50
    14.00  9.96 model=10.00
     6.00  7.24 model= 6.00
     4.00  4.26 model= 5.00
    12.00 10.84 model= 9.00
     7.00  4.82 model= 6.50
     5.00  5.68 model= 5.50

## Regression applied to engine data

This is the crux of the problem.

```python
a, b = regression(
    [s.tach for s in tach_readings],
    [s.engine for s in tach_readings]
)
```

```python
f"engine = {a:.1f} + {b:.2f} × tach"
```

    'engine = -92.9 + 0.91 × tach'

```python
def engine_2(s: Sample):
    return -92.9 + 0.91 * s.tach
```

```python
diffs_2 = [
    engine_2(s) - s.engine for s in tach_readings
]
```

```python
min(diffs_2), max(diffs_2), mean(diffs_2)
```

    (-77.90000000000009, 57.09999999999991, -4.114285714285789)

This is much nicer. The differences are less than 80 RPM, and they vary in sign between negative and positive. The averge is less than 5 RPM.

```python
est_2 = [engine_2(s) for s in tach_readings]
```

```python
data, = plt.plot(x, y, 'o', color='blue',
                 label='source data')
model, = plt.plot(x, est_2, '+', color='black',
                  label='engine_2: $e=-92.9 + 0.91t$')
plt.legend(handles=[data, model])
plt.ylabel('engine RPM')
plt.xlabel('tachometer display')
plt.title('Regression Model')
```

This shows how well the new model fits the available data points.

## Summary

We followed a typical process for working with data in a Python. We opened a Jupyter Lab notebook, and loaded the data.

We then defined some functions in the notebook. In the long run, it's slightly better to save the functions in a module file and import the module into the notebook. When getting started with new data or a new problem domain, defining functions in the notebook keeps everything in one place.

The process of defining the functions was done by first pasting the relevant background information into markdown cell. Then defining a function that matched the definition, then testing the function to be sure it worked.

Once the preliminaries of data and processing were completed, we could move on to applying the processing to the available data.

First, we saw a good correlation between the measurements taken with the optical device and the display shown on the tachometer. This told us we were using the optical device correctly, and had not introduced some  measurement error.

Second, we saw that our initial model was consistently wrong about predicting actual engine RPM's from the tachometer display.

Third, we used linear regression to create a better model. This made consistently correct predictions of RPM given tachometer display.

The results are all in a single notebook for further review and follow-on work.

```python

```


```python
import sys
import math
import sympy as sy
from uncertainties import ufloat
from scipy import stats
from iminuit import Minuit                             # The actual fitting tool, better than scipy's
from scipy.stats import binom, poisson, norm
import numpy as np
import matplotlib.pyplot as plt
sys.path.append(r'/home/saim/External_Functions')
from ExternalFunctions import Chi2Regression, BinnedLH, UnbinnedLH
from ExternalFunctions import nice_string_output, add_text_to_ax    # Useful functions to print fit results on figure
```

```python
# Plotting stuff
plt.rcParams['font.size'] = 18
plt.style.use(['science', 'notebook', 'grid'])

pink = '#e377c2'
blue = '#1f77b4'
golden = '#ff7f0e'
green = '#2ca02c'
red = '#d62728'
purple = '#9467bd'
light_blue = '#17becf'
```

```python
r = np.random
r.seed(42)

N_points = 10000     # Number of random points produced
x_all = r.normal(loc = 0.2,
                 scale = 1.1,
                 size = N_points)

Nbins = 100
binwidth_gauss = np.ptp(x_all) / Nbins
x_min, x_max = np.min(x_all), np.max(x_all)
```

```python
# Fitting function
#def func_gaussian_llh(x, N, mu, sigma) :
    #return N * norm.pdf(x, mu, sigma)

# Fitting function which is NOT normalised but has normalisation constants "N" in,
# and includes the bin width:

def func_gauss_norm(x, N, mu, sigma) :
    norm = binwidth_gauss * N / np.sqrt(2.0 * np.pi) / sigma
    z = (x - mu) / sigma

    return norm * np.exp(-0.5 * (z**2))

# Fitting function which is NOT normalised but has normalisation constants "N" in,
# but does NOT includes the bin width:

def func_gauss_llh(x, N, mu, sigma) :
    norm = N / np.sqrt(2.0 * np.pi) / sigma
    z = (x - mu) / sigma

    return norm * np.exp(-0.5 * (z**2))
```

```python
def MLE_hist(data, Nbins, mini, maxi):

    counts, bin_edges = np.histogram(data,
                                     bins = Nbins,
                                     range = (mini, maxi),
                                     density = False)

    x = (bin_edges[1:] + bin_edges[:-1]) / 2
    y = counts
    sy = np.sqrt(y)

    return x, y, sy

x1, y1, sy1 = MLE_hist(x_all, Nbins, x_min, x_max)

print(np.mean(x_all), np.std(x_all))
```

    0.19765041829473118 1.1037534267429865

```python
# extended=True because we have our own normalization in our fit function
bllh_object_fit = BinnedLH(func_gauss_llh,
                           x_all,
                           bins = Nbins,
                           extended = True)

minuit_fit_bllh = Minuit(bllh_object_fit,
                         N = N_points,
                         mu = 0.204,
                         sigma = 1)

minuit_fit_bllh.errordef = Minuit.LIKELIHOOD
minuit_fit_bllh.migrad()
```

    /home/saim/External_Functions/ExternalFunctions.py:277: UserWarning: x is really small return 0
      warnings.warn('x is really small return 0')

         FCN = 49.23
         Nfcn = 44 (44 total)

         EDM = 2.52e-05 (Goal: 0.0001)

         Valid Minimum
         Valid Parameters
         No Parameters at limit

         Below EDM threshold (goal x 10)
         Below call limit

         Hesse ok
         Has Covariance
         Accurate
         Pos. def.
         Not forced

         Name
         Value
         Hesse Error
         Minos Error-
         Minos Error+
         Limit-
         Limit+
         Fixed

         0
         N
         10.0e3
         0.1e3

         1
         mu
         0.198
         0.011

         2
         sigma
         1.105
         0.008

```python
# Produce the points for drawing the fit:
xaxis_bllh = np.linspace(x_min, x_max, Nbins)
yaxis_bllh = func_gauss_norm(xaxis_bllh, *minuit_fit_bllh.values[:])

# Init plot object
fig, ax = plt.subplots(figsize=(15, 9))

ax.errorbar(x1, y1, sy1, fmt = '.', color = blue, label = 'Data')
ax.plot(xaxis_bllh, yaxis_bllh, '-', color = golden, label = 'Fit')

ax.set(xlabel = "Value",
       ylabel = "Frequency",
       title = "")

ax.legend(loc = 'upper left', fontsize=19);
fig.tight_layout()
```

```python
# TODO
# Try all with various functional forms
```

```python
ullhfit = UnbinnedLH(func_gauss_llh,
                     x_all,
                     bound = (x_min, x_max),
                     extended = True)

minuit_fit_ullh = Minuit(ullhfit,
                     N = N_points,
                     mu = 0.204,
                     sigma = 1)

minuit_fit_ullh.errordef = Minuit.LIKELIHOOD
minuit_fit_ullh.migrad()           # Perform the actual fit
```

         FCN = -6.693e+04
         Nfcn = 46 (46 total)

         EDM = 2.75e-05 (Goal: 0.0001)

         Valid Minimum
         Valid Parameters
         No Parameters at limit

         Below EDM threshold (goal x 10)
         Below call limit

         Hesse ok
         Has Covariance
         Accurate
         Pos. def.
         Not forced

         Name
         Value
         Hesse Error
         Minos Error-
         Minos Error+
         Limit-
         Limit+
         Fixed

         0
         N
         10.0e3
         0.1e3

         1
         mu
         0.198
         0.011

         2
         sigma
         1.105
         0.008

```python
# Produce the points for drawing the fit:
xaxis_ullh = np.linspace(x_min, x_max, N_points)
yaxis_ullh = func_gauss_norm(xaxis_ullh, *minuit_fit_ullh.values[:])

# Init plot object
fig, ax = plt.subplots(figsize=(15, 9))

ax.errorbar(x1, y1, sy1, fmt = '.', color = blue, label = 'Data')

ax.plot(xaxis_ullh,
        yaxis_ullh,
        '-g',
        label = 'Unbinned LLH fit model result',
        color = green)

ax.set(xlabel = "Value",
       ylabel = "Frequency",
       title = "")

ax.legend(loc = 'upper left', fontsize=19);
fig.tight_layout()
```

```python
print(*minuit_fit_ullh.values[:])
```

    10001.115740347039 0.19771011351981246 1.1045541341503253

```python

```

```python
# Fit exp, power, T-dist, double gauss
```

## Exponential Fit

```python
N_exp = 10000     # Number of random points produced
x_exp = r.exponential(np.e, N_exp)

exp_bins = 100
binwidth_exp = np.ptp(x_exp) / exp_bins
exp_min, exp_max = np.min(x_exp), np.max(x_exp)
```

```python
def exp_pdf(x, N, tau):
    return N * binwidth_exp / tau * np.exp(-x/tau)

def exp_pdf_llh(x, N, tau):
    return N / tau * np.exp(-x/tau)
```

```python
# extended=True because we have our own normalization in our fit function
bllh_exp_object = BinnedLH(exp_pdf_llh,
                           x_exp,
                           bins = exp_bins,
                           extended = True)

exp_fitted_bllh = Minuit(bllh_exp_object,
                         N = N_exp,
                         tau = 2.5)

exp_fitted_bllh.errordef = Minuit.LIKELIHOOD
exp_fitted_bllh.migrad()
```

    /home/saim/External_Functions/ExternalFunctions.py:277: UserWarning: x is really small return 0
      warnings.warn('x is really small return 0')

         FCN = 47.69
         Nfcn = 32 (32 total)

         EDM = 5.05e-07 (Goal: 0.0001)

         Valid Minimum
         Valid Parameters
         No Parameters at limit

         Below EDM threshold (goal x 10)
         Below call limit

         Hesse ok
         Has Covariance
         Accurate
         Pos. def.
         Not forced

         Name
         Value
         Hesse Error
         Minos Error-
         Minos Error+
         Limit-
         Limit+
         Fixed

         0
         N
         10.0e3
         0.1e3

         1
         tau
         2.792
         0.028

```python
# Produce the points for drawing the fit:
exp_x_ax = np.linspace(exp_min, exp_max)
exp_y_ax = exp_pdf(exp_x_ax, *exp_fitted_bllh.values[:])
```

```python
# Produce figure with histogram (with error bars) and fit overlayed:
fig, ax = plt.subplots(figsize=(14, 6))

# Binning data and plotting it
x2, y2, sy2 = MLE_hist(x_exp, exp_bins, exp_min, exp_max)
ax.errorbar(x2, y2, sy2, fmt = '.', color = blue, label = 'Data')

ax.plot(exp_x_ax, exp_y_ax, '-', color = golden, label = 'Fit')

ax.set(xlabel = "Value",
       ylabel = "Frequency",
       title = "")

ax.legend(loc = 'lower right',
          fontsize=14);

fig.tight_layout()
```

```python
# REMEMBER THAT LLH DOES NOT CARE ABOUT ZERO BINS
```


# A <del>Simple</del> Least Squares Regression

```python
import matplotlib.pyplot as plt
import numpy as np

from sklearn.datasets import make_regression
from sklearn.linear_model import LinearRegression
from statsmodels.formula.api import ols
from ipywidgets import interactive

plt.xkcd()
RANDOM_SEED = 1
```

## Intro

As you probably know the main task of Machine Learning algorithms is to make a predictions based on a past data. The basic scenario sounds like this:

> Imagine you have two variables: X (let it be years of experience of an ML engineer) and Y (salary of ML specialist). You want to build a model to predict the salary of an engineer based on his years of experience as the new data comes in.

This is what called a **regression** task (we are predicting **continuous** variable). Most of the times the good place to start is Linear Regression, since it's considered to be a simple and intuitive model. But how simple is that? I. Despite its simplicity, linear regression can be the worst model for prediction when it was modeled in an incorrect way. In this tutorial we are going to look at the math behind **Ordinary least squares** method, its assumptions and properties and when things can go wrong.

```python
# gernerate two random variables of size 50
n = 50
years, salary = make_regression(n_samples=n,
                                n_features=1,
                                noise=50,
                                random_state=RANDOM_SEED)
years = years.reshape(n,) + 2.5
salary = salary + 1000
```

We have simulated two sample with years of experience and salary of size 50. Looking at the data, what would be the salary for someone with 1 year of experience? Looking at this graph or from your prior knowledge you might assume that there is a **linear** relationship between years of experience and the salary. And according to the data the most relevant guess would be in a $[800,950]$ range.

```python
plt.figure(figsize=(10,4))
plt.scatter(years, salary, label='Observations')
plt.scatter([1], [850], color='red', label='New Data')
plt.text(x=0.7, y=865, s="(1, ???)")
plt.vlines(x=1, ymin=0, ymax=850, linestyle='--',
           linewidth=1, color='red')
plt.hlines(y=850, xmin=-1, xmax=1, linestyle='--',
           linewidth=1, color='red')

plt.xlim([0,5])
plt.ylim([750, 1250])
plt.xlabel('Years of Experience')
plt.ylabel('Salary (% per month)')
plt.title('Relationship of X and Y')
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['right'].set_visible(False)
plt.legend()
plt.show()
```

You also might remember that the line equation can we written as $y = kx + b$. In context of our problem:

$$y = \beta_0 + \beta_1 x \tag{1}$$

* $y$ - salary
* $x$ - years of experience
* $\beta_0$ - intercept (value of interception with $y$ axis)
* $\beta_1$ - slope (the rate of change)

You can check how different values of intercept and slope affect the line in this interactive demo:

```python
def simple_line_plot(intercept, slope):
    y_space = intercept + slope*x_space
    plt.figure(figsize=(8,4))
    plt.plot(x_space, y_space)
    plt.xlabel('X')
    plt.ylabel('Y')
    plt.xlim([-10,10])
    plt.ylim([-10,10])
    plt.text(x=7, y=8,
             s=f"y={intercept} + {slope: .1f}x",
             color='red')
    plt.gca().spines['top'].set_visible(False)
    plt.gca().spines['right'].set_visible(False)
    plt.plot()

x_space = np.linspace(start=-10, stop=10, num=1000)
interactive_plot = interactive(simple_line_plot,
                               intercept=(-5, 5),
                               slope=(-3, 3, 0.5))
interactive_plot
```

    interactive(children=(IntSlider(value=0, description='intercept', max=5, min=-5), FloatSlider(value=0.0, descr…

## General Form

But as we can see, even the relationship seems to be positive (as salary increases as experience goes up), the relationship is not exactly linear and there is some noise (error $\epsilon$). Let's consider a more general case when we have $k$ independent variables of $n$ observations in a matrix form. In such case:

* $y$ - is a vector of dependent (predicted) variable of size $(n \times 1)$
* $x$ - is a matrix of independent variable of size $(n \times k)$
* $\beta$ - is a vector of parameters of size $(k \times 1)$
* $\epsilon$ - is a vector of errors of size $(n \times 1)$

\begin{equation}
\begin{bmatrix}
y_1 \\
y_2 \\
... \\
y_n
\end{bmatrix} =
\begin{bmatrix}
1 & x_{11} & ... & x_{k1} \\
1 & x_{12} & ... & x_{k2} \\
... & ... & ... & ... \\
1 & x_{1n} & ... & x_{kn}
\end{bmatrix}
\begin{bmatrix}
\beta_1 \\
\beta_2 \\
... \\
\beta_k
\end{bmatrix} +
\begin{bmatrix}
\epsilon_1 \\
\epsilon_2 \\
... \\
\epsilon_n
\end{bmatrix}
\tag{2}
\end{equation}

Or more simply:

$$y = X \beta + \epsilon \tag{3}$$

*Note, that the intercept is added into $\beta$ vector that's why we have a row of ones in $X$ matrix.*

The goal is to find the estimated values ($b$) of population parameters ($\beta$).

## Ordinary Least Squares

We can achieve that with the help of **Ordinary Least Squares** method. This method fits a line through the data points that minimizes the sum of the squared differences between observed values and corresponding fitted values. So we need the minimum value of mean squared error:

$$\hat{y} = X b \tag{4}$$

$$\epsilon = y - \hat{y} = y - Xb \tag{5}$$

* $\hat{y}$ - predicted values

The sum of squared residuals (**RSS**):

\begin{equation}
\begin{bmatrix}
\epsilon_1 & \epsilon_2 & ... & \epsilon_n
\end{bmatrix}
\begin{bmatrix}
\epsilon_1 \\
\epsilon_2 \\
... \\
\epsilon_n
\end{bmatrix}
= [\epsilon_1 \epsilon_1 + \epsilon_2 \epsilon_2 + ... + \epsilon_n \epsilon_n]
\tag{6}
\end{equation}

Since RSS is just a scalar we know that:

$$\epsilon^T = \epsilon = y - Xb = (y - Xb)^T \tag{7}$$

Then we can use this equation to find the RSS formula:

$$\epsilon^T \epsilon = (y - Xb)^T (y - Xb) $$
$$ = (y^T - X^T b^T)(y - Xb)$$
$$ = y^T y - y^T Xb - X^T b^T y + X^T b^T X b $$
$$ = y^T y - 2X^T b^T y + X^T b^T X b \tag{8}$$

To find the $b$ that minimizes the sum of squared residuals, we need to take the derivative of $\epsilon^T \epsilon$ with respect to $b$.

$$\frac{\partial \epsilon^T \epsilon}{\partial b} = -2 X^T y + 2 X^T X b = 0 \tag{9}$$

And this gives us:

$$ X^T X b = X^T y \tag{10}$$

To find $b$ we can multiply each side of the equation by $(X^T X)^{-1}$:

$$ (X^T X)^{-1} (X^T X) b = (X^T X)^{-1} X^T y \tag{11}$$

Since by definition $(X^T X)^{-1} (X^T X) = I$, where $I$ is identity matrix with ones on a main diagonal and zeros elsewhere.

$$ I b = (X^T X)^{-1} X^T y$$

$$ b = (X^T X)^{-1} X^T y \tag{12}$$

Let's check a simple case:

\begin{equation}
X = \begin{bmatrix}
1 & 1 \\
1 & 2
\end{bmatrix},
y = \begin{bmatrix}
3 \\
5
\end{bmatrix}
\end{equation}

As we can see the regression line should be defined as $y = 1 + 2x$

Calculations:

1.
\begin{equation}
X^T X = \begin{bmatrix}
1 & 1 \\
1 & 2
\end{bmatrix}
\begin{bmatrix}
1 & 1 \\
1 & 2
\end{bmatrix} =
\begin{bmatrix}
2 & 3 \\
3 & 5
\end{bmatrix}
\end{equation}

2.
\begin{equation}
(X^T X)^{-1} = \frac{1}{2 \times 5 - 3 \times 3} \begin{bmatrix}
5 & 3 \\
-3 & 2
\end{bmatrix} =
\begin{bmatrix}
5 & 3 \\
-3 & 2
\end{bmatrix}
\end{equation}

3.
\begin{equation}
X^T y = \begin{bmatrix}
1 & 1 \\
1 & 2
\end{bmatrix}
\begin{bmatrix}
3 \\
5
\end{bmatrix} =
\begin{bmatrix}
8 \\
13
\end{bmatrix}
\end{equation}

4.
\begin{equation}
(X^T X)^{-1} X^T y =
\begin{bmatrix}
5 & -3 \\
-3 & 2
\end{bmatrix}
\begin{bmatrix}
8 \\
13
\end{bmatrix} =
\begin{bmatrix}
1 \\
2
\end{bmatrix}
\end{equation}

Exactly what we have expected!

```python
# numpy calculations
x = np.array([[1, 1], [1, 2]])
y = np.array([[3], [5]])
b = np.linalg.inv(x.T@x) @ (x.T@y)
print(b)
```

    [[1.]
     [2.]]

## Properties of OLS

If we rewrite the equation $(10)$ using $y = Xb + \epsilon$ we get:

$$ X^T X b = X^T (Xb + \epsilon) = X^T Xb + X^T \epsilon$$

$$ X^T \epsilon = 0 \tag{13}$$

Now let's take a look at the first element in $X^T \epsilon$ vector. We can see that it equals to:

$$1 \times \epsilon_1 + 1 \times \epsilon_2 + ... 1 \times \epsilon_n = 0 \tag{14}$$

Which means that **the sum of residuals $\sum_{i=1}^n \epsilon_i$ equals to 0**. The same applies for the mean $\bar{\epsilon} = 0$.

If we divide regression equation $(3)$ by number of observations $n$ we get:

$$\frac{y}{n} = \frac{X}{n}b + \frac{\epsilon}{n}$$

$$\bar{y} = \bar{x}b \tag{15}$$

Which means that **the least squares line always goes through $(\bar{x}, \bar{y})$ point**.

## Multicollinearity

Looks pretty simple so far, right? But let's take a look when things can go wrong.

Recall the equation $(12)$. There might be cases when the inverse matrix $(X^T X)^{-1}$ doesn't exist.

* if rank\* of a matrix $X^T X$ is less then $n$, meaning that one or more of the independent variables are a linear combination of the other variables (**multicollinearity**).
* if $n<k$ (number of variables is larger than the number of observations).
* the determinant of $X^T X$ is zero.

\* The rank of a matrix is defined as the maximum number of linearly independent column vectors in the matrix or the maximum number of linearly independent row vectors in the matrix.

```python
# example 1
x = np.array(
    [[1, 2, 4],
     [1, 3, 6],
     [1, 2, 5]])

print(f"Matrix rank: {np.linalg.matrix_rank(x.T@x)}")
try:
    print(f"Inverse matrix:\n{np.linalg.inv(x.T@x)}")
except np.linalg.LinAlgError as e:
    print(f"Inverse matrix doesn't exist. {e}")
```

    Matrix rank: 3
    Inverse matrix:
    [[13.  1. -3.]
     [ 1.  6. -3.]
     [-3. -3.  2.]]

```python
# example 2
x = np.array(
    [[1, 2, 4],
     [1, 3, 6],
     [1, 2.5, 5]])

print(f"Matrix rank: {np.linalg.matrix_rank(x.T@x)}")
try:
    np.linalg.inv(x.T@x)
except np.linalg.LinAlgError as e:
    print(f"Inverse matrix doesn't exist. {e} was given")
```

    Matrix rank: 2
    Inverse matrix doesn't exist. Singular matrix was given

```python
# example 3
x = np.array(
    [[1, 2, 4],
     [1, 3, 6]])

print(f"Matrix rank: {np.linalg.matrix_rank(x.T@x)}")
try:
    print(f"Inverse matrix:\n{np.linalg.inv(x.T@x)}")
except np.linalg.LinAlgError as e:
    print(f"Inverse matrix doesn't exist. {e}")
```

    Matrix rank: 2
    Inverse matrix doesn't exist. Singular matrix

```python
# example 4
x = np.array(
    [[2, 4],
     [3, 6],
     [2, 5]])

print(f"Matrix rank: {np.linalg.matrix_rank(x.T@x)}")
try:
    print(f"Inverse matrix:\n{np.linalg.inv(x.T@x)}")
except np.linalg.LinAlgError as e:
    print(f"Inverse matrix doesn't exist. {e}")
```

    Matrix rank: 2
    Inverse matrix:
    [[ 5.92307692 -2.76923077]
     [-2.76923077  1.30769231]]

In the case of multicollinearity there is not going to be a single solution for $(X^T X)^{-1}$, but rather an infinite number of them.

## Making a Prediction

Now we can apply the OLS method on our experience-salary example and find out the expected salary for the engineer with 1 year of experience.

```python
# add column with ones
X = np.vstack([np.ones(shape=n), years]).T
b = np.linalg.inv(X.T@X) @ (X.T@salary)
y_hat = X@b
print(f"Model: y ={b[0]: .2f} +{b[1]: .2f}x")
```

    Model: y = 812.11 + 79.52x

```python
# for a smooth line
x_space = np.linspace(years.min(), years.max(), 1000)
y_space = b[0] + b[1]*x_space

# value to predict
x_new = 1
y_new = b[0] + b[1]*x_new
```

```python
plt.figure(figsize=(10,4))
plt.scatter(years, salary, label='Observations')
plt.plot(x_space, y_space, '--', color='black',
         linewidth=1, label='Fitted Line')
plt.scatter([x_new], [y_new],
            color='red', label='New Data')
plt.text(x=0.7, y=920, s=f"(1, {y_new: .2f})")

plt.xlim([0,5])
plt.ylim([750, 1250])
plt.xlabel('Years of Experience')
plt.ylabel('Salary (% per month)')
plt.title('Relationship of X and Y')
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['right'].set_visible(False)
plt.legend()
plt.show()
```

Based on a fitted model, engineer with 1 year of experience is expected to earn 891.64$ per month.

### `sklearn` realization

And obviously you don't have to do this calculations by hand since there is already a `sklearn` realization:

```python
# sklearn realization
ols_model = LinearRegression(fit_intercept=True)
ols_model.fit(years.reshape(n,1), salary)
print(f"Model: y ={ols_model.intercept_: .2f} +{ols_model.coef_[0]: .2f}x")
```

    Model: y = 812.11 + 79.52x

```python
# prediction
print(ols_model.predict(np.array(x_new).reshape(1,1)))
```

    [891.63525261]

## Assumptions for Linear Regression

1. A **linear** relationship between $y$ and $X$.
2. The dependent variables $X$ should be **linearly independent** (no perfect multicollinearity).
3. Approximate **normality of residuals distribution** centered at 0, $\epsilon \sim N(0, \sigma)$, $E[\epsilon] = 0$.
4. **Constant variability** (heteroscedasticity).

Some possible ways for assumptions check:

1. Linear relationship between 2 variables can be easily inspected on a scatter plot.
2. Highly correlated variables can be found in the correlation matrix $R_{xx}$.
3. Simplest way is to plot a histogram of residuals after model was fitted.
4. Scatter plot the residuals against the predicted values. There should be no "fan" shape or some other funny shapes.

```python
errors = salary - y_hat

plt.figure(figsize=(14,4))
plt.subplot(121)
plt.hist(errors, bins=30)
plt.axvline(0, linestyle='--', color='black')
plt.xlabel('Residual value')
plt.ylabel('Count')
plt.title('Residuals Distribution')

plt.subplot(122)
plt.scatter(y_hat, errors)
plt.axhline(0, linestyle='--', color='black')
plt.xlabel('Fitted Value')
plt.ylabel('Residual')
plt.title('Residual vs Predicted Value')
plt.show()
```

Another important piece of information to keep in mind that fitted line can be affected by **outliers**. Assume that you got the information about three new engineers. Despite the fact that they have worked for 4 years already, their salary is still low. See what happens to regression line when we include this observations into model:

```python
# add 3 new points
years_outlier = np.concatenate([years, [4, 4.1, 4.2]])
salary_outlier = np.concatenate([salary, [800, 810, 900]])

# add column with ones
X = np.vstack([np.ones(shape=n+3), years_outlier]).T
b = np.linalg.inv(X.T@X) @ (X.T@salary_outlier)
y_hat = X@b
print(f"Model: y ={b[0]: .2f} +{b[1]: .2f}x")
```

    Model: y = 859.96 + 54.23x

```python
y_space_outlier = b[0] + b[1]*x_space

plt.figure(figsize=(10,4))
plt.scatter(years_outlier, salary_outlier, label='Observations')
plt.plot(x_space, y_space, '--', color='black',
         linewidth=1, label='Fitted Line (w/o outliers)')
plt.plot(x_space, y_space_outlier, '--', color='red',
         linewidth=1, label='Fitted Line (w/ outliers)')
plt.scatter(4.1, 850, s=6000, facecolors='none', edgecolors='r')
plt.xlim([0,5])
plt.ylim([750, 1250])
plt.xlabel('Years of Experience')
plt.ylabel('Salary (% per month)')
plt.title('Relationship of X and Y')
plt.gca().spines['top'].set_visible(False)
plt.gca().spines['right'].set_visible(False)
plt.legend()
plt.show()
```

There are two possible types of outliers:

1. **Leverage point** - observation that is located away from the point cloud, but *doesn't affect* the slope of the regression line.
2. **Influential point** - observation that *affects* the slope of the regression line.

## Goodness of Fit

After you have fitted the model you might want to check how good your model is. One of the most common metrics is **R square** (coefficient of determination), which shows the percentage of dependent variable variation explained by the model. It can be find as:

$$R^2 = r_{xy}^T R_{xx}^{-1} r_{xy} \tag{16}$$

* $r_{xy}$ - vector of correlations between independent variables and dependent variable $[r_{x_1y} r_{x_2y} ... r_{x_ky}]^T$
* $R_{xx}$ - correlation matrix between predictor variables

\begin{equation}
R = \begin{bmatrix}
r_{x_1 x_1} & r_{x_1 x_2} & ... & r_{x_1 x_k} \\
r_{x_2 x_1} & r_{x_2 x_2} & ... & r_{x_2 x_k} \\
... & ... & ... & ... \\
r_{x_k x_1} & r_{x_k x_2} & ... & r_{x_k x_k} \\
\end{bmatrix}
\tag{17}
\end{equation}

If there is only one independent variable $x$, then $R^2$ equals to squared correlation value between $x$ and $y$.

However, there are some limitations of $R^2$ - it always increases whenever a new variable is added to the model. So even if you add some irrelevant feature to $X$ (for example height of the person for salary prediction) $R^2$ will stay the same of van even increase a bit.

```python
r_xy = np.corrcoef(years, salary)[0, 1] # correlation
R_sq = r_xy**2

print(f"{R_sq*100: .2f}% of salary variance change can be explained by the model.")
```

     78.19% of salary variance change can be explained by the model.

```python
# sklearn
print(f"R-squared: {ols_model.score(years.reshape(n,1), salary): .4f}")
```

    R-squared:  0.7819

Example with 5 predictors:

```python
k = 5
x, y = make_regression(n_samples=n, n_features=k,
                       noise=50, random_state=RANDOM_SEED)
x = np.concatenate([np.ones(shape=(n,1)), x], axis=1)
```

```python
r_xy = np.corrcoef(x[:,1:], y, rowvar=False)[-1, :-1]
R_xx = np.corrcoef(x[:, 1:], rowvar=False)
R_sq = r_xy.T@np.linalg.inv(R_xx)@r_xy

print(f"{R_sq*100: .2f}% of salary variance change can be explained by the model.")
```

     73.76% of salary variance change can be explained by the model.

```python
# sklearn realization
ols_model = LinearRegression(fit_intercept=False)
ols_model.fit(x, y)
print(f"R-squared: {ols_model.score(x, y): .4f}")
```

    R-squared:  0.7376

## Summary

To sum it up we can generalize steps for OLS regression model fitting steps:

* Before the fitting:
  1. Check the relationship between dependent and independent variables. It should be linear.
  2. Check the relationship between each independent variables combinations. No variable should be a linear combination of another.
  2. Check for outliers.
* After the fitting:
  1. Check the distribution of residuals (errors). It should be normal with the center at zero.
  2. Check for the constant variance (no weird patterns on a residual-fitted value plot).
  3. Check your $R^2$.


# Quantum Channel Capacity

Date : December 15, 2021

This notebook contains material supporting a paper, currently titled *Five Starter Pieces: Quantum Information Science via Semi-definite Programs*, by Vikesh Siddhu (vsiddhu@protonmail.com) and Sridhar Tayur (stayur@cmu.edu). The paper is available on this **[arXiv](http://arxiv.org/abs/2112.08276)** link. The arXiv paper is released there is under the **[arXiv.org perpetual, non-exclusive license](https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html)**, and this code is released under the **[MIT license](https://opensource.org/licenses/MIT)**.

This notebook depends upon various packages including [numpy](https://numpy.org/) >= 1.19.5, [picos](https://picos-api.gitlab.io/picos/index.html) >= 2.2.55, and [cvxopt](http://cvxopt.org/) >= 1.2.5.

[](https://colab.research.google.com/github/vsiddhu/SDP-Quantum-OR/blob/master/Notebook%205%20-%20Quantum%20Channel%20Capacity.ipynb)

### Introduction
Any linear map $\mathcal{M}:\hat{ \mathcal{H}_a} \mapsto \hat{ \mathcal{H}_b}$ has a Choi-Jamiolkowski representation
$$
\mathcal{J}(\mathcal{M})_{ba} = \mathcal{M} \otimes \mathcal{I}_a (\gamma)
$$
where $\gamma = \sum_{ij} | ii \rangle \langle  jj|$.

A quantum channel $\mathcal{B}:\hat{ \mathcal{H}_a} \mapsto \hat{ \mathcal{H}_b}$ with complementary channel
$\mathcal{B}^c:\hat{ \mathcal{H}_a} \mapsto \hat{ \mathcal{H}_c}$ is $\epsilon$ degradable when there is another quantum channel $\mathcal{C}:\hat{ \mathcal{H}_b} \mapsto \hat{ \mathcal{H}_c}$ such that,

$$
|| \mathcal{C} \circ \mathcal{B} - \mathcal{B}^c ||_{\diamond} \leq \epsilon,
$$

where $|| \mathcal{M} ||_{\diamond}$ is the diamond norm of a map. Then the smallest $\epsilon$ for which $\mathcal{B}$ is $\epsilon$ degradable is given by twice the optimum value of the semi-definite program

 \begin{align}
 \begin{aligned}
   \text{mininimize} \; & \mu &\\
   \text{subject to} \; & \rm Tr_{c}(Z_{ca}) \preceq \mu I_a, &\\
  & \rm Tr_{c}(J_{cb}(\mathcal{C})) = I_b, &\\
  & Z_{ca} \succeq \mathcal{J}_{ca}(\mathcal{B}^c) - \mathcal{J}_{ca}(\mathcal{C} \circ \mathcal{B}), &\\
  & Z_{ca} \succeq 0, & \\
  \text{and} \; & \mathcal{J}_{cb}(\mathcal{C}) \succeq 0 &,
  \end{aligned}
  \end{align}

  where $I_a$ and $I_b$ are the identity on $\mathcal{H}_a$ and $\mathcal{H}_b$ respectively.
In what follows, consider various examples of this SDP.

If $\mathcal{B}$ is $\epsilon$-degradable and has channel coherent information $\mathcal{Q}^{(1)}(\mathcal{B})$,
then

\begin{equation}
|\mathcal{Q}(\mathcal{B}) - \mathcal{Q}^{(1)}(\mathcal{B})|
    \leq \epsilon \log(d_c - 1)/2 + \epsilon \log(d_c) + h(\epsilon/2) + (1 + \epsilon/2)h(2/(2+ \epsilon))
\end{equation}
where $h(x) = -\big( x \log x +(1-x) \log (1+x) \big)$ is the binary entropy function.

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
import HelperFunctionsforNotebook5 as crp
import picos as pic
import cvxopt as cvx
```

```python
print('Solvers supported on this installation of picos:', pic.solvers.all_solvers().keys())

```

    Solvers supported on this installation of picos: dict_keys(['cplex', 'cvxopt', 'ecos', 'glpk', 'gurobi', 'mosek', 'mskfsn', 'scip', 'smcp'])

```python
print('Solvers available to picos on this machine :', pic.solvers.available_solvers())
```

    Solvers available to picos on this machine : ['cvxopt', 'mosek', 'mskfsn']

```python
# For Google Colab use, commands installing packages
try:
  import google.colab
  IN_COLAB = True
except:
  IN_COLAB = False

# Let's start with Pyomo
if IN_COLAB:
    !pip install -q picos
    !pip install -q cvxopt
```

### Example 1
Compute smallest $\epsilon$ for which the qubit dephasing channel
\begin{equation}
    \mathcal{F}_{q}(\rho) = (1-q) \rho + q Z \rho Z
\end{equation}
$0 \leq q \leq 1$ is $\epsilon$ degradable.

```python
q = np.random.rand()

#Kraus Operators for qubit dephasing channel
k1 = np.sqrt(1-q)*np.array([[1.,0.],[0.,1.]])
k2 = np.sqrt(q)*np.array([[1.,0.],[0.,-1.]])
krsLstF = [k1,k2]

(dc,db,da) = np.shape(krsLstF)

#Choi-Jamiolkowski representations of channel and its complement
(jbaF, jcaF) = crp.krausToChoiJ(krsLstF)

```

```python
#Constants
#----------
JbaFPic = pic.Constant("J(B)_ba", jbaF)
JcaFPic = pic.Constant("J(B)_ca", jcaF)

iMatA = pic.Constant('Ia', np.eye(da))
iMatB = pic.Constant('Ib', np.eye(db))
shpCA = (dc*da,dc*da)
shpCB = (dc*db,dc*db)

#Variables
#----------
ZPic = pic.HermitianVariable("Zca", shpCA)
JPic = pic.HermitianVariable("Jcb", shpCB)
mu = pic.RealVariable("mu")

```

```python
prob1P = pic.Problem()

#Constraint
#----------
prob1P.add_constraint(ZPic >> 0)
prob1P.add_constraint(JPic >> 0)

ZaPic = pic.partial_trace(ZPic,subsystems=(0), dimensions=(dc,da))
JbPic = pic.partial_trace(JPic,subsystems=(0), dimensions=(dc,db))

prob1P.add_constraint(ZaPic << mu*iMatA)
prob1P.add_constraint(JbPic == iMatB)

JcaPic = crp.choiJOfChanInSeriesPic(JbaFPic,JPic,da,db,dc)
prob1P.add_constraint(ZPic >> JcaFPic-JcaPic)
```

    <4×4 Complex LMI Constraint: Zca ≽ J(B)_ca - shuffled(shuffled(Jcb,ikjl,(2,2,2,2),C)·shuffled(J(B)_ba,ikjl,(2,2,2,2),C),ikjl,(2,2,2,2),C)>

```python
#Objective
#----------
prob1P.set_objective('min',mu)

#User readable view of the problem being composed in PICOS'
print(prob1P)
```

    --------------------------------------------------------------------------------------------------
    Complex Semidefinite Program
      minimize mu
      over
        4×4 hermitian variable Jcb, Zca
        1×1 real variable mu
      subject to
        Zca ≽ 0
        Jcb ≽ 0
        Zca.{tr([2×2])⊗[2×2]} ≼ mu·Ia
        Jcb.{tr([2×2])⊗[2×2]} = Ib
        Zca ≽ J(B)_ca -
          shuffled(shuffled(Jcb,ikjl,(2,2,2,2),C)·shuffled(J(B)_ba,ikjl,(2,2,2,2),C),ikjl,(2,2,2,2),C)
    --------------------------------------------------------------------------------------------------

```python
#Solve the problem using cvxopt as a solver
prob1P.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
#Solver claims to have found optimal saolution
mu1P =  prob1P.value
eps1 = 2*mu1P
print(eps1)
```

    6.20984358392016e-10

```python
print('Least epsilon for which the dephasing channel is epsilon degradable')
print('Using SDP = ', eps1)
print('Difference between the SDP value and the algberaic values', abs(eps1 - 0))
```

    Least epsilon for which the dephasing channel is epsilon degradable
    Using SDP =  6.20984358392016e-10
    Difference between the SDP value and the algberaic values 6.20984358392016e-10

The qubit dephasing channel,

$$\mathcal{F}_{q}(\rho) = (1-q) \rho + q Z \rho Z, $$

is degradable for any $0 \leq q \leq 1$. This degradability implies that
$\mathcal{F}_{q}$ is $\epsilon$ degradable for $\epsilon = 0$, as confirmed by
the numerics above. Given $\epsilon = 0$, the bound

\begin{equation}
|\mathcal{Q}(\mathcal{F}_q) - \mathcal{Q}^{(1)}(\mathcal{F}_q)|
    \leq \epsilon \log(d_c - 1)/2 + \epsilon \log(d_c) + h(\epsilon/2) + (1 + \epsilon/2)h(2/(2+ \epsilon))
\end{equation}

trivially evaluates to
\begin{equation}
   |\mathcal{Q}(\mathcal{F}_q) - \mathcal{Q}^{(1)}(\mathcal{F}_q)| =0
\end{equation}

Using the above equation, along with the expression for the channel coherent information, we get
\begin{equation}
    \mathcal{Q}(\mathcal{F}_q) = \mathcal{Q}^{(1)}(\mathcal{F}_{q}) = 1 - h(q).
\end{equation}

### Example 2
Compute smallest $\epsilon$ for which the qubit erasure channel
\begin{equation}
    \mathcal{E}_{p}(\rho) = (1-p) \rho + p \rm Tr(\rho) |e\rangle\langle e|
\end{equation}
with erasure probability $0 \leq p \leq 1$ is $\epsilon$ degradable.

```python
#Consider a random erasure probability less than half
p = np.random.rand()/2

#Kraus Operators for Erasure
k1 = np.sqrt(p)*np.array([[0.,0.],[0.,0.],[1.,0.]])
k2 = np.sqrt(p)*np.array([[0.,0.],[0.,0.],[0.,1.]])
k3 = np.sqrt(1-p)*np.array([[1.,0.],[0.,1.],[0.,0.]])
krsLstE = [k1,k2,k3]

(dc,db,da) = np.shape(krsLstE)

#Choi-Jamiolkowski representations of channel and its complement
(jbaE, jcaE) = crp.krausToChoiJ(krsLstE)
```

```python
#Constants
#----------
JbaEPic = pic.Constant("J(B)_ba", jbaE)
JcaEPic = pic.Constant("J(B)_ca", jcaE)
iMatA = pic.Constant('Ia', np.eye(da))
iMatB = pic.Constant('Ib', np.eye(db))
shpCA = (dc*da,dc*da)
shpCB = (dc*db,dc*db)

#Variables
#----------
ZPic = pic.HermitianVariable("Zca", shpCA)
JPic = pic.HermitianVariable("Jcb", shpCB)
mu = pic.RealVariable("mu")

```

```python
prob2P = pic.Problem()
```

```python
#Constraint
#----------
prob2P.add_constraint(ZPic >> 0)
prob2P.add_constraint(JPic >> 0)

ZaPic = pic.partial_trace(ZPic,subsystems=(0), dimensions=(dc,da))
JbPic = pic.partial_trace(JPic,subsystems=(0), dimensions=(dc,db))

prob2P.add_constraint(ZaPic << mu*iMatA)
prob2P.add_constraint(JbPic == iMatB)

JcaPic = crp.choiJOfChanInSeriesPic(JbaEPic,JPic,da,db,dc)
prob2P.add_constraint(ZPic >> JcaEPic-JcaPic)

```

    <6×6 Complex LMI Constraint: Zca ≽ J(B)_ca - reshaped(shuffled(shuffled(Jcb,ikjl,(3,3,3,3),C)·reshaped(shuffled(J(B)_ba,ikjl,(3,2,3,2),C)ᵀ, 4×9)ᵀ,ikjl,(3,3,2,2),C)ᵀ, 6×6)ᵀ>

```python
#Objective
#----------
prob2P.set_objective('min',mu)

#User readable view of the problem being composed in PICOS'
print(prob2P)
```

    ----------------------------------------------------------------------------------------------------
    Complex Semidefinite Program
      minimize mu
      over
        6×6 hermitian variable Zca
        9×9 hermitian variable Jcb
        1×1 real variable mu
      subject to
        Zca ≽ 0
        Jcb ≽ 0
        Zca.{tr([3×3])⊗[2×2]} ≼ mu·Ia
        Jcb.{tr([3×3])⊗[3×3]} = Ib
        Zca ≽ J(B)_ca -
          reshaped(shuffled(shuffled(Jcb,ikjl,(3,3,3,3),C)·reshaped(shuffled(J(B)_ba,ikjl,(3,2,3,2),C)ᵀ,
          4×9)ᵀ,ikjl,(3,3,2,2),C)ᵀ, 6×6)ᵀ
    ----------------------------------------------------------------------------------------------------

```python
#Solve the problem using cvxopt as a solver
prob2P.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
#Solver claims to have found optimal saolution
mu2P =  prob2P.value
eps2 = 2*mu2P
```

```python
print('Using SDP = ', eps2)
print('Bound on difference between capacity and coherent information =', crp.boundFunc(eps2,dc))
```

    Using SDP =  1.2914823168265619e-09
    Bound on difference between capacity and coherent information = 4.3982699098789774e-08

The qubit erasure channel $\mathcal{E}_{p}$ has coherent information

\begin{equation}
    \mathcal{Q}^{(1)}(\mathcal{E}_{p}) = \max(1 - 2p,0).
\end{equation}

The channel is degradable for erasure probability $0 \leq p \leq 1/2$. This degradability implies that $\mathcal{E}_{p}$ is $\epsilon$ degradable for $\epsilon = 0$, as confirmed by the numerics above when $0 \leq p \leq 1/2$. For values of $0 \leq p \leq 1/2$, the bound

\begin{equation}
|\mathcal{Q}(\mathcal{E}_p) - \mathcal{Q}^{(1)}(\mathcal{E}_{p})|
    \leq \epsilon \log(d_c - 1)/2 + \epsilon \log(d_c) + h(\epsilon/2) + (1 + \epsilon/2)h(2/(2+ \epsilon))
\end{equation}

trivially evaluates to
\begin{equation}
   |\mathcal{Q}(\mathcal{E}_p) - \mathcal{Q}^{(1)}(\mathcal{E}_p)| =0
\end{equation}
For $1/2 \leq p \leq 1$, $\epsilon$ is non-zero,the bound is also non-zero. It turns out that even for $1/2 \leq p \leq 1$, $\mathcal{Q}(\mathcal{E}_p) = \mathcal{Q}^{(1)}(\mathcal{E}_p)$, due to anti-degradability of the
erasure channel for erasure probability $1/2 \leq p \leq 1$.

### Example 3
Compute smallest $\epsilon$ for which the qubit depolarizing channel
\begin{equation}
    \Delta(\rho) = (1-p) \rho + \frac{p}{3} (X \rho X + Y \rho Y  + Z \rho Z),
\end{equation}
$0 \leq p \leq 1$ is $\epsilon$ degradable. The depolarizing channel above can also be written using the parameter $\lambda = 1 - 4p/3$ as
\begin{equation}
    \Delta(\rho) = \lambda \rho + \frac{(1-\lambda)}{2} \rm Tr(\rho) I.
\end{equation}

```python
#Example 3
p = np.random.rand()/10

#Kraus Operators for Depolarizing Channel
k1 = np.sqrt(1-p)*np.array([[1.,0.],[0.,1.]])
k2 = np.sqrt(p/3)*np.array([[0.,1.],[1.,0.]])
k3 = np.sqrt(p/3)*np.array([[0.,-1j],[1j,0.]])
k4 = np.sqrt(p/3)*np.array([[1.,0.],[0.,-1.]])

krsLstD = [k1,k2,k3,k4]
(dc,db,da) = np.shape(krsLstD)

(jbaD, jcaD) = crp.krausToChoiJ(krsLstD)
```

```python
#Constants
#----------
JbaDPic = pic.Constant("J(B)_ba", jbaD)
JcaDPic = pic.Constant("J(B)_ca", jcaD)
iMatA = pic.Constant('Ia', np.eye(da))
iMatB = pic.Constant('Ib', np.eye(db))

#Variables
#----------
shpCA = (dc*da,dc*da)
shpCB = (dc*db,dc*db)

ZPic = pic.HermitianVariable("Zca", shpCA)
JPic = pic.HermitianVariable("Jcb", shpCB)
mu = pic.RealVariable("mu")
```

```python
prob3P = pic.Problem()
```

```python
#Constraint
#----------
prob3P.add_constraint(ZPic >> 0)
prob3P.add_constraint(JPic >> 0)

ZaPic = pic.partial_trace(ZPic,subsystems=(0), dimensions=(dc,da))
JbPic = pic.partial_trace(JPic,subsystems=(0), dimensions=(dc,db))

prob3P.add_constraint(ZaPic << mu*iMatA)
prob3P.add_constraint(JbPic == iMatB)

JcaPic = crp.choiJOfChanInSeriesPic(JbaDPic,JPic,da,db,dc)
prob3P.add_constraint(ZPic >> JcaDPic-JcaPic)

```

    <8×8 Complex LMI Constraint: Zca ≽ J(B)_ca - reshaped(shuffled(reshaped(shuffled(Jcb,ikjl,(4,2,4,2),C)ᵀ, 4×16)ᵀ·shuffled(J(B)_ba,ikjl,(2,2,2,2),C),ikjl,(4,4,2,2),C)ᵀ, 8×8)ᵀ>

```python
#Objective
#----------
prob3P.set_objective('min',mu)

#User readable view of the problem being composed in PICOS'
print(prob3P)
```

    -------------------------------------------------------------------
    Complex Semidefinite Program
      minimize mu
      over
        8×8 hermitian variable Jcb, Zca
        1×1 real variable mu
      subject to
        Zca ≽ 0
        Jcb ≽ 0
        Zca.{tr([4×4])⊗[2×2]} ≼ mu·Ia
        Jcb.{tr([4×4])⊗[2×2]} = Ib
        Zca ≽ J(B)_ca -
          reshaped(shuffled(reshaped(shuffled(Jcb,ikjl,(4,2,4,2),C)ᵀ,
          4×16)ᵀ·shuffled(J(B)_ba,ikjl,(2,2,2,2),C),ikjl,(4,4,2,2),C)ᵀ,
          8×8)ᵀ
    -------------------------------------------------------------------

```python
#Solve the problem using cvxopt as a solver
prob3P.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
#Solver claims to have found optimal solution
mu3P =  prob3P.value
eps3 = 2*mu3P
```

```python
print('Least epsilon for which the depolarizing channel is epsilon degradable')
print('Using the SDP = ', eps3)
print('Bound on difference between capacity and coherent information =', crp.boundFunc(eps3,dc))
```

    Least epsilon for which the depolarizing channel is epsilon degradable
    Using the SDP =  0.0030743301525582175
    Bound on difference between capacity and coherent information = 0.041751518313526206

The qubit depolarizing channel is not degradable and the least value of $\epsilon$ for which the channel is $\epsilon$ degradable is found numerically, as demonstrated above.

The channel coherent information
\begin{equation}
    \mathcal{Q}^{(1)}(\Delta) = \max(1 - h(p) - p \log 3, 0),
\end{equation}
along with $\epsilon$ computed using the SDP can be used to obtain a bound,

\begin{equation}
|\mathcal{Q}(\Delta) - \mathcal{Q}^{(1)}(\Delta)|
    \leq \epsilon \log(d_c - 1)/2 + \epsilon \log(d_c) + h(\epsilon/2) + (1 + \epsilon/2)h(2/(2+ \epsilon)),
\end{equation}
which is usually not tight, except when $p = 0$ or $p = 1$.

Notebook 5 - Quantum Channel Capacity.ipynb