# Statistics

## 1.1 Sales for the past week was the following amounts: [3505, 2400, 3027, 2798, 3700, 3250, 2689]. Without using library functions, what is the mean, variance, and standard deviation of of sales from last week? (for extra bonus points, write your own function that can calculate these two values for any sized list)

```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import random
from math import sqrt

sales = [3505, 2400, 3027, 2798, 3700, 3250, 2689]
s = len(sales)
mean = sum(sales) / len(sales)
# mean calculation
sales.sort()
if s % 2 == 0:
  median1 = sales[s//2]
  median2 = sales[s//2 - 1]
  median = (median1 + median2)/2
else:
  median = sales[s//2]
print("Sales median is:" + str(median))

#calculating variance
sales_var = sum((xi - median) ** 2 for xi in sales) / (len(sales) -1)
print("Sales Variance is: " + str(sales_var))

#calculating standard deviation
# sales_std = sqrt(sum([(number - mean(sales)) ** 2 for number in sales]) / (len(sales) - 1))
# print("The standard deviation is: " + str(sales_std))
# print("The standard deviation is: " + str(np.std(sales)))

sales_std = np.std(sales, ddof=1)
sales_std
```

    Sales median is:3027
    Sales Variance is: 215159.33333333334

    463.02041505953576

## 1.2 Find the covariance between last week's sales numbers and the number of customers that entered the store last week: [127, 80, 105, 92, 120, 115, 93] (you may use librray functions for calculating the covariance since we didn't specifically talk about its formula)

```
customers = [127, 80, 105, 92, 120, 115, 93]
```

```
numer = np.cov(sales, customers)
numer
```

    array([[214387.9047619 ,   -869.64285714],
           [  -869.64285714,    290.95238095]])

## 1.3 Find the standard deviation of customers who entered the store last week. Then, use the standard deviations of both sales and customers to standardize the covariance to find the correlation coefficient that summarizes the relationship between sales and customers. (You may use library functions to check your work.)

```
df = pd.DataFrame({
    'sales' : sales,
    'customers' : customers
    })
# df['customers'] = pd.Series[customers]
df.head()
```

      sales
      customers

      0
      2400
      127

      1
      2689
      80

      2
      2798
      105

      3
      3027
      92

      4
      3250
      120

```
customer_std = np.std(customers)
print(customer_std)
```

    15.792015549069118

```
r = numer / (sales_std * customer_std)
r
```

    array([[29.31990623, -0.11893323],
           [-0.11893323,  0.03979094]])

```
np.corrcoef(sales, customers)
```

    array([[ 1.        , -0.11011078],
           [-0.11011078,  1.        ]])

```
df.corr()
```

      sales
      customers

      sales
      1.000000
      -0.110111

      customers
      -0.110111
      1.000000

## 1.4 Use pandas to import a cleaned version of the titanic dataset from the following link: [Titanic Dataset](https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv)

## Calculate the variance-covariance matrix and correlation matrix for the titanic dataset's numeric columns. (you can encode some of the categorical variables and include them as a stretch goal if you finish early)

```
df = pd.read_csv('https://raw.githubusercontent.com/Geoyi/Cleaning-Titanic-Data/master/titanic_clean.csv')
df.head()
```

      Unnamed: 0
      pclass
      survived
      name
      sex
      age
      sibsp
      parch
      ticket
      fare
      cabin
      embarked
      boat
      body
      home.dest
      has_cabin_number

      0
      1
      1.0
      1.0
      Allen, Miss. Elisabeth Walton
      female
      29.0000
      0.0
      0.0
      24160
      211.3375
      B5
      S
      2
      NaN
      St Louis, MO
      1

      1
      2
      1.0
      1.0
      Allison, Master. Hudson Trevor
      male
      0.9167
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      11
      NaN
      Montreal, PQ / Chesterville, ON
      1

      2
      3
      1.0
      0.0
      Allison, Miss. Helen Loraine
      female
      2.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      NaN
      NaN
      Montreal, PQ / Chesterville, ON
      1

      3
      4
      1.0
      0.0
      Allison, Mr. Hudson Joshua Creighton
      male
      30.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      NaN
      135.0
      Montreal, PQ / Chesterville, ON
      1

      4
      5
      1.0
      0.0
      Allison, Mrs. Hudson J C (Bessie Waldo Daniels)
      female
      25.0000
      1.0
      2.0
      113781
      151.5500
      C22 C26
      S
      NaN
      NaN
      Montreal, PQ / Chesterville, ON
      1

```
df.cov()
```

      Unnamed: 0
      pclass
      survived
      age
      sibsp
      parch
      fare
      body
      has_cabin_number

      Unnamed: 0
      143117.500000
      284.357034
      -53.967125
      -1442.939812
      25.828746
      1.172783
      -9410.735123
      591.579132
      -95.438885

      pclass
      284.357034
      0.701969
      -0.127248
      -3.954605
      0.053090
      0.013287
      -24.227788
      -2.876653
      -0.249992

      survived
      -53.967125
      -0.127248
      0.236250
      -0.314343
      -0.014088
      0.034776
      6.146023
      0.000000
      0.061406

      age
      -1442.939812
      -3.954605
      -0.314343
      165.850021
      -2.559806
      -1.459378
      114.416613
      81.622922
      1.463138

      sibsp
      25.828746
      0.053090
      -0.014088
      -2.559806
      1.085052
      0.336833
      8.641768
      -8.708471
      -0.003946

      parch
      1.172783
      0.013287
      0.034776
      -1.459378
      0.336833
      0.749195
      9.928031
      4.237190
      0.013316

      fare
      -9410.735123
      -24.227788
      6.146023
      114.416613
      8.641768
      9.928031
      2678.959738
      -179.164684
      10.976961

      body
      591.579132
      -2.876653
      0.000000
      81.622922
      -8.708471
      4.237190
      -179.164684
      9544.688567
      3.625689

      has_cabin_number
      -95.438885
      -0.249992
      0.061406
      1.463138
      -0.003946
      0.013316
      10.976961
      3.625689
      0.174613

# Orthogonality

## 2.1 Plot two vectors that are orthogonal to each other. What is a synonym for orthogonal?

A synonym for orthogonal is perpindicular

```
vector_1 = [0, 3]
vector_2 = [3, 0]

plt.arrow(0, 0, vector_1[0], vector_1[1], color='red')
plt.arrow(0, 0, vector_2[0], vector_2[1], color='green')
plt.xlim(-2, 5)
plt.ylim(-2, 5)
plt.title("Yeet")
plt.show()
```

## 2.2 Are the following vectors orthogonal? Why or why not?

\begin{align}
a = \begin{bmatrix} -5 \\ 3 \\ 7 \end{bmatrix}
\qquad
b = \begin{bmatrix} 6 \\ -8 \\ 2 \end{bmatrix}
\end{align}

```
a = np.array([
              [-5],
              [3],
              [7]
              ])

b = np.array([
              [6],
              [-8],
              [2]
              ])

# a = [-5, 3, 7]

np.dot(a.T, b)
```

    array([[-40]])

#Not orthogonal since dotproduct !=0

## 2.3 Compute the following values: What do these quantities have in common?

## What is $||c||^2$?

## What is $c \cdot c$?

## What is $c^{T}c$?

\begin{align}
c = \begin{bmatrix} 2 & -15 & 6 & 20 \end{bmatrix}
\end{align}

```
c = np.array([2, -15, 6, 20])

magnitude_c = np.linalg.norm(c)
mag_sqrd_c = magnitude_c**2
print('The magnitude of c squared is: ' + str(mag_sqrd_c))
```

    The magnitude of c squared is: 665.0

```
print(np.dot(c, c))
```

    665

```
c_tpose = c.T
c_tpose * c
```

    array([  4, 225,  36, 400])

#They all add up to 665

# Unit Vectors

## 3.1 Using Latex, write the following vectors as a linear combination of scalars and unit vectors:

\begin{align}
d = \begin{bmatrix} 7 \\ 12 \end{bmatrix}
e = \begin{bmatrix} 2 \\ 11 \\ -8  \end{bmatrix}
\end{align}

#Linear Combinations:
\begin{align}
d = \begin{bmatrix} 7 \\ 12 \end{bmatrix} = 7\begin{bmatrix} 1 \\ 0 \end{bmatrix} + 12\begin{bmatrix} 0 \\ 1 \end{bmatrix} = 7\hat{i} + 3\hat{j}
\end{align}

\begin{align}
e = \begin{bmatrix} 2 \\ 11 \\ -8  \end{bmatrix} = 2\begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix} + 11\begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix} + (-8)\begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix} = 2\hat{i} + 11\hat{j} + (-8)\hat{k}
\end{align}

## 3.2 Turn vector $f$ into a unit vector:

\begin{align}
f = \begin{bmatrix} 4 & 12 & 11 & 9 & 2 \end{bmatrix}
\end{align}

```
f = np.array([4, 12, 11, 9, 2])

norm = np.linalg.norm(f)
norm
f / norm
```

    array([0.20908335, 0.62725005, 0.57497921, 0.47043754, 0.10454167])

# Linear Independence / Dependence

## 4.1 Plot two vectors that are linearly dependent and two vectors that are linearly independent (bonus points if done in $\mathbb{R}^3$).

```
from mpl_toolkits.mplot3d import Axes3D

# Axis Bounds
plt.xlim(-1,15)
plt.ylim(-1,15)

# dependent Vectors
i_hat = [6, 0, 0]

j_hat = np.multiply(3, i_hat)

l_hat = [6, 12, 4]
m_hat = [4, 3, 2]
# Fix Axes
plt.axes().set_aspect('equal')

# PLot Vectors
plt.arrow(0, 0, i_hat[0], i_hat[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')
plt.arrow(0, 0, j_hat[0], j_hat[1], linewidth=3, head_width=.05, head_length=0.05, color ='blue')
plt.arrow(0, 0, l_hat[0], l_hat[1], linewidth=3, head_width=.05, head_length=0.05, color ='black')
plt.arrow(0, 0, m_hat[0], m_hat[1], linewidth=3, head_width=.05, head_length=0.05, color ='green')
plt.title("Vectors, man", color='white')
plt.show();

```

```

```

# Span

## 5.1 What is the span of the following vectors?

\begin{align}
g = \begin{bmatrix} 1 & 2 \end{bmatrix}
\qquad
h = \begin{bmatrix} 4 & 8 \end{bmatrix}
\end{align}

```
g = [1, 2]
h = [4, 8]
array = np.array([g + h])
np.linalg.matrix_rank(array)
```

    1

## 5.2 What is the span of $\{l, m, n\}$?

\begin{align}
l = \begin{bmatrix} 1 & 2 & 3 \end{bmatrix}
\qquad
m = \begin{bmatrix} -1 & 0 & 7 \end{bmatrix}
\qquad
n = \begin{bmatrix} 4 & 8  & 2\end{bmatrix}
\end{align}

```
l = [1, 2, 3]
m = [-1, 0, 7]
n = [4, 8, 2]

array = np.array([l + m + n]).reshape(3, 3).T
array
np.linalg.matrix_rank(array)
```

    3

# Basis

## 6.1 Graph two vectors that form a basis for $\mathbb{R}^2$

```
# Plot Linearly Dependent Vectors

# Axis Bounds
plt.xlim(-2,3.5)
plt.ylim(-1,3)

# Original Vector
a = [-1.5,.5]
b = [3, 1]

# Plot Vectors
plt.arrow(0,0, a[0], a[1], linewidth=3, head_width=.05, head_length=0.05, color ='blue')
plt.arrow(0,0, b[0], b[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("Vectors that form a basis for R**2")
plt.show()
```

## 6.2 What does it mean to form a basis?

To form a basis means that the vectors have a point in which they are [0,0]

# Rank

## 7.1 What is the Rank of P?

\begin{align}
P = \begin{bmatrix}
1 & 2 & 3 \\
 -1 & 0 & 7 \\
4 & 8  & 2
\end{bmatrix}
\end{align}

```
p = np.array([
              [1, 2, 3],
              [-1, 0, 7],
              [4, 8, 2]
              ])

np.linalg.matrix_rank(p)
```

    3

## 7.2 What does the rank of a matrix tell us?

The rank of a matrix tells us the scaler by which the matrix can be multiplied and reach all points within the 2d array

# Linear Projections

## 8.1 Line $L$ is formed by all of the vectors that can be created by scaling vector $v$
\begin{align}
v = \begin{bmatrix} 1 & 3 \end{bmatrix}
\end{align}

\begin{align}
w = \begin{bmatrix} -1 & 2 \end{bmatrix}
\end{align}

## find $proj_{L}(w)$

## graph your projected vector to check your work (make sure your axis are square/even)

```
# Axis Bounds
plt.xlim(-1.1,4)
plt.ylim(-1.1,4)

# Original Vector
v = [1,3]
w = [-1,2]
proj = np.multiply(,v)
x_minus_proj = w-proj

# Set axes
axes = plt.gca()
plt.axes().set_aspect('equal')

# Get Vals for L
x_vals = np.array(axes.get_xlim())
y_vals = 1/2*x_vals

# Plot Vectors and L
plt.plot(x_vals, y_vals, '--', color='b', linewidth=1)
plt.arrow(0, 0, proj[0], proj[1], linewidth=3, head_width=.05, head_length=0.05, color ='gray')
plt.arrow(0, 0, v[0], v[1], linewidth=3, head_width=.05, head_length=0.05, color ='green')
plt.arrow(0, 0, w[0], w[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')
plt.arrow(proj[0], proj[1], x_minus_proj[0], x_minus_proj[1], linewidth=3, head_width=.05, head_length=0.05, color = 'yellow')

plt.title("non x-axis projection")
plt.show()
```

# Stretch Goal

## For vectors that begin at the origin, the coordinates of where the vector ends can be interpreted as regular data points. (See 3Blue1Brown videos about Spans, Basis, etc.)

## Write a function that can calculate the linear projection of each point (x,y) (vector) onto the line y=x. run the function and plot the original points in blue and the new projected points on the line y=x in red.

## For extra points plot the orthogonal vectors as a dashed line from the original blue points to the projected red points.

```
import pandas as pd
import matplotlib.pyplot as plt

# Creating a dataframe for you to work with -Feel free to not use the dataframe if you don't want to.
x_values = [1, 4, 7, 3, 9, 4, 5 ]
y_values = [4, 2, 5, 0, 8, 2, 8]

data = {"x": x_values, "y": y_values}

df = pd.DataFrame(data)

df.head()

plt.scatter(df.x, df.y)
plt.show()
```

```

```


# Inverse Linear Regression Proof

In this notebook we want to check if both ways of 'inverse linear regression' lead to the same result!

Write the function that uses expenses as input and sqft_living as output!
* Upload and preprocess the data
* Write a function to compute the Simple Linear Regression weights with sqft_living as output and price as input
* Write a function to make predictions of the output given the input feature of $1.000.000

Then write a function that uses sqft_living as input and price as output and invert it!
* Write a function to compute the Simple Linear Regression weights with price as output and sqft_living as input
* Write a function to make predictions of the input given the output feature of $1.000.000

# Upload and preprocess the data

Dataset on house sales data (housing.csv).

```python
import pandas as pd
data = pd.read_csv('housing.csv')
# Look at the table to check potential features
data[:10]
```

      id
      date
      price
      bedrooms
      bathrooms
      sqft_living
      sqft_lot
      floors
      waterfront
      view
      ...
      grade
      sqft_above
      sqft_basement
      yr_built
      yr_renovated
      zipcode
      lat
      long
      sqft_living15
      sqft_lot15

      0
      7129300520
      20141013T000000
      221900.0
      3
      1.00
      1180
      5650
      1.0
      0
      0
      ...
      7
      1180
      0
      1955
      0
      98178
      47.5112
      -122.257
      1340
      5650

      1
      6414100192
      20141209T000000
      538000.0
      3
      2.25
      2570
      7242
      2.0
      0
      0
      ...
      7
      2170
      400
      1951
      1991
      98125
      47.7210
      -122.319
      1690
      7639

      2
      5631500400
      20150225T000000
      180000.0
      2
      1.00
      770
      10000
      1.0
      0
      0
      ...
      6
      770
      0
      1933
      0
      98028
      47.7379
      -122.233
      2720
      8062

      3
      2487200875
      20141209T000000
      604000.0
      4
      3.00
      1960
      5000
      1.0
      0
      0
      ...
      7
      1050
      910
      1965
      0
      98136
      47.5208
      -122.393
      1360
      5000

      4
      1954400510
      20150218T000000
      510000.0
      3
      2.00
      1680
      8080
      1.0
      0
      0
      ...
      8
      1680
      0
      1987
      0
      98074
      47.6168
      -122.045
      1800
      7503

      5
      7237550310
      20140512T000000
      1225000.0
      4
      4.50
      5420
      101930
      1.0
      0
      0
      ...
      11
      3890
      1530
      2001
      0
      98053
      47.6561
      -122.005
      4760
      101930

      6
      1321400060
      20140627T000000
      257500.0
      3
      2.25
      1715
      6819
      2.0
      0
      0
      ...
      7
      1715
      0
      1995
      0
      98003
      47.3097
      -122.327
      2238
      6819

      7
      2008000270
      20150115T000000
      291850.0
      3
      1.50
      1060
      9711
      1.0
      0
      0
      ...
      7
      1060
      0
      1963
      0
      98198
      47.4095
      -122.315
      1650
      9711

      8
      2414600126
      20150415T000000
      229500.0
      3
      1.00
      1780
      7470
      1.0
      0
      0
      ...
      7
      1050
      730
      1960
      0
      98146
      47.5123
      -122.337
      1780
      8113

      9
      3793500160
      20150312T000000
      323000.0
      3
      2.50
      1890
      6560
      2.0
      0
      0
      ...
      7
      1890
      0
      2003
      0
      98038
      47.3684
      -122.031
      2390
      7570

10 rows × 21 columns

# Split data into training and testing

```python
from sklearn.model_selection import train_test_split
# Split data set into 80% train and 20% test data
train_data, test_data = train_test_split(data, test_size=0.2, random_state=0)
# Look at the shape to check the split ratio
train_data.shape, test_data.shape
```

    ((17290, 21), (4323, 21))

# Build a generic simple linear regression function

Let's build a closed form solution (function) to compute the slope and intercept for a simple linear regression.

Sample regression:
\begin{equation}
\hat{y}_i = \hat{i} + \hat{s} x_i
\end{equation}

Sample intercept:
\begin{equation}
\hat{i} = \bar{y} – \hat{s} \bar{x}
\end{equation}

Sample slope:
\begin{equation}
\hat{s} = \frac{\sum(x_i – \bar{x}) (y_i – \bar{y})} {\sum(x_i – \bar{x})^2}
\end{equation}

Here are the sample means:

\begin{equation}
\bar{x} = \frac{1}{n}\sum_{i=1}^n x_i
\end{equation}

\begin{equation}
\bar{y} = \frac{1}{n}\sum_{i=1}^n y_i
\end{equation}

```python
from time import time

X = train_data['price'].values
Y = train_data['sqft_living'].values
```

### What a Statistician would do:

```python
def simple_linear_regression(X, Y):

    # Compute the means:
    assert len(X) == len(Y)

    n = len(X)
    x_mean = sum(X) / n
    y_mean = sum(Y) / n

    # Compute the slope:
    slope = sum((X - x_mean) * (Y - y_mean)) / sum((X - x_mean)**2)

    # Compute the intercept:
    intercept = y_mean - slope * x_mean

    return (intercept, slope)
```

```python
t0 = time()
sqft_intercept, sqft_slope = simple_linear_regression(X, Y)
t1 = time()

print('Elapsed time: {:.2f} milliseconds\n'.format((t1 - t0) * 1000))
print('Intercept: ' + str(sqft_intercept))
print('Slope: ' + str(sqft_slope))
```

    Elapsed time: 5.48 milliseconds

    Intercept: 1134.23538705004
    Slope: 0.0017449919139801982

### What a Machine Learning Engineer would do:

```python
from sklearn.linear_model import LinearRegression

def simple_linear_regression_sklearn(X, Y):
    X_2D = X.reshape(-1, 1)  # Convert to 2D array for Sklearn
    model = LinearRegression().fit(X_2D, Y)
    return (model.intercept_, model.coef_[0])
```

```python
t0 = time()
sqft_intercept, sqft_slope = simple_linear_regression_sklearn(X, Y)
t1 = time()

print('Elapsed time: {:.2f} milliseconds\n'.format((t1 - t0) * 1000))
print('Intercept: ' + str(sqft_intercept))
print('Slope: ' + str(sqft_slope))
```

    Elapsed time: 1.36 milliseconds

    Intercept: 1134.235387050036
    Slope: 0.0017449919139802053

### What a Computer Scientist would do:

```python
def simple_linear_regression_optimized(input_feature, output):

    n = len(input_feature)

    # compute the sum of input_feature and output
    x_sum = input_feature.sum()
    y_sum = output.sum()

    # compute the product of the output and the input_feature and its sum
    xy = input_feature * output
    xy_sum = xy.sum()

    # compute the squared value of the input_feature and its sum
    x_squared = input_feature * input_feature
    x_squared_sum = x_squared.sum()

    # use the formula for the slope
    slope = (xy_sum - x_sum * y_sum / n) / (x_squared_sum - x_sum * x_sum / n)

    # use the formula for the intercept
    intercept = y_sum / n - slope * x_sum / n

    return (intercept, slope)
```

```python
t0 = time()
sqft_intercept, sqft_slope = simple_linear_regression_optimized(X, Y)
t1 = time()

print('Elapsed time: {:.2f} milliseconds\n'.format((t1 - t0) * 1000))
print('Intercept: ' + str(sqft_intercept))
print('Slope: ' + str(sqft_slope))
```

    Elapsed time: 0.35 milliseconds

    Intercept: 1134.235387050036
    Slope: 0.001744991913980205

### Plot data and function

# Predicting values

Now let's use that function of a simple regression model for predicting prices based on sqft - rembember that we train on train_data!

With the model parameters 'intercept & slope' we can now write a function to return the predicted output - given the input_feature, slope and intercept:

```python
def get_regression_predictions(input_feature, intercept, slope):
    # calculate the predicted values:
    predicted_values = intercept + slope * input_feature
    return predicted_values
```

```python
import matplotlib.pyplot as plt
import numpy as np

def plot_function(intercept, slope):
    axes = plt.gca() # get current axes
    x_vals = np.array(axes.get_xlim())
    y_vals = get_regression_predictions(x_vals, intercept, slope)
    plt.plot(x_vals, y_vals, 'r-')
```

```python
# Plot feature relations
plt.figure(figsize=(5,10))

plt.plot(X, Y, 'o')
plt.title('price vs. sqft_living')
plt.xlabel('sqft_living')
plt.ylabel('price')

plot_function(sqft_intercept, sqft_slope)
```

# Additional Task: Go to your 2a insurance notebook from yesterday and fit a regression line into your plot of bmi vs. expenses

Now that we have this function let's make a prediction.

* What is the estimated expense for an entry according to the model we estiamted above?

```python
import locale
locale.setlocale(locale.LC_ALL, 'en_US')

my_budget = 1000000
estimated_sqft_living = get_regression_predictions(my_budget, sqft_intercept, sqft_slope)

my_budget_currency = locale.currency(my_budget, grouping=True)
print("The estimated sqft_living for a budget of {} is {:.2f}".format(my_budget_currency, estimated_sqft_living))
```

    The estimated sqft_living for a budget of $1,000,000.00 is 2879.23

# Inverse Regression Prediction

The inverse process is calculated by solving `output = intercept + slope * input_feature` for `input_feature`.

```python
def inverse_regression_predictions(output, intercept, slope):
    # Use this equation to compute the inverse predictions:
    estimated_feature = (output - intercept) / slope
    return estimated_feature
```

```python
my_sqft = estimated_sqft_living
estimated_budget = inverse_regression_predictions(my_sqft, sqft_intercept, sqft_slope)

estimated_budget_currency = locale.currency(my_budget, grouping=True)
print("The estimated budget for a living of {:.2f} sqft is {}".format(estimated_sqft_living, estimated_budget_currency))
```

    The estimated budget for a living of 2879.23 sqft is $1,000,000.00

Train an inverse model that predicts the budget based on the sqft_livingroom:

```python
budget_intercept, budget_slope = simple_linear_regression(Y, X)

print('Intercept: ' + str(budget_intercept))
print('Slope: ' + str(budget_slope))
```

    Intercept: -48257.06359102961
    Slope: 283.96855715513044

```python
# Plot feature relations
plt.figure(figsize=(10,5))

plt.plot(Y, X, 'o')
plt.title('price vs. sqft_living')
plt.xlabel('sqft_living')
plt.ylabel('price')

plot_function(budget_intercept, budget_slope)
```

Compare the output of the inverse model with an inverse prediction, against the first model:

```python
my_budget_2 = 1000000
estimated_sqft_living_2 = inverse_regression_predictions(my_budget_2, budget_intercept, budget_slope)

my_budget_2_currency = locale.currency(my_budget_2, grouping=True)
print("The estimated sqft_living for a budget of {} is {:.2f}".format(my_budget_2_currency, estimated_sqft_living_2))
```

    The estimated sqft_living for a budget of $1,000,000.00 is 3691.45

The short answer is [here](https://stats.stackexchange.com/a/20560):

```python
plt.figure(figsize=(15,10))
plt.imshow(plt.imread('regression.png'))
```

The long answer is [here](https://stats.stackexchange.com/a/20556):

Given $n$ data points $(x_i,y_i), i = 1,2,\ldots n$, in the plane, let us draw a straight line $y = ax+b$.  If we predict $ax_i+b$ as the value $\hat{y}_i$ of $y_i$, then the _error_ is $(y_i-\hat{y}_i) = (y_i-ax_i-b)$, the _squared error_ is $(y_i-ax_i-b)^2$, and the _total squared error_ $\sum_{i=1}^n (y_i-ax_i-b)^2$. We ask

>What choice of $a$ and $b$ minimizes
$S =\displaystyle\sum_{i=1}^n (y_i-ax_i-b)^2$ ?

Since $(y_i-ax_i-b)$ is the vertical distance of $(x_i,y_i)$ from the straight line, we are asking for the line such that the sum of the squares of the vertical distances of the points from the line is as small as possible.  Now $S$  is a quadratic function of both $a$ and $b$ and attains its minimum value when $a$ and $b$ are such that

$$
\begin{align*}
\frac{\partial S}{\partial a} &= 2\sum_{i=1}^n (y_i-ax_i-b)(-x_i) &= 0\\
\frac{\partial S}{\partial b} &= 2\sum_{i=1}^n (y_i-ax_i-b)(-1) &= 0
\end{align*}
$$

From the second equation, we get

$$
b = \frac{1}{n}\sum_{i=1}^n (y_i - ax_i) = \mu_y - a\mu_x
$$

where $\displaystyle \mu_y = \frac{1}{n}\sum_{i=1}^n y_i, ~ \mu_x = \frac{1}{n}\sum_{i=1}^n x_i$ are the arithmetic average values of the $y_i$'s and the $x_i$'s respectively. Substituting into the first equation, we get

$$
a = \frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{
\left( \frac{1}{n}\sum_{i=1}^n x_i^2\right)  -\mu_x^2}.
$$

Thus, the line that minimizes $S$ can be expressed as

$$
y = ax+b = \mu_y +
\left(\frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{
\left( \frac{1}{n}\sum_{i=1}^n x_i^2\right)  -\mu_x^2}\right)
(x - \mu_x),
$$

and the minimum value of $S$ is

$$
S_{\min} =
\frac{\left[\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right)  -\mu_y^2\right]
\left[\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right)  -\mu_x^2\right]
-
\left[\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right)
-\mu_x\mu_y\right]^2}{\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right)  -\mu_x^2}.
$$

If we interchange the roles of $x$ and $y$, draw a line $x = \hat{a}y + \hat{b}$, and ask for the values of
$\hat{a}$ and $\hat{b}$ that minimize

$$
T = \sum_{i=1}^n (x_i - \hat{a}y_i - \hat{b})^2,
$$

that is, we want the line such that the sum of the squares of the _horizontal_ distances of the points from the line is as small as possible, then we get

$$
x = \hat{a}y+\hat{b} = \mu_x +
\left(\frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{
\left( \frac{1}{n}\sum_{i=1}^n y_i^2\right)  -\mu_y^2}\right)
(y - \mu_y)
$$

and the minimum value of $T$ is

$$
T_{\min} =
\frac{\left[\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right)  -\mu_y^2\right]
\left[\left(\frac{1}{n}\sum_{i=1}^n x_i^2\right)  -\mu_x^2\right]
-
\left[\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right)
-\mu_x\mu_y\right]^2}{\left(\frac{1}{n}\sum_{i=1}^n y_i^2\right)  -\mu_y^2}.
$$

Note that both lines pass through the point $(\mu_x,\mu_y)$ but the slopes are
$$
a =
\frac{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}{
\left( \frac{1}{n}\sum_{i=1}^n x_i^2\right)  -\mu_x^2},~~
\hat{a}^{-1} = \frac{
\left( \frac{1}{n}\sum_{i=1}^n y_i^2\right)  -\mu_y^2}{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y}
$$
are different in general. The slopes are the same when all the points $(x_i,y_i)$ lie on the same straight line.  To see this, note that

$$
\hat{a}^{-1} - a = \frac{S_{\min}}{\left(\frac{1}{n}\sum_{i=1}^n x_iy_i\right) -\mu_x\mu_y} = 0 \Rightarrow S_{\min} = 0 \Rightarrow y_i=ax_i+b, i=1,2,\ldots, n.
$$

```python

```


# The Discrete Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Fast Convolution

The linear convolution of signals is a basic building block in many practical applications. The straightforward convolution of two finite-length signals $x[k]$ and $h[k]$ has considerable numerical complexity. This has led to the development of various algorithms that realize the convolution with lower complexity. The basic concept of the *fast convolution* is to exploit the [convolution theorem](theorems.ipynb#Convolution-Theorem) of the discrete Fourier transform (DFT). This theorem states that the periodic convolution of two signals is equal to a scalar multiplication of their spectra. The scalar multiplication has considerably less numerical operations that the convolution. The transformation of the signals can be performed efficiently by the [fast Fourier transform](fast_fourier_transform.ipynb) (FFT).

Since the scalar multiplication of the spectra realizes a periodic convolution, special care has to be taken to realize a linear convolution in the spectral domain. The equivalence between linear and periodic convolution is discussed in the following.

### Equivalence of Linear and Periodic Convolution

The [linear convolution](../discrete_systems/linear_convolution.ipynb#Finite-Length-Signals) of a causal signal $x_L[k]$ of length $L$ with a causal signal $h_N[k]$ of length $N$ reads

\begin{equation}
y[k] = x_L[k] * h_N[k] = \sum_{\kappa = 0}^{L-1} x_L[\kappa] \; h_N[k - \kappa] = \sum_{\kappa = 0}^{N-1} h_N[\kappa] \; x_L[k - \kappa]
\end{equation}

The resulting signal $y[k]$ is of finite length $M = N+L-1$. Without loss of generality it is assumed in the following that $N \leq L$. The computation of $y[k]$ for $k=0,1, \dots, M-1$ requires $M \cdot N$ multiplications and $M \cdot (N-1)$ additions. The computational complexity of the convolution is consequently [on the order of](https://en.wikipedia.org/wiki/Big_O_notation) $\mathcal{O}(M \cdot N)$.

The periodic convolution of the two signals $x_L[k]$ and $h_N[k]$ is defined as

\begin{equation}
x_L[k] \circledast_P h_N[k] = \sum_{\kappa = 0}^{N-1} h_N[\kappa] \cdot \tilde{x}[k-\kappa]
\end{equation}

where $\tilde{x}[k]$ denotes the periodic summation of $x_L[k]$ with period $P$

\begin{equation}
\tilde{x}[k] = \sum_{\nu = -\infty}^{\infty} x_L[k - \nu P]
\end{equation}

The result of the circular convolution is periodic with period $P$. To compute the linear convolution by a periodic convolution, one has to take care that the result of the linear convolution fits into one period of the periodic convolution. Hence, the periodicity has to be chosen as $P \geq M$ where $M = N+L-1$. This can be achieved by zero-padding of $x_L[k]$ to the total length $M$ resulting in the signal $x_M[k]$ of length $M$ which is defined as

\begin{equation}
x_M[k] = \begin{cases}
x_L[k] & \text{for } 0 \leq k < L \\
0 & \text{for } L \leq k < M
\end{cases}
\end{equation}

and similar for $h_N[k]$ resulting in the zero-padded signal $h_M[k]$ which is defined as

\begin{equation}
h_M[k] = \begin{cases}
h_N[k] & \text{for } 0 \leq k < N \\
0 & \text{for } N \leq k < M
\end{cases}
\end{equation}

Using these signals, the linear and periodic convolution are equivalent for the first $M$ samples $k = 0,1,\dots, M-1$

\begin{equation}
x_L[k] * h_N[k] = x_M[k] \circledast_M h_M[k]
\end{equation}

#### Example

The following example computes the linear, periodic and linear by periodic convolution of two signals $x[k] = \text{rect}_L[k]$ and $h[k] = \text{rect}_N[k]$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from tools import cconv

L = 8  # length of signal x[k]
N = 10  # length of signal h[k]
P = 14 # periodicity of periodic convolution

# generate signals
x = np.ones(L)
h = np.ones(N)

# linear convolution
y1 = np.convolve(x, h, 'full')
# periodic convolution
y2 = cconv(x, h, P)
# linear convolution via periodic convolution
xp = np.append(x, np.zeros(N-1))
hp = np.append(h, np.zeros(L-1))
y3 = cconv(xp, hp, L+N-1)

# plot results
def plot_signal(x):
    plt.stem(x)
    plt.xlabel('$k$')
    plt.ylabel('$y[k]$')
    plt.xlim([0, N+L])
    plt.gca().margins(y=0.1)

plt.figure(figsize = (10, 8))
plt.subplot(3,1,1)
plot_signal(y1)
plt.title('Linear convolution')

plt.subplot(3,1,2)
plot_signal(y2)
plt.title('Periodic convolution with period $P=%d$'%P)

plt.subplot(3,1,3)
plot_signal(y3)
plt.title('Linear convolution as periodic convolution')
plt.tight_layout()
```

**Exercise**

* Change the lengths `L`, `N` and `P` and check how the results for the different convolutions change.

### The Fast Convolution Algorithm

Using the above derived equality of the linear and periodic convolution one can express the linear convolution $y[k] = x_L[k] * h_N[k]$ by the DFT as

$$ y[k] = \text{IDFT}_M \{ \; \text{DFT}_M\{ x_M[k] \} \cdot \text{DFT}_M\{ h_M[k] \} \; \} $$

The resulting algorithm is composed of the following steps

1. Zero-padding of the two input signals $x_L[k]$ and $h_N[k]$ to at least a total length of $M \geq N+L-1$

2. Computation of the DFTs $X[\mu]$ and $H[\mu]$ using a FFT of length $M$

3. Multiplication of the spectra $Y[\mu] = X[\mu] \cdot H[\mu]$

4. Inverse DFT of $Y[\mu]$ using an inverse FFT of length $M$

The algorithm requires two DFTs of length $M$, $M$ complex multiplications and one IDFT of length $M$. On first sight this does not seem to be an improvement, since one DFT/IDFT requires $M^2$ complex multiplications and $M \cdot (M-1)$ complex additions. The overall numerical complexity is hence in the order of $\mathcal{O}(M^2)$. The DFT can be realized efficiently by the [fast Fourier transformation](fast_fourier_transform.ipynb) (FFT), which lowers the number of numerical operations for each DFT/IDFT significantly. The actual gain depends on the particular implementation of the FFT. Many FFTs are most efficient for lengths which are a power of two. It therefore can make sense, in terms of the number of numerical operations, to choose $M$ as a power of two instead of the shortest possible length $N+L-1$. In this case, the numerical complexity of the radix-2 algorithm is on the order of $\mathcal{O}(M \log_2 M)$.

The introduced algorithm is known as *fast convolution* due to its computational efficiency when realized by the FFT. For real valued signals $x[k] \in \mathbb{R}$ and $h[k] \in \mathbb{R}$ the number of numerical operations can be reduced further by using a real valued FFT.

#### Example

The implementation of the fast convolution algorithm is straightforward. In the following example the fast convolution of two real-valued signals $x[k] = \text{rect}_L[k]$ and $h[k] = \text{rect}_N[k]$ is shown. The real valued FFT/IFFT is consequently used. Most implementations of the FFT include the zero-padding to a given length $M$, e.g as in `numpy` by `numpy.fft.rfft(x, M)`.

```python
L = 8  # length of signal x[k]
N = 10  # length of signal h[k]

# generate signals
x = np.ones(L)
h = np.ones(N)

# fast convolution
M = N+L-1
y = np.fft.irfft(np.fft.rfft(x, M)*np.fft.rfft(h, M))

# show result
plt.figure(figsize=(10, 3))
plt.stem(y)
plt.xlabel('k')
plt.ylabel('y[k]');
```

### Benchmark

It was already argued that the numerical complexity of the fast convolution is considerably lower due to the usage of the FFT. As measure, the gain in terms of execution time with respect to the linear convolution is evaluated in the following. Both algorithms are executed for the convolution of two real-valued signals $x_L[k]$ and $h_N[k]$ of length $L=N=2^n$ for $n \in \mathbb{N}$. The length of the FFTs/IFFT was chosen as $M=2^{n+1}$. The results depend heavily on the implementation of the FFT and the hardware used. Note that the execution of the following script may take some time.

```python
import timeit

n = np.arange(17)  # lengths = 2**n to evaluate
reps = 20  # number of repetitions for timeit

gain = np.zeros(len(n))
for N in n:
    length = 2**N
    # setup environment for timeit
    tsetup = 'import numpy as np; from numpy.fft import rfft, irfft; \
            x=np.random.randn(%d); h=np.random.randn(%d)' % (length, length)
    # direct convolution
    tc = timeit.timeit('np.convolve(x, x, "full")', setup=tsetup, number=reps)
    # fast convolution
    tf = timeit.timeit('irfft(rfft(x, %d) * rfft(h, %d))' % (2*length, 2*length), setup=tsetup, number=reps)
    # speedup by using the fast convolution
    gain[N] = tc/tf

# show the results
plt.figure(figsize = (15, 10))
plt.barh(n-.5, gain, log=True)
plt.plot([1, 1], [-1, n[-1]+1], 'r-')
plt.yticks(n, 2**n)
plt.xlabel('Gain of fast convolution')
plt.ylabel('Length of signals')
plt.title('Comparison of execution times between direct and fast convolution')
plt.grid()
```

**Exercise**

* For which lengths is the fast convolution faster than the linear convolution?
* Why is it slower below a given signal length?
* Is the trend of the gain as expected from above considerations?

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


```python
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
```

```python
import numpy as np
import numpy.linalg as la
```

# Dimension Reduction

```python
np.random.seed(123)
np.set_printoptions(3)
```

### PCA from scratch

Principal Components Analysis (PCA) basically means to find and rank all the eigenvalues and eigenvectors of a covariance matrix. This is useful because high-dimensional data (with $p$ features) may have nearly all their variation in a small number of dimensions $k$, i.e. in the subspace spanned by the eigenvectors of the covariance matrix that have the $k$ largest eigenvalues. If we project the original data into this subspace, we can have a dimension reduction (from $p$ to $k$) with hopefully little loss of information.

For zero-centered vectors,

\begin{align}
\text{Cov}(X, Y) &= \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1} \\
  &= \frac{\sum_{i=1}^nX_iY_i}{n-1} \\
  &= \frac{XY^T}{n-1}
\end{align}

and so the covariance matrix for a data set X that has zero mean in each feature vector is just $XX^T/(n-1)$.

In other words, we can also get the eigendecomposition of the covariance matrix from the positive semi-definite matrix $XX^T$.

We will take advantage of this when we cover the SVD later in the course.

Note: Here we are using a matrix of **row** vectors

```python
n = 100
x1, x2, x3 = np.random.normal(0, 10, (3, n))
x4 = x1 + np.random.normal(size=x1.shape)
x5 = (x1 + x2)/2 + np.random.normal(size=x1.shape)
x6 = (x1 + x2 + x3)/3 + np.random.normal(size=x1.shape)
```

#### For PCA calculations, each column is an observation

```python
xs = np.c_[x1, x2, x3, x4, x5, x6].T
```

```python
xs[:, :10]
```

#### Center each observation

```python
xc = xs - np.mean(xs, 1)[:, np.newaxis]
```

```python
xc[:, :10]
```

#### Covariance

Remember the formula for covariance

$$
\text{Cov}(X, Y) = \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1}
$$

where $\text{Cov}(X, X)$ is the sample variance of $X$.

```python
cov = (xc @ xc.T)/(n-1)
```

```python
cov
```

#### Check

```python
np.cov(xs)
```

#### Eigendecomposition

```python
e, v = la.eigh(cov)
```

```python
idx = np.argsort(e)[::-1]
```

```python
e = e[idx]
v = v[:, idx]
```

#### Explain the magnitude of the eigenvalues

Note that $x4, x5, x6$ are linear combinations of $x1, x2, x3$ with some added noise, and hence the last 3 eigenvalues are small.

```python
plt.stem(e)
pass
```

#### The eigenvalues and eigenvectors give a factorization of the covariance matrix

```python
v @ np.diag(e) @ v.T
```

### Geometry of PCA

### Algebra of PCA

Note that $Q^T X$ results in a new data set that is uncorrelated.

```python
m = np.zeros(2)
s = np.array([[1, 0.8], [0.8, 1]])
x = np.random.multivariate_normal(m, s, n).T
```

```python
x.shape
```

#### Calculate covariance matrix from centered observations

```python
xc = (x - x.mean(1)[:, np.newaxis])
```

```python
cov = (xc @ xc.T)/(n-1)
```

```python
cov
```

#### Find eigendecoposition

```python
e, v = la.eigh(cov)
idx = np.argsort(e)[::-1]
e = e[idx]
v = v[:, idx]
```

#### In original coordinates

```python
plt.scatter(x[0], x[1], alpha=0.5)
for e_, v_ in zip(e, v.T):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('x', fontsize=14)
plt.ylabel('y', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
pass
```

#### After change of basis

```python
yc = v.T @ xc
```

```python
plt.scatter(yc[0,:], yc[1,:], alpha=0.5)
for e_, v_ in zip(e, np.eye(2)):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('PC1', fontsize=14)
plt.ylabel('PC2', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
pass
```

#### Check

```python
from sklearn.decomposition import PCA
```

```python
pca = PCA()
```

#### Note that the PCA from scikit-learn works with feature vectors, not observation vectors

```python
z = pca.fit_transform(x.T)
```

#### Eigenvectors from PCA

```python
pca.components_
```

#### Eigenvalues from PCA

```python
pca.explained_variance_
```

#### Explained variance

This is just a consequence of the invariance of the trace under change of basis. Since the original diagnonal entries in the covariance matrix are the variances of the featrues, the sum of the eigenvalues must also be the sum of the orignal varainces. In other words, the cumulateive proportion of the top $n$ eigenvaluee is the "explained variance" of the first $n$ principal components.

```python
e/e.sum()
```

```python
pca.explained_variance_ratio_
```

#### The principal components are identical to our home-brew version, up to a flip in direction of eigenvectors

```python
plt.scatter(z[:, 0], z[:, 1], alpha=0.5)
for e_, v_ in zip(e, np.eye(2)):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('PC1', fontsize=14)
plt.ylabel('PC2', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
pass
```

```python
plt.subplot(121)
plt.scatter(-z[:, 0], -z[:, 1], alpha=0.5)
for e_, v_ in zip(e, np.eye(2)):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('PC1', fontsize=14)
plt.ylabel('PC2', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
plt.title('Scikit-learn PCA (flipped)')

plt.subplot(122)
plt.scatter(yc[0,:], yc[1,:], alpha=0.5)
for e_, v_ in zip(e, np.eye(2)):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xlabel('PC1', fontsize=14)
plt.ylabel('PC2', fontsize=14)
plt.axis('square')
plt.axis([-3,3,-3,3])
plt.title('Homebrew PCA')
plt.tight_layout()
pass
```


# Chapter 5. Applications

## 5.1 Linear Algebra

### 5.1.1 Solving a linear system

```python
import numpy as np
from numpy.linalg import solve

A = np.array([[5, -3, 2], [-3, 8, 4], [2, 4, -9]], dtype=float)
b = np.array([10, 20, 9], dtype=float)
x = solve(A, b)
print x
c = np.dot(A, x)
print c
```

    [ 3.44419134  3.19817768  1.18678815]
    [ 10.  20.   9.]

## 5.1.2 Gaussian elimination

```python
from sympy import symbols, Function, Matrix, init_printing

x, y, z, t = symbols('x y z t')          # Symbols representing real numbers
k, m, n = symbols('k m n', integer=True) # Symbols representing integers
f, g, h = symbols('f g h', cls=Function) # Symbols repesenting function names
init_printing()

A = np.array([[5, -3, 2], [-3, 8, 4], [2, 4, -9]], dtype=float)
b = np.array([[10], [20], [9]], dtype=float)
print A, b
C = np.concatenate((A, b), axis=1)
Cr = Matrix(C).rref()
print Cr
```

    [[ 5. -3.  2.]
     [-3.  8.  4.]
     [ 2.  4. -9.]] [[ 10.]
     [ 20.]
     [  9.]]
    (Matrix([
    [1.0,   0,   0, 3.44419134396355],
    [  0, 1.0,   0, 3.19817767653759],
    [  0,   0, 1.0, 1.18678815489749]]), [0, 1, 2])

### 5.1.3 Finding eigenvalues and eigenvectors

```python
A = np.array([[5, -3, 2], [-3, 8, 4], [4, 2, -9]], dtype=float)
[V, D] = np.linalg.eig(A)
print 'Eigenvalues\n', V
print 'Eigenvectors\n', D
```

    Eigenvalues
    [-10.22062687   4.42464106   9.7959858 ]
    Eigenvectors
    [[ 0.17254241  0.87060637 -0.53754162]
     [ 0.23822812  0.37738965  0.84287515]
     [-0.95575963  0.31562891 -0.02470826]]

### 5.1.4 Matrix factorizations

```python
A = np.array([[5, -3, 2], [-3, 8, 4], [4, 2, -9]], dtype=float)
[Q, R] = np.linalg.qr(A)
print R
```

    [[-7.07106781  4.38406204  5.37401154]
     [-0.         -7.60131568  2.04701405]
     [ 0.          0.         -8.24194962]]

```python
[U, D, V] = np.linalg.svd(A)
print 'U =\n', U
print 'D =\n', D
print 'V =\n', V
```

    U =
    [[ 0.19746661  0.4956575   0.84577218]
     [-0.68275873 -0.54956546  0.48147515]
     [ 0.70345395 -0.67253361  0.22989366]]
    D =
    [ 11.44271549   8.82037309   4.38922325]
    V =
    [[ 0.51119203 -0.40615899 -0.75744147]
     [ 0.16290121 -0.81953034  0.5493935 ]
     [ 0.84388737  0.40423371  0.35277359]]

```python
A = np.array([[5, -3, 2], [-3, 8, 4], [2, 4, 9]], dtype=float)
R = np.linalg.cholesky(A)
print R
```

    [[ 2.23606798  0.          0.        ]
     [-1.34164079  2.48997992  0.        ]
     [ 0.89442719  2.08837026  1.95926253]]

## 5.2 Curve Fitting and Interpolation

```python
import matplotlib.pyplot as plt
%matplotlib inline

x = np.array([5, 10, 20, 50, 100], dtype=float)
y = np.array([15, 33, 53, 140, 301], dtype=float)

p1 = np.polyfit(x, y, 1)
y1 = np.polyval(p1, x)

p2 = np.polyfit(x, y, 2)
y2 = np.polyval(p2, x)

p3 = np.polyfit(x, y, 3)
y3 = np.polyval(p3, x)

plt.plot(x, y, 'bo--', label='Raw Data')
plt.plot(x, y1, label='n = 1')
plt.plot(x, y2, label='n = 2')
plt.plot(x, y3, label='n = 3')
plt.grid()
plt.xlabel('x')
plt.ylabel('y')
plt.legend(loc=2)
plt.show()
```

## 5.4 Numerical Integration (Quadrature)

```python
import scipy.integrate as spint

def erfcousin(x):
    return np.exp(-x**2)

y = spint.quad(erfcousin, 1/2.0, 3/2.0)
print y
```

    (0.39490738721210866, 4.3843527387744516e-15)

### 5.4.1 Double integration

```python
def f(x, y):
    return 1.0 - 6.0 * x**2 * y

I = spint.dblquad(f, 0, 2.0, lambda x : -1, lambda x : 1)
print I
```

    (-3.9999999999999996, 5.55733536161384e-14)

## 5.5 Ordinary DIfferential Equations

```python
def simpmode(t, x):
    return x + t

t = np.arange(0, 2, 0.1)
x0 = 0
x = spint.odeint(simpmode, x0, t)

plt.plot(t, x)
plt.grid()
plt.show()
```

## 5.6 Nonlinear Algebraic Equations

Solve the following transcendental equation
$$\sin(x) = e^x - 5$$
Write the equation in standard form:
$$\sin(x) - e^x + 5 \Rightarrow f(x) = \sin(x) - e^x + 5$$

```python
import scipy.optimize as spopt

def transf(x):
    return np.sin(x) - np.exp(x) + 5

r = spopt.newton(transf, 1)
print r, transf(r)
```

    1.78784147462 1.01252339846e-13

spopt.root(transf, 1)
### 5.6.1 Roots of polynomials

Polynomial equation
$$x^5 - 3x^3 + x^2 - 9 = 0$$
The coefficients are
$$(1) x^5 + (0) x^4 + (-3) x^3 + (1) x^2 + (0) x + (-9) = 0$$

```python
p = np.poly1d([1, 0, -3, 1, 0, -9.0])
print p.r
```

    [ 1.93160185+0.j          0.58983307+1.19342825j  0.58983307-1.19342825j
     -1.55563399+0.4573533j  -1.55563399-0.4573533j ]

```python
print np.roots(p)
```

    [ 1.93160185+0.j          0.58983307+1.19342825j  0.58983307-1.19342825j
     -1.55563399+0.4573533j  -1.55563399-0.4573533j ]

```python

```


In this notebook we fetch one week of Swarm data from VirES and perform a basic spherical harmonic analysis to retrieve a main field model.

We use [`design_gauss()`](https://github.com/ancklo/ChaosMagPy/blob/6c1077f901d7827a355da366a4df952f7e602681/chaosmagpy/model_utils.py#L496) from [`chaosmagpy`](https://github.com/ancklo/ChaosMagPy) v0.1 (Clemens Kloss)

```python
import datetime as dt
import numpy as np
import matplotlib.pyplot as plt
from viresclient import SwarmRequest
import chaosmagpy as cp
```

# Fetch filtered data as an xarray Dataset

... explain subsampling and data selection ("filters") ...

```python
request = SwarmRequest()

request.set_collection("SW_OPER_MAGA_LR_1B")
request.set_products(
    measurements=["B_NEC"],
    sampling_step="PT10S"
)
request.set_range_filter('Kp', 0, 3)

ds = request.get_between(
    start_time=dt.datetime(2019,2,1),
    end_time=dt.datetime(2019,2,8)
).as_xarray()
ds
```

    [1/1] Processing:  100%|██████████|  [ Elapsed: 00:01, Remaining: 00:00 ]
          Downloading: 100%|██████████|  [ Elapsed: 00:00, Remaining: 00:00 ] (2.966MB)

    <xarray.Dataset>
    Dimensions:     (Timestamp: 51839, dim: 3)
    Coordinates:
      * Timestamp   (Timestamp) datetime64[ns] 2019-02-01T00:00:10 ... 2019-02-07T23:59:50
    Dimensions without coordinates: dim
    Data variables:
        Spacecraft  (Timestamp) <U1 'A' 'A' 'A' 'A' 'A' 'A' ... 'A' 'A' 'A' 'A' 'A'
        Latitude    (Timestamp) float64 45.32 45.96 46.61 47.25 ... 34.4 35.04 35.69
        Longitude   (Timestamp) float64 -174.4 -174.4 -174.4 ... 175.4 175.4 175.4
        Radius      (Timestamp) float64 6.809e+06 6.808e+06 ... 6.81e+06 6.81e+06
        B_NEC       (Timestamp, dim) float64 1.913e+04 2.076e+03 ... 2.532e+04
    Attributes:
        Sources:         ['SW_OPER_AUX_KP__2__19980101T013000_20190824T073000_000...
        MagneticModels:  []
        RangeFilters:    ['Kp:0,3']

# Inspect data plots... (tbc)

```python
ds["B_NEC"][:, 2].plot();
```

# Converting between B_NEC and (B$_r$, B$_\theta$, B$_\phi$)

B_NEC contains components (North, East, Centre), i.e. pointing Northwards, Eastwards, and radially downwards.

B$_r$ points radially upwards. B$_\theta$ points in the direction of increasing colatitude ($\theta$), i.e. Southwards. B$_\phi$ points in the direction of increasing longitude ($\phi$), i.e. Eastward.

So B$_r$, B$_\theta$, B$_\phi$ = `-ds["B_NEC"][:, 2]`, `-ds["B_NEC"][:, 0]`, `ds["B_NEC"][:, 1]`

```python
B_radius, B_theta, B_phi = -ds["B_NEC"][:, 2], -ds["B_NEC"][:, 0], ds["B_NEC"][:, 1]
```

# Build Gauss design matrix

```python
nmax = 13

A_radius, A_theta, A_phi = cp.model_utils.design_gauss(
    ds["Radius"]/1e3,  # Input must be in km
    90-ds["Latitude"], # as colatitude
    ds["Longitude"],
    nmax
)
```

```python
G = np.vstack((A_radius, A_theta, A_phi))
G.shape
```

    (155517, 195)

```python
d = np.hstack((B_radius, B_theta, B_phi))
d.shape
```

    (155517,)

# Least squares inversion

We want to solve for $m$:
$$
\begin{align}
 d     &= G m \\
 G^T d &= G^T G m \\
 m     &= (G^T G)^{-1} G^T d
\end{align}
$$

```python
gauss_coeffs = np.linalg.inv(G.T @ G) @ (G.T @ d)
```

alternatively:

```python
gauss_coeffs = np.linalg.lstsq(G, d, rcond=None)[0]
```

# Power spectrum

```python
power_spec = cp.model_utils.power_spectrum(gauss_coeffs)
fig, ax = plt.subplots(1, 1)
ax.semilogy(np.arange(1,14), power_spec, 'b-', label="My model")
ax.set_xlabel("SH degree")
ax.set_ylabel("Power")
ax.set_title("Mauersberger–Lowes spectrum")
ax.legend()
ax.grid();
```

# Next: evaluation and plotting components, ... ?


## First Assignment

#### 1) Apply the appropriate string methods to the **x** variable (as '.upper') to change it exactly to: "$Dichlorodiphenyltrichloroethane$".

```python
x = "DiClOrod  IFeNi  lTRicLOr oETaNo  DiChlorod iPHeny lTrichL oroEThaNe"
```

```python
# first try. there might be a way where I don't need to create now variables for each step?
xnew = x.replace(" ", "")
xnew_stripped = xnew[27:]
xnew_stripped_lower = xnew_stripped.lower()
xfinal = xnew_stripped.capitalize()
print(xfinal)
```

    Dichlorodiphenyltrichloroethane

```python
#other solution presented in class(not my own work):
y = "DiClOrod  IFeNi  lTRicLOr oETaNo  DiChlorod iPHeny lTrichL oroEThaNe"
y.replace(" ", "")[27: ].title()

```

    'Dichlorodiphenyltrichloroethane'

#### 2) Assign respectively the values: 'word', 15, 3.14 and 'list' to variables A, B, C and D in a single line of code. Then, print them in that same order on a single line separated by a space, using only one print statement.

```python
#so second try with tuple:
A, B, C, D = "word", 15, 3.14, "list"
print(A, B, C, D)
```

    word 15 3.14 list

#### 3) Use the **input()** function to receive an input in the form **'68.4 1.71'**, that is, two floating point numbers in a line separated by space. Then, assign these numbers to the variables **w** and **h** respectively, which represent an individual's weight and height (hint: take a look at the '.split()' method). With this data, calculate the individual's Body Mass Index (BMI) from the following relationship:

\begin{equation}
BMI = \dfrac{weight}{height^2}
\end{equation}

```python
x,y = input("Enter your weight in kg and heigt in m separated by a space").split()
bmi = float(x)/float(y)**2
print("This is your BMI:", bmi)
```

#### This value can also be classified according to ranges of values, following to the table below. Use conditional structures to classify and print the classification assigned to the individual.

<\center>

(source: https://healthtravelguide.com/bmi-calculator/)

```python
#x,y = input("Enter your weight in kg and heigt in m separated by a space").split()
#bmi = float(x)/float(y)**2
#print("This is your BMI:", bmi)

if (bmi < 18.5):
        print ("Your nutritional status is 'Underweight'")
elif (bmi >= 18.5) and (bmi <= 24.9):
         print ("Your nutritional status is 'Normal weight'")
elif (bmi >= 25.0) and (bmi <= 29.9):
         print("Your nutritional status is 'Pre-obesity'")
elif (bmi >= 30.0) and (bmi <= 34,9):
        print("Your nutritional status is 'Obersity class I'")
elif (bmi >= 35.0) and (bmi <= 39,9):
        print("Your nutritional status is 'Obersity class II'")
elif (bmi >= 40):
        print("Your nutritional status is 'Obersity class III'")
else:
        print("your BMI is to low to calculate")
```

#### 4) Receive an integer as an input and, using a loop, calculate the factorial of this number, that is, the product of all the integers from one to the number provided.

```python
value = int(input())
factorial = 1
for number in range(1,value+1):
    factorian = factorial * number
print(factorial)
```

     3

    1

#### 5) Using a while loop and the input function, read an indefinite number of integers until the number read is -1. Present the sum of all these numbers in the form of a print, excluding the -1 read at the end.

```python
num = 0
sum_num = 0

while num != -1:
    sum_num = sum_num + num
    num = int(input())
print(f"Sum= {sum_num}")
```

     3
     4
     5
     1
     -1

    Sum= 13

#### 6) Read the **first name** of an employee, his **amount of hours worked** and the **salary per hour** in a single line separated by commas. Next, calculate the **total salary** for this employee and show it to two decimal places.

```python
#first try. only one decimal place
x = ("Nora", 40, 10)
(name, hours, salery) = x
name = x[0]
hours = x[1]
salery= x[2]

salerytotal = float(hours*salery)
print(salerytotal)
```

    400.0

#### 7) Read three floating point values **A**, **B** and **C** respectively. Then calculate itens a, b, c, d and e:

```python
numbers = input("Select three numbers: ").split()
A = int(numbers[0])
B = int(numbers[1])
C = int(numbers[2])
```

    Select three numbers:  1 3 4

    a) the area of the triangle rectangle with A as the base and C as the height.

```python
#A=h*b/2

area_triangle = (A*C)/2
print(area_triangle)
```

    2.0

    b) the area of the circle of radius C. (pi = 3.14159)

```python
#A=h*b/2

area_triangle = (A*C)/2
print(area_triangle)
```

    2.0

    c) the area of the trapezoid that has A and B for bases and C for height.

```python
#A=pi*r^2

area_circle = 3.14159*C**2
print(area_circle)
```

    50.26544

    d) the area of the square that has side B.

```python
#A = (a+b/2)*h

area_trapezoid = ((A+B)/2)*C
print(area_trapezoid)
```

    8.0

    e) the area of the rectangle that has sides A and B.

```python
#A = wl
area_rectangle = A*B
print(area_rectangle)
```

    3

#### 8) Read **the values a, b and c** and calculate the **roots of the second degree equation** $ax^{2}+bx+c=0$ using [this formula](https://en.wikipedia.org/wiki/Quadratic_equation). If it is not possible to calculate the roots, display the message **“There are no real roots”**.

```python
numbers = input("Select three numbers: ").split()
a = int(numbers[0])
b = int(numbers[1])
c = int(numbers[2])

# Python program to find roots of quadratic equation
import math

# function for finding roots
def roots( a, b, c):

    # calculating discriminant using formula
    dis = b * b - 4 * a * c
    sqrt = math.sqrt(abs(dis))

    # checking condition for discriminant
    if dis > 0:
        print(" real and different roots ")
        print((-b + sqrt)/(2 * a))
        print((-b - sqrt)/(2 * a))

    elif dis == 0:
        print(" real and equal roots")
        print(-b / (2 * a))

    # when discriminant is less than 0
    else:
        print("There are no real roots")

roots(a, b, c)
```

    Select three numbers:  3 5 6

    There are no real roots

#### 9) Read four floating point numerical values corresponding to the coordinates of two geographical coordinates in the cartesian plane. Each point will come in a line with its coordinates separated by space. Then calculate and show the distance between these two points.

(obs: $d=\sqrt{(x_1-x_2)^2 + (y_1-y_2)^2}$)

```python
numbers = input("Select four values for two coordinates: ").split()
x1 = float(numbers[0])
y1 = float(numbers[1])
x2 = float(numbers[2])
y2 = float(numbers[3])

d = math.sqrt((x1 - x2)**2 + (y1-y2)**2)
print(d)
```

    Select four values for two coordinates:  2 3 4 5

    2.8284271247461903

#### 10) Read **two floating point numbers** on a line that represent **coordinates of a cartesian point**. With this, use **conditional structures** to determine if you are at the origin, printing the message **'origin'**; in one of the axes, printing **'x axis'** or **'y axis'**; or in one of the four quadrants, printing **'q1'**, **'q2**', **'q3'** or **'q4'**.

```python
numbers = input("Select two numbers: ").split()
x = float(numbers[0])
y = float(numbers[1])

#check for quadrants:
if x > 0 and y > 0:
    print("q1")
elif x < 0 and y > 0:
    print("q2")
elif x < 0 and y < 0:
    print("q3")
elif x > 0 and y < 0:
    print("q4")
elif x == 0 and y ==0:
    print("origin")
elif x and y == 0:
    print ("origin")
elif x == 0 and y != 0:
    print("x axis")
elif x != 0 and y == 0:
    print("y axis")
```

    Select two numbers:  1 2

    q1

#### 11) Read an integer that represents a phone code for international dialing.
#### Then, inform to which country the code belongs to, considering the generated table below:
(You just need to consider the first 10 entries)

```python
import pandas as pd
df = pd.read_html('https://en.wikipedia.org/wiki/Telephone_numbers_in_Europe')[1]
df = df.iloc[:,:2]
df.head(20)
```

      Country
      Country calling code

      0
      Austria
      43

      1
      Belgium
      32

      2
      Bulgaria
      359

      3
      Croatia
      385

      4
      Cyprus
      357

      5
      Czech Republic
      420

      6
      Denmark
      45

      7
      Estonia
      372

      8
      Finland
      358

      9
      France
      33

      10
      Germany
      49

      11
      Greece
      30

      12
      Hungary
      36

      13
      Iceland
      354

      14
      Ireland
      353

      15
      Italy
      39

      16
      Latvia
      371

      17
      Liechtenstein
      423

      18
      Lithuania
      370

      19
      Luxembourg
      352

```python
some_value = input("Specify the phone code: ")
my_subset = df["Country calling code"] == some_value
country = df.loc[my_subset]["Country"].values.tolist()[0]
print(country)
```

    Specify the phone code:  43

    Austria

#### 12) Write a piece of code that reads 6 numbers in a row. Next, show the number of positive values entered. On the next line, print the average of the values to one decimal place.

```python
numbers = [float(y) for y in input("Give me six values separates by space").split()]
sum_numbers = 0
positiv = 0
for i in numbers:
    sum_numbers = sum_numbers + i
    if i > 0:
        positiv = positiv +1
print(f"Count of positives: {positiv}")
print(f"Average: {sum_numbers/6}")
```

    Give me six values separates by space 1 4 5 -2 4 3

    Count of positives: 5
    Average: 2.5

#### 13) Read an integer **N**. Then print the **square of each of the even values**, from 1 to N, including N, if applicable, arranged one per line.

```python
n = input("Give me an integer: ")
N = int(n[0])
for x in range(1, N+1):
    if x%2 == 0:
        print(x**2)
```

    Give me an integer:  8

    4
    16
    36
    64

#### 14) Using **input()**, read an integer and print its classification as **'even / odd'** and **'positive / negative'** . The two classes for the number must be printed on the same line separated by a space. In the case of zero, print only **'null'**.

```python
#first try.
def even_odd(num):
    num_int = input("Provide a numeric: ")
    num = int(num_int)
    if num%2 == 0 and num > 0:
        return "even positive"
    if num%2 != 0 and num > 0:
        return "odd positive"
    if num%2 == 0 and num < 0:
        return "even negative"
    if num%2 != 0 and num < 0:
        return "odd negative"
    if num == 0:
        return "null"

even_odd(x)
```

    Provide a numeric:  7

    'odd positive'

```python
# now with classes, but needed google to figure out the tenary operation.
num = int(input("Provide a numeric: "))
group1 = "even" if num%2 == 0 else "odd"
group2 = "positive" if num > 0 else "negative"
if number == 0:
    print ("null")
else:
    print(group1, group2)

```

    Provide a numeric:  7

    odd positive

## Challenge
#### 15) Ordering problems are recurrent in the history of programming. Over time, several algorithms have been developed to fulfill this function. The simplest of these algorithms is the [**Bubble Sort**](https://en.wikipedia.org/wiki/Bubble_sort), which is based on comparisons of elements two by two in a loop of passes through the elements. Your mission, if you decide to accept it, will be to input six whole numbers ramdonly ordered. Then implement the **Bubble Sort** principle to order these six numbers **using only loops and conditionals**.
#### At the end, print the six numbers in ascending order on a single line separated by spaces.

```python
#does not work...
def bubbelsort(array):
    n = len(array)
    for i in range(n -1):
        for j in (0, n - i - 1):
            if array[j] > array[j + 1] :
                array[j], array[j + 1] = array[j + 1], array[j]
    print(array.sort(reverse = Treu))

array = [4, 2, 5, 6, 8, 1]
bubbelsort(array)
```

```python

```


## [Problem 51](https://projecteuler.net/problem=51)
Prime digit replacements

```python
from sympy import sieve
import numpy as np
```

```python
N=6
primes=[i for i in sieve.primerange(10**(N-1),10**N)]
#1を3回使うものの桁
diffs=[(k,j,i) for i in range(1,N) for j in range(i+1,N) for k in range(j+1,N)]
```

```python
intersections=[]
for diff_kji in diffs:
    k,j,i=diff_kji
    diff=10**k+10**j+10**i
    prime_list=[_i for _i in primes if str(_i)[N-k-1]==str(_i)[N-j-1]==str(_i)[N-i-1]]
    prime_set=set(prime_list)
    prime_array=np.array(prime_list)
    #3連続は絶対にあるはず
    #intersection=[[{_i,_i+diff,_i+2*diff},[int(_k) for counter,_k in enumerate(str(_i)) if 6-counter-1 not in {k,j,i}]] for _i in prime_list if {_i,_i+diff,_i+2*diff} <= prime_set]
    ketas=[N-counter-1 for counter in range(N) if N-counter-1 not in {k,j,i}]
    intersection=[[int(_k) for counter,_k in enumerate(str(_i)) if N-counter-1 not in {k,j,i}] for _i in prime_list if {_i,_i+diff,_i+2*diff} <= prime_set]
    possibles=[10**ketas[0]*_i[0]+10**ketas[1]*_i[1]+10**ketas[2]*_i[2] for _i in intersection]
    possibles=[set([possible+_i*diff for _i in range(10)]) for possible in possibles]
    for p in possibles:
        if len(p & prime_set)==8:
            print(p & prime_set)
```

    {121313, 323333, 525353, 929393, 222323, 424343, 626363, 828383}
    {121313, 323333, 525353, 929393, 222323, 424343, 626363, 828383}
    {121313, 323333, 525353, 929393, 222323, 424343, 626363, 828383}
    {121313, 323333, 525353, 929393, 222323, 424343, 626363, 828383}

## [Problem 53](https://projecteuler.net/problem=53)
Combinatoric selections

```python
import numpy as np
```

```python
dp=[[1]]
for n in range(1,101):
    l=[1]
    for r in range(1,n):
        l.append(dp[n-1][r-1]+dp[n-1][r])
    l.append(1)
    dp.append(l.copy())
```

```python
ans_list=[1 for l in dp for x in l if x>10**6]
print(len(ans_list))
```

    4075

```python

```


# Characterization of Systems in the Spectral Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Bode Plots

The Fourier domain transfer function $H(j \omega)$ characterizes the transmission properties of a linear time-incariant (LTI) system with respect to an [harmonic exponential signal](../continuous_signals/standard_signals.ipynb#Complex-Exponential-Signal) $e^{j \omega t}$ with angular frequency $\omega = 2 \pi f$. In order to illustrate the characteristics of an LTI system, often the magnitude $| H(j \omega) |$ and phase $\varphi_H(j \omega)$ of the transfer function is regarded separately. Decomposing the output signal $Y(j \omega) = X(j \omega) \cdot H(j \omega)$ into its magnitude $| Y(j \omega) |$ and phase $\varphi_Y(j \omega)$ yields

\begin{align}
| Y(j \omega) | &= | X(j \omega) | \cdot | H(j \omega) | \\
\varphi_Y(j \omega) &= \varphi_X(j \omega) + \varphi_H(j \omega)
\end{align}

where $X(j \omega)$ denotes the input signal, and $| X(j \omega) |$ and $\varphi_X(j \omega)$ its magnitude and phase, respectively. It can be concluded from above equations, that the magnitude $| H(j \omega) |$ provides the frequency-dependent attenuation of the magnitude $| X(j \omega) |$ of the input signal by the system, while $\varphi_H(j \omega)$ provides the introduced phase-shift.

A commonly used graphical illustration of the system properties in terms of the magnitude and phase of the transfer function are [*bode plots*](https://en.wikipedia.org/wiki/Bode_plot). Here the logarithmic magnitude of the transfer function $20 \log_{10} | H(j \omega) |$ in [decibels](https://en.wikipedia.org/wiki/Decibel) is plotted against the logarithm of the frequency $\omega$ or $f$. The phase $\varphi_H(j \omega)$ is plotted linearly against the logarithm of the frequency. A transfer function $H(s)$ which is a rational function in $s$ can be represented [in terms of its poles and zeros](../laplace_transform/definition.ipynb#Representation). Applying this representation to the transfer function $H(j \omega)$ in the Fourier domain yields

\begin{equation}
H(j \omega) = K \cdot \frac{\prod_{\mu=0}^{Q} (j \omega - s_{0 \mu})}{\prod_{\nu=0}^{P} (j \omega - s_{\infty \nu})}
\end{equation}

where $s_{0 \mu}$ and $s_{\infty \nu}$ denote the $\mu$-th zero and $\nu$-th pole of $H(s)$, and $Q$ and $P$ the total number of zeros and poles, respectively. The logarithm of the magnitude and the phase can then be expressed as

\begin{align}
\log_{10} | H(j \omega) | &= \sum_{\mu=0}^{Q} \log_{10} |j \omega - s_{0 \mu}| - \sum_{\nu=0}^{P} \log_{10} |j \omega - s_{\infty \nu}| + \log_{10} |K|   \\
\varphi_H(j \omega) &= \sum_{\mu=0}^{Q} \arg (j \omega - s_{0 \mu}) - \sum_{\nu=0}^{P} \arg (j \omega - s_{\infty \nu})
\end{align}

where $\arg(\cdot)$ denotes the [argument](https://en.wikipedia.org/wiki/Argument_%28complex_analysis%29) (phase) of a complex function. It can be concluded from above result, that the individual contributions of the poles and zeros to the bode plot can be superimposed. This fact can be exploited to sketch approximated bode plots illustrating the properties of an LTI system for a given set of poles and zeros. Rules for the asymptotic behavior of the magnitude and phase for single poles and zeros are available in the [literature](https://en.wikipedia.org/wiki/Bode_plot#Rules_for_handmade_Bode_plot). For instance it is stated that a single pole/zero results in a magnitude response in the bode plot that decreases/increases with a slope of 20 dB per decade. These rules provide insights into the influence of the location and order of poles and zeros on the properties of systems. They are of benefit when designing systems with given magnitude and phase properties.

**Example - Second order low-pass filter**

The bode plot of the [2nd order low-pass filter](../laplace_transform/network_analysis.ipynb#Example:-Second-Order-Low-Pass-Filter) is plotted. The transfer function $H(j \omega)$ of the low-pass filter can be derived from its Laplace domain counterpart $H(s)$ by

\begin{equation}
H(j \omega) = \frac{1}{C L s^2 + C R s + 1} \bigg\rvert_{s = j \omega}
\end{equation}

```python
import sympy as sym
%matplotlib inline
sym.init_printing()

s = sym.symbols('s')
w = sym.symbols('omega', real=True)
R, L, C = sym.symbols('R L C', positive=True)

H = 1/(C*L*s**2 + C*R*s + 1)
H = H.subs(s, sym.I * w)
H
```

The bode plot is generated for the normalized values $R = 1$, $L = 0.5$ and $C = 0.4$

```python
import matplotlib.pyplot as plt
from sympy.plotting.plot import MatplotlibBackend

def logx_plot(Hw, wrange, **kwargs):
    'create plot with logarithmic x-axis'
    p = sym.plot(Hw, wrange, show=False, **kwargs)
    backend = MatplotlibBackend(p)
    backend.process_series()
    backend.ax[0].spines['left'].set_position(('axes', 0))
    backend.ax[0].spines['bottom'].set_position(('axes', 0))
    plt.xscale('log')
    plt.grid(which='both')

def db(x):
    'compute dB value'
    return 20 * sym.log(sym.Abs(x), 10)

RLC = {R: 1, L: sym.Rational('.5'), C: sym.Rational('.4')}
logx_plot(db(H.subs(RLC)), (w, 0.1, 10),
          xlabel='$\omega$', ylabel='$20 \log_{10} | H(j \omega) |$ in dB')
logx_plot(sym.arg(H.subs(RLC)), (w, 0.1, 10),
          xlabel='$\omega$', ylabel=r'$\varphi(j \omega)$')
```

**Example - Second order system**

The bode plot of a second-order LTI system with the following transfer function

\begin{equation}
H(s) = \frac{(s - s_0)(s - s_0^*)}{(s - s_\infty)(s - s_\infty^*)}
\end{equation}

with

\begin{align}
s_0 &= \sigma_0 + j \omega_0  &\text{with } \quad &\sigma_0 = 2000, \omega_0 = 2 \pi \cdot 100 \\
s_\infty  &= \sigma_\infty + j \omega_\infty & &\sigma_\infty = 20000, \omega_\infty = 2 \pi \cdot 4000
\end{align}

is constructed in this example. First the contribution of the pair of complex conjugate zeros $s_0$ and $s_0^*$ to the magnitude of the transfer function $H(j \omega)$ is computed and plotted over the frequency $f$ using $\omega = 2 \pi f$.

```python
f = sym.symbols('f', real=True)
s = sym.I * 2*sym.pi*f
s_0 = 2000 + sym.I * 2*sym.pi*100

H1 = (s - s_0)*(s - sym.conjugate(s_0))
Hlog1 = db(H1)
logx_plot(Hlog1, (f, 10, 10000), xlabel='$f$',
          ylabel='$20 \log_{10} |H_0(j \omega)|$ in dB')
```

Now the contribution of the pair of complex conjugate poles $s_\infty$ and $s_\infty^*$ is computed and plotted

```python
s_inf = 20000 + sym.I * 2*sym.pi*4000

H2 = 1/((s - s_inf)*(s - sym.conjugate(s_inf)))
Hlog2 = db(H2)
logx_plot(Hlog2, (f, 10, 20000), xlabel='$f$',
          ylabel='$20 \log_{10} |H_\infty(j \omega)|$ in dB')
```

The (logarithmic) magnitude frequency response of the system is given by superposition of the individual contributions from the zeros and the poles

```python
Hlog = Hlog1 + Hlog2
logx_plot(Hlog, (f, 10, 50000), xlabel='$f$',
          ylabel='$20 \log_{10} |H(j \omega)|$ in dB')
```

**Exercise**

* Examine the bode plots for the contributions of the zero and the pole:
    * What regions show approximately linear behavior over the frequency $f$?
    * What is the slope of these regions per octave/decade?
    * At which frequency is the transition between the regions of linear behavior? How is the frequency related to the imaginary part of the zero/pole?

* Examine the bode plot of the system. How is it related to the bode plots of the individual zero/pole?

* Move the pole and/or zero closer to the imaginary axis by changing the values $\sigma_\infty$ and/or $\sigma_0$. What changes?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


## Problem Set 2: Linear Discriminant Functions and Support Vector Machines

## Introduction
The MNIST database (Modified National Institute of Standards and Technology Database) is a large collection of handwritten digits which happens to be a state of the art data set to train and test machine learning models using image processing.It consist of 60000 Training samples and 10000 Testing Samples. Each sample image consist of normalized $28X28$ pixels grayscale image stored as 784 dimensional vector for each sample.This is a way of dimension reduction done by the MNIST to use it for model developments.

### Preparing the datasets
For loading the given MNIST datasets we will be using the mnist library to fetch the data.The MNIST() function will be used to fetch the datasets from the directory './data'.

```python
import numpy as np
import cv2
from mnist.loader import MNIST
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

m = MNIST('./data')
```

Now we load the different datasets as the X-Training and Y-Training set which has 60000 samples which will be used to train the classification model and then the X-Test and Y-test data set which has 10000 samples will be used to test the model. After seperating the datasets , the data sets will be converted to ND array and as float values which will be benificial for the computations.

```python
xtrain,ytrain = m.load_training()
xtest,ytest = m.load_testing()
xtrain = np.asarray(xtrain).astype(np.float32)
ytrain = np.asarray(ytrain).astype(np.float32)
xtest = np.asarray(xtest).astype(np.float32)
ytest = np.asarray(ytest).astype(np.float32)

```

### Part 1 :Train the Multi Class Support Vector Classifier with 1-Norm and Dot Product Kernel
For training the linear SVM an L1 norm will be used which is the Hinge loss.The training data is normalized before training it to decrease the convergence time and increase accuracy. The SVC Classification minimization function is given as
\begin{equation}
C\sum_{i=1,n}L(f(x_i),y_i)+\Omega(w)
\end{equation}
Where $C$ is the amount which sets the impact of the regularization. This will be set by the process of cross validation. We will be using the LinearSVC function to fit in the training data to build the SVC Classifier and the StandardScaler function will be used to normalize the input data during training and prediction using the model.

```python
from sklearn.svm import LinearSVC
from sklearn.preprocessing import StandardScaler
scl = StandardScaler()
scl.fit(xtrain)
svc = LinearSVC(loss = 'hinge',C = 1,max_iter = 2000)
# svc.fit(xtrain, ytrain)
svc.fit(scl.transform(xtrain), ytrain)
```

    c:\users\moham\appdata\local\programs\python\python38\lib\site-packages\sklearn\svm\_base.py:946: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.
      warnings.warn("Liblinear failed to converge, increase "

    LinearSVC(C=1, class_weight=None, dual=True, fit_intercept=True,
              intercept_scaling=1, loss='hinge', max_iter=2000, multi_class='ovr',
              penalty='l2', random_state=None, tol=0.0001, verbose=0)

Through cross validation the $C$ parameter is found to be $1$ when a maximum iteration of 2000 is set to during building the model.We will predict the target values using the SVC Classifier built with the testing sample dataset and then compute the accuracy and error with respect to the true target values.

```python
ypred = np.zeros(ytest.shape)
xtest_n = scl.transform(xtest)
for i in range(0,ypred.shape[0]):
    x = np.array([xtest_n[i]])
    ypred[i] = svc.predict(x)
true = 0
error = 0
for i in range(0,ytest.shape[0]):
    error = error + ((ytest[i]-ypred[i])**2)**0.5
    if(ytest[i] == ypred[i]):
        true = true+1

accuracy = true/ypred.shape[0]
Error = error/ypred.shape[0]

print('Accuracy - ',accuracy*100,'%')
print('Error - ',Error*100,'%')
```

    Accuracy -  91.82000000000001 %
    Error -  29.79 %

### Part 2 :Lagrange dual problem

The Given Problem is
$Features$ $(x_1,y_1),....(x_N,y_N)$ $where y_1,...y_N\in{-1,1}$ which is also the primal problem is given as
\begin{equation}
Minimize \; w^T.w+C\sum_{i=1}^N\xi_i \; where
\\
Subject\;to\;y_i(w^T.x_i)\geq1-\xi_i
\\
and\;\;\xi_i\geq0
\end{equation}
The inequality constraints can be expressed as
\begin{equation}
2-y_i(w^T.x_i)-\xi_i\leq0
\\
1-\xi\leq0
\end{equation}
To solve it using the Dual Method we will be converting it to Lagrange form.
The Lagrange form of the problem is given as
\begin{equation}
L(w,\alpha) = w^T.w+C\sum_{i=1}^N\xi_i + \alpha_1(2-y_i(w^T.x_i)-\xi_i)+\alpha_2(1-\xi)
\end{equation}
where $\alpha_1$ & $\alpha_2$ are the Lagrange Multipliers or the KKT multipliers where $\alpha_i\geq0$.
Set the partial derivative function to Zero.
\begin{equation}
\partial_wL = 2w - \alpha_1y_ix_i= 0
\\
w = \frac{\alpha_1y_ix_i}{2}
\end{equation}
The marigin is given as
\begin{equation}
\gamma = y\frac{w^Tw+b}{||w||}
\end{equation}
In SVM the a criterion is defined by a decision surface that maximally far away from any data point and the distance between the decision surface to the closest data point is called the marigin.The marigin needs to be maximized because the points nearer to the decision surface has high uncertainity with the decision function which classifies the data point where as a large marigin gives low uncertain classifications.This allows to handle small miscalculations and errors in the data sets.
Solving Dual problem is beneficial over primal problem because in Primal problem we only obtain the optimal $w$ where as in the Dual problem we take $\alpha_i$ in consideration.The $\alpha_i$ is useful because it decides the importance of the subjected features and computation of the classification can be faster than computing the scalar product $w^Tx$ directly in the case of the primal problem.

```python

```


```python
import networkx as nx
import matplotlib.pyplot as plt
import numpy as np
import random
import sympy
```

    Bad key "text.kerning_factor" on line 4 in
    /home/sc/anaconda3/envs/old_nx/lib/python3.6/site-packages/matplotlib/mpl-data/stylelib/_classic_test_patch.mplstyle.
    You probably need to get an updated matplotlibrc file from
    http://github.com/matplotlib/matplotlib/blob/master/matplotlibrc.template
    or from the matplotlib source distribution

```python
max_B = 10000 # maximum budget
no_runs = 100 # no. of runs to average

np.random.seed(0)
random.seed(42)
```

```python
edges = np.loadtxt('facebook_combined.txt',dtype=int)

G = nx.Graph()
G.add_edges_from(edges)

G_no_edges=G.number_of_edges()
G_no_nodes=G.number_of_nodes()
print(G_no_nodes,G_no_edges)
```

    4039 88234

```python
from louvain import detect_communities, modularity
'''
def draw_communities(G, node_size=90, alpha=1, k=None, randomized=False):
    partition = detect_communities(G, randomized=randomized)
    print("Modularity for best partition:", modularity(G, partition))
    community_map = {}
    for community, nodes in enumerate(partition):
        for node in nodes:
            community_map[node] = community

    cmap = plt.get_cmap("jet")
    plt.figure(figsize=(15, 15))
    pos = nx.spring_layout(G, k=k)
    indexed = [community_map.get(node) for node in G]
    plt.axis("off")
    nx.draw_networkx_nodes(G, pos=pos, cmap=cmap, node_color=indexed, node_size=node_size, alpha=alpha)
    nx.draw_networkx_edges(G, pos=pos, alpha=0.2)
    for x in range(len(pos)):
        pos[x] = pos[x]+np.array([0.02,0])
    labels = dict([(n, n) for n in G.nodes()]) #add labels
    #_ = nx.draw_networkx_labels(G, pos, labels=labels, font_color='#000000', font_size=15) #draw labels
draw_communities(G)
'''
```

    '\ndef draw_communities(G, node_size=90, alpha=1, k=None, randomized=False):\n    partition = detect_communities(G, randomized=randomized)\n    print("Modularity for best partition:", modularity(G, partition))\n    community_map = {}\n    for community, nodes in enumerate(partition):\n        for node in nodes:\n            community_map[node] = community\n    \n    cmap = plt.get_cmap("jet")\n    plt.figure(figsize=(15, 15))\n    pos = nx.spring_layout(G, k=k)\n    indexed = [community_map.get(node) for node in G]\n    plt.axis("off")\n    nx.draw_networkx_nodes(G, pos=pos, cmap=cmap, node_color=indexed, node_size=node_size, alpha=alpha)\n    nx.draw_networkx_edges(G, pos=pos, alpha=0.2)\n    for x in range(len(pos)):\n        pos[x] = pos[x]+np.array([0.02,0])\n    labels = dict([(n, n) for n in G.nodes()]) #add labels\n    #_ = nx.draw_networkx_labels(G, pos, labels=labels, font_color=\'#000000\', font_size=15) #draw labels\ndraw_communities(G)\n'

```python
partition = detect_communities(G, randomized=False)
community_map = {}
node_map = {}

for community, nodes in enumerate(partition):
    for node in nodes:
        node_map[node] = community
    community_map[community] = nodes
```

```python
K = 4

def MHRR_sampling(G,B,U=None):
    restart_ind = [10]
    t=20
    while restart_ind[-1]<B:
        restart_ind.append(restart_ind[-1]+int(K*np.log(t)))
        t+=1

    est_MH= []
    est_MH_t = 0
    sample = np.random.choice(G.nodes())
    est_MH_t += node_fn(sample)
    est_MH.append(est_MH_t)
    for ii in range(2,B+1):
        if ii in restart_ind:
            if U is None:
                sample = np.random.choice(community_map[np.random.choice([x for x in list(community_map.keys()) if x!=node_map[sample]])])
            else:
                z = [node_map[x] for x in U].index(node_map[sample])
                sample = np.random.choice(U[:z]+U[z+1:])
        else:
            neighbors = list(nx.neighbors(G,sample))
            sample_t = np.random.choice(neighbors)
            if np.random.rand() <= (G.degree(sample)/G.degree(sample_t)):
                sample = sample_t

        est_MH_t += node_fn(sample)
        est_MH.append(est_MH_t/ii)
    return np.array(est_MH)
```

```python
def MHRR_sampling_n(G,B,U=None):
    restart_ind = [10]
    t=11
    while t<B:
        if np.random.random()<50/(50+t):
            restart_ind.append(t)
        t+=1

    est_MH= []
    est_MH_t = 0
    sample = np.random.choice(G.nodes())
    est_MH_t += node_fn(sample)
    est_MH.append(est_MH_t)
    for ii in range(2,B+1):
        if ii in restart_ind:
            if U is None:
                sample = np.random.choice(community_map[np.random.choice([x for x in list(community_map.keys()) if x!=node_map[sample]])])
            else:
                z = [node_map[x] for x in U].index(node_map[sample])
                sample = np.random.choice(U[:z]+U[z+1:])
        else:
            neighbors = list(nx.neighbors(G,sample))
            sample_t = np.random.choice(neighbors)
            if np.random.rand() <= (G.degree(sample)/G.degree(sample_t)):
                sample = sample_t

        est_MH_t += node_fn(sample)
        est_MH.append(est_MH_t/ii)
    return np.array(est_MH)

def MHRR_sampling_log(G,B,U=None):
    restart_ind = [10]
    t=11
    while t<B:
        if np.random.random()<1/(1+2*np.log(t)):
            restart_ind.append(t)
        t+=1

    est_MH= []
    est_MH_t = 0
    sample = np.random.choice(G.nodes())
    est_MH_t += node_fn(sample)
    est_MH.append(est_MH_t)
    for ii in range(2,B+1):
        if ii in restart_ind:
            if U is None:
                sample = np.random.choice(community_map[np.random.choice([x for x in list(community_map.keys()) if x!=node_map[sample]])])
            else:
                z = [node_map[x] for x in U].index(node_map[sample])
                sample = np.random.choice(U[:z]+U[z+1:])
        else:
            neighbors = list(nx.neighbors(G,sample))
            sample_t = np.random.choice(neighbors)
            if np.random.rand() <= (G.degree(sample)/G.degree(sample_t)):
                sample = sample_t

        est_MH_t += node_fn(sample)
        est_MH.append(est_MH_t/ii)
    return np.array(est_MH)
```

## Finding High Degree Nodes in Clusters

```python
U = []

for com in list(community_map.keys()):
    nodes = community_map[com]
    maxd = -1
    maxn = -1
    for node in nodes:
        if G.degree(node)>maxd:
            maxd = G.degree(node)
            maxn = node
    U.append(maxn)
```

## F(v) = int(node_map[node] == 1)

```python
def node_fn(node):
    return int(node_map[node]==1)
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.10670958157959891

```python
MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B,U)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling_n(G,max_B,U)-F_org)**2
MSE_mhrr_n = MSE_mhrr_t/(no_runs)
MSE_mhrr_n = np.sqrt(MSE_mhrr_n)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling_log(G,max_B,U)-F_org)**2
MSE_mhrr_log = MSE_mhrr_t/(no_runs)
MSE_mhrr_log = np.sqrt(MSE_mhrr_log)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='red',linewidth=1.5,label='MHRR-T')
plt.plot(np.array(list(range(len(MSE_mhrr_n)))),MSE_mhrr_n,color='blue',linewidth=1.5,label='nMHRR-T')
plt.plot(np.array(list(range(len(MSE_mhrr_log)))),MSE_mhrr_log,color='black',linewidth=1.5,label='logMHRR-T')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(top=1)
plt.grid()
```

## F(v) = int(G.degree(node)>100)

```python
def node_fn(node):
    return int(G.degree(node)>100)
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.11908888338697697

```python
MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B,U)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling_n(G,max_B,U)-F_org)**2
MSE_mhrr_n = MSE_mhrr_t/(no_runs)
MSE_mhrr_n = np.sqrt(MSE_mhrr_n)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling_log(G,max_B,U)-F_org)**2
MSE_mhrr_log = MSE_mhrr_t/(no_runs)
MSE_mhrr_log = np.sqrt(MSE_mhrr_log)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='red',linewidth=1.5,label='MHRR-T')
plt.plot(np.array(list(range(len(MSE_mhrr_n)))),MSE_mhrr_n,color='blue',linewidth=1.5,label='nMHRR-T')
plt.plot(np.array(list(range(len(MSE_mhrr_log)))),MSE_mhrr_log,color='black',linewidth=1.5,label='logMHRR-T')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(top=1)
plt.grid()
```

## F(v) = isprime(v)

```python
def node_fn(node):
    return int(sympy.isprime(G.degree(node)))
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.2904184204010894

```python
MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B,U)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling_n(G,max_B,U)-F_org)**2
MSE_mhrr_n = MSE_mhrr_t/(no_runs)
MSE_mhrr_n = np.sqrt(MSE_mhrr_n)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling_log(G,max_B,U)-F_org)**2
MSE_mhrr_log = MSE_mhrr_t/(no_runs)
MSE_mhrr_log = np.sqrt(MSE_mhrr_log)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='red',linewidth=1.5,label='MHRR-T')
plt.plot(np.array(list(range(len(MSE_mhrr_n)))),MSE_mhrr_n,color='blue',linewidth=1.5,label='nMHRR-T')
plt.plot(np.array(list(range(len(MSE_mhrr_log)))),MSE_mhrr_log,color='black',linewidth=1.5,label='logMHRR-T')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(top=1)
plt.grid()
```

## F(v) = random

```python
fn_mapping = np.random.exponential(1,size=(G_no_nodes))
def node_fn(node):
    return fn_mapping[node]

F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.9946721335672434

```python
MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B,U)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling_n(G,max_B,U)-F_org)**2
MSE_mhrr_n = MSE_mhrr_t/(no_runs)
MSE_mhrr_n = np.sqrt(MSE_mhrr_n)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling_log(G,max_B,U)-F_org)**2
MSE_mhrr_log = MSE_mhrr_t/(no_runs)
MSE_mhrr_log = np.sqrt(MSE_mhrr_log)/F_org

plt.figure(figsize=(10,8))
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='red',linewidth=1.5,label='MHRR-T')
plt.plot(np.array(list(range(len(MSE_mhrr_n)))),MSE_mhrr_n,color='blue',linewidth=1.5,label='nMHRR-T')
plt.plot(np.array(list(range(len(MSE_mhrr_log)))),MSE_mhrr_log,color='black',linewidth=1.5,label='logMHRR-T')

legend = plt.legend(loc='best', shadow=True, fontsize='xx-large')
legend.get_frame().set_facecolor('0.90')
for legobj in legend.legendHandles:
    legobj.set_linewidth(2.5)

plt.ylim(top=1)
plt.grid()
```


# Forced Convection over a flat plate

A flat plate of width w=1 m is maintained at a uniform surface temperature, 𝑇_𝑠= 230oC,
 by using independently controlled, electrical strip heaters, each of which is 50 mm long.  If
 atmospheric air at 25oC flows over the plate at a velocity of 60 m/s, at what heater is the
 electrical input a maximum? What is the value of this input?

# Given

```python
import numpy as np

from thermal_system_calculation import FluidState, Fluid

width = 1 # m
length_heater = 0.05 # m
temp_surface = 230 # degC
temp_air = 25 # degC
velocity_air = 60 # m/s
```

# Find
At what heater is the electrical input a maximum?

What is the value of this maximum heat input?

# Schematic

# Assumption
1. The temperature is uniform at the surface.
2. Steady state conditions
3. Uniform and constant properties
4. Uniform heat transfer over a single heater

# Properties

```python
temp_boundary = np.mean([temp_air, temp_surface]) + 273.15
air_properties = FluidState(
    fluid=Fluid.AIR,
    temp_k=temp_boundary,
    characteristic_length=length_heater
)
thermal_conductivity_air = air_properties.k
specific_heat_capacity_air = air_properties.cp
dynamic_viscosity_air = air_properties.dynamic_viscosity
density_air = air_properties.density
print(f"Air at {temp_boundary} K has: \n"
      f"\t- thermal conductivity: {thermal_conductivity_air:.3f} W/mK\n"
      f"\t- dynamic viscosity: {dynamic_viscosity_air:.5e} Pa s \n"
      f"\t- density: {density_air:.3f} kg/m3")

```

    Air at 400.65 K has:
    	- thermal conductivity: 0.033 W/mK
    	- dynamic viscosity: 2.30830e-05 Pa s
    	- density: 0.881 kg/m3

# Analysis
As the temperature on the surface is give, we can calculate the heat transfer by the Newton's
cooling law.
$$q_\mathrm{conv} = h A \left(T_s - T_\infty\right)$$
and this heat transfer has to be balanced by the heat input from each heater.

We just learned that h is mainly determined by development of the boundary layers and the degree of
turbulence of the fluid flow. We need to search for maximum points, the entrance of the
 plate and near the flow transition from laminar flow to turbulence is. See below figure.

For the first heater, we can determine average heat transfer coefficients by convection and heat
transfer rate,
$\overline{h}_1$ by,
$$
\begin{align}
    \overline{Nu}_1 &= 0.664 Re_1^{1/2} Pr^{1/3} \\
    \overline{h}_1 &= \frac{\overline{Nu}_1 k}{L_1}
\end{align}
$$

```python
# Let's define the functions for Reynold's number, Prandtl number and Nusselts number.
def get_reynolds_number(
        velocity: float,
        length: float,
        density: float,
        dynamic_viscosity
) -> float:
    return velocity * length * density / dynamic_viscosity

def get_prandtl_number(
        specific_heat_capacity: float,
        dynamic_viscosity: float,
        thermal_conductivity: float
) -> float:
    return specific_heat_capacity * dynamic_viscosity / thermal_conductivity

def get_h_from_nusselt_number(
        nusselt_number: float,
        thermal_conductivity: float,
        length: float
) -> float:
    return nusselt_number * thermal_conductivity / length

re1 = get_reynolds_number(velocity_air, length_heater, density_air, dynamic_viscosity_air)
pr = get_prandtl_number(specific_heat_capacity_air, dynamic_viscosity_air, thermal_conductivity_air)
nu1 = 0.664 * re1**0.5 * pr**(1/3)
h1 = get_h_from_nusselt_number(nu1, thermal_conductivity_air, length_heater)
q1 = h1 * width * length_heater * (temp_surface - temp_air)

print(f"The heat input from the first heater is {q1:.3f} W.")
```

    The heat input from the first heater is 1369.114 W.

The next possible maximum heat input will happen when the air flow becomes fully turbulent. We
will use a criteria $Re_c = 500000$ for the full turbulence flow. We can find a characteristic
length for this $Re_c$.

```python
reynolds_number_at_full_turbulence = 500000
length_full_turbulence = reynolds_number_at_full_turbulence * dynamic_viscosity_air / \
                         (density_air * velocity_air)
plate_number_full_turbulence = int(np.ceil(length_full_turbulence / length_heater))
print(f"The flow is fully turbulent after {length_full_turbulence:.3f} m, "
      f"at {plate_number_full_turbulence}th heater.")
```

    The flow is fully turbulent after 0.218 m, at 5th heater.

On the 5th heater, the plate above has both laminar and turbulent flow. It has mixed boundary
layer conditions. We can obtain the local heat convection above the 5th heater by
$$
\begin{align}
    q_5 &= q_{1-5} - q_{1-4} \\
        &= \overline{h}_{1-5} L_5 w \left(T_s - T_\infty\right) - \overline{h}_{1-4}L_4 w \left(T_s - T_\infty\right) \\
        &= \left(\overline{h}_{1-5} L_5 - \overline{h}_{1-4} L_4\right) w \left(T_s - T_\infty\right)
\end{align}
$$
where we obtain $\overline{h}_{1-5}$ from the mixed boundary layer conditions and $\overline{h}_{1-4}$
from the laminar boundary layer condition.

$$
\begin{align}
    \overline{Nu}_{1-5} &= \left(0.037 Re_5^{4/5} - 871\right)Pr^{1/3} \\
    \overline{Nu}_{1-4} &= 0.664 Re_4^{1/2} * Pr^{1/3}
\end{align}
$$

```python
# h_1_5 Average convection heat transfer coefficient from the beginning to the end of the 5th heater
char_length_5 = length_heater * 5
re5 = get_reynolds_number(
    velocity=velocity_air,
    length=char_length_5,
    density=density_air,
    dynamic_viscosity=dynamic_viscosity_air
)
# Pradntl number does not change with the flow condition.
nu_1_5 = (0.037 * re5**0.8 - 871) * pr**(1/3)
h_1_5 = get_h_from_nusselt_number(nu_1_5, thermal_conductivity_air, char_length_5)

# h_1_4 Average convection heat transfer coefficient from the beginning to the end of the 4th heater
char_length_4 = length_heater * 4
re4 = get_reynolds_number(
    velocity=velocity_air,
    length=char_length_4,
    density=density_air,
    dynamic_viscosity=dynamic_viscosity_air
)
nu_1_4 = 0.664 * re4**0.5 * pr**(1/3)
h_1_4 = get_h_from_nusselt_number(nu_1_4, thermal_conductivity_air, char_length_4)

q_5 = (h_1_5 * char_length_5 - h_1_4 * char_length_4) * width * (temp_surface - temp_air)

print(f"The heat input from the fifth heater is {q_5:.3f} W.")
```

    The heat input from the fifth heater is 1058.754 W.

It is also possible that the heat transfer right after the transition to fully turbulent flow
since at the fifth section, it has a mixed boundary condition. We can see the heat transfer at
the sixth section in the same way as in the fifth section.h

$$q_6 = \left(\overline{h}_{1-6} L_6 - \overline{h}_{1-5} L_5\right) w \left(T_s - T_\infty\right)$$

```python
char_length_6 = length_heater * 6
re6 = get_reynolds_number(
    velocity=velocity_air,
    length=char_length_6,
    density=density_air,
    dynamic_viscosity=dynamic_viscosity_air
)
nu_1_6 = (0.037 * re6**0.8 - 871) * pr**(1/3)
h_1_6 = get_h_from_nusselt_number(nu_1_6, thermal_conductivity_air, char_length_6)

q_6 = (h_1_6 * char_length_6 - h_1_5 * char_length_5) * width * (temp_surface - temp_air)

print(f"The heat input from the sixth heater is {q_6:.3f} W.")

```

    The heat input from the sixth heater is 1429.741 W.

Therefore, the maximum heat input is required at the sixth heater.

```python

```