_Lambda School Data Science_

# Ordinary Least Squares Regression

## What is Linear Regression?

Linear Regression is a statistical model that seeks to describe the relationship between some y variable and one or more x variables.

In the simplest case, linear regression seeks to fit a straight line through a cloud of points. This line is referred to as the "regression line" or "line of best fit." This line tries to summarize the relationship between our X and Y in a way that enables us to use the equation for that line to make predictions.

### Synonyms for "y variable"
- Dependent Variable
- Response Variable
- Outcome Variable
- Predicted Variable
- Measured Variable
- Explained Variable
- Label
- Target

### Synonyms for "x variable"
- Independent Variable
- Explanatory Variable
- Regressor
- Covariate
- Feature

# Simple Linear Regresion (bivariate)

## Making Predictions

Say that we were trying to create a model that captured the relationship between temperature outside and ice cream sales. In Machine Learning our goal is often different that of other flavors of Linear Regression Analysis, because we're trying to fit a model to this data with the intention of making **predictions** on new data (in the future) that we don't have yet.

## What are we trying to predict?

So if we had measured ice cream sales and the temprature outside on 11 different days, at the end of our modeling **what would be the thing that we would want to predict? - Ice Cream Sales or Temperature?**

We would probably want to be measuring temperature with the intention of using that to **forecast** ice cream sales. If we were able to successfully forecast ice cream sales from temperature, this might help us know beforehand how much ice cream to make or how many cones to buy or on which days to open our store, etc. Being able to make predictions accurately has a lot of business implications. This is why making accurate predictions is so valuable (And in large part is why data scientists are paid so well).

### Y Variable Intuition

We want the thing that we're trying to predict to serve as our **y** variable. This is why it's sometimes called the "predicted variable." We call it the "dependent" variable because our prediction for how much ice cream we're going to sell "depends" on the temperature outside.

### X Variable Intuition

All other variables that we use to predict our y variable (we're going to start off just using one) we call our **x** variables. These are called our "independent" variables because they don't *depend* on y, they "explain" y. Hence they are also referred to as our "explanatory" variables.

```python
%matplotlib inline
from ipywidgets import interact
from matplotlib.patches import Rectangle
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
import statsmodels.api as sm
```

```python
columns = ['Year','Incumbent Party Candidate','Other Candidate','Incumbent Party Vote Share']

data = [[1952,"Stevenson","Eisenhower",44.6],
        [1956,"Eisenhower","Stevenson",57.76],
        [1960,"Nixon","Kennedy",49.91],
        [1964,"Johnson","Goldwater",61.34],
        [1968,"Humphrey","Nixon",49.60],
        [1972,"Nixon","McGovern",61.79],
        [1976,"Ford","Carter",48.95],
        [1980,"Carter","Reagan",44.70],
        [1984,"Reagan","Mondale",59.17],
        [1988,"Bush, Sr.","Dukakis",53.94],
        [1992,"Bush, Sr.","Clinton",46.55],
        [1996,"Clinton","Dole",54.74],
        [2000,"Gore","Bush, Jr.",50.27],
        [2004,"Bush, Jr.","Kerry",51.24],
        [2008,"McCain","Obama",46.32],
        [2012,"Obama","Romney",52.00]]

df = pd.DataFrame(data=data, columns=columns)
```

```python
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share

      0
      1952
      Stevenson
      Eisenhower
      44.60

      1
      1956
      Eisenhower
      Stevenson
      57.76

      2
      1960
      Nixon
      Kennedy
      49.91

      3
      1964
      Johnson
      Goldwater
      61.34

      4
      1968
      Humphrey
      Nixon
      49.60

      5
      1972
      Nixon
      McGovern
      61.79

      6
      1976
      Ford
      Carter
      48.95

      7
      1980
      Carter
      Reagan
      44.70

      8
      1984
      Reagan
      Mondale
      59.17

      9
      1988
      Bush, Sr.
      Dukakis
      53.94

      10
      1992
      Bush, Sr.
      Clinton
      46.55

      11
      1996
      Clinton
      Dole
      54.74

      12
      2000
      Gore
      Bush, Jr.
      50.27

      13
      2004
      Bush, Jr.
      Kerry
      51.24

      14
      2008
      McCain
      Obama
      46.32

      15
      2012
      Obama
      Romney
      52.00

```python
df.plot(x='Year', y='Incumbent Party Vote Share', kind='scatter');
```

```python
df['Incumbent Party Vote Share'].describe()
```

    count    16.000000
    mean     52.055000
    std       5.608951
    min      44.600000
    25%      48.350000
    50%      50.755000
    75%      55.495000
    max      61.790000
    Name: Incumbent Party Vote Share, dtype: float64

```python
target = 'Incumbent Party Vote Share'
df['Prediction'] = df[target].mean()
df['Error'] = df['Prediction'] - df[target]
```

```python
df['Error'].sum()
```

    -1.4210854715202004e-14

```python
df['Absolute Error'] = df['Error'].abs()
df['Absolute Error'].sum()
```

    72.82

```python
df['Absolute Error'].mean()
```

    4.55125

```python
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error

      0
      1952
      Stevenson
      Eisenhower
      44.60
      52.055
      7.455
      7.455

      1
      1956
      Eisenhower
      Stevenson
      57.76
      52.055
      -5.705
      5.705

      2
      1960
      Nixon
      Kennedy
      49.91
      52.055
      2.145
      2.145

      3
      1964
      Johnson
      Goldwater
      61.34
      52.055
      -9.285
      9.285

      4
      1968
      Humphrey
      Nixon
      49.60
      52.055
      2.455
      2.455

      5
      1972
      Nixon
      McGovern
      61.79
      52.055
      -9.735
      9.735

      6
      1976
      Ford
      Carter
      48.95
      52.055
      3.105
      3.105

      7
      1980
      Carter
      Reagan
      44.70
      52.055
      7.355
      7.355

      8
      1984
      Reagan
      Mondale
      59.17
      52.055
      -7.115
      7.115

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      52.055
      -1.885
      1.885

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      52.055
      5.505
      5.505

      11
      1996
      Clinton
      Dole
      54.74
      52.055
      -2.685
      2.685

      12
      2000
      Gore
      Bush, Jr.
      50.27
      52.055
      1.785
      1.785

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      52.055
      0.815
      0.815

      14
      2008
      McCain
      Obama
      46.32
      52.055
      5.735
      5.735

      15
      2012
      Obama
      Romney
      52.00
      52.055
      0.055
      0.055

```python
mean_absolute_error(y_true=df[target], y_pred=df['Prediction'])
```

    4.55125

```python
r2_score(y_true=df[target], y_pred=df['Prediction'])
```

    0.0

# R Squared:  $R^2$

One final attribute of linear regressions that we're going to talk about today is a measure of goodness of fit known as $R^2$ or R-squared. $R^2$ is a statistical measure of how close the data are fitted to our regression line. A helpful interpretation for the $R^2$ is the percentage of the dependent variable that is explained by the model.

In other words, the $R^2$ is the percentage of y that is explained by the x variables included in the model. For this reason the $R^2$ is also known as the "coefficient of determination," because it explains how much of y is explained (or determined) by our x varaibles. We won't go into the calculation of $R^2$ today, just know that a higher $R^2$ percentage is nearly always better and indicates a model that fits the data more closely.

# Add data

```python
columns = ['Year','Average Recent Growth in Personal Incomes']

data = [[1952,2.40],
        [1956,2.89],
        [1960, .85],
        [1964,4.21],
        [1968,3.02],
        [1972,3.62],
        [1976,1.08],
        [1980,-.39],
        [1984,3.86],
        [1988,2.27],
        [1992, .38],
        [1996,1.04],
        [2000,2.36],
        [2004,1.72],
        [2008, .10],
        [2012, .95]]

growth = pd.DataFrame(data=data, columns=columns)
```

```python
df = df.merge(growth)
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error
      Average Recent Growth in Personal Incomes

      0
      1952
      Stevenson
      Eisenhower
      44.60
      52.055
      7.455
      7.455
      2.40

      1
      1956
      Eisenhower
      Stevenson
      57.76
      52.055
      -5.705
      5.705
      2.89

      2
      1960
      Nixon
      Kennedy
      49.91
      52.055
      2.145
      2.145
      0.85

      3
      1964
      Johnson
      Goldwater
      61.34
      52.055
      -9.285
      9.285
      4.21

      4
      1968
      Humphrey
      Nixon
      49.60
      52.055
      2.455
      2.455
      3.02

      5
      1972
      Nixon
      McGovern
      61.79
      52.055
      -9.735
      9.735
      3.62

      6
      1976
      Ford
      Carter
      48.95
      52.055
      3.105
      3.105
      1.08

      7
      1980
      Carter
      Reagan
      44.70
      52.055
      7.355
      7.355
      -0.39

      8
      1984
      Reagan
      Mondale
      59.17
      52.055
      -7.115
      7.115
      3.86

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      52.055
      -1.885
      1.885
      2.27

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      52.055
      5.505
      5.505
      0.38

      11
      1996
      Clinton
      Dole
      54.74
      52.055
      -2.685
      2.685
      1.04

      12
      2000
      Gore
      Bush, Jr.
      50.27
      52.055
      1.785
      1.785
      2.36

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      52.055
      0.815
      0.815
      1.72

      14
      2008
      McCain
      Obama
      46.32
      52.055
      5.735
      5.735
      0.10

      15
      2012
      Obama
      Romney
      52.00
      52.055
      0.055
      0.055
      0.95

```python
feature = 'Average Recent Growth in Personal Incomes'
```

```python
df.plot(x=feature, y=target, kind='scatter');
```

We can see from the scatterplot that these data points seem to follow a somewhat linear relationship. This means that we could probably summarize their relationship well by fitting a line of best fit to these points. Lets do it.

## The Equation for a Line

As we know a common equation for a line is:

\begin{align}
y = mx + b
\end{align}

Where $m$ is the slope of our line and $b$ is the y-intercept.

If we want to plot a line through our cloud of points we figure out what these two values should be. Linear Regression seeks to **estimate** the slope and intercept values that describe a line that best fits the data points.

```python
m = 4
b = 44
df['Prediction'] = m * df[feature] + b
df['Error'] = df['Prediction'] - df[target]
df['Absolute Error'] = df['Error'].abs()
df['Absolute Error'].sum()
```

    45.27999999999999

```python
df['Absolute Error'].mean()
```

    2.829999999999999

```python
r2_score(y_true=df[target], y_pred=df['Prediction'])
```

    0.5178779627255485

```python
ax = df.plot(x=feature, y=target, kind='scatter')
df.plot(x=feature, y='Prediction', kind='line', ax=ax);
```

```python
def regression(m, b):
    df['Prediction'] = m * df[feature] + b
    df['Error'] = df['Prediction'] - df[target]
    df['Absolute Error'] = df['Error'].abs()
    sum_absolute_error = df['Absolute Error'].sum()

    title = f'Sum of absolute errors: {sum_absolute_error}'
    ax = df.plot(x=feature, y=target, kind='scatter', title=title, figsize=(7, 7))
    df.plot(x=feature, y='Prediction', kind='line', ax=ax)

regression(m=4, b=48)
```

## Residual Error

The residual error is the distance between points in our dataset and our regression line.

```python
def regression(m, b):
    df['Prediction'] = m * df[feature] + b
    df['Error'] = df['Prediction'] - df[target]
    df['Absolute Error'] = df['Error'].abs()
    sum_absolute_error = df['Absolute Error'].sum()

    title = f'Sum of absolute errors: {sum_absolute_error}'
    ax = df.plot(x=feature, y=target, kind='scatter', title=title, figsize=(7, 7))
    df.plot(x=feature, y='Prediction', kind='line', ax=ax)

    for x, y1, y2 in zip(df[feature], df[target], df['Prediction']):
        ax.plot((x, x), (y1, y2), color='grey')

regression(m=3, b=46)
```

```python
interact(regression, m=(-10,10,0.5), b=(40,60,0.5));
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=10.0, min=-10.0, step=0.5), FloatSlider(valu…

```python
df['Square Error'] = df['Error'] **2
```

```python
def regression(m, b):
    df['Prediction'] = m * df[feature] + b
    df['Error'] = df['Prediction'] - df[target]
    df['Absolute Error'] = df['Error'].abs()
    df['Square Error'] = df['Error'] **2
    sum_square_error = df['Square Error'].sum()

    title = f'Sum of square errors: {sum_square_error}'
    ax = df.plot(x=feature, y=target, kind='scatter', title=title, figsize=(7, 7))
    df.plot(x=feature, y='Prediction', kind='line', ax=ax)

    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)

    for x, y1, y2 in zip(df[feature], df[target], df['Prediction']):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))
```

```python
interact(regression, m=(-10,10,0.5), b=(40,60,0.5));
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=10.0, min=-10.0, step=0.5), FloatSlider(valu…

```python
b = 46
ms = np.arange(-10,10,0.5)
sses = []

for m in ms:
    predictions = m * df[feature] + b
    errors = predictions - df[target]
    square_errors = errors ** 2
    sse = square_errors.sum()
    sses.append(sse)

hypotheses = pd.DataFrame({'Slope': ms})
hypotheses['Intercept'] = b
hypotheses['Sum of Square Errors'] = sses

hypotheses.sort_values(by='Sum of Square Errors')
```

      Slope
      Intercept
      Sum of Square Errors

      26
      3.0
      46
      200.48220

      27
      3.5
      46
      209.41375

      25
      2.5
      46
      234.96115

      28
      4.0
      46
      261.75580

      24
      2.0
      46
      312.85060

      29
      4.5
      46
      357.50835

      23
      1.5
      46
      434.15055

      30
      5.0
      46
      496.67140

      22
      1.0
      46
      598.86100

      31
      5.5
      46
      679.24495

      21
      0.5
      46
      806.98195

      32
      6.0
      46
      905.22900

      20
      0.0
      46
      1058.51340

      33
      6.5
      46
      1174.62355

      19
      -0.5
      46
      1353.45535

      34
      7.0
      46
      1487.42860

      18
      -1.0
      46
      1691.80780

      35
      7.5
      46
      1843.64415

      17
      -1.5
      46
      2073.57075

      36
      8.0
      46
      2243.27020

      16
      -2.0
      46
      2498.74420

      37
      8.5
      46
      2686.30675

      15
      -2.5
      46
      2967.32815

      38
      9.0
      46
      3172.75380

      14
      -3.0
      46
      3479.32260

      39
      9.5
      46
      3702.61135

      13
      -3.5
      46
      4034.72755

      12
      -4.0
      46
      4633.54300

      11
      -4.5
      46
      5275.76895

      10
      -5.0
      46
      5961.40540

      9
      -5.5
      46
      6690.45235

      8
      -6.0
      46
      7462.90980

      7
      -6.5
      46
      8278.77775

      6
      -7.0
      46
      9138.05620

      5
      -7.5
      46
      10040.74515

      4
      -8.0
      46
      10986.84460

      3
      -8.5
      46
      11976.35455

      2
      -9.0
      46
      13009.27500

      1
      -9.5
      46
      14085.60595

      0
      -10.0
      46
      15205.34740

```python
hypotheses.plot(x='Slope', y='Sum of Square Errors',
                title=f'Intercept={b}');
```

```python
X = df[[feature]]
y = df[target]

X.shape, y.shape
```

    ((16, 1), (16,))

```python
model = LinearRegression()
model.fit(X, y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None,
             normalize=False)

```python
model.coef_, model.intercept_
```

    (array([3.06052805]), 46.247648016800795)

```python
model.predict([[0]])
```

    array([46.24764802])

```python
model.predict([[1]])
```

    array([49.30817607])

```python
model.predict([[2]])
```

    array([52.36870413])

```python
model.predict([[3]])
```

    array([55.42923218])

```python
model.predict(X)
```

    array([53.59291535, 55.09257409, 48.84909686, 59.13247113, 55.49044274,
           57.32675957, 49.55301832, 45.05404208, 58.06128631, 53.1950467 ,
           47.41064868, 49.43059719, 53.47049423, 51.51175627, 46.55370082,
           49.15514967])

```python
df['Prediction'] = model.predict(X)
```

```python
ax = df.plot(x=feature, y=target, kind='scatter', title='sklearn LinearRegression')
df.plot(x=feature, y='Prediction', kind='line', ax=ax);
```

```python
df['Error'] = df['Prediction'] - y
```

```python
df['Absolute Error'] = df['Error'].abs()
df['Square Error'] = df['Error'] ** 2
```

```python
df['Square Error'].mean()
```

    12.392042143389562

```python
mean_squared_error(y_true=y, y_pred=model.predict(X))
```

    12.39204214338956

```python
np.sqrt(mean_squared_error(y, model.predict(X)))
```

    3.5202332512760512

```python
model.score(X, y)
```

    0.5798462099485426

```python
r2_score(y, model.predict(X))
```

    0.5798462099485426

### Statsmodels

https://www.statsmodels.org/dev/examples/notebooks/generated/ols.html

```python
X
```

      Average Recent Growth in Personal Incomes

      0
      2.40

      1
      2.89

      2
      0.85

      3
      4.21

      4
      3.02

      5
      3.62

      6
      1.08

      7
      -0.39

      8
      3.86

      9
      2.27

      10
      0.38

      11
      1.04

      12
      2.36

      13
      1.72

      14
      0.10

      15
      0.95

```python
sm.add_constant(X)
```

      const
      Average Recent Growth in Personal Incomes

      0
      1.0
      2.40

      1
      1.0
      2.89

      2
      1.0
      0.85

      3
      1.0
      4.21

      4
      1.0
      3.02

      5
      1.0
      3.62

      6
      1.0
      1.08

      7
      1.0
      -0.39

      8
      1.0
      3.86

      9
      1.0
      2.27

      10
      1.0
      0.38

      11
      1.0
      1.04

      12
      1.0
      2.36

      13
      1.0
      1.72

      14
      1.0
      0.10

      15
      1.0
      0.95

```python
model = sm.OLS(y, sm.add_constant(X))
results = model.fit()
print(results.summary())
```

                                    OLS Regression Results
    ======================================================================================
    Dep. Variable:     Incumbent Party Vote Share   R-squared:                       0.580
    Model:                                    OLS   Adj. R-squared:                  0.550
    Method:                         Least Squares   F-statistic:                     19.32
    Date:                        Mon, 29 Apr 2019   Prob (F-statistic):           0.000610
    Time:                                12:42:56   Log-Likelihood:                -42.839
    No. Observations:                          16   AIC:                             89.68
    Df Residuals:                              14   BIC:                             91.22
    Df Model:                                   1
    Covariance Type:                    nonrobust
    =============================================================================================================
                                                    coef    std err          t      P>|t|      [0.025      0.975]
    -------------------------------------------------------------------------------------------------------------
    const                                        46.2476      1.622     28.514      0.000      42.769      49.726
    Average Recent Growth in Personal Incomes     3.0605      0.696      4.396      0.001       1.567       4.554
    ==============================================================================
    Omnibus:                        5.392   Durbin-Watson:                   2.379
    Prob(Omnibus):                  0.067   Jarque-Bera (JB):                2.828
    Skew:                          -0.961   Prob(JB):                        0.243
    Kurtosis:                       3.738   Cond. No.                         4.54
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

    /anaconda3/lib/python3.7/site-packages/scipy/stats/stats.py:1416: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=16
      "anyway, n=%i" % int(n))

# The Anatomy of Linear Regression

- Intercept: The $b$ value in our line equation $y=mx+b$
- Slope: The $m$ value in our line equation $y=mx+b$. These two values together define our regression line.

- $\hat{y}$ : A prediction
- Line of Best Fit (Regression Line)
- Predicted (fitted) Values: Points on our regression line
- Observed Values: Points from our dataset
- Error: The distance between predicted and observed values.

# More Formal Notation

We have talked about a line of regression being represented like a regular line $y=mx+b$ but as we get to more complicated versions we're going to need to extend this equation. So lets establish the proper terminology.

**X** - Independent Variable, predictor variable, explanatory variable, regressor, covariate

**Y** - Response variable, predicted variable, measured vairable, explained variable, outcome variable

$\beta_0$ - "Beta Naught" or "Beta Zero", the intercept value. This is how much of y would exist if X were zero. This is sometimes represented by the letter "a" but I hate that. So it's "Beta 0" during my lecture.

$\beta_1$ - "Beta One" The primary coefficient of interest. This values is the slope of the line that is estimated by "minimizing the sum of the squared errors/residuals" - We'll get to that.

$\epsilon$ - "Epsilon" The "error term", random noise, things outside of our model that affect y.

# How Does it do it?

## Minimizing the Sum of the Squared Error

The most common method of estimating our $\beta$ parameters  is what's known as "Ordinary Least Squares" (OLS). (There are different methods of arriving at a line of best fit). OLS estimates the parameters that minimize the squared distance between each point in our dataset and our line of best fit.

\begin{align}
SSE = \sum(y_i - \hat{y})^2
\end{align}

## Linear Algebra!

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

### Lets calculate our $\beta$ coefficients with numpy!

```python
X = sm.add_constant(df[feature]).values
print('X')
print(X)

y = df[target].values[:, np.newaxis]
print('y')
print(y)

X_transpose = X.T
print('X Transpose')
print(X_transpose)

X_transpose_X = X_transpose @ X
print('X Transpose X')
print(X_transpose_X)

X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
print('X Transpose X Inverse')
print(X_transpose_X_inverse)

X_transpose_y = X_transpose @ y
print('X Transpose y')
print(X_transpose_y)

beta_hat = X_transpose_X_inverse @ X_transpose_y
print('Beta Hat')
print(beta_hat)
```

    X
    [[ 1.    2.4 ]
     [ 1.    2.89]
     [ 1.    0.85]
     [ 1.    4.21]
     [ 1.    3.02]
     [ 1.    3.62]
     [ 1.    1.08]
     [ 1.   -0.39]
     [ 1.    3.86]
     [ 1.    2.27]
     [ 1.    0.38]
     [ 1.    1.04]
     [ 1.    2.36]
     [ 1.    1.72]
     [ 1.    0.1 ]
     [ 1.    0.95]]
    y
    [[44.6 ]
     [57.76]
     [49.91]
     [61.34]
     [49.6 ]
     [61.79]
     [48.95]
     [44.7 ]
     [59.17]
     [53.94]
     [46.55]
     [54.74]
     [50.27]
     [51.24]
     [46.32]
     [52.  ]]
    X Transpose
    [[ 1.    1.    1.    1.    1.    1.    1.    1.    1.    1.    1.    1.
       1.    1.    1.    1.  ]
     [ 2.4   2.89  0.85  4.21  3.02  3.62  1.08 -0.39  3.86  2.27  0.38  1.04
       2.36  1.72  0.1   0.95]]
    X Transpose X
    [[16.    30.36 ]
     [30.36  86.821]]
    X Transpose X Inverse
    [[ 0.18575056 -0.06495418]
     [-0.06495418  0.03423145]]
    X Transpose y
    [[ 832.88  ]
     [1669.7967]]
    Beta Hat
    [[46.24764802]
     [ 3.06052805]]

# Multiple Regression

Simple or bivariate linear regression involves a single $x$ variable and a single $y$ variable. However, we can have many $x$ variables. A linear regression model that involves multiple x variables is known as **Multiple** Regression - NOT MULTIVARIATE!

```python
df.sort_values(by='Square Error', ascending=False)
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error
      Average Recent Growth in Personal Incomes
      Square Error

      0
      1952
      Stevenson
      Eisenhower
      44.60
      53.592915
      8.992915
      8.992915
      2.40
      80.872526

      4
      1968
      Humphrey
      Nixon
      49.60
      55.490443
      5.890443
      5.890443
      3.02
      34.697316

      11
      1996
      Clinton
      Dole
      54.74
      49.430597
      -5.309403
      5.309403
      1.04
      28.189758

      5
      1972
      Nixon
      McGovern
      61.79
      57.326760
      -4.463240
      4.463240
      3.62
      19.920515

      12
      2000
      Gore
      Bush, Jr.
      50.27
      53.470494
      3.200494
      3.200494
      2.36
      10.243163

      15
      2012
      Obama
      Romney
      52.00
      49.155150
      -2.844850
      2.844850
      0.95
      8.093173

      1
      1956
      Eisenhower
      Stevenson
      57.76
      55.092574
      -2.667426
      2.667426
      2.89
      7.115161

      3
      1964
      Johnson
      Goldwater
      61.34
      59.132471
      -2.207529
      2.207529
      4.21
      4.873184

      8
      1984
      Reagan
      Mondale
      59.17
      58.061286
      -1.108714
      1.108714
      3.86
      1.229246

      2
      1960
      Nixon
      Kennedy
      49.91
      48.849097
      -1.060903
      1.060903
      0.85
      1.125515

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      47.410649
      0.860649
      0.860649
      0.38
      0.740716

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      53.195047
      -0.744953
      0.744953
      2.27
      0.554955

      6
      1976
      Ford
      Carter
      48.95
      49.553018
      0.603018
      0.603018
      1.08
      0.363631

      7
      1980
      Carter
      Reagan
      44.70
      45.054042
      0.354042
      0.354042
      -0.39
      0.125346

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      51.511756
      0.271756
      0.271756
      1.72
      0.073851

      14
      2008
      McCain
      Obama
      46.32
      46.553701
      0.233701
      0.233701
      0.10
      0.054616

```python
"""
Fatalities denotes the cumulative number of American military
fatalities per millions of US population the in Korea, Vietnam,
Iraq and Afghanistan wars during the presidential terms
preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and
2012 elections.

http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf
"""

columns = ['Year','US Military Fatalities per Million']

data = [[1952,190],
        [1956,  0],
        [1960,  0],
        [1964,  1],
        [1968,146],
        [1972,  0],
        [1976,  2],
        [1980,  0],
        [1984,  0],
        [1988,  0],
        [1992,  0],
        [1996,  0],
        [2000,  0],
        [2004,  4],
        [2008, 14],
        [2012,  5]]

deaths = pd.DataFrame(data=data, columns=columns)
```

```python
df = df.merge(deaths)
```

```python
features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million']

target = 'Incumbent Party Vote Share'

X = df[features]
y = df[target]

model = sm.OLS(y, sm.add_constant(X))
results = model.fit()
print(results.summary())
```

                                    OLS Regression Results
    ======================================================================================
    Dep. Variable:     Incumbent Party Vote Share   R-squared:                       0.871
    Model:                                    OLS   Adj. R-squared:                  0.851
    Method:                         Least Squares   F-statistic:                     43.77
    Date:                        Mon, 29 Apr 2019   Prob (F-statistic):           1.68e-06
    Time:                                12:57:54   Log-Likelihood:                -33.412
    No. Observations:                          16   AIC:                             72.82
    Df Residuals:                              13   BIC:                             75.14
    Df Model:                                   2
    Covariance Type:                    nonrobust
    =============================================================================================================
                                                    coef    std err          t      P>|t|      [0.025      0.975]
    -------------------------------------------------------------------------------------------------------------
    const                                        46.6621      0.937     49.806      0.000      44.638      48.686
    Average Recent Growth in Personal Incomes     3.4820      0.408      8.527      0.000       2.600       4.364
    US Military Fatalities per Million           -0.0537      0.010     -5.408      0.000      -0.075      -0.032
    ==============================================================================
    Omnibus:                        2.475   Durbin-Watson:                   2.607
    Prob(Omnibus):                  0.290   Jarque-Bera (JB):                0.666
    Skew:                           0.096   Prob(JB):                        0.717
    Kurtosis:                       3.981   Cond. No.                         110.
    ==============================================================================

    Warnings:
    [1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

    /anaconda3/lib/python3.7/site-packages/scipy/stats/stats.py:1416: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=16
      "anyway, n=%i" % int(n))

```python
model = LinearRegression()
model.fit(X, y)
print('Intercept:', model.intercept_)
pd.Series(model.coef_, features)
```

    Intercept: 46.662064098015115

    Average Recent Growth in Personal Incomes    3.481956
    US Military Fatalities per Million          -0.053661
    dtype: float64

```python
np.sqrt(mean_squared_error(y, model.predict(X)))
```

    1.9528760602450586

# Train / Test Split

```python
train = df.query('Year < 2008')
test  = df.query('Year >= 2008')
X_train = train[features]
y_train = train[target]
X_test  = test[features]
y_test  = test[target]
X_train.shape, y_train.shape, X_test.shape, y_test.shape
```

    ((14, 2), (14,), (2, 2), (2,))

```python
model.fit(X_train, y_train)
model.predict(X_test)
```

    array([45.86970509, 49.39965918])

```python
test
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Prediction
      Error
      Absolute Error
      Average Recent Growth in Personal Incomes
      Square Error
      US Military Fatalities per Million

      14
      2008
      McCain
      Obama
      46.32
      46.553701
      0.233701
      0.233701
      0.10
      0.054616
      14

      15
      2012
      Obama
      Romney
      52.00
      49.155150
      -2.844850
      2.844850
      0.95
      8.093173
      5

### More about the "Bread & Peace" model
- https://fivethirtyeight.com/features/what-do-economic-models-really-tell-us-about-elections/
- https://statmodeling.stat.columbia.edu/2007/12/15/bread_and_peace/
- https://avehtari.github.io/RAOS-Examples/ElectionsEconomy/hibbs.html
- https://douglas-hibbs.com/
- http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf

# Dimensionality in Linear Regression!

Muliple Regression is simply an extension of the bivariate case. The reason why we see the bivariate case demonstrated so often is simply because it's easier to graph and all of the intuition from the bivariate case is the same as we keep on adding explanatory variables.

As we increase the number of $x$ values in our model we are simply fitting a n-1-dimensional plane to an n-dimensional cloud of points within an n-dimensional hypercube.

# Interpreting Coefficients

One of Linear Regression's strengths is that the parameters of the model (coefficients) are readily interpretable and useful. Not only do they describe the relationship between x and y but they put a number on just how much x is associated with y. We should be careful to not speak about this relationshiop in terms of causality because these coefficients are in fact correlative measures. We would need a host of additional techniques in order to estimate a causal effect using linear regression (econometrics).

\begin{align}
\hat{\beta} = \frac{Cov(x,y)}{Var(y)}
\end{align}

Going back to the two equations for the two models that we have estimated so far, lets replace their beta values with their actual values to see if we can make sense of how to interpret these beta coefficients.

## Bivariate Model

$y_i = \beta_0 + \beta_1temperature + \epsilon$

$sales_i = -596.2 + 24.69temperature + \epsilon$

What might $\beta_0$ in this model represent? It represents the level of sales that we would have if temperature were 0. Since this is negative one way of interpreting it is that it's so cold outside that you would have to pay people to eat ice cream. A more appropriate interpretation is probably that the ice cream store owner should close his store down long before the temperature reaches 0 degrees farenheit (-17.7 celsius). The owner can compare his predicted sales with his costs of doing business to know how warm the weather has to get before he should open his store.

What might the $beta_1$ in this model reprsent? it represents the increase in sales for each degree of temperature increase. For every degree that the temperature goes up outside he has $25 more in sales.

## Multiple Regression Model

$y_i = \beta_0 + \beta_1age_i + \beta_2weight_i + \epsilon$

$BloodPressure_i = 30.99+ .86age_i + .33weight_i + \epsilon$

The interpretation of coefficients in this example are similar. The intercept value repesents the blood pressure a person would have if they were 0 years old and weighed 0 pounds. This not a super useful interpretation. If we look at our data it is unlikely that we have any measurements like these in the dataset. This means that our interpretation of our intercept likely comes from extrapolating the regression line (plane). Coefficients having straightforward interpretations is a strength of linear regression if we're careful about extrapolation and only interpreting our data within the context that it was gathered.

The interpretation of our other coefficients can be a useful indicator for how much a person similar to those in our dataset's blood pressure will go up on average with each additional year of age and pound of weight.

# Basic Model Validation

One of the downsides of relying on $R^2$ too much is that although it tells you when you're fitting the data well, it doesn't tell you when you're *overfitting* the data. The best way to tell if you're overfitting the data is to get some data that your model hasn't seen yet, and evaluate how your predictions do. This is essentially what "model validation" is.

# Why is Linear Regression so Important?

## Popularity

Linear Regression is an extremely popular technique that every data scientist **needs** to understand. It's not the most advanced technique and there are supervised learning techniques that will obtain a higher accuracy, but where it lacks in accuracy it makes up for it in interpretability and simplicity.

## Interpretability

Few other models possess coefficients that are so directly linked to their variables with a such a clear interpretation. Tomorrow we're going to learn about ways to make them even easier to interpret.

## Simplicity

A linear regression model can be communicated just by writing out its equation. It's kind of incredible that such high dimensional relationships can be described from just a linear combination of variables and coefficients.


```python
# ORIGINAL RGB FILE
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import random
import pystan

from IPython.display import display

from matplotlib import rc
rc("font", family="serif", size=11)
```

First we will read in a file that contains the overall list of all the available datasets, with corresponding star IDs, $\nu_{max}$ and $\sigma_{\nu_{max}}$ values. From this we will take a random selection for our analysis.

```python
random.seed(121)
#output = pd.read_csv('../data/output_1000stars.csv', usecols=range(1,4))
output = pd.read_csv('../data/output_back_filesremoved.csv', usecols=['kic','numax', 'numax_err'])
output = output.loc[(np.abs(output['numax'] - 75) < 30.0)].reset_index(drop=True)
# Select a random number of stars from the file, recording their IDs
IDs = [random.choice(output['kic']) for i in range(6)]
IDs = [10802837,12003742,2283075,4760954,11358669,11521197]
# Create a list of the numax values corresponding to each ID
Numax = [(output.loc[(output['kic'] == IDs[i])]).iloc[0]['numax'] for i in range(len(IDs))]
Numax_err = [(output.loc[(output['kic'] == IDs[i])]).iloc[0]['numax_err'] for i in range(len(IDs))]
```

Using our list of selected stars, we can then pull the respective files containing the induvidual mode frequencies and errors. The different files will contain varying numbers of modes; in order to successfully parse the data through Stan, one must ensure that all the arrays are of the same length. To do this, for every dataset that contains less modes than the most complete dataset, we can produce additional modes based on the average frequency spacing $\left<\Delta\nu\right>$ for each star, ensuring to assign these points with a large uncertainty so that Stan does not use this as restrictive points during the fit (they are essentially ignored).

```python
modesID = [pd.read_csv('../data/rgbmodes/modes_'+str(IDs[i])+'.csv', usecols=['f0', 'f0_err', 'A0'])
           for i in np.arange(0,len(IDs),1)]
lenmodes = [len(modesID[i]) for i in range(len(IDs))]
maxmodes = max(lenmodes)
arr_n = np.zeros([len(IDs),maxmodes])
arr_freq = np.zeros([len(IDs),maxmodes])
arr_freqerr = np.zeros([len(IDs),maxmodes])

dnu_avgID = []
for i in np.arange(0,len(IDs),1):
    modesID[i] = modesID[i].sort_values(by=['f0'])
    modesID[i] = modesID[i].set_index(np.arange(0,len(modesID[i]),1))
    modesID[i]['dnu'] = (modesID[i]['f0'].diff(2).shift(-1))/2
    dnu_avg = (np.mean(modesID[i]['dnu']))
    dnu_avgID.append(dnu_avg)

    n_min = int(modesID[i]['f0'].min() / dnu_avg)
    n = np.arange(n_min, n_min+len(modesID[i]), 1)
    modesID[i].insert(loc=0, column='n', value=n)

    # Loop to ensure all arrays are the same length
    if lenmodes[i] < maxmodes:
        l = lenmodes[i]
        while l < maxmodes:
            newrow = {'n': int(np.max(modesID[i]['n'])+1),
                      'f0': np.max(modesID[i]['f0'])+dnu_avgID[i],
                      'f0_err': 100000}
            modesID[i] = modesID[i].append(newrow, ignore_index=True)
            l += 1

    arr_n[i,:] = modesID[i]['n']
    arr_freq[i,:] = modesID[i]['f0']
    arr_freqerr[i,:] = modesID[i]['f0_err']

    #display(modesID[i])

dnu_avgarr = np.asarray(dnu_avgID)

#Define start parameters
eps = []
for i in range(len(IDs)):
    epsilon1 = np.median((modesID[i].f0 % dnu_avg) / dnu_avg)
    eps.append(epsilon1)
epsilon = np.asarray(eps)

numax_obs = Numax
nmax = numax_obs/dnu_avgarr - epsilon
alpha = 0.015*dnu_avgarr**(-0.32)
A = 0.06*dnu_avgarr**(-0.88)
G = np.ones([len(IDs)])*3.08
tau = np.ones([len(IDs)])*10
phi = np.ones([len(IDs)])*1.71
```

The data for the different stars can be viewed on an Échelle Diagram.

```python
plt.figure(1)
for i in np.arange(0,len(IDs),1):
    plt.scatter(modesID[i]['f0'] % dnu_avgID[i], modesID[i]['f0'], label=r'ID = '+str(IDs[i]))

plt.ylabel('Frequency')
plt.xlabel(r'Mod. Freq. Spacing ('+ str(u"%.3f" % dnu_avg) +') $\mu$Hz')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.5)
#plt.savefig('seminar/manyrgbs.pdf')

plt.figure(2)
for i in np.arange(0,len(IDs),1):
    plt.scatter(modesID[i]['n'], modesID[i]['f0'], label=r'ID = '+str(IDs[i]))
plt.xlabel('Radial Order ($n$)')
plt.ylabel('Frequency')
plt.show()
```

With all our required data, we can parse the data into a stan model constructed to perform a hierarchical Bayesian model. Hyperpriors have been introduced on a number of parameters based on values in the Vrard (2014) paper.

The model follows the form:
\begin{equation}
    \left(n + \epsilon + \frac{\alpha}{2}\left(n-n_{max}\right)^2 + \frac{\mathcal{AG}}{2\pi}\sin\left(\frac{2\pi(n-n_{max})}{\mathcal{G}} + \phi \right)\right) e^{\frac{n}{\tau}}
\end{equation}
where
\begin{equation}
    n_{max} = \frac{\nu_{max}}{\Delta\nu} - \epsilon
\end{equation}

$$ A = A_0 e^{-n_{max}/\tau} \longrightarrow A_0 = A_{n_{max}} e^{n_{max}/\tau}$$

```python
code = '''
functions {
    real glitch(real n, real dnu, real numax, real epsilon, real alpha, real A, real G, real phi, real tau){
        real nmax = numax/dnu - epsilon;
        return (n + epsilon + alpha/2 * (n-nmax)^2 +
                A*G/(2*pi()) * sin((2*pi()*(n-nmax))/G + phi)) * dnu * exp(-(n-nmax)/tau);
    }
}
data {
    int N;  //number of stars
    int M; //number of modes
    real n[N, M];
    real freq[N, M];
    real freq_err[N, M];
    real numax_obs[N];
    real numax_err[N];
    real dnu_guess[N];
}
parameters {
    real dnu[N];
    real numax[N];
    real<lower = -2.0*pi(), upper = 2.0*pi()> phi[N];
    //real<lower = 0> A[N];
    real epsilon[N];
    real<lower = 0> Anmax[N];
    real G[N];
    real<lower = 0> tau[N];

    // Hierarchical Parameters
    //real epsilon_std[N];
    //real<lower=0> eps_std;
    //real epsA;
    //real epsB;
    real alpha_std[N];
    real<lower=0> al_std;
    real alA;
    real alB;
    //real A_std[N];
    //real<lower=0> A_err;
    //real AA;
    //real AB;
    //real G_std[N];
    //real<lower=0> G_err;
    //real GA;

}

transformed parameters {
    //real epsilon[N];
    real alpha[N];
    //real A[N];
    //real G[N];
    real A0[N];

    for (i in 1:N){
        //epsilon[i] = epsilon_std[i] * eps_std + (epsA + epsB * log(dnu[i]));
        alpha[i] = alpha_std[i] * al_std + (alA * (dnu[i])^(-alB));
        //A[i] = A_std[i] * A_err + (AA * (dnu[i])^(-AB));
        //G[i] = G_std[i] * G_err + GA;
        A0[i] = Anmax[i] * exp((numax[i]/dnu[i] - epsilon[i])/tau[i]);
    }
}

model {
    real mod[M];
    for (i in 1:N){
        for (j in 1:M){
            mod[j] = glitch(n[i,j], dnu[i], numax[i], epsilon[i], alpha[i], A0[i], G[i], phi[i], tau[i]);
            // mod[j] ~ normal(freq[i], freq_err[i]);
        }
        freq[i,:] ~ normal(mod, freq_err[i,:]);
        dnu[i] ~ normal(dnu_guess[i], dnu_guess[i]*0.001);
        Anmax[i] ~ lognormal(log(0.06*dnu[i]^(-0.88)), 0.4);
        epsilon[i] ~ normal(0.601 + 0.632*log10(dnu[i]), 0.5);
    }

    //epsilon ~ uniform(-1.0, 2.0);
    //nmax ~ normal(10, 4);
    //alpha ~ lognormal(log(0.015*dnu^(-0.32)), 0.3);

    G ~ normal(3.08, 0.65);

    numax ~ normal(numax_obs, numax_err);

    tau ~ normal(10, 10);

    // Hierarchical Hyperpriors
    //epsilon_std ~ normal(0, 1);
    //eps_std ~ normal(0, 0.5);
    //epsA ~ normal(0.601, 0.25);
    //epsB ~ normal(0.632, 0.25);
    alpha_std ~ normal(0, 1);
    al_std ~ normal(0, 0.5);
    alA ~ normal(0.015, 0.005);
    alB ~ normal(0.32, 0.08);
    //A_std ~ normal(0, 1);
    //A_err ~ normal(0, 0.5);
    //AA ~ normal(0.06, 0.02);
    //AB ~ normal(0.88, 0.05);
    //G_std ~ normal(0, 1);
    //G_err ~ normal(0.65, 0.5);
    //GA ~ normal(3.08, 0.1);
}
'''
sm = pystan.StanModel(model_code=code)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_cf9481015a01108502163a589ab98401 NOW.

```python
no_tau = pd.read_csv('no_tau_models.csv', names=['dnu', 'numax', 'epsilon', 'alpha', 'A', 'G', 'phi', 'tau'])

stan_data = {'N': len(IDs),
         'M': maxmodes,
         'n': arr_n,
         'freq': arr_freq,
         'freq_err': arr_freqerr,
         'numax_obs': Numax,
         'numax_err': Numax_err,
         'dnu_guess': dnu_avgarr
        }
start = {'dnu': dnu_avgarr,
         'numax': Numax,
         'eps_std': 0.01,
         'al_std': 0.01,
         'A_err': 0.01,
         #'G_err': 0.65,
         'epsilon': no_tau['epsilon'].values,
         'alpha': no_tau['alpha'].values,
         #'A0': no_tau['A'].values,
         'G': no_tau['G'].values,
         'phi': no_tau['phi'].values,
         #'epsA': 0.601,
         #'epsB': 0.632,
         'alA': 0.015,
         'alB': 0.32,
         #'AA': 0.06,
         #'AB': 0.88,
         #'GA': 3.08
         'tau': tau
    }
nchains = 4

fit = sm.sampling(data=stan_data, iter=5000, chains=nchains, init=[start for n in range(nchains)])
```

    WARNING:pystan:n_eff / iter for parameter dnu[6] is 0.000803667072128739!
    WARNING:pystan:n_eff / iter for parameter numax[3] is 0.0002939991322524567!
    WARNING:pystan:n_eff / iter for parameter phi[3] is 0.00028714195023308664!
    WARNING:pystan:n_eff / iter for parameter phi[4] is 0.00021163273606235834!
    WARNING:pystan:n_eff / iter for parameter phi[5] is 0.000627734710984984!
    WARNING:pystan:n_eff / iter for parameter phi[6] is 0.00021906152380652726!
    WARNING:pystan:n_eff / iter for parameter epsilon[3] is 0.0002996592295814148!
    WARNING:pystan:n_eff / iter for parameter epsilon[6] is 0.0009458853227326667!
    WARNING:pystan:n_eff / iter for parameter Anmax[4] is 0.0005249897668759926!
    WARNING:pystan:n_eff / iter for parameter Anmax[5] is 0.0002586545925958755!
    WARNING:pystan:n_eff / iter for parameter Anmax[6] is 0.0002047872096681584!
    WARNING:pystan:n_eff / iter for parameter G[3] is 0.0003138239153518612!
    WARNING:pystan:n_eff / iter for parameter G[4] is 0.00020548854706328945!
    WARNING:pystan:n_eff / iter for parameter G[5] is 0.00020392659954961804!
    WARNING:pystan:n_eff / iter for parameter G[6] is 0.00020064500168141732!
    WARNING:pystan:n_eff / iter for parameter tau[3] is 0.0009112483559484401!
    WARNING:pystan:n_eff / iter for parameter tau[6] is 0.00021415533086381594!
    WARNING:pystan:n_eff / iter for parameter alpha_std[3] is 0.00032988067377525744!
    WARNING:pystan:n_eff / iter for parameter alpha_std[6] is 0.0006334445358226462!
    WARNING:pystan:n_eff / iter for parameter alpha[3] is 0.0002949070540381632!
    WARNING:pystan:n_eff / iter for parameter alpha[6] is 0.0003673384007587611!
    WARNING:pystan:n_eff / iter for parameter A0[4] is 0.000526238642008954!
    WARNING:pystan:n_eff / iter for parameter A0[5] is 0.00025866402339226477!
    WARNING:pystan:n_eff / iter for parameter A0[6] is 0.00020563335146305258!
    WARNING:pystan:n_eff / iter for parameter lp__ is 0.00021471160584088608!
    WARNING:pystan:n_eff / iter below 0.001 indicates that the effective sample size has likely been overestimated
    WARNING:pystan:Rhat for parameter dnu[6] is 1.1473182959374602!
    WARNING:pystan:Rhat for parameter numax[3] is 2.2143559959067933!
    WARNING:pystan:Rhat for parameter phi[3] is 2.285310394133709!
    WARNING:pystan:Rhat for parameter phi[4] is 4.05969478014307!
    WARNING:pystan:Rhat for parameter phi[5] is 1.2016191256203406!
    WARNING:pystan:Rhat for parameter phi[6] is 3.3078525426673275!
    WARNING:pystan:Rhat for parameter epsilon[3] is 2.0724139684759475!
    WARNING:pystan:Rhat for parameter epsilon[6] is 1.1519590800273134!
    WARNING:pystan:Rhat for parameter Anmax[4] is 1.2571251451572731!
    WARNING:pystan:Rhat for parameter Anmax[5] is 2.0150119225734424!
    WARNING:pystan:Rhat for parameter Anmax[6] is 6.439264784751745!
    WARNING:pystan:Rhat for parameter G[3] is 2.039909413370057!
    WARNING:pystan:Rhat for parameter G[4] is 5.7971617443406815!
    WARNING:pystan:Rhat for parameter G[5] is 6.840000376970124!
    WARNING:pystan:Rhat for parameter G[6] is 17.14401909831593!
    WARNING:pystan:Rhat for parameter tau[3] is 1.1711170483756155!
    WARNING:pystan:Rhat for parameter tau[6] is 3.696976774582099!
    WARNING:pystan:Rhat for parameter alpha_std[3] is 1.8888368602742591!
    WARNING:pystan:Rhat for parameter alpha_std[6] is 1.2082130914823273!
    WARNING:pystan:Rhat for parameter alpha[3] is 2.1674512132440227!
    WARNING:pystan:Rhat for parameter alpha[6] is 1.4463994925633896!
    WARNING:pystan:Rhat for parameter A0[4] is 1.2563569923405402!
    WARNING:pystan:Rhat for parameter A0[5] is 2.014841757862956!
    WARNING:pystan:Rhat for parameter A0[6] is 5.94433771508894!
    WARNING:pystan:Rhat for parameter lp__ is 4.3197350515638675!
    WARNING:pystan:Rhat above 1.1 or below 0.9 indicates that the chains very likely have not mixed
    WARNING:pystan:7 of 10000 iterations ended with a divergence (0.07%).
    WARNING:pystan:Try running with adapt_delta larger than 0.8 to remove the divergences.
    WARNING:pystan:2500 of 10000 iterations saturated the maximum tree depth of 10 (25.0%)
    WARNING:pystan:Run again with max_treedepth larger than 10 to avoid saturation

```python
print(fit)
```

    Inference for Stan model: anon_model_6cc1e34c5a01d1e0b2b90681b89693fe.
    4 chains, each with iter=5000; warmup=2500; thin=1;
    post-warmup draws per chain=2500, total post-warmup draws=10000.

                   mean se_mean     sd   2.5%    25%    50%    75%  97.5%  n_eff   Rhat
    dnu[1]         5.49  1.1e-3 5.8e-3   5.48   5.48   5.49   5.49    5.5     28   1.06
    dnu[2]         5.42  1.2e-4 5.3e-3   5.41   5.42   5.42   5.42   5.43   1810   1.01
    dnu[3]         6.45  6.2e-3   0.02   6.43   6.44   6.44   6.45   6.51      7    1.6
    dnu[4]         4.77  1.1e-4 4.7e-3   4.77   4.77   4.77   4.78   4.78   1745    1.0
    dnu[5]         8.64  9.8e-4 6.2e-3   8.63   8.64   8.64   8.65   8.65     39   1.07
    dnu[6]         6.17  2.3e-3 6.8e-3   6.15   6.16   6.17   6.17   6.18      9   1.14
    numax[1]      54.97    1.33   1.91  52.59  53.09  54.89  56.85  57.53      2   5.51
    numax[2]      52.36  8.6e-3   0.27  51.82  52.18  52.37  52.55   52.9   1025   1.01
    numax[3]      65.91    3.09   4.65  59.42  61.23  68.94  70.25  71.52      2   3.28
    numax[4]       40.9  7.2e-3   0.27  40.37  40.72   40.9  41.08  41.44   1418   1.01
    numax[5]      95.09    0.06    0.3   94.5  94.89  95.09  95.31  95.68     22    1.1
    numax[6]      67.64    0.02   0.33  66.95  67.44  67.64  67.84  68.29    190   1.03
    phi[1]         2.89    1.53   2.18   0.61   0.75   1.99   5.05   5.93      2   6.09
    phi[2]        -3.51    0.61   0.87  -4.33  -4.08  -3.93   -2.9  -1.83      2   5.49
    phi[3]        -0.06    0.44   0.81  -1.02  -0.73  -0.56   0.92   1.01      3   2.07
    phi[4]        -2.74    0.73   1.09  -4.88  -3.55  -2.42  -1.75  -1.35      2   3.18
    phi[5]         2.95    1.31   1.86   1.52   1.79   1.97   4.41   6.24      2  13.32
    phi[6]         -0.1    0.97   2.13  -5.87  -0.29  -0.06   1.47   2.92      5   1.43
    epsilon[1]    -0.01    0.07   0.11  -0.15  -0.11  -0.06   0.08    0.2      3   2.14
    epsilon[2]     0.13  2.8e-4   0.01   0.11   0.13   0.13   0.14   0.15   1524   1.01
    epsilon[3]     0.22    0.11   0.17-5.9e-4   0.05    0.3   0.38   0.45      2    2.7
    epsilon[4]     0.89  2.6e-4   0.01   0.87   0.89   0.89    0.9   0.91   1590    1.0
    epsilon[5]    -0.09  3.8e-4 7.9e-3   -0.1  -0.09  -0.09  -0.08  -0.07    438   1.02
    epsilon[6]     0.13    0.03   0.07   0.03   0.09   0.12   0.16    0.3      6   1.28
    Anmax[1]       1.16    0.73   1.04   0.11   0.13   0.78   2.16   2.69      2   5.39
    Anmax[2]       0.25    0.02   0.04   0.19   0.22   0.25   0.28   0.35      5   1.32
    Anmax[3]       0.11    0.01   0.03   0.03    0.1   0.11   0.12   0.14      6   1.97
    Anmax[4]       0.16    0.05   0.08   0.08   0.11   0.13   0.19   0.36      2   2.47
    Anmax[5]       0.03  6.6e-3   0.01 5.3e-3   0.02   0.03   0.04   0.04      2   2.79
    Anmax[6]      11.47   12.85  18.32   0.09    0.1   0.98  18.46  49.06      2   8.11
    G[1]           4.38    2.35   3.33   1.05   1.06   3.72   7.67   8.39      2  12.55
    G[2]           2.11     0.1   0.15   1.83    2.0   2.18    2.2   2.25      2   5.77
    G[3]          11.16    1.99   3.18   3.01   9.09  13.02  13.82  14.51      3   3.32
    G[4]           1.63    0.43    0.6   0.63   1.12   1.81   2.02   2.34      2   19.2
    G[5]           1.44    0.69   0.97  -0.17   0.69   1.67   2.11   2.63      2  25.61
    G[6]           6.36    4.12   5.83   0.05    0.5   6.04  12.17  12.74      2  26.03
    tau[1]        91.22    10.9  16.52  65.59  75.74  92.04 106.28 116.98      2   2.69
    tau[2]       175.12    0.14   6.03  163.6 171.02  175.0 179.21 187.07   1929    1.0
    tau[3]       153.29    9.25  22.42  132.0 141.24 147.62 155.38 234.88      6   1.96
    tau[4]       159.35     0.1   5.76 148.49 155.41 159.23 163.15 171.14   3163    1.0
    tau[5]       312.07    0.54   5.32 301.68 308.41 312.02 315.88 322.16     97   1.04
    tau[6]       104.67   14.07  20.77   70.2  85.44 108.91 123.81 133.85      2   3.38
    alpha_std[1]   1.28    0.33   0.66   0.33   0.77   1.18    1.7   2.76      4   1.43
    alpha_std[2]   0.79    0.18   0.38   0.26   0.51   0.71   1.01   1.73      5   1.36
    alpha_std[3]   0.31    0.59   1.02  -1.23  -0.46  -0.08   1.12    2.4      3   1.94
    alpha_std[4]   0.69    0.16   0.33   0.22   0.44   0.62   0.87    1.5      5   1.36
    alpha_std[5]   0.18    0.04   0.11   0.02    0.1   0.16   0.24   0.44      7   1.19
    alpha_std[6]   0.34    0.16    0.3  -0.13   0.14   0.29   0.48   1.07      4   1.52
    al_std         0.06    0.01   0.03   0.02   0.04   0.05   0.07   0.14      7   1.22
    alA            0.02  7.4e-5 5.1e-3 6.4e-3   0.01   0.02   0.02   0.03   4660    1.0
    alB            0.31  1.2e-3   0.08   0.15   0.25   0.31   0.36   0.47   4553    1.0
    alpha[1]       0.08    0.03   0.04   0.03   0.04   0.07   0.12   0.13      2   4.87
    alpha[2]       0.05  2.6e-5 1.8e-3   0.04   0.04   0.05   0.05   0.05   4584    1.0
    alpha[3]       0.03    0.03   0.05  -0.03  -0.01 4.9e-3   0.08   0.09      3   2.67
    alpha[4]       0.04  3.9e-5 1.6e-3   0.04   0.04   0.04   0.04   0.04   1586   1.01
    alpha[5]       0.02  3.2e-4 7.3e-4   0.02   0.02   0.02   0.02   0.02      5   1.27
    alpha[6]       0.02  5.4e-3 9.8e-3 5.3e-3   0.02   0.03   0.03   0.04      3   1.62
    A0[1]          1.32    0.84    1.2   0.12   0.15   0.87   2.47   3.13      2   5.02
    A0[2]          0.27    0.02   0.04    0.2   0.24   0.26    0.3   0.36      5   1.32
    A0[3]          0.12    0.01   0.03   0.03   0.11   0.12   0.13   0.16      5   1.99
    A0[4]          0.17    0.05   0.08   0.08   0.11   0.14    0.2   0.37      2   2.46
    A0[5]          0.03  6.9e-3   0.01 5.5e-3   0.02   0.03   0.04   0.04      2   2.79
    A0[6]         12.94   14.47  20.67    0.1   0.11   1.09  20.33  55.83      2   7.55
    lp__          -2464   92.34 141.57  -2865  -2533  -2436  -2363  -2305      2   4.12

    Samples were drawn using NUTS at Fri Feb 15 10:51:52 2019.
    For each parameter, n_eff is a crude measure of effective sample size,
    and Rhat is the potential scale reduction factor on split chains (at
    convergence, Rhat=1).

```python
fit.plot()
plt.show()
```

```python
#%matplotlib inline
def model(n,dnu,numax,eps,alpha,A,G,phi,tau):
    nmax = numax/dnu - eps
    s = (n + eps + ((alpha/2)*(nmax-n)**2) +
         (((A*G)/(2*np.pi)) * np.sin((2*np.pi*(n-nmax))/G + phi)))*dnu*np.exp(-(n-nmax)/(tau))
    return s

fig = plt.figure(1, figsize=(12,4))
ax  = fig.add_subplot(111)
ax.set_position([0.1,0.1,0.6,0.85])

import itertools
colours = itertools.cycle(('r', 'orange', 'olive', 'teal', 'sienna', 'indigo'))

for i in np.arange(0,len(IDs),1):
    # Iterate point colour
    colour = next(colours)
    # Plot all frequencies, and errorbars for the 'true' data
    ax.scatter(modesID[i]['f0'] % dnu_avgID[i], modesID[i]['f0'],
                label='', zorder=-1, facecolors='none', edgecolors=colour)
    pts = modesID[i].loc[(modesID[i]['f0_err'] < 10e3)]
    ax.scatter(pts['f0'] % dnu_avgID[i], pts['f0'],
                label=r'ID = '+str(IDs[i]), c=colour)
    ax.errorbar(pts['f0'] % dnu_avgID[i], pts['f0'], xerr=pts['f0_err'],
                        zorder=0, fmt="none", label='', c='k',
                        capsize=2, markersize=4, elinewidth=1)

    # Plot the modelled fit
    g2 = model(arr_n[i],
           np.mean(fit['dnu'], axis=0)[i],
           np.mean(fit['numax'], axis=0)[i],
           np.mean(fit['epsilon'], axis=0)[i],
           np.mean(fit['alpha'], axis=0)[i],
           np.mean(fit['A0'], axis=0)[i],
           np.mean(fit['G'], axis=0)[i],
           np.mean(fit['phi'], axis=0)[i],
           np.mean(fit['tau'], axis=0)[i]
              )

    ax.plot(g2 % np.mean(fit['dnu'], axis=0)[i], g2,
             c=colour)
'''
g1 = model(arr_n[0],
           np.mean(fit['dnu'], axis=0)[0],
           np.mean(fit['nmax'], axis=0)[0],
           np.mean(fit['epsilon'], axis=0)[0],
           np.mean(fit['alpha'], axis=0)[0],
           np.mean(fit['A'], axis=0)[0],
           np.mean(fit['G'], axis=0)[0],
           np.mean(fit['phi'], axis=0)[0],
           np.mean(fit['tau'], axis=0)[0])

plt.plot(g1 % np.mean(fit['dnu'], axis=0)[0], g1,
             label='Fit_2D')

'''

print(arr_n[0])
print(np.mean(fit['dnu'], axis=0))
print(dnu_avgID)
print(np.mean(fit['numax'], axis=0))
print(np.mean(fit['epsilon'], axis=0))
print(np.mean(fit['alpha'], axis=0))
print(np.mean(fit['A0'], axis=0))
print(np.mean(fit['G'], axis=0))
print(np.mean(fit['phi'], axis=0))
#print(np.mean(fit['tau'], axis=0))

plt.legend()
plt.ylabel('Frequency')
plt.xlabel(r'Mod. Freq. Spacing ($\mu$Hz)')
plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
#plt.savefig('seminar/somergbseizure.pdf', bbox='tight_layout')
```

```python
%matplotlib inline
plt.figure(2)
for i in np.arange(0,len(IDs),1):
    plt.scatter(modesID[i]['n'], modesID[i]['f0'], label=r'ID = '+str(IDs[i]))
plt.xlabel('Radial Order ($n$)')
plt.ylabel('Frequency')
plt.legend()
plt.show()

fig, ax = plt.subplots(len(IDs))
for i in np.arange(0,len(IDs),1):
    ax[i].errorbar(modesID[i]['n'], modesID[i]['f0'] - g2, yerr=modesID[i]['f0_err']*10)
    #ax[i].set_xlim([6,20])
    #ax[i].set_ylim([-3.5, 3.5])
    ax[i].set_ylim([-15, 15])

plt.show()
```

```python
no_osc = np.loadtxt('/home/daniel/Documents/y4project/Dan_notebooks/no_osc_models.csv', delimiter=',')

fig = plt.figure(1, figsize=(12,4))
ax  = fig.add_subplot(111)
ax.set_position([0.1,0.1,0.6,0.85])

import itertools
colours = itertools.cycle(('r', 'orange', 'olive', 'teal', 'sienna', 'indigo'))

osc = []
for i in np.arange(0,len(IDs),1):
    colour = next(colours)
    ax.scatter(modesID[i]['f0'] % dnu_avgID[i], modesID[i]['f0'],
                label='', zorder=-1, facecolors='none', edgecolors=colour)
    pts = modesID[i].loc[(modesID[i]['f0_err'] < 10e3)]
    ax.scatter(pts['f0'] % dnu_avgID[i], pts['f0'],
                label=r'ID = '+str(IDs[i]), c=colour)
    ax.errorbar(pts['f0'] % dnu_avgID[i], pts['f0'], xerr=pts['f0_err'],
                        zorder=0, fmt="none", label='', c='k',
                        capsize=2, markersize=4, elinewidth=1)

    # Plot the modelled fit
    g2 = model(arr_n[i],
           np.mean(fit['dnu'], axis=0)[i],
           np.mean(fit['numax'], axis=0)[i],
           np.mean(fit['epsilon'], axis=0)[i],
           np.mean(fit['alpha'], axis=0)[i],
           np.mean(fit['A0'], axis=0)[i],
           np.mean(fit['G'], axis=0)[i],
           np.mean(fit['phi'], axis=0)[i],
           np.mean(fit['tau'], axis=0)[i]
              )
    ax.plot(g2 % np.mean(fit['dnu'], axis=0)[i], g2,
             c=colour)
    ax.plot(no_osc[i] % dnu_avgID[i], no_osc[i], linestyle=':', c=colour)
    osc.append(g2)

colours = itertools.cycle(('r', 'orange', 'olive', 'teal', 'sienna', 'indigo'))
fig, ax = plt.subplots(len(IDs), figsize=(5,14))
for i in np.arange(0,len(IDs),1):
    #ax[i].scatter(no_osc[i], (no_osc[i] % dnu_avgID[i]))
    #ax[i].scatter(osc[i], (osc[i] % dnu_avgID[i]))
    ax[i].scatter(no_osc[i], (no_osc[i] % dnu_avgID[i])-(osc[i] % np.mean(fit['dnu'], axis=0)[i]), c=next(colours))
plt.xlabel(r'Freqency ($\mu$Hz)')
plt.show()
```

```python
no_tau = np.loadtxt('/home/daniel/Documents/y4project/Dan_notebooks/no_tau_models.csv', delimiter=',')
no_tau = pd.read_csv('no_tau_models.csv', names=['dnu', 'numax', 'epsilon', 'alpha', 'A', 'G', 'phi', 'tau'])
no_tau
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>dnu</th>
      <th>numax</th>
      <th>epsilon</th>
      <th>alpha</th>
      <th>A</th>
      <th>G</th>
      <th>phi</th>
      <th>tau</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>5.483315</td>
      <td>57.263652</td>
      <td>0.029768</td>
      <td>0.014692</td>
      <td>0.037512</td>
      <td>2.700290</td>
      <td>0.852972</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>1</th>
      <td>5.394925</td>
      <td>59.846463</td>
      <td>0.169461</td>
      <td>0.013854</td>
      <td>0.077494</td>
      <td>2.586622</td>
      <td>-0.043143</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6.449046</td>
      <td>67.676914</td>
      <td>0.169412</td>
      <td>0.013481</td>
      <td>0.029490</td>
      <td>2.673891</td>
      <td>1.557981</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>3</th>
      <td>4.747841</td>
      <td>45.746180</td>
      <td>0.917655</td>
      <td>0.013181</td>
      <td>0.038654</td>
      <td>3.097227</td>
      <td>0.140495</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>4</th>
      <td>8.432162</td>
      <td>98.099118</td>
      <td>0.188873</td>
      <td>0.011033</td>
      <td>0.021136</td>
      <td>2.814720</td>
      <td>2.384268</td>
      <td>NaN</td>
    </tr>
    <tr>
      <th>5</th>
      <td>6.166735</td>
      <td>67.642849</td>
      <td>0.103137</td>
      <td>0.011543</td>
      <td>0.039744</td>
      <td>2.377497</td>
      <td>2.624294</td>
      <td>NaN</td>
    </tr>
  </tbody>
</table>
</div>

```python

```


```python
# Erasmus+ ICCT project (2018-1-SI01-KA203-047081)

# Toggle cell visibility

from IPython.display import HTML
tag = HTML('''
Promijeni vidljivost <a href="javascript:code_toggle()">ovdje</a>.''')
display(tag)
```

Promijeni vidljivost ovdje.

```python
%matplotlib notebook
import numpy as np
import control as control
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from ipywidgets import widgets
from ipywidgets import interact
import scipy.signal as signal
import sympy as sym

```

## Mehanički sustavi

#### Opći model masa-opruga-prigušivač
> Model masa-opruga-prigušivač sastoji se od diskretnih čvorova mase raspoređenih po objektu i međusobno povezanih mrežom opruga i prigušivača. Ovaj je model pogodan za opisivanje objekata sa složenim svojstvima materijala kao što su nelinearnost i viskoelastičnost. (izvor: [Wikipedia](https://en.wikipedia.org/wiki/Mass-spring-damper_model "Mass-spring-model"))
#### Model 1/4 automobila
> Model 1/4 automobila se koristi za analizu kvalitete automobilskih sustava ovjesa. Masa $m_1$ je "opružena masa", koja predstavlja jednu četvrtinu mase vozila koju podupire sustav ovjesa. Masa $m_2$ je "neopružena masa", koja predstavlja opterećenje jednog kotača i poluosovinskog sklopa, uključujući amortizer i opruge. Krutost i prigušenje sustava ovjesa modeliraju se idealnom konstantom opruge $k_1$ i koeficijentom trenja $B$, respektivno. Krutost gume modelirana je konstantom opruge $k_2$. (izvor: [Chegg Study](https://www.chegg.com/homework-help/questions-and-answers/figure-p230-shows-1-4-car-model-used-analyze-ride-quality-automotive-suspension-systems-ma-q26244005 "1/4 car model"))

---

### Kako koristiti ovaj interaktivni primjer?
1. Izaberite između ponuđenih modela *masa-opruga-prigušivač* i *1/4 automobila* klikom na odgovarajući gumb.
2. Odaberite funkciju sile $F$ među ponuđenima: *step funkcija*, *impulsna funkcija*, *rampa funkcija* i *funkcija sinus*.
3. Pomičite klizače da biste definirali vrijednosti mase ($m$; $m_1$ i $m_2$), koeficijenata opruge ($k$; $k_1$ i $k_2$), konstante prigušenja ($B$), pojačanje ulaznog signala i početne uvjete ($x_0$, $\dot{x}_0$, $y_0$, $\dot{y}_0$).

        Model masa-opruga-prigušivač
        Model 1/4 automobila

```python
# create figure
fig = plt.figure(figsize=(9.8, 4),num='Mehanički sustavi')

# add sublot
ax = fig.add_subplot(111)
ax.set_title('Vremenski odziv')
ax.set_ylabel('ulaz, izlaz')
ax.set_xlabel('$t$ [s]')

ax.grid(which='both', axis='both', color='lightgray')

inputf, = ax.plot([], [])
responsef, = ax.plot([], [])
responsef2, = ax.plot([], [])
arrowf, = ax.plot([],[])

style = {'description_width': 'initial'}

selectSystem=widgets.ToggleButtons(
    options=[('masa-opruga-prigušivač',0),('1/4 automobila',1)],
    description='Odaberi sustav: ', style=style) # define toggle buttons
selectForce = widgets.ToggleButtons(
    options=[('step funkcija', 0), ('impulsna funkcija', 1), ('rampa funkcija', 2), ('funkcija sinus', 3)],
    description='Odaberi $F$ funkciju: ', style=style)
display(selectSystem)
display(selectForce)

def build_model(M,K,B,M1,M2,B1,K1,K2,amp,x0,xpika0,y0,ypika0,select_System,index):

    num_of_samples = 1000
    total_time = 25
    t = np.linspace(0, total_time, num_of_samples) # time for which response is calculated (start, stop, step)

    global inputf, responsef, responsef2, arrowf

    if select_System==0:

        system0 = control.TransferFunction([1], [M, B, K])

        if index==0:
            inputfunc = np.ones(len(t))*amp
            inputfunc[0]=0
            time, response, xx = control.forced_response(system0, t, inputfunc, X0=[xpika0,x0])

        elif index==1:
            inputfunc=signal.unit_impulse(1000, 0)*amp
            time, response, xx = control.forced_response(system0, t, inputfunc, X0=[xpika0,x0])

        elif index==2:
            inputfunc=t;
            time, response, xx = control.forced_response(system0, t, inputfunc, X0=[xpika0,x0])

        elif index==3:
            inputfunc=np.sin(t)*amp
            time, response, xx = control.forced_response(system0, t, inputfunc, X0=[xpika0,x0])

    elif select_System==1:

        system1=control.TransferFunction([M2, B1, K1+K2], [M1*M2, M1*B1+M2*B1, M2*K1+M1*(K1+K2), K2*B1, K1*K2])
        system2=control.TransferFunction([B1*K1*M2**2, B1**2*K1*M2, B1*K1**2*M2 + 2*B1*K1*K2*M2,
                                          B1**2*K1*K2, B1*K1**2*K2 + B1*K1*K2**2],
                                         [M1**2*M2**2, B1*M1**2*M2 + 2*B1*M1*M2**2,
                                          B1**2*M1*M2 + B1**2*M2**2 + K1*M1**2*M2 + 2*K1*M1*M2**2 + 2*K2*M1**2*M2 + K2*M1*M2**2,
                                          2*B1*K1*M1*M2 + 2*B1*K1*M2**2 + B1*K2*M1**2 + 5*B1*K2*M1*M2 + B1*K2*M2**2,
                                          B1**2*K2*M1 + 2*B1**2*K2*M2 + K1**2*M1*M2 + K1**2*M2**2 + K1*K2*M1**2 + 5*K1*K2*M1*M2 + K1*K2*M2**2 + K2**2*M1**2 + 2*K2**2*M1*M2,
                                          2*B1*K1*K2*M1 + 4*B1*K1*K2*M2 + 3*B1*K2**2*M1 + 2*B1*K2**2*M2,
                                          B1**2*K2**2 + K1**2*K2*M1 + 2*K1**2*K2*M2 + 3*K1*K2**2*M1 + 2*K1*K2**2*M2 + K2**3*M1,
                                          2*B1*K1*K2**2 + B1*K2**3,
                                          K1**2*K2**2 + K1*K2**3])
        if index==0:
            inputfunc = np.ones(len(t))*amp
            inputfunc[0]=0
            time, response, xx = control.forced_response(system1, t, inputfunc, X0=[0,0,xpika0,x0])
            time2, response2, xx2 = control.forced_response(system2, t, inputfunc, X0=[0,0,0,0,0,0,ypika0,y0])

        elif index==1:
            inputfunc=signal.unit_impulse(1000, 0)*amp
            time, response, xx = control.forced_response(system1, t, inputfunc, X0=[0,0,xpika0,x0])
            time2, response2, xx2 = control.forced_response(system2, t, inputfunc, X0=[0,0,0,0,0,0,ypika0,y0])

        elif index==2:
            inputfunc=t;
            time, response, xx = control.forced_response(system1, t, inputfunc, X0=[0,0,xpika0,x0])
            time2, response2, xx2 = control.forced_response(system2, t, inputfunc, X0=[0,0,0,0,0,0,ypika0,y0])

        elif index==3:
            inputfunc=np.sin(t)*amp
            time, response, xx = control.forced_response(system1, t, inputfunc, X0=[0,0,xpika0,x0])
            time2, response2, xx2 = control.forced_response(system2, t, inputfunc, X0=[0,0,0,0,0,0,ypika0,y0])

    ax.lines.remove(responsef)
    ax.lines.remove(inputf)
    ax.lines.remove(responsef2)
    ax.lines.remove(arrowf)

    inputf, = ax.plot(t,inputfunc,label='$F$',color='C0')
    responsef, = ax.plot(time, response,label='$x$',color='C3')

    if select_System==1:
        responsef2, = ax.plot(time, response2,label='$y$',color='C2')
    elif select_System==0:
        responsef2, = ax.plot([],[])

    if index==1:
        if amp>0:
            arrowf, = ax.plot([-0.1,0,0.1],[amp-((amp*0.05)/2),amp,amp-((amp*0.05)/2)],color='C0',linewidth=4)
        elif amp==0:
            arrowf, = ax.plot([],[])
        elif amp<0:
            arrowf, = ax.plot([-0.1,0,0.1],[amp-((amp*0.05)/2),amp,amp-(amp*(0.05)/2)],color='C0',linewidth=4)
    else:
        arrowf, = ax.plot([],[])

    ax.relim()
    ax.autoscale_view()

    ax.legend()

def update_sliders(index):
    global m1_slider, b1_slider, k1_slider, m21_slider, m22_slider, b2_slider, k21_slider, k22_slider
    global x0_slider, xpika0_slider, y0_slider, ypika0_slider

    m1val = [0.1,0.1,0.1,0.1]
    k1val = [1,1,1,1]
    b1val = [0.1,0.1,0.1,0.1]
    m21val = [0.1,0.1,0.1,0.1]
    m22val = [0.1,0.1,0.1,0.1]
    b2val = [0.1,0.1,0.1,0.1]
    k21val = [1,1,1,1]
    k22val = [1,1,1,1]
    x0val = [0,0,0,0]
    xpika0val = [0,0,0,0]
    y0val = [0,0,0,0]
    ypika0val = [0,0,0,0]

    m1_slider.value = m1val[index]
    k1_slider.value = k1val[index]
    b1_slider.value = b1val[index]
    m21_slider.value = m21val[index]
    m22_slider.value = m22val[index]
    b2_slider.value = b2val[index]
    k21_slider.value = k21val[index]
    k22_slider.value = k22val[index]
    x0_slider.value = x0val[index]
    xpika0_slider.value = xpika0val[index]
    y0_slider.value = y0val[index]
    ypika0_slider.value = ypika0val[index]

def draw_controllers(type_select,index):

    global m1_slider, b1_slider, k1_slider, m21_slider, m22_slider, b2_slider, k21_slider, k22_slider
    global x0_slider, xpika0_slider, y0_slider, ypika0_slider

    x0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='$x_0$ [dm]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
    xpika0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='${\dot{x}}_0$ [dm/s]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)

    if type_select==0:

        amp_slider = widgets.FloatSlider(value=1.,min=-2.,max=2.,step=0.1,
            description='Pojačanje ulaznog signala:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',style=style)

        m1_slider = widgets.FloatSlider(value=.1, min=.01, max=1., step=.01,
                                        description='$m$ [kg]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
        k1_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
                                       description='$k$ [N/m]:',disabled=False,continuous_update=False,
                                       orientation='horizontal',readout=True,readout_format='.1f',)
        b1_slider = widgets.FloatSlider(value=.1,min=0.0,max=0.5,step=.01,
                                        description='$B$ [Ns/m]:',disabled=False,continuous_update=False,
                                        rientation='horizontal',readout=True,readout_format='.2f',)
        m21_slider = widgets.FloatSlider(value=.1,min=.01,max=1.,step=.01,
            description='$m_1$ [kg]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        m22_slider = widgets.FloatSlider(value=.1,min=.0,max=1.,step=.01,
            description='$m_2$ [kg]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        b2_slider = widgets.FloatSlider(value=.1,min=0.0,max=2,step=.01,
            description='$B$ [Ns/m]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        k21_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
            description='$k_1$ [N/m]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',
        )
        k22_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
            description='$k_2$ [N/m]:',disabled=True,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',
        )

        y0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='$y_0$ [dm]:',disabled=True,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
        ypika0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='${\dot{y}}_0$ [dm/s]:',disabled=True,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
    elif type_select==1:

        amp_slider = widgets.FloatSlider(value=1.,min=-2.,max=2.,step=0.1,
            description='Pojačanje ulaznog signala:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',style=style)

        m1_slider = widgets.FloatSlider(value=.1, min=.01, max=1., step=.01,
                                        description='$m$ [kg]:',disabled=True,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
        k1_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
                                       description='$k$ [N/m]:',disabled=True,continuous_update=False,
                                       orientation='horizontal',readout=True,readout_format='.1f',)
        b1_slider = widgets.FloatSlider(value=.1,min=0.0,max=0.5,step=.01,
                                        description='$B$ [Ns/m]:',disabled=True,continuous_update=False,
                                        rientation='horizontal',readout=True,readout_format='.2f',)
        m21_slider = widgets.FloatSlider(value=.1,min=.01,max=1.,step=.01,
            description='$m_1$ [kg]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        m22_slider = widgets.FloatSlider(value=.1,min=.0,max=1.,step=.01,
            description='$m_2$ [kg]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        b2_slider = widgets.FloatSlider(value=.1,min=0.0,max=2,step=.01,
            description='$B$ [Ns/m]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.2f',
        )
        k21_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
            description='$k_1$ [N/m]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',
        )
        k22_slider = widgets.FloatSlider(value=1.,min=0.,max=20.,step=.1,
            description='$k_2$ [N/m]:',disabled=False,continuous_update=False,orientation='horizontal',readout=True,readout_format='.1f',
        )

        y0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='$y_0$ [dm]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
        ypika0_slider=widgets.FloatSlider(value=0, min=-1, max=1., step=.1,
                                        description='${\dot{y}}_0$ [dm/s]:',disabled=False,continuous_update=False,
                                        orientation='horizontal',readout=True,readout_format='.2f',)
    input_data = widgets.interactive_output(build_model, {'M':m1_slider, 'K':k1_slider, 'B':b1_slider, 'M1':m21_slider,
                                                    'M2':m22_slider, 'B1':b2_slider, 'K1':k21_slider, 'K2':k22_slider, 'amp':amp_slider,
                                                    'x0':x0_slider,'xpika0':xpika0_slider,'y0':y0_slider,'ypika0':ypika0_slider,
                                                    'select_System':selectSystem,'index':selectForce})

    input_data2 = widgets.interactive_output(update_sliders, {'index':selectForce})

    box_layout = widgets.Layout(border='1px solid black',
                    width='auto',
                    height='',
                    flex_flow='row',
                    display='flex')

    buttons1=widgets.HBox([widgets.VBox([amp_slider],layout=widgets.Layout(width='auto')),
                         widgets.VBox([x0_slider,xpika0_slider]),
                         widgets.VBox([y0_slider,ypika0_slider])],layout=box_layout)
    display(widgets.VBox([widgets.Label('Odaberite vrijednosti pojačanja ulaznog signala i početnih uvjeta:'), buttons1]))
    display(widgets.HBox([widgets.VBox([m1_slider,k1_slider,b1_slider], layout=widgets.Layout(width='45%')),
                          widgets.VBox([m21_slider,m22_slider,k21_slider,k22_slider,b2_slider], layout=widgets.Layout(width='45%'))]), input_data)

widgets.interactive_output(draw_controllers, {'type_select':selectSystem,'index':selectForce})
```

    <IPython.core.display.Javascript object>

    ToggleButtons(description='Odaberi sustav: ', options=(('masa-opruga-prigušivač', 0), ('1/4 automobila', 1)), …

    ToggleButtons(description='Odaberi $F$ funkciju: ', options=(('step funkcija', 0), ('impulsna funkcija', 1), (…

    Output()

```python

```