_Lambda School Data Science_

# Polynomial & Log-Linear Regression

## "Linear" Regression?

Which of the following is a linear regression model?

**All** of these functional forms can be fit using Linear Regression. The "Linear" in linear regression refers to the linear form of the equation.

### Linear Combinations

Remember when we rewrote vectors as a **linear combination** of scalars and unit vectors?

\begin{align}
v = \begin{bmatrix}2 \\ 3\end{bmatrix} = 2 \begin{bmatrix} 1 \\ 0 \end{bmatrix} + 3 \begin{bmatrix} 0 \\ 1\end{bmatrix} = 2\hat{i} + 3\hat{j}
\end{align}

The syntax where we have a scalar (think coefficient) multiplying some vector (unit vector in this case) and all of them being added together is what makes this a linear combination.

### Linear Equations

A "Linear Equation" is any equation that takes the following form:

\begin{align}
a_1x_1 + \ldots + a_nx_n + b = 0
\end{align}

Does this look familiar? A linear equation is one where we have $x_1, \ldots, x_n$ unknowns and $b, a_1, \ldots, a_n$ coefficients which are considered parameters of the equation. "The solutions of such an equation are the values that, when substituted to the unknowns, make the equality true."

[Linear Equation Wikipedia](https://en.wikipedia.org/wiki/Linear_equation)

Linear Regression is **linear** not because it can only plot straight lines and fit straight-line patterns in data, but because the form of the equation used to represent our regression is in the form of a **Linear Equation**.

### _So how do we fit curved data?_

# Example #1: Moore's Law dataset

#### Background
- https://en.wikipedia.org/wiki/Moore%27s_law
- https://en.wikipedia.org/wiki/Transistor_count

#### Scrape HTML tables with Pandas!
- https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_html.html
- https://medium.com/@ageitgey/quick-tip-the-easiest-way-to-grab-data-out-of-a-web-page-in-python-7153cecfca58

#### More web scraping options
- https://automatetheboringstuff.com/chapter11/

### Imports

```python
%matplotlib inline
from ipywidgets import interact
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
import statsmodels.api as sm
```

### Get data

```python
tables = pd.read_html('https://en.wikipedia.org/wiki/Transistor_count', header=0)
print([table.shape for table in tables])
```

    [(139, 6), (6, 6), (2, 5), (80, 6), (13, 7), (27, 3), (0, 2), (15, 7), (29, 2), (0, 2), (3, 2), (2, 2), (2, 2), (5, 2), (7, 2)]

```python
moore = tables[0]
moore
```

      Processor
      Transistor count
      Date of introduction
      Designer
      Process
      Area

      0
      Intel 4004
      2300
      1971
      Intel
      10,000 nm
      12 mm²

      1
      Intel 8008
      3500
      1972
      Intel
      10,000 nm
      14 mm²

      2
      MOS Technology 6502
      3,510[3]
      1975
      MOS Technology
      8,000 nm
      21 mm²

      3
      Motorola 6800
      4100
      1974
      Motorola
      6,000 nm
      16 mm²

      4
      Intel 8080
      4500
      1974
      Intel
      6,000 nm
      20 mm²

      5
      RCA 1802
      5000
      1974
      RCA
      5,000 nm
      27 mm²

      6
      TMS 1000
      8000
      1974[4]
      Texas Instruments
      8,000 nm
      11 mm²

      7
      Intel 8085
      6500
      1976
      Intel
      3,000 nm
      20 mm²

      8
      Zilog Z80
      8500
      1976
      Zilog
      4,000 nm
      18 mm²

      9
      Motorola 6809
      9000
      1978
      Motorola
      5,000 nm
      21 mm²

      10
      Intel 8086
      29000
      1978
      Intel
      3,000 nm
      33 mm²

      11
      Intel 8088
      29000
      1979
      Intel
      3,000 nm
      33 mm²

      12
      WDC 65C02
      11,500[5]
      1981
      WDC
      3,000 nm
      6 mm²

      13
      Intel 80186
      55000
      1982
      Intel
      3,000 nm
      60 mm²

      14
      Motorola 68000
      68,000[citation needed]
      1979
      Motorola
      3,500 nm
      44 mm²

      15
      Intel 80286
      134000
      1982
      Intel
      1,500 nm
      49 mm²

      16
      WDC 65C816
      22,000[6]
      1983
      WDC
      3000 nm[7]
      9 mm²

      17
      Motorola 68020
      190,000[8]
      1984
      Motorola
      2,000 nm
      85 mm²

      18
      Intel 80386
      275000
      1985
      Intel
      1,500 nm
      104 mm²

      19
      ARM 1
      25,000[8]
      1985
      Acorn
      3,000 nm
      50 mm²

      20
      Novix NC4016
      16,000[9]
      1985[10]
      Harris Corporation
      3,000 nm[11]
      NaN

      21
      ARM 2
      30,000[8]
      1986
      Acorn
      2,000 nm
      30 mm²

      22
      Motorola 68030
      273000
      1987
      Motorola
      800 nm
      102 mm²

      23
      TI Explorer's 32-bit Lisp machine chip
      553,000[12]
      1987
      Texas Instruments
      2,000 nm[13]
      NaN

      24
      DEC WRL MultiTitan
      180,000[14]
      1988
      DEC WRL
      1,500 nm
      61 mm²

      25
      Intel i960
      250,000[15]
      1988
      Intel
      600 nm
      NaN

      26
      Intel 80486
      1180235
      1989
      Intel
      1000 nm
      173 mm²

      27
      ARM 3
      310000
      1989
      Acorn
      1,500 nm
      87 mm²

      28
      68040
      1200000
      1990
      Motorola
      650 nm
      152 mm²

      29
      R4000
      1350000
      1991
      MIPS
      1,000 nm
      213 mm²

      ...
      ...
      ...
      ...
      ...
      ...
      ...

      109
      IBM z13
      3990000000
      2015
      IBM
      22 nm
      678 mm²

      110
      12-core POWER8
      4200000000
      2013
      IBM
      22 nm
      650 mm²

      111
      Apple A11 Bionic (hexa-core ARM64 "mobile SoC")
      4300000000
      2017
      Apple
      10 nm
      89.23 mm²

      112
      15-core Xeon Ivy Bridge-EX
      4,310,000,000[42]
      2014
      Intel
      22 nm
      541 mm²

      113
      Zeppelin SoC Ryzen
      4,800,000,000[43]
      2017
      AMD
      14 nm
      192 mm²

      114
      Ryzen 5 1600 Ryzen
      4,800,000,000[44]
      2017
      AMD
      14 nm
      213 mm²

      115
      Ryzen 5 1600 X Ryzen
      4,800,000,000[45]
      2017
      AMD
      14 nm
      213 mm²

      116
      61-core Xeon Phi
      5,000,000,000[46]
      2012
      Intel
      22 nm
      720 mm²

      117
      Xbox One main SoC
      5000000000
      2013
      Microsoft/AMD
      28 nm
      363 mm²

      118
      18-core Xeon Haswell-E5
      5,560,000,000[47]
      2014
      Intel
      22 nm
      661 mm²

      119
      IBM z14
      6100000000
      2017
      IBM
      14 nm
      696 mm²

      120
      Apple A12 Bionic (hexa-core ARM64 "mobile SoC")
      6,900,000,000[48][49]
      2018
      Apple
      7 nm
      83.27 mm2

      121
      HiSilicon Kirin 960 (octa-core ARM64 "mobile S...
      4,000,000,000[50]
      2016
      Huawei
      16 nm
      110.00 mm2

      122
      HiSilicon Kirin 980 (octa-core ARM64 "mobile S...
      6,900,000,000[51]
      2018
      Huawei
      7 nm
      74.13 mm2

      123
      HiSilicon Kirin 970 (octa-core ARM64 "mobile S...
      5,500,000,000[52]
      2017
      Huawei
      10 nm
      96.72 mm2

      124
      HiSilicon Kirin 710 (octa-core ARM64 "mobile S...
      5,500,000,000[53]
      2018
      Huawei
      12 nm
      NaN

      125
      Xbox One X (Project Scorpio) main SoC
      7,000,000,000[54]
      2017
      Microsoft/AMD
      16 nm
      360 mm²[54]

      126
      IBM z13 Storage Controller
      7100000000
      2015
      IBM
      22 nm
      678 mm²

      127
      28-core Xeon Platinum 8180
      8,000,000,000[55]
      2017
      Intel
      14 nm
      NaN

      128
      22-core Xeon Broadwell-E5
      7,200,000,000[56]
      2016
      Intel
      14 nm
      456 mm²

      129
      POWER9
      8000000000
      2017
      IBM
      14 nm
      695 mm²

      130
      72-core Xeon Phi
      8000000000
      2016
      Intel
      14 nm
      683 mm²

      131
      IBM z14 Storage Controller
      9700000000
      2017
      IBM
      14 nm
      696 mm²

      132
      32-core SPARC M7
      10,000,000,000[57]
      2015
      Oracle
      20 nm
      NaN

      133
      Apple A12X Bionic (octa-core ARM64 "mobile SoC")
      10,000,000,000[58]
      2018
      Apple
      7 nm
      122 mm2

      134
      Apple A10X Fusion (hexa-core ARM64 "mobile SoC")
      4,300,000,000[59]
      2017
      Apple
      10 nm
      96.40 mm2

      135
      Centriq 2400
      18,000,000,000[60]
      2017
      Qualcomm
      10 nm
      398 mm2

      136
      32-core AMD Epyc
      19200000000
      2017
      AMD
      14 nm
      768 mm2

      137
      GC2 IPU
      23600000000
      2018
      Graphcore
      16 nm
      825 mm2

      138
      Tegra Xavier SoC
      9,000,000,000[61]
      2018
      Nvidia
      12 nm
      350 mm²

139 rows × 6 columns

### Clean data

```python
moore.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 139 entries, 0 to 138
    Data columns (total 6 columns):
    Processor               139 non-null object
    Transistor count        135 non-null object
    Date of introduction    139 non-null object
    Designer                139 non-null object
    Process                 139 non-null object
    Area                    130 non-null object
    dtypes: object(6)
    memory usage: 6.6+ KB

```python
feature = 'Date of introduction'
target  = 'Transistor count'

moore = moore.dropna(subset=[feature, target]).copy()

for column in [feature, target]:
    moore[column] = (moore[column]
                     .str.split('[').str[0]  # Remove citations
                     .str.replace(r'\D','')  # Remove non-digit characters
                     .astype(int))

moore = moore.sort_values(by=feature)
```

### Visualize

```python
sns.distplot(moore[target]);
```

```python
moore.plot(x=feature, y=target, kind='scatter', alpha=0.5);
```

```python
moore.plot(x=feature, y=target, kind='scatter', alpha=0.5, logy=True);
```

### Ordinary Least Squares Regression

```python
ax = moore.plot(x=feature, y=target, kind='scatter', alpha=0.5)

X = moore[[feature]]
y = moore[target]
model = LinearRegression()
model.fit(X, y)
ax.plot(X, model.predict(X))
print('R^2', model.score(X, y))
```

### Log-Linear Regression

```python
ax = moore.plot(x=feature, y=target, kind='scatter', alpha=0.5, logy=True)

X = moore[[feature]]
y = np.log(moore[target]) # Apply natural log function to the target
model = LinearRegression()
model.fit(X, y)
y_pred = np.exp(model.predict(X)) # Apply exponential function (inverse of natural log) to the predictions
ax.plot(X, y_pred)
print('R^2', model.score(X, y))
```

```python
ax = moore.plot(x=feature, y=target, kind='scatter', alpha=0.5)
ax.plot(X, y_pred)
print('R^2', model.score(X, y))
```

```python
sns.distplot(y);
```

### Polynomial Regression

```python
# Make polynomial feature
moore['Date of introduction ** 2'] = moore['Date of introduction'] ** 2

features = ['Date of introduction', 'Date of introduction ** 2']
X = moore[features]
y = moore[target]
model = LinearRegression()
model.fit(X, y)
ax = moore.plot(x='Date of introduction', y=target, kind='scatter', alpha=0.5)
ax.plot(X['Date of introduction'], model.predict(X))
print('R^2', model.score(X, y))
```

```python
# Generalize for higher degree polynomials, and print equation

def polynomial_regression(degrees=2):

    # Make polynomial features
    feature = 'Date of introduction'
    polynomial_features = []
    for degree in range(2, degrees+1):
        name = f'{feature} ** {degree}'
        moore[name] = moore[feature] ** degree
        polynomial_features.append(name)

    features = [feature] + polynomial_features
    target  = 'Transistor count'
    X = moore[features]
    y = moore[target]
    model = LinearRegression()
    model.fit(X, y)

    ax = moore.plot(x=feature, y=target, kind='scatter', alpha=0.5)
    ax.plot(moore[feature], model.predict(X))
    betas = [model.intercept_] + model.coef_.tolist()
    equation = ' + '.join(f'{beta}x**{i}' for i, beta in enumerate(betas))
    print(equation)
    print('R^2', model.score(X, y))

polynomial_regression(3)
```

```python
interact(polynomial_regression, degrees=(1,10));
```

# Polynomial Regression, explained

Just as multiple regression was an extension of the bivariate case, Polynomial Regression is an extention of multiple regression and can be used to fit data to any (curved) shape. This is one of the reasons why data exploration is so important. You won't know that you need to fit a polynomial function to a feature unless you have examined its distribution.

[Why is polynomial regression considered a special case of multiple linear regression?](https://stats.stackexchange.com/questions/92065/why-is-polynomial-regression-considered-a-special-case-of-multiple-linear-regres)

# Example #1: King County Housing Data

[from Kaggle](https://www.kaggle.com/harlfoxem/housesalesprediction)

```python
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/datasets/master/kc_house_data.csv')
print(df.shape)
df.head().T
```

## Find a "curved" feature in the dataset

```python
target = 'price'
numeric_columns = df.select_dtypes(include='number').columns
for feature in numeric_columns.drop(target):
    sns.scatterplot(x=feature, y=target, data=df, alpha=0.1)
    plt.show()
```

## Make or "engineer" a new grade_squared feature

```python
df['grade_squared'] = df['grade']**2
for feature in ['grade', 'grade_squared']:
    sns.scatterplot(x=feature, y=target, data=df, alpha=0.1)
    plt.show()
```

## Test the fit of a polynomial regression to that feature

First we'll fit a regular bivariate regression line and calculate its $R^2$ to get a baseline. Since we want to know if this generated feature is improving our model or not we'll first run our code without it so that we have something to compare to.

$price_i = \beta_0 + \beta_1grade_i + \epsilon_i$

```python
# Separate dependent and independent variables
target = 'price'
features = ['grade']
y = df[target]
X = df[features]

def run_linear_model(X, y):
    # Split into test and train data
    X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.80, test_size=0.20, random_state=42)

    # Fit model using train data
    model = LinearRegression()
    model.fit(X_train, y_train)

    # Make predictions using test features
    y_pred = model.predict(X_test)

    # Compare predictions to test target
    rmse = (np.sqrt(mean_squared_error(y_test, y_pred)))
    r2 = r2_score(y_test, y_pred)

    print('Root Mean Squared Error', rmse)
    print('R^2 Score', r2)
    print('Intercept', model.intercept_)
    coefficients = pd.Series(model.coef_, X_train.columns)
    print(coefficients.to_string())

run_linear_model(X, y)
```

## Lets try fitting grade_squared as a bivariate model

$price_i = \beta_0 + \beta_1grade^{2}_i + \epsilon_i$

```python
# Separate dependent and independent variables
target = 'price'
features = ['grade_squared']
y = df[target]
X = df[features]

run_linear_model(X, y)
```

## Multiple Regression using both grade and grade_squred

$price_i = \beta_0 + \beta_1 grade_i + \beta_2grade^{2}_i + \epsilon_i$

```python
# Separate dependent and independent variables
target = 'price'
features = ['grade', 'grade_squared']
y = df[target]
X = df[features]

run_linear_model(X, y)
```

# How to find non-linear features

## 1) Domain Knowledge (think about your variables and hypothesize)

This is why having domain knowledge about the problem that you're trying to solve is something that's so important. In the context of home prices, variables that have a curved structure often are that way due to some form of diminishing returns increases in certain amenities. Lets think about the following variables:

- Lot Size:

The more land you're willing to buy all at once the cheaper it will be on a per-acre basis (Saving money when you buy in bulk). This trend carries through to small to medium sized lots but but with a more shallow curve.

- Square Footage:

Square footage of a home sees a similar pattern. The value an additional 100 square feet in small homes (imagine the difference between say a 800 sq foot home and a 900 sq foot home) makes a big difference to buyers, where as an additional 100 square feet in a mansion probably isn't valued quite as highly.

- Age:

Just like how the prices of new cars drop steeply in the first few years, the value of homes due to age drop quickly in the first few years after its built and then less quickly as time goes on. This is not a linear pattern and needs to be fitted by a polynomial model.

## 2) Visual Inspection

We already talked about how generating scatterplots or other graphs is vital in our data exploration stage and can lead us to identify possible candidates for polynomials. Here I just wanted to share one more tip that can help you analyze scatterplots when you have a lot of data.

If you have so much data that it's hard to tell what's going on in your scatterplot, then sample your data and regenerate them to get a better idea.

```python
# Sample our dataframe to take 1/20th the values
sampled = df.sample(frac=0.05, replace=True, random_state=42)

target = 'price'
numeric_columns = df.select_dtypes(include='number').columns
for feature in numeric_columns.drop(target):
    sns.scatterplot(x=feature, y=target, data=sampled, alpha=0.2)
    plt.show()
```

## 3) Inspect the distribution of residuals

## An Aside: The "Hedonic Housing Model"

Using Linear Regression to model home prices is a very common use of predictive linear regression modeling. It's so common fact that it has its own name: The Hedonic Housing Model. In the Hedonic Housing model it is well understood and reiterated that certain features tend to be curved in nature and these polynomial features (like the ones mentioned above) have all been well explored in real estate prediction circles. This is another of how domain knowledge can give you an edge. The best way to gain domain knowledge is to dive in and try and solve one particular kind of problem, and pick up little tips and tidbits as time goes on.

# Log-Linear Regression

In a log-linear regression model, we take the natural log of all of our y variable and use that as our y vector instead of the raw y values. Why would we do that?

<https://www.kaggle.com/c/house-prices-advanced-regression-techniques/discussion/60861>

## 1) To reduce skew in y

Where we have variables with lots of relatively low values and few high values (like with home prices) we would expect to see our data more clustered on the left-hand side with a long tail extending to the right up into the expensive homes. The fact of the matter is that we will be able to make better predictions if we can normalize our data to some degree and one way to do this is by taking the natural log of it.

```python
df['ln_price'] = np.log(df['price'])
```

```python
sns.distplot(df['price']);
```

```python
sns.distplot(df['ln_price']);
```

## New distribution of variables

```python
target = 'ln_price'
numeric_columns = df.select_dtypes(include='number').columns
for feature in numeric_columns.drop(target):
    sns.scatterplot(x=feature, y=target, data=df, alpha=0.1)
    plt.show()
```

## 2) Make coefficients easier to interpret

Transforming our price values in this way won't change our model's ability to generate predictions, but what it **will** do is change the interpretation of all of our x-coefficients. This will change our x-coefficients from have an elasticity type interpretation (a raw dollar amount change if there is a 1 unit increase in x) to having a percentage-terms interpretation. Lets demonstrate and talk about this further.

## 3) Make our errors easier to interpret

Errors that have been calculated on variables that are in log form can also be interpreted roughly as percentage error. We've been using percentages all our lives and they have immediate meaning to us. This is why I prefer log-linear regression models when possible.

<https://people.duke.edu/~rnau/411log.htm>

Feature Engineering is a big topic in machine learning. We won't be able to cover every aspect of it today, but hopefully we can give you a strong idea of what it is and how to go about it.

[Understanding Feature Engineering Part 1](https://towardsdatascience.com/understanding-feature-engineering-part-1-continuous-numeric-data-da4e47099a7b)

Feature engineering is key to success in predictive modeling. It is the process by which we take existing features and combine them or alter them in ways that will expose additional signal to our model. Feature engineering is all about making the most of the data that we already had.

```python
# Log-Linear Regression
# Separate dependent and independent variables
target = 'ln_price'
features = ['grade']
y = df[target]
X = df[features]

run_linear_model(X, y)
```

This means that a one unit increase in the grade of a home increases its sale price by 31%. Often it is much easier to interpret coefficients in this manner than in the regular way.

```python
## Log-Linear Regression
# Separate dependent and independent variables
target = 'ln_price'
features = ['grade', 'grade_squared']
y = df[target]
X = df[features]

run_linear_model(X, y)
```

Our RMSE is really "small" now because it now represents error in percentage terms. We're on average about 37% off in our predictions of house prices.

Our coefficients can also be understood in percentage terms which makes the coefficients on our regression much more digestable at a glance.

## A note about $R^2$

$R^2$ If we add any feature to our model (even nearly meaningless ones) our $R^2$ will improve. For this reason raw $R^2$ is not the ultimate measure of goodness of fit. It is informative but completely secondary to our Root-Mean-Squared-Error (in predictive modeling). While a higher $R^2$ is generally better, this isn't the thing that we're trying to optimize. We care more about minimizing RMSE than maximizing $R^2$

### "Kitchen Sink" models

You may hear the term "kitchen sink" regression model used. This refers to a regression model that throws every available explanatory variable into the model in an attempt to improve it without much thought to whether those variables should really be considered as affecting y. Kitchen Sink models will have a higher $R^2$ than others but will have higher standard errors (estimates about particular coefficients may be less precise).

Therefore, you tend to see "Kitchen Sink" models when the only priority is predictive accuracy and not interpretability.

### Alternative measures of Goodness-of-fit

Efforts have been made to improve upon $R^2$ due to these limitations. A metric called "Adjusted $R^2$" seeks to account for the number of explanatory variables included in a model and adjust $R^2$ accordingly. This is something that you can look up if you're curious. I just wanted to make you aware of it more than anything.

## A note about dirty linear regression data

### Linear Regression models can only process numeric values

### Your data must be free of NaN values before being passed to the algorithm

(some data cleaning will be required in today's assignment)

# Feature Engineering

### Polynomial Regression requires feature engineering

You've already seen an example of feature engineering today when we created the $grade^2$ variable. We took an existing feature and used it to generate a new feature that exposed the data to the model in a slightly different way.

"Feature engineering is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in improved model accuracy on unseen data." - [Jason Brownlee](https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/)

### What features could we engineer with the King County dataset?

```python
df.head().T
```

- **[date]** The date is in a format that is not super useful to us. If were to extract the year we could then take the difference between year and yr_built to find the age of the home. We could also include the squared term of the home age since we know (from our hedonic housing model domain knowledge) that home age typically is not linear.
- **[bedrooms]** & **[bathrooms]** We maybe use a combined measure of bedrooms and bathrooms, or find an average room square footage by taking total number of rooms and dividing by the square footage.
- **[sqft_living]** **[sqft_lot]** The difference between lot square footage and home square footage ought to also give us a rough measure of the size of the yard. Rough measures are fine as long as the engineered features expose some new shred of meaning to our model.
- **[floors]** We could calculate an average number of square feet per floor
- **[lat]** **[long]** There are all kinds of things that we could do with the latitude and longitude especially if we use some kind of outside API or external dataset to bring in new features associated with the location of the homes. This would take a lot of work but these could potentially be very powerful features.

### Kaggle is one of the best places to get feature engineering ideas

Look at Kernels: https://www.kaggle.com/c/house-prices-advanced-regression-techniques/kernels

Here's just one example: https://www.kaggle.com/thevachar/house-price-regression-and-feature-engineering

Kaggle has definitions of the columns too: https://www.kaggle.com/harlfoxem/housesalesprediction

### Start with baseline, before feature engineering

```python
df['ln_price'] = np.log(df['price'])
target = 'ln_price'
numeric_columns = df.select_dtypes(include='number').columns
features = numeric_columns.drop([target, 'price', 'id'])
features
```

```python
# Non-Feature Engineered Baseline Model
y = df[target]
X = df[features]
run_linear_model(X, y)
```

### Engineer a new feature

```python
df['age'] = 2015 - df['yr_built']
```

```python
numeric_columns = df.select_dtypes(include='number').columns
features = numeric_columns.drop([target, 'price', 'id'])
y = df[target]
X = df[features]
run_linear_model(X, y)
```

### Can you engineer more features?

## More topics

### Interaction Terms

An interaction occurs when an independent variable has a different effect on the outcome depending on the values of another independent variable.

Lets look at an example where we were trying to estimate the level of satisfaction that a person would have when eating some kind of food with a condiment (sauce) on it.

$satisfaction_i = \beta_0 + \beta_1 food_i + \beta_2condiment_i + \epsilon$

Imagine that we have two foods: Ice Cream and Hot Dogs, and we also have to condiments: hot fudge and mustard.

$\beta_1$ in this example is trying to capture the effect of on satisfaction between eating hot dogs vs eating ice cream, and $\beta_2$ is trying to capture the effect of putting hot fudge (chocolate sauce) vs mustard on your food.

$\beta_2$ is a little more problematic in this scenario. If someone were to come up to you and ask if you preferred hot fudge or mustard on your food, how would you answer?

You would probably say something like **"IT DEPENDS ON WHAT THE FOOD IS."** This means that the effect of our x variables on y (satisfaction) depends on the combination of food and condiment. I don't know about you guys, but I wouldn't be as satisfied if I had hot fudge on my hot dog or mustard on my ice cream.

An interaction terms is something that we add to our regression to account for these "It Depends" moments between two x variables. We do this by multiplying the two of them together or *interacting* them with each other to capture the implications of the different combinations taking place.

$satisfaction_i = \beta_0 + \beta_1 food_i + \beta_2condiment_i + \beta_3(food\times condiment_i) + \epsilon$

<http://statisticsbyjim.com/regression/interaction-effects/>

### Removing Outliers

To remove outliers via the 1.5*Interquartile-Range method. The first step is to calculate the IQR for each variable.

The IQR is the difference between the 25th and 75th percentiles of the feature.

Find the IQR and multiply it by 1.5

Then add the 1.5*IQR to the 3rd quartile (75th percentile). Anything above that range is an outlier.
Subtract 1.5*IQR from the 1st quartile (25th percentile). Anything below that value is also an outlier.

You want to minimize outliers in your dataset, so remove them by dropping observations that contain outliers in key features.

Typically you will wan to remove outliers before doing anything else with your dataset. We haven't focused on this strongly yet in the class, but coefficients get strongly biased by outliers so if you want to really have accurate predictions, remove outliers before you begin your feature engineering and modeling.

# Major Takeaways

- Polynomial Regression
  - Linear Regression can fit curved lines.
  - Including squared and cubed terms can improve fit and accuracy.

- Log-linear Regression
  - ln(y) helps normalize data with a skewed y variable.
  - ln(y) changes interpretation of coefficients and errors to be percentages.

- Feature Engineering
  - Generating Features improves model accuracy if done well.
  - This is where creativity and domain knowledge really pay off.
  - When you think that certain combinations of x variables might affect y differently than how they do separately. Include an interaction term.


# Learning the Essence of Linear Algebra

Linear algebra is a branch of mathematics that has applications in almost all scientific domains. But what is it exactly? Linear algebra is the study of linear maps on finite-dimensional vector space, this definition is "impeccably correct but not at all helpful." So what is really linear algebra?

Let's define the two words first:
    - Linear: of the first degree with respect to one or more variables. Something that follows an expected sequence that is graphically represented with a stright line. The word comes from latin and it means "pertaining or resembling a line"
    - Algebra: a study of mathematical symbols and rules for manipulating these symbols. Originates from the arabic word al-jabir meaning "reunion of broken parts."

So linear algebra is the study of objects that represent expected sequences. The objects that we will use are the following: scalars, vectors, matrices and tensors:
    - Scalars: a quantity without direction but only magnitude (something you can use to scale something else)
    - Vectors: a quantity that has a magnitude as well as direction, something you can use to determine the position of one point relative to another. An ordered array of numbers, list of values. The python package numpy can be used to create vectors (which are called arrays in numpy)
    - Matrices: a 2D array of numbers. A table, or list of lists: [ [list1], [list2] ]
    - Tensors: an array of numbers arranged on a regular grid with a variable number of axes. A table of tables, a list of lists of lists: [ [ [], [] ], [ [], [] ] ]

```python
import numpy as np
import matplotlib.pyplot as plt
from jupyterthemes import jtplot
!jt -t monokai
!jt -t monokai -f fira -fs 13 -nf ptsans -nfs 11 -N -kl -cursw 5 -cursc r -cellw 95% -T

!jtplot.style(theme=’monokai’, context=’notebook’, ticks=True, grid=False)
```

    /bin/sh: -c: line 0: syntax error near unexpected token `theme=’monokai’,'
    /bin/sh: -c: line 0: `jtplot.style(theme=’monokai’, context=’notebook’, ticks=True, grid=False)'

```python
def plotVectors(vecs, cols=None, alpha=1):
    """
    Plot set of vectors.

    Parameters
    ----------
    vecs : array-like
        Coordinates of the vectors to plot. Each vectors is in an array. For
        instance: [[1, 3], [2, 2]] can be used to plot 2 vectors.
    cols : array-like
        Colors of the vectors. For instance: ['red', 'blue'] will display the
        first vector in red and the second in blue.
    alpha : float
        Opacity of vectors

    Returns:

    fig : instance of matplotlib.figure.Figure
        The figure of the vectors
    """
    plt.figure()
    plt.axvline(x=0, color='#A9A9A9', zorder=0)
    plt.axhline(y=0, color='#A9A9A9', zorder=0)

    for i in range(len(vecs)):
        x = np.concatenate([[0,0],vecs[i]])
        if cols:
            plt.quiver([x[0]],
                       [x[1]],
                       [x[2]],
                       [x[3]],
                       angles='xy', scale_units='xy', scale=1, color=cols[i],
                       alpha=alpha)
        else:
            plt.quiver([x[0]],
                       [x[1]],
                       [x[2]],
                       [x[3]],
                       angles='xy', scale_units='xy', scale=1,
                       alpha=alpha)
# Source: https://hadrienj.github.io/posts/Deep-Learning-Book-Series-2.11-The-determinant/

orange = '#FF9A13'
blue = '#1190FF'

i = [0, 1]
j = [1, 0]

plotVectors([i, j], [[blue], [orange]])
plt.xlim(-0.5, 3)
plt.ylim(-0.5, 3)
plt.show()
```

## Properties of Linear Objects

Now that we have defined a handful of linear objects, we can start thinking of what sorts of properties susch objects have.

### Basis Vectors

Suppose you have a two dimensional vector of length 1 represented by  ${{\hat {\imath }}}$  on one direction and  ${{\hat {\jmath }}}$  on the other; thus representing a vector as  ${{\hat {\imath }}}$ +  ${{\hat {\jmath }}}$. These two letters can be thought of as the grids x and y on a 2D coordinate system, thus representing a way to get to a point on the xy grid.

### Linear Combinations

If you take the basis vectors ${{\hat {\imath }}}$ and ${{\hat {\jmath }}}$ and scale them on each axis, you will be able to land on any point on the grid. This way of scaling basis vectors in order to reach a point is called a linear combination.

### Span

The span of of a vector is defined as the space the vector is able to reach as a result of scaling. If you have a vector represented by v and w, the span of this vector is then the entire 2D grid since you can reach any point on the grid by linearly combining v and w.
    - The span of a 1D vector is an infinite line
    - The span of a 2D vector is an infinite plane
    - The span of a 3D vector is an infinite 3D space $R^3$

Properties such as this is what allows linear algebra to represent attributes of reality, making it so useful.

    - Formally, the basis of a vector space (1D, 2D, etc.) is a set of linearly independent vectors that span the full space
    - i and j are the basis of the 2D vector space
    - i, j and k are the basis for the 3D vector space etc.

### Linear Independence

If two vectors are linearly inependent, it means that they represent independent directions in the vector space. Here are different ways of thinking about this:
    - If you have a vector, vector 1, pointing in direction 1, then vector 2 would be independent of vector one if and only if it points in a different direction
    - Independent columns of of matrix A means the only solution to Av = 0 is v = 0
    - If I have independent columns in my matrix then the null space of the matrix is just the zero vector

### Basis for Subspace
    - Independent vectors that span the space
    - The number of dimension is the number of basis vectors we need
    - Dimension: number of basis vectors for the subspace
        - 1D dimension has exactly 1 basis vector
        - 2D dimension has exactly 2 basis vectors
    - Every basis has the same number and that number is the dimension
    - Take 3D space, and a subspace of a plane, the plan can only have 2 independent vectors
        - Every basis for this plane contains only and exactly 2 which is why the dimension of the plane is 2

#### Useful Things to Remember
    - A set with only one vector is linearly independent if one vector is not the zero vector
    - A set containing two vectors is linearly independent so long as one vector is not a multiple of the other
    - A set containing 'n x 1' vectors is linearly dependent if the number of vectors is greather than n
        - [1, 2] and [3, 5] are linearly inependent but whatever third vector you add will end up being linearly dependent
    - A set containing the zero vector is always linearly dependent
    - If the vectors are linearly independent then removing an arbitrary vector, v_i does not affect the linear independence
    - If the set only contains distinct unit vectors, the set has linear independence

#### Sources:
https://www.youtube.com/watch?v=eeMJg4uI7o0

https://hadrienj.github.io/posts/Deep-Learning-Book-Series-2.4-Linear-Dependence-and-Span/

https://www.youtube.com/watch?v=50aWVJBkJNs

#### How do you know if two vectors are linealy independent?

If two vectors are linearly independent, it means that they are pointing in different directions. In other words, numerically, they can't be multiples (a scaled version) of eachother.

The vectors $\{v_1, v_2, ..., v_n\}$ are linearly independent if the equation involving linear combination involving:

    $a_1v_1 + a_2v_2 + ... + a_nv_n = 0$

is true only when the scalars $a_i$ are all equal to zero. Pictorially, this means you are reducing each vector unit by unit from all sides and checking if the vectors are pointing the same direction.

```python
# TODO: write python code that will figure out linearly dependent vectors
```

## The Algebra of Linear Objects

Since linear algebra is the study of mathematical operations on linear objects, we will now investigate such operations and what they mean. These operations are also called linear transformations, since they are able to transform the object from one form to another.

### Linear Transformation

 Transformation is simply a function that takes in a vector and transforms it into a vector of a different form. Here, only transformations that are linear will be discussed. A linear transformation will keep the lines straight, such that gridlines are kept parallel and evenly spaced. In order to describe linear transformations in a numerical way, we will use basis vectors.

 Suppose you have the following equation: $Ax = v$, where A is a matrix and v is a vector. The question is: in what way should $A$ transform $x$ such that it lands on $v$. The matrix $A$ is what will execute the linear transformation.

$
\begin{align}
\begin{bmatrix}
a & b  \\
c & d
\end{bmatrix}\begin{bmatrix}
x  \\
y
\end{bmatrix} &= x\begin{bmatrix}
a  \\
c
\end{bmatrix} + y\begin{bmatrix}
b  \\
d
\end{bmatrix}\\
\begin{bmatrix}
a & b  \\
c & d
\end{bmatrix}\begin{bmatrix}
x  \\
y
\end{bmatrix} &= \begin{bmatrix}
ax & + & by  \\
cx & + & dy
\end{bmatrix}
\end{align}$

 The above equation shows how the vector $[x  y]^T$ is linearly transformed by the matrix on the left. So the vector x is transformed by $[a c]$ and $y$ is transformed by $[b d]$.

 If the columns of the matrix $A$ are linearly dependent, this will mean that the vector will be transformed in such a way that it would end up on a line.

- A matrix is basically a function that has the ability to linearly transform an input vector to an output vector

```python
xvals = np.linspace(-10, 10, 10)
yvals = np.linspace(-3, 3, 10)
xygrid = np.column_stack([[x, y] for x in xvals for y in yvals])

fig, axes = plt.subplots(nrows=1, ncols=5, figsize=(20, 5), facecolor='w')

axes[0].scatter(xygrid[0], xygrid[1], s=36,  edgecolor="none")
axes[0].grid(True)
axes[0].axis("equal")
axes[0].set_title("Original grid in x-y space")

transformers = [
    np.column_stack([[0, -1], [1, 0]]),
    np.column_stack([[1, 0], [0, 1]]),
    np.column_stack([[1000, 0], [0, 1000]]),
    np.column_stack([[0.0001, -1], [-1, 0.0001]])
]

for i in range(len(transformers)):
    result = np.dot(transformers[i], xygrid)
    axes[i+1].scatter(result[0], result[1], s=36, edgecolor="none")
    axes[i+1].grid(True)
    axes[i+1].axis("equal")
    axes[i+1].set_title("Transformed Grid {0}".format(i+1))

```

### Composition

Multiplying two matricies represents applying one linear transformation followed by another. Numerically this means multiplying two matricies together such that you end up with one matrix that is a composition of the two matricies.

$\begin{bmatrix}
a & b  \\
c & d
\end{bmatrix}\begin{bmatrix}
e & f  \\
g & h
\end{bmatrix}  =
\begin{bmatrix}
ae + bg & af + bh  \\
ce + dg & cf + dh
\end{bmatrix}$

Keep in mind that the order in which the matricies compose matters! This makes sense because if you think of a matrix as something that transforms a vector, then the order in which this transformation happens matters because space is not symmetric all the way. Although it might be the case that order does not matter if you have enough transformations to render it symmetric.

#### In code the numpy dot product can be used:

```python
import numpy as np
matrix_transformer = np.array([ # Bisymmetric matrix
    [ 1, 2, 3 ],
    [ 2, 5, 2 ],
    [ 3, 2, 1 ]
])

input_vector = np.array([1, 2, 3])
output1 = matrix_transformer.dot(input_vector)
output2 = input_vector.dot(matrix_transformer)

print("""Output vector is {0} on the left
And {1} on the right because of symmetry""".format(output1, output2))
```

    Output vector is [14 18 10] on the left
    And [14 18 10] on the right because of symmetry

### The Determinant

The degree to which a matrix scales an area/volume as a result of transforming a vector is called the determinant. The determinant is a property of a matrix and it $determines$ how much areas/volumes (3D) change as a result of transformation imposed by the matrix. This change in area/volume occurs because the vector represents a certain area within the grid. This means that if the determinant of a matrix is zero, then the transformation imposed by such matrix squizzes the area onto a dot. This is not exactly right because the determinant can take on negative values but this has to do with direction and technicality; the essense of this example maintains validity. Let's just say that the absolute value of the determinant is what represents the area\volume.

What would it mean if the determinant of a matrix is zero? Well, if the determinant is zero it means that the space enclosed by the vectors (area\volume) would converge to a point (or a flat plane in 3D, also a point is possible in 3D). When the determinant is zero, it means that the columns of the matrix are linearly dependent. This is because the matrix transforms given vectors in such a way that the vectors end up either on the same direction or on to a point.

$$det\bigg(\begin{bmatrix}
a & b  \\
c & d
\end{bmatrix}\bigg) = ad - bc$$

If you multiply two matricies and then take their determinant: $det(M_1M_2)$ it would be the same as if you take respective determinants and then multiply: $det(M_1)det(M_2)$. Why is that?

$$det(M_1M_2) = det(M_1)det(M_2)$$

Well when you multiply the two matrices you will end up with a matrix that contains information about how to transform a vector, when you take the determinant of that you will have information about such transformation. The information of the determinant remains the same even if you apply it independently with each matrix given they are in the right order.

How much information regarding the matrix is stored in a determinant? The determinant is a function that takes in a square matrix and outputs a number. If two matrices have the same determinant, it doesn't mean that they are the same; it just means the same thing as if two rectangles have the same area.

In code:

```python
# The determinant can be computed with numpy
import numpy as np

M = np.array([[6, 1, 1], [4, -2, 5], [2,8,7]])
det = np.linalg.det(M)
print(det)
```

    -306.0

### Inverse Matrices

### Column Space

# Continue Here:

### Norms

### Special Matrices and Vectors

### Eigendecomposition

### The Trace Operator

```python

```

Learning the Essence of Linear Algebra.ipynb


# Лабораторная работа 11. ООП.

# Упражнение 1. Формула.

Напишите класс "Формула" для создания символьной записи формулы и её вычисления. Класс должен содержать атрибуты:
- список операций производимых над переменной "x",
- метод для вывода всех операторов и вывода определённого оператора по его номеру,
- метод для составления формулы как строки скомпанованной из заданных операторов,
- метод для изменения оператора,
- метод для подстановки значения x и вычисления значения формулы (например, с помощью функции `eval()` :-) ).

```python
import numpy as np
import math
import scipy
import sympy as sm

class Formula:
    def about():
        print('Класс формул. Работает следующим образом:\n\t1) вводятся переменные, используемые в формуле\n\t2) вводится формула')
        print('\t3) теперь можно выполнять алгебраические операции и вычислять значения.')

    def __init__(self, raw = None, symbols = None):
        if raw is None:
            self.input()
        else:
            self.symbols = []
            self.raw = raw
        self.addSymbols()
        self.tryToComplete()

    def input(self):
        print('В следующих строках введите переменные и константы (символьные обозначения, по одному в строке, например: x). ' +
              'Когда закончите ввод, оставьте строку ввода пустой.')
        i = True
        self.symbols = []
        while i:
            k = input('Введите переменную или константу: ')
            if k != '':
                self.symbols.append(k)
            else:
                i = False
        self.raw = input('Теперь введите формулу. Используйте операции языка Python, а также функции, определённые в пакетах:\n\t' +
                         '1) NumPy (обращение - np.X)\n\t2) Scipy(обращение - scipy.X)\n\t3) SymPy (обращение - sm.X)\n\t' +
                         '4) math (обращение - math.X)\n')

    def addSymbols(self):
        for symbol in self.symbols:
            exec('%s = sm.symbols("%s")' % (symbol, symbol))
            globals()[symbol] = eval(symbol)

    def tryToComplete(self):
        try:
            self.fr = eval(self.raw)
        except:
            raise AttributeError('Формулу невозможно привести, так как Python не обладает необходимыми методами.')

    def out(self):
        print(self.fr)

    def f(self, **kwargs):
        for arg in kwargs.keys():
            exec('%s = %s' % (arg, kwargs[arg]))
        return eval('%s' % self.fr)
```

```python
Formula.about()
```

    Класс формул. Работает следующим образом:
    	1) вводятся переменные, используемые в формуле
    	2) вводится формула
    	3) теперь можно выполнять алгебраические операции и вычислять значения.

```python
A = Formula()
```

    В следующих строках введите переменные и константы (символьные обозначения, по одному в строке, например: x). Когда закончите ввод, оставьте строку ввода пустой.
    Введите переменную или константу: x
    Введите переменную или константу:
    Теперь введите формулу. Используйте операции языка Python, а также функции, определённые в пакетах:
    	1) NumPy (обращение - np.X)
    	2) Scipy(обращение - scipy.X)
    	3) SymPy (обращение - sm.X)
    	4) math (обращение - math.X)
    x ** 2 - 16

```python
A.out()
```

    x**2 - 16

```python
print(A.f(x = 4, log = 'math.log', sin = 'math.sin'))
```

    0

```python
print(A.f(x = 19, log = 'math.log', sin = 'math.sin'))
```

    -0.17654666791480436

```python
B = Formula()
```

    В следующих строках введите переменные и константы (символьные обозначения, по одному в строке, например: x). Когда закончите ввод, оставьте строку ввода пустой.
    Введите переменную или константу: f
    Введите переменную или константу: x
    Введите переменную или константу:
    Теперь введите формулу. Используйте операции языка Python, а также функции, определённые в пакетах:
    	1) NumPy (обращение - np.X)
    	2) Scipy(обращение - scipy.X)
    	3) SymPy (обращение - sm.X)
    	4) math (обращение - math.X)
    f - x ** 3 + 16 * x ** 2

```python
print(B.f(x = 4))
```

    f + 192.0

# Упражнение 2. Больше формул!

Определите для объектов класса "Формула" алгебраические операции: `+`, `-`, `*`, `/`.

```python
import numpy as np
import math
import scipy
import sympy as sm
```

```python
class Formula:
    def about():
        print('Класс формул. Работает следующим образом:\n\t1) вводятся переменные, используемые в формуле\n\t2) вводится формула')
        print('\t3) теперь можно выполнять алгебраические операции и вычислять значения.')

    def __init__(self, raw = None, symbols = None, notSetup = False):
        if notSetup:
            return
        if raw is None:
            self.input()
        else:
            self.symbols = symbols
            self.raw = raw
        self.addSymbols()
        self.tryToComplete()

    def input(self):
        print('В следующих строках введите переменные и константы (символьные обозначения, по одному в строке, например: x). ' +
              'Когда закончите ввод, оставьте строку ввода пустой.')
        i = True
        self.symbols = []
        while i:
            k = input('Введите переменную или константу: ')
            if k != '':
                self.symbols.append(k)
            else:
                i = False
        self.raw = input('Теперь введите формулу. Используйте операции языка Python, а также функции, определённые в пакетах:\n\t' +
                         '1) NumPy (обращение - np.X)\n\t2) Scipy(обращение - scipy.X)\n\t3) SymPy (обращение - sm.X)\n\t' +
                         '4) math (обращение - math.X)\n')

    def addSymbols(self):
        for symbol in self.symbols:
            exec('%s = sm.symbols("%s")' % (symbol, symbol))
            globals()[symbol] = eval(symbol)

    def tryToComplete(self):
        try:
            self.fr = eval(self.raw)
        except:
            raise AttributeError('Формулу невозможно привести, так как Python не обладает необходимыми методами.')

    def out(self):
        print(self.fr)

    def f(self, **kwargs):
        for arg in kwargs.keys():
            exec('%s = %f' % (arg, kwargs[arg]))
        return eval('%s' % self.fr)

    def __add__(self1, self2):
        self = Formula(None, None, True)
        self.symbols = list(set(self1.symbols + self2.symbols))
        self.addSymbols()
        self.fr = self1.fr + self2.fr
        return self

    def __sub__(self1, self2):
        self = Formula(None, None, True)
        self.symbols = list(set(self1.symbols + self2.symbols))
        self.addSymbols()
        self.fr = self1.fr - self2.fr
        return self

    def __mul__(self1, self2):
        self = Formula(None, None, True)
        self.symbols = list(set(self1.symbols + self2.symbols))
        self.addSymbols()
        self.fr = self1.fr * self2.fr
        return self

    def __truediv__(self1, self2):
        self = Formula(None, None, True)
        self.symbols = list(set(self1.symbols + self2.symbols))
        self.addSymbols()
        self.fr = self1.fr / self2.fr
        return self
```

```python
F1 = Formula(raw = 'x ** 2 - 16', symbols = ['x'])
F2 = Formula(raw = 'x ** 2 + 16', symbols = ['x'])
(F1 + F2).out()
```

    2*x**2

# Упражнение 3. Вспомнить всё.

# Тортик.

За столом сидят `n` гостей (вводится с клавиатуры), перед которыми стоит пирог. Пирог и его части можно делить только пополам. Определите, сколько раз нужно делить пирог на ещё более мелкие части, чтобы:
- каждому из гостей достался хотя бы 1 кусок;
- как минимум половине гостей досталось по 2 куска;
- каждому гостю досталось по 1 куску и при этом ещё хотя бы 10 кусков осталось в запасе.

```python
from math import ceil, log

def firstAppropiation(n):
    return ceil(log(n, 2))

def secondAppropiation(n):
    return ceil(log(1.5 * n, 2))

def thirdAppropiation(n):
    return ceil(log(n + 10, 2))
```

```python
thirdAppropiation(5)
```

    4

# Явный Фибоначчи.

Для `n`-го члена в последовательности Фибоначчи существует явная формула:

$$x_n = \frac{1}{2} \left( \left( \frac{1 + \sqrt{5}}{2} \right)^{n+1} - \left( \frac{1 - \sqrt{5}}{2} \right)^{n+1} \right) $$

Поскольку операции с вещественными числами происходят с конечной точностью, то с ростом `n` результат вычисления по этой формуле будет все больше отличаться от настоящего числа Фибоначчи. Найдите `n`, начиная с которого, отличие от истинного значения превысит `0.001`.

```python
epsilon = float(input('Введите кастомную точность: '))
```

    Введите кастомную точность: 0.001

```python
from math import sqrt as SQ
```

```python
def Fibonacci(n):
    fib = [0] * max(2, n)
    fib[0] = 1
    fib[1] = 1
    for i in range(2, n):
        fib[i] = fib[i - 1] + fib[i - 2]
    return fib[n - 1]
```

```python
#def ClearFibonacci(n):
#    return 0.5 * (((1 + 5 ** 0.5)/2) ** (n + 1) - ((1 - 5 ** 0.5)/2) ** (n + 1))
def ClearFibonacci(n):
    return 0.5 * (((1 + SQ(5))/2) ** (n + 1) - ((1 - SQ(5))/2) ** (n + 1))
```

```python
def trouverSaPlaceDansLaVie():
    print('Процедура находит n, начиная с которого результаты Fibonacci и ClearFibonacci отличаются более чем на epsilon, ' +
          'простым перебором.')
    n = 1 # начинаем поиск с третьего члена
    while True:
        if abs(Fibonacci(n) - ClearFibonacci(n - 1)) <= epsilon:
            n += 1
        else:
            print(n)
            print(Fibonacci(n))
            print(ClearFibonacci(n - 1))
            return
```

```python
trouverSaPlaceDansLaVie()
```

    Процедура находит n, начиная с которого результаты Fibonacci и ClearFibonacci отличаются более чем на epsilon, простым перебором.
    1
    1
    1.118033988749895

В то время как я подумал о неисправленной программе (где у меня появлялось переполнение памяти), я нашёл это:

> The problem is that you're using doubles to calculate the value and the doubles are overflowing. Doubles give exact solutions only to about the 85th Fibonacci number.
If you want a fast and accurate calculation you're better off using an algorithm based on a better recurrence relationship, and using python bignum integers.

> **Перевод:** Проблема в том, что вы используете double для вычисления значения, а double переполняется. Двойные числа дают точные решения только около 85-го числа Фибоначчи.
Если вы хотите быстрых и точных вычислений, вам лучше использовать алгоритм, основанный на лучших отношениях повторяемости, и использовать целые числа Python Bignum.

При этом:

> If you do want to perform your calculations using doubles and the explicit formulae, then the formulae can be tweaked to give something faster that doesn't overflow until about the 1500th fibonacci number, and remains the same accuracy as your version:

> **Перевод:** Если вы хотите выполнить свои вычисления с использованием двойных чисел и явных формул, то формулы можно настроить так, чтобы получить что-то более быстрое, которое не переполняется до 1500-го числа Фибоначчи и остается той же точности, что и ваша версия.

```
def fib(n):
    return round( ((1+math.sqrt(5))/2)**n / math.sqrt(5) )
```

Решил улучшить, так сказать.

```python
def updatedClearFibonacci(n):
    return ((1 + 5 ** 0.5)/2) ** n / (5 ** 0.5)
```

```python
def trouverSaPlaceDansLaVie2():
    print('Процедура находит n, начиная с которого результаты Fibonacci и ClearFibonacci отличаются более чем на epsilon, ' +
          'простым перебором.')
    n = 3 # начинаем поиск с третьего члена
    while True:
        if abs(Fibonacci(n) - updatedClearFibonacci(n)) <= epsilon:
            n += 1
        else:
            print(n)
            print(Fibonacci(n))
            print(updatedClearFibonacci(n))
            return
```

```python
trouverSaPlaceDansLaVie2()
```

    Процедура находит n, начиная с которого результаты Fibonacci и ClearFibonacci отличаются более чем на epsilon, простым перебором.
    3
    2
    1.8944271909999157

ВАУ! Формула даёт сбой на третьем же элементе ахахаха

Зато нашёл такой код:

```
def fib(n, memo={0:0, 1:1}):
    if n not in memo:
        memo[n] = fib(n-1) + fib(n-2)
    return memo[n]
```

И слегка подучил Markdown по [этому туториалу.](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)

Теперь поищем ещё чего-нибудь. Например:

> Python supports arbitrarily large integers naturally:

*`10 ** 1000`

100000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000*

You could even get, for example of a huge integer value, fib(4000000).
But still it does not (for now) supports an arbitrarily large float !!
If you need one big, large, float then check up on the decimal Module. There are examples of use on these forums: [OverflowError: (34, 'Result too large')](https://stackoverflow.com/questions/20201706/overflowerror-34-result-too-large)

> **Перевод:** Python поддерживает произвольно большие целые числа. Вы можете даже получить, например, огромное целочисленное значение fib (4000000). Но все же он (пока) не поддерживает сколь угодно большое нецелочисленное число! Если вам нужно одно такое, проверьте модуль decimal. На форуме есть примеры использования.

По ссылке нам предлагают удивительный код:

```
import decimal
decimal.getcontext().prec = 100
def pi():
    pi = decimal.Decimal(0)
    for k in range(350):
        pi += (decimal.Decimal(4)/(decimal.Decimal(8)*decimal.Decimal(k+1))...)
```

Проверим?

```python
from decimal import Decimal
import decimal

decimal.getcontext().prec = 100

def completedClearFibonacci(n):
    return Decimal(0.5) * ((Decimal(1 + 5 ** 0.5)/2) ** Decimal(n + 1) - (Decimal(1 - 5 ** 0.5)/2) ** Decimal(n + 1))
```

```python
def fib(n, memo={0:0, 1:1}):
    if n not in memo:
        memo[n] = fib(n-1) + fib(n-2)
    return memo[n]
```

```python
def trouverSaPlaceDansLaVie_FINAL():
    print('Процедура находит n, начиная с которого результаты Fibonacci и ClearFibonacci отличаются более чем на epsilon, ' +
          'простым перебором.')
    n = 3 # начинаем поиск с третьего члена
    while True:
        if abs(fib(n) - completedClearFibonacci(n - 1)) <= epsilon:
            n += 1
        else:
            print(n)
            print(fib(n))
            print(completedClearFibonacci(n - 1))
            return
```

```python
trouverSaPlaceDansLaVie_FINAL()
```

    Процедура находит n, начиная с которого результаты Fibonacci и ClearFibonacci отличаются более чем на epsilon, простым перебором.
    3
    2
    2.236067977499789940854357834

Знаков действительно стало больше. Но я не понял, когда формула начинает считать число точно.

Вообще, всё, что я понял из этого задания - что Python изначально считает корни с конечной точностью. Или формула не работает.

**А В ВИКИПЕДИИ-ТО ДЕЛЯТ НЕ НА ДВА, А НА КОРЕНЬ ИЗ ПЯТИ!**

$$x_n = \frac{\left( \frac{1 + \sqrt{5}}{2} \right)^{n+1} - \left( \frac{1 - \sqrt{5}}{2} \right)^{n+1}}{\sqrt{5}} $$

```python
from math import sqrt as SQ
```

```python
def Fibonacci(n):
    fib = [0]*max(2,n)
    fib[0] = 1
    fib[1] = 1
    for i in range(2, n):
        fib[i] = fib[i - 1] + fib[i - 2]
    return fib[n - 1]
```

```python
def ClearFibonacci(n):
    return (((1 + SQ(5))/2) ** (n + 1) - ((1 - SQ(5))/2) ** (n + 1)) / SQ(5)

print(ClearFibonacci(0))
print(ClearFibonacci(1))
print(ClearFibonacci(2))
print(ClearFibonacci(3))
```

    1.0
    1.0
    2.0
    3.0000000000000004

```python
def trouverSaPlaceDansLaVie():
    print('Процедура находит n, начиная с которого результаты Fibonacci и ClearFibonacci отличаются более чем на epsilon, ' +
          'простым перебором.')
    n = 1 # начинаем поиск с первого члена
    while True:
        if abs(Fibonacci(n) - ClearFibonacci(n - 1)) <= epsilon:
            n += 1
        else:
            print(n)
            print(Fibonacci(n))
            print(ClearFibonacci(n - 1))
            return
```

```python
trouverSaPlaceDansLaVie()
```

    Процедура находит n, начиная с которого результаты Fibonacci и ClearFibonacci отличаются более чем на epsilon, простым перебором.
    58
    591286729879
    591286729879.0012

# Кубики.

В детском садике `n` детей играют в следующую игру. Перед ними гора из `m` кубиков, первый ребёнок вынимает из кучи 1 кубик, каждый последующий ребёнок в два раза больше предыдущего, и так по кругу. Если число кубиков, которые нужно вынуть, превышает 25, из него вычитается 25 и отсчёт идёт от уменьшенного числа, например, вместо 32 кубиков будет вынуто 7, затем 14 и т. д. Проигравшим считается тот, кто не смог вытащить нужное число кубиков (в куче осталось недостаточно). Определите проигравшего.

```python
def numberGenerator(init = 1, max = 25):
    sequence = []
    while init not in sequence:
        if init <= max:
            sequence.append(init)
        else:
            init -= max
            if init in sequence:
                break
            sequence.append(init)
        init *= 2
    return sequence
```

```python
print('Все возможные ходы детей последовательно перечислены здесь:', ', '.join(map(str, numberGenerator())) + '.')
```

    Все возможные ходы детей последовательно перечислены здесь: 1, 2, 4, 8, 16, 7, 14, 3, 6, 12, 24, 23, 21, 17, 9, 18, 11, 22, 19, 13.

```python
def bunchSolver(cubes, kids):
    init = 1 #int(input('Введите количество кубиков, которое вынимает первый ребёнок: '))
    max = 25 #int(input('Введите максимальное количество кубиков, которое может вытянуть ребёнок: '))
    sequence = numberGenerator(init, max)
    currentStep = 0
    while cubes >= 0:
        cubes -= sequence[currentStep % len(sequence)]
        currentStep += 1
    return (currentStep) % kids or kids
```

```python
cubes = int(input('Введите количество кубиков: '))
kids = int(input('Введите число детей (минимум один ребёнок): '))
print('Проиграл %i-й ребёнок.' % bunchSolver(cubes, kids))
```

    Введите количество кубиков: 25
    Введите число детей (минимум один ребёнок): 3
    Проиграл 2-й ребёнок.

```python
print('Детей - %i, кубиков - %i. Проиграл %i-й ребёнок.' % (2, 3, bunchSolver(3, 2)))
print('Детей - %i, кубиков - %i. Проиграл %i-й ребёнок.' % (3, 1, bunchSolver(1, 3)))
print('Детей - %i, кубиков - %i. Проиграл %i-й ребёнок.' % (5, 15, bunchSolver(15, 5)))
```

    Детей - 2, кубиков - 3. Проиграл 1-й ребёнок.
    Детей - 3, кубиков - 1. Проиграл 2-й ребёнок.
    Детей - 5, кубиков - 15. Проиграл 5-й ребёнок.


SymPyBotics
===========

Symbolic Framework for Modeling and Identification of Robot Dynamics

Uses [Sympy](http://sympy.org) and [Numpy](http://www.numpy.org/) libraries.

[](https://travis-ci.org/cdsousa/SymPyBotics)

##### Citation:
[](http://dx.doi.org/10.5281/zenodo.11365)

Example
-------

Definition of a 2 DOF example robot:

```python
import sympy
import sympybotics
```

```python
rbtdef = sympybotics.RobotDef('Example Robot', # robot name
                               [('-pi/2', 0, 0, 'q+pi/2'),  # list of tuples with Denavit-Hartenberg parameters
                                ( 'pi/2', 0, 0, 'q-pi/2')], # (alpha, a, d, theta)
                               dh_convention='standard' # either 'standard' or 'modified'
                              )
```

```python
rbtdef.frictionmodel = {'Coulomb', 'viscous'} # options are None or a combination of 'Coulomb', 'viscous' and 'offset'
rbtdef.gravityacc = sympy.Matrix([0.0, 0.0, -9.81]) # optional, this is the default value
```

`L` is the link inertia tensor computed about the link frame;
`l` is the link first moment of inertia;
`m` is the link mass.
These are the so-called barycentric parameters, with respect to which the dynamic model is linear.

Generation of geometric, kinematic and dynamic models:

```python
rbtdef.dynparms()
```

    [L_1xx,
     L_1xy,
     L_1xz,
     L_1yy,
     L_1yz,
     L_1zz,
     l_1x,
     l_1y,
     l_1z,
     m_1,
     fv_1,
     fc_1,
     L_2xx,
     L_2xy,
     L_2xz,
     L_2yy,
     L_2yz,
     L_2zz,
     l_2x,
     l_2y,
     l_2z,
     m_2,
     fv_2,
     fc_2]

```python
rbt = sympybotics.RobotDynCode(rbtdef, verbose=True)
```

    generating geometric model
    generating forward kinematics code: transformation matrix
    generating forward kinematics code: position vector
    generating forward kinematics code: rotation matrix
    generating kinematic model
    generating jacobian code
    generating dynamics model
    generating inverse dynamics code
    generating gravity term code
    generating coriolis term code
    generating coriolis matrix code
    generating inertia matrix code
    generating regressor matrix code
    generating friction term code
    done

```python
rbt.geo.T[-1]
```

$\displaystyle \left[\begin{matrix}- \sin{\left(q_{1} \right)} \sin{\left(q_{2} \right)} & - \cos{\left(q_{1} \right)} & \sin{\left(q_{1} \right)} \cos{\left(q_{2} \right)} & 0\\\sin{\left(q_{2} \right)} \cos{\left(q_{1} \right)} & - \sin{\left(q_{1} \right)} & - \cos{\left(q_{1} \right)} \cos{\left(q_{2} \right)} & 0\\\cos{\left(q_{2} \right)} & 0 & \sin{\left(q_{2} \right)} & 0\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
rbt.kin.J[-1]
```

$\displaystyle \left[\begin{matrix}0 & 0\\0 & 0\\0 & 0\\0 & - \cos{\left(q_{1} \right)}\\0 & - \sin{\left(q_{1} \right)}\\1 & 0\end{matrix}\right]$

C function generation:

```python
tau_str = sympybotics.robotcodegen.robot_code_to_func('C', rbt.invdyn_code, 'tau_out', 'tau', rbtdef)
```

Doing print(tau_str), function code will be output:

```python
print(tau_str)
```

    void tau( double* tau_out, const double* parms, const double* q, const double* dq, const double* ddq )
    {
      double x0 = -ddq[0];
      double x1 = sin(q[1]);
      double x2 = dq[0];
      double x3 = x1*x2;
      double x4 = -x3;
      double x5 = cos(q[1]);
      double x6 = -x0;
      double x7 = dq[1]*x4 + x5*x6;
      double x8 = x2*x5;
      double x9 = dq[1]*x8 + x1*x6;
      double x10 = dq[1]*parms[16] + parms[14]*x8 + parms[17]*x3;
      double x11 = 9.8100000000000005*x1;
      double x12 = dq[1]*parms[15] + parms[13]*x8 + parms[16]*x3;
      double x13 = dq[1]*parms[13] + parms[12]*x8 + parms[14]*x3;
      double x14 = 9.8100000000000005*x5;
    //
      tau_out[0] = dq[0]*parms[10] + parms[11]*(((dq[0]) > 0) - ((dq[0]) < 0)) - parms[3]*x0 + x1*(ddq[1]*parms[16] - dq[1]*x13 + parms[14]*x7 + parms[17]*x9 - parms[19]*x14 + x12*x8) + x5*(ddq[1]*parms[13] + dq[1]*x10 + parms[12]*x7 + parms[14]*x9 + parms[19]*x11 + x12*x4);
      tau_out[1] = ddq[1]*parms[15] + dq[1]*parms[22] + parms[13]*x7 + parms[16]*x9 - parms[18]*x11 + parms[20]*x14 + parms[23]*(((dq[1]) > 0) - ((dq[1]) < 0)) - x10*x8 + x13*x3;
    //
      return;
    }

Dynamic base parameters:

```python
rbt.calc_base_parms(verbose=True)
rbt.dyn.baseparms
```

    calculating base parameters and regressor code
    done

$\displaystyle \left[\begin{matrix}L_{1yy} + L_{2zz}\\fv_{1}\\fc_{1}\\L_{2xx} - L_{2zz}\\L_{2xy}\\L_{2xz}\\L_{2yy}\\L_{2yz}\\l_{2x}\\l_{2z}\\fv_{2}\\fc_{2}\end{matrix}\right]$

Author
------

[Cristóvão Duarte Sousa](https://github.com/cdsousa)

Install
-------

From git source:

    git clone https://github.com/cdsousa/SymPyBotics.git
    cd sympybotics
    python setup.py install

License
-------

New BSD license. See [License File](LICENSE.txt)

```python

```


```python
# Header starts here.
from sympy.physics.units import *
from sympy import *

# Rounding:
import decimal
from decimal import Decimal as DX
from copy import deepcopy
def iso_round(obj, pv, rounding=decimal.ROUND_HALF_EVEN):
    import sympy
    """
    Rounding acc. to DIN EN ISO 80000-1:2013-08
    place value = Rundestellenwert
    """
    assert pv in set([
        # place value   #  round to:
        1,              #  1
        0.1,            #  1st digit after decimal
        0.01,           #  2nd
        0.001,          #  3rd
        0.0001,         #  4th
        0.00001,        #  5th
        0.000001,       #  6th
        0.0000001,      #  7th
        0.00000001,     #  8th
        0.000000001,    #  9th
        0.0000000001,   # 10th
        ])
    objc = deepcopy(obj)
    try:
        tmp = DX(str(float(objc)))
        objc = tmp.quantize(DX(str(pv)), rounding=rounding)
    except:
        for i in range(len(objc)):
            tmp = DX(str(float(objc[i])))
            objc[i] = tmp.quantize(DX(str(pv)), rounding=rounding)
    return objc

# LateX:
kwargs = {}
kwargs["mat_str"] = "bmatrix"
kwargs["mat_delim"] = ""

# kwargs["symbol_names"] = {FB: "F^{\mathsf B}", }

# Units:
(k, M, G ) = ( 10**3, 10**6, 10**9 )
(mm, cm) = ( m/1000, m/100 )
Newton = kg*m/s**2
Pa     = Newton/m**2
MPa    = M*Pa
GPa    = G*Pa
kN     = k*Newton
deg    = pi/180

half = S(1)/2

# Header ends here.
#
# https://colab.research.google.com/github/kassbohm/tm-snippets/blob/master/ipynb/TM_2/4_BB/2_BL/2.4.2.I-FEM_cc.ipynb

a, EI, F, q, c = var('a, EI, F, q, c')

spring = True
# spring = False

sub_list = [
    (a, 3 *m),
    (q, 3 *kN/m),
    (EI, 200 *GPa* 6500 *cm**4),
    (F, 10 *kN),
    (c, 1 *kN / mm),
    ]

a2 = a*a
a3 = a*a*a

K44 = EI/a3
K44 *= Matrix(
[
[  4*a2 ,  -6*a ,  2*a2 ,   6*a ],
[ -6*a  ,  12   , -6*a  , -12   ],
[  2*a2 ,  -6*a ,  4*a2 ,   6*a ],
[  6*a  , -12   ,  6*a    , 12   ],
]
)
tmp = K44/EI*a**3
l = var("l")
tmp = tmp.subs(a,l)
pprint(tmp)
pprint(latex(tmp,**kwargs))
exit()
K = K44[0:2, 0:2]

p1,w1 = var("p1,w1")
M2,F2 = var("M2,F2")
u = Matrix([p1,w1])

if spring:
    f = Matrix([0, F - c*w1])
else:
    f = Matrix([0, F]) + q*Matrix([-a2/12, a/2])

unknowns = [p1, w1]
eq = Eq(K*u , f)
pprint(eq)
sol = solve(eq, unknowns)

p1, w1 = sol[p1], sol[w1]

pprint("\nψ₁:")
tmp = p1
pprint(tmp)
pprint("\nψ₁ / deg:")
pprint(p1)
tmp = tmp.subs(sub_list)
tmp *= 180/pi
tmp = iso_round(tmp, 0.01)
pprint(tmp)

pprint("\nw₁:")
tmp = w1
pprint(tmp)
pprint("\nw₁ / mm:")
tmp = tmp.subs(sub_list)
tmp /= mm
tmp = iso_round(tmp, 0.01)
pprint(tmp)

if spring:
    exit()

K = K44[2:, 0:2]
u = Matrix([p1,w1])
f = Matrix([M2, F2]) + q*Matrix([a2/12, a/2])
unknowns = [M2, F2]
eq = Eq(K*u , f)
pprint(eq)
sol = solve(eq, unknowns)

M2, F2 = sol[M2], sol[F2]

pprint("\nM₂:")
tmp = M2
pprint(tmp)
pprint("\nM₂ / kNm:")
tmp = tmp.subs(sub_list)
tmp /= k*Newton*m
pprint(tmp)

pprint("\nF₂:")
tmp = F2
pprint(tmp)
pprint("\nF₂ / kN:")
tmp = tmp.subs(sub_list)
tmp /= kN
pprint(tmp)

# ⎡ 4⋅EI⋅p₁   6⋅EI⋅w₁  ⎤   ⎡  2    ⎤
# ⎢ ─────── - ───────  ⎥   ⎢-a ⋅q  ⎥
# ⎢    a          2    ⎥   ⎢────── ⎥
# ⎢              a     ⎥   ⎢  12   ⎥
# ⎢                    ⎥ = ⎢       ⎥
# ⎢  6⋅EI⋅p₁   12⋅EI⋅w₁⎥   ⎢    a⋅q⎥
# ⎢- ─────── + ────────⎥   ⎢F + ───⎥
# ⎢      2         3   ⎥   ⎣     2 ⎦
# ⎣     a         a    ⎦
#
# ψ₁:
#  2
# a ⋅(3⋅F + a⋅q)
# ──────────────
#      6⋅EI
#
# ψ₁ / deg:
# 0.26
#
# w₁:
#  3 ⎛F   a⋅q⎞
# a ⋅⎜─ + ───⎟
#    ⎝3    8 ⎠
# ────────────
#      EI
#
# w₁ / mm:
# 9.26
#                                     ⎡      2  ⎤
# ⎡      ⎛F   a⋅q⎞   a⋅(3⋅F + a⋅q)⎤   ⎢     a ⋅q⎥
# ⎢- 6⋅a⋅⎜─ + ───⎟ + ─────────────⎥   ⎢M₂ + ────⎥
# ⎢      ⎝3    8 ⎠         3      ⎥   ⎢      12 ⎥
# ⎢                               ⎥ = ⎢         ⎥
# ⎢                a⋅q            ⎥   ⎢     a⋅q ⎥
# ⎢           -F - ───            ⎥   ⎢F₂ + ─── ⎥
# ⎣                 2             ⎦   ⎣      2  ⎦
#
# M₂:
# -a⋅(2⋅F + a⋅q)
# ───────────────
#        2
#
# M₂ / kNm:
# -87/2
#
# F₂:
# -F - a⋅q
#
# F₂ / kN:
# -19

# spring = True:
#
# ⎡ 4⋅EI⋅p₁   6⋅EI⋅w₁  ⎤
# ⎢ ─────── - ───────  ⎥
# ⎢    a          2    ⎥
# ⎢              a     ⎥   ⎡   0    ⎤
# ⎢                    ⎥ = ⎢        ⎥
# ⎢  6⋅EI⋅p₁   12⋅EI⋅w₁⎥   ⎣F - c⋅w₁⎦
# ⎢- ─────── + ────────⎥
# ⎢      2         3   ⎥
# ⎣     a         a    ⎦
#
# ψ₁:
#           2
#      3⋅F⋅a
# ───────────────
#   ⎛        3  ⎞
# 2⋅⎝3⋅EI + a ⋅c⎠
#
# ψ₁ / deg:
#           2
#      3⋅F⋅a
# ───────────────
#   ⎛        3  ⎞
# 2⋅⎝3⋅EI + a ⋅c⎠
# 0.12
#
# w₁:
#        3
#     F⋅a
# ───────────
#         3
# 3⋅EI + a ⋅c
#
# w₁ / mm:
# 4.09

```


```python
import math
import json
import sympy as sp
from sympy.utilities.lambdify import lambdify
import numpy as np
import matplotlib.pyplot as plt

import openrtdynamics2.lang as dy
import openrtdynamics2.py_execute as dyexe
from openrtdynamics2.ORTDtoNumpy import ORTDtoNumpy

from vehicle_lib.vehicle_lib import *
import vehicle_lib.path_transformations as pt
import vehicle_lib.motion_primitives as mp
```

```python
Ts = 0.01
```

```python
# load track data
with open("track_data/simple_track.json", "r") as read_file:
    track_data = json.load(read_file)

```

# Under construction...

```python

```

```python

```

```python

```

# Kinematic bicycle model

The dynamic system equations are given by

$
\dot X
=
f(x,y,\psi)
=
\begin{pmatrix}
  \dot x \\
  \dot y \\
  \dot \psi
\end{pmatrix}
=
\begin{pmatrix}
  v \cos( \delta + \psi) \\
  v \sin( \delta + \psi) \\
  v / l_r \sin( \delta ), \\
\end{pmatrix}
$

with the state vector

$ X = [ x, y, \psi ]^T $.

Herein, $x$ and $y$ denote the coordinates of the vehicle front axle in cartesian space and $\psi$ the vehicle body orientation angle. The system inputs are the steering angle $\delta$ and the vehicle velocity $v$. Finally, the parameter $l_r$ denotes the wheelbase, which is the length in-between front and rear axle.

```python
x, y, v, delta, psi, l_r, T_s, n = sp.symbols('x y v delta psi l_r T_s n')

x_dot   = v * sp.cos( delta + psi )
y_dot   = v * sp.sin( delta + psi )
psi_dot = v / l_r * sp.sin( delta )

# system function f
f = sp.Matrix([ x_dot, y_dot, psi_dot ])

# state vector
X_bic = sp.Matrix( [ x, y, psi ])

# input vector
U_bic = sp.Matrix( [ delta, v ])
```

```python
f
```

$\displaystyle \left[\begin{matrix}v \cos{\left(\delta + \psi \right)}\\v \sin{\left(\delta + \psi \right)}\\\frac{v \sin{\left(\delta \right)}}{l_{r}}\end{matrix}\right]$

# Discretization of the continunous model

By applying Euler-forward discretization

$ {X}[k+1] = \underbrace{ {X}[k] + T_s  \dot{X} }_{f_{dscr}} $,

the continuous system is time-discretized with the sampling time $T_s$ yielding the discrete system funtion $f_{dscr}$.

```python
# apply Euler forward
f_dscr = sp.Matrix( [x,y,psi]) + T_s * f
```

```python
f_dscr
```

$\displaystyle \left[\begin{matrix}T_{s} v \cos{\left(\delta + \psi \right)} + x\\T_{s} v \sin{\left(\delta + \psi \right)} + y\\\frac{T_{s} v \sin{\left(\delta \right)}}{l_{r}} + \psi\end{matrix}\right]$

# Analytically compute the Jacobian matrices

A linearization of the non-linear system function around a dynamic set point is calculated by deriving the jacobian matrices w.r.t. the state vector $X$ and each system input $\delta$ and $v$:

continuous case
$
  A = \frac{ \partial f }{ \partial X},
  \qquad
  B = \frac{ \partial f }{ \partial [ \delta, v ]^T },
$

discrete-time case
$
  A_{dscr} = \frac{ \partial f_{dscr} }{ \partial X},
  \qquad
  B_{dscr} = \frac{ \partial f_{dscr} }{ \partial [ \delta, v ]^T },
$

```python
# continuous system matrices
A = f.jacobian(X_bic)
B = f.jacobian(U_bic)

# discrete system matrices
A_dscr = f_dscr.jacobian(X_bic)
B_dscr = f_dscr.jacobian(U_bic)
```

```python
A_dscr
```

$\displaystyle \left[\begin{matrix}1 & 0 & - T_{s} v \sin{\left(\delta + \psi \right)}\\0 & 1 & T_{s} v \cos{\left(\delta + \psi \right)}\\0 & 0 & 1\end{matrix}\right]$

```python
B_dscr
```

$\displaystyle \left[\begin{matrix}- T_{s} v \sin{\left(\delta + \psi \right)} & T_{s} \cos{\left(\delta + \psi \right)}\\T_{s} v \cos{\left(\delta + \psi \right)} & T_{s} \sin{\left(\delta + \psi \right)}\\\frac{T_{s} v \cos{\left(\delta \right)}}{l_{r}} & \frac{T_{s} \sin{\left(\delta \right)}}{l_{r}}\end{matrix}\right]$

# Create functions that generate the matrices A, B, and the system function f

Create python functions with which the symbolically dervied matrices and system function can be evaluated.

```python
variables = (T_s,l_r, x,y,psi,v,delta)

array2mat = [{'ImmutableDenseMatrix': np.matrix}, 'numpy']

A_dscr_fn = lambdify( variables, A_dscr, modules=array2mat)
B_dscr_fn = lambdify( variables, B_dscr, modules=array2mat)
f_dscr_fn = lambdify( variables, f_dscr, modules=array2mat)

```

```python
A_dscr_fn(0.01, 3.0, 0.1,0.2,0.4,10,0.1)
```

    matrix([[ 1.        ,  0.        , -0.04794255],
            [ 0.        ,  1.        ,  0.08775826],
            [ 0.        ,  0.        ,  1.        ]])

```python
B_dscr_fn(0.01, 3.0, 0.1,0.2,0.4,10,0.1)
```

    matrix([[-0.04794255,  0.00877583],
            [ 0.08775826,  0.00479426],
            [ 0.03316681,  0.00033278]])

```python
f_dscr_fn(0.01, 3.0, 0.1,0.2,0.4,10,0.1)
```

    matrix([[0.18775826],
            [0.24794255],
            [0.40332778]])

```python

```

```python

```

# Run a simulation to generate test data

Set-up a simulation of a vehicle (bicycle model). The vehicle is controlled to follow a given path. In addition, an intended lateral distance $\Delta l$ is modulated by applying a pre-defined profile to the reference $\Delta l_r$.

```python
path_transform = pt.LateralPathTransformer(wheelbase=3.0)
```

    compiling system store_input_data (level 1)...
    compiling system tracker_loop (level 3)...
    compiling system Subsystem1000 (level 2)...
    compiling system controller (level 2)...
    compiling system simulation_model (level 2)...
    compiling system process_data (level 1)...
    compiling system simulation (level 0)...
    Generated code will be written to generated/tmp1 .

```python
lateral_profile = mp.generate_one_dimensional_motion(Ts=Ts, T_phase1=1, T_phase2=3, T_phase3=1, T_phase4=3, T_phase5=1)

mp.plot_lateral_profile(lateral_profile)
```

```python
output_path = path_transform.run_lateral_path_transformer( track_data, lateral_profile )

plt.figure(figsize=(8,4), dpi=100)
plt.plot( output_path['X'], output_path['Y']+0.1 )
plt.plot( track_data['X'], track_data['Y'] )
plt.legend(['manipulated (output) path', 'original (input) path'])
plt.grid()
plt.xlabel('x [m]')
plt.ylabel('y [m]')
plt.show()
```

```python
output_path.keys()
```

    dict_keys(['D', 'X', 'Y', 'PSI', 'K', 'D_STAR', 'V_DELTA_DOT', 'V_DELTA', 'V_PSI_DOT', 'V_PSI', 'V_VELOCITY'])

# Add noise to the model (sensing noise)

Simulate measurement noise which is, e.g., introduced by GPS.

```python
# X/Y positioning noise (normal distribution)
N = len( output_path['X'] )
eta_x = np.random.normal(0, 0.1, N) * 1
eta_y = np.random.normal(0, 0.1, N) * 1

x_meas       = eta_x + output_path['X']
y_meas       = eta_y + output_path['Y']
psi_meas     = output_path['V_PSI']
psi_dot_meas = output_path['V_PSI_DOT']
v_meas       = output_path['V_VELOCITY']

plt.figure(figsize=(12,8), dpi=70)
plt.plot(x_meas, y_meas)
plt.plot(track_data['X'], track_data['Y'])
plt.show()

plt.figure(figsize=(12,8), dpi=70)
plt.plot(psi_meas)
plt.show()
```

# Extended Kalman filter

The extended Kalman filter is applied to the linearized model descibed by the matrices $A_{dscr}$ and $B_{dscr}$ and given the system function $f_{dscr}$. The simulated data serves as measured data and is the input to the filter.

```python
f_dscr
```

$\displaystyle \left[\begin{matrix}T_{s} v \cos{\left(\delta + \psi \right)} + x\\T_{s} v \sin{\left(\delta + \psi \right)} + y\\\frac{T_{s} v \sin{\left(\delta \right)}}{l_{r}} + \psi\end{matrix}\right]$

```python
A_dscr
```

$\displaystyle \left[\begin{matrix}1 & 0 & - T_{s} v \sin{\left(\delta + \psi \right)}\\0 & 1 & T_{s} v \cos{\left(\delta + \psi \right)}\\0 & 0 & 1\end{matrix}\right]$

```python
B_dscr
```

$\displaystyle \left[\begin{matrix}- T_{s} v \sin{\left(\delta + \psi \right)} & T_{s} \cos{\left(\delta + \psi \right)}\\T_{s} v \cos{\left(\delta + \psi \right)} & T_{s} \sin{\left(\delta + \psi \right)}\\\frac{T_{s} v \cos{\left(\delta \right)}}{l_{r}} & \frac{T_{s} \sin{\left(\delta \right)}}{l_{r}}\end{matrix}\right]$

The implemented filter in form of a loop

```python
l_r = 3.0

# allocate space to store the filter results
results = {'delta' : np.zeros(N), 'x' : np.zeros(N), 'y' : np.zeros(N), 'psi' : np.zeros(N) }

# the guess/estimate of the initial states
X = np.matrix([ [0.5], [0.5], [0.1] ])

P = np.matrix([  [0.1, 0,   0     ],
                 [0,   0.1, 0     ],
                 [0,   0,   0.1   ]  ])

# covariance of the noise w addtitive to the states
Q = 0.00001*np.matrix([  [1, 0, 0 ],
                         [0, 1, 0 ],
                         [0, 0, 1 ]  ])

# covariance of the noise v in the measured system output signal
R = np.matrix([  [0.1, 0   ],
                 [0  , 0.1 ]  ])

for i in range(0,N):

    # measured input signals
    v       = v_meas[i]
    x       = x_meas[i]
    y       = y_meas[i]
    psi_dot = psi_dot_meas[i]

    # compute steering angle by the inverse for the vehicle orientation change
    delta = math.asin( psi_dot * l_r / v )

    # system output vector (x, y)
    z = np.matrix([ [x], [y] ])

    # pridiction step using the non-linear model (f_dscr)
    # x(k-1|k-1) --> x(k|k-1)

    X[0] = X[0] + Ts * ( v * math.cos( X[2] + delta ) )
    X[1] = X[1] + Ts * ( v * math.sin( X[2] + delta ) )
    X[2] = X[2] + Ts * ( v / l_r * math.sin(delta) )

    # optionally use the auto-generated python function for evaluation
    # X = f_dscr_fn( Ts, l_r, float(X[0]), float(X[1]), float(X[2]), v, delta )

    # evaluate jacobi matrices A_dscr and B_dscr
    F = np.matrix([  [1, 0, -Ts*v*math.sin(delta+X[2])  ],
                     [0, 1,  Ts*v*math.cos(delta+X[2])  ],
                     [0, 0, 1                           ]   ])

    # optionally use the auto-generated python function for evaluation
    # F = A_dscr_fn( Ts, l_r, float(X[0]), float(X[1]), float(X[2]), v, delta )

    B = np.matrix([  [-Ts*v*math.sin(delta+X[2]), Ts*math.cos(delta+X[2])      ],
                     [ Ts*v*math.cos(delta+X[2]), Ts*math.sin(delta+X[2])      ],
                     [Ts*v/l_r * math.cos(delta), Ts/l_r * math.sin(delta)     ]   ])

    # optionally use the auto-generated python function for evaluation
    # B = B_dscr_fn( Ts, l_r, float(X[0]), float(X[1]), float(X[2]), v, delta )

    # the system output matrix: returns X and Y when multiplied with the state vector X
    # which are compared to the measurements
    H = np.matrix([  [1,0,0],
                     [0,1,0] ])

    # prdicted state covariance P(k|k-1)
    P = F*P*F.transpose() + Q

    # estimation output residual vector
    e = z - H*X

    # Kalman gain
    S = H*P*H.transpose() + R
    K = P*H.transpose() * np.linalg.inv( S )

    # post priori state X(k|k)
    X = X + K*e

    # post priori covariance
    P = (np.eye(3) - K*H) * P

    # store results
    results['delta'][i] = delta
    results['x'][i] = X[0]
    results['y'][i] = X[1]
    results['psi'][i] = X[2]

# show results
plt.figure(figsize=(12,8), dpi=79)
#plt.plot(x_meas, y_meas, '+')
plt.plot(output_path['X'], output_path['Y'], 'g')
plt.plot(results['x'], results['y'], 'r')
plt.show()

plt.figure(figsize=(12,8), dpi=70)
plt.plot(results['psi'])
plt.plot(output_path['V_PSI'], 'g')
plt.show()
```

```python

```