_Lambda School Data Science — Linear Models_

# Understanding Linear Regression

#### Objectives
- understand how ordinary least squares regression minimizes the sum of squared errors
- understand how linear algebra can solve ordinary least squares regression
- get and interpret coefficients of a linear model
- visualize a line of best fit in 2D, and hyperplane in 3D
- use regression metrics: MSE, RMSE, MAE, R^2

#### Extra Links
- [Statistics 101: Simple Linear Regression](https://www.youtube.com/watch?v=ZkjP5RJLQF4) (20 minute video)
- [_An Introduction to Statistical Learning_](http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf), Chapter 3.1, Simple Linear Regression, & 3.2, Multiple Linear Regression
- Priceonomics, [The Discovery of Statistical Regression](https://priceonomics.com/the-discovery-of-statistical-regression/)
- Priceonomics, [Why the Father of Modern Statistics Didn’t Believe Smoking Caused Cancer](https://priceonomics.com/why-the-father-of-modern-statistics-didnt-believe/)
- Harvard Business Review, [When to Act on a Correlation, and When Not To](https://hbr.org/2014/03/when-to-act-on-a-correlation-and-when-not-to)
- [xkcd 552: Correlation](https://www.explainxkcd.com/wiki/index.php/552:_Correlation)
- [xkcd 1725: Linear Regression](https://www.explainxkcd.com/wiki/index.php/1725:_Linear_Regression)

## What is Linear Regression?

[Linear Regression](https://en.wikipedia.org/wiki/Linear_regression) is a statistical model that seeks to describe the relationship between some y variable and one or more x variables.

In the simplest case, linear regression seeks to fit a straight line through a cloud of points. This line is referred to as the "regression line" or "line of best fit." This line tries to summarize the relationship between our X and Y in a way that enables us to use the equation for that line to make predictions.

```python
%matplotlib inline
import seaborn as sns
x = [10,  8, 13,  9, 11, 14,  6,  4, 12,  7,  5]
y = [ 8,  6,  7,  8,  8,  9,  7,  4, 10,  4,  5]
sns.regplot(x, y);
```

### Synonyms for "y variable"
- Dependent Variable
- Response Variable
- Outcome Variable
- Predicted Variable
- Measured Variable
- Explained Variable
- Label
- Target

### Synonyms for "x variable"
- Independent Variable
- Explanatory Variable
- Regressor
- Covariate
- Feature

## Simple Linear Regresion

#### Making Predictions

Say that we were trying to create a model that captured the relationship between temperature outside and ice cream sales. In Machine Learning our goal is often different that of other flavors of Linear Regression Analysis, because we're trying to fit a model to this data with the intention of making **predictions** on new data (in the future) that we don't have yet.

#### What are we trying to predict?

So if we had measured ice cream sales and the temprature outside on 11 different days, at the end of our modeling **what would be the thing that we would want to predict? - Ice Cream Sales or Temperature?**

We would probably want to be measuring temperature with the intention of using that to **forecast** ice cream sales. If we were able to successfully forecast ice cream sales from temperature, this might help us know beforehand how much ice cream to make or how many cones to buy or on which days to open our store, etc. Being able to make predictions accurately has a lot of business implications. This is why making accurate predictions is so valuable (And in large part is why data scientists are paid so well).

#### y variable intuition

We want the thing that we're trying to predict to serve as our **y** variable. This is why it's sometimes called the "predicted variable." We call it the "dependent" variable because our prediction for how much ice cream we're going to sell "depends" on the temperature outside.

#### x variable intuition

All other variables that we use to predict our y variable (we're going to start off just using one) we call our **x** variables. These are called our "independent" variables because they don't *depend* on y, they "explain" y. Hence they are also referred to as our "explanatory" variables.

## Example: Predict presidential election voting

#### Douglas Hibbs, [Background Information on the ‘Bread and Peace’ Model of Voting in Postwar US Presidential Elections](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)

> Aggregate two-party vote shares going to candidates of the party holding the presidency during the postwar era are well explained by just two fundamental determinants:

> (1) Positively by weighted-average growth of per capita real disposable personal income over the term.
> (2) Negatively by cumulative US military fatalities (scaled to population) owing to unprovoked, hostile deployments of American armed forces in foreign wars.

#### Data sources
- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

```python
import pandas as pd

columns = ['Year','Incumbent Party Candidate','Other Candidate','Incumbent Party Vote Share']

data = [[1952,"Stevenson","Eisenhower",44.6],
        [1956,"Eisenhower","Stevenson",57.76],
        [1960,"Nixon","Kennedy",49.91],
        [1964,"Johnson","Goldwater",61.34],
        [1968,"Humphrey","Nixon",49.60],
        [1972,"Nixon","McGovern",61.79],
        [1976,"Ford","Carter",48.95],
        [1980,"Carter","Reagan",44.70],
        [1984,"Reagan","Mondale",59.17],
        [1988,"Bush, Sr.","Dukakis",53.94],
        [1992,"Bush, Sr.","Clinton",46.55],
        [1996,"Clinton","Dole",54.74],
        [2000,"Gore","Bush, Jr.",50.27],
        [2004,"Bush, Jr.","Kerry",51.24],
        [2008,"McCain","Obama",46.32],
        [2012,"Obama","Romney",52.00],
        [2016,"Clinton","Trump",48.2]]

votes = pd.DataFrame(data=data, columns=columns)
```

```python
columns = ['Year','Average Recent Growth in Personal Incomes']

data = [[1952,2.40],
        [1956,2.89],
        [1960, .85],
        [1964,4.21],
        [1968,3.02],
        [1972,3.62],
        [1976,1.08],
        [1980,-.39],
        [1984,3.86],
        [1988,2.27],
        [1992, .38],
        [1996,1.04],
        [2000,2.36],
        [2004,1.72],
        [2008, .10],
        [2012, .95],
        [2016, .10]]

growth = pd.DataFrame(data=data, columns=columns)
```

```python
"""
Fatalities denotes the cumulative number of American military
fatalities per millions of US population the in Korea, Vietnam,
Iraq and Afghanistan wars during the presidential terms
preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and
2012 elections.

http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf
"""

columns = ['Year','US Military Fatalities per Million']

data = [[1952,190],
        [1956,  0],
        [1960,  0],
        [1964,  1],
        [1968,146],
        [1972,  0],
        [1976,  2],
        [1980,  0],
        [1984,  0],
        [1988,  0],
        [1992,  0],
        [1996,  0],
        [2000,  0],
        [2004,  4],
        [2008, 14],
        [2012,  5],
        [2016,  5]]

deaths = pd.DataFrame(data=data, columns=columns)
```

```python
df = votes.merge(growth).merge(deaths)
```

### Plot univariate correlations
[Seaborn tutorial: Visualizing linear relationships](https://seaborn.pydata.org/tutorial/regression.html)

```python
%matplotlib inline
import seaborn as sns

target = 'Incumbent Party Vote Share'
features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million']

for feature in features:
    sns.lmplot(x=feature, y=target, data=df)
```

We can see from the scatterplot that these data points seem to follow a somewhat linear relationship for the "Average Recent Growth in Personal Incomes" feature. This means that we could probably summarize their relationship well by fitting a line of best fit to these points. Lets do it.

## The Equation for a Line

A common equation for a line is:

\begin{align}
y = mx + b
\end{align}

Where $m$ is the slope of our line and $b$ is the y-intercept.

If we want to plot a line through our cloud of points we figure out what these two values should be. Linear Regression seeks to **estimate** the slope and intercept values that describe a line that best fits the data points.

## The Anatomy of Linear Regression

- Intercept: The $b$ value in our line equation $y=mx+b$
- Slope: The $m$ value in our line equation $y=mx+b$. These two values together define our regression line.
- $\hat{y}$ : A prediction
- Line of Best Fit (Regression Line)
- Predicted (fitted) Values: Points on our regression line
- Observed Values: Points from our dataset
- Error: The distance between predicted and observed values.

Ordinary Least Squares Regression is a way to solve for $m$ and $b$.

Let's start by seeing what would happen if we just guessed and checked some values for $m$ and $b$.

What's the line of "best" fit look like? What's the error?

```python
# TODO
```

## R Squared:  $R^2$

One final attribute of linear regressions that we're going to talk about today is a measure of goodness of fit known as $R^2$ or R-squared. $R^2$ is a statistical measure of how close the data are fitted to our regression line. A helpful interpretation for the $R^2$ is the percentage of the dependent variable that is explained by the model.

In other words, the $R^2$ is the percentage of y that is explained by the x variables included in the model. For this reason the $R^2$ is also known as the "coefficient of determination," because it explains how much of y is explained (or determined) by our x varaibles. We won't go into the calculation of $R^2$ today, just know that a higher $R^2$ percentage is nearly always better and indicates a model that fits the data more closely.

## Residual Error

The residual error is the distance between points in our dataset and our regression line.

```python
def regression_residuals(df, feature, target, m, b):
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    plt.scatter(x, y, label='y_true')
    plt.plot(x, y_pred, label='y_pred')
    plt.legend()

    # Plot residual errors
    for x, y1, y2 in zip(x, y, y_pred):
        plt.plot((x, x), (y1, y2), color='grey')

    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

regression_residuals(df, feature='Average Recent Growth in Personal Incomes',
                     target='Incumbent Party Vote Share', m=3, b=46)
```

```python
from ipywidgets import interact, fixed

interact(regression_residuals,
         df=fixed(df),
         feature=fixed('Average Recent Growth in Personal Incomes'),
         target=fixed('Incumbent Party Vote Share'),
         m=(-10,10,0.5),
         b=(40,60,0.5));
```

## More Formal Notation

We have talked about a line of regression being represented like a regular line $y=mx+b$ but as we get to more complicated versions we're going to need to extend this equation. So lets establish the proper terminology.

**X** - Independent Variable, predictor variable, explanatory variable, regressor, covariate

**Y** - Response variable, predicted variable, measured vairable, explained variable, outcome variable

$\beta_0$ - "Beta Naught" or "Beta Zero", the intercept value. This is how much of y would exist if X were zero. This is sometimes represented by the letter "a" but I hate that. So it's "Beta 0" during my lecture.

$\beta_1$ - "Beta One" The primary coefficient of interest. This values is the slope of the line that is estimated by "minimizing the sum of the squared errors/residuals" - We'll get to that.

$\epsilon$ - "Epsilon" The "error term", random noise, things outside of our model that affect y.

## Minimizing the Sum of the Squared Error

The most common method of estimating our $\beta$ parameters  is what's known as "Ordinary Least Squares" (OLS). (There are different methods of arriving at a line of best fit). OLS estimates the parameters that minimize the squared distance between each point in our dataset and our line of best fit.

\begin{align}
SSE = \sum(y_i - \hat{y})^2
\end{align}

```python
from matplotlib.patches import Rectangle
import numpy as np
from sklearn.metrics import mean_squared_error

def regression_squared_errors(df, feature, target, m, b):
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    fig = plt.figure(figsize=(7,7))
    ax = plt.axes()
    ax.scatter(x, y, label='y_true')
    ax.plot(x, y_pred, label='y_pred')
    ax.legend()

    # Plot square errors
    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)
    for x, y1, y2 in zip(x, y, y_pred):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))

    mse = mean_squared_error(y, y_pred)
    rmse = np.sqrt(mse)
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Squared Error:', mse)
    print('Root Mean Squared Error:', rmse)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

regression_squared_errors(df, feature='Average Recent Growth in Personal Incomes',
           target='Incumbent Party Vote Share', m=3, b=46)
```

```python
interact(regression_squared_errors,
         df=fixed(df),
         feature=fixed('Average Recent Growth in Personal Incomes'),
         target=fixed('Incumbent Party Vote Share'),
         m=(-10,10,0.5),
         b=(40,60,0.5));
```

## Hypotheses

```python
b = 46
ms = np.arange(-10,10,0.5)
sses = []
feature = 'Average Recent Growth in Personal Incomes'

for m in ms:
    predictions = m * df[feature] + b
    errors = predictions - df[target]
    square_errors = errors ** 2
    sse = square_errors.sum()
    sses.append(sse)

hypotheses = pd.DataFrame({'Slope': ms})
hypotheses['Intercept'] = b
hypotheses['Sum of Square Errors'] = sses

hypotheses
```

```python
hypotheses.plot(x='Slope', y='Sum of Square Errors',
                title=f'Intercept={b}');
```

## Scikit-learn

#### Jake VanderPlas, [Python Data Science Handbook, Chapter 5.2, Introducing Scikit-Learn](https://jakevdp.github.io/PythonDataScienceHandbook/05.02-introducing-scikit-learn.html), Scikit-Learn's Estimator API

> Most commonly, the steps in using the Scikit-Learn estimator API are as follows (we will step through a handful of detailed examples in the sections that follow).

> 1. Choose a class of model by importing the appropriate estimator class from Scikit-Learn.
> 2. Choose model hyperparameters by instantiating this class with desired values.
> 3. Arrange data into a features matrix and target vector following the discussion above.
> 4. Fit the model to your data by calling the `fit()` method of the model instance.
> 5. Apply the Model to new data: For supervised learning, often we predict labels for unknown data using the `predict()` method.

```python
# TODO
```

## Linear Algebra!

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

\begin{align}
\hat{\beta} = (X^{T}X)^{-1}X^{T}y
\end{align}

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

### Lets calculate our $\beta$ coefficients with numpy!

```python
from statsmodels.api import add_constant

X = add_constant(df[feature].values)
print('X')
print(X)

y = df[target].values[:, np.newaxis]
print('y')
print(y)

X_transpose = X.T
print('X Transpose')
print(X_transpose)

X_transpose_X = X_transpose @ X
print('X Transpose X')
print(X_transpose_X)

X_transpose_X_inverse = np.linalg.inv(X_transpose_X)
print('X Transpose X Inverse')
print(X_transpose_X_inverse)

X_transpose_y = X_transpose @ y
print('X Transpose y')
print(X_transpose_y)

beta_hat = X_transpose_X_inverse @ X_transpose_y
print('Beta Hat')
print(beta_hat)
```

# Multiple Regression

Simple or bivariate linear regression involves a single $x$ variable and a single $y$ variable. However, we can have many $x$ variables. A linear regression model that involves multiple x variables is known as **Multiple** Regression (NOT MULTIVARIATE).

\begin{align}
y = \beta_0 + \beta_1 X_1 + \beta_2 X_2 + ... + \beta_n X_n + \epsilon
\end{align}

```python
# TODO
```

## Visualize hyperplane of best fit in 3D

```python
# https://stackoverflow.com/a/47230966
# Plotly notebook mode with google colaboratory
# You need to define this function
# And call it in each offline plotting cell

def configure_plotly_browser_state():
    import IPython
    display(IPython.core.display.HTML('''

        '''))
```

```python
import itertools
import plotly.graph_objs as go
from plotly.offline import init_notebook_mode, iplot
init_notebook_mode(connected=True)

def viz3D(fitted_model, X, features, target='', num=100):
    """
    Visualize model predictions in 3D, for regression model fit on 2 features

    Parameters
    ----------
    fitted_model : scikit-learn model, already fitted
    X : pandas dataframe, which was used to fit model
    features : list of strings, column names of the 2 features
    target : string, name of target
    num : int, number of grid points for each feature

    References
    ----------
    https://plot.ly/python/3d-charts/
    """
    feature1, feature2 = features
    min1, max1 = X[feature1].min(), X[feature1].max()
    min2, max2 = X[feature2].min(), X[feature2].max()
    x1 = np.linspace(min1, max1, num)
    x2 = np.linspace(min2, max2, num)
    combos = list(itertools.product(x1, x2))
    Z = fitted_model.predict(combos).reshape(num, num)

    configure_plotly_browser_state()
    data = [go.Surface(x=x1, y=x2, z=Z)]
    layout = go.Layout(
        scene={'xaxis': {'title': feature1, 'range': [min1,max1], 'showticklabels': True},
               'yaxis': {'title': feature2, 'range': [min2,max2], 'showticklabels': True},
               'zaxis': {'title': target, 'showticklabels': True}},
    )
    fig = go.Figure(data=data, layout=layout)
    iplot(fig)
```

```python
# TODO
```

## Dimensionality in Linear Regression

Muliple Regression is simply an extension of the bivariate case. The reason why we see the bivariate case demonstrated so often is simply because it's easier to graph and all of the intuition from the bivariate case is the same as we keep on adding explanatory variables.

As we increase the number of $x$ values in our model we are simply fitting a n-1-dimensional plane to an n-dimensional cloud of points within an n-dimensional hypercube.

## Interpreting Coefficients

One of Linear Regression's strengths is that the parameters of the model (coefficients) are readily interpretable and useful. Not only do they describe the relationship between x and y but they put a number on just how much x is associated with y. We should be careful to not speak about this relationshiop in terms of causality because these coefficients are in fact correlative measures. We would need a host of additional techniques in order to estimate a causal effect using linear regression (econometrics).

\begin{align}
\hat{\beta} = \frac{Cov(x,y)}{Var(y)}
\end{align}

## Why is Linear Regression so Important?

### Popularity

Linear Regression is an extremely popular technique that every data scientist **needs** to understand. It's not the most advanced technique and there are supervised learning techniques that will obtain a higher accuracy, but where it lacks in accuracy it makes up for it in interpretability and simplicity.

### Interpretability

Few other models possess coefficients that are so directly linked to their variables with a such a clear interpretation. Tomorrow we're going to learn about ways to make them even easier to interpret.

### Simplicity

A linear regression model can be communicated just by writing out its equation. It's kind of incredible that such high dimensional relationships can be described from just a linear combination of variables and coefficients.

# Assignment
- Continue to predict New York City apartment rents. This is your last assignment with this dataset.
- You may select any number of features. You are encouraged to engineer new features.
- Get and plot your model's coefficients.
- Report your Root Mean Squared Error, Mean Absolute Error, and R^2 Score, for your Train and Test sets. Share your scores with your cohort on Slack!
- Fit a model with 2 features, and visualize the plane of best fit in 3D.
- Commit your notebook to your fork of the repo.

## Stretch Goals

Study more about Linear Regression. Here are two helpful links. If you find more links, share your favorites with your cohort on Slack.

1. Watch this 20 minute video that just hit 1 million views: Brandon Foltz, Statistics 101: Simple Linear Regression (https://www.youtube.com/watch?v=ZkjP5RJLQF4)
2. Skim _An Introduction to Statistical Learning_, Chapter 3.1, Simple Linear Regression, & Chapter 3.2, Multiple Linear Regression (http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf)

In your 3D visualization, can you include the actual datapoints, like in [this notebook](https://nbviewer.jupyter.org/urls/s3.amazonaws.com/datarobotblog/notebooks/multiple_regression_in_python.ipynb)? Can you also include the residual lines from the datapoints to the plane of the best fit, like in _An Introduction to Statistical Learning?_ This would be hard to do, but awesome!

Can you get creative with feature engineering? Share with your cohort on Slack. We mentioned some feature ideas at the end of last lesson, but didn't demonstrate how to engineer them. So here are some example solutions:

```python
# Does apartment have a non-empty description?
df['description'] = df['description'].str.strip().fillna('')
df['has_description'] = df['description'] != ''

# How long is the description?
df['description_length'] = df['description'].str.len()

# How many total perks does each apartment have?
perk_cols = ['elevator', 'cats_allowed', 'hardwood_floors', 'dogs_allowed',
             'doorman', 'dishwasher', 'no_fee', 'laundry_in_building',
             'fitness_center', 'pre-war', 'laundry_in_unit', 'roof_deck',
             'outdoor_space', 'dining_room', 'high_speed_internet', 'balcony',
             'swimming_pool', 'new_construction', 'exclusive', 'terrace',
             'loft', 'garden_patio', 'common_outdoor_space',
             'wheelchair_access']
df['perk_count'] = df[perk_cols].sum(axis=1)

# Are pets allowed?
df['pets_allowed'] = (df['cats_allowed']==1) | (df['dogs_allowed']==1)
```


### Functions

```
#             Detect Intersection             #
import math
from sklearn.preprocessing import MinMaxScaler
from sklearn.cluster import KMeans
from sympy import Line

def line(p1, p2):
    A = (p1[1] - p2[1])
    B = (p2[0] - p1[0])
    C = (p1[0]*p2[1] - p2[0]*p1[1])
    return A, B, -C

def intersection2(L1, L2):
    D  = L1[0] * L2[1] - L1[1] * L2[0]
    Dx = L1[2] * L2[1] - L1[1] * L2[2]
    Dy = L1[0] * L2[2] - L1[2] * L2[0]
    if D != 0:
        x = Dx / D
        y = Dy / D
        return x,y
    else:
        return False

def regression(img, x, y, color=(255, 0, 0)):

  y_at_border = np.array([0, img.shape[0]])
  p = np.polyfit(y, x, deg=1)
  x_at_border = np.poly1d(p)(y_at_border)

  cv2.line(img, (int(x_at_border[0]), int(y_at_border[0])), (int(x_at_border[1]), int(y_at_border[1])), color, 2)

  return x_at_border, y_at_border

def drawLines(img, lines, color=(255,0,0)):
  """
  Draw lines on an image
  """

  centroids = list()
  r_xs = list()
  r_ys = list()
  for line_ in lines:
      for rho,theta in line_:
          a = np.cos(theta)
          b = np.sin(theta)
          x0 = a*rho
          y0 = b*rho
          x1 = int(x0 + 1000*(-b))
          y1 = int(y0 + 1000*(a))
          x2 = int(x0 - 1000*(-b))
          y2 = int(y0 - 1000*(a))

          slope = (y1 - y0) / float(x1 - x0)
          angle = math.degrees(math.atan(slope))
          if abs(angle) > 80:
            # print(img.shape[1])
            h_layout = line((0, 0), (img.shape[1], 0))
            h_layout_lower = line((0, img.shape[0]), (img.shape[1], img.shape[0]))
            r = intersection2(h_layout, line((x1, y1), (x2, y2)))
            r_lower = intersection2(h_layout_lower, line((x1, y1), (x2, y2)))
            # cv2.line(img, (int(x1), int(y1)), (int(x2), int(y2)), color, 2)
            cv2.line(img, (int(r[0]), int(r[1])), (int(r_lower[0]), int(r_lower[1])), color, 2)
            center_p = (int((r[0] + r_lower[0]) / 2), int((r[1] + r_lower[1])/ 2))
            centroids.append(center_p)

            r_xs.append((r[0], r_lower[0]))
            r_ys.append((r[1], r_lower[1]))
            cv2.circle(img, center_p, 10, (255, 0, 255), -1)

  cv2.line(img, (int(0), int(0)), (int(0), int(img.shape[0])), color, 2)
  cv2.line(img, (int(img.shape[1]), int(0)), (int(img.shape[1]), int(img.shape[0])), color, 2)
  cv2.circle(img, (0, int(img.shape[0] / 2)), 10, (255, 0, 255), -1)
  cv2.circle(img, (img.shape[1], int(img.shape[0] / 2)), 10, (255, 0, 255), -1)
  centroids.append((0, int(img.shape[0] / 2)))
  centroids.append((img.shape[1], int(img.shape[0] / 2)))

  return r_xs, r_ys, centroids

from scipy.spatial import distance as sci_dist

def order_points(pts):

  xSorted = pts[np.argsort(pts[:, 0]), :]
  leftMost = xSorted[:2, :]
  rightMost = xSorted[2:, :]
  leftMost = leftMost[np.argsort(leftMost[:, 1]), :]
  (tl, bl) = leftMost
  D = sci_dist.cdist(tl[np.newaxis], rightMost, "euclidean")[0]
  (br, tr) = rightMost[np.argsort(D)[::-1], :]

  return np.array([tl, tr, br, bl], dtype="float32")

```

```
from PIL import Image
import os
import matplotlib.pyplot as plt
import numpy as np
from XiaohuLuVPDetection.lu_vp_detect.vp_detection import VPDetection
import time
# import os
import cv2
# import pylab as pl
from skimage import morphology as mp
import sys

length_thresh = 50
principal_point = None
focal_length = 1300 # 1102.79
seed = 1300
vpd = VPDetection(length_thresh, principal_point, focal_length, seed)

# org_path = 'datasets/lsun/images/' # -> original image
# layout_path = 'drn_d_105_024_val_ms/images/' # -> layout image (result image frome above model file)
# refer_path = '../refer_data/wall/myxkehu1kfzggursepnk0tfiyps8zbs5umvzv8d92r6hhxejgawebwsufssgov5q-.jpg'
# val_image_list = os.listdir(layout_path)

#     Refering for one image data   #
def refering(org_path, layout_path, refer_path):

  # print(image)
  img = Image.open(layout_path + image)
  # print(type(img))
  img_np = np.invert(np.asarray(img))
  # print(img_np.max(), img_np.min())
  ret, thr = cv2.threshold(img_np, 254, 255, cv2.THRESH_BINARY_INV)

  org = Image.open(org_path + image)
  org_np = np.asarray(org)

  vps = vpd.find_vps(org_path + image)
  vl_img = vpd.create_debug_VP_image()

  # print(image)
  # plt.subplot(131)
  # plt.imshow(org)
  # plt.show()
  img_size = (org_np.shape[1], org_np.shape[0])

  scale_factor = 6
  print('scale_factor :', scale_factor)

  refer = Image.open(refer_path)
  refer = np.asarray(refer)

  #     refer_size와 img_size가 동일하거나 refer_size가 작은 경우를 고려해야한다.     #
  refer = np.tile(refer, (scale_factor, scale_factor, 1))
  size_ratio = math.floor(min((refer.shape[0] / (org_np.shape[0] * 1.5)), (refer.shape[1] / (org_np.shape[1] * 1.5))))
  refer = Image.fromarray(refer).resize((int(refer.shape[1] / size_ratio), int(refer.shape[0] / size_ratio)))

  tun = thr
  skl = mp.medial_axis(tun).astype(np.uint8) * 255
  rho = 1
  theta = np.pi/180
  thresh = 50
  lines = cv2.HoughLines(skl, rho, theta, thresh)

  # print(lines)
  # plt.imshow(skl)
  # plt.show()

  # Draw all Hough lines in red
  img_with_all_lines = np.copy(skl)
  img_with_all_lines = cv2.cvtColor(img_with_all_lines, cv2.COLOR_GRAY2RGB)
  # plt.imshow(img_with_all_lines)
  # plt.show()
  org_np = org_np.astype(np.uint8)
  org_np2 = org_np.copy()

  start = time.time()

  r_x_list, r_y_list, centroids_list = drawLines(org_np2, lines)
  drawLines(img_with_all_lines, lines)

  vps = vpd.find_vps(skl)
  vl_img, vl_list = vpd.create_debug_VP_image()

  centroids_data = np.array(centroids_list)[:, [0]]
  # print('len(r_x_list), len(r_y_list) :', len(r_x_list), len(r_y_list))
  print('centroids_data.shape :', centroids_data.shape)
  mms = MinMaxScaler()
  cen_data =mms.fit_transform(centroids_data)
  # print('cen_data :', cen_data)
  K = range(2, 6)
  s_dist = list()
  for k in K:
    if cen_data.shape[0] < k:
      break
    km = KMeans(n_clusters=k)
    km = km.fit(cen_data)
    inertia = km.inertia_
    s_dist.append(inertia)

    # if inertia <= 0:
    #   print('vertical line number =', k)
    #   break
  mms2 = MinMaxScaler()
  mms_s_dist = mms2.fit_transform(np.array(s_dist).reshape(-1, 1))
  k_thresh = 0.1
  # plt.plot(range(len(mms_s_dist)), mms_s_dist)
  # plt.show()

  if cen_data.shape[0] > 2:
    for i in range(len(mms_s_dist)):
      if mms_s_dist[i] < k_thresh:
        k = i + 2
        break
  else:
    k = 2

  #     k Confirmation : Comparing k.cluster_centers_ dist  #
  while True:

    km = KMeans(n_clusters=k)
    km = km.fit(cen_data)
    if k <= 2:
      break

    cluster_centroids = km.cluster_centers_
    # print('cluster_centroids :', cluster_centroids)
    error_exist = False
    for i in range(len(cluster_centroids) - 1):
      for j in range(i + 1, len(cluster_centroids)):
        if abs(cluster_centroids[i] - cluster_centroids[j]) < 0.05:
          error_exist = True
    if error_exist:
      k -= 1
    else:
      break

  print('k is', k)

  predict_cen = km.predict(cen_data)
  print(predict_cen)

  keys = list(range(k))
  for rm in predict_cen[-2:]:
    keys.remove(rm)
  print('keys :', keys)

  skl_rgb = cv2.cvtColor(skl, cv2.COLOR_GRAY2RGB)
  skl_copy = np.copy(skl_rgb)

  # black_plane = np.zeros(skl_rgb.shape).astype(np.uint8)
  black_plane2 = np.zeros(skl_rgb.shape).astype(np.uint8)

  reg_xs = list()
  reg_ys = list()
  for key in keys:
    temp_rx = tuple()
    temp_ry = tuple()
    for pred_key, r_x, r_y in zip(predict_cen[:-2], r_x_list, r_y_list):
      if key == pred_key:
        temp_rx += r_x
        temp_ry += r_y

    #       Regression Multiple Lines     #
    border_x, border_y = regression(org_np2, temp_rx, temp_ry)
    # regression(black_plane, temp_rx, temp_ry)
    # print(border_x, border_y)

    reg_xs.append(border_x)
    reg_ys.append(border_y)

  #           VL Segmentation             #
  # Angle between line1 and x-axis
  # print(len(vl_list))
  v_lines = list()
  h_lines = list()

  for vl in vl_list:
    x0, y0, x1, y1 = vl
    slope = (y1 - y0) / float(x1 - x0)
    angle = math.degrees(math.atan(slope))
    # print(angle)

    if abs(angle) > 80:
      # cv2.line(skl_copy, (int(x1), int(y1)), (int(x0), int(y0)), (255, 0, 0), 3,
      #                  cv2.LINE_AA)
      v_lines.append(vl)

    elif abs(angle) < 70:
      # cv2.line(skl_copy, (int(x1), int(y1)), (int(x0), int(y0)), (0, 0, 255), 2,
      #                  cv2.LINE_AA)
      h_lines.append(vl)

  # print('len(v_lines) :', len(v_lines))
  # print('len(h_lines) :', len(h_lines))

  #         Extend v_lines and draw h_lines       #
  all_closest_inters = list()
  all_centroid_inters = list()
  h_border = line((0, 0), (skl_copy.shape[1], 0))
  h_border_lower = line((0, skl_copy.shape[0]), (skl_copy.shape[1], skl_copy.shape[0]))
  for reg_x, reg_y in zip(reg_xs, reg_ys):

      left_vh_intersections = list()
      right_vh_intersections = list()

      left_angle = list()
      right_angle = list()

      vline = line((reg_x[0], reg_y[0]), (reg_x[1], reg_y[1]))
      intersections = list()
      for h_line in h_lines:
        hline = line(h_line[:2], h_line[2:])
        center_h_x = (h_line[0] + h_line[2]) / 2
        center_v_x = (reg_x[0] + reg_x[1]) / 2

        slope = (h_line[3] - h_line[1]) / float(h_line[2] - h_line[0])
        angle = math.degrees(math.atan(slope))

        if center_h_x < center_v_x:
          left_vh_intersections.append(intersection2(vline, hline))
          left_angle.append(angle)
        else:
          right_vh_intersections.append(intersection2(vline, hline))
          right_angle.append(angle)

      print('len(left_vh_intersections) :', len(left_vh_intersections))
      print('len(right_vh_intersections) :', len(right_vh_intersections))

      #     Find Close Intersection Points in vline     #
      close_points = list()
      close_thr = 30
      angle_gap = 10
      for (ix, iy), l_angle in zip(left_vh_intersections, left_angle):
        for (jx, jy), r_angle in zip(right_vh_intersections, right_angle):
          dist = math.hypot(jx - ix, jy - iy)
          # print(dist)
          if dist < close_thr:
            # print(l_angle, r_angle)
            if abs(l_angle - r_angle) > angle_gap:
              intersections.append((ix, iy, jx, jy))
              # cv2.circle(black_plane, (int(ix), int(iy)), 5, (255, 0, 255), -1)
              # cv2.circle(black_plane, (int(jx), int(jy)), 5, (255, 255, 0), -1)

      #     Find intersections between horizontal border and vertical lines    #
      r = intersection2(h_border, vline)
      r_lower = intersection2(h_border_lower, vline)
      # print(r)
      # print(r + r)

      intersections.append(r + r)
      intersections.append(r_lower + r_lower)
      intersections_ = intersections.copy()

      imms = MinMaxScaler()
      intersections = np.array(intersections)[:, [1, 3]]
      inter_data =imms.fit_transform(intersections)
      K = range(2, 5)
      s_dist = list()
      for k in K:
        if inter_data.shape[0] < k:
          break
        km = KMeans(n_clusters=k)
        km = km.fit(inter_data)
        inertia = km.inertia_
        s_dist.append(inertia)
      # plt.plot(s_dist)
      # plt.show()

      print(s_dist)
      mms2 = MinMaxScaler()
      mms_s_dist = mms2.fit_transform(np.array(s_dist).reshape(-1, 1))
      k_thresh = 0.025
      # print(mms_s_dist)
      # plt.plot(range(len(mms_s_dist)), mms_s_dist)
      # plt.show()

      if inter_data.shape[0] > 2:
        for i in range(len(mms_s_dist)):
          if mms_s_dist[i] < k_thresh:
            k = i + 2
            break
          elif i == len(mms_s_dist) - 1:
            k = 4
      else:
        k = 2

      #     k Confirmation : Comparing k.cluster_centers_ dist  #
      while True:
        km = KMeans(n_clusters=k)
        km = km.fit(inter_data)
        if k <= 2:
          break
        cluster_centroids = km.cluster_centers_[:, [1]]
        # print('cluster_centroids :', cluster_centroids)
        error_exist = False
        for i in range(len(cluster_centroids) - 1):
          for j in range(i + 1, len(cluster_centroids)):
            if abs(cluster_centroids[i] - cluster_centroids[j]) < 0.05:
              error_exist = True
        if error_exist:
          k -= 1
        else:
          break

      print('k is ', k)
      predict_inter = km.predict(inter_data)
      print(predict_inter)

      keys = list(range(k))
      for rm in predict_inter[-2:]:
        keys.remove(rm)
      print('keys :', keys)

      #     해당 키 안에서의 closest intersection 두쌍의 centroid를 구하면 된다.      #
      centroid_inters = list()
      closest_inters = list()
      temp_black = black_plane2.copy()
      for key in keys:
        temp_inter_left = list()
        temp_inter_right = list()

        for pred_key, inter_point in zip(predict_inter[:-2], intersections_[:-2]):
          if key == pred_key:
            temp_inter_left.append(inter_point[:2])
            temp_inter_right.append(inter_point[2:])
          # else:
          #   cv2.circle(black_plane2, (int(inter_point[0]), int(inter_point[1])), 5, (255, 0, 255), -1)
        # plt.imshow(temp_black)
        # plt.show()

        print('len(temp_inter_left) :', len(temp_inter_left))
        print('len(temp_inter_right) :', len(temp_inter_right))
        min_dist = close_thr
        closest_p = None
        closest_inter = None
        for ix, iy in temp_inter_left:
          for jx, jy in temp_inter_right:
            dist = math.hypot(jx - ix, jy - iy)
            # print(dist)
            if dist < min_dist:
                min_dist = dist
                closest_p = ((ix + jx) / 2, (iy + jy) / 2)
                closest_inter = [ix, iy, jx, jy]
                # closest_inter = (ix, iy, jx, jy)

              # print('min_dist :', min_dist)

        if closest_p:
          centroid_inters.append(closest_p)
          closest_inters.append(closest_inter)
          # cv2.circle(black_plane, (int(closest_p[0]), int(closest_p[1])), 5, (0, 255, 0), -1)

      #       여기서 append하는 closeest_inters는 항상 2개를 유지해야한다.    #
      #       단, intersection이 1개이면 나중에 middle section refering 혼란을 방지하기 위해 좀 다른 형식으로     #
      #       border intersection을 숨겨서 보내준다.      #
      #       intersection이 존재하지 않는 vline은 없앤다.      #
      if len(closest_inters) != 0:
        if len(closest_inters) == 1:
          #       check ceil / floor type inter       #
          #       ceil condition      #
          if (closest_inters[0][1] + closest_inters[0][3]) / 2 < (1 / 3) * org_np.shape[0]:
            opposite_inters = r_lower
          else:
            opposite_inters = r
          closest_inters = [closest_inters[0] + list(opposite_inters)]
          # print('closest_inters :', closest_inters)

        all_centroid_inters.append(centroid_inters)
        all_closest_inters.append(closest_inters)

  h_intersections = list()
  v_border = line((0, 0), (0, skl_copy.shape[0]))
  v_border_lower = line((skl_copy.shape[1], 0), (skl_copy.shape[1], skl_copy.shape[0]))
  for h_line in h_lines:

      #       Find Intersection between v_line, h_line      #
      vh_intersections_x = list()
      vh_intersections_y = list()

      hline = line(h_line[:2], h_line[2:])
      for reg_x, reg_y in zip(reg_xs, reg_ys):
        vline = line((reg_x[0], reg_y[0]), (reg_x[1], reg_y[1]))

        #     Extract only x - coordination     #
        vh_intersections_x.append(intersection2(vline, hline)[0])
        vh_intersections_y.append(intersection2(vline, hline)[1])

      # h_x = np.array([h_line[0], h_line[2]])
      # h_y = np.array([h_line[1], h_line[3]])
      # ex_h_x, ex_h_y = extended(h_x, h_y, 500)
      # ex_h_line = line((int(h_x[0]), int(h_y[0])), (int(h_x[1]), int(h_y[1])))
      r = intersection2(v_border, hline)
      r_lower = intersection2(v_border_lower, hline)

      vh_intersections_x.append(r[0])
      vh_intersections_y.append(r[1])
      vh_intersections_x.append(r_lower[0])
      vh_intersections_y.append(r_lower[1])

      sorted_vh_inter_x = sorted(vh_intersections_x)
      # print('vh_intersections_x :', vh_intersections_x)
      # print('sorted_vh_inter_x :', sorted_vh_inter_x)

      center_h_x = (h_line[0] + h_line[2]) / 2
      # print('center_h_x :', center_h_x)

      #       hline 상의 교차점을 찾아내서 범위 내에서 연결한다.      #
      for i in range(1, len(sorted_vh_inter_x)):

        if sorted_vh_inter_x[i - 1] <= center_h_x <= sorted_vh_inter_x[i]:
          lx, ly = sorted_vh_inter_x[i - 1], vh_intersections_y[vh_intersections_x.index(sorted_vh_inter_x[i - 1])]
          rx, ry = sorted_vh_inter_x[i], vh_intersections_y[vh_intersections_x.index(sorted_vh_inter_x[i])]
          # print('lx, ly, rx, ry :', lx, ly, rx, ry)
          #       이곳의 lx, ly, rx, ry 는 close_p 다. 좌우로 나뉘어진 lx, ly가 아니다.     #

            #       정제된 교차점을 만드는 hline만 사용해야한다.        #
          for inters in all_closest_inters:
            for inter in inters:
              if lx in inter or rx in inter:
                h_intersections.append((lx, ly, rx, ry))
                # cv2.line(black_plane, (int(lx), int(ly)), (int(rx), int(ry)), (0, 0, 255), 1, cv2.LINE_AA)

  #       vline 기준으로 구획해야한다. -> 가장 왼쪽 / 오른쪽 vline + 구역 먼저 찾아 작업하기     #
  #       vline 별로 구획 나누기      #
  h_intersections = list(set(h_intersections))
  print('np.array(all_closest_inters).shape :', np.array(all_closest_inters).shape)
  all_closest_inters = np.array(all_closest_inters)
  print('len(all_closest_inters) :', len(all_closest_inters))

  center_xs = list()
  for inters in all_closest_inters:
    #     all_closest_inter가 정렬되지 않았다면 inters type = list()      #
    sum_x = 0
    for inter in inters:
      sum_x += inter[0] + inter[2]
    center_xs.append(sum_x / len(inters) * 2)
  sorted_center_xs = sorted(center_xs)

  sorted_index = list()
  for center_x in sorted_center_xs:
    # print('center_x :', center_x)
    sorted_index.append(center_xs.index(center_x))
  # print('sorted_index :', sorted_index)
  #       sorted_center_xs 순서로 all_closest_inter를 정렬한다.     #
  #       index list를 추출해서 for 문으로 all_closest_inter의 inters를 추출해
  #       sorted_all ... 에 append 시킨다.      #
  sorted_all_closest_inters = list()
  sorted_all_centroid_inters = list()
  for s_index in sorted_index:
    sorted_all_closest_inters.append(all_closest_inters[s_index])
    sorted_all_centroid_inters.append(all_centroid_inters[s_index])
  all_closest_inters = sorted_all_closest_inters
  all_centroid_inters = sorted_all_centroid_inters

  for inters_i, inters in enumerate(all_closest_inters):

    # print('inters :', inters)
    inter_x = np.array(inters)[:, [0, 2]]
    inter_y = np.array(inters)[:, [1, 3]]

    #      vline 별로 양옆으로 작업을 하면 len(vline) = 1의 작업을 반복할 필요가 없어진다.    #
    iter = False
    while True:

      four_inters = list()
      find_pair = True
      centroid_inters = all_centroid_inters[inters_i]

      if not iter:
        #       vline 우측 session      #
        #   vline 우편 좌표   #
        final_xs = inter_x[:, [1]].reshape(-1, )
        final_ys = inter_y[:, [1]].reshape(-1, )
        # print(final_xs)
        # four_inters.append([final_xs[0], final_ys[0]])
        four_inters.append(centroid_inters[0])
        # print(four_inters)

        #       intersection이 1개이고 오른쪽 끝 vline이면, border intersection을 추가해준다.   #
        #       오른쪽 끝이 아니고 현재 vline의 교차점과 다음 vline의 교차점이같은 위치에 없으면 평행 copy,    #
        #       현재 교차점이 없고 다음 교차점도 없으면 border intersection     #
        #       둘다 있으면 추가      #
        #        border inter parallel copy     #

        if len(inters[0]) == 6:
          print('len(inter[0]) == 6')
          #       오른쪽 끝 vline 이면        #
          if inters_i == len(all_closest_inters) - 1:
            print('inters_i == len(all_closest_inters) - 1')
            four_inters.append(inters[0][-2:])
            four_inters.append([org_np.shape[1], inters[0][-1]])
          else:
            find_pair = False
            next_inters = np.array(all_closest_inters[inters_i + 1])
            next_centroid_inters = np.array(all_centroid_inters[inters_i + 1])
            print(next_centroid_inters)
            four_inters.append(next_centroid_inters[0])
            if len(next_inters) == 2:
              print('len(next_inters) == 2')
              #     없는 부분 평행 copy   #
              four_inters.append(next_centroid_inters[1])

              #     1. 없는 부분이 어디인지 확인해야한다.      #
              #     2. copy할 부분의 인덱스 번호를 확인해야한다.      #
              if inters[0][-1] == 0: # -> 천장 부분 교차점이 없다.
                print(type(next_inters))
                if np.mean(next_inters[[0], [1, 3]]) < np.mean(next_inters[[1], [1, 3]]):
                  y_in = np.mean(next_inters[[0], [1, 3]])
                else:
                  y_in = np.mean(next_inters[[1], [1, 3]])
              else:   # -> 바닥 교차점이 없다.
                if np.mean(next_inters[[0], [1, 3]]) < np.mean(next_inters[[1], [1, 3]]):
                  y_in = np.mean(next_inters[[1], [1, 3]])
                else:
                  y_in = np.mean(next_inters[[0], [1, 3]])

              x = (inters[0][0], inters[0][2])
              y = (inters[0][1], inters[0][3])
              p = np.polyfit(y, x , deg=1)
              x_out = np.poly1d(p)(y_in)
              four_inters.append([x_out, y_in])

            else:
              if inters[0][-1] == next_inters[0][-1]:
                print('inters[0][-1] == next_inters[0][-1]')
                #     없는 부분 border intersection     #
                four_inters.append(inters[0][-2:])
                four_inters.append(next_inters[0][-2:])

              else: # 다른 위치
                #     없는 부분 평행 copy     #
                x = (inters[0][0], inters[0][2])
                y = (inters[0][1], inters[0][3])
                p = np.polyfit(y, x , deg=1)
                y_in = np.mean(next_inters[0, [1, 3]])
                x_out = np.poly1d(p)(y_in)
                four_inters.append([x_out, y_in])

                x = (next_inters[0][0], next_inters[0][2])
                y = (next_inters[0][1], next_inters[0][3])
                p = np.polyfit(y, x , deg=1)
                y_in = np.mean(inters[0, [1, 3]])
                x_out = np.poly1d(p)(y_in)
                four_inters.append([x_out, y_in])

        #      len vline inters = 2     #
        else:
          # four_inters.append([final_xs[1], final_ys[1]])
          four_inters.append(centroid_inters[1])

          #       오른쪽 끝 vline 이면      #
          if inters_i == len(all_closest_inters) - 1:
            print('inters_i == len(all_closest_inters) - 1')
          else:
            find_pair = False
            inters = np.array(inters)
            next_inters = np.array(all_closest_inters[inters_i + 1])
            next_centroid_inters = np.array(all_centroid_inters[inters_i + 1])
            four_inters.append(next_centroid_inters[0])
            if len(next_inters) == 2:
              print('len(next_inters) == 2')
              four_inters.append(next_centroid_inters[1])
            else:
              #     1. 없는 부분이 어디인지 확인해야한다.      #
              #     2. copy할 부분의 인덱스 번호를 확인해야한다.      #
              if next_inters[0][-1] == 0: # -> 천장 부분 교차점이 없다.
                print('next_inters[0] :', next_inters[0])
                print('type(next_inters) :', type(next_inters))
                inters = np.array(inters)
                if np.mean(inters[[0], [1, 3]]) < np.mean(inters[[1], [1, 3]]):
                  y_in = np.mean(inters[[0], [1, 3]])
                else:
                  y_in = np.mean(inters[[1], [1, 3]])
              else:   # -> 바닥 교차점이 없다.
                if np.mean(inters[0, [1, 3]]) < np.mean(inters[[[1]], [1, 3]]):
                  y_in = np.mean(inters[[1], [1, 3]])
                else:
                  y_in = np.mean(inters[[0], [1, 3]])

              x = (next_inters[0][0], next_inters[0][2])
              y = (next_inters[0][1], next_inters[0][3])
              p = np.polyfit(y, x , deg=1)
              x_out = np.poly1d(p)(y_in)
              four_inters.append([x_out, y_in])

      #     i = 0 에 한해서만 왼쪽으로도 refering 진행, 나머지는 오른쪽으로만     #
      else:
        #     한 vline에 대해 분포하는 모든 intersection에 대한 pair는 찾아주어야 한다.   #
        #   vline 좌편 좌표   #
        final_xs = inter_x[:, [0]].reshape(-1, )
        final_ys = inter_y[:, [0]].reshape(-1, )
        # print(final_xs)
        four_inters.append(centroid_inters[0])
        # print(four_inters)

        #       intersection이 1개이면, border intersection을 추가해준다.   #
        #        border inter parallel copy     #
        if len(inters[0]) == 6:
          print('inters[0][-2:] :', inters[0][-2:])
          four_inters.append(inters[0][-2:])
          four_inters.append([0, inters[0][-1]])
        else:
          four_inters.append(centroid_inters[1])
        # print(four_inters)

      print('four_inters :', four_inters)
      print('h_intersections :', h_intersections)
      #             Find intersection pairs by h_intersections            #
      if find_pair:
        #       맨 좌우 vline일 경우에만 해당하는데 => pair를 찾아주는게 아니라 v_border와의      #
        #       교차점을 찾아주는게 맞는 방향이다.      #
        for h_inter in h_intersections:
          for final_x, final_y in zip(final_xs, final_ys):
            if final_x in h_inter and final_y in h_inter:
              hline = line(h_inter[:2], h_inter[2:])
              if iter:
                print('intersection2(v_border, hline) :', intersection2(v_border, hline))
                four_inters.append(list(intersection2(v_border, hline)))
              else:
                four_inters.append(list(intersection2(v_border_lower, hline)))

      #              Four Intersection 완성               #
      print('four_inters :', four_inters)
      if len(four_inters) != 4:
        print('Error in four_inter == 4')
        raise Exception
        break

      #                                   Refering                                 #
      #     Find Top left / right & Bottom left / right       #
      #     ->  tl, tr, bl, br     #
      four_inters = np.array(four_inters)
      [tl, tr, br, bl] = order_points(four_inters)
      # print(four_inters.shape)
      top_length = math.hypot(tl[0] - tr[0], tl[1] - tr[1])
      bottom_length = math.hypot(bl[0] - br[0], bl[1] - br[1])
      max_hlength = max(top_length, bottom_length)

      #     1. Compare left / right height     #
      l_height = bl[1] - tl[1]
      r_height = br[1] - tr[1]

      #     2. Extend Shorter Height      #
      if l_height <= r_height:
        shorter_points = [bl, tl]
        longer_points = [br, tr]
      else:
        shorter_points = [br, tr]
        longer_points = [bl, tl]

      ex_shorter_points = np.zeros_like(shorter_points)
      ex_longer_points = np.zeros_like(longer_points)

      print('shorter_points :', shorter_points)
      x = (shorter_points[0][0], shorter_points[1][0])
      y = (shorter_points[0][1], shorter_points[1][1])
      print('shorter x, y :', x, y)

      y_ext = np.array([org_np.shape[0], 0])
      p = np.polyfit(y, x , deg=1)
      x_ext = np.poly1d(p)(y_ext)

      ex_shorter_points[0] = [x_ext[0], y_ext[0]]
      ex_shorter_points[1] = [x_ext[1], y_ext[1]]

      #     Find intersection between (longer + shorter points)'s parallel line and longer line   #
      long_short_lower = Line(longer_points[0], shorter_points[0])
      long_short = Line(longer_points[1], shorter_points[1])
      p_line_lower = long_short_lower.parallel_line(ex_shorter_points[0])
      p_line = long_short.parallel_line(ex_shorter_points[1])
      longer_line = Line(longer_points[0], longer_points[1])

      # print('ex_shorter_points[1] in p_line ? :', ex_shorter_points[1] in p_line)
      # print('p_line_lower.intersection(longer_line) :', p_line_lower.intersection(longer_line)[0][0])
      ex_longer_points[0] = [p_line_lower.intersection(longer_line)[0][0], p_line_lower.intersection(longer_line)[0][1]]
      ex_longer_points[1] = [p_line.intersection(longer_line)[0][0], p_line.intersection(longer_line)[0][1]]

      if l_height <= r_height:
        ex_bl, ex_tl = ex_shorter_points
        ex_br, ex_tr = ex_longer_points
      else:
        ex_br, ex_tr = ex_shorter_points
        ex_bl, ex_tl = ex_longer_points

      for ex_p in [ex_bl, ex_br, ex_tl, ex_tr]:
        print(ex_p)
        # cv2.circle(black_plane2, (int(ex_p[0]), int(ex_p[1])), 10, (255, 0, 255), -1)

      min_x, max_x = min(ex_bl[0], ex_tl[0]), max(ex_br[0], ex_tr[0])
      min_y, max_y = min(ex_tl[1], ex_tr[1]), max(ex_bl[1], ex_br[1])
      src_x = max((max_x - min_x), max_hlength) * math.sqrt(2)
      print('src_x :',  src_x)

      refered = np.asarray(refer)[:int(max_y - min_y), :int(src_x)]

      # plt.imshow(refered)
      # plt.title('refered')
      # plt.show()

      #   tl, tr, br, bl    #
      #     refer를 위해 src_x => 0 으로 맞춰준다.    #
      src = np.array([
          [0, 0],
          [src_x, 0],
          [src_x, max_y - min_y],
          [0, max_y - min_y]], dtype = "float32")
      dst = np.array([
                      [ex_tl[0] - min_x, ex_tl[1] - min_y],
                      [ex_tr[0] - min_x, ex_tr[1] - min_y],
                      [ex_br[0] - min_x, ex_br[1] - min_y],
                      [ex_bl[0] - min_x, ex_bl[1] - min_y]], dtype = "float32")

      print('src :', src)
      print('dst :', dst)

      # compute the perspective transform matrix and then apply it
      matrix = cv2.getPerspectiveTransform(src, dst)
      refered = cv2.warpPerspective(refered, matrix, (refered.shape[1], refered.shape[0]))

      # print('refered.min() :', refered.min())
      # print(refered[[0], [0]])
      # print('refered.shape :', refered.shape)
      # plt.imshow(refered)
      # plt.title('warfed')
      # plt.xlim(min_x, max_x)
      # plt.xlim(0, max_x - min_x)
      # plt.ylim(max_y, min_y)
      # plt.xlim(-1000, 1000)
      # plt.ylim(1000, -1000)
      # plt.show()

      refer_crop = refered[int(abs(min_y)): int(abs(min_y)) + org_np.shape[0], :]
      # print('refer_crop.shape :', refer_crop.shape)

      # plt.imshow(refer_crop)
      # plt.title('cropped refer')
      # plt.show()

      #     Crop Section From Cropped Refer     #
      #         src_x => 0으로 맞춰줬던 것을 되돌린다.      #
      print('int(min_x) :', int(min_x))
      print('int(min_y) :', int(min_y))

      for i in range(refer_crop.shape[1]):
        for j in range(refer_crop.shape[0]):
          if sum(refer_crop[j][i]) != 0:
            if i + int(min_x) < black_plane2.shape[1]:
              black_plane2[j][i + int(min_x)] = refer_crop[j][i]

      #     i != 0 인 경우 break     #
      if inters_i == 0 and not iter:
        iter = True
        print('iter :', iter)
      else:
        break

  if len(all_closest_inters) == 0:
    refered = np.asarray(refer)[:black_plane2.shape[1], :black_plane2.shape[0]]
    black_plane2 = refered

  else:
    #         Refer의 빈 부분은 original image로 채운다.       #
    for i in range(black_plane2.shape[1]):
      for j in range(black_plane2.shape[0]):
        if sum(black_plane2[j][i]) == 0:
          black_plane2[j][i] = org_np[j][i]

  print('elapsed time :', time.time() - start)

  # plt.figure(figsize=(10, 8))

  # plt.subplot(121)
  # plt.imshow(org_np2)
  # plt.xlim(-10, org_np.shape[0] + 10)

  # # plt.subplot(132)
  # # plt.imshow(black_plane)

  # plt.subplot(122)
  # plt.imshow(black_plane2)
  # plt.show()

  return black_plane2
```

    Output hidden; open in https://colab.research.google.com to view.

```

```


## Demo: Expected Free Energy minimization for controlling a Duffing oscillator

This project considers a [Duffing oscillator](hkps://en.wikipedia.org/wiki/Duffing_equation), a driven damped harmonic oscillator with a cubic nonlinearity in its spring stiffness component. The continuous-time dynamics of the system are:

$$\begin{align}
m \frac{d^2 x(t)}{dt^2} + c \frac{d x(t)}{dt} + a x(t) + b x^3(t) =&\ u(t) + w(t) \\
y(t) =&\ x(t) + e(t)
\end{align}$$

where
$$\begin{align}
m     =&\ \text{mass} \\
c     =&\ \text{damping} \\
a     =&\ \text{linear stiffness} \\
b     =&\ \text{nonlinear stiffness} \\
y(t)    =&\ \text{observation (displacement)} \\
x(t)    =&\ \text{state (displacement)} \\
u(t)    =&\ \text{force} \\
v(t)    =&\ \text{measurement noise} \\
w(t)    =&\ \text{process noise}
\end{align}$$

The process noise is a Wiener process, where the increment is Gaussian distributed $w(t) \sim \mathcal{N}(0, \zeta^{-1}dt)$ with $\zeta$ representing the precision of the process. The measurement noise is also a Wiener process, with $v(t) \sim \mathcal{N}(0, \xi^{-1}dt)$ and $\xi$ as precision parameter.

We cast this to the following discrete-time system:

$$\begin{align}
x_k =&\ \theta_1 x_{k-1} + \theta_2 x_{k-1}^3 + \theta_3 x_{k-2} + \eta u_k + w_k \\
y_k =&\ x_k + e_k
\end{align}$$

where the coefficients $\theta = (\theta_1, \dots, \theta_3)$ and $\eta$ are nonlinear combinations of the physical parameters (i.e. mass, damping and stiffness).

```julia
using Pkg
Pkg.activate(".")
Pkg.instantiate()
```

      Activating environment at `C:\Syndr\Wouter\Onderzoek\Demonstraties\tue\actinf-oscillator\Project.toml`
        Updating registry at `C:\Users\Wouter Kouw\.julia\registries\General`
       Installed Functors ─────── v0.2.2
       Installed NNlib ────────── v0.7.24
       Installed ChainRulesCore ─ v0.10.11
       Installed Zygote ───────── v0.6.15
       Installed ChainRules ───── v0.8.22
       Installed Plots ────────── v1.19.0
        Updating `C:\Syndr\Wouter\Onderzoek\Demonstraties\tue\actinf-oscillator\Project.toml`
      [336ed68f] + CSV v0.8.5
      [8f4d0f93] + Conda v1.5.2
      [a93c6f00] + DataFrames v1.2.0
      [31c24e10] + Distributions v0.24.18
      [ced4e74d] + DistributionsAD v0.6.28
      [587475ba] + Flux v0.12.1
      [9fc3f58a] + ForneyLab v0.11.2
      [f6369f11] + ForwardDiff v0.10.18
      [43dcc890] + GaussianDistributions v0.5.1
      [5fb14364] + OhMyREPL v0.5.10
      [90014a1f] + PDMats v0.11.1
      [91a5bcdd] + Plots v1.19.0
      [92933f4c] + ProgressMeter v1.7.1
      [438e738f] + PyCall v1.92.3
      [d330b81b] + PyPlot v2.9.0
      [295af30f] + Revise v3.1.17
      [276daf66] + SpecialFunctions v0.10.3
      [2913bbd2] + StatsBase v0.33.8
      [4c63d2b9] + StatsFuns v0.9.8
      [e88e6eb3] + Zygote v0.6.15
        Updating `C:\Syndr\Wouter\Onderzoek\Demonstraties\tue\actinf-oscillator\Manifest.toml`
      [621f4979] + AbstractFFTs v1.0.1
      [1520ce14] + AbstractTrees v0.3.4
      [79e6a3ab] + Adapt v3.3.1
      [ab4f0b2a] + BFloat16s v0.1.0
      [fa961155] + CEnum v0.4.1
      [336ed68f] + CSV v0.8.5
      [052768ef] + CUDA v2.6.3
      [082447d4] + ChainRules v0.8.22
      [d360d2e6] + ChainRulesCore v0.10.11
      [da1fd8a2] + CodeTracking v1.0.5
      [944b1d66] + CodecZlib v0.7.0
      [35d6a980] + ColorSchemes v3.13.0
      [3da002f7] + ColorTypes v0.11.0
      [5ae59095] + Colors v0.12.8
      [bbf7d656] + CommonSubexpressions v0.3.0
      [34da2185] + Compat v3.31.0
      [8f4d0f93] + Conda v1.5.2
      [d38c429a] + Contour v0.5.7
      [a8cc5b0e] + Crayons v4.0.4
      [9a962f9c] + DataAPI v1.7.0
      [a93c6f00] + DataFrames v1.2.0
      [864edb3b] + DataStructures v0.18.9
      [e2d170a0] + DataValueInterfaces v1.0.0
      [163ba53b] + DiffResults v1.0.3
      [b552c78f] + DiffRules v1.0.2
      [31c24e10] + Distributions v0.24.18
      [ced4e74d] + DistributionsAD v0.6.28
      [ffbed154] + DocStringExtensions v0.8.5
      [e30172f5] + Documenter v0.25.5
      [e2ba6199] + ExprTools v0.1.6
      [c87230d0] + FFMPEG v0.4.1
      [1a297f60] + FillArrays v0.11.9
      [53c48c17] + FixedPointNumbers v0.8.4
      [587475ba] + Flux v0.12.1
      [59287772] + Formatting v0.4.2
      [9fc3f58a] + ForneyLab v0.11.2
      [f6369f11] + ForwardDiff v0.10.18
      [d9f16b24] + Functors v0.2.2
      [0c68f7d7] + GPUArrays v6.4.1
      [61eb1bfa] + GPUCompiler v0.10.0
      [28b8d3ca] + GR v0.57.5
      [43dcc890] + GaussianDistributions v0.5.1
      [5c1252a2] + GeometryBasics v0.3.13
      [42e2da0e] + Grisu v1.0.2
      [cd3eb016] + HTTP v0.9.12
      [b5f81e59] + IOCapture v0.1.1
      [7869d1d1] + IRTools v0.4.3
      [83e8ac13] + IniFile v0.5.0
      [41ab1584] + InvertedIndices v1.0.0
      [c8e1da08] + IterTools v1.3.0
      [82899510] + IteratorInterfaceExtensions v1.0.0
      [1019f520] + JLFzf v0.1.3
      [692b3bcd] + JLLWrappers v1.3.0
      [682c06a0] + JSON v0.21.1
      [aa1ae85d] + JuliaInterpreter v0.8.18
      [e5e0dc1b] + Juno v0.8.4
      [929cbde3] + LLVM v3.9.0
      [b964fa9f] + LaTeXStrings v1.2.1
      [23fbe1c1] + Latexify v0.15.6
      [2ab3a3ac] + LogExpFunctions v0.2.5
      [6f1432cf] + LoweredCodeUtils v2.1.0
      [1914dd2f] + MacroTools v0.5.6
      [739be429] + MbedTLS v1.0.3
      [442fdcdd] + Measures v0.3.1
      [e89f7d12] + Media v0.5.0
      [c03570c3] + Memoize v0.4.4
      [e1d29d7a] + Missings v1.0.0
      [872c559c] + NNlib v0.7.24
      [77ba4419] + NaNMath v0.3.5
      [5fb14364] + OhMyREPL v0.5.10
      [bac558e1] + OrderedCollections v1.4.1
      [90014a1f] + PDMats v0.11.1
      [69de0a69] + Parsers v1.1.0
      [b98c9c47] + Pipe v1.3.0
      [ccf2f8ad] + PlotThemes v2.0.1
      [995b91a9] + PlotUtils v1.0.11
      [91a5bcdd] + Plots v1.19.0
      [2dfb63ee] + PooledArrays v1.2.1
      [21216c6a] + Preferences v1.2.2
      [08abe8d2] + PrettyTables v1.1.0
      [92933f4c] + ProgressMeter v1.7.1
      [438e738f] + PyCall v1.92.3
      [d330b81b] + PyPlot v2.9.0
      [1fd47b50] + QuadGK v2.4.1
      [3cdcf5f2] + RecipesBase v1.1.1
      [01d81517] + RecipesPipeline v0.3.4
      [189a3867] + Reexport v1.1.0
      [ae029012] + Requires v1.1.3
      [295af30f] + Revise v3.1.17
      [79098fc4] + Rmath v0.7.0
      [6c6a2e73] + Scratch v1.1.0
      [91c51154] + SentinelArrays v1.3.3
      [992d4aef] + Showoff v1.0.3
      [a2af1166] + SortingAlgorithms v1.0.1
      [276daf66] + SpecialFunctions v0.10.3
      [90137ffa] + StaticArrays v1.2.6
      [82ae8749] + StatsAPI v1.0.0
      [2913bbd2] + StatsBase v0.33.8
      [4c63d2b9] + StatsFuns v0.9.8
      [09ab397b] + StructArrays v0.6.0
      [3783bdb8] + TableTraits v1.0.1
      [bd369af6] + Tables v1.4.4
      [a759f4b9] + TimerOutputs v0.5.12
      [0796e94c] + Tokenize v0.5.18
      [3bb67fe8] + TranscodingStreams v0.9.5
      [5c2747f8] + URIs v1.3.0
      [81def892] + VersionParsing v1.2.0
      [a5390f91] + ZipFile v0.9.3
      [e88e6eb3] + Zygote v0.6.15
      [700de1a5] + ZygoteRules v0.2.1
      [6e34b625] + Bzip2_jll v1.0.6+5
      [83423d85] + Cairo_jll v1.16.0+6
      [5ae413db] + EarCut_jll v2.1.5+1
      [2e619515] + Expat_jll v2.2.10+0
      [b22a6f82] + FFMPEG_jll v4.3.1+4
      [a3f928ae] + Fontconfig_jll v2.13.1+14
      [d7e528f0] + FreeType2_jll v2.10.1+5
      [559328eb] + FriBidi_jll v1.0.10+0
      [0656b61e] + GLFW_jll v3.3.5+0
      [d2c73de3] + GR_jll v0.57.3+0
      [78b55507] + Gettext_jll v0.21.0+0
      [7746bdde] + Glib_jll v2.68.1+0
      [aacddb02] + JpegTurbo_jll v2.1.0+0
      [c1c5ebd0] + LAME_jll v3.100.1+0
      [dd4b983a] + LZO_jll v2.10.1+0
      [dd192d2f] + LibVPX_jll v1.10.0+0
      [e9f186c6] + Libffi_jll v3.2.2+0
      [d4300ac3] + Libgcrypt_jll v1.8.7+0
      [7e76a0d4] + Libglvnd_jll v1.3.0+3
      [7add5ba3] + Libgpg_error_jll v1.42.0+0
      [94ce4f54] + Libiconv_jll v1.16.1+1
      [4b2f31a3] + Libmount_jll v2.35.0+0
      [89763e89] + Libtiff_jll v4.3.0+0
      [38a345b3] + Libuuid_jll v2.36.0+0
      [e7412a2a] + Ogg_jll v1.3.5+0
      [458c3c95] + OpenSSL_jll v1.1.10+0
      [efe28fd5] + OpenSpecFun_jll v0.5.5+0
      [91d4177d] + Opus_jll v1.3.2+0
      [2f80f16e] + PCRE_jll v8.44.0+0
      [30392449] + Pixman_jll v0.40.1+0
      [ea2cea3b] + Qt5Base_jll v5.15.3+0
      [f50d1b31] + Rmath_jll v0.3.0+0
      [a2964d1f] + Wayland_jll v1.19.0+0
      [2381bf8a] + Wayland_protocols_jll v1.18.0+4
      [02c8fc9c] + XML2_jll v2.9.12+0
      [aed1982a] + XSLT_jll v1.1.34+0
      [4f6342f7] + Xorg_libX11_jll v1.6.9+4
      [0c0b7dd1] + Xorg_libXau_jll v1.0.9+4
      [935fb764] + Xorg_libXcursor_jll v1.2.0+4
      [a3789734] + Xorg_libXdmcp_jll v1.1.3+4
      [1082639a] + Xorg_libXext_jll v1.3.4+4
      [d091e8ba] + Xorg_libXfixes_jll v5.0.3+4
      [a51aa0fd] + Xorg_libXi_jll v1.7.10+4
      [d1454406] + Xorg_libXinerama_jll v1.1.4+4
      [ec84b674] + Xorg_libXrandr_jll v1.5.2+4
      [ea2f1a96] + Xorg_libXrender_jll v0.9.10+4
      [14d82f49] + Xorg_libpthread_stubs_jll v0.1.0+3
      [c7cfdc94] + Xorg_libxcb_jll v1.13.0+3
      [cc61e674] + Xorg_libxkbfile_jll v1.1.0+4
      [12413925] + Xorg_xcb_util_image_jll v0.4.0+1
      [2def613f] + Xorg_xcb_util_jll v0.4.0+1
      [975044d2] + Xorg_xcb_util_keysyms_jll v0.4.0+1
      [0d47668e] + Xorg_xcb_util_renderutil_jll v0.3.9+1
      [c22f9ab0] + Xorg_xcb_util_wm_jll v0.4.1+1
      [35661453] + Xorg_xkbcomp_jll v1.4.2+4
      [33bec58e] + Xorg_xkeyboard_config_jll v2.27.0+4
      [c5fb5394] + Xorg_xtrans_jll v1.4.0+3
      [3161d3a3] + Zstd_jll v1.5.0+0
      [214eeab7] + fzf_jll v0.24.4+0
      [0ac62f75] + libass_jll v0.14.0+4
      [f638f0a6] + libfdk_aac_jll v0.1.6+4
      [b53b4c65] + libpng_jll v1.6.38+0
      [f27f6e37] + libvorbis_jll v1.3.7+0
      [1270edf5] + x264_jll v2020.7.14+2
      [dfaa095f] + x265_jll v3.0.0+3
      [d8fb68d0] + xkbcommon_jll v0.9.1+5
      [0dad84c5] + ArgTools
      [56f22d72] + Artifacts
      [2a0f44e3] + Base64
      [ade2ca70] + Dates
      [8bb1440f] + DelimitedFiles
      [8ba89e20] + Distributed
      [f43a241f] + Downloads
      [7b1f6079] + FileWatching
      [9fa8497b] + Future
      [b77e0a4c] + InteractiveUtils
      [4af54fe1] + LazyArtifacts
      [b27032c2] + LibCURL
      [76f85450] + LibGit2
      [8f399da3] + Libdl
      [37e2e46d] + LinearAlgebra
      [56ddb016] + Logging
      [d6f4376e] + Markdown
      [a63ad114] + Mmap
      [ca575930] + NetworkOptions
      [44cfe95a] + Pkg
      [de0858da] + Printf
      [9abbd945] + Profile
      [3fa0cd96] + REPL
      [9a3f8284] + Random
      [ea8e919c] + SHA
      [9e88b42a] + Serialization
      [1a1011a3] + SharedArrays
      [6462fe0b] + Sockets
      [2f01184e] + SparseArrays
      [10745b16] + Statistics
      [4607b0f0] + SuiteSparse
      [fa267f1f] + TOML
      [a4e569a6] + Tar
      [8dfed614] + Test
      [cf7118a7] + UUIDs
      [4ec0a83e] + Unicode
      [e66e0078] + CompilerSupportLibraries_jll
      [deac9b47] + LibCURL_jll
      [29816b5a] + LibSSH2_jll
      [c8ffd9c3] + MbedTLS_jll
      [14a3606d] + MozillaCACerts_jll
      [83775a58] + Zlib_jll
      [8e850ede] + nghttp2_jll
      [3f19e933] + p7zip_jll

```julia
using Plots
pyplot()
viz = true;
```

    ┌ Info: Precompiling Plots [91a5bcdd-55d7-5caf-9e0b-520d859cae80]
    └ @ Base loading.jl:1317

### Simulation environment

```julia
# Dynamical parameters
mass = 1.2
damping = 0.4
stiffness_lin = 0.9
stiffness_nnl = 0.8

# Process noise precision
ζ_true = 1e4

# Measurement noise precision
ξ_true = 1e3

# Pack parameters
sys_params = (mass, damping, stiffness_lin, stiffness_nnl, ζ_true, ξ_true);

# State transition coefficients
θ1 = (2*mass + damping - stiffness_lin)/(mass + damping)
θ2 = -stiffness_nnl/(mass + damping)
θ3 = -mass/(mass + damping)
θ_true = Array{Float64}([θ1, θ2, θ3])

# Control coefficient
η_true = 1/(mass + damping)

# Pack substituted variables
subs_params = (θ_true, η_true, ζ_true, ξ_true);
```

```julia
function sim_sys(input, state, sys_params)
   "Simulate dynamic system"

    # Unpack state
    x_kmin1, x_kmin2 = state

    # Unpack system parameters
    m,c,a,b,ζ,ξ = sys_params

    # Draw noises
    w_k = sqrt(inv(ζ))*randn(1,)[1]
    v_k = sqrt(inv(ξ))*randn(1,)[1]

    # State transition
    x_k = (2*m + c - a)/(m+c)*x_kmin1 + (-b)/(m+c)*x_kmin1^3 + -m/(m+c)*x_kmin2 + 1/(m+c)*input + w_k

    # Generate observation
    y_k = x_k + v_k

    return y_k, x_k
end
```

    sim_sys (generic function with 1 method)

```julia
# Test simulation environment

# Time
T = 200

# Input signal
input = sin.((1:T)./ (6*π))

# Preallocate arrays
states = zeros(T,)
output = zeros(T,)

for k = 4:T
    output[k], states[k] = sim_sys(input[k], [states[k-1], states[k-2]], sys_params)
end

p100 = plot(1:T, states, linewidth=2, color="blue", label="states", size=(700,400))
plot!(1:T, input, linewidth=2, color="red", label="input", size=(700,400))
scatter!(1:T, output, color="black", label="output")
```

```julia
savefig(p100, "figures/example-input-output_seq1.png")
# savefig(p100, "figures/example-input-output_seq1.pdf")
```

## Model

We cast the above system into matrix form:

$$ \underbrace{\begin{bmatrix} x_{k} \\ x_{k-1} \end{bmatrix}}_{z_k} = \underbrace{\begin{bmatrix} 0 & 0 \\ 1 & 0 \end{bmatrix}}_{S} \underbrace{\begin{bmatrix} x_{k-1} \\ x_{k-2} \end{bmatrix}}_{z_{k-1}} + \underbrace{\begin{bmatrix} 1 \\ 0 \end{bmatrix}}_{s} g(\theta, z_{k-1}) + \begin{bmatrix} 1 \\ 0 \end{bmatrix} \eta u_t + \begin{bmatrix} 1 \\ 0 \end{bmatrix} \tilde{w}_t \, ,$$

with $g(\theta, z_{k-1}) = \theta_1 x_{k-1} + \theta_2 x_{k-2}$ and $\tilde{w}_k \sim \mathcal{N}(0, \zeta^{-1})$. The system is now a nonlinear autoregressive process:

$$z_k = f(\theta, z_{k-1}, \eta, u_k) + \tilde{w}_k$$

where $f(\theta, z_{k-1}, \eta, u_k) = S z_{k-1} + s g(\theta, z_{k-1}) + s \eta u_k$.

### Likelihood

Integrating out $\tilde{w}_t$ and $v_t$ produces a Gaussian state transition node:

$$\begin{align}
z_k \sim&\ \mathcal{N}(f(\theta, z_{k-1}, \eta, u_k), V(\zeta)) \\
y_k \sim&\ \mathcal{N}(s^{\top} z_k, \xi^{-1}) \, ,
\end{align}$$

where $V(\zeta) = \begin{bmatrix} \zeta^{-1} & 0 \\ 0 & \epsilon \end{bmatrix}$ and $W(\zeta) = V(\zeta)^{-1} = \begin{bmatrix} \zeta & 0 \\ 0 & \epsilon^{-1} \end{bmatrix}$.

### Approximating the nonlinearity

The nonlinearity is approximated using a first-order Taylor expansion;

$$ g(\theta, x) = g(m_{\theta}, m_x) + J_{x}(m_{\theta}, m_x)^{\top}(x - m_x) + J_{\theta}(m_{\theta}, m_x)^{\top}(\theta - m_{\theta}) \, ,$$

where $J_x$ denotes the partial derivative of $g$ with respect to $x$ and $J_{\theta}$ w.r.t. $\theta$. Note that our current $g$ is linear in $\theta$ and one could argue that the approximation is unnecessary. However, this form is more general and the first-order Taylor is exact anyway.

### Priors

We choose the following priors:

$$\begin{align}
p(\theta) = \text{Normal}(m^{0}_{\theta}, V^{0}_{\theta}) \, , \quad
p(\eta) = \text{Normal}(m^{0}_{\eta}, v^{0}_{\eta}) \, , \quad
p(\zeta)= \text{Gamma}(a^{0}_\zeta, b^{0}_\zeta) \, , \quad
p(\xi)= \text{Gamma}(a^{0}_\xi, b^{0}_\xi) \, .
\end{align}$$

### Recognition model

The recognition model will follow the generative model:

$$\begin{align}
q(\theta) = \text{Normal}(m_{\theta}, V_{\theta}) \ , \quad
q(\eta) = \text{Normal}(m_{\eta}, v_{\eta}) \ , \quad
q(\zeta)= \text{Gamma}(a_\zeta, b_\zeta) \, , \quad
q(\xi)= \text{Gamma}(a_\xi, b_\xi) \, .
\end{align}$$

### Variational free energy

#TODO

## Implementation

We implemented the model in [ForneyLab.jl](hkps://github.com/biaslab/ForneyLab.jl) with a custom node called Nonlinear Latent Autoregressive eXogenous input (NLARX) model.

```julia
using LinearAlgebra
using ProgressMeter
using Zygote
using Flux: update!, Descent, ADAM

using ForneyLab
using NLARX

include("util.jl");
```

    ┌ Info: Precompiling Zygote [e88e6eb3-aa80-5325-afca-941959d7151f]
    └ @ Base loading.jl:1317
    ┌ Info: Precompiling Flux [587475ba-b771-5e3f-ad9e-33799f191a9c]
    └ @ Base loading.jl:1317
    WARNING: both ExprTools and LLVM export "parameters"; uses of it in module GPUCompiler must be qualified
    ┌ Warning: Error requiring `CUDA` from `Zygote`
    │   exception = (ArgumentError("Package Zygote does not have CUDA in its dependencies:\n- If you have Zygote checked out for development and have\n  added CUDA as a dependency but haven't updated your primary\n  environment's manifest file, try `Pkg.resolve()`.\n- Otherwise you may need to report an issue with Zygote"), Union{Ptr{Nothing}, Base.InterpreterIP}[Ptr{Nothing} @0x00000000610a3edb, Ptr{Nothing} @0x000000000277da06, Ptr{Nothing} @0x000000000277f72c, Ptr{Nothing} @0x00000000027617b2, Ptr{Nothing} @0x00000000027621c5, Base.InterpreterIP in top-level CodeInfo for Zygote at statement 8, Ptr{Nothing} @0x000000000277e7c7, Ptr{Nothing} @0x000000000278024f, Ptr{Nothing} @0x00000000641d808f, Ptr{Nothing} @0x00000000641d80b3, Ptr{Nothing} @0x000000000276103c, Ptr{Nothing} @0x0000000002760c34, Ptr{Nothing} @0x0000000002761588, Ptr{Nothing} @0x0000000002761b75, Ptr{Nothing} @0x0000000002762060, Base.InterpreterIP in MethodInstance for err(::Any, ::Module, ::String) at statement 2, Ptr{Nothing} @0x00000000641d7fe2, Ptr{Nothing} @0x00000000641d8003, Ptr{Nothing} @0x000000000276103c, Ptr{Nothing} @0x0000000002760c34, Ptr{Nothing} @0x00000000027619da, Ptr{Nothing} @0x0000000002761b75, Ptr{Nothing} @0x0000000002762060, Base.InterpreterIP in MethodInstance for withpath(::Any, ::String) at statement 10, Ptr{Nothing} @0x00000000641d7f46, Ptr{Nothing} @0x00000000641d7f73, Ptr{Nothing} @0x00000000027554e6, Ptr{Nothing} @0x0000000064647d11, Ptr{Nothing} @0x000000000276103c, Ptr{Nothing} @0x0000000002760c34, Ptr{Nothing} @0x0000000002761588, Ptr{Nothing} @0x0000000002762060, Base.InterpreterIP in MethodInstance for loadpkg(::Base.PkgId) at statement 6, Ptr{Nothing} @0x00000000027554e6, Ptr{Nothing} @0x000000006109fe88, Ptr{Nothing} @0x00000000610a064e, Ptr{Nothing} @0x00000000610a1ac3, Ptr{Nothing} @0x00000000610a3073, Ptr{Nothing} @0x00000000610a3d53, Ptr{Nothing} @0x000000000277da06, Ptr{Nothing} @0x000000000277dbe4, Ptr{Nothing} @0x000000000277f185, Ptr{Nothing} @0x000000000277f301, Ptr{Nothing} @0x000000000278024f, Ptr{Nothing} @0x00000000610e4ec5, Ptr{Nothing} @0x00000000610e529f, Ptr{Nothing} @0x000000006465127d, Ptr{Nothing} @0x00000000027554e6, Ptr{Nothing} @0x00000000610d5ca2, Ptr{Nothing} @0x00000000610d5ed6, Ptr{Nothing} @0x00000000610d5ef3, Ptr{Nothing} @0x0000000002766284])
    └ @ Requires C:\Users\Wouter Kouw\.julia\packages\Requires\7Ncym\src\require.jl:49
    ┌ Info: Precompiling ForneyLab [9fc3f58a-c2cc-5bff-9419-6a294fefdca9]
    └ @ Base loading.jl:1317
    ┌ Info: Precompiling NLARX [63ebac55-cc67-47e6-977a-6a1f35aa9955]
    └ @ Base loading.jl:1317

```julia
graph1 = FactorGraph()

# Static parameters
@RV θ ~ GaussianMeanPrecision(placeholder(:m_θ, dims=(3,)), placeholder(:w_θ, dims=(3,3)))
@RV η ~ GaussianMeanPrecision(placeholder(:m_η), placeholder(:w_η))
@RV ζ ~ Gamma(placeholder(:a_ζ), placeholder(:b_ζ))
@RV ξ ~ Gamma(placeholder(:a_ξ), placeholder(:b_ξ))

# Nonlinearity
g(θ, x) = θ[1]*x[1] + θ[2]*x[1]^3 + θ[3]*x[2]

# State prior
@RV z_kmin1 ~ GaussianMeanPrecision(placeholder(:m_z, dims=(2,)), placeholder(:w_z, dims=(2, 2)), id=:z_kmin1)

# Autoregressive node
@RV z_k ~ NLatentAutoregressiveX(θ, z_kmin1, η, placeholder(:u_k), ζ, g=g, id=:z_k)

# Specify likelihood
@RV y_k ~ GaussianMeanPrecision(dot([1. , 0.], z_k), ξ, id=:y_k)

# Placeholder for observation
placeholder(y_k, :y_k);

# Draw time-slice subgraph
# ForneyLab.draw(graph1)
```

```julia
# Specify recognition model
q1 = PosteriorFactorization(z_k, z_kmin1, θ, η, ζ, ξ, ids=[:z_k, :z_kmin1, :θ, :η, :ζ, :ξ])
algo1 = messagePassingAlgorithm([z_k, z_kmin1, θ, η, ζ, ξ], q1, free_energy=true)

# Compile inference algorithm
source_code1 = algorithmSourceCode(algo1, free_energy=true)
eval(Meta.parse(source_code1));
# println(source_code1)
```

### Infer parameters on training data

```julia
# Goal state (mean and std dev)
goal_state = (-.35, 0.001)

# Time horizon
T_trn = 300

# Planning horizon
plan_horizon = 2

# Number of gradient steps in EFE min
num_steps = 100

# Number of variational updates in VFE min
num_iterations = 10

# Initialize input-output arrays
inputs = zeros(T_trn,)
states = zeros(T_trn,)
outputs = zeros(T_trn,)

# Initialize marginal and data dicts
data = Dict()
marginals = Dict()

# Initialize free energy tracking array
free_energy_trn = zeros(T_trn, num_iterations)

# Initialize arrays of parameterizations
params_z = (zeros(2,T_trn+1), repeat(eye(2), outer=(1,1,T_trn+1)))
params_θ = (zeros(3,T_trn+1), repeat(eye(3), outer=(1,1,T_trn+1)))
params_η = (ones(1,T_trn+1), ones(1,T_trn+1))
params_ζ = (1e2*ones(1,T_trn+1), 1e0*ones(1,T_trn+1))
params_ξ = (1e4*ones(1,T_trn+1), 1e0*ones(1,T_trn+1))

# Perform inference at each time-step
@showprogress for k = 1:T_trn

    "Control"

    # Pack current model params
    model_params = (params_θ[1][:,k], params_η[1][1,k], params_ζ[1][k]/params_ζ[2][k], params_ξ[1][k]/params_ξ[2][k])

    # Previous state
    state_kmin1 = [params_z[1][:,k], params_z[2][:,:,k]]

    # Minimize EFE to select input signal
    policy, G = minEFE(state_kmin1, goal_state, model_params, g, num_iters=num_steps, plan_horizon=plan_horizon)

    # Only execute first action in policy
    inputs[k] = policy[1]

    # Execute input and observe output
    outputs[k], states[k] = sim_sys(inputs[k], params_z[1][:,k], sys_params)

#     "Test identification in case of designed control"
#     inputs[k] = sin(k ./ (6*π))
#     outputs[k], states[k] = sim_sys(inputs[k], params_z[1][:,k], sys_params)

    "Identification"

    # Initialize marginals
    marginals[:z_kmin1] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=params_z[1][:,k], w=params_z[2][:,:,k])
    marginals[:z_k] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=params_z[1][:,k], w=params_z[2][:,:,k])
    marginals[:θ] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=params_θ[1][:,k], w=params_θ[2][:,:,k])
    marginals[:η] = ProbabilityDistribution(Univariate, GaussianMeanPrecision, m=params_η[1][1,k], w=params_η[2][1,k])
    marginals[:ζ] = ProbabilityDistribution(Univariate, Gamma, a=params_ζ[1][1,k], b=params_ζ[2][1,k])
    marginals[:ξ] = ProbabilityDistribution(Univariate, Gamma, a=params_ξ[1][1,k], b=params_ξ[2][1,k])

    data = Dict(:y_k => outputs[k],
                :u_k => inputs[k],
                :m_z => params_z[1][:,k],
                :w_z => params_z[2][:,:,k],
                :m_θ => params_θ[1][:,k],
                :w_θ => params_θ[2][:,:,k],
                :m_η => params_η[1][1,k],
                :w_η => params_η[2][1,k],
                :a_ζ => params_ζ[1][1,k],
                :b_ζ => params_ζ[2][1,k],
                :a_ξ => params_ξ[1][1,k],
                :b_ξ => params_ξ[2][1,k])

    # Iterate variational parameter updates
    for i = 1:num_iterations

        # Update parameters
        stepη!(data, marginals)
        stepθ!(data, marginals)

        # Update states
        stepz_k!(data, marginals)
        stepz_kmin1!(data, marginals)

        # Update noise
        stepζ!(data, marginals)
        stepξ!(data, marginals)

        # Compute free energy
        free_energy_trn[k, i] = freeEnergy(data, marginals)

    end

    # Store current parameterizations of marginals
    params_z[1][:,k+1] = ForneyLab.unsafeMean(marginals[:z_k])
    params_z[2][:,:,k+1] = marginals[:z_k].params[:w]
    params_θ[1][:,k+1] = ForneyLab.unsafeMean(marginals[:θ])
    params_θ[2][:,:,k+1] = marginals[:θ].params[:w]
    params_η[1][1,k+1] = ForneyLab.unsafeMean(marginals[:η])
    params_η[2][1,k+1] = marginals[:η].params[:w]
    params_ζ[1][1,k+1] = marginals[:ζ].params[:a]
    params_ζ[2][1,k+1] = marginals[:ζ].params[:b]
    params_ξ[1][1,k+1] = marginals[:ξ].params[:a]
    params_ξ[2][1,k+1] = marginals[:ξ].params[:b]

end
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:14

```julia
p1 = plot(1:T_trn, inputs, color="red", label="inputs", ylims=[-1, 1.])
p2 = plot(1:T_trn, states, color="blue", label="states", ylims=[-1, 1.])
plot!(1:T_trn, params_z[1][1,2:end], color="purple", label="inferred", ylims=[-1, 1.])
plot!(1:T_trn, repeat([goal_state[1]], outer=(1,T_trn))', ribbon=[goal_state[2], goal_state[2]], label="goal state")
plot(p1, p2, layout=(2,1), size=(800,600))
```

```julia
# Scatter FE during training
p24 = plot(1:T_trn, free_energy_trn[:,end], color="black", xlabel="time (t)", ylabel="F[q]", label="", title="Free energy at training time")
```

```julia
# Scatter FE during training
p34 = plot(mean(free_energy_trn, dims=1)', color="black", xlabel="time (t)", ylabel="F[q]", label="", title="Average free energy over iterations")
```

### Validate identified system

We validate the identified system by computing simulation error on the validation set.

```julia
# Time horizon
T_val = 300

# Input signal
# input_val = [mean(sin.(t./ (0.1:.1:6 .*π))) for t in 1:T_val]
input_val = sin.((1:T_val) ./ (6*π))

# Preallocate arrays
states_val = zeros(T_val,)
output_val = zeros(T_val,)

for k = 4:T_val
    output_val[k], states_val[k] = sim_sys(input_val[k], [states_val[k-1], states_val[k-2]], sys_params)
end

# Write to file
```

```julia
# Prediction graph
graph2 = FactorGraph()

# Autoregressive node
@RV z_pred ~ NLatentAutoregressiveX(placeholder(:θ, dims=(3,)), placeholder(:z_tmin1, dims=(2,)), placeholder(:η), placeholder(:u_t), placeholder(:ζ), g=g, id=:z_pred_t)

# Draw time-slice subgraph
# ForneyLab.draw(graph2)

# Inference algorithm
q2 = PosteriorFactorization(z_pred, ids=[:z_pred])
algo2 = messagePassingAlgorithm([z_pred], q2, free_energy=true)
source_code2 = algorithmSourceCode(algo2, free_energy=true)
eval(Meta.parse(source_code2));
# println(source_code2)
```

```julia
# Initialize free energy tracking array
free_energy_pred = zeros(T_val, num_iterations)

# Initialize future state arrays
params_preds = (zeros(2, T_val), repeat(.1 .*float(eye(2)), outer=(1,1,T_val)))

# Start simulation with known output
params_preds[1][1,2] = output_val[2]
params_preds[1][2,2] = output_val[1]

@showprogress for k = 3:T_val

    # Initialize marginals
    marginals[:z_pred] = ProbabilityDistribution(Multivariate, GaussianMeanPrecision, m=params_preds[1][:,k], w=params_preds[2][:,:,k])

    # Clamp data
    data = Dict(:u_t => input_val[k],
                :z_tmin1 => params_preds[1][:,k-1],
                :θ => params_θ[1][:,end],
                :η => params_η[1][end],
                :ζ => params_ζ[1][end]/params_ζ[2][end])

    # Iterate variational parameter updates
    for i = 1:num_iterations

        # Make prediction
        stepz_pred!(data, marginals)

        # Compute free energy
        free_energy_pred[k, i] = freeEnergy(data, marginals)
    end

    # Store current parameterizations of marginals
    params_preds[1][:,k] = ForneyLab.unsafeMean(marginals[:z_pred])
    params_preds[2][:,:,k] = marginals[:z_pred].params[:w]

end
```

    Progress: 100%|█████████████████████████████████████████| Time: 0:00:00

### Visualize results

```julia
# Mean and std dev of predictions
predictions_mean = params_preds[1][1,:]
predictions_std = sqrt.(inv.(params_preds[2][1,1,:]))

# Plot predictions
p23 = scatter(1:T_val, output_val, label="observations", xlabel="time (t)", color="black")
plot!(1:T_val, predictions_mean, ribbon=[predictions_std, predictions_std], label="predictions", color="red")
```

```julia
Plots.savefig(p23, "figures/simulation_error.png")
```

```julia
# Compute prediction error
sq_pred_error = (predictions_mean[2:end] .- output_val[2:end]).^2

# Simulation error
MSE_sim = mean(sq_pred_error)

# Scatter error over time
p24 = scatter(1:T_val, sq_pred_error, color="black", xlabel="time (t)", ylabel="Prediction error", label="")
title!("MSE = "*string(MSE_sim))
```

```julia
Plots.savefig(p24, "figures/pred-errors.png")
```

```julia
# Scatter FE during validation
p24 = plot(1:T_val, free_energy_pred[:,end], color="black", xlabel="time (t)", ylabel="F[q]", label="", title="Free energy of predictions")
```

```julia

```