 Creating the Best Fantasy Basketball Lineup

 Introduction

Fantasy basketball, and fantasy sports in general, is an extremely popular game where participants compete against each other by virtually selecting players from real upcoming games to create a lineup. Typically, the goal is to assemble the best all-star team, meaning participants are allowed to select players from different professional teams that compete in the same league to play in each coach's virtual team.

A basketball team typically consists of five players: a point guard (PG), a shooting guard (SG), a small forward (SF), a power forward (PF), and a center (C). Each player, assigned to these positions, can have different characteristics and objectives in the basketball game. A shooting guard might specialize in scoring, while a point guard's objective might be to run the game-plan and distribute the ball efficiently. To help compare players across different positions we need to find a common metric to assess each player's performance -- and we'll use fantasy points to do this.

In a follow-on example we will extend the optimization model to reflect actual fantasy basketball competitions found on major sites.

 Objective and Prerequisites

In this example, we'll take on the role of a basketball coach and learn how to find the optimal lineup of National Basketball Association (NBA) players in the context of fantasy basketball. The goal is to select the five players who are going to perform the best in the NBA games played on December 25, 2017, by simultaneously satisfying player and position eligibility and budget constraints.

We aim to demonstrate that machine learning and mathematical optimization are closely connected, and how they can be integrated to derive optimal decisions. Simply starting from a dataset, we will apply machine learning techniques to make predictions about the performance of basketball players. But when taking into account the problem constraints, directly utilizing these forecasts to make player selections isn't straightforward. This is where mathematical optimization and the Gurobi Optimizer thrive to prove what the best viable lineup is, by formulating and solving a mixed-integer programming (MIP) problem that will maximize the lineup's **total fantasy points**.

This example is for beginners in mathematical optimization who may have some experience in data handling and predictive modeling and experience using Python.

The presented problem requires the installation of the following Python packages:

- **pandas** for data analysis and manipulation
- **numpy** for calculations
- **matplotlib** and **seaborn** for plotting and visualizing information
- **scikit-learn** for accessing data science predictive tools
- **gurobipy** for utilizing Gurobi to formulate and solve the optimization problem

 Problem Statement and Solution Approach

Given the regular season historical performances (box scores) of NBA players for the seasons 2016-2017 and 2017-2018, the eligible positions for each of the players, and a budget restriction on how much total salary we are allowed to spend for our team, select the five NBA players who are going to have the best performance on the NBA games played on Christmas day of the 2017-2018 season.

Among the eligible players that are going to play that day, we need to select a point guard, a shooting guard, a shooting forward, a power forward, and a center.

The solution to the problem consists of two components: 1) **fantasy points forecast** and 2) **lineup optimization**.

 Fantasy Points Forecast

Let's start by loading the necessary libraries to solve our problem.

```python
%pip install gurobipy
```

```python
import pandas as pd                                       #importing pandas
import numpy as np                                        #importing numpy
import matplotlib.pyplot as plt                           #importing matplotlib
import seaborn as sns                                     #importing seaborn
from sklearn.model_selection import train_test_split      #importing scikit-learn's function for data splitting
from sklearn.linear_model import LinearRegression         #importing scikit-learn's linear regressor function
from sklearn.neural_network import MLPRegressor           #importing scikit-learn's neural network function
from sklearn.ensemble import GradientBoostingRegressor    #importing scikit-learn's gradient booster regressor function
from sklearn.metrics import mean_squared_error            #importing scikit-learn's root mean squared error function for model evaluation
from sklearn.model_selection import cross_validate        #improting scikit-learn's cross validation function
from gurobipy import Model, GRB, quicksum                 #importing Gurobi
```

We are going to use two publicly available datasets from [Kaggle](https://www.kaggle.com). The [first dataset](https://www.kaggle.com/pablote/nba-enhanced-stats) contains the historical information of the players' and teams' performances and will be for training and testing our predictive model. The [second dataset](https://www.kaggle.com/alandu20/daily-fantasy-basketball-draftkings) includes the salaries of players, as well as their eligible position. Note that *salaries* here are determined by sites that host the fantasy sports contests and have nothing to do with what the players are actually paid.

We have slightly preprocessed the two datasets to have consistency in the players' names as well as including solely the columns that we would need for the purposes of this project. Also, we utilize observations where athletes have played for at least 3 minutes in a basketball game. Both of the updated versions of the datasets are available in our **repository**.

We begin by reading the boxscores dataset, which includes the regular season performances for all players from season 2016-2017 up to the games played on December 25th, 2017. For games before that date, we calculated each player's total fantasy points using box score values as follows:

$FP = Points + 1.5 \times Assists - 0.5 \times Turnovers + Steals + 2 \times Blocks + 1.25 \times Rebounds + 0.5 \times ThreePointer$

This value is already calculated in the **boxscores_dataset.csv** file.

```python
boxscores = pd.read_csv('https://raw.githubusercontent.com/yurchisin/modeling-examples/master/fantasy_basketball_1_2/boxscores_dataset.csv')     #load boxscores dataset
boxscores = boxscores[(boxscores.playMin>=3) | (boxscores.playMin.isnull())]
```

In general, fantasy points are connected with features related to the utilization and efficiency of each player and we can examine some of these relationships through visualization. Below are a few scatterplots, but feel free to add more visuals to get a deeper understanding of what's happening in the data.

```python
fig, (FGA, FGM, FTM, Min) = plt.subplots(1, 4, figsize=(14,5))
fig.tight_layout()

FGA.scatter(boxscores['playFGA'], boxscores['FantasyPoints'], c='blue', alpha = .2)
FGM.scatter(boxscores['playFGM'], boxscores['FantasyPoints'], c='lightblue', alpha = .2)
FTM.scatter(boxscores['playFTM'], boxscores['FantasyPoints'], c='coral', alpha = .2)
Min.scatter(boxscores['playMin'], boxscores['FantasyPoints'], c='purple', alpha = .2)

FGA.set_xlabel('Field Goal Attempts')
FGM.set_xlabel('Field Goals Made')
FTM.set_xlabel('Free Throws Made')
Min.set_xlabel('Minutes Played')

FGA.set_ylabel('Fantasy Points');
```

A distribution plot of the true fantasy points of the players for the previous games is shown below:

```python
hplot = sns.histplot(boxscores['FantasyPoints'], color="blue", label="Fantasy Points", kde=True, stat="density", linewidth=0, bins=20)
hplot.set_xlabel("Fantasy Points", fontsize = 12)
hplot.set_ylabel("Density", fontsize = 12)
sns.set(rc={"figure.figsize":(14, 5)})
```

When building our predictive model, fantasy points scored in a game is a good target (or independent) variable. But what about the model's features?

One approach is to use boxscore data for a player's previous several games to find the average of key stats over that span. We will begin with the average of the three previous games for: points, assists, turnovers, steals, blocks, total rebounds, field goal attempts, free throw attempts, 2-pointer \%, 3-pointer \%, free throw \%, minutes played, days-off and the true fantasy points.

For example, this calculation for the assists at the $k^{th}$ game is:

$Average \hspace{0.1cm} Assists_k = \frac{Assists_{k-1}+Assists_{k-2}+Assists_{k-3}}{3}$

```python
horizon=3

for column_name in ['playPTS','playAST','playTO','playSTL','playBLK','playTRB','playFGA','playFTA','play2P%','play3P%','playFT%','playMin','teamDayOff','FantasyPoints']:
    boxscores['moving' + column_name] = boxscores.groupby(['playDispNm'])[column_name].transform(lambda x: x.rolling(horizon, 1).mean().shift(1))                           #lagged moving average of numeric features
```

This gives us a set of features to start with, though we still would want to perform feature selection for our predictive model. Changing the number of lag games may improve your predictive model -- so give it a try!

Because we are using a lagged average, there is not any historical information for the first game played for each player, so we drop these observations.

```python
boxscores.dropna(subset = ["movingplayPTS"], inplace=True)               #drop the first observation for each player
```

To start with feature selection, we can look at the correlation between our newly derived variables, which is shown below:

```python
sns.set(rc = {'figure.figsize':(15,8)})
sns.heatmap(boxscores[['movingplayPTS', 'movingplayAST','movingplayTO','movingplaySTL','movingplayBLK','movingplayTRB','movingplayFGA','movingplayFTA','movingplay2P%','movingplay3P%','movingplayFT%','movingplayMin','movingteamDayOff','movingFantasyPoints']].corr(),annot=True)
plt.show()
```

We observe that some of the features are highly correlated. For example, the average of free throw attempts with the points average.

We are also investigating the utilization of two categorical variables. Specifically, if the current game is home or away and if the player will start or come from the bench. We transform the categorical variables to 0-1 variables.

```python
boxscores['dummyTeamLoc'] = pd.get_dummies(data=boxscores['teamLoc'],drop_first=True)    #1 if the game is a home game, 0 if it is an away game
boxscores['dummyplayStat'] = pd.get_dummies(data=boxscores['playStat'],drop_first=True)  #1 if the player starts, 0 if the player comes from the bench
```

Now that the dataset has been updated, we move into forecasting the fantasy points for the players who are going to play on December 25, 2017.

```python
forecasting_data = boxscores[boxscores.gmDate != '2017-12-25']  #for model training, we exclude observation on December 25, 2017
```

We split the data set using 80\% of the data for training and 20\% for testing. For forecasting we try three models: linear regression, a neural network and a gradient boosting regressor. In addition to the heat map, we based out statistical significance tests and looked into multicollinearity using variance inflation factors (VIF). Cross-validation is also used to identify potential overfitting.

The features that will be used for our predictive models are the average of the player's: assists, turnovers, steals, blocks, total rebounds, free throw attempts, free throw %, as well as whether the player will start or will come from the bench.

```python
X = forecasting_data[['movingplayAST','movingplayTO','movingplaySTL','movingplayBLK','movingplayTRB','movingplayFTA','movingplayFT%','dummyplayStat']]  #select the features that will be used for model training
y = forecasting_data['FantasyPoints']  #target set
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=4)      #dataset splitting

linear_regressor = LinearRegression()                                                         #load linear regressor
linear_regressor.fit(X_train, y_train)                                                        #train the linear regression model
linear_regression_validation = cross_validate(linear_regressor, X_train, y_train, cv=5, return_train_score=True, return_estimator=True)

mlp = MLPRegressor(hidden_layer_sizes=(5,5), activation='relu')                               #load neural network
mlp.fit(X_train,y_train)                                                                      #train the neural network with a ReLU function and two hidden layers with 5 nodes each
mlp_validation = cross_validate(mlp, X_train, y_train, cv=5, return_train_score=True, return_estimator=True)

gb = GradientBoostingRegressor()                                                              #load a gradient boosting regressor
gb.fit(X_train, y_train)                                                                      #train a gradient boosting model
gb_validation = cross_validate(gb, X_train, y_train, cv=5, return_train_score=True, return_estimator=True)
```

    /Users/yurchisin/opt/anaconda3/lib/python3.9/site-packages/sklearn/neural_network/_multilayer_perceptron.py:614: ConvergenceWarning: Stochastic Optimizer: Maximum iterations (200) reached and the optimization hasn't converged yet.
      warnings.warn(

```python
gb_validation['test_score']                            #print CV test score across each fold
```

    array([0.50957097, 0.49910472, 0.48128309, 0.49759688, 0.50076644])

We observe that the model performs consistently across each of the folds. Now, to evaluate the performance of the models we calculate their mean squared error (MSE) values.

```python
linear_regression_predictions = linear_regressor.predict(X_test)                              #make predictions based on the test set for the linear regression model
mlp_predictions = mlp.predict(X_test)                                                         #make predictions based on the test set for the neural network model
gb_predictions = gb.predict(X_test)                                                           #make predictions based on the test set for the gradient boosting model

linear_regression_mse = mean_squared_error(y_test, linear_regression_predictions)             #calculate the MSE for the linear regression model
mlp_mse = mean_squared_error(y_test, mlp_predictions)                                         #calculate the MSE for the neural network model
gb_mse = mean_squared_error(y_test, gb_predictions)                                           #calculate the MSE for the gradient boosting model

results = {'Linear Regression':[linear_regression_mse],'ReLU Neural Network':[mlp_mse],'Gradient Boosting Regressor':[gb_mse]}
modeling_results = pd.DataFrame(data=results,index=['MSE'])

modeling_results
```

      Linear Regression
      ReLU Neural Network
      Gradient Boosting Regressor

      MSE
      91.077312
      90.16697
      90.670528

Each of the model types perform similarly well as shown from their mean squared error values. Also, plots of the predicted value error with respect to the predicted fantasy points are shown in the following figures:

```python
fig, (LR, FNN, GBR) = plt.subplots(1, 3,figsize=(15,5));
fig.tight_layout()

LR.scatter(x = linear_regression_predictions, y = y_test - linear_regression_predictions,color='red',alpha=0.06)
FNN.scatter(x = mlp_predictions, y = y_test - mlp_predictions, color='green',alpha=0.06)
GBR.scatter(x = gb_predictions, y = y_test - gb_predictions, color='blue',alpha=0.06)

LR.set_xlabel('Linear Regression Predicted Fantasy Points')
FNN.set_xlabel('Neural Network Predicted Fantasy Points')
GBR.set_xlabel('Gradient Boosting Predicted Fantasy Points')

LR.set_ylabel('Linear Regression Residual')
FNN.set_ylabel('Neural Network Residual')
GBR.set_ylabel('Gradient Boosting Residual')
```

As we see from the above plots and the test scores, our models can only partially explain the variance of the fantasy points. That is to be expected since there are so many factors that affect the performance of the players. By a slight margin the gradient boosting regressor had the lowest MSE, so we'll refit that model to the complete data set and use that to predict fantasy points.

After that, we'll append the predicted fantasy points to the salary data to set up the optimization part of the solution.

```python
gb_final = GradientBoostingRegressor(random_state=4)
gb_final.fit(X, y)

optimization_dataset = boxscores
optimization_dataset['PredictedFantasyPoints'] = gb_final.predict(boxscores[['movingplayAST','movingplayTO','movingplaySTL','movingplayBLK','movingplayTRB','movingplayFTA','movingplayFT%','dummyplayStat']])

player_results = pd.read_csv('https://raw.githubusercontent.com/yurchisin/modeling-examples/master/fantasy_basketball_1_2/target_games.csv')
player_list = list(player_results['Player'].unique())
col = pd.DataFrame()

for player in player_list:
    player_flag = player
    optimization_data_per_player = optimization_dataset.loc[(optimization_dataset['playDispNm']==player)&(optimization_dataset['gmDate']=='2017-12-25')]
    col = col.append(optimization_data_per_player)

player_results['PredictedFantasyPoints'] = col['PredictedFantasyPoints'].values
```

We also calculate another column which is the predicted fantasy points divided by the salary of each player.

```python
pd.set_option('display.expand_frame_repr', False)
player_results['Points/Salary Ratio'] = 1000*player_results['PredictedFantasyPoints']/player_results['Salary']    #we multiple the fantasy vs salary ratio by 1000 for better visualization
player_results.sort_values(by='PredictedFantasyPoints',ascending=False).head(5)
```

      Player
      Pos
      Team
      Opp
      Salary
      PredictedFantasyPoints
      Points/Salary Ratio

      4
      Joel Embiid
      C
      PHI
      @ NYK
      9500
      51.313689
      5.401441

      0
      James Harden
      PG
      HOU
      @ OKC
      11100
      48.809577
      4.397259

      1
      LeBron James
      SF
      CLE
      @ GSW
      11000
      48.149718
      4.377247

      2
      Russell Westbrook
      PG
      OKC
      vs HOU
      10900
      44.007224
      4.037360

      3
      Kevin Durant
      SF
      GSW
      vs CLE
      10500
      43.438575
      4.137007

So how many potential lineups are there? A lot.

$25PG \times 23SF \times 22SG \times 19PF \times 9C = 2,163,150$ combinations

Using brute force to calculate the total fantasy points for each lineup would take a very, very long time. We also need to consider the salary constraint (e.g. $30,000). A greedy approach (i.e. always taking the best player available) would select the first three players that have the highest fantasy points (Embiid, James, and Harden), but then realize that the salary cap will be violated. Using the points to salary ratio can also lead to the same issue.

This is where optimization becomes most advantageous, due to its ability to efficiently explore the space of options in a very efficient manner and provably showing you have an optimal solution.

 Optimal Lineup Selection

Now that we have predictions for the players in our slate of NBA games we can work on determining our optimal lineup.

First, we'll make some definitions as these are helpful for us to set up our optimization model. For starters, the subscript $i$ will be used to denote individual players across the set of all players we can choose from.

**Input Parameters**

$p_{i}$: the predicted fantasy points of player $i$

$s_{i}$: the salary of player $i$

$S$: our total available salary

Here is the code to set up the indices and parameters:

```python
indices = player_results.Player
points = dict(zip(indices, player_results.PredictedFantasyPoints))
salaries = dict(zip(indices, player_results.Salary))
S = 30000

m = Model();        # this defines the model that we'll add to as we finish the formulation
```

**Decision Variables**

The goal of this problem is to determine whether each player is in our final lineup. This an example of when *binary variables* are used in optimization, to represent a **yes** or **no** decision we want to make. This is modeled as $y_{i} = 1$: if player $i$ is selected; and $y_{i} = 0$ otherwise.

Here is where we add a set of decision variables to our model in gurobipy:

```python
y = m.addVars(player_results.Player, vtype=GRB.BINARY, name="y")
```

**Objective Function**

The objective of our problem is to maximize the total fantasy points of our lineup. Since we are using binary variables, if a player is selected ($y_{i} = 1$), then their predicted points will contribute to our lineup using $p_{i} \cdot y_{i}$. If $y_{i} = 0$ then $p_{i} \cdot y_{i} = 0$ as well. Summing across all players will give us the function to maximize, written as:

\begin{align}
 Max \hspace{0.2cm} Z = \sum p_{i} \cdot y_{i}
\end{align}

Below is one way to write a summation using the quicksum function in gurobypi (you can get other ways to work) which we add to our model as the objective:

```python
# since we are maximizing points the last argument here is GRB.MAXIMIZE
m.setObjective(quicksum(points[i]*y[i] for i in indices), GRB.MAXIMIZE)
```

**Constraints**

We need to guarantee that each position has exactly one player assigned. Since our decision variables are each 0 or 1, we can model this by adding a constraint for each position summing across the eligible players (and decision variables) requiring that sum to be equal to 1:

\begin{align}
\sum_{i \space eligible} y_{i} = 1
\end{align}

Looping over the positions will add a constraint for each while the condition in the quicksum function will tell the model to only sum over players if they are eligible for that position:

```python
player_position_map = list(zip(player_results.Player, player_results.Pos))
for j in player_results.Pos:
    m.addConstr(quicksum([y[i] for i, pos in player_position_map if pos==j])==1)
```

Additionally, we need to ensure that the total salary must not exceed a certain threshold:

\begin{align}
\sum s_{i} \cdot y_{i} \leq S
\end{align}

```python
m.addConstr(quicksum(salaries[i]*y[i] for i in indices) <= S, name="salary");
#the budget for the selected team must not exceed 30,000
```

Now we've added all of the required parts to our optimization model and we are ready to solve.

```python
m.optimize()  #we optimize our model
```

    Gurobi Optimizer version 9.5.1 build v9.5.1rc2 (mac64[rosetta2])
    Thread count: 8 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 97 rows, 96 columns and 2082 nonzeros
    Model fingerprint: 0xb9f2d21c
    Variable types: 0 continuous, 96 integer (96 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+04]
      Objective range  [7e+00, 5e+01]
      Bounds range     [1e+00, 1e+00]
      RHS range        [1e+00, 3e+04]
    Found heuristic solution: objective 126.1046919
    Presolve removed 91 rows and 57 columns
    Presolve time: 0.00s
    Presolved: 6 rows, 39 columns, 78 nonzeros
    Found heuristic solution: objective 156.7777076
    Variable types: 0 continuous, 39 integer (39 binary)

    Root relaxation: objective 1.736178e+02, 7 iterations, 0.00 seconds (0.00 work units)

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0  173.61779    0    2  156.77771  173.61779  10.7%     -    0s
    H    0     0                     171.9189291  173.61779  0.99%     -    0s
         0     0  172.93546    0    4  171.91893  172.93546  0.59%     -    0s

    Cutting planes:
      Gomory: 1
      GUB cover: 1

    Explored 1 nodes (10 simplex iterations) in 0.03 seconds (0.00 work units)
    Thread count was 8 (of 8 available processors)

    Solution count 3: 171.919 156.778 126.105

    Optimal solution found (tolerance 1.00e-04)
    Best objective 1.719189291363e+02, best bound 1.719189291363e+02, gap 0.0000%

Much of the output above can be ignored for now, but will provide valuable information as you create larger and more complicated models. Let's focus on the last two lines which says an optimal solution was found and its objective value.

We need to extract the solution from the model to find which players are selected. Using getVars() gets the value of each decision variable, so we loop through the decision variables checking if its value (using .x) is greater than 0. While in the loop we record info about each player selected in a data frame.

```python
results = pd.DataFrame()

for v in m.getVars():
    if v.x > 1e-6:
        results = results.append(player_results.iloc[v.index][['Player','Pos','PredictedFantasyPoints','Salary']])
        print(v.varName, v.x)

print('Total fantasy score: ', m.objVal)
```

    y[Joel Embiid] 1.0
    y[Dario Saric] 1.0
    y[Trevor Ariza] 1.0
    y[Jarrett Jack] 1.0
    y[Markieff Morris] 1.0
    Total fantasy score:  171.91892913632242

Because the games were played, we also look up the true fantasy points of the players selected for the games on December 25, 2017, and we add that to our results dataframe.

```python
results['True Fantasy Points'] = [53.5,17.25,28.5,15.5,29.25]
results
```

      Player
      Pos
      PredictedFantasyPoints
      Salary
      True Fantasy Points

      4
      Joel Embiid
      C
      51.313689
      9500.0
      53.50

      19
      Dario Saric
      PF
      40.505486
      6200.0
      17.25

      36
      Trevor Ariza
      SG
      26.354157
      5000.0
      28.50

      43
      Jarrett Jack
      PG
      27.780012
      4600.0
      15.50

      49
      Markieff Morris
      SF
      25.965585
      4400.0
      29.25

The best total fantasy points based on *this initial set* of predictions is **$171.92$** by selecting **Jarrett Jack** at point guard, **Trevor Ariza** for small guard, **Markieff Morris** to be the shooting forward, **Dario Saric** as power forward and **Joel Embiid** to be the center of our team, with a total salary of $\$29,700$,

Let's quickly revisit the idea of putting together a lineup in a *greedy* fashion, meaning we start with the *best available* choice and repeat. Notice that players we predicted to have very high scoring games, such as James Harden or Lebron James, didn't make the optimal roster, so going after just the high scorers can be suboptimal.

If you were to re-sort the salary table by the ratio of predicted points to salary, we'd see PF Jordan Bell and PG Josh Hart atop the list. But neither of those players are in the optimal lineup as well, so even using that metric can lead to a suboptimal solution.

This shows the power of mathematical optimization. Our simple model was able to consider over 2 million possibilities and arrive at an optimal solution that would have been overlooked if we were to apply some intuitive decision rules when creating a lineup.

 Conclusion

In this example you learned how to optimize your basketball lineup in the context of fantasy sports. Starting with box score data, we constructed models that predict the future fantasy points of NBA athletes and then developed an optimization formulation to calculate the best lineup subject to budget and position eligibility constraints.

Through this example, our aim was to clearly present some of the connections between data science and mathematical optimization by starting with data and ending with a decision. We also saw how easy it is to make suboptimal decisions even when applying sensible rules.

We also have a more advanced fantasy basketball problem that uses the same predictive element but expands the optimization component to reflect real fantasy basketball contests from popular sites.


Contenido bajo licencia Creative Commons BY 4.0 y código bajo licencia MIT. © Juan Gómez y Nicolás Guarín-Zapata 2019-2020. Este material es parte del curso Modelación Computacional en el programa de Ingeniería Civil de la Universidad EAFIT.

# Determinación de raíces

## Introducción

El problema de determinar las raíces o ceros de una función aparece de manera frecuente en múltiples problemas de ingeniería. En este notebook discutiremos 2 de los métodos mas usados para su solución tales como el método de Bisección y el método de Newton-Raphson. El primero es interesante debido a su carácter intuitivo, mientras que el segundo, además de su mejor desempeño, es la base para la solución de problemas no-lineales de mayor complejidad como los que aparecen en métodos de elementos finitos. Para presentar los métodos utilizaremos un ejemplo tomado de un problema de la mecánica de los medios continuos.

**Al completar este notebook usted debería estar en la capacidad de:**

* Reconocer en detalle los aspectos algorítmicos de los métodos de búsqueda de raíces de Bisección y Newton-Raphson.

* Formular apropiadamente un problema como uno de búsqueda de raíces.

* Reconocer la diferencia entre convergencia y divergencia de un proceso iterativo.

* Resolver manualmente y con la ayuda del computador problemas de búsqueda de raíces.

## Ejemplo: Determinación de tensiones extremas en un sólido elástico

En un problema de análisis de tensiones para un sólido elástico se encuentra que los máximos esfuerzos $\sigma$ son las raíces del polinomio caracteristico correspondiente a la siguiente función:

\begin{equation}
f(\sigma)=\sigma ^3 -200 \sigma ^2 -100000 \sigma
\end{equation}

y en donde $\sigma \in [-500 , 500].$ Para efectos de verificar si una condición de diseño del sólido es apropiada se requiere determinar los valores extremos.[Ref 1](#Referencias)

```python
# Esta línea permite tener gráficos interactivos en el notebook
%matplotlib notebook
```

```python
import numpy as np
import matplotlib.pyplot as plt
```

### Paso 1: localización inicial de las raíces

Inicialmente haremos una búsqueda o localización, de baja precisión, de las raíces para posteriormente afinar los valores usando métodos más formales. Como estrategia inicial para localizar las raíces es conveniente graficar la función. En el siguiente bloque de código usaremos la función **lambda** de Python para definir la función y su derivada.

```python
fun  = lambda x: x**3 - 200.0*x**2 - 100000.0*x
deriv = lambda x: 3.0*x**2 - 400.0*x - 100000.0
```

Y luego la graficamos.

```python
npts = 200
x = np.linspace(-500, 500, npts)
y = fun(x)
plt.figure()
plt.plot(x, y)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.lines.Line2D at 0x1913dfb3f28>]

**Preguntas**

- Modificar los ejes de la gráfica y adicionar una cuadricula para facilitar la  identificación aproximada de las raíces.

- ¿Cuáles son los valores aproximados de las raíces que se obtienen de la gráfica?

Para hacer la búsqueda inicial usaremos el algoritmo de _bracketing_ el cual recibe como parámetros de entrada la función, los extremos del intervalo de búsqueda y el número de intervalos a considerar. En el siguiente bloque de código definimos la función **bracketing()** de manera que esta pueda ser utilizada de forma general.

```python
# Comentar apropiadamente el siguiente código
def bracketing(fun, a, b, N):
    """Estimate the root of a real function using bracketing

    Parameters
    ----------
    fun : function
        Function.
    a : float
        Initial point of the interval.
    b : float
        Final point of the interval.
    N : int
        Number of subdivisions for the interval

    Returns
    -------
    xR : list
        List with intervals where a change of sign was found.
    msg : string
        Message regarding the success of the method.
    """
    msg = "Maximum number of iterations reached."
    dx = (b - a)/(N - 1)
    iroot = 0
    x2 = a
    xR = []
    for i in range(0, N):
        x1 = x2
        x2 = x1 + dx
        if (fun(x1) * fun(x2)) < 0:
            msg = "A change of sign was found."
            iroot = iroot + 1
            xR.append([x1, x2])
    return xR, msg
```

En el uso de la rutina de búsqueda partiremos el dominio del problema en $N = 100$ sub-intervalos de manera que tendremos:

$$\triangle\sigma=\frac{b-a}{100}\equiv\frac{1000}{100}$$

**Preguntas**

- ¿Qué tan cercanas son las raíces encontradas por la rutina con las identificadas en la gráfica?

- ¿Qué pasa si se cambia el valor del numero de sub-intervalos en la operación de bracketing?

```python
xR, msg = bracketing(fun, -500, 500, 10)
xR
```

    [[-277.7777777777778, -166.6666666666667],
     [-55.5555555555556, 55.555555555555515],
     [388.8888888888888, 499.9999999999999]]

### Paso 2a: Afinación de la raíz por el método de bisección

El método de bisección consiste en la partición por mitades e identificación de la mitad en la que se presenta el cambio de signo, como se muestra en la figura. El método cada vez toma un sub-intervalo de menor tamaño variando la posición del punto $x_3$.

A continuación se presenta el método de bisección que toma como parámetros de entrada la función; los valores extremos del intervalo de búsqueda $a$ y $b$; y 2 valores de tolerancia usados para detectar la presencia de una raíz ya sea por el tamaño del intervalo de búsqueda o por la cercanía con cero del valor de la función.

```python
# Comentar apropiadamente el siguiente código
def bisection(fun, a, b, niter=50, ftol=1e-12, verbose=False):
    """Use bisection method to estimate the root of a real function

    Parameters
    ----------
    fun : function
        Function.
    grad : function
        Derivative of the function.
    a : float
        Initial point of the interval.
    b : float
        Final point of the interval.
    niter : int, optional
        Maximum number of iterations.
    ftol : float, optional
        Tolerance for the root.
    verbose : bool, optional
        If True, prints each iteration.

    Returns
    -------
    x : float
        Approximated root.
    msg : string
        Message regarding the success of the method.
    """
    if fun(a) * fun(b) > 0:
        c = None
        msg = "The function should have a sign change in the interval."
    else:
        for cont in range(niter):
            c = 0.5*(a + b)
            if verbose:
                print("n: {}, x: {}".format(cont, c))
            if abs(fun(c)) < ftol:
                msg = "Root found with desired accuracy."
                break
            elif fun(a) * fun(c) < 0:
                b = c
            elif fun(b) * fun(c) < 0:
                a = c
            msg = "Maximum number of iterations reached."
    return c, msg
```

```python
raiz, _ = bisection(fun, -300.0, -200.0)
raiz
```

    -231.6624790355399

```python
raiz, _ = bisection(fun, -100.0, 0.100)
raiz
```

    -7.723180237198344e-15

```python
raiz, _ = bisection(fun,  300.0, 500.0)
raiz
```

    431.66247903554

**Pregunta**

Para al menos una de las raíces pruebe con diferentes rangos de localización de la raíz e indique el desempeño del algoritmo en términos del número de iteraciones.

### Paso 2b: Afinación de la raíz por el método de Newton-Raphson

Recordemos el método de Newton-Raphson el cual consiste en extender la recta tangente en el punto actual $x_i$ hasta que cruce el cero para luego hacer la aproximación en la abscisa de dicho punto de cruce (ver figura). Considerando la pendiente en $x_i$ se tiene que:

$$
f'(x_i)\equiv m=\frac{0-f(x_i)}{x_{i+1}-x_i}
$$

resultando en la iteración de Newton-Raphson:

$$
x_{i+1}=x_i-\frac{f(x_i)}{f'(x_i)}
$$

En el siguiente bloque de código se implementa la iteración de Newton-Raphson. Note que además de la función, el algoritmo requiere como parámetros de entrada la primera derivada y un valor inicial de la raíz. El algoritmo puede fallar por alcanzar el máximo número de iteraciones permitido o por llegar a puntos en donde la derivada es cercana a cero.

```python
# Comentar apropiadamente el siguiente código

def newton(fun, grad, x, niter=50, ftol=1e-8, verbose=False):
    """Use Newton method to estimate the root of a real function

    Parameters
    ----------
    fun : function
        Function.
    grad : function
        Derivative of the function.
    x : float
        Initial estimate.
    niter : int, optional
        Maximum number of iterations.
    ftol : float, optional
        Tolerance for the root.
    verbose : bool, optional
        If True, prints each iteration.

    Returns
    -------
    x : array
        Approximated root.
    msg : string
        Message regarding the success of the method.
    """
    msg = "Maximum number of iterations reached."
    for cont in range(niter):
        if abs(grad(x)) < ftol:
            x = None
            msg = "Derivative near to zero."
            break
        if verbose:
            print("n: {}, x: {}".format(cont, x))
        x = x - fun(x)/grad(x)
        if abs(fun(x)) < ftol:
            msg = "Root found with desired accuracy."
            break
    return x, msg
```

Probemos con diferentes puntos iniciales.

```python
newton(fun, deriv, -230.0, verbose=True)
```

    n: 0, x: -230.0
    n: 1, x: -231.67883211678833
    n: 2, x: -231.6624805928219

    (-231.66247903554, 'Root found with desired accuracy.')

```python
newton(fun, deriv, -51.0, verbose=True)

```

    n: 0, x: -51.0
    n: 1, x: 10.94059640374946
    n: 2, x: 0.20496836097227877
    n: 3, x: 8.378324920088609e-05
    n: 4, x: 1.4039249228220071e-11

    (3.9420326066980225e-25, 'Root found with desired accuracy.')

```python
newton(fun, deriv, 430.0, verbose=False)

```

    (431.66247903554, 'Root found with desired accuracy.')

## Glosario de términos

**Raíz de una función $f(x)$:** Valor (o valores) de $x$ para los cuales se da la condición $f(x)=0$.

**Convergencia:** Aproximaciones sucesivas a un valor buscado a través de un proceso iterativo.

**Tolerancia:** Número real para definir el cero computacional en un algoritmo.

**Método de bisección:** Algoritmo que localiza la raíz de una función $f(x)$ a través de particiones sucesivas del intervalo de búsqueda.

**Método de Newton-Raphson:** Algoritmo que localiza la raíz de una función $f(x)$ a partir de la ecuación de la tangente en una aproximación inicial de la raíz.

## Actividad para la clase

El método de Newton-Raphson resulta de la ecuación de la tangente en un punto $x_i  $asumido como raíz de la función. Este método tiene como desventaja la necesidad de requerir la primera derivada de la función $f'(x)$, la cual no siempre esta disponible. Se puede formular también una iteración alternativa que evite el uso de $f'(x)$ si se hace uso de la recta secante entre 2 puntos $x_i$ y $x_{i+1}$ como se ilustra en la figura.

Para derivar el método partimos de dos puntos $x_i$ y $x_{i-1}$, y encontramos la recta
secante que pasa por estos. Extendiendo la misma hasta su cruce con cero resulta en:

$$m = \frac{0 - f(x_i)}{x_{i + 1} - x_{i}}\, ,$$

y sabiendo que $m = (f(x_i) - f(x_{i - 1}))/(x_{i} - x_{i - 1})$, se tiene la siguiente iteración:

$$x_{i + 1} = x_i - f(x_i) \frac{x_{i} - x_{i - 1}}{f(x_i) - f(x_{i - 1})}\, .$$

1. Modificar la rutina de Newton-Raphson de manera que determine la raíz de $f(x)$ usando el método de la secante.

2. Para la función del ejemplo determinar sus raíces usando los métodos de Newton-Raphson y de la secante y compare su desempeño en términos de precisión y número de iteraciones.

A continuación se brinda una plantilla para la actividad.

```python
def secante(fun, x0, x1, niter=50, ftol=1e-8):
    """Use secant method to estimate the root of a real function

    Parameters
    ----------
    fun : function
        Function.
    x0 : float
        Initial estimate.
    x1 : float
        Initial estimate.
    niter : int, optional
        Maximum number of iterations.
    ftol : float, optional
        Tolerance for the root.

    Returns
    -------
    x : array
        Approximated root.
    """
    for cont in range(niter): # Modificar
        grad = 1 # Aproximacion al gradiente
        x = x - fun(x)/grad
        if abs(fun(x)) < ftol:
            break
    return x
```

## Referencias

1. Gómez, Juan., Sierra, César., Vergara, Juan., Sáenz, Mario., and Guarín-Zapata, Nicolás. (2018) [Notas de clase: Mecánica del medio continuo](https://github.com/AppliedMechanics-EAFIT/Notas-MMC/raw/master/notas_de_clase/notas_medios.pdf). Universidad EAFIT.

2. Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1996). Numerical recipes in Fortran 90. Cambridge: Cambridge university press.

## Formato del notebook

La siguiente celda cambia el formato del Notebook.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>


```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import control.matlab as cm
import sympy as sy
```

# Preparations
Continuous time state space system
\begin{align}
\dot{x} &= Ax + Bu\\
y &= Cx,
\end{align}
with
\begin{align}
A &= \begin{bmatrix} 2 & -2\\ 1 & 0 \end{bmatrix}\\
B &= \begin{bmatrix} 1\\0\end{bmatrix}\\
C &= \begin{bmatrix} 0 & 1\end{bmatrix}
\end{align}
Want closed loop system with poles in $p_{1,2} = -2 \pm j$
using statefeedback $u = Lx + mr$.

The sampling time is $h=0.1 s$.
## Preparation exercise 1
Determine $L$ so that eigenvalues of $A+BL$ as specified

Note that the system is on controllable form. It thus have the characteristic equation
\begin{equation}
s^2 -2s + 2.
\end{equation}
With state feedback, the system is still on controllable form:
\begin{equation}
A+BL = \begin{bmatrix} 2+l_1 & -2+l_2\\1 & 0 \end{bmatrix}
\end{equation}
with characteristic equation
\begin{equation}
s^2 + (-2-l_1)s + (2-l_2).
\end{equation}
The desired characteristic equation is
\begin{equation}
(s+2+i)(s+2-i) = s^2 + 4s + 5.
\end{equation}
Identifying the coefficients gives
\begin{align}
l_1 &= -6\\
l_2 &= -3
\end{align}

```python
A = np.array([[2, -2], [1, 0]])
B = np.array([[1],[0]])
C = np.array([[0,1]])
D = np.array([[0]])
```

```python
As = sy.Matrix(A)
Bs = sy.Matrix(B)
Cs = sy.Matrix(C)
l1,l2 = sy.symbols("l1 l2")
L = sy.Matrix([[l1, l2]])
Ac = A+B*L
pc = Ac.eigenvals()
pc
```

    {l1/2 - sqrt(l1**2 + 4*l1 + 4*l2 - 4)/2 + 1: 1,
     l1/2 + sqrt(l1**2 + 4*l1 + 4*l2 - 4)/2 + 1: 1}

```python
l1l2 = sy.solve([l1/2+1+2, l1**2+4*l1+4*l2-4 + 4], (l1,l2))
L = np.array([float(l) for l in l1l2[0]])
L
```

    array([-6., -3.])

```python
Ac2 = Ac.subs(l1,-6).subs(l2,-3)
Ac2.eigenvals()
```

    {-2 - I: 1, -2 + I: 1}

## Preparations exercise 2
Derive a discrete time representation with sampling time $h=0.1$
Solution to the continuous time system starting from a known state $x(0)$ at time $t=0$ is given by
\begin{equation}
x(t) = \mathrm{e}^{At}x(0) + \int_0^{t}\mathrm{e}^{A(t-\tau)}Bu(\tau)d\tau.
\end{equation}
Without loss of generality, we can assume $t$ to denote the (continous) time elapsed since the last sampling instant $k$. Assuming that $u$ is constant between sampling instants  $u(t)=u(0)$ for $t<h$, we get the solution at time $t=h$
\begin{equation}
x(h) = \mathrm{e}^{Ah}x(0) + \Big(\int_0^{h}\mathrm{e}^{A(h-\tau)}Bd\tau\Big) u(0) = \Phi(h)x(0) + \Gamma(h)u(0).
\end{equation}
See Åstrom & Wittenmark 3.2.
We can for instance compute by series expansion
\begin{equation}
\Psi = \int_0^h\mathrm{e}^{A\tau}d\tau = Ih + \frac{Ah^2}{2!} + \frac{A^2h^3}{3!} + \cdots
\end{equation}
\begin{align}
\Phi &= I + A\Psi\\
\Gamma &= \Psi B
\end{align}

```python
tol = 1e-12
h = 0.1
hp = h
term = np.identity(2)*h
Psi = term
k=1.0
while np.linalg.norm(term) > tol:
    k += 1.0
    term = np.dot(A,term)*h/k
    Psi += term
    hp *= h

print k
Phi = np.identity(2) + np.dot(A,Psi)
Gamma = np.dot(Psi, B)

print Phi
print Gamma
```

    9.0
    [[ 1.20998266 -0.22066598]
     [ 0.11033299  0.98931668]]
    [[ 0.11033299]
     [ 0.00534166]]

## Preparation exercise 3
Determine discrete time poles that correspond to the continuous time ones.

The mapping from continuous to discrete time poles is given by (3.54) in Å&W section 3.6
\begin{equation}
\lambda_i(\Phi) = \mathrm{e}^{\lambda_i(A)h}
\end{equation}

```python
z1 = np.exp((-2-1j)*h)
z2 = np.exp((-2+1j)*h)
print z1
print z2
```

    (0.814640509554-0.0817366883936j)
    (0.814640509554+0.0817366883936j)

Find the control law $L$ for the discrete time case using Ackermans formula
\begin{equation}
L = \begin{bmatrix} 0  & \cdots & 0 & 1\end{bmatrix} W_c^{-1}P(\Phi),
\end{equation}
where $W_c$ is the controllability matrix for the system, i.e.
\begin{equation}
W_c = \begin{bmatrix} \Gamma & \Phi\Gamma & \cdots & \Phi^{n-1}\Gamma \end{bmatrix},
\end{equation}
and $P(\Phi)$ is the characteristic equation of the closed loop system with $\Phi$ as argument. That is,
\begin{equation}
P(\Phi) = \Phi^n + p_1\Phi^{n-1} + \cdots + p_nI,
\end{equation}
with
\begin{equation}
P(z) = z^n + p_1z^{n-1} + \cdots + p_n = 0
\end{equation}
giving the desired poles of the closed loop system.

```python
p1 = -z1-z2
p2 = z1*z2
Wc = np.hstack((Gamma, np.dot(Phi,Gamma)))
P = np.dot(Phi, Phi) + p1*Phi + p2*np.identity(2)
WinvP = np.dot(np.linalg.inv(Wc), P)
Ld = -np.real(WinvP[-1,:])
Ld.shape = (1,2)
print Ld

Phic = Phi + np.dot(Gamma, Ld)
np.linalg.eig(Phic)
```

    [[-5.08339449 -1.71336368]]

    (array([ 0.81464051+0.08173669j,  0.81464051-0.08173669j]),
     array([[ 0.91172384+0.j        ,  0.91172384-0.j        ],
            [-0.36834206-0.18188944j, -0.36834206+0.18188944j]]))

# Computer experiments
## Computer exercise 1
Controller design for the continuous time system

```python
sys_c = cm.ss(A,B,C,D)
sys_d = cm.ss(Phi, Gamma, C, D, h)
Lc = -cm.place(A, B, [-2+1j, -2-1j])
print (L-Lc)
print L
print Lc
print np.linalg.eig(A+np.dot(B,Lc))
```

    [[ -1.77635684e-15   1.77635684e-15]]
    [-6. -3.]
    [[-6. -3.]]
    (array([-2.+1.j, -2.-1.j]), array([[ 0.91287093+0.j        ,  0.91287093-0.j        ],
           [-0.36514837-0.18257419j, -0.36514837+0.18257419j]]))

## Computer exercise 2
Controller design for the discrete time system

```python
sys_d2 = cm.c2d(sys_c, h)
print sys_d
print sys_d2
Ld2 = -cm.place(Phi,Gamma,[z1,z2])
print (Ld-Ld2)
print Ld
print Ld2
print np.linalg.eig(Phi+np.dot(Gamma,Ld2))
print z1
```

    A = [[ 1.20998266 -0.22066598]
     [ 0.11033299  0.98931668]]

    B = [[ 0.11033299]
     [ 0.00534166]]

    C = [[0 1]]

    D = [[0]]

    dt = 0.1

    A = [[ 1.20998266 -0.22066598]
     [ 0.11033299  0.98931668]]

    B = [[ 0.11033299]
     [ 0.00534166]]

    C = [[0 1]]

    D = [[0]]

    dt = 0.1

    [[ -8.88178420e-16  -3.10862447e-15]]
    [[-5.08339449 -1.71336368]]
    [[-5.08339449 -1.71336368]]
    (array([ 0.81464051+0.08173669j,  0.81464051-0.08173669j]), array([[ 0.91172384+0.j        ,  0.91172384-0.j        ],
           [-0.36834206-0.18188944j, -0.36834206+0.18188944j]]))
    (0.814640509554-0.0817366883936j)

## Computer exercise 3 Simulation
Create closed loop continuous time system with unit dc gain

```python
Ac = A + np.dot(B,Lc)
sys_cl = cm.ss(Ac, B, C, D)
dcg = cm.dcgain(sys_cl)
m = 1.0/dcg[0,0]
sys_cl = sys_cl*m
print cm.dcgain(sys_cl)

```

    [[ 1.]]

Discrete system with same feedback gain as continuous time system, and with correct feedback gain

```python
sys_cl_d1 = cm.ss(Phi+np.dot(Gamma,Lc), Gamma*m, C, D, h )
sys_cl_d2 = cm.ss(Phi+np.dot(Gamma,Ld), Gamma, C, D, h )
dcg_d = np.dot(np.dot(C, np.linalg.inv(np.identity(2)-sys_cl_d2.A)), sys_cl_d2.B)
md = 1.0/dcg_d[0,0]
sys_cl_d2 = md * sys_cl_d2

```

Simulate the three systems

```python
Tc = np.linspace(0,10,100)
Td = h*np.arange(100)
(yc, tc) = cm.step(sys_cl, Tc)
(yd1, td1) = cm.step(sys_cl_d1, Td)
(yd2, td2) = cm.step(sys_cl_d2, Td)
yy1 = np.ravel(yd1)
yy2 = np.ravel(yd2)

plt.plot(tc,yc)
plt.plot(td1,np.ravel(yd1)/1)
plt.plot(td2,np.ravel(yd2)/yy2[-1])
plt.xlim([0, 4])
plt.legend(["cont", "discrete w cont param", "discrete w discr param"], loc="lower right")

```

```python
Tc = np.linspace(0,10,100)
Td = h*np.arange(100)
(yc, tc) = cm.impulse(sys_cl, Tc)
(yd1, td1) = cm.impulse(sys_cl_d1, Td)
(yd2, td2) = cm.impulse(sys_cl_d2, Td)

plt.plot(tc,yc)
plt.plot(td1,np.ravel(yd1))
plt.plot(td2,np.ravel(yd2))
plt.legend(["cont", "discrete w cont param", "discrete w discr param"], loc="upper right")

```

# Effect of sampling interval
Try different values of sampling time

```python
(yc, tc) = cm.step(sys_cl, Tc)
plt.plot(tc,yc)
for hh in [0.01, 0.1, 0.2, 0.4, 0.8, 1.6]:
    sys_d2 = cm.c2d(sys_c, hh)
    z1 = np.exp((-2-1j)*hh)
    z2 = np.exp((-2+1j)*hh)
    F = sys_d2.A
    G = sys_d2.B

    Ld2 = -cm.place(F,G,[z1,z2])
    print Ld2
    sys_cl_d = cm.ss(F+np.dot(G,Ld2), G, C, D, hh )
    # Find dc gain of the discrete time system (See below)
    dcg_d = np.dot(np.dot(C, np.linalg.inv(np.identity(2)-sys_cl_d.A)), sys_cl_d.B)
    md = 1.0/dcg_d[0,0]
    sys_cl_d = md * sys_cl_d
    print md
    print cm.dcgain(sys_cl_d)
    Td = np.arange(1000)*hh
    (yd1, td1) = cm.step(sys_cl_d, Td)
    yy = np.ravel(yd1)
    #plt.plot(td1,yy/yy[-1])
    plt.plot(td1,yy)
    plt.xlim([0, 10])
```

# DC gain for discrete system
In steady state with step input we have $x(k+1) = x(k)$
\begin{equation}
x(k) = \Phi x(k) + m\Gamma
\end{equation}
Hence,
\begin{equation}
x(\infty) = m(1-\Phi)^{-1}\Gamma,
\end{equation}
and so
\begin{equation}
y(\infty) = mC(1-\Phi)^{-1}\Gamma.
\end{equation}
For this to be equal to 1, we must have
\begin{equation}
m = \frac{1}{C(1-\Phi)^{-1}\Gamma}.
\end{equation}

```python

```


# Разработать программу, способную решать систему линейных алгебраических уравнений, содержащую n уравнений и n неизвестных, методом Крамера.

\begin{equation}
\left \{ \begin{aligned}
& a_{11}x_1+a_{12}x_2+a_{13}x_3+\ldots+a_{1n}x_n=b_1;\\
& a_{21}x_1+a_{22}x_2+a_{23}x_3+\ldots+a_{2n}x_n=b_2;\\
& \ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots\ldots \\
& a_{n1}x_1+a_{n2}x_2+a_{n3}x_3+\ldots+a_{nn}x_n=b_n.
\end{aligned} \right.
\end{equation}

**Входные данные:**

`0≤i,j<n`

В первой строке вводится целое число n, количсетво уравнений и неизвестных.

Далее во входном потоке идет n строк по n вещественных чисел aij, коэффициенты уравнения, разделенные пробелом.

В последней строке вводятся n вещественных чисел bi, свободные члены, разделенные пробелом.

**Выходные данные:**

В одной строке вывести корни данного уравнения, разделенные пробелом.

**Пример входных данных:**

3

2.0 -1.0 0.0

-1.0 1.0 4.0

1.0 2.0 3.0

0.0 13.0 14.0

**Пример выходных данных:**

1.0 2.0 3.0

#### Разберем, как решаются уравнения методом Крамера.

Пусть есть система уравнений:

\begin{equation}
\left \{ \begin{aligned}
& a_{11}x_1+a_{12}x_2+a_{13}x_3=b_1\\
& a_{21}x_1+a_{22}x_2+a_{23}x_3=b_2\\
& a_{31}x_1+a_{32}x_2+a_{33}x_3=b_3
\end{aligned} \right.
\end{equation}

Для нахождения решения данного уравнения нам необходимо найти следующий определеители матрицы матрицы:

\begin{equation}
\Delta = \begin{vmatrix} a_{11} a_{12} a_{13}\\  a_{21} a_{22} a_{23}\\  a_{31} a_{32} a_{33} \end{vmatrix}
\Delta_1 = \begin{vmatrix} b_{1} a_{12} a_{13}\\ b_{2} a_{22} a_{23}\\ b_{3} a_{32} a_{33} \end{vmatrix}
\Delta_2 = \begin{vmatrix} a_{11} b_{1} a_{13}\\ a_{21} b_{2} a_{23}\\ a_{31} b_{3} a_{33} \end{vmatrix}
\Delta_3 = \begin{vmatrix} a_{11} a_{12} b_{1}\\ a_{21} a_{22} b_{2}\\ a_{31} a_{32} b_{3} \end{vmatrix}
\end{equation}

В определителях столбец коэффициентов при соответствующей неизвестной заменяется столбцом свободных членов системы.

`Решение:`

\begin{equation}
x_1 = \dfrac{\Delta_1}{\Delta} , x_2 = \dfrac{\Delta_2}{\Delta} , x_3 = \dfrac{\Delta_3}{\Delta}
\end{equation}

##### Для решения поставленной задачи необходимо разработать 2 функции:

Первая функция `будет находить определитель матрицы.`

Вторая функция `будет подставлять столбец свободных членов в соответсвующие поизиции и считать определитель от них.`

Для реализации первой функции я воспользовался частным случаем теоремы Лапласа.
Его суть заключается в следующем:
Он позволяет представить определитель квадратной матрицы в виде суммы произведений элементов любой её строки или столбца на их алгебраические дополнения.

Представленая реализация выполняет следующее:
Идет по столбцу матрицы от 0 до n
Для каждого элемента aij в столбце формирует его алгебраическое дополнение(Алгебраическим дополнением Aij элемента aij матрицы A n− го порядка называется число равное произведению минора Mij на (−1)i+j:

\begin{equation}
a_{ij}=(-1)^{i+j}*M_{ij}
\end{equation}

Данное алгебраическое дополнение умножается на сам элмент.

При `рекурсивном` вызове функции мы доходим до матрицы размера 1x1 - определителем такой матрицы будет являться едеинственный элемент матрицы.

\begin{equation}
\det A=\sum\limits_{j=1}^na_{ij}A_{ij}
\end{equation}

Для реализации второй я "вставлял" массив b на необходимую позицию и запоминал значение определителя, для последющего вычисления значений x по формуле:

\begin{equation}
x_1 = \dfrac{\Delta_1}{\Delta} , x_2 = \dfrac{\Delta_2}{\Delta} , x_3 = \dfrac{\Delta_3}{\Delta}
\end{equation}

```python
# Функция для нахождения определителя матрицы

def determinant(matrix):
    if len(matrix) == 1:
        return matrix[0][0]
    else:
        n = len(matrix)
        summa = 0
        for i in range(n):
            minor = []
            for x in range(n):
                temporary = []
                for y in range(n):
                    if x != i and y != 0:
                        temporary.append(matrix[x][y])
                if len(temporary) == n - 1:
                    minor.append(temporary)
            if i % 2 == 0:
                summa += matrix[i][0] * determinant(minor)
            else:
                summa -= matrix[i][0] * determinant(minor)
        return summa
```

```python
# Функция для решения СЛАУ

def slae_solver(x, y):
    det1 = determinant(x)
    if det1 == 0:
        return None
    n = len(x)
    ans = []
    deters = []
    for k in range(n):
        matrix = []
        for i, line in enumerate(x):
            temp = []
            for j, var in enumerate(line):
                if j == k:
                    temp.append(y[i])
                else:
                    temp.append(var)
            matrix.append(temp)
        deters.append(determinant(matrix))
    for det in deters:
        ans.append(det/det1)
    return ans
```

```python
# Считываем данные

n = int(input())
a = [list(map(float, input().split())) for i in range(n)]
b = list(map(float, input().split()))
```

    3
    2.0 -1.0 0.0
    -1.0 1.0 4.0
    1.0 2.0 3.0
    0.0 13.0 14.0

```python
answer = slae_solver(a, b)
for elem in answer:
    print(elem, end = " ")
```

    1.0 2.0 3.0


### If you want to quickly see how to fit your own data, jump down to the "Fit model to data" section

# Multi-population recurrent switching linear dynamical systems overview

This notebook goes through the simulation example shown in our manuscript (Figure 2A,B).

Below, we briefly describe the model. We also recommend looking at the "Recurrent SLDS" notebook, which provides more details on the standard rSLDS.

**1. Data**.
Let $y_t^{_{(j)}}$ denote a vector of activity measurements of the $N_j$ neurons in population $j$ in time bin $t$.

**2. Emissions**.
let $x_t^{_{(j)}}$ denote a continuous latent state of population $j$ at time $t$. The population states may differ in dimensionality~$D_j$, since populations may differ in size and complexity. The observed activity of population $j$ is modeled with a generalized linear model,
\begin{align}
    E[y_t^{(j)}] &= f(C_j x_t^{(j)} + d_j),
\end{align}
where each population has its own linear mapping parameterized by $\{C_j, d_j\}$. In this notebook, we use a Poisson GLM. Inputs can also be passed into this GLM, as described in the rSLDS notebook.

There are multi-population emissions classes that will be loaded in the example below.

**3. Continuous State Update (Dynamics)**.
The dynamics of a switching linear dynamical system are piecewise linear, with the linear dynamics at a given time determined by a discrete state, (more on discrete states below).

\begin{align}
    x_t \sim
    A^{(z_t)} x_{t-1} + b^{(z_t)}
\end{align}

where $z_t$ is the discrete state, $A^{(z_t)}$ is the dynamics for that discrete state, and $x_t$ contains the latents from all populations, $[x_t^1, x_t^2, ..., x_t^J]$. We ignore the noise term here for simplicity.

Having unique continuous latents for each population allows us to decompose the dynamics in an interpretable manner. We model the temporal dynamics of the continuous states as
\begin{align}
    x_t^{(j)} \sim
    A_{(j \: to \: j)}^{(z_t)} x_{t-1}^{(j)}
    + \sum_{i \neq j} A_{(i \: to \: j)}^{(z_t)} x_{t-1}^{(i)}
    + b_j^{(z_t)}.
\end{align}

In the full dynamics matrices, $A^{(z_t)}$ we will show in the example below, the on-diagonal blocks represent the internal dynamics, $A_{(j \: to \: j)}^{(z_t)}$ and the off-diagonal blocks represent the external dynamics, $A_{(i \: to \: j)}^{(z_t)}$.

**4. Discrete State Update (Transitions)**.
Recurrent transitions are based on the continuous latent state. Our recurrent transitions have a sticky component, $S$ that determines the probabilities of staying in a state, and a switching component, $R$, that determines the probabilities of switching to states. In the model we use in this notebook:

\begin{align}
    p(z_t = i \mid z_{t-1} = j, x_{t-1}) &= \mathrm{softmax}\bigg\{ \Big( \big(R x_{t-1}\big) + r\Big) \odot (1 - e_{z_{t-1}})  + \Big( \big(S x_{t-1} \big) + s \Big) \odot e_{z_{t-1}}  \bigg\},
\end{align}

where $e_{z_{t-1}} \in \{0,1\}^K$ is a one-hot encoding of $z_{t-1}$.

To understand which populations are contributing to the transitions, we can decompose this equation:

\begin{align}
    p(z_t = i \mid z_{t-1} = j, x_{t-1}) &= \mathrm{softmax}\bigg\{ \Big( \sum_{j=1}^J \big(R_j x_{t-1}^{(j)}\big) + r\Big) \odot (1 - e_{z_{t-1}})  + \Big( \sum_{j=1}^J \big(S_j x_{t-1}^{(j)} \big) + s \Big) \odot e_{z_{t-1}}  \bigg\},
\end{align}
where, for example, $R_j x_{t-1}^{(j)}$ contains the contribution of population $j$ towards switching to each state.

Additionally, we can include a dependency on the previous discrete state. This is included in the code package, but is not used in the example below.

\begin{align}
    p(z_t = i \mid z_{t-1} = j, x_{t-1}) &= \mathrm{softmax}\bigg\{ \log(P_{j,i}) +  \big(R x_{t-1}\big) \odot (1 - e_{z_{t-1}})  + \big(S x_{t-1} \big) \odot e_{z_{t-1}}  \bigg\},
\end{align}

There are sticky multi-population emissions classes that will be loaded in the example below.

**5. Model fitting**.
We fit the model with variational laplace EM - see the "Variational Laplace EM for SLDS Tutorial" for more information.

## Import packages, including multipopulation extensions

```python
try:
    import ssm
except:
    !pip install git+https://github.com/lindermanlab/ssm.git#egg=ssm
    import ssm
```

```python
#### General packages

from matplotlib import pyplot as plt
%matplotlib inline
import autograd.numpy as np
import autograd.numpy.random as npr

import seaborn as sns
sns.set_style("white")
sns.set_context("talk")
sns.set_style('ticks',{"xtick.major.size":8,
"ytick.major.size":8})
from ssm.plots import gradient_cmap, white_to_color_cmap

color_names = [
    "purple",
    "red",
    "amber",
    "faded green",
    "windows blue",
    "orange"
    ]

colors = sns.xkcd_palette(color_names)
cmap = gradient_cmap(colors)
```

```python
#### SSM PACKAGES ###

import ssm
from ssm.variational import SLDSMeanFieldVariationalPosterior, SLDSTriDiagVariationalPosterior, \
    SLDSStructuredMeanFieldVariationalPosterior
from ssm.util import random_rotation, find_permutation, relu

#Load from extensions
from ssm.extensions.mp_srslds.emissions_ext import GaussianOrthogonalCompoundEmissions, PoissonOrthogonalCompoundEmissions
from ssm.extensions.mp_srslds.transitions_ext import StickyRecurrentOnlyTransitions, StickyRecurrentTransitions
```

## Simulate (somewhat realistic) data

### Set parameters of simulation

```python
K=3 #Number of discrete states

num_gr=3 #Number of populations
num_per_gr=5 #Number of latents per population
neur_per_gr=75 #Number of neurons per population

t_end=3000 #number of time bins
num_trials=1 #number of trials
```

```python
np.random.seed(108) #To create replicable dynamics

alphas=.03+.1*np.random.rand(K) #Determines the distribution of values in the dynamics matrix, for each discrete state
print('alphas:', alphas)

sparsity=.33 #Proportion of non-diagonal blocks in the dynamics matrix that are 0

e1=.1 #Amount of noise in the dynamics
```

    alphas: [0.05336111 0.03343876 0.08800889]

### Get new emissions and transitions classes for the simulated data

```python
#Vector containing number of latents per population
D_vec=[]
for i in range(num_gr):
    D_vec.append(num_per_gr)

#Vector containing number of neurons per population
N_vec=[]
for i in range(num_gr):
    N_vec.append(neur_per_gr)

D=np.sum(D_vec)
num_gr=len(D_vec)
D_vec_cumsum = np.concatenate(([0], np.cumsum(D_vec)))

#Get new multipopulation emissions class for the simulation

# gauss_comp_emissions=GaussianOrthogonalCompoundEmissions(N=np.sum(N_vec),K=1,D=np.sum(D_vec),D_vec=D_vec,N_vec=N_vec)
poiss_comp_emissions=PoissonOrthogonalCompoundEmissions(N=np.sum(N_vec),K=1,D=np.sum(D_vec),D_vec=D_vec,N_vec=N_vec,link='softplus')

#Get transitions class
true_sro_trans=StickyRecurrentOnlyTransitions(K=K,D=np.sum(D_vec))
```

### Create simulated data

```python
np.random.seed(10) #To create replicable simulations

A_masks=[]

A_all=np.zeros([K,D,D]) #Initialize dynamics matrix
b_all=np.zeros([K,D]) #Initialize dynamics offset

#Create initial ground truth model, that we will modify
true_slds = ssm.SLDS(N=np.sum(N_vec),K=K,D=int(np.sum(D_vec)),
             dynamics="gaussian",
             emissions=poiss_comp_emissions,
             transitions=true_sro_trans)

#Create ground truth transitions
v=.2+.2*np.random.rand(1)
for k in range(K):
    inc=np.copy(k)
    true_slds.transitions.Rs[k,D_vec_cumsum[inc]:D_vec_cumsum[inc]+1]=v
    true_slds.transitions.Ss[k,D_vec_cumsum[inc]:D_vec_cumsum[inc]+1]=v-.1

true_slds.transitions.r=0*np.ones([K,1])
true_slds.transitions.s=5*np.ones([K,1])

#Create ground truth dynamics for each state
for k in range(K):

    ##Create dynamics##
    alpha=alphas[k]

    A_mask=np.random.rand(num_gr,num_gr)>sparsity #Make some blocks of the dynamics matrix 0

    A_masks.append(A_mask)

    for i in range(num_gr):
        A_mask[i,i]=1

    A0=np.zeros([D,D])
    for i in range(D-1):
        A0[i,i+1:]=alpha*np.random.randn(D-1-i)
    A0=(A0-A0.T)

    for i in range(num_gr):
        A0[D_vec_cumsum[i]:D_vec_cumsum[i+1],D_vec_cumsum[i]:D_vec_cumsum[i+1]]=2*A0[D_vec_cumsum[i]:D_vec_cumsum[i+1],D_vec_cumsum[i]:D_vec_cumsum[i+1]]

    A0=A0+np.identity(D)
    A=A0*np.kron(A_mask, np.ones((num_per_gr, num_per_gr)))

    A=A/(np.max(np.abs(np.linalg.eigvals(A)))+.01) #.97

    b=1*np.random.rand(D)

    A_all[k]=A
    b_all[k]=b

true_slds.dynamics.As=A_all
true_slds.dynamics.bs=b_all

zs, xs, _ = true_slds.sample(t_end) #Sample discrete and continuous latents from model for simulation

#Get spike trains that have an average firing rate of 0.25 per bin
tmp=np.mean(relu(np.dot(true_slds.emissions.Cs[0],xs.T)+.1*true_slds.emissions.ds[0][:,None]).T)
mult=.25/tmp
lams=relu(mult*np.dot(true_slds.emissions.Cs[0],xs.T)+.1*true_slds.emissions.ds[0][:,None]).T
ys=np.random.poisson(lams) #Get spiking activity based on poisson statistics
```

## Plot simulated data

### Dynamics matrices ($A^z$)

```python
# vmin,vmax=[-1,1]
vmin,vmax=[-.5,.5] #zoom in to see colors more clearly

for k in range(K):

    plt.figure(figsize=(4,4))
    plt.imshow(true_slds.dynamics.As[k], aspect='auto', interpolation="none", vmin=vmin, vmax=vmax, cmap='RdBu')
    offset=-.5
    for nf in D_vec:
        plt.plot([-0.5, D-0.5], [offset, offset], '-k')
        plt.plot([offset, offset], [-0.5, D-0.5], '-k')
        offset += nf
    plt.xticks([])
    plt.yticks([])
    plt.title('Actual State '+str(k))
```

### Discrete states ($z$)

```python
plt.figure(figsize=(8, 4))
plt.subplot(211)
plt.imshow(zs[None,:], aspect="auto", cmap=cmap, vmin=0, vmax=len(colors)-1)
plt.xlim(0, t_end)
plt.ylabel("$z_{\\mathrm{true}}$")
plt.yticks([])
```

### Transitions (in a shorter time window)
The contribution of population $j$ to staying in a state is $S_j x^{j}$ and the contribution to switching to a state is $R_j x^{j}$

```python
dur=200
st_t=650
end_t=st_t+dur
```

```python
plt.figure(figsize=(8, 4))

j=0

plt.subplot(211)
for g in range(K):
    plt.plot(np.dot(xs[st_t:end_t,D_vec_cumsum[g]:D_vec_cumsum[g+1]],true_slds.transitions.Rs[j,D_vec_cumsum[g]:D_vec_cumsum[g+1]].T))
plt.xlim(0, dur)
plt.ylabel('Contribution towards \n switching to \n purple state',rotation=60)
plt.xticks([])
plt.yticks([])

j=1
plt.subplot(212)
for g in range(K):
    plt.plot(np.dot(xs[st_t:end_t,D_vec_cumsum[g]:D_vec_cumsum[g+1]],true_slds.transitions.Ss[j,D_vec_cumsum[g]:D_vec_cumsum[g+1]].T))
plt.xlim(0, dur)
plt.ylabel('Contribution towards \n staying in red state',rotation=60)
plt.legend(['Pop 1','Pop 2','Pop 3'])
plt.yticks([])
```

### Continuous latents ($x$) and spikes ($y$) for an example population

```python
plt.figure(figsize=(8, 4))

plt.subplot(211)
plt.plot(xs[st_t:end_t,:num_per_gr]) #Show latents of first group
plt.xticks([])

plt.subplot(212)
plt.plot(ys[st_t:end_t,:10]) #Show first 10 neurons

```

## Fit model to data

#### To create the emissions classes for the multipopulation models, we need vectors containing the number of continuous latents per population ("D_vec") and neurons per population ("N_vec")

```python
num_gr=3 #Number of populations
num_per_gr=5 #Number of latents per population
neur_per_gr=75 #Number of neurons per population

#Vector containing number of latents per population
D_vec=[]
for i in range(num_gr):
    D_vec.append(num_per_gr)

#Vector containing number of neurons per population
N_vec=[]
for i in range(num_gr):
    N_vec.append(neur_per_gr)
```

#### Now create the multipopulation emissions and transitions classes for our model

```python
#Get new multipopulation emissions class
poiss_comp_emissions=PoissonOrthogonalCompoundEmissions(N=np.sum(N_vec),K=1,D=np.sum(D_vec),D_vec=D_vec,N_vec=N_vec,link='softplus')

#Get new transitions class
sro_trans=StickyRecurrentOnlyTransitions(K=K,D=np.sum(D_vec), l2_penalty_similarity=10, l1_penalty=10)
#The above l2 penalty is on the similarity between R and S (its assuming the activity to switch into a state is similar to activity to stay in a state)
#The L1 penalty is on the entries of R and S
```

Note that another new emissions class is "GaussianOrthogonalCompoundEmissions"

Note that another new transitions class is "StickyRecurrentTransitions"

#### Now declare and fit the model

```python
K=3 #Number of discrete states

rslds = ssm.SLDS(N=np.sum(N_vec),K=K,D=np.sum(D_vec),
             dynamics="gaussian",
             emissions=poiss_comp_emissions,
             transitions=sro_trans,
             dynamics_kwargs=dict(l2_penalty_A=100)) #Regularization on the dynamics matrix

q_elbos_ar, q_ar = rslds.fit(ys, method="laplace_em",
                             variational_posterior="structured_meanfield",
                             continuous_optimizer='newton',
                             initialize=True,
                             num_init_restarts=10,
                             num_iters=30,
                             alpha=0.25)
```

    HBox(children=(IntProgress(value=0, max=10), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    HBox(children=(IntProgress(value=0, max=30), HTML(value='')))

```python
plt.plot(q_elbos_ar[1:])
plt.xlabel("Iteration")
plt.ylabel("ELBO")
```

## Align solution with simulation for plotting

```python
#The recovered discrete states can be permuted in any way.
#Find permutation to match the discrete states in the model and the ground truth
z_inferred=rslds.most_likely_states(q_ar.mean_continuous_states[0],ys)
rslds.permute(find_permutation(zs, z_inferred))
z_inferred2=rslds.most_likely_states(q_ar.mean_continuous_states[0],ys)
```

```python
#Each population's latents can be multiplied by an arbitrary rotation matrix
#Additionally, there may be a change in scaling between the simulation ground truth and recovered latents,
#because the simulation didn't constrain the effective emissions (C) matrix to be orthonormal like in the model

from sklearn.linear_model import LinearRegression

R=np.zeros([D,D])
for g in range(num_gr):
    lr=LinearRegression(fit_intercept=False)
    lr.fit(q_ar.mean_continuous_states[0][:,D_vec_cumsum[g]:D_vec_cumsum[g+1]],xs[:,D_vec_cumsum[g]:D_vec_cumsum[g+1]])
    R[D_vec_cumsum[g]:D_vec_cumsum[g+1],D_vec_cumsum[g]:D_vec_cumsum[g+1]]=lr.coef_
```

## Plot results

### Discrete states ($z$)

```python
plt.figure(figsize=(8, 4))
plt.subplot(211)
plt.imshow(zs[None,:], aspect="auto", cmap=cmap, vmin=0, vmax=len(colors)-1)
plt.xlim(0, t_end)
plt.ylabel("$z_{\\mathrm{true}}$")
plt.yticks([])

plt.subplot(212)
plt.imshow(z_inferred2[None,:], aspect="auto", cmap=cmap, vmin=0, vmax=len(colors)-1)
plt.xlim(0, t_end)
plt.ylabel("$z_{\\mathrm{inferred}}$")
plt.yticks([])
plt.xlabel("time")

plt.tight_layout()
```

```python
print('Discrete state accuracy: ', np.mean(zs==z_inferred2))
```

    Discrete state accuracy:  0.978

#### Shorter time window

```python
plt.figure(figsize=(4, 2))
plt.subplot(211)
plt.imshow(zs[None,st_t:end_t], aspect="auto", cmap=cmap, vmin=0, vmax=len(colors)-1)
plt.xlim(0, dur)
plt.ylabel("$z_{\\mathrm{true}}$")
plt.yticks([])
plt.xticks([])

plt.subplot(212)
plt.imshow(z_inferred2[None,st_t:end_t], aspect="auto", cmap=cmap, vmin=0, vmax=len(colors)-1)
plt.xlim(0, dur)
plt.ylabel("$z_{\\mathrm{inferred}}$")
plt.yticks([])
```

### Dynamics matrices ($A^z$)

We show the A matrix from when the continuous latents are aligned to ground truth, demonstrating the ability to recover the ground truth dynamics.

We also show the original recovered A matrix, which demonstrates that we can learn about the block structure, regardless of scaling/rotations.

```python
plt.figure(figsize=(12, 12))

q=1

for k in range(K):

    plt.subplot(3,3,q)
    plt.imshow(true_slds.dynamics.As[k], aspect='auto', interpolation="none", vmin=-.5, vmax=.5, cmap='RdBu')
    offset=-.5
    for nf in D_vec:
        plt.plot([-0.5, D-0.5], [offset, offset], '-k')
        plt.plot([offset, offset], [-0.5, D-0.5], '-k')
        offset += nf
    plt.xticks([])
    plt.yticks([])
    plt.title('Actual State '+str(k))

    q=q+1

    plt.subplot(3,3,q)
#     plt.imshow(rslds.dynamics.As[k], aspect='auto', interpolation="none", vmin=-.5, vmax=.5, cmap='RdBu')
    plt.imshow(R@rslds.dynamics.As[k]@np.linalg.inv(R), aspect='auto', interpolation="none", vmin=-.5, vmax=.5, cmap='RdBu')

    offset=-.5
    for nf in D_vec:
        plt.plot([-0.5, D-0.5], [offset, offset], '-k')
        plt.plot([offset, offset], [-0.5, D-0.5], '-k')
        offset += nf
    plt.xticks([])
    plt.yticks([])
    plt.title('Aligned Predicted State '+str(k))
#     plt.savefig(folder+'dyn_est'+str(k)+'.pdf')

    q=q+1

    plt.subplot(3,3,q)
#     plt.imshow(rslds.dynamics.As[k], aspect='auto', interpolation="none", vmin=-.5, vmax=.5, cmap='RdBu')
    plt.imshow(rslds.dynamics.As[k], aspect='auto', interpolation="none", vmin=-.5, vmax=.5, cmap='RdBu')

    offset=-.5
    for nf in D_vec:
        plt.plot([-0.5, D-0.5], [offset, offset], '-k')
        plt.plot([offset, offset], [-0.5, D-0.5], '-k')
        offset += nf
    plt.xticks([])
    plt.yticks([])
    plt.title('Raw Predicted State '+str(k))
#     plt.savefig(folder+'dyn_est'+str(k)+'.pdf')

    q=q+1

```

### Transitions
The contribution of population $j$ to staying in a state is $S_j x^{j}$ and the contribution to switching to a state is $R_j x^{j}$

```python
plt.figure(figsize=(15, 4))

### Actual

j=0

plt.subplot(221)
for g in range(K):
    plt.plot(np.dot(xs[st_t:end_t,D_vec_cumsum[g]:D_vec_cumsum[g+1]],true_slds.transitions.Rs[j,D_vec_cumsum[g]:D_vec_cumsum[g+1]].T))
plt.xlim(0, dur)
plt.ylabel('Contribution towards \n switching to \n purple state',rotation=60)
plt.xticks([])
plt.yticks([])
plt.title('Actual')

j=1
plt.subplot(223)
for g in range(K):
    plt.plot(np.dot(xs[st_t:end_t,D_vec_cumsum[g]:D_vec_cumsum[g+1]],true_slds.transitions.Ss[j,D_vec_cumsum[g]:D_vec_cumsum[g+1]].T))
plt.xlim(0, dur)
plt.ylabel('Contribution towards \n staying in red state',rotation=60)
plt.legend(['Pop 1','Pop 2','Pop 3'])
plt.yticks([])

### Predicted

j=0

plt.subplot(222)
for g in range(K):
    plt.plot(np.dot(q_ar.mean_continuous_states[0][st_t:end_t,D_vec_cumsum[g]:D_vec_cumsum[g+1]],rslds.transitions.Rs[j,D_vec_cumsum[g]:D_vec_cumsum[g+1]].T))
plt.xlim(0, dur)
# plt.ylabel('Contribution towards \n switching to \n purple state',rotation=60)
plt.xticks([])
plt.yticks([])
plt.title('Predicted')

j=1
plt.subplot(224)
for g in range(K):
    plt.plot(np.dot(q_ar.mean_continuous_states[0][st_t:end_t,D_vec_cumsum[g]:D_vec_cumsum[g+1]],rslds.transitions.Ss[j,D_vec_cumsum[g]:D_vec_cumsum[g+1]].T))
plt.xlim(0, dur)
# plt.ylabel('Stay in Red')
# plt.xticks([])
plt.yticks([])
```

### Example fit of neural activity ($y$)

```python
preds=rslds.smooth(q_ar.mean_continuous_states[0],ys) #get predictions

nrn=0 #Example neuron
plt.plot(ys[st_t:end_t,nrn],alpha=.5) #true spiking activity
plt.plot(lams[st_t:end_t,nrn]) #true firing rate
plt.plot(preds[st_t:end_t,nrn]) #predicted firing

plt.legend(['True Spikes','True FR','Predicted FR'])

```


```python
# Init matplotlib

%matplotlib inline

import matplotlib
```

```python
FIG_SIZE = 4

def plot_linear_system(Ab, bounds=None, ax=None, color='r', alpha=1):
    """Plot the linear system defined by the extended matrix `Ab`."""

    # Check arguments #############

    assert Ab.ndim == 2
    assert Ab.shape[1] == 3

    if bounds is None:
        if ax is None:
            bounds = np.array([[-10, 10],
                               [-10, 10]])
        else:
            bounds = np.array(ax.axis()).reshape([2,2])

    assert bounds.ndim == 2
    assert bounds.shape == (2, 2)
    assert bounds[0,0] < bounds[0,1]
    assert bounds[1,0] < bounds[1,1]

    # Setup the plot ##############

    if ax is None:
        fig, ax = plt.subplots(figsize=(FIG_SIZE, FIG_SIZE))

    ax.axis('equal')

    ax.axis(bounds.flatten())
    ax.set_xticks(np.arange(*bounds[0], 1))
    ax.set_yticks(np.arange(*bounds[0], 1))

    ax.axhline(y=0, color='k')
    ax.axvline(x=0, color='k')

    # Compute points to plot ######

    x = bounds[0]            # horizontal axis limits

    Abnv = Ab[Ab[:,1] != 0]  # linear equations having a non-null x_2 coefficient (i.e. non-vertical line)
    Abv = Ab[Ab[:,1] == 0]   # linear equations having a null x_2 coefficient (i.e. vertical line)

    # Plot non-vertical lines #####

    yp1 = (Abnv[:,2] - Abnv[:,0] * x[0]) / Abnv[:,1]
    yp2 = (Abnv[:,2] - Abnv[:,0] * x[1]) / Abnv[:,1]
    plt.plot(x, np.array([yp1, yp2]), "-{}".format(color), alpha=alpha)

    # Plot vertical lines #########

    for ab in Abv:
        plt.axvline(ab[2] / ab[0], color=color, alpha=alpha)

    # TODO: add label (e.g. "2x + 3y = 1")

    plt.grid(True)

    return ax
```

```python
pre_str1 = r"""
$
\left\{
\begin{align}
"""

post_str1 = r"""\end{align}
\right.
$
"""

pre_str2 = r"""
$
\begin{pmatrix}
"""

post_str2 = r"""\end{pmatrix}
$
"""

def complete_matrix_to_latex(Ab):
    #6 x_1 + 7 x_2 &  = 11

    sys_str = ""

    for eq in Ab:

        eq_str = ""

        for coef_index, coef in enumerate(eq[:-1]):

            if coef != 0:
                if len(eq_str) > 0:
                    eq_str += "+ "
                if coef == 1:
                    eq_str += "x_{" + str(coef_index + 1) + "} "
                elif coef == -1:
                    eq_str += "-x_{" + str(coef_index + 1) + "} "
                else:
                    eq_str += str(coef) + " x_{" + str(coef_index + 1) + "} "

        eq_str = eq_str.replace("+ -", "- ")
        eq_str += "& = {} \\\\\n".format(eq[-1])
        sys_str += eq_str

    sys_str = pre_str1 + sys_str + post_str1

    mat_str = ""

    for eq in Ab:

        eq_str = ""

        for coef_index, coef in enumerate(eq):

            if len(eq_str) > 0:
                eq_str += " & "
            eq_str += str(coef)

        eq_str += "\\\\\n"
        mat_str += eq_str

    mat_str = pre_str2 + mat_str + post_str2

    return sys_str + "\n" + mat_str

```

## Méthode par élimination (ou combinaison)

### Exemple 1

```python
Ab = np.array([[ 1,  5,  7],
               [-2, -7, -5]])

print(complete_matrix_to_latex(Ab))
plot_linear_system(Ab);
```

```python
Ab = np.array([[1, 5, 7],
               [0, 3, 9]])

print(complete_matrix_to_latex(Ab))
plot_linear_system(Ab);
```

```python
Ab = np.array([[1, 5, 7],
               [0, 1, 3]])

print(complete_matrix_to_latex(Ab))
plot_linear_system(Ab);
```

```python
Ab = np.array([[1, 0, -8],
               [0, 1,  3]])

print(complete_matrix_to_latex(Ab))
plot_linear_system(Ab);
```


```
# default_exp
```

```
#hide
#all_slow
```

# 2-D posterior toy example

> Demonstration of posterior error bounds for a 2-D toy example, for various levels of sensor noise

In this notebook we will demonstrate the error bounds for the statFEM posterior for the toy example introduced in twoDim. We first import some required packages.

```
from dolfin import *
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.cm as cm
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = (10,6)
from scipy.linalg import sqrtm
import sympy; sympy.init_printing()
from tqdm.notebook import tqdm
# code for displaying matrices nicely
def display_matrix(m):
    display(sympy.Matrix(m))
# import required functions from twoDim
from statFEM.twoDim import mean_assembler, kernMat, cov_assembler, m_post, gen_sensor, MyExpression, m_post_fem_assembler, c_post, c_post_fem_assembler
```

Since we again to not have access to the true distribution we will use the method outlined in the section 2-D prior toy example to approximate the convergence rate. We now set up the required code.

First we set up the mean and kernel functions for the random forcing term $f$.

```
# set up mean and kernel functions
f_bar = Constant(1.0)

l_f = 0.4
σ_f = 0.1

def k_f(x):
    return (σ_f**2)*np.exp(-(x**2)/(2*(l_f**2)))

def m_f(x):
    return 1.0
```

We now set up a reference grid on which we will compare the covariance matrices.

```
N = 41
x_range = np.linspace(0,1,N)
grid = np.array([[x,y] for x in x_range for y in x_range])
```

```
ϵ = 0.001
```

We now set up the sensor grid and generate some sensor data.

```
%%time
s = 25
s_sqrt = int(np.round(np.sqrt(s)))
Y_range = np.linspace(0.01,0.99,s_sqrt)
Y = np.array([[x,y] for x in Y_range for y in Y_range])
J_fine = 120
np.random.seed(42)
v_dat = gen_sensor(ϵ,m_f,k_f,Y,J_fine,False,True)
```

    CPU times: user 1min 28s, sys: 11.8 s, total: 1min 40s
    Wall time: 22.1 s

We now set up a function to get the statFEM prior and posterior for a FE mesh size $h$, using functions from twoDim.

```
def fem_prior(h,f_bar,k_f,grid):
    J = int(np.round(1/h))
    μ = mean_assembler(h,f_bar)
    Σ = cov_assembler(J,k_f,grid,False,True)
    return μ,Σ
```

```
def fem_posterior(h,f_bar,k_f,ϵ,Y,v_dat,grid):
    J = int(np.round(1/h))
    m_post_fem = m_post_fem_assembler(J,f_bar,k_f,ϵ,Y,v_dat)
    μ = MyExpression()
    μ.f = m_post_fem
    Σ = c_post_fem_assembler(J,k_f,grid,Y,ϵ,False,True)
    return μ,Σ
```

We now see if the sensor level noise causes the posterior to look different from prior.

```
h = 0.05
μ_prior, Σ_prior = fem_prior(h,f_bar,k_f,grid)
μ_posterior, Σ_posterior = fem_posterior(h,f_bar,k_f,ϵ,Y,v_dat,grid)
```

```
#hide_input
fig, (ax1,ax2,ax3) = plt.subplots(1,3,figsize=(24,6))

sns.heatmap(Σ_prior,cbar=True,
              annot=False,
              xticklabels=False,
              yticklabels=False,
              cmap=cm.viridis,
              ax=ax1)
ax1.set_title('statFEM prior covariance matrix, ϵ: %.5f' % ϵ)

sns.heatmap(Σ_posterior,cbar=True,
              annot=False,
              xticklabels=False,
              yticklabels=False,
              cmap=cm.viridis,
              ax=ax2)
ax2.set_title('statFEM posterior covariance matrix, ϵ: %.5f' % ϵ)

ax3.plot(np.diag(Σ_prior),label='prior')
ax3.plot(np.diag(Σ_posterior),label='posterior')
ax3.grid()
ax3.set_xlabel("i")
ax3.set_ylabel("\operatorname{var}(u(x_i))")
ax3.set_title("Variance at each grid point")
ax3.legend()

plt.show()
```

We now set up a function to compare the covariance functions on the reference grid.

```
def compute_cov_diff(C1,C2,tol=1e-10):
    N = np.sqrt(C1.shape[0])
    #N = C1.shape[0]
    C1_sqrt = np.real(sqrtm(C1))
    rel_error_1 = np.linalg.norm(C1_sqrt @ C1_sqrt - C1)/np.linalg.norm(C1)
    assert rel_error_1 < tol

    C12 = C1_sqrt @ C2 @ C1_sqrt
    C12_sqrt = np.real(sqrtm(C12))
    rel_error_12 = np.linalg.norm(C12_sqrt @ C12_sqrt - C12)/np.linalg.norm(C12)
    assert rel_error_12 < tol

    hSq = (1/(N-1))**2
    return hSq*(np.trace(C1) + np.trace(C2) - 2*np.trace(C12_sqrt))
```

We now set up a function to compute the Wasserstein distance between the statFEM posteriors.

```
def W(μ_1,μ_2,Σ_1,Σ_2,J_norm):
    mean_error = errornorm(μ_1,μ_2,'L2',mesh=UnitSquareMesh(J_norm,J_norm))
    cov_error = compute_cov_diff(Σ_1,Σ_2)
    cov_error = np.sqrt(np.abs(cov_error))
    error = mean_error + cov_error
    return error
```

In the interests of memory efficiency we will again create a function which will compute the ratios of errors by succesively refining the FE mesh as done in the section 2-D prior toy example.

```
def refine(h,n,f_bar,k_f,ϵ,Y,v_dat,grid,J_norm):
    # set up empty lists to hold h-values and errors (this being the ratios)
    h_range = []
    errors = []
    # get the statFEM posterior for h and h/2
    μ_1, Σ_1 = fem_posterior(h,f_bar,k_f,ϵ,Y,v_dat,grid)
    μ_2, Σ_2 = fem_posterior(h/2,f_bar,k_f,ϵ,Y,v_dat,grid)
    # compute the distance between these and store in numerator variable
    numerator = W(μ_1,μ_2,Σ_1,Σ_2,J_norm)
    # succesively refine the mesh by halving and do this n times
    for i in tqdm(range(n)):
        # store mean and cov for h/2 in storage for h
        μ_1, Σ_1 = μ_2, Σ_2
        # in storage for h/2 store mean and cov for h/4
        μ_2, Σ_2 = fem_posterior(h/4,f_bar,k_f,ϵ,Y,v_dat,grid)
        # compute the distance between the posteriors for h/2 and h/4
        # and store in denominator variable
        denominator = W(μ_1,μ_2,Σ_1,Σ_2,J_norm)
        # compute the ratio and store in error
        error = numerator/denominator
        # append the current value of h and the ratio
        h_range.append(h)
        errors.append(error)
        # store denominator in numerator and halve h
        numerator = denominator
        h = h/2
    # return the list of h-values together with the ratios for these values
    return h_range,errors
```

We now set up a list of starting $h$ values and number of refinements $n$ to get a decent number of ratios to approximate $p$. We also set up the `J_norm` variable needed to control the grid on which the mean error is computed in the Wasserstein distance.

```
my_list = [(0.25,4),(0.2,3),(0.175,3),(0.176,3),(0.177,3),(0.178,3),(0.179,3),(0.18,3),(0.21,3),(0.215,3),(0.1,2),(0.3,4),(0.31,4),(0.315,4),(0.24,3),(0.245,3),(0.14,2),(0.16,2),(0.15,2)]
J_norm = 40
```

#hide

The following cells use `pickle` to load the results, avoiding having to wait for the results to compute.

```
#hide
# import pickle
```

#hide

If this next cell is run the dictionary `results` will be filled in from the file `2D_posterior_results_full_noise_0.001`. The code below can then be run to produce the final figure without having to wait for the results to be computed.

```
#hide
# with open('2D_posterior_results_full_noise_0.001','rb') as f:
#     results = pickle.load(f)
# h_range = results['h']
# errors = results['ratios']
```

We now compute the results:

#hide

This next cell shouldn't be run if you load the results.

```
%%time
h_range = []
errors = []
np.random.seed(235)
for h,n in tqdm(my_list,desc='Outer loop'):
    h_range_tmp, errors_tmp = refine(h,n,f_bar,k_f,ϵ,Y,v_dat,grid,J_norm)
    h_range.extend(h_range_tmp)
    errors.extend(errors_tmp)
```

    Outer loop:   0%|          | 0/19 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/2 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/4 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/3 [00:00<?, ?it/s]

      0%|          | 0/2 [00:00<?, ?it/s]

      0%|          | 0/2 [00:00<?, ?it/s]

      0%|          | 0/2 [00:00<?, ?it/s]

    CPU times: user 16h 51min 12s, sys: 1d 13h 56min 24s, total: 2d 6h 47min 36s
    Wall time: 7h 9min 40s

#hide

Run the following cell to save the results.

```
#hide
# results = {'h':h_range,'ratios':errors}
# with open('2D_posterior_results_full_noise_0.001','wb') as f:
#     pickle.dump(results,f)
```

We now sort these results by $h$.

```
h_range_array = np.array(h_range)
errors_array = np.array(errors)
```

```
argInd = np.argsort(h_range_array)
hs = h_range_array[argInd]
es = errors_array[argInd]
hs,hs_ind = np.unique(hs,return_index=True)
es = es[hs_ind]
```

We now plot the base-2 logarithm of the ratios against $h$ below:

```
#hide_input
plt.plot(hs,np.log2(es),c='r')
plt.xlabel('$h$')
plt.ylabel('$\operatorname{LR}(h)$')
plt.title('Base-2 log plot of the ratios against $h$')
plt.hlines(2.0,hs[0],hs[-1],colors='black',linestyles='--')
plt.xlim(hs[-1],hs[0])
plt.grid()
plt.savefig('2D_posterior_results.png',dpi=300,bbox_inches='tight',facecolor='w')
plt.show()
```

We can see from the above plot that the logarithms seems to be approaching $p=2$ as $h$ gets small just as predicted. However, the results aren't that smooth and they haven't seemed to settle on $p=2$ yet. This can be due to memory constraints meaning we cannot use small enough $h$. We thus smooth the above results by taking a cumulative average of the ratios and then applying $\log_2$. We take the rolling average starting with large $h$. We choose a cut-off point of $h=0.15$, i.e. we discard any results for $h>0.15$.

The results are shown below:

From the smoothed results above we can see more clearly that the ratios are converging to around $p=2$.

```
#hide_input
# get the cumulative average of es:
i = -18
es_trimmed = es[:i]
HS = hs[:i]
pos = np.arange(0,35,step=5)
es_av = np.log2(np.cumsum(es_trimmed[::-1])/np.arange(1,len(es_trimmed)+1))
plt.plot(es_av,c='r')
plt.hlines(2.0,0,len(es_av),colors='black',linestyles='--')
plt.xticks(pos,HS[::-1][pos])
plt.xlabel('index')
plt.title('Base-2 logarithms of the cumulative averages of the ratios for $h\leq 0.15$')
plt.grid()
plt.savefig('2D_posterior_results_smoothed.png',dpi=300,bbox_inches='tight',facecolor='w')
plt.show()
```

From the smoothed results above we can see more clearly that the ratios are converging to around $p=2$. In fact, discarding the values corresponding to $h>0.15$ seems to result in the the rolling average converging to a value slightly greater than $2$.

```

```