# Python: Cluster Robust Double Machine Learning

## Motivation

In many empirical applications, errors exhibit a clustered structure such that the usual i.i.d. assumption does not hold anymore. In order to perform valid statistical inference, researchers have to account for clustering. In this notebook, we will shortly emphasize the consequences of clustered data on inference based on the double machine learning (DML) approach as has been considered in [Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815). We will demonstrate how users of the [DoubleML](https://docs.doubleml.org/stable/index.html) package can account for one- and two-way clustering in their analysis.

Clustered errors in terms of one or multiple dimensions might arise in many empirical applications. For example, in a cross-sectional study, errors might be correlated (i) within regions (one-way clustering) or (ii) within regions and industries at the same time (two-way clustering). Another example for two-way clustering, discussed in [Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815), refers to market share data with market shares being subject to shocks on the market and product level at the same time. We refer to [Cameron et al. (2011)](https://doi.org/10.1198/jbes.2010.07136) for an introduction to multiway clustering and a illustrative list of empirical examples.

## Clustering and double machine learning

Clustering creates a challenge to the double machine learning (DML) approach in terms of

1. a necessary adjustment of the formulae used for estimation of the variance covariance matrix, standard errors, p-values etc., and,
2. an adjusted resampling scheme for the cross-fitting algorithm.

The first point equally applies to classical statistical models, for example a linear regression model (see, for example [Cameron et al. 2011](https://doi.org/10.1198/jbes.2010.07136)). The second point arises because the clustering implies a correlation of errors from train and test samples if the standard cross-fitting procedure suggested in [Chernozhukov et al. (2018)](https://doi.org/10.1111/ectj.12097) was employed. The DML approach builds on independent sample splits into partitions that are used for training of the machine learning (ML) model learners and generation of predictions that are eventually used for solving the score function. For a motivation of the necessity of sample splitting, we refer to the illustration example in the [user guide](
https://docs.doubleml.org/stable/guide/basics.html#sample-splitting-to-remove-bias-induced-by-overfitting) as well as to the explanation in [Chernozhukov et al. (2018)](https://doi.org/10.1111/ectj.12097) .

In order to achieve independent data splits in a setting with one-way or multi-way clustering, [Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815) develop an updated  $K$-fold sample splitting procedure that ensures independent sample splits: The data set is split into disjoint partitions in terms of all clustering dimensions. For example, in a situation with two-way clustering, the data is split into $K^2$ folds. The machine learning models are then trained on a specific fold and used for generation of predictions in hold-out samples. Thereby, the sample splitting procedure ensures that the hold-out samples do not contain observations of the same clusters as used for training.

```python
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
from matplotlib.colors import ListedColormap
import seaborn as sns

from sklearn.model_selection import KFold, RepeatedKFold
from sklearn.base import clone

from sklearn.linear_model import LassoCV

from doubleml import DoubleMLClusterData, DoubleMLData, DoubleMLPLIV

from doubleml.datasets import make_pliv_multiway_cluster_CKMS2021
```

## A Motivating Example: Two-Way Cluster Robust DML

In a first part, we show how the two-way cluster robust double machine learning (DML) ([Chiang et al. 2021](https://doi.org/10.1080/07350015.2021.1895815)) can be implemented with the [DoubleML](https://docs.doubleml.org/stable/index.html) package.
[Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815) consider double-indexed data

\begin{equation}
\lbrace W_{ij}: i \in \lbrace 1, \ldots, N \rbrace, j \in \lbrace 1, \ldots, M \rbrace \rbrace
\end{equation}

and the partially linear IV regression model (PLIV)

$$\begin{aligned}
Y_{ij} = D_{ij} \theta_0 +  g_0(X_{ij}) + \epsilon_{ij}, & &\mathbb{E}(\epsilon_{ij} | X_{ij}, Z_{ij}) = 0, \\
Z_{ij} = m_0(X_{ij}) + v_{ij}, & &\mathbb{E}(v_{ij} | X_{ij}) = 0.
\end{aligned}$$

### Simulate two-way cluster data

We use the PLIV data generating process described in Section 4.1 of [Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815).
The DGP is defined as
$$\begin{aligned}
Z_{ij} &= X_{ij}' \xi_0 + V_{ij}, \\
D_{ij} &= Z_{ij}' \pi_{10} + X_{ij}' \pi_{20} + v_{ij}, \\
Y_{ij} &= D_{ij} \theta + X_{ij}' \zeta_0 + \varepsilon_{ij},
\end{aligned}$$
with
$$\begin{aligned}
X_{ij} &= (1 - \omega_1^X - \omega_2^X) \alpha_{ij}^X
+ \omega_1^X \alpha_{i}^X + \omega_2^X \alpha_{j}^X, \\
\varepsilon_{ij} &= (1 - \omega_1^\varepsilon - \omega_2^\varepsilon) \alpha_{ij}^\varepsilon
+ \omega_1^\varepsilon \alpha_{i}^\varepsilon + \omega_2^\varepsilon \alpha_{j}^\varepsilon, \\
v_{ij} &= (1 - \omega_1^v - \omega_2^v) \alpha_{ij}^v
+ \omega_1^v \alpha_{i}^v + \omega_2^v \alpha_{j}^v, \\
V_{ij} &= (1 - \omega_1^V - \omega_2^V) \alpha_{ij}^V
+ \omega_1^V \alpha_{i}^V + \omega_2^V \alpha_{j}^V,
\end{aligned}$$
and $\alpha_{ij}^X, \alpha_{i}^X, \alpha_{j}^X \sim \mathcal{N}(0, \Sigma)$
where  $\Sigma$ is a $p_x \times p_x$ matrix with entries
$\Sigma_{kj} = s_X^{|j-k|}$.
Further
$$\begin{aligned}
\left(\begin{matrix} \alpha_{ij}^\varepsilon \\ \alpha_{ij}^v \end{matrix}\right),
\left(\begin{matrix} \alpha_{i}^\varepsilon \\ \alpha_{i}^v \end{matrix}\right),
\left(\begin{matrix} \alpha_{j}^\varepsilon \\ \alpha_{j}^v \end{matrix}\right)
\sim \mathcal{N}\left(0, \left(\begin{matrix} 1 & s_{\varepsilon v} \\
s_{\varepsilon v} & 1 \end{matrix} \right) \right)
\end{aligned}$$
and $\alpha_{ij}^V, \alpha_{i}^V, \alpha_{j}^V \sim \mathcal{N}(0, 1)$.

Data from this DGP can be generated with the [make_pliv_multiway_cluster_CKMS2021()](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_pliv_multiway_cluster_CKMS2021.html#doubleml.datasets.make_pliv_multiway_cluster_CKMS2021) function from [DoubleML](https://docs.doubleml.org/stable/index.html).

```python
# Set the simulation parameters
N = 25  # number of observations (first dimension)
M = 25  # number of observations (second dimension)
dim_X = 100  # dimension of X
np.random.seed(3141) # set seed

obj_dml_data = make_pliv_multiway_cluster_CKMS2021(N, M, dim_X)
```

### Data-Backend for Cluster Data
The implementation of cluster robust double machine learning is based on a special data-backend called [DoubleMLClusterData](https://docs.doubleml.org/stable/api/generated/doubleml.DoubleMLClusterData.html#doubleml.DoubleMLClusterData). As compared to the standard data-backend [DoubleMLData](https://docs.doubleml.org/dev/api/generated/doubleml.DoubleMLData.html), users can specify the clustering variables during instantiation of a  [DoubleMLClusterData](https://docs.doubleml.org/stable/api/generated/doubleml.DoubleMLClusterData.html#doubleml.DoubleMLClusterData) object. The estimation framework will subsequently account for the provided clustering options.

```python
# The simulated data is of type DoubleMLClusterData
print(obj_dml_data)
```

```python
# The cluster variables are part of the DataFrame
obj_dml_data.data.head()
```

### Initialize the objects of class `DoubleMLPLIV`

```python
# Set machine learning methods for m, g & r
learner = LassoCV()
ml_g = clone(learner)
ml_m = clone(learner)
ml_r = clone(learner)

# initialize the DoubleMLPLIV object
dml_pliv_obj = DoubleMLPLIV(obj_dml_data,
                            ml_g, ml_m, ml_r,
                            n_folds=3)
```

```python
print(dml_pliv_obj)
```

### Define Helper Functions for Plotting

```python
#discrete color scheme
x = sns.color_palette("RdBu_r", 7)
cMap = ListedColormap([x[0], x[3], x[6]])
plt.rcParams['figure.figsize'] = 15, 12
sns.set(font_scale=1.3)
```

```python
def plt_smpls(smpls, n_folds_per_cluster):
    df = pd.DataFrame(np.zeros([N*M, n_folds_per_cluster*n_folds_per_cluster]))
    for i_split, this_split_ind in enumerate(smpls):
        df.loc[this_split_ind[0], i_split] = -1.
        df.loc[this_split_ind[1], i_split] = 1.

    ax = sns.heatmap(df, cmap=cMap);
    ax.invert_yaxis();
    ax.set_ylim([0, N*M]);
    ax.set_xlabel('Fold')
    ax.set_ylabel('Observation')
    colorbar = ax.collections[0].colorbar
    colorbar.set_ticks([-0.667, 0, 0.667])
    colorbar.set_ticklabels(['Nuisance', '', 'Score'])
```

```python
def plt_smpls_cluster(smpls_cluster, n_folds_per_cluster):
    for i_split in range(len(smpls_cluster)):
        plt.subplot(n_folds_per_cluster, n_folds_per_cluster, i_split + 1)
        df = pd.DataFrame(np.zeros([N*M, 1]),
                      index = pd.MultiIndex.from_product([range(N), range(M)]),
                      columns=['value'])

        df.loc[pd.MultiIndex.from_product(smpls_cluster[i_split][0]), :] = -1.
        df.loc[pd.MultiIndex.from_product(smpls_cluster[i_split][1]), :] = 1.

        df_wide = df.reset_index().pivot(index="level_0", columns="level_1", values="value")
        df_wide.index.name=''
        df_wide.columns.name=''

        ax = sns.heatmap(df_wide, cmap=cMap);
        ax.invert_yaxis();
        ax.set_ylim([0, M]);
        colorbar = ax.collections[0].colorbar
        colorbar.set_ticks([-0.667, 0, 0.667])

        l = i_split % n_folds_per_cluster + 1
        k = np.floor_divide(i_split, n_folds_per_cluster) + 1
        title = f'Nuisance: $I_{{{k}}}^C \\times J_{{{l}}}^C$; Score: $I_{{{k}}} \\times J_{{{l}}}$'
        ax.set_title(title)
        if l == n_folds_per_cluster:
            colorbar.set_ticklabels(['Nuisance', '', 'Score'])
        else:
            colorbar.set_ticklabels(['', '', ''])
        if l == 1:
            ax.set_ylabel('First Cluster Variable $k$')
        if k == 3:
            ax.set_xlabel('Second Cluster Variable $\ell$')
    plt.tight_layout()
```

### Cluster Robust Cross Fitting
A key element of cluster robust DML ([Chiang et al. 2021](https://doi.org/10.1080/07350015.2021.1895815)) is a special sample splitting used for the cross-fitting.
In case of two-way clustering, we assume $N$ clusters in the first dimension and $M$ clusters in the second dimension.

For $K$-fold cross-fitting, [Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815) proposed to randomly partition $[N]:=\{1,\ldots,N\}$ into $K$ subsets $\{I_1, \ldots, I_K\}$ and $[M]:=\{1,\ldots,N\}$ into $K$ subsets $\{J_1, \ldots, J_K\}$.
Effectively, one then considers $K^2$ folds.
Basically for each $(k, \ell) \in \{1, \ldots, K\} \times \{1, \ldots, K\}$, the nuisance functions are estimated for all double-indexed observations in $([N]\setminus I_K) \times ([M]\setminus J_\ell)$, i.e.,
$$
\hat{\eta}_{k\ell} = \hat{\eta}\left((W_{ij})_{(i,j)\in ([N]\setminus I_K) \times ([M]\setminus J_\ell)}\right)
$$
The causal parameter is then estimated as usual by solving a moment condition with a Neyman orthogonal score function.
For two-way cluster robust double machine learning with algorithm [DML2](https://docs.doubleml.org/stable/guide/algorithms.html#algorithm-dml2) this results in solving
$$
\frac{1}{K^2} \sum_{k=1}^{K} \sum_{\ell=1}^{K} \frac{1}{|I_k| |J_\ell|} \sum_{(i,j) \in I_K \times J_\ell}
\psi(W_{ij}, \tilde{\theta}_0, \hat{\eta}_{k\ell}) = 0
$$
for $\tilde{\theta}_0$.
Here $|I_k|$ denotes the cardinality, i.e., the number of clusters in the $k$-th fold for the first cluster variable.

We can visualize the sample splitting of the $N \cdot M = 625$ observations into $K \cdot K = 9$ folds. The following heat map illustrates the partitioned data set that is split into $K=9$ folds. The horizontal axis corresponds to the fold indices and the vertical axis to the indices of the observations. A blue field indicates that the observation $i$ is used for fitting the nuisance part, red indicates that the fold is used for prediction generation and white means that an observation is left out from the sample splitting.

For example, the first observation as displayed on the very bottom of the figure (using Python indexing starting at `0`) is used for training of the nuisance parts in the first (`0`), the third (`2`), fourth (`3`) and sixth (`5`) fold and used for generation of the predictions in fold eight (`7`). At the same time the observation is left out from the sample splitting procedure in folds two (`1`), five (`4`), seven (`6`) and nine (`8`).

```python
plt_smpls(dml_pliv_obj.smpls[0], dml_pliv_obj._n_folds_per_cluster)
```

If we visualize the sample splitting in terms of the cluster variables, the partitioning of the data into $9$ folds $I_k \times J_\ell$ becomes clear.
The identifiers for the first cluster variable $[N]:=\{1,\ldots,N\}$ have been randomly partioned into $K=3$ folds denoted by $\{I_1, I_2, I_3\}$ and the identifiers for the second cluster variable $[M]:=\{1,\ldots,M\}$ have also been randomly partioned into $K=3$ folds denoted by $\{J_1, J_2, J_3\}$.
By considering every combination $I_k \times J_\ell$ for $1 \leq k, \ell \leq K = 3$ we effectively base the cross-fitting on $9$ folds.

We now want to focus on the top-left sub-plot showing the partitioning of the cluster data for the first fold.
The $x$-axis corresponds to the first cluster variable and the $y$-axis to the second cluster variable.
Observations with cluster variables $(i,j) \in I_K \times J_\ell$ are used for estimation of the target parameter $\tilde{\theta}_0$ by solving a Neyman orthogonal score function.
For estimation of the nuisance function, we only use observation where neither the first cluster variable is in $I_K$ nor the second cluster variable is in $J_\ell$, i.e., we use observations indexed by $(i,j)\in ([N]\setminus I_K) \times ([M]\setminus J_\ell)$ to estimate the nuisance functions
$$
\hat{\eta}_{k\ell} = \hat{\eta}\left((W_{ij})_{(i,j)\in ([N]\setminus I_K) \times ([M]\setminus J_\ell)}\right).
$$
This way we guarantee that there are never observations from the same cluster (first and/or second cluster dimension) in the sample for the nuisance function estimation (blue) and at the same time in the sample for solving the score function (red). As a result of this special sample splitting proposed by [Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815), the observations in the score (red) and nuisance (blue) sample can be considered independent and the standard cross-fitting approach for double machine learning can be applied.

```python
plt_smpls_cluster(dml_pliv_obj.smpls_cluster[0], dml_pliv_obj._n_folds_per_cluster)
```

### Cluster Robust Standard Errors
In the abstract base class `DoubleML` the estimation of cluster robust standard errors is implemented for all supported double machine learning models.
It is based on the assumption of a linear Neyman orthogonal score function.
We use the notation $n \wedge m := \min\{n,m\}$.
For the the asymptotic variance of
$\sqrt{\underline{C}}(\tilde{\theta_0} - \theta_0)$ with
$\underline{C} := N \wedge M$
[Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815) then propose the following estimator
$$
\hat{\sigma}^2 = \hat{J}^{-1} \hat{\Gamma} \hat{J}^{-1}
$$
where
$$
\begin{aligned}
\hat{\Gamma} = \frac{1}{K^2} \sum_{(k, \ell) \in[K]^2}
\Bigg[ \frac{|I_k| \wedge |J_\ell|}{(|I_k||J_\ell|)^2}
\bigg(&\sum_{i \in I_k} \sum_{j \in J_\ell} \sum_{j' \in J_\ell}
\psi(W_{ij}; \tilde{\theta}, \hat{\eta}_{k \ell}) \psi(W_{ij'}; \tilde{\theta}_0, \hat{\eta}_{k \ell}) \\
&+ \sum_{i \in I_k} \sum_{i' \in I_k} \sum_{j \in J_\ell}
\psi(W_{ij}; \tilde{\theta}, \hat{\eta}_{k \ell}) \psi(W_{i'j}; \tilde{\theta}_0, \hat{\eta}_{k \ell})
\bigg)
\Bigg]
\end{aligned}$$
and
$$
\begin{aligned}
\hat{J} = \frac{1}{K^2} \sum_{(k, \ell) \in[K]^2} \frac{1}{|I_k||J_\ell|}
\sum_{i \in I_k} \sum_{j \in J_\ell}
\psi_a(W_{ij}; \tilde{\theta}_0, \hat{\eta}_{k \ell}).
\end{aligned}
$$
A $(1-\alpha)$ confidence interval is then given by ([Chiang et al. 2021](https://doi.org/10.1080/07350015.2021.1895815))
$$\begin{aligned}
\left[
\tilde{\theta} \pm \Phi^{-1}(1-\alpha/2) \sqrt{\hat{\sigma}^2 / \underline{C}}
\right]
\end{aligned}
$$
with $\underline{C} = N \wedge M$.

```python
# Estimate the PLIV model with cluster robust double machine learning
dml_pliv_obj.fit()
dml_pliv_obj.summary
```

## (One-Way) Cluster Robust Double Machine Learing

We again use the PLIV data generating process described in Section 4.1 of [Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815).
To obtain one-way clustered data, we set the following weights to zero
$$
\omega_2^X = \omega_2^\varepsilon = \omega_2^v = \omega_2^V = 0.
$$
Again we can simulate this data with [make_pliv_multiway_cluster_CKMS2021()](https://docs.doubleml.org/stable/api/generated/doubleml.datasets.make_pliv_multiway_cluster_CKMS2021.html#doubleml.datasets.make_pliv_multiway_cluster_CKMS2021). To prepare the data-backend for one-way clustering, we only have to alter the `cluster_cols` to be `'cluster_var_i'`.

```python
obj_dml_data = make_pliv_multiway_cluster_CKMS2021(N, M, dim_X,
                                                   omega_X=np.array([0.25, 0]),
                                                   omega_epsilon=np.array([0.25, 0]),
                                                   omega_v=np.array([0.25, 0]),
                                                   omega_V=np.array([0.25, 0]))
```

```python
obj_dml_data.cluster_cols = 'cluster_var_i'
print(obj_dml_data)
```

```python
# Set machine learning methods for m & g
learner = LassoCV()
ml_g = clone(learner)
ml_m = clone(learner)
ml_r = clone(learner)

# initialize the DoubleMLPLIV object
dml_pliv_obj = DoubleMLPLIV(obj_dml_data,
                            ml_g, ml_m, ml_r,
                            n_folds=3)
```

```python
dml_pliv_obj.fit()
dml_pliv_obj.summary
```

## Real-Data Application
As a real-data application we revist the consumer demand example from [Chiang et al. (2021)](https://doi.org/10.1080/07350015.2021.1895815).
The U.S. automobile data of [Berry, Levinsohn, and Pakes (1995)](https://doi.org/10.2307/2171802) is obtained from the `R` package [hdm](https://cran.r-project.org/web/packages/hdm/index.html). In this example, we consider different specifications for the cluster dimensions.

### Load and Process Data

```python
from sklearn.preprocessing import PolynomialFeatures
from rpy2.robjects.packages import importr, data
from rpy2.robjects import r, pandas2ri
pandas2ri.activate()
```

```python
hdm = importr('hdm')
blp_data = data(hdm).fetch('BLP')['BLP'][0]
```

```python
x_cols = ['hpwt', 'air', 'mpd', 'space']
blp_data
```

```python
def construct_iv(blp_data, x_cols=['hpwt', 'air', 'mpd', 'space']):
    n = blp_data.shape[0]
    p = len(x_cols)

    firmid = blp_data['firm.id'].values
    cdid = blp_data['cdid'].values
    id_var = blp_data['id'].values
    X = blp_data[x_cols]

    sum_other = pd.DataFrame(columns=['sum.other.' + var for var in x_cols],
                             index=blp_data.index)
    sum_rival = pd.DataFrame(columns=['sum.rival.' + var for var in x_cols],
                             index=blp_data.index)

    for i in range(n):
        other_ind = (firmid == firmid[i]) & (cdid == cdid[i]) & (id_var != id_var[i])
        rival_ind = (firmid != firmid[i]) & (cdid == cdid[i])
        for j in range(p):
            sum_other.iloc[i, j] = X.iloc[:,j][other_ind].sum()
            sum_rival.iloc[i, j] = X.iloc[:,j][rival_ind].sum()

    return pd.concat((sum_other, sum_rival), axis=1)
```

```python
iv_vars = construct_iv(blp_data, x_cols=['hpwt', 'air', 'mpd', 'space'])
```

```python
poly = PolynomialFeatures(degree=3, include_bias=False)
data_transf = poly.fit_transform(blp_data[x_cols])
x_cols_poly = poly.get_feature_names(x_cols)
data_transf = pd.DataFrame(data_transf, columns=x_cols_poly)
data_transf.index = blp_data.index
```

```python
sel_cols_chiang = list(np.setdiff1d(data_transf.columns,
                                    ['hpwt air mpd', 'hpwt air space',
                                     'hpwt mpd space', 'air mpd space']))
sel_cols_chiang
```

```python
blp_data['log_p'] = np.log(blp_data['price'] + 11.761)

y_col = 'y'
d_col = 'log_p'
cluster_cols = ['model.id', 'cdid']
all_z_cols = ['sum.other.hpwt', 'sum.other.mpd', 'sum.other.space']
z_col = all_z_cols[0]
```

```python
dml_df = pd.concat((blp_data[[y_col] + [d_col] + cluster_cols],
                    data_transf[sel_cols_chiang],
                    iv_vars[all_z_cols]),
                   axis=1)
```

```python
dml_df.shape
```

### Initialize `DoubleMLClusterData` object

```python
dml_data = DoubleMLClusterData(dml_df,
                               y_col=y_col,
                               d_cols=d_col,
                               z_cols=z_col,
                               cluster_cols=cluster_cols,
                               x_cols=sel_cols_chiang)
```

```python
print(dml_data)
```

```python
learner = LassoCV(max_iter=50000)
```

```python
res_df = pd.DataFrame()
n_rep = 10
```

### Two-Way Clustering with Respect to Product and Market

```python
dml_data.z_cols = z_col
dml_data.cluster_cols = ['model.id', 'cdid']
dml_pliv = DoubleMLPLIV(dml_data,
                        clone(learner), clone(learner), clone(learner),
                        n_folds=2, n_rep=n_rep)
dml_pliv.fit()
res = dml_pliv.summary.reset_index(drop=True)
res['z_col'] = dml_data.z_cols[0]
res['clustering'] = 'two-way'
res_df = res_df.append(res)
```

### One-Way Clustering with Respect to the Product

```python
dml_data.z_cols = z_col
dml_data.cluster_cols = 'model.id'
dml_pliv = DoubleMLPLIV(dml_data,
                        clone(learner), clone(learner), clone(learner),
                        n_folds=4, n_rep=n_rep)
dml_pliv.fit()
res = dml_pliv.summary.reset_index(drop=True)
res['z_col'] = dml_data.z_cols[0]
res['clustering'] = 'one-way-product'
res_df = res_df.append(res)
```

### One-Way Clustering with Respect to the Market

```python
dml_data.z_cols = z_col
dml_data.cluster_cols = 'cdid'
dml_pliv = DoubleMLPLIV(dml_data,
                        clone(learner), clone(learner), clone(learner),
                        n_folds=4, n_rep=n_rep)
dml_pliv.fit()
res = dml_pliv.summary.reset_index(drop=True)
res['z_col'] = dml_data.z_cols[0]
res['clustering'] = 'one-way-market'
res_df = res_df.append(res)
```

### No Clustering / Zero-Way Clustering

```python
dml_data = DoubleMLData(dml_df,
                        y_col=y_col,
                        d_cols=d_col,
                        z_cols=z_col,
                        x_cols=sel_cols_chiang)
```

```python
print(dml_data)
```

```python
dml_data.z_cols = z_col
dml_pliv = DoubleMLPLIV(dml_data,
                        clone(learner), clone(learner), clone(learner),
                        n_folds=4, n_rep=n_rep)
dml_pliv.fit()
res = dml_pliv.summary.reset_index(drop=True)
res['z_col'] = dml_data.z_cols[0]
res['clustering'] = 'zero-way'
res_df = res_df.append(res)
```

### Application Results

```python
res_df
```

## References
Berry, S., Levinsohn, J., and Pakes, A. (1995), Automobile Prices in Market
Equilibrium, Econometrica: Journal of the Econometric Society, 63, 841-890, doi: [10.2307/2171802](https://doi.org/10.2307/2171802).

Cameron, A. C., Gelbach, J. B. and Miller, D. L. (2011), Robust Inference with Multiway Clustering, Journal of Business & Economic Statistics, 29:2, 238-249, doi: [10.1198/jbes.2010.07136](https://doi.org/10.1198/jbes.2010.07136).

Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W. and Robins, J. (2018), Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21: C1-C68, doi: [10.1111/ectj.12097](https://doi.org/10.1111/ectj.12097).

Chiang, H. D., Kato K., Ma, Y. and Sasaki, Y. (2021), Multiway Cluster Robust Double/Debiased Machine Learning, Journal of Business & Economic Statistics, doi: [10.1080/07350015.2021.1895815](https://doi.org/10.1080/07350015.2021.1895815), arXiv: [1909.03489](https://arxiv.org/abs/1909.03489).


```python
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import display, HTML, IFrame, YouTubeVideo
from ipywidgets import interact,fixed
import pandas as pd
from numpy import cos,sin,pi,tan,log,exp,sqrt,array,linspace,arange
from mpl_toolkits import mplot3d
# from mpl_toolkits.mplot3d.art3d import Poly3DCollection

from ipywidgets import interact

plt.rcParams["figure.figsize"] = [7,7]

from numpy.linalg import norm

%matplotlib inline

# Uncomment the one that corresponds to your Jupyter theme
plt.style.use('dark_background')
# plt.style.use('fivethirtyeight')
# plt.style.use('Solarize_Light2')
```

$\renewcommand{\vec}{\mathbf}$

### Exercises

  1. Wheat production $W$ in a given year depends on the average temperature $T$ and the annual rainfall $R$. Scientists estimate that the average temperature is rising at a rate of $0.15^\circ$C/year and rainfall is decreasing at a rate of $0.1$ cm/year. They also estimate that at current production levels, $\partial W/\partial T = -2$ and $\partial W/\partial R = 8$.

    1. What is the significance of the signs of these partial derivatives?

    As temperature goes up, wheat production decreases. More rain, on the other hand, more wheat.

    2. Estimate the current rate of change of wheat production, $dW/dt$.

$$\frac{dW}{dt} = \frac{\partial W}{\partial T}\frac{dT}{dt} + \frac{\partial W}{\partial R}\frac{dR}{dt} = -2(0.15) + 8(-0.1) = -1.1 \text{ wheats} / \text{year}$$

  2. Suppose
\begin{align}
z &= z(x,y) \\
x &= x(u,v) \\
y &= y(u,v) \\
u &= u(s,t) \\
v &= v(s,t) \\
\end{align}
are all differentiable. Find an expression for $\frac{\partial z}{\partial s}$.

$$ \frac{\partial z}{\partial s} = \frac{\partial z}{\partial x}\frac{\partial x}{\partial u}\frac{\partial u}{\partial s}
+ \frac{\partial z}{\partial x}\frac{\partial x}{\partial v}\frac{\partial v}{\partial s}
+ \frac{\partial z}{\partial y}\frac{\partial y}{\partial u}\frac{\partial u}{\partial s}
+ \frac{\partial z}{\partial y}\frac{\partial y}{\partial v}\frac{\partial v}{\partial s} $$

## Example

If $g:\RR\to\RR$ is any smooth function, show that $f(x,y) = g(x^2+y^2)$ is radially symmetric. That is, $\frac{\partial f}{\partial \theta} =0$

$$\frac{\partial f }{\partial \theta} = \frac{\partial}{\partial x} (g(x^2 + y^2)) \frac{\partial x}{\partial \theta}
+ \frac{\partial}{\partial y} (g(x^2 + y^2)) \frac{\partial y}{\partial \theta} $$

$$ = g'(x^2 + y^2)2x (-r \sin \theta) + g'(x^2 + y^2)2y (-r \cos \theta) $$

$$ = g'(x^2 + y^2)( -2xy + 2yx) = 0 $$

<p style="padding-bottom:40%;"> </p>

## Example

Find the slope of the tangent line to

$$ x \sin(y) - \frac12 = \sqrt{2} - 2\cos(xy)$$ at the point $\left(\frac12,\frac\pi2\right)$.

```python
x = y = np.linspace(-pi,pi,102)
x,y = np.meshgrid(x,y)
z = x*sin(y) + 2*cos(x*y) - sqrt(2) - 1/2
plt.figure(figsize=(7,7))

cp = plt.contour(x,y,z,levels=arange(-3,3.4,.5),alpha=.5,colors='y')

cp = plt.contour(x,y,z,levels=[0],colors='y')
# plt.clabel(cp,fmt="%d");
x = np.linspace(-2.5,3.5,102)
plt.plot(x,pi/2 + (x-1/2) * (sqrt(2) - pi),color='r');
plt.grid(True)
plt.scatter(1/2,pi/2)
plt.xlim(-pi,pi)
plt.ylim(-pi,pi);
```

$$F(x,y) = x \sin y + 2\cos(xy) = \frac12 + \sqrt 2 $$

$$ \frac{dy}{dx} = \left.-\frac{F_x}{F_y} \right\rvert_{(1/2,\pi/2)} = \left.-\frac{\sin y -2\sin(xy)y}{x\cos y -2 \sin(xy)x}\right\rvert_{(1/2,\pi/2)} $$

$$ = - \frac{1 - \frac{\pi}{\sqrt2}}{-\frac{1}{\sqrt2}} = \sqrt2 - \pi $$

### Example

Differentiate the function $$f(t) = \int_0^t e^{-tx^2}dx.$$

**Solution** This is a funny example as it is ostensibly a one-variable calculus problem. $x$ is just a dummy variable so the only variable to differentiate here is $t$, but you are not likely to find this example in a Calculus 1 text.

```python
@interact
def _(t = (0.,3.,0.05)):
    x = np.linspace(0,3,200)
    plt.plot(x,exp(-x**2),label = "$e^{-x^2}$")
    plt.plot(x,exp(-t*x**2),label = "$e^{-tx^2}$")
    y = np.array([0] + list(np.linspace(0,t,150)) + [t])
    z = exp(-t*y**2)
    z[0] = 0
    z[-1] = 0
    plt.fill(y,z)

    plt.legend();
```

    interactive(children=(FloatSlider(value=1.5, description='t', max=3.0, step=0.05), Output()), _dom_classes=('w…

We cannot only apply the Fundamental Theorem of Calculus here directly as $t$ appears in both the limits and the integrand. So instead, we define

$$F(a,b) = \int_0^a e^{-bx^2}dx$$

to separate those roles and then realize $f(t) = F(t,t)$ so we apply the chain rule

$$f'(t) = F_a(t,t) + F_b(t,t)$$ where of course here $\frac{da}{dt} = 1 = \frac{db}{dt}$. The first partial is computed via FTC and the second by differentiating under the integral sign. And thus,

$$f'(t) = e^{-t^3} + \int_0^t (-x^2)e^{-tx^2}\,dx $$

which is not beautiful but can be evaluated to arbitrary precision.

```python
from scipy.integrate import quad

def fprime(t):
    val = quad(lambda x: (-x**2)*exp(-t*x**2),0,t)[0]
    return exp(-t**3) + val

fprime(1)
```

    0.17840709535094998

```python
t = np.linspace(0,3,200)
plt.figure(figsize=(8,8))
plt.plot(t, [fprime(tt) for tt in t],label="$df/dt$")
plt.plot(t, [quad(lambda x: exp(-tt*x**2),0,tt)[0] for tt in t],label="$f$")
plt.legend();
plt.plot(t, 0*t);
```

```python

```


## PRESCAL Updating Design

Original work: https://github.com/dongwookim-ml/almc
Paper: https://arxiv.org/pdf/1608.05921.pdf

To speed up the algorithm, we re-design the updating steps.

We consider:

1. We might do not need to update posteriors for all entities and relations.

2. For a sequential updating algorithm (Thompson sampling), it doesn't make sense to use all observed labels in each iteration. i.e.

$$P_t = l({x}_t) P_{t-1}$$
where $x_t$ is the label observed in $t^{th}$ iteration.

Based on the above consideration, we come up with the following design ideas:

### Design 1

Assume we observe $x_{ijk}$ in $t^{th}$ iteration, we only update the posterior of $e_i, e_j, r_k$ using the new label $x_{ijk}$.

$\textbf{Prior}$:

$$P(\mathbf{e_i}|\sigma_e) = \mathcal{N}(\mathbf{e_i}| \mathbf{u_e}, {\sigma_e}^2 I_D)$$
$$P(\mathbf{R_k}|\sigma_r) = \mathcal{MN}(\mathbf{R_k}| \mathbf{u_r}, {\sigma_r} I_D, {\sigma_r} I_D)$$
or eqivalently,
$$P(\mathbf{r_k}|\sigma_r) = \mathcal{N}(\mathbf{r_k}| \mathbf{u_r}, {\sigma_r}^2 I_{D^2})$$
where $r_k = vec(R_k) \in \mathcal{R}^{D^2,1}$

$\textbf{Likelihood}$:

$$p(x_{ikj}|\mathbf{e_i, e_j}, R_k) = \mathcal{N}(x_{ikj}| \mathbf{e_i}^T R_k \mathbf{e_j}, \sigma_x^2)$$
using the identity $\mathbf{e_i}^T R_k \mathbf{e_j} = r_k^T \mathbf{e_i} \otimes \mathbf{e_j}$,
$$p(x_{ikj}|\mathbf{e_i, e_j, r_k}) = \mathcal{N}(x_{ikj}| \mathbf{r_k}^T \mathbf{e_i} \otimes \mathbf{e_j}, \sigma_x^2)$$

$\textbf{Entity Posterior}$:

$$P(\mathbf{e_i}|x_{ikj}, \mathbf{e_j}, R_k, \sigma_e) = \mathcal{N}(\mathbf{e_i}| m_{eN}, s_{eN}) \propto P(\mathbf{e_i}|\sigma_e)P(x_{ikj}|\mathbf{e_i, e_j}, R_k) = \mathcal{N}(\mathbf{e_i}| \mathbf{u_e}, {\sigma_e}^2 I_D) \mathcal{N}(x_{ikj}| \mathbf{e_i}^T R_k \mathbf{e_j}, \sigma_x^2)$$

We know for $c \mathcal{N}(\mathbf{x|c, C}) = \mathcal{N}(\mathbf{x|a, A})\mathcal{N}(\mathbf{x|b, B})$,

\begin{equation}
\mathbf{C = {(A^{-1} + B ^{-1)})}^{-1}}
\end{equation}
$$\mathbf{c = C(A^{-1}a + B^{-1}b)}$$

So the goal is to transform $\mathcal{N}(x_{ikj}| r_k^T \mathbf{e_i} \otimes \mathbf{e_j}, \sigma_x^2)$ into $\mathcal{N}(\mathbf{e_i}|  M x_{ikj}, \sigma_x^2 MM^T)$

Assume $R_k \mathbf{e_j}$ is column full rank,
$$x_{ikj} = \mathbf{e_i^T}R_k\mathbf{e_j} \Leftrightarrow \mathbf{e_i} = (R_k \mathbf{e_j})^{-T}x_{ikj}$$

$$\mathcal{N}(\mathbf{e_i}|  M x_{ikj}, \sigma_x^2 MM^T) = \mathcal{N}(\mathbf{e_i}| (R_k \mathbf{e_j})^{-T}x_{ikj}, \sigma_x^2 ((R_k \mathbf{e_j})(R_k \mathbf{e_j})^T)^{-1})$$

$$s_{eN} = (\sigma_e^{-2} I_D +  \sigma_x^{-2} (R_k \mathbf{e_j})(R_k \mathbf{e_j})^T)^{-1}$$

$$m_{eN} = s_{eN} (\sigma_e^{-2} \mathbf{u_e} +  \sigma_x^{-2} (R_k \mathbf{e_j}) x_{ikj} )$$

Similarly, assum $\mathbf{e_i}^T R_k$ is column full rank, for $P(e_j|x_{ikj}, \mathbf{e_i}, R_k, \sigma_e)$ we have

$$s_{eN} = (\sigma_e^{-2} I_D +  \sigma_x^{-2} (\mathbf{e_i}^T R_k)^T(\mathbf{e_i}^T R_k))^{-1}$$

$$m_{eN} = s_{eN} (\sigma_e^{-2} \mathbf{u_e} +  \sigma_x^{-2} (\mathbf{e_i}^T R_k)^T x_{ikj} )$$

$\textbf{Relation Posterior}:$

$$P(\mathbf{r_k}|x_{ikj}, \mathbf{e_i, e_j}, \sigma_r) = \mathcal{N}(\mathbf{r_k}|m_{rN}, s_{rN}) \propto P(\mathbf{r_k|\sigma_r})P(x_{ikj}|\mathbf{e_i, e_j, r_k}) = \mathcal{N}(\mathbf{r_k}| \mathbf{u_r}, {\sigma_r}^2 I_{D^2}) \mathcal{N}(x_{ikj}| \mathbf{r_k}^T \mathbf{e_i} \otimes \mathbf{e_j}, \sigma_x^2)$$

Similarly, assume $\mathbf{e_i} \otimes \mathbf{e_j}$ is column full rank,

$$ x_{ikj} = \mathbf{r_k}^T \mathbf{e_i} \otimes \mathbf{e_j} \Leftrightarrow \mathbf{r_k} = (\mathbf{e_i} \otimes \mathbf{e_j}) ^{-T} x_{ikj}$$

$$\mathcal{N}(\mathbf{r_k}|M x_{ikj}, \sigma^2 MM^T) = \mathcal{N}(\mathbf{r_k}| (\mathbf{e_i} \otimes \mathbf{e_j}) ^{-T} x_{ikj}, \sigma_x^{2} ((\mathbf{e_i} \otimes \mathbf{e_j}) (\mathbf{e_i} \otimes \mathbf{e_j}) ^T)^{-1} )$$

$$s_{rN} = (\sigma_r^{-2}I_D + \sigma_x^{-2} (\mathbf{e_i} \otimes \mathbf{e_j}) (\mathbf{e_i} \otimes \mathbf{e_j}) ^T)^{-1}$$

$$m_{rN} = s_{rN}(\sigma_r^{-2} \mathbf{u_r} + \sigma_x^{-2} (\mathbf{e_i} \otimes \mathbf{e_j}))$$

### Design 2

```python

```


## Smith-Hutton Problem

We want to solve the following PDE:

\begin{equation}
    u \frac{\partial \phi}{\partial x} + v \frac{\partial \phi}{\partial y} = 0
\end{equation}

The independen variables (i.e, $x$, $y$) are used as input values for the NN, and the solution (i.e. $\phi(x,y)$) is the output. In order to find the solution, at each step the NN outputs are derived w.r.t the inputs. Then, a loss function that matches the PDE is built and the weights are updated accordingly. If the loss function goes to zero, we can assume that our NN is indeed the solution to our PDE. We will try to find a general solution for different values of $u$, so it will be set also as an input. The geometry of the problem is as follows:

```
# autoreload nangs
%reload_ext autoreload
%autoreload 2
%matplotlib inline
```

```
#imports
import math
import numpy as np
import matplotlib.pyplot as plt
import torch
```

First we define our PDE and set the values for training.

```
from nangs.pde import PDE
from nangs.bocos import PeriodicBoco, DirichletBoco, NeumannBoco
from nangs.solutions import MLP

class MyPDE(PDE):
    def __init__(self, inputs=None, outputs=None):
        super().__init__(inputs, outputs)
    def computePDELoss(self, grads, inputs, outputs, params):
        dpdx, dpdy = grads['p']['x'], grads['p']['y']
        x, y = inputs['x'], inputs['y']
        u, v = 2*y*(1-x**2), -2*x*(1-y**2)
        return [u*dpdx + v*dpdy]

# instanciate pde
pde = MyPDE(inputs=['x', 'y'], outputs=['p'])

# define input values
x = np.linspace(-1,1,60)
y = np.linspace(0,1,30)

pde.setValues({'x': x[1:-1], 'y': y[1:-1]})
```

Boundary conditions.

```
# left and rigth b.c
ALPHA = 10
x1 = np.array([-1, 1])
p = np.zeros(2*len(y))
for i in range(2*len(y)): p[i] = 1. - math.tanh(ALPHA)
boco = DirichletBoco('left_rigth', {'x': x1, 'y': y}, {'p': p})
pde.addBoco(boco)

# top b.c
p = np.zeros(len(x))
for i in range(len(x)): p[i] = 1. - math.tanh(ALPHA)
boco = DirichletBoco('top', {'x': x, 'y': y[-1:]}, {'p': p})
pde.addBoco(boco)

# bottom b.c
x1, x2, p1 = [], [], []
for i in range(len(x)):
    if x[i] < 0:
        x1.append(x[i])
        p1.append(1.+math.tanh(ALPHA*(2*x[i]+1)))
    else:
        x2.append(x[i])
x1, x2, p1 = np.array(x1), np.array(x2), np.array(p1)

# bottom left b.c
boco = DirichletBoco('bottom_left', {'x': x1, 'y': y[:1]}, {'p': p1})
pde.addBoco(boco)

# bottom right b.c
boco = NeumannBoco('bottom_right', {'x': x2, 'y': y[:1]}, grads={'p': 'y'})
pde.addBoco(boco)
```

```
plt.plot(x1,p1)
plt.xlabel('x')
plt.ylabel('p      ', rotation=np.pi/2)
plt.title('Solution at y=0')
plt.grid()
plt.show()
```

The objective is to reproduce the inflow ($y$=0 and $x<0$) at the outlet ($y$=0 and $x>0$).

Now we define a topology for our solution and set the training parameters. Then we can find a solution for our PDE.

```
# define solution topology
mlp = MLP(pde.n_inputs, pde.n_outputs, 5, 2048)
optimizer = torch.optim.Adam(mlp.parameters(), lr=3e-4)
pde.compile(mlp, optimizer)
```

```
# find the solution
hist = pde.solve()
```

Epoch 1/30 Losses 0.35488
 PDE  [ 0.03867 ]
 left_rigth 0.01197
 top 0.00970
 bottom_left 0.27590
 bottom_right 0.01865
Epoch 2/30 Losses 0.04847
 PDE  [ 0.01998 ]
 left_rigth 0.00002
 top 0.00002
 bottom_left 0.02552
 bottom_right 0.00294
Epoch 3/30 Losses 0.02898
 PDE  [ 0.01335 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.01449
 bottom_right 0.00111
Epoch 4/30 Losses 0.02053
 PDE  [ 0.01016 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.00727
 bottom_right 0.00308
Epoch 5/30 Losses 0.01547
 PDE  [ 0.00805 ]
 left_rigth 0.00002
 top 0.00001
 bottom_left 0.00539
 bottom_right 0.00200
Epoch 6/30 Losses 0.01908
 PDE  [ 0.00914 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.00651
 bottom_right 0.00339
Epoch 7/30 Losses 0.01439
 PDE  [ 0.00614 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.00542
 bottom_right 0.00280
Epoch 8/30 Losses 0.00852
 PDE  [ 0.00485 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00249
 bottom_right 0.00117
Epoch 9/30 Losses 0.00758
 PDE  [ 0.00421 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.00219
 bottom_right 0.00117
Epoch 10/30 Losses 0.02043
 PDE  [ 0.00952 ]
 left_rigth 0.00005
 top 0.00005
 bottom_left 0.00928
 bottom_right 0.00154
Epoch 11/30 Losses 0.03349
 PDE  [ 0.01048 ]
 left_rigth 0.00002
 top 0.00002
 bottom_left 0.02116
 bottom_right 0.00182
Epoch 12/30 Losses 0.00724
 PDE  [ 0.00399 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00175
 bottom_right 0.00150
Epoch 13/30 Losses 0.00572
 PDE  [ 0.00308 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00121
 bottom_right 0.00143
Epoch 14/30 Losses 0.00490
 PDE  [ 0.00314 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00110
 bottom_right 0.00065
Epoch 15/30 Losses 0.00719
 PDE  [ 0.00439 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.00205
 bottom_right 0.00074
Epoch 16/30 Losses 0.00584
 PDE  [ 0.00312 ]
 left_rigth 0.00000
 top 0.00001
 bottom_left 0.00155
 bottom_right 0.00115
Epoch 17/30 Losses 0.00584
 PDE  [ 0.00343 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00130
 bottom_right 0.00110
Epoch 18/30 Losses 0.00854
 PDE  [ 0.00421 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.00262
 bottom_right 0.00169
Epoch 19/30 Losses 0.00347
 PDE  [ 0.00220 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00085
 bottom_right 0.00042
Epoch 20/30 Losses 0.00486
 PDE  [ 0.00302 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00118
 bottom_right 0.00066
Epoch 21/30 Losses 0.00503
 PDE  [ 0.00299 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00123
 bottom_right 0.00081
Epoch 22/30 Losses 0.00625
 PDE  [ 0.00415 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00166
 bottom_right 0.00044
Epoch 23/30 Losses 0.00420
 PDE  [ 0.00241 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00096
 bottom_right 0.00083
Epoch 24/30 Losses 0.00993
 PDE  [ 0.00554 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.00324
 bottom_right 0.00114
Epoch 25/30 Losses 0.00462
 PDE  [ 0.00239 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00115
 bottom_right 0.00108
Epoch 26/30 Losses 0.00392
 PDE  [ 0.00243 ]
 left_rigth 0.00001
 top 0.00000
 bottom_left 0.00091
 bottom_right 0.00057
Epoch 27/30 Losses 0.00488
 PDE  [ 0.00269 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00126
 bottom_right 0.00093
Epoch 28/30 Losses 0.00635
 PDE  [ 0.00322 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.00205
 bottom_right 0.00106
Epoch 29/30 Losses 0.00571
 PDE  [ 0.00355 ]
 left_rigth 0.00000
 top 0.00000
 bottom_left 0.00153
 bottom_right 0.00063
Epoch 30/30 Losses 0.00748
 PDE  [ 0.00333 ]
 left_rigth 0.00001
 top 0.00001
 bottom_left 0.00207
 bottom_right 0.00207

    /opt/conda/lib/python3.7/site-packages/numpy/core/fromnumeric.py:3257: RuntimeWarning: Mean of empty slice.
      out=out, **kwargs)
    /opt/conda/lib/python3.7/site-packages/numpy/core/_methods.py:161: RuntimeWarning: invalid value encountered in double_scalars
      ret = ret.dtype.type(ret / rcount)

```
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,5))
ax1.plot(hist['train_loss'], label="train_loss")
ax1.grid(True)
ax1.legend()
ax1.set_yscale("log")
for boco in pde.bocos:
    ax2.plot(hist['bocos'][boco.name], label=boco.name)
ax2.legend()
ax2.grid(True)
ax2.set_yscale("log")
plt.show()
```

Finally, we can evaluate our solution.

```
# evaluate the solution
x = np.linspace(-1,1,100)
y = np.linspace(0,1,50)
pde.evaluate({'x': x, 'y': y})
p = pde.outputs['p']
plt.imshow(p.reshape((len(y),len(x))), vmin=p.min(), vmax=p.max(), origin='lower',
           extent=[x.min(), x.max(), y.min(), y.max()])
plt.colorbar()
plt.show()
```

```
x = np.linspace(-1,1,200)
y = np.array([0])
pde.evaluate({'x': x, 'y': y})
p = pde.outputs['p']
plt.plot(x1,p1)
plt.plot(x1+1,np.flip(p1), label="exact")
plt.plot(x, p, '.k', label="predicted")
plt.legend()
plt.xlabel('x')
plt.ylabel('p      ', rotation=np.pi/2)
plt.title('Solution at y=0')
plt.grid()
plt.show()
```

```

```

```

```


# INF-495, v0.01, Claudio Torres, ctorres@inf.utfsm.cl. DI-UTFSM
## Textbook: Computational Mathematical Modeling, An Integrated Approach Across Scales by Daniela Calvetti and Erkki Somersalo.

# Chapter 1

```python
import numpy as np
import scipy.sparse.linalg as sp
import sympy as sym
from scipy.linalg import toeplitz

import ipywidgets as widgets
from ipywidgets import IntSlider

import matplotlib.pyplot as plt
%matplotlib inline
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter

plt.style.use('ggplot')

import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
sym.init_printing()

from scipy.integrate import odeint
from ipywidgets import interact
```

## Hints:
I strongly suggest to review the following jupyter notebook:
- [Intro to ODE] https://github.com/tclaudioe/Scientific-Computing/blob/master/SC2/U3_ODE.ipynb

## 1.2 Ordinary differential equations

A differenetial equation, in this case a scalar initial value problem (IVP), can be defined as follows:
\begin{align*}
\dfrac{\mathrm{d}x}{\mathrm{d}t}(t) &= f(t,x(t)),\\
x(0) &= x_0.
\end{align*}
For a known function $f(t,x)$ and $x_0$.
Particularly one could integrate both sides of the first equation as follows,
\begin{align*}
\int_0^{h} \dfrac{\mathrm{d}x}{\mathrm{d}t}(t)\mathrm{d}t & = \int_0^{h} f(t,x(t))\,\mathrm{d}t,\\
x(h)-x(0) &= \int_0^{h} f(t,x(t))\,\mathrm{d}t,
\end{align*}
where the Fundamental Theorem of Calculus was used.
Now, we solve for the unknow value $x(h)$:
\begin{align*}
x(h) &= x_0 + \int_0^{h} f(t,x(t))\,\mathrm{d}t.
\end{align*}
Up to this point, we have not made any approximation.
From this point, one can derive several numerical method, depending how you approximate the integral.
For instance, if we consider $h=\Delta t$ and we use a Reimann sum from the left (only one interval from $0$ to $\Delta t$), we obtain the method called Forward Euler.
But if we do the same but with the Reimann sum from the right, we obtain backward Euler.
Another two options that are easy to see are the midpoint rule and the trapezoidal rule.
Notice that in some cases mentioned, you would need a non-linear equation (or a system in higher dimension) for each time step.

### Example 1

Considers the following IVP:
\begin{align*}
\dfrac{\mathrm{d}x}{\mathrm{d}t}(t) &= -\alpha\,x,\\
x(0) &= x_0,
\end{align*}
for which we know the solution:
\begin{align*}
x(t) &= x_0\,\exp(-\alpha\,t)
\end{align*}

```python
# Warning: In general, we will use \dot{y}=f(t,y), y(0)=y_0,
# so don't get mixed up with the notation from the textbook.

def f_example_1_interact(alpha_input=1,T_max=5,p=2):
    # Example 1, numerically.
    def f_example_1(t, y, alpha):
        return -alpha*y

    # Initial condition
    y0=1
    # time where we want your solution
    t = np.linspace(0, T_max, 100)

    sol = odeint(f_example_1, y0, t, args=(alpha_input,), tfirst=True)

    plt.figure()
    plt.plot(t, sol, 'b', label='y(t)')
    for i in np.arange(1,np.ceil(T_max/(np.log(p)/alpha_input))):
        plt.axvline(x=i*np.log(p)/alpha_input)
        print('t: ',i*np.log(p)/alpha_input,', exp(t): ',np.exp(-alpha_input*i*np.log(p)/alpha_input))
    plt.legend(loc='best')
    plt.xlabel('t')
    plt.ylabel(r'$\exp(-\alpha\,t)$')
    plt.title('What are the red lines?')
    plt.grid(True)
    plt.show()
interact(f_example_1_interact,alpha_input=(0.1,10,0.1),T_max=(0.1,100,0.1),p=(2,10,1))
```

    interactive(children=(FloatSlider(value=1.0, description='alpha_input', max=10.0, min=0.1), FloatSlider(value=…

    <function __main__.f_example_1_interact(alpha_input=1, T_max=5, p=2)>

### Example 2

Considers the following IVP:
\begin{align*}
\dfrac{\mathrm{d}x}{\mathrm{d}t}(t) &= \sqrt{x}\\
x(0) &= 0,
\end{align*}

Recall that $2^{-1074}$ is the smallest number greater than 0 that ``double precision'' can store. See:
- https://github.com/tclaudioe/Scientific-Computing/blob/master/SC1/03_floating_point_arithmetic.ipynb

```python
print(2**-1074,', ',2**-1075)
```

    5e-324 ,  0.0

```python
# Example 2, numerically.
def f_example_2(t,y):
    return np.sqrt(y)

# Initial condition
y0=0.0
# time where we want your solution
t = np.linspace(0, 1, 100)
sol_a = odeint(f_example_2, y0, t, tfirst=True)
sol_b = odeint(f_example_2, y0+2**-1074, t, tfirst=True)

plt.figure()
plt.plot(t, sol_a, 'b', label='y_a(t)')
plt.plot(t, sol_b, 'r', label='y_b(t)')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid(True)
plt.show()
```

### Example 3

Considers the following IVP:
\begin{align*}
\dfrac{\mathrm{d}x}{\mathrm{d}t}(t) &= x^2\\
x(0) &= 1,
\end{align*}
for which we know the solution:
\begin{align*}
x(t) &= \dfrac{1}{1-t}
\end{align*}

```python
def f_example_3_interact(T_max=0.5):
    # Example 3, numerically.
    def f_example_3(t,y):
        return y*y

    # Initial condition
    y0=1
    # time where we want your solution
    t = np.linspace(0, T_max, 100)
    sol = odeint(f_example_3, y0, t, tfirst=True)

    plt.figure()
    plt.plot(t, sol, 'b', label='y(t)')
    plt.legend(loc='best')
    plt.xlabel('t')
    plt.grid(True)
    plt.show()
interact(f_example_3_interact,T_max=(0.1,1,0.001))
```

    interactive(children=(FloatSlider(value=0.5, description='T_max', max=1.0, min=0.1, step=0.001), Output()), _d…

    <function __main__.f_example_3_interact(T_max=0.5)>

### Example 4

See classnotes!

```python
def f_example_4(t,y,alpha):
    return alpha*(1/2-y/(1+y))

# Initial condition
y0=3
# time where we want your solution
t = np.linspace(0, 10, 100)

plt.figure(figsize=(10,8))
for j in np.arange(1,6):
    sol= odeint(f_example_4, y0, t, args=(4/j,), tfirst=True)
    plt.plot(t, sol, label=r'$\xi(t)$'+r', $\tau= $'+str(j))
    plt.axvline(x=j,color='k')
    plt.text(j,0.5,r'$\tau= $'+str(j))
plt.legend(loc='best')
plt.ylabel('Concentration')
plt.xlabel('t')
plt.ylim((0,3))
plt.grid(True)
plt.show()
```

```python
def f_example_4b(t,y,alpha,phi_in):
    return phi_in-alpha*(y/(1+y))

# Initial condition
y0=3
# time where we want your solution
t = np.linspace(0, 1, 100)

plt.figure(figsize=(10,8))
sol= odeint(f_example_4b, y0, t, args=(4,5), tfirst=True)
plt.plot(t, sol, label=r'$\xi(t)$')
plt.legend(loc='best')
plt.ylabel('Concentration')
plt.xlabel('t')
plt.title(r'What happens then if $\phi_{in}\geq\alpha$?')
plt.grid(True)
plt.show()
```

### Example 7, for completeness.

Plotting:
\begin{align*}
f(t) &= \dfrac{t^2}{\pi^2+t^2}\,\sin(5\,t),\quad 0 \leq t \leq 4\,\pi
\end{align*}

```python
t = np.linspace(0,4*np.pi,100)

f = lambda t: (t**2/(np.pi**2+t**2))*np.sin(t)

plt.figure()
plt.plot(t,f(t),'b-',label='f(t)')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid(True)
plt.show()
```

### Example 8, for completeness.

Plotting:
\begin{align*}
f(t,\tau) &= \exp(-t/\tau)\,\sin(2\,\pi\,t),\quad 0 \leq t \leq 2.
\end{align*}

```python
t = np.linspace(0,2,100)

f = lambda t,tau: np.exp(-t/tau)*np.sin(2*np.pi*t)

plt.figure()
plt.plot(t,f(t,1),'b-',label=r'$f(t,\tau=1)$')
plt.plot(t,f(t,0.5),'r--',label=r'$f(t,\tau=0.5)$')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid(True)
plt.show()
```

### Example 9, for completeness.

Plotting:
\begin{align*}
f(t,u) &= u^2\,\dfrac{t}{1+t},\quad 0 \leq t, u \in \mathbb{R}.
\end{align*}

```python
elev_widget = IntSlider(min=0, max=180, step=10, value=40)
azim_widget = IntSlider(min=0, max=360, step=10, value=230)

def example_9_interact(elev=40,azim=230):
    f = lambda t,u: (u**2)*(t/(1+t))

    nt, nu = (100, 50)
    t = np.linspace(0, 5, nt)
    u = np.linspace(-1, 1, nu)
    tt, uu = np.meshgrid(t, u)
    zz = f(tt,uu)

    fig = plt.figure(figsize=(10,10))
    ax  = fig.gca(projection='3d')

    surf = ax.plot_surface(tt,uu,zz, cmap=cm.coolwarm, linewidth=0, antialiased=False)
    ax.view_init(elev,azim)
    ax.set_xlabel('t')
    ax.set_ylabel('u')
    ax.set_zlabel('f')
    plt.grid(True)
    plt.show()
interact(example_9_interact,elev=elev_widget,azim=azim_widget)
```

    interactive(children=(IntSlider(value=40, description='elev', max=180, step=10), IntSlider(value=230, descript…

    <function __main__.example_9_interact(elev=40, azim=230)>

### Example 10, for completeness.

\begin{equation*}
f(t)=
\begin{cases}
t\,\sin(t), & 0\leq t < 2, \\
\dfrac{\sin(\pi\,t)}{t}, & 2 < t.
\end{cases}
\end{equation*}

```python
t = np.linspace(0,4,100)

def f(t):
    if t<=2:
        return t*np.sin(t)
    else:
        return np.sin(np.pi*t)/t

fv=np.vectorize(f, otypes=[np.float64])

plt.figure()
plt.plot(t,fv(t),'b-',label='f(t)')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid(True)
plt.show()
```

### Example 11, for completeness.

Write a function that return $f(t)=\exp(-\alpha\,t^2)$ and $f'(t)=-2\,\alpha\,t\,\exp(-\alpha\,t^2)$.

```python
def f_and_fprime(t,alpha):
    f = lambda t: np.exp(-alpha*t**2)
    fp = lambda t: -2*alpha*t*f(t)
    return np.array([f(t),fp(t)])

print(f_and_fprime(t=1,alpha=1))
```

    [ 0.36787944 -0.73575888]

## Homework

- Plot the output of Example 11.
- Example 12, 13 and 14.
- Solve numerically the following problem and plot the solution:
Let $y_1(t)$ be the number of rabbits at time $t$ and $y_2(t)$ be the number of foxes at time $t$.
Consider they are in an isolated island and the rabiits have unlimited grass to eat and the foxes eat rabbits.
So, if there are not foxes, the population will grow without limits, this meains, $\dot{y_1}(t)=y_1(t)$, this means the numbers of rabbits grow exponentially.
Now, if there are not rabbits, we have $\dot{y_2}(t)=-y_2(t)$, since the population of foxes will decay without food. but, how do the two population could interact?

\begin{align*}
\dot{y_1}(t)&=(1-y_2(t)/\mu_2)\,y_1(t)\\
\dot{y_2}(t)&=-(1-y_1(t)/\mu_1)\,y_2(t)\\
y_1(0)&=400\\
y_2(0)&=100
\end{align*}

donde $\mu_1=300$ y $\mu_2=200$ son constantes de normalización.

```python

```


# Rapport

ETAPES:

a)Partie analyse lineaire:

    1-discrétisation de la structure
    2-création des matrices K  et F
    3-mise en place des conditions aux limites
    4-resolution de K*U=F => obtention de U

b)Partie analyse du flambage

    1-construction de la matrice G
    2-resolution de l'equation (K-lambda*G)*U=0 grace à la méthode puissance itérée
    => obtention de lambda
    3-Vérification de notre coefficient lambda

Dans un premier temps, on a créé un modèle de notre structure en la discrétisant.

Nous avons par un cas simple, soit une poutre.

Nous avons décomposé une poutre en $n_{e}$ segments. Elle est constituée d'**éléments**, separés par $n$ **noeuds** avec pour coordonnées $\{(x_I,y_I)\}_{I=1}^{n}$.

Nous avons ensuite appliqué cette discrétisation à une structure plus complexe, le portique.

# I-Partie analyse du flambement de structure

## 1) Construction de la matrice G

```python
%matplotlib inline
from sympy.interactive import printing
printing.init_printing()
from frame import *
import sympy as sp
import numpy as np
import scipy.sparse as sparse
import scipy.sparse.linalg as linalg
```

Après avoir trouvé la **solution fondamentale** du problème, nous avons commencé la résolution du problème de flambement de notre structure.

La résolution de ce problème consiste à trouver le coefficient multiplicateur de charges, $\lambda$.

Pour cela, on doit résoudre le problème :
$$
(K-\lambda G)U = 0
$$

avec $G$ la **matrice de rigidité géométrique**

On construit $G$ à l'aide la formule suivante:
$$
G^{(e)}_{ij}=-\bar N_0^{(e)}\int_0^LS_{1i}'(s)S_{1j}'(s) ds,\quad i,j=1,\ldots,6
$$
avec $N_0^{(e)}$ la valeur normale de l'effort pour un élément $e$

Nous avons, tout d'abord, récupéré les valeurs des efforts normaux pour chaque élément puis nous les avons mis dans un tableau.

```python
class Frame_Buckling1(LinearFrame):

    def N_local_stress(self,element):

        """
        Returns the normal force of an element.
        """
        Ke= self.K_local()
        Ue= self.U_e_local_coord(element)
        F=Ke*Ue
        N_local = F[3]
        return N_local

    def N_local_stress_tot(self):

        """
        Returns the normal force of all elements.
        """
        Ns=[self.N_local_stress(e)for e in range (self.nelements)]
        return Ns

```

Nous avons utilisé le principe d'héritage pour créer cette classe. En effet, cette classe dépend de la classe utilisée précédemment, $LinearFrame$.

Après avoir récupéré l'effort normal pour chaque élément $e$, nous avons codé 3 méthodes pour obtenir la matrice G.
La première méthode transcrit l'expression de G écrite au-dessus en code, la deuxième fait que G va être exprimé dans le repère global et la dernière méthode assemble le tout pour donner la matrice globale G de notre strucutre.

```python
class Frame_Buckling2(Frame_Buckling1):
    def G_local(self):
        """
        Returns the local geometric stiffness matrix
        """
        L = sp.Symbol('L')
        s = sp.Symbol('s')
        S=self.S()
        Ge=sp.Matrix([[sp.integrate(S[1,i_local].diff(s)*S[1,j_local].diff(s),(s,0,L) )for i_local in range(6)] for j_local in range(6)])
        return Ge

    def G_local_rotated(self):
        """
        Gives the analytical expression of the local geometric stiffness matrix in the global coordinate system
        as a function of the orientation angle alpha
        """
        alpha = sp.Symbol("alpha")
        R = self.rotation_matrix(alpha)
        Ge = R.transpose()*self.G_local()*R
        return Ge

    def assemble_G(self):
        """
        Returns the global geometric stiffness matrix
        """
        Ge = self.G_local_rotated()
        G = np.zeros([self.ndof,self.ndof])
        N0=self.N_local_stress_tot()
        for e in range(self.nelements):
            Gen = -N0[e].subs({'EI': self.EI[e], 'ES': self.ES[e], 'L': self.Ls[e], 'alpha': self.angles[e]})*Ge.subs({'EI': self.EI[e], 'ES': self.ES[e], 'L': self.Ls[e], 'alpha': self.angles[e]})
            for i_local in range(6):
                for j_local in range(6):
                    G[self.dof_map(e, i_local),self.dof_map(e, j_local)] += Gen[i_local,j_local]
        return G

```

 Nous appliquons ensuite à la matrice G les conditions de bords. Nous avons réutilisé le même principe que pour K.

```python
class Frame_Buckling(Frame_Buckling2):
    def bc_apply_G(self,G,blocked_dof):
        for (dof) in enumerate(blocked_dof):
            Gbc = G
            Gbc[dof, :] = 0
            Gbc[:, dof] = 0
            Gbc[dof, dof] = 1
        return Gbc
```

## 2)Validation du modèle numérique

Après avoir créer toutes ces méthodes, nous les avons testé sur un cas simple où nous pouvions comparer le modèle numérique et le modèle théorique, nous avons pour cela choisi la poutre en encastrement d'un côté et en appui simple glissant de l'autre soumise à une force de 1N suivant x.

Vous trouverez ci-dessous l'analyse linéaire :

```python
E=1.3 #en MPa
h=7.5 #en mm
b=20. #en mm
I=b*(h**3)/12 #en mm^4
S=b*h  #en mm^2

n_elements = 10
xnodes = np.linspace(0,1000,n_elements + 1)
ynodes = np.linspace(0,0,n_elements + 1)
nodes = np.array([xnodes,ynodes]).transpose()
n_nodes = xnodes.size
elements=np.array([[0,1],[1,2],[2,3],[3,4],[4,5],[5,6],[6,7],[7,8],[8,9],[9,10]])

frame= Frame_Buckling(nodes,elements)
frame.plot_with_label()

ne = frame.nelements
ndof = frame.ndof
EI = np.ones(ne)*E*I
ES = np.ones(ne)*E*S
f_x = 0*np.ones(ne)
f_y = 0*np.ones(ne)
frame.set_distributed_loads(f_x, f_y)
frame.set_stiffness(EI, ES)
blocked_dof = np.array([0, 1, ndof-2])
bc_values = np.array([0, 0, 0])

K = frame.assemble_K()
F=frame.assemble_F()
#F[12]=F[12]-.5*EI[0]*np.pi**2
F[ndof-3]=F[ndof-3]-1.
Kbc, Fbc = frame.bc_apply(K, F, blocked_dof, bc_values)
Usol = np.linalg.solve(Kbc,Fbc)

frame.set_displacement(Usol)
frame.plot_with_label()
frame.plot_displaced()
```

Voici l'analyse du flambement de cette structure :

```python
Gbc=frame.assemble_G()
G=frame.bc_apply_G(Gbc,blocked_dof)
```

Nous avons appris lors de ces séances de projet que pour résoudre le problème $(K-\lambda G)U = 0$ il fallait obtenir les valeurs propres de la matrice $G^{-1}K$. Nous avons pour cela deux méthodes de résolution numériques, la méthode la puissance itérée et l'algorithme QR. Or, dans notre cas, la matrice G est singulière. Il nous est donc impossible de calculer un inverse.

Pour contourner ce problème, nous avons utilisé la bibliothèque sparse présente dans scipy.
Dans cette bibliothèque, il y a une fonction qui permet d'avoir les valeurs propres à partir d'une certaine sans sprendre en compte le caractère singulier de G.

Voici la résolution du problème aux valeurs propres:

```python
Ks = sparse.csr_matrix(K)
Gs = sparse.csr_matrix(G)
val, vect = linalg.eigsh(Ks, 5, Gs, which = 'LA', sigma =4.)
print(val)
frame.set_displacement(10*vect[:,0])
frame.plot_with_label()
frame.plot_displaced()
```

Comme vous pouvez le constatez, la première valeur propre obtenue correspond aussi au premier coefficient multiplicateur de charges, $\lambda_{1}$. On a donc $\lambda_{1num}=8,7.10^{-3}$

Nous avons utilisé ce que nous avons vu en cours pour calculer la valeur théorique de $\lambda$ en utilisant les mêmes conditions que pour le modèle numérique.

```python
E*I*np.pi**2/1000**2
```

Nous avons obtenu que $\lambda_{1théo}=9.10^{-3}$.

Nous avons pu constater que notre valeur théorique et notre valeur numérique était semblable aux approximations près. Nous pouvons donc conclure que notre modèle numérique est bon et qu'il peut être utilisé pour calculer les structures qui nous intéressse.

## 3)Comparaison de notre modèle numérique avec notre modèle expérimentale

Après avoir validé notre modèle numérique, nous l'avons appliqué aux deux structures de l'exercice 2, soit le portique et le portique renforcé. Voici la discrétisation du problème :

```python
Lx=55. #en mm
Lyh=60. #en mm
Lyb=45. #en mm
nodes= np.array([[0.,0.],[0.,Lyb],[0.,Lyh+Lyb],[Lx/2,Lyh+Lyb],[Lx,Lyh+Lyb],[Lx,Lyb],[Lx,0.]])
elements=np.array([[0,1],[1,5],[1,2],[2,3],[3,4],[4,5],[5,6]])

frame= Frame_Buckling(nodes,elements)
frame.plot_with_label()
```

Pour le portique renforcé, nous avons appliqué une force de $1N$ au noeud 3 en y  et nous avons modifié le module de Young E des éléments 1,3 et 4 en le multipliant par 100.

Pour le portique, nous avons appliqué deux forces de $1N$ au noeud 2 et 4 en y.

Nous avons utilisé ensuite le même principe de résolution que pour le 2).

Pour le portique renforcé, nous avons obtenu un $\lambda_{num1}=4,0$.

Pour le portique, nous avons obtenu un $\lambda_{num2}=1,8$.

Dans les conditions prises pour le portique renforcé, il faudrait 400g au niveau du noeud 3 pour que la structure flambe.

Dans les conditions prises pour le portique, il faudrait 180g au niveau des noeuds 2 et 4  pour que la structure flambe.

Nous avons testé expérimentalement sur le portique et le portique pour voir si nos modèles numériques sont une modélisation acceptable de notre structure.

Voici la photo pour le portique renforcé, nous avons mis une masse totale de 400g au dessus de notre structure qui corresponderait à notre noeud 3 sur notre modélisation :

Pour vérifier le flambement de notre structure, nous avons appuyé doucement sur le côté de la structure et nous avons remarqué qu'il y avait un grand déplacement de notre structure ce qui n'était pas le cas quand nous mettions pas de masse.

Nous avons obtenu $\lambda_{exp1}=4,0$ pour le portique renforcé.

Nous avons fait de même pour le portique et nous avons trouvé $\lambda_{exp2}=1,7$.

Nous trouvons donc des valeurs valeurs proches entre notre modèle numérique avec $\lambda_{num1}=4,0$ et $\lambda_{num2}=1,8$ ,et notre modèle expérimental avec $\lambda_{exp1}=4,0$ et $\lambda_{exp2}=1,7$.

# Conclusion

Pour conclure, nous avons pu comparer nos modèles expérimentaux et nos modèles numériques et voir qu'ils se comportaient de la même façon même si les mesures sur les modèles expérimentaux restent très approximatives.
Les approximations faites dans notre modèle numérique ne sont donc pas aberrantes.

Nous avons par la suite essayé notre modélisation numérique sur d'autres structures afin de savoir laquelle pouvait supporter un $\lambda$ maximal et nous sommes arrivés à la structure ci-dessous qui flambe à $\lambda=20$ soit 2kg.Cette disposition serait la disposition optimale pour maximiser le $\lambda$ de notre modélisation d'un immeuble à deux étages.

```python
nodes= np.array([[0.,0.],[0.,Lyb],[0.,Lyh+Lyb],[Lx/2,Lyh+Lyb],[Lx,Lyh+Lyb],[Lx,Lyb],[Lx,0.]])
elements=np.array([[0,1],[1,5],[1,2],[2,3],[3,4],[4,5],[2,5],[0,5],[5,6]])

frame= Frame_Buckling(nodes,elements)
frame.plot_with_label()
```


# Fourier Series: Straight Lines and Polygons
This notebook shows how to compute the Fourier series of straight lines and polygons.

### Content
1. [Prerequisites](#prerequisites)
2. [Representing Straight Lines](#representation)
    1. Polygons
3. [Fourier Series of Straight Lines](#fourier_computation)
    1. [Proof](#proof)
    2. [Implementation](#implementation)
4. [Example](#example)

```python
%matplotlib inline

# Initial imports
import numpy as np
import matplotlib.pyplot as plt
```

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});

MathJax.Hub.Queue(
  ["resetEquationNumbers", MathJax.InputJax.TeX],
  ["PreProcess", MathJax.Hub],
  ["Reprocess", MathJax.Hub]
);
```

    <IPython.core.display.Javascript object>

## Prerequisites
This notebook shows how to compute the Fourier series of straight lines and polygones. For this, you should be familiar with:
- The general computation of [Fourier series of curves](Fourier-Series-of-Curves-Background.ipynb) and especially the case of piecewiese defines curves.
- Complex numbers, especially the representation of 2-dimensional points as complex numbers
- Computation of integrals of curves

## Representing Straight Lines
A straight line $g:[0,1]\to\mathbb{C}$ from point $p_0\in\mathbb{C}$ to point $p_1\in\mathbb{C}$ with $g(0)=p_0$ and $g(1) = p_1$ can be written as

$$
\begin{equation}
    g(t) = (1-t)\cdot p_0 + t\cdot p_1 \label{eqn:straigt_line_01}
\end{equation}
$$

For the computation of the Fourier Series, we need a curve defined over the interval $[0,2\pi]$; for a polygon, which is a piecewise concatenation of straight lines, each piece $i$ would be defined over an interval $[t_i,t_{i+1}]$. For this reason, we generalize ($\ref{eqn:straigt_line_01}$) to a function $g:[a,b]\to\mathbb{C}$ with $g(a)=p_0$ and $g(b) = p_1$:

$$
\begin{align}
    g(t) &= \frac{b-t}{b-a}\cdot p_0 + \frac{t-a}{b-a}\cdot p_1 \label{eqn:straigt_line_ab}\\
         &= \frac{p_0b - p_1a}{b-a} + \frac{p_1-p_0}{b-a}t \label{eqn:straigt_line_by_t}
\end{align}
$$

### Polygons
Polygons are a sequence of straight lines, connecting corner points $p_0,\ldots,p_n$ with $p_n = p_0$ for closing the curve.

## Fourier Series of Straight Lines
As described in the [introduction of Fourier series of curves](Fourier-Series-of-Curves-Background.ipynb) we need to solve the integral $\int_a^b g(t)e^{-i\lambda t}dt$ for an arbitrary straight line $g$ and any choice of $a$, $b$ and $\lambda$ in order to compute the Fourier series of straight lines and polygons.

_Case 1:_ $\lambda=0$:
$$
    \int_a^b g(t)e^{-i\lambda t}dt = \frac{1}{2}\left(p_1 + p_2)(b - a)\right)
$$

_Case 2:_ $\lambda\neq 0$:
$$
    \int_a^b g(t)e^{-i\lambda t}dt = \frac{ie^{-i\lambda b}}{\lambda}p_2 - \frac{ie^{-i\lambda a}}{\lambda} p_1
           +\frac{1}{\lambda^2}\frac{p_2-p_1}{b-a}\left(e^{-i\lambda b}-e^{-i\lambda a}\right)
$$

### Proof:
Using (\ref{eqn:straigt_line_by_t}), we can compute the integral as:
_Case 1:_ $\lambda=0$:
$$
\begin{align*}
    \int_a^b g(t)e^{-i\lambda t}dt &= \int_a^b g(t)dt \\
        &= \frac{p_1b - p_2a}{b-a}\left(b-a\right) + \frac{p_2-p_1}{b-a}\frac{1}{2}\left(b^2-a^2\right)\\
        &= \left(p_1b - p_2a\right) + \frac{1}{2}\left(p_2-p_1\right)\left(a+b\right)\\
        &= \frac{1}{2}\left(p_1b - p_2a + p_2b - p_1a\right)\\
        &= \frac{1}{2}\left(p_1 + p_2\right)\left(b - a\right)
\end{align*}
$$

_Case 2:_ $\lambda\neq 0$:
$$
\begin{align*}
    \int_a^b g(t)e^{-i\lambda t}dt &= \int_a^b \left(\frac{p_1b - p_2a}{b-a} + \frac{p_2-p_1}{b-a}t\right)e^{-i\lambda t}dt\\
        &= \frac{p_1b - p_2a}{b-a} \int_a^b e^{-i\lambda t}dt + \frac{p_2-p_1}{b-a}\int_a^b te^{-i\lambda t}dt\\
        &= \frac{p_1b - p_2a}{b-a} \left[\frac{e^{-i\lambda t}}{-i\lambda}\right]_{t=a}^b + \frac{p_2-p_1}{b-a}\left(\left[\frac{te^{-i\lambda t}}{-i\lambda}\right]_{t=a}^b-\int_a^b \frac{e^{-i\lambda t}}{-i\lambda}dt\right)\\
        &= \frac{i}{\lambda}\frac{p_1b - p_2a}{b-a} \left(e^{-i\lambda b}-e^{-i\lambda a}\right) + \frac{i}{\lambda}\frac{p_2-p_1}{b-a}\left(be^{-i\lambda b}-ae^{-i\lambda a}\right) + \frac{1}{\lambda^2}\frac{p_2-p_1}{b-a}\left(e^{-i\lambda b}-e^{-i\lambda a}\right)\\
        &=\\
        &= \frac{e^{-i\lambda b}}{b-a} \frac{i}{\lambda}\left(p_1b - p_2a + p_2b - p_1b\right)-\frac{e^{-i\lambda a}}{b-a} \frac{i}{\lambda}\left(p_1b - p_2a + p_2a - p_1a\right) + \frac{1}{\lambda^2}\frac{p_2-p_1}{b-a}\left(e^{-i\lambda b}-e^{-i\lambda a}\right)\\
        &= \frac{ie^{-i\lambda b}}{\lambda}p_2 - \frac{ie^{-i\lambda a}}{\lambda} p_1 +\frac{1}{\lambda^2}\frac{p_2-p_1}{b-a}\left(e^{-i\lambda b}-e^{-i\lambda a}\right)
\end{align*}
$$

### Implementation
Beginning with a single staight line segment

```python
def transform_straight_line(p1, p2, l, a, b):
    i = 1j   # Just for shorter notations

    l = np.asarray(l)
    result = np.zeros(shape=l.shape, dtype=np.complex)

    # Handle Case k != 0
    l_ = l[l!=0]
    result[l!=0] = i * np.exp(-i*l_*b) * p2 / l_ \
           - i * np.exp(-i*l_*a) * p1 / l_ \
           + (p2 - p1) * (np.exp(-i*l_*b) - np.exp(-i*l_*a)) / (l_*l_*(b - a))

    # Handle case k=0
    result[l==0] = (p1 + p2) * (b - a) / 2

    # Return results
    return result
```

Extending this to closed sequences of lines. Here we scale the period so that the length $T=2\pi$.

_Note:_ There are different ways to determine the segment-borders $t_0,\ldots,t_m$. Here we use the length of the lines.

```python
def transform_polygon(p, n):
    m = len(p)   # Number of Segments (if we close the path)

    # Close the curve: add the beginning point at the end
    p = np.reshape(p, (-1))
    p = p[list(range(m)) + [0]]

    # Length of the segments
    l = np.abs(p[1:] - p[:-1])

    # Compute t_0 to t_m based on the lengths with t_0 = 0 and t_m = 2\pi
    t = l.cumsum() / l.sum()
    t = 2 * np.pi * np.concatenate([[0], t])

    # get vector of k
    k = np.arange(-n,n+1)

    c = sum([transform_straight_line(p[i], p[i+1], k, t[i], t[i+1]) for i in range(m)])

    return c, k

```

Get the fourier approximation as a function.
This will be used to plot the fourier approximation.

```python
def get_fourier_fct(c, k):
    # Reshape the fourier coefficients row vectors
    c = np.reshape(c, (1,-1))
    k = np.reshape(k, (1,-1))

    def fct(t):
        # Reshape the input values into a column vector
        t = np.reshape(t, (-1,1))

        return np.sum(c * np.exp(1j * k * t), axis=1) / (2 * np.pi)

    return fct
```

## Example: Triangle
Works also with other polygons, just add or modify points.

```python
# Corner points of the triangle as complex numbers
p = [-1 -1j,
     0.5 + 3j,
     12
    ]
```

```python
N = 12

# Compute coefficients c_{-N} to c_N
c,k = transform_polygon(p, N)

# Get approximations (limited to different values n=1,...,N)
fcts = [get_fourier_fct(c[np.abs(k) <= n], k[np.abs(k) <= n]) for n in range(1,N+1)]
```

Plotting the results

```python
# plot closed curve of complex
def plotcc(p, *args, **kwargs):
    # close curve
    m = len(p)
    p = np.reshape(p, (-1))
    p = p[list(range(m)) + [0]]     # Closing the curve

    # Complex to real x and y vectors
    x, y = np.real(p), np.imag(p)

    plt.plot(x, y, *args, **kwargs)
```

```python
plt.figure(figsize=(30,20))

# Number of points
nT = 1000

t = 2 * np.pi * np.arange(0,1, 1/nT)

for n in range(1, N+1):
    pf = fcts[n-1](t)

    plt.subplot(3,4,n)
    plotcc(p)
    plotcc(pf)
    plt.title(f"n={n}")

plt.show()
```


## Teoría de perturbaciones

Consiste en resolver un sistema perturbado(se conoce la solución al no perturbado), y donde el interés es conocer la contribución de la parte perturbada $H'$ al nuevo sistema total.

$$ H = H^{0} + H'$$

Para sistemas no degenerados, la corrección a la energía a primer orden se calcula como

$$ E_{n}^{(1)} = \int\psi_{n}^{(0)*} H' \psi_{n}^{(0)}d\tau$$

** Tarea 1 : Programar esta ecuación si conoces $H^{0}$ y sus soluciones. **

```python
### Solución
from sympy import *
from sympy import init_printing; init_printing(use_latex = 'mathjax')
import numpy as np
from sympy.physics.quantum.constants import hbar
import scipy.integrate as integ
import scipy.constants as cons
var('x r phi theta r_1 r_2 phi_1 phi_2 theta_1 theta_2 gamma')
var('n l m', positive = 'True', integer = 'True')
var('m_e Z a_0 q L epsilon_0 omega', constant = 'True')
var('c d', real = 'True', constant = 'True')
import sympy.physics.hydrogen as hyd
import sympy.functions.special.spherical_harmonics as sph
import sympy.physics.qho_1d as qho
```

```python
#Átomo de Hidrogeno
def Hidrogeno():
    Yhyd = hyd.R_nl(n, l, r, Z)
    Arm = sph.Ynm(m, l, theta, phi)  #Función de onda átomo de hidrógeno
    Ehyd = hyd.E_nl(n, Z) * (hbar ** 2 / (a_0 ** 2 * m_e)) #Energía en eV
    return Yhyd, Arm, Ehyd
```

```python
#Oscilador armónico 1D
#H = c * x ** 3 + d * x ** 4
def oscilador():
    Eqho = qho.E_n(n, omega)
    Yqho = qho.psi_n(n, x, m, omega)
    return Yqho, Eqho
```

```python
#Particula en una caja 1D (box)
#H = x ** 2
def caja():
    Ebox = (hbar * pi * n) ** 2 / ((2* m_e) * L **2)
    Ybox = sqrt(2 / L) * sin(n * pi * x / L)
    return Ybox, Ebox
```

```python
#General y Menú
print('Para que sistema desea resolver: 1. Oscilador armónico 1D, 2. Partícula en una caja 1D, 3. Atómo de Hidrogeno')
sis = input('Introduzca el número de opción')
print('Ahora toca elegir la perturbación H´ (para el atomo de hidrogeno (corrección relativista) presione cualquier tecla)')
H = sympify(input('Introduzca la perturbación'))
aprox = 5 #que aproximación quieres + 1
if sis == '1':
    Y0, E0 = oscilador()
    nbase = 0
    linf = -oo
    lsup = oo
    ww = 1e15
    def psis():
        EqhoC = integrate((Y0* H * Y0).subs(n, nbase), (x, linf, lsup)).args[0][0]
        YqhoS = np.asarray([(integrate((Y0.subs(n, i)* H * Y0.subs(n, nbase)), (x, -oo, oo))).args[0][0] \
                            for i in range(0,aprox) if i != nbase])
        EqhoS = np.asarray([E0.subs([(n, nbase), (omega, ww), (hbar, cons.hbar)]) \
                            - E0.subs([(n, i), (omega, ww), (hbar, cons.hbar)]) for i in range(0,aprox) if i != nbase])
        Cqho = YqhoS / EqhoS
        YqhoC = [Cqho[i-nbase-1] * Y0.subs(n, i) for i in range(0, aprox) if i != nbase]
        PsiQho = np.sum(YqhoC)
        Cqho2 = YqhoS ** 2 / EqhoS #Oscilador
        Eqho2 = np.sum(Cqho2)
        return EqhoC, Eqho2, PsiQho
elif sis == '2':
    Y0, E0 = caja()
    nbase = 1
    linf = 0
    lsup = L
    LL = 1e-9
    def inte(x, LL, mm, nn, pii):
        return Y0.subs([(L, LL), (n, nn), (pi, pii)])*H*Y0.subs([(L, LL), (n, mm), (pi, pii)])
    def psis():
        EnBox = (integrate(Y0*H*Y0, (x, linf, lsup))).simplify()
        YboxS = np.asarray([integ.quad(lambdify(x, inte(x, LL, i, nbase, np.pi), 'numpy'), 0, LL) \
                            for i in range(1,aprox) if i != nbase])
        EboxS = np.asarray([E0.subs([(L, LL), (pi, np.pi), (n, nbase), (m_e, cons.m_e), (hbar, cons.hbar)]) \
                            - E0.subs([(L, LL), (pi, np.pi), (n, i), (m_e, cons.m_e), (hbar, cons.hbar)]) \
                            for i in range(1,aprox) if i != nbase])
        Cbox = YboxS[:,0] / EboxS
        YboxC = [Cbox[i-2] * Y0.subs(n, i) for i in range(1, aprox) if i != nbase]
        PsiBox = np.sum(YboxC)
        Cbox2 = (YboxS[:,0]) ** 2 / EboxS #CAJA
        Ebox2 = np.sum(Cbox2)
        return EnBox, Ebox2, PsiBox
elif sis == '3':
    Y0, A0, E0 = Hidrogeno()
    nbase = 1
    lbase = 0
    mbase = 0
    linf = 0
    lsup = oo
    def psis():
        Rad = Y0.subs([(l, lbase), (Z, 1)])
        RadHRad = -Rad.subs(n, nbase) *( 1/ (8*m_e**3*c**2)) * hbar ** 4  * diff(Rad.subs(n, nbase), r, 4)
        EhydC = integrate(RadHRad*r**2, (r, 0, oo))
        EhydC = EhydC.subs([(hbar, cons.hbar), (m_e, cons.m_e), (c, cons.c)]) / (4 * cons.epsilon_0) ** 4
        YhydS = np.asarray([(integrate(-Rad.subs(n, i) *( 1/ (8*m_e**3*c**2)) * hbar ** 4  * diff(Rad.subs(n, nbase), r, 4), (r, 0, oo))) \
                   for i in range(1,aprox) if i != nbase])
        EhydS = np.asarray([E0.subs([(n, nbase), (Z, 1)]) - E0.subs([(n, i), (Z, 1)]) for i in range(1,aprox) if i != nbase])
        Chyd = YhydS / EhydS
        YhydC = [Chyd[i-nbase-1] * Rad.subs([(n, i)]) for i in range(1, aprox) if i != nbase]
        PsiHyd = np.sum(YhydC)
        Chyd2 = YhydS ** 2 / EhydS
        Ehyd2 = np.sum(Chyd2)
        return EhydC, Ehyd2, PsiHyd
else:
    print('Selección inválida')

```

    Para que sistema desea resolver: 1. Oscilador armónico 1D, 2. Partícula en una caja 1D, 3. Atómo de Hidrogeno
    Introduzca el número de opción3
    Ahora toca elegir la perturbación H´ (para el atomo de hidrogeno (corrección relativista) presione cualquier tecla)
    Introduzca la perturbación1

```python
E1, E2, Psi = psis()
```

```python
Psi
```

$$\frac{2 \hbar^{2} a_{0}^{2} e^{- \frac{r}{2}}}{27 c^{2} m_{e}^{2}} \left(- r + 2\right) + \frac{7 \hbar^{2} a_{0}^{2} e^{- \frac{r}{3}}}{576 c^{2} m_{e}^{2}} \left(\frac{2 r^{2}}{9} - 2 r + 3\right) + \frac{34 \hbar^{2} a_{0}^{2} e^{- \frac{r}{4}}}{9375 c^{2} m_{e}^{2}} \left(- \frac{r^{3}}{48} + \frac{r^{2}}{2} - 3 r + 4\right)$$

```python
E1
```

$$-1.44634324025469 \cdot 10^{-22}$$

```python
E2
```

$$- \frac{42820217573 \hbar^{6} a_{0}^{2}}{1944000000000 c^{4} m_{e}^{5}}$$

Y la corrección a la función de onda, también a primer orden, se obtiene como:

 $$ \psi_{n}^{(1)} = \sum_{m\neq n} \frac{\langle\psi_{m}^{(0)} | H' | \psi_{n}^{(0)} \rangle}{E_{n}^{(0)} - E_{m}^{(0)}} \psi_{m}^{(0)}$$

**Tarea 2: Programar esta ecuación si conoces $H^{0}$ y sus soluciones. **

```python
### Solución en tarea 1
```

**Tarea 3: Investigue las soluciones a segundo orden y también programe las soluciones. **

```python
### Solución en tarea 1
```

**Tarea 4.  Resolver el átomo de helio aplicando los programas anteriores.**

```python
var('r_12')
#Obtencion perturbación átomo He
H = 1 / r_12
#r12 = sqrt(r_1**2 + r_2**2 - 2 * r_1 * r_2 * cos(gamma))
#parte Radial
Rad = hyd.R_nl(1, 0, r, 2)
#Numérico (Revisar)
#Irad = (Rad1 * Rad2 * H * Rad1 * Rad2).subs([(q, cons.e), (epsilon_0, cons.epsilon_0), (pi, np.pi), (r_1-r_2, r12)])
#Ir = lambdify((r_1, r_2, gamma), Irad)
#IRAD = integ.nquad(Ir, [[0, np.inf], [0, np.inf], [0, np.pi]])
#EHe = IRAD[0] * ( 1/2) #Multiplicar por 1 / (cons.epsilon_0) para las unidades
I1 = integrate((Rad*H*Rad* r ** 2).subs([(r, r_1), (r_12, r_2)]), (r_1, 0, r_2)) + \
                integrate((Rad*H*Rad* r ** 2).subs([(r, r_1), (r_12, r_1)]), (r_1, r_2, oo))
EHe = integrate((Rad*Rad * r ** 2).subs(r, r_2) * (I1), (r_2, 0, oo)) / 2  #para pasar de Hartree a eV
```

```python
EHe * (2 * 27.2114) #para pasar de Hartree a eV
```

$$34.01425$$

**Tarea 5: Método variacional-perturbativo. **

Este método nos permite estimar de forma precisa $E^{(2)}$ y correcciones perturbativas de la energía de órdenes más elevados para el estado fundamental del sistema, sin evaluar sumas infinitas. Ver ecuación 9.38 del libro.

```python
var('zeta A')
ph = hyd.R_nl(1, 0, r_1, zeta) * hyd.R_nl(1, 0, r_2, zeta)
H =  (zeta - Z)*(1/ r_1 + 1 / r_2)
```

```python
In1 = integrate(ph * H * ph * r_1**2 * r_2 ** 2, (r_1, 0, oo)).args[0][0]
Sand1 = integrate(In1, (r_2, 0, oo)).args[0][0] + EHe * zeta - zeta ** 2
ZetaVal = solve((diff(Sand1, zeta, 1)), zeta)[0]
Correc = Sand1.subs(zeta, ZetaVal).simplify()
Correc.subs(Z, 2) * 27.2114 #para poner las unidades de eV
```

$$-77.48871328125$$

**Resolver el átomo de helio, considerando este método (sección 9.4), como mejor le parezca. **

**Tarea 6. Revisar sección 9.7. **

Inicialmente a mano, y sengunda instancia favor de intentar programar sección del problema, i.e. integral de Coulomb  e integral de intercambio.

## Siguiente: Segunda parte, Octubre
Simetrías moleculares y Hartree-Fock

```python

```


+ This notebook is part of lecture 22 *Diagonalization and powers of A* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
from sympy import init_printing, Matrix, symbols, eye, Rational
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Diagonalizing a matrix
# Powers of a matrix A

## Definition

* If A is a *n*×*n*, then a non-zero vector **x** in ℝn is called an *eigenvector* of the matrix A if A**x** is a scalar multiple of **x**
* What this suggests is that if you consider the column vector **x** and multiply it by a scalar (here called λ) (which is then parallel to **x**, just of different length) it results in the same solution as multiplying the matrix A by **x**
* Let's try another explanation: if a matrix A, multiplied with a (column) vector (**x**) results in a scalar multiple of that same (column) vector (and is thus parallel to that (column) vector) then this (column) vector is an eigenvector of the matrix A
    * In essence this multiplication of a matrix with a (column) vector produces another vector on the same line as the original vector
    * Depending on the value of this scalar the resulting vector might point in the opposite direction and be shorter or longer than the original
* This scalar multiple is called the eigenvalue
* Matrices can have more than one eigenvalue and eigenvector

## Derivations

* We need to insert an identity matrix of size *n* into the equation that describes the explanation above
$$ {A}\underline{x}={\lambda}\underline{x} \\ {A}\underline{x}={\lambda}{I}\underline{x} \\ {A}\underline{x}-{\lambda}{I}\underline{x}=\underline{0} \\ \left({A}-{\lambda}{I}\right)\underline{x}=\underline{0} $$

* Look at this carefully and you'll notice that we are suggesting the nullspace (eigenspace) of the matrix (A-λI)
* This matrix has to be singular, i.e. have a determinant of 0
$$ \left|{A}-{\lambda}{I}\right|=0 $$
* Solving this equation (called the characteristic equation) will give us the eigenvalues (λ's)
* It will always be a polynomial in λ (called the characteristic polynomial of A), with a leading coefficient of 1 and a degree of *n* corresponding to the size of A
$$ {p}\left({\lambda}\right)={\lambda}^{n}+{c}_{1}{\lambda}^{n-1}+\dots+{c}_{n} $$
* Substituting them back into...
$$ \left({A}\underline{x}-{\lambda}{I}\right)\underline{x}=\underline{0} $$
* ... allows us to calculate the eigenvector(s) **x**

* Let's look at the following matrix A
$$ A=\begin{bmatrix} 0 & 0 & -2 \\ 1 & 2 & 1 \\ 1 & 0 & 3 \end{bmatrix}\\ A-\lambda I=\begin{bmatrix} 0 & 0 & -2 \\ 1 & 2 & 1 \\ 1 & 0 & 3 \end{bmatrix}-\begin{bmatrix} \lambda  & 0 & 0 \\ 0 & \lambda  & 0 \\ 0 & 0 & \lambda  \end{bmatrix}=\begin{bmatrix} -\lambda  & 0 & -2 \\ 1 & 2-\lambda  & 1 \\ 1 & 0 & 3-\lambda  \end{bmatrix}\\ \begin{vmatrix} -\lambda  & 0 & -2 \\ 1 & 2-\lambda  & 1 \\ 1 & 0 & 3-\lambda  \end{vmatrix}=0\\ { \lambda  }^{ 3 }-5{ \lambda  }^{ 2 }+8\lambda -4=0\\ { \lambda  }_{ 1 }=1,\quad { \lambda  }_{ 2 }={ \lambda  }_{ 3 }=2 $$

* Let's start with the first eigenvalue, which is equal to 1 and replace it in A-λI

```python
A = Matrix([[-1, 0 ,-2], [1, 1, 1], [1, 0, 2]])
A
```

* We now need the nullspace of this matrix

```python
A.nullspace()
```

* We knew that this would be 1-dimensional after looking at the row-reduced form

```python
A.rref()
```

* It has rank 2 (two pivot column and 1 free variable

* Now for the other 2 eigenvalues, both equaling 2

```python
A = Matrix([[-2, 0, -2], [1, 0, 1], [1, 0, 1]])
A
```

```python
A.nullspace()
```

```python
A.rref()
```

* Only a single pivot column, therefor rank of 1 and two independent (free) variables

* Corresponding to the first eigenvalue we have a single eigenvector that is the basis for a 1-dimensional (line) eigenspace in ℝ3
* Corresponding to the second (and third) eigenvalues we have two basis vectors for a 2-dimensional plane in ℝ3
* Since we are talking about subspaces, we must note that the zero vector must be in both eigenspaces (type of nullspace), but isn't an eigenvector

## The eigenvalues of triangular (upper and lower) and diagonal matrices

* The eigenvalue of these type of matrices are exactly the entries along the main diagonal

## Real and complex eigenvalues

* There will be characteristic polynomials resulting in complex roots
* The consequences of real-valued eigenvalues for a square matrix A of size *n* are the following
    * The system (A-λI)**x**=**0** has non-trivial solutions
    * There is a non-zero vector **x** in ℝn such that A**x**=λ**x**

## The eigenvector matrix S and eigenvalue matrix Λ

* We need to create S from the (column) eigenvectors such that the following holds
$$ {S}^{-1}{A}{S}=\Lambda $$

* As such, S should be square of size *n*×*n* and invertible, so we need *n* independent eigenvectors

* Suppose we have *n* linearly independent eigenvectors of A
* Put them in the columns of S and calculate AS
$$ AS=A\begin{bmatrix} \vdots  & \vdots  & \vdots  & \vdots  \\ \vdots  & \vdots  & \vdots  & \vdots  \\ { x }_{ 1 } & { x }_{ 2 } & \dots  & { x }_{ n } \\ \vdots  & \vdots  & \vdots  & \vdots  \end{bmatrix}=\begin{bmatrix} \vdots  & \vdots  & \vdots  & \vdots  \\ \vdots  & \vdots  & \vdots  & \vdots  \\ { { \lambda  }_{ 1 }x }_{ 1 } & { \lambda  }_{ 2 }{ x }_{ 2 } & \dots  & { \lambda  }_{ n }{ x }_{ n } \\ \vdots  & \vdots  & \vdots  & \vdots  \end{bmatrix}=\begin{bmatrix} \vdots  & \vdots  & \vdots  & \vdots  \\ \vdots  & \vdots  & \vdots  & \vdots  \\ { x }_{ 1 } & { x }_{ 2 } & \dots  & { x }_{ n } \\ \vdots  & \vdots  & \vdots  & \vdots  \end{bmatrix}\begin{bmatrix} { \lambda  }_{ 1 } & 0 & 0 & 0 \\ 0 & { \lambda  }_{ 2 } & 0 & 0 \\ \vdots  & \vdots  & { \dots  } & \vdots  \\ 0 & 0 & 0 & { \lambda  }_{ n } \end{bmatrix}\\ AS=S\Lambda  $$

* From this we have the following
$$ AS=S\Lambda \\ { S }^{ -1 }AS=\Lambda \\ A=S\Lambda { S }^{ -1 } $$
* Later I will use the computer variable D for this diagonal matrix Λ

## The power of a matrix (only for *n* independent eigenvectors)

* We saw in the example section of the last lecture that the following holds
$$ {A}^{2}\underline{x}={\lambda}^{2}{x} $$
* The eigenvectors are the same for A and A2
* We can also see the following
$$ { A }^{ 2 }=S\Lambda { S }^{ -1 }S\Lambda { S }^{ -1 }=S{ \Lambda  }^{ 2 }{ S }^{ -1 } $$

* The power need not be 2, but any *k* which will have S-1 appearing *k*-1 times

* We thus have the following theorems
$$ { A }^{ k }\rightarrow 0\quad \because \quad k\rightarrow \infty ;\quad \left| { \lambda  }_{ i } \right| $$
* ...and...
* If *k* is a positive integer, λ is an eigenvalue of the matrix A, and **x** is a corresponding eigenvector, then λk is an eigenvalue of Ak and **x** is a corresponding eigenvector

## What makes a matrix diagonalizable

* In discussing diagonalization we are concerned with finding a basis for ℝn that consists of eigenvectors of a given square matrix of size *n*
* These bases can tell us about geometric properties of A and it can simplify numerical computations involving A

* We need to answer two question (which are actually the same)
    * Given a square matrix of size *n*, is there a basis for ℝn consisting of eigenvectors?
    * Given a square matrix of size *n*, it there and invertible matrix S, such that S-1AS is a diagonal matrix? (It is the same matrix S referred to above)
* If such a matrix S exists, it is said to diagonalize A (and we will call the resultant diagonal matrix D)

* In short the answer to the above question(s) is yes if A has *n* independent eigenvectors
    * This happens if all λ's are different (none are repeated) (not totally excluded if they are repeated though)

* If they are repeated, we still might have independent eigenvectors, i.e. any size identity matrix (because it is already diagonal)

```python
A = eye(5)
A
```

```python
A.eigenvals()
```

```python
A.eigenvects()
```

* Here we look at a triangular matrix, though

```python
A = Matrix([[2, 1], [0, 2]])
A.eigenvals()
```

```python
A.eigenvects()
```

* We can use python™ code to calculate the diagonalized matrix

```python
A = Matrix([[3, -2,  4, -2], [5,  3, -3, -2], [5, -2,  2, -2], [5, -2, -3, 3]])
A
```

```python
S, D = A.diagonalize()
```

```python
S # S, such that A = S times D times the inverse of S
```

```python
D # The diagonal
```

```python
S * D * S.inv() == A # Checking to see if our statement above is correct
```

```python
S.inv() * A * S == D # Checking to see if our statement above is correct
```

```python
A.eigenvals()
```

* Remember Λ from above?
    * The eigenvalues are precisely the entries along the main diagonal of the diagonal matrix

* To produce the required diagonal matrix manually then will require computing *n* linearly independent eigenvectors for matrix A of size *n* (assuming that it is diagonalizable), creating a matrix with its columns equal to these eigenvectors (called matrix S) and performing the equation S-1AS to calculate the diagonal matrix D ( or Λ)

* Back to the topic of what makes a matrix diagonalizable

* Suppose we have an equation that starts with some vector and every subsequent vector is a matrix A time the previous vector
$$ \underline{u}_{k+1}={A}\underline{u}_{k} $$

* From this arises the following
$$ { \underline { u }  }_{ 1 }=A{ \underline { u }  }_{ 0 }\\ { \underline { u }  }_{ 2 }=A{ A\underline { u }  }_{ 0 }={ A }^{ 2 }{ \underline { u }  }_{ 0 }\\ { \underline { u }  }_{ k }={ A }^{ k }{ \underline { u }  }_{ 0 } $$

* To really solve this problem, rewrite **u**0 as follows (a certain scalar times an eigenvector)
$$ { \underline { u }  }_{ 0 }={ c }_{ 1 }\underline{ x }_{ 1 }+{ c }_{ 2 }\underline{ x }_{ 2 }+\dots +{ c }_{ n }\underline{ x }_{ n } = {S}\underline{c} $$
* Where the S**c** is a linear combination of the individual eigenvectors

* Now multiply both sides by A
$$ A{ \underline { u }  }_{ 0 }={ c }_{ 1 }{A}\underline{ x }_{ 1 }+{ c }_{ 2 }{A}\underline{ x }_{ 2 }+\dots +{ c }_{ n }{A}\underline{ x }_{ n } \\ A{ \underline { u }  }_{ 0 }={ c }_{ 1 }{ \lambda  }_{ 1 }\underline{ x }_{ 1 }+{ c }_{ 2 }{ \lambda  }_{ 2 }\underline{ x }_{ 2 }+\dots +{ c }_{ n }{ \lambda  }_{ n }\underline{ x }_{ n } $$
* Taking a power of A now (i.e. *k*) would be akin to taking each eigenvalue to that power
$$ {A}^{k}{ \underline { u }  }_{ 0 }={ c }_{ 1 }{ \lambda  }_{ 1 }^{k}\underline{ x }_{ 1 }+{ c }_{ 2 }{ \lambda  }_{ 2 }^{k}\underline{ x }_{ 2 }+\dots +{ c }_{ n }{ \lambda  }_{ n }^{k}\underline{ x }_{ n } $$
* This can be written as
$$ \underline{u}_{k} = {A}^{k}\underline{u}_{0}={\Lambda}^{k}{S}\underline{c} $$

* As an example consider the Fibonacci numbers: 0, 1, 1, 2, 3, 5, 8, 13, ...
* What would the 100th number be?
* Consider the following
$$ {F}_{k+2}={F}_{k+1}+{F}_{k} $$
* This is a (second-order) difference equation; think of this example as similar to a second-order differential equation (without derivatives)
* By adding a second equation Fk+1=Fk+1, consider **u**k to be the following vector
$$ \underline{u}_{k}=\begin{bmatrix} {F}_{k+1} \\ {F}_{k} \end{bmatrix} $$
* This means the following
$$ \underline{u}_{k+1}=\begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix} \begin{bmatrix} {F}_{k+1} \\ {F}_{k} \end{bmatrix}=\begin{bmatrix} {F}_{k+1}+{F}_{k} \\ {F}_{k+1} \end{bmatrix} \\ \underline{u}_{k+1}=\begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix}\underline{u}_{k} $$

```python
A = Matrix([[1, 1], [1, 0]])
A
```

```python
A.eigenvals()
```

```python
A.eigenvects()
```

```python
S, D = A.diagonalize()
```

```python
D
```

* From above we remember the following
$$ \underline{u}_{k} = {A}^{k}\underline{u}_{0}={\Lambda}^{k}{S}\underline{c} $$
* We have **u**0 contains the first two values

```python
u_zero =  Matrix([1, 0])
u_100 = A ** 100 * u_zero
u_100 # The top value is the 100th Fibonacci number
```

```python
u_four = A ** 4 * u_zero
u_four # If the first number is 0 the the fourth number would be the top value
```

## Example problems

### Example problem 1

* Find an equation for Ck where C is given by the following matrix
$$  $$
* Calculate C100 when *a*=*b*=-1

#### Solution

```python
a, b, k = symbols('a b k')
```

```python
C = Matrix([[2 * b - a, a - b], [2 * b - 2 * a, 2 * a - b]])
C
```

* We remember the following
$$ {A}^{k}={S}{\Lambda}^{k}{S}^{-1} $$
* Where Λ is denoted by the computer variable D

```python
S, D = C.diagonalize()
```

```python
S
```

* Python™ is not always good at simplifying these
* If you look at it carefully you will note the following

```python
S = Matrix([[1, Rational(1, 2)], [1, 1]])
S
```

```python
D
```

```python
D = Matrix([[b, 0], [0, a]])
D
```

* For the values given, we have the following

```python
C = Matrix([[-1, 0],  [0, -1]])
C
```

```python
S, D = C.diagonalize()
```

```python
D
```

```python
D ** 100
```

```python
S * (D ** 100) * S.inv()
```

* Doing the same, but with eigenvalues and eigenvectors

```python
C = Matrix([[2 * b - a, a - b], [2 * b - 2 * a, 2 * a - b]])
C
```

```python
C.eigenvals()
```

* This simplifies the λ1 = *b* and λ2 = *a*
* That makes Λ (or D) the following

```python
D = Matrix([[b, 0], [0, a]])
D
```

```python
C.eigenvects() # The solution is two tuples, with each being eigenvalue, eigenvector
```

* This simplifies to the following eigenvalue matrix S

```python
S = Matrix([[1, Rational(1, 2)], [1, 1]])
S
```

* We can see if we can get back to C

```python
S * D * S.inv()
```

```python
S * D * S.inv() == C
```

* Python™ won't to Dk for you, but it's easy to do yourself

```python
D = Matrix([[b ** k, 0], [0, a ** k]])
D
```

* Now we can compute SΛS-1

```python
S * D * S.inv()
```

* Placing the given values into this equation will give you the same solution for C100 as above

```python

```