# BT2101 Introduction to Logistic Regression

#### Version: Python 3

## 1 Goal:

In this notebook, we will explore logistic regression using:
* Gradient ascent method (because you cannot get closed-form solutions)
* Open-source package: `scikit-learn`

For the gradient descent method, you will:
* Use numpy to write functions
* Write a likelihood function
* Write a derivative function
* Write an output function
* Write a gradient ascent function
* Add a constant column of 1's as intercept term
* Use the gradient ascent function to get regression estimators

```python
# -*- coding:utf-8 -*-
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from math import sqrt
from __future__ import division
%matplotlib inline
```

### 1.1 Summary of Logistic Regression

From the lecture class, we know that a typical logistic regression model of *N* observations and *p* predictors:

\begin{align}
\text{Logit}(P(y_{i}=1)) &= log(\frac{P(y_{i}=1)}{1-P(y_{i}=1)}) \\
& = \beta_0 + \sum_{j=1}^p x_j \beta_j \\
& = X\beta
\end{align}

Rewrite the model and we can get this function:

$$ P(y_{i}=1) = \frac{e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}}{1+e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}} $$

We aim to **Maximize** the (log-)likelihood function:

$$ l(\beta) = \prod_{i=1}^{N}[\frac{e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}}{1+e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}}]^{y_i}[\frac{1}{1+e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}}]^{(1-y_i)} $$
Or
$$ ll(\beta) = \sum_{i=1}^{N}[-log(1+e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j})+y_{i}(\beta_{0}+\sum_{j=1}^{p}x_j\beta_j)] $$

Let us **Maximize** the log-likelihood function $ll(\beta)$ and get derivative with respect to $\beta_j$:

\begin{align*}
\frac{\partial ll(\beta)}{\partial\beta_j} & = -\sum_{i=1}^{N}\frac{e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}}{1+e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}}x_{ij}+\sum_{i=1}^{N}y_{i}x_{ij} \\
& = \sum_{i=1}^{N}(y_{i}-\frac{e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}}{1+e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}})x_{ij} \\
& = \sum_{i=1}^{N}(y_{i}-P(y_{i}=1))x_{ij} \\
\end{align*}

Remember that the predicted score $P(y_{i}=1)$ is calculated as $\frac{e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}}{1+e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j}}=\frac{1}{1+e^{-(\beta_{0}+\sum_{j=1}^{p}x_{j}\beta_{j})}}$

Then we can write a predicted score function first.

```python
def score(feature_matrix, weight_vector):
    '''This function is used to calculate predicted probability or score.

    Inputs:
    1) feature_matrix: A matrix of selected features;
    2) weight_vector: A vector of coefficients for selected features;

    Outputs:
    1) score: A vector of predicted probabilities or scores

    '''

    score = 1/(1+np.exp(-np.dot(feature_matrix, weight_vector)))

    return score
```

```python
# Let us have a test
features = np.array([[2.0, 1.5], [4.8, 5.2]])
weights = np.array([1.0, 1.0])
weights = np.ones((2, 1))
score(features, weights)
```

    array([[0.97068777],
           [0.9999546 ]])

## 2 Gradient Ascent
### 2.1 Computing Derivatives

How can we compute the derivatives of the log-likelihood function?

We already know that the derivative with respect to $\beta_j$ is:

\begin{align*}
\frac{\partial ll(\beta)}{\partial\beta_j} & = \sum_{i=1}^{N}(y_{i}-P(y_{i}=1))x_{ij} \\
\end{align*}

```python
def derivative(feature_matrix, error_vector):
    '''This function is used to calculate the derivatives for features.
    Inputs:
    1) feature_matrix: Data matrix of features (j = 0,...,p)
    2) error_vector: A vector of errors of N observations

    Outputs:
    1) derive: Derivative for this feature j

    '''

    derive = np.dot(feature_matrix.T, error_vector)

    return derive
```

```python
# Let us have a test
features = np.array([[2.0], [4.8]])
weights = np.ones((1, 1))
true_outputs = np.array([[1], [0]])
predict_score = score(features, weights)
errors = true_outputs - predict_score

print ("True output is: ")
print (true_outputs)
print ("Predicted score is: ")
print (predict_score)
print ("Error is: ")
print (errors)
print ("Derivative is: ")
print (derivative(features, errors))
```

    True output is:
    [[1]
     [0]]
    Predicted score is:
    [[0.88079708]
     [0.99183743]]
    Error is:
    [[ 0.11920292]
     [-0.99183743]]
    Derivative is:
    [[-4.52241381]]

### 2.2 Computing Log-likehood

In the lecture class, we know the log likelihood function is computed as:

\begin{align*}
ll(\beta) & = \sum_{i=1}^{N}[-log(1+e^{\beta_0 + \sum_{j=1}^{p}x_j\beta_j})+y_{i}(\beta_{0}+\sum_{j=1}^{p}x_j\beta_j)] \\
\end{align*}

```python
def log_likelihood(feature_matrix, weight_vector, output_vector):
    '''This function is used to calculate log-likelihood value of given features, output and coefficients
    Inputs:
    1) feature_matrix: A matrix of selected features
    2) weight_vector: A vector of coefficients
    3) output_vector: A vector of true outputs

    Outputs:
    1) ll: Log-likelihood value

    '''

    # Linear term: Xbeta
    X_beta = np.dot(feature_matrix, weight_vector)

    # Sum term: The term in the sum_up bracket
    sum_term = -np.log(1+np.exp(X_beta)) + output_vector*X_beta

    # Get log-likelihood
    ll = np.sum(sum_term)

    return ll

```

### 2.3 Procedures for Gradient Ascent

Step 0: Initialize $ \beta = (\beta_{0},\dots,\beta_{p}) $;

Step 1: Calculate $ \beta_{j} \leftarrow \beta_{j} + Stepsize\times(\sum_{i=1}^{N}(y_{i}-P(y_{i}=1))x_{ij}) $, for j = 1,...,p;

Step 2: If not converged, go back to Step 1;

Step 3: Get $ \beta $.

```python
def Logistic_GA(initial_weights, feature_matrix, true_output, step_size, tolerance, n_iter):
    '''This function is used to iteratively calculate coefficients for logistic regression model using Gradient Aescent.
    Inputs:
    1) initial_weights: Initial regression coefficients
    2) feature_matrix: A matrix of selected features
    3) true_output: A vector of true outputs -> [0, 1]
    4) step_size: Size of step for each iteration of gradient search
    5) tolerance: Indicate converging condition
    6) n_iter: Maximum number of iterations

    Outputs:
    1) weights: Estimated coefficients.

    '''

    weights = np.array(initial_weights, dtype=np.float64)

    for i in range(n_iter):
        # Calculate predicted scores, or probabilitis of 1's
        prediction = score(feature_matrix, weights)

        # Calculate errors
        error = true_output - prediction

        # Calculating derivatives for weights
        derivative_vector = derivative(feature_matrix, error)

        # Updating weights
        weights += step_size * derivative_vector

        # Converging conditions: L2 norm for derivatives
        sum_squared_gradient = np.sum(np.power(derivative_vector,2))

        # Verify whether converging early and Report Log-likelihood values
        if np.sqrt(sum_squared_gradient) < tolerance:
            # Calculate log-likelihood
            loglike = log_likelihood(feature_matrix, weights, true_output)

            # Print out log-likelihood
            print ('Iteration %10d: Log-likelihood = %.4f' % (i, loglike))

            return weights

        elif (i<=20) or (i<=100 and i%10==0) or (i<=1000 and i%100==0) or (i==n_iter):
            # Calculate log-likelihood
            loglike = log_likelihood(feature_matrix, weights, true_output)

            # Print out log-likelihood
            print ('Iteration %10d: Log-likelihood = %.4f' % (i, loglike))

    return weights

```

## 3 Convert Pandas DataFrame to Numpy Array

We need to:

1. Convert Pandas DataFrame into a Numpy Array/Matrix to do internal calculations;
2. Augment this Array/Matrix by adding 1's column in the first column, in order to calculate the intercept.

```python
def convert_data(df, feature_names, output_name):
    '''This function is used to convert pandas dataframe to numpy array/matrix, and augment it with 1's column as intercept.
    Inputs:
    1) df: Original data in the format of pandas dataframe
    2) feature_names: A list of names of selected features
    3) output_name: Name of selected outputs

    Outputs:
    1) feature_matrix: Augmented feature matrix
    2) output_vector: A vector of true outputs

    '''

    # Feature names of Augmented matrix
    augment_feature_names = ['intercept'] + feature_names

    # Augmented feature matrix by adding constant 1's as intercept term, and reorder the feature matrix
    df['intercept'] = 1
    feature_matrix = df[augment_feature_names]
    n, k = feature_matrix.shape # n: number of observations; k: number of weights

    # Convert selected feature matrix and output vector to Numpy Array
    feature_matrix = feature_matrix.values.reshape((n, k))
    output_vector = df[output_name].values.reshape((n, 1))

    return (feature_matrix, output_vector)
```

## 4 Load Dataset and Conduct Estimations

Our dataset can be obtained from [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/datasets/Iris).

### 4.1 Attribute Information:

Attribute Information:

1. sepal length in cm
2. sepal width in cm
3. petal length in cm
4. petal width in cm
5. class:

-- Iris Setosa

-- Iris Versicolour

-- Iris Virginica

### Suppose: 1:class='Iris-setosa'; 0:otherwise
### It means we want to predict whether class is 'Iris-setosa' (Y=1) or not (Y=0)

```python
# Import dataset into Python Pandas DataFrame
filepath = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'
colnames = ['sepal_length', 'sepal_width', 'petal_length', 'petal_width', 'class']
data = pd.read_csv(filepath, header=None, names=colnames, dtype={'sepal_length':np.float64, 'sepal_width':np.float64, 'petal_length':np.float64, 'petal_width':np.float64})
data.head(n=5)
```

      sepal_length
      sepal_width
      petal_length
      petal_width
      class

      0
      5.1
      3.5
      1.4
      0.2
      Iris-setosa

      1
      4.9
      3.0
      1.4
      0.2
      Iris-setosa

      2
      4.7
      3.2
      1.3
      0.2
      Iris-setosa

      3
      4.6
      3.1
      1.5
      0.2
      Iris-setosa

      4
      5.0
      3.6
      1.4
      0.2
      Iris-setosa

```python
data.dropna() # Drop null values
data.describe()
```

      sepal_length
      sepal_width
      petal_length
      petal_width

      count
      150.000000
      150.000000
      150.000000
      150.000000

      mean
      5.843333
      3.054000
      3.758667
      1.198667

      std
      0.828066
      0.433594
      1.764420
      0.763161

      min
      4.300000
      2.000000
      1.000000
      0.100000

      25%
      5.100000
      2.800000
      1.600000
      0.300000

      50%
      5.800000
      3.000000
      4.350000
      1.300000

      75%
      6.400000
      3.300000
      5.100000
      1.800000

      max
      7.900000
      4.400000
      6.900000
      2.500000

```python
# Create an output dummy
data['Iris_setosa'] = data['class'].map(lambda x: 1 if x=='Iris-setosa' else 0)
```

```python
data.describe()
```

      sepal_length
      sepal_width
      petal_length
      petal_width
      Iris_setosa

      count
      150.000000
      150.000000
      150.000000
      150.000000
      150.000000

      mean
      5.843333
      3.054000
      3.758667
      1.198667
      0.333333

      std
      0.828066
      0.433594
      1.764420
      0.763161
      0.472984

      min
      4.300000
      2.000000
      1.000000
      0.100000
      0.000000

      25%
      5.100000
      2.800000
      1.600000
      0.300000
      0.000000

      50%
      5.800000
      3.000000
      4.350000
      1.300000
      0.000000

      75%
      6.400000
      3.300000
      5.100000
      1.800000
      1.000000

      max
      7.900000
      4.400000
      6.900000
      2.500000
      1.000000

```python
# Select features and output
feature_list =  ['sepal_width', 'petal_length']
output_label = 'Iris_setosa'

features = convert_data(data, feature_list, output_label)[0]
output = convert_data(data, feature_list, output_label)[1]
```

Now set up initial parameters and try gradient ascent:

```python
# Set up
initial_weights = np.ones((len(feature_list)+1, 1))
step_size = 7e-5
tolerance = 2e-9
iteration = 1000
```

```python
# Try estimation with gradient descent
coefficients = Logistic_GA(initial_weights, features, output, step_size, tolerance, iteration)
print ('\nIntercept ','X1', 'X2')
coefficients
```

    Iteration          0: Log-likelihood = -854.7068
    Iteration          1: Log-likelihood = -831.4416
    Iteration          2: Log-likelihood = -808.1852
    Iteration          3: Log-likelihood = -784.9390
    Iteration          4: Log-likelihood = -761.7047
    Iteration          5: Log-likelihood = -738.4844
    Iteration          6: Log-likelihood = -715.2805
    Iteration          7: Log-likelihood = -692.0960
    Iteration          8: Log-likelihood = -668.9344
    Iteration          9: Log-likelihood = -645.7998
    Iteration         10: Log-likelihood = -622.6974
    Iteration         11: Log-likelihood = -599.6331
    Iteration         12: Log-likelihood = -576.6144
    Iteration         13: Log-likelihood = -553.6500
    Iteration         14: Log-likelihood = -530.7506
    Iteration         15: Log-likelihood = -507.9292
    Iteration         16: Log-likelihood = -485.2014
    Iteration         17: Log-likelihood = -462.5862
    Iteration         18: Log-likelihood = -440.1063
    Iteration         19: Log-likelihood = -417.7895
    Iteration         20: Log-likelihood = -395.6688
    Iteration         30: Log-likelihood = -198.0342
    Iteration         40: Log-likelihood = -91.9918
    Iteration         50: Log-likelihood = -60.1710
    Iteration         60: Log-likelihood = -49.9844
    Iteration         70: Log-likelihood = -45.4959
    Iteration         80: Log-likelihood = -42.8597
    Iteration         90: Log-likelihood = -40.9528
    Iteration        100: Log-likelihood = -39.3869
    Iteration        200: Log-likelihood = -29.1200
    Iteration        300: Log-likelihood = -23.0344
    Iteration        400: Log-likelihood = -19.0276
    Iteration        500: Log-likelihood = -16.2086
    Iteration        600: Log-likelihood = -14.1239
    Iteration        700: Log-likelihood = -12.5222
    Iteration        800: Log-likelihood = -11.2539
    Iteration        900: Log-likelihood = -10.2252

    Intercept  X1 X2

    array([[ 0.89483748],
           [ 1.06595578],
           [-1.57802064]])

You can try different `initial_weights`, `step_size`, `tolerance`, or `iteration`.

You can also use different features to run logistic regression model.

Large step size may make gradient search fluctuate too much, which is more difficult to converge.

Small tolerance may increase the number of iterations, but will result in more accurate estimators.

## 5 Logistic Classifier

In this section, you will:
* Use `scikit-learn` to run logistic regression model
* Do train-test split
* Understand ROC curve and AUC
* Understand confusion matrix (TP, FP, TN and FN), and performance measures (e.g., accuracy, precision, etc.)

### 5.1 Scikit-Learn
The package `scikit-learn` can be found at http://scikit-learn.org/stable/index.html.

Please install the package first.

```python
from sklearn import datasets
from sklearn.cross_validation import train_test_split
from sklearn.linear_model import LogisticRegression
```

    C:\Users\bensj\Anaconda3\lib\site-packages\sklearn\cross_validation.py:41: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
      "This module will be removed in 0.20.", DeprecationWarning)

```python
# Load a dataset
X = data[['sepal_width', 'petal_length']]
y = data['Iris_setosa']
```

```python
# Do train-test split: 70% train and 30% test
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
```

```python
# Fit the model on train data: Using L2-regularization
lr = LogisticRegression(fit_intercept=True, max_iter=1000, tol=2e-9, penalty='l2', C=100, random_state=0)
lr.fit(X=X_train, y=y_train)
```

    LogisticRegression(C=100, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=1000, multi_class='ovr', n_jobs=1,
              penalty='l2', random_state=0, solver='liblinear', tol=2e-09,
              verbose=0, warm_start=False)

```python
# Get coefficients
print (lr.intercept_, lr.coef_)
```

    [1.13752285] [[ 4.1323274  -5.10560028]]

```python
# Predict outputs for test data
y_pred = lr.predict(X_test)
y_pred
```

    array([0, 0, 1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0,
           1, 1, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 1,
           1], dtype=int64)

```python
# Create confusion matrix
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_pred)
TN, FP, FN, TP = cm.ravel()
print (cm)
print (TN, FP, FN, TP)
```

    [[29  0]
     [ 0 16]]
    29 0 0 16

```python
# ROC and AUC
from sklearn.metrics import roc_curve, auc

# Get predicted scores Pr(y=1): Used as thresholds for calculating TP Rate and FP Rate
# lr.classes_
score = lr.predict_proba(X_test)[:, 1]

# Plot ROC Curve
fpr, tpr, thresholds = roc_curve(y_test, score) # fpr: FP Rate, tpr: TP Rate, thresholds: Pr(y=1)
roc_auc = auc(fpr, tpr)

plt.plot(fpr, tpr, label='AUC = %0.2f'% roc_auc)
plt.legend(loc='lower right')
plt.plot([0,1],[0,1],'r--')
plt.xlim([-0.1,1.1])
plt.ylim([-0.1,1.1])
plt.title('Receiver operating characteristic')
plt.ylabel('True Positive Rate')
plt.xlabel('False Positive Rate')
plt.show()
```

More about Logistic Regression can be found at http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html

## 6 Assignments

### Using K-Fold Cross Validation to Do Model Evaluation

#### K-Fold Cross Validation:

K-Fold cross validation: http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html

```python
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn import tree
from sklearn import datasets
```

Let's use a standard binary classification dataset from the UCI machine learning repository "Haberman's Survival Data Set" (https://archive.ics.uci.edu/ml/datasets/Haberman's+Survival). The dataset has one binary output and three numeric input variables of varying scales. The details about the dataset can be found [here](https://archive.ics.uci.edu/ml/datasets/Haberman's+Survival).

```python
# Load dataset:
url = "https://archive.ics.uci.edu/ml/machine-learning-databases/haberman/haberman.data"
names = ['age', 'year_operation', 'positive_detected', 'survive']
dataframe = pd.read_csv(url, names=names)
dataframe['survive'] = dataframe['survive'].map(lambda x:1 if x==1 else 0)  # =1, survive after 5 years; =0, die within 5 years
dataframe.head()
```

      age
      year_operation
      positive_detected
      survive

      0
      30
      64
      1
      1

      1
      30
      62
      3
      1

      2
      30
      65
      0
      1

      3
      31
      59
      2
      1

      4
      31
      65
      4
      1

```python
# Descriptive analysis
dataframe.describe()
```

      age
      year_operation
      positive_detected
      survive

      count
      306.000000
      306.000000
      306.000000
      306.000000

      mean
      52.457516
      62.852941
      4.026144
      0.735294

      std
      10.803452
      3.249405
      7.189654
      0.441899

      min
      30.000000
      58.000000
      0.000000
      0.000000

      25%
      44.000000
      60.000000
      0.000000
      0.000000

      50%
      52.000000
      63.000000
      1.000000
      1.000000

      75%
      60.750000
      65.750000
      4.000000
      1.000000

      max
      83.000000
      69.000000
      52.000000
      1.000000

```python
# Features and output
features = ['age', 'year_operation', 'positive_detected']
output = 'survive'
```

#### Step 1. Do K-fold Cross Validation
* Set `n_splits=5`: 5-fold cross validation
* Set `random_state` to `12345`

Hint: Use `KFold` function

```python
# Create a 5-fold cross validation
kf = KFold(n_splits = 5, shuffle = False, random_state = 12345)
```

#### Step 2. Create Multiple Models

* Model 1: Logistic regression with L2 regularization
* Model 2: Logistic regression with L1 regularization (Let's use the same parameter values as Model 1, except `penalty`)

```python
# Model 1: Logistic regression with L2 regularization
model_1 = LogisticRegression(fit_intercept=True, max_iter=1000, tol=2e-9, penalty='l2', C=100, random_state=12345)
```

```python
# Model 2: Logistic regression with L1 regularization (Let's use the same parameter values as Model 1, except penalty)
model_2 = LogisticRegression(fit_intercept=True, max_iter=1000, tol=2e-9, penalty='l1', C=100, random_state=12345)
```

#### Step 3. Conduct K-fold Cross-Validation on These Models

Hint: Use `cross_val_score` function

http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html

```python
# Cross-validation on Model 1
cv_model_1 = cross_val_score(model_1, # Cross-validation on model_1
                             dataframe[features], # Feature matrix
                             dataframe[output], # Output vector
                             cv=kf, # Cross-validation technique
                             scoring='accuracy' # Model performance metrics: accuracy
                            )
```

```python
# Cross-validation on Model 2
cv_model_2 = cross_val_score(model_2, # Cross-validation on model_2
                             dataframe[features], # Feature matrix
                             dataframe[output], # Output vector
                             cv=kf, # Cross-validation technique
                             scoring='accuracy' # Model performance metrics: accuracy
                            )
```

#### Step 4. Report and Evaluate Mean Accuracy of These Models

```python
# Report average cross-validation accuracy of Model 1
print(sum(cv_model_1)/len(cv_model_1))

# Report average cross-validation accuracy of Model 2
print(sum(cv_model_2)/len(cv_model_2))

```

    0.7317821258593337
    0.7350608143839238

#### Step 5. Based on the cross-validation accuacy, which model do your prefer ?

```python
# Model 2. Model 2 provides a higher accuracy using lasso regression as compared to model 1 (ridge regression).
```

```python
## 7 References

[1] Jason Brownlee, 2018, [Machine Learning Algorithms from Scratch with Python](https://machinelearningmastery.com/machine-learning-algorithms-from-scratch/). <br/>
[2] Peter Harrington, 2012. Machine Learning in Action. Shelter Island, NY: Manning Publications Co.
```


# WT operating under yaw misalignment

In case the wind turbine rotor is not perpendicular to the inflow, its operation and effects on the flow field will be different. In general it is a quite complicated process. In PyWake the effects are divided into four subeffects that are handled invididually:

1. Change of operation due to reduced inflow wind speed
2. Reduced deficit, $deficit_{normal}$, due to reduced inflow wind speed, ($C_{T,n} \rightarrow C_{T,x}$)
3. Reduced deficit, $deficit_{downwind}$, due to misalignment between thrust and downwind direction
4. Deflection of wake deficit, $deficit_{deflected}$, due to traversal thrust component reaction

```python
# Install PyWake if needed
try:
    import py_wake
except ModuleNotFoundError:
    !pip install git+https://gitlab.windenergy.dtu.dk/TOPFARM/PyWake.git
```

```python
import numpy as np
import matplotlib.pyplot as plt
from py_wake.examples.data.hornsrev1 import V80
from py_wake.flow_map import HorizontalGrid
from py_wake.tests.test_files import tfp
from py_wake.wind_farm_models import PropagateDownwind, All2AllIterative
from py_wake.deficit_models import FugaYawDeficit
from py_wake.deflection_models import FugaDeflection
from py_wake.examples.data.hornsrev1 import Hornsrev1Site
from py_wake.flow_map import XYGrid

from ipywidgets import interact
from ipywidgets import IntSlider
wt = V80()
site = Hornsrev1Site()
```

## Change of operation due to reduced inflow wind speed

The inflow perpendicular to the rotor is reduced by $cos\theta$.

This means that the pitch and rotor speed setting of the WT will be different. In PyWake the WT operation (pitch and rotor speed settings) is modeled in terms of the Power and CT curves. If these curves are specified in terms of the the wind speed normal to the rotor, then the wind speed must be multiplied by $cos\theta$ before looking up the power and ct. This is one of two effects modeled by the `SimpleYawModel`, which is an additional model that is applied as default to most PowerCtFunctions, see [here](#SimpleYawModel) for more details.

Note, that below rated rotor speed, the CT curve is typically rather flat and reducing the wind speed has therefore limited effect on the CT value in this region. The power, on the other hand, is limited by the maximum power and therefore only affected in the region below rated wind speed (rated wind speed will increase with yaw misalignment)

```python
u = np.arange(3,25,.1)
ax1,ax2 = plt.subplots(1,2, figsize=(15,5))[1]
for yaw in [0,15,30]:
    ax1.plot(u,wt.ct(u*np.cos(np.deg2rad(yaw))), label='%d deg yaw misalignment'%yaw)
    ax2.plot(u,wt.power(u*np.cos(np.deg2rad(yaw))), label='%d deg yaw misalignment'%yaw)
for ax in (ax1, ax2):
    ax.legend()
    ax.set_xlabel('Wind speed [m/s]')
ax1.set_ylabel('CT')
ax2.set_ylabel('Power [W]')
```

Furthermore, misalignement between the thrust force (axial induction) and the inflow leads to higher wind speed at the rotor plane, which also affects the WT operation. This effect, however, is not considered in PyWake.

## Reduced deficit due to reduced inflow wind speed ($C_{T,n} \rightarrow C_{T,x}$)

The wake deficit is caused by a reaction to the thrust force, which slows down the inflow.

The thrust force normal to the rotor plane is

\begin{align}
T_n & =\frac{1}{2}\rho C_{T,n} A (U cos\theta)^2
\end{align}

In non-aligned inflow, the thrust force that slows down the flow, i.e. in the mean wind direction is

\begin{align}
T_x & =\frac{1}{2}\rho C_ {T,x} \left(A cos\theta \right) U^2
\end{align}

From these two equations we can find the relationship between the thrust coefficient in the rotor-normal direction, $C_{T,n}$, and the thrust coefficient in the down-wind direction, $C_{T,x}$.

\begin{align}
T_x & = T_n cos\theta \\
\frac{1}{2} \rho C_{T,x} \left(A cos\theta \right) U^2 & = \frac{1}{2}\rho C_{T,n} A (U cos\theta)^2 cos\theta\\
C_ {T,x}  & = C_{T,n}  \cos^2\theta
\end{align}

This is the second effect modeled by the `SimpleYawModel`, which is an additional model that is applied as default to most PowerCtFunctions, see [here](#SimpleYawModel) for more details.

## SimpleYawModel

The `SimpleYawModel` is an additional model, which as default is applied by
- `PowerCtFunction`
- `PowerCtTabular`
- `PowerCtNDTabular`
- `PowerCtXr`
- `CubePowerSimpleCt`

It handles effects 1. and 2. by:

1. Compute/look up $C_{T,n}$ based on wind speed $U\cos\theta$
2. Returns $C_{T,x} = C_{T,n} \cos^2\theta$

## Reduced deficit due to misalignment between thrust and downwind direction

Most engineering models calculate the deficit normal to the rotor plane, while the deficit impacting downstream WT is the deficit in the downstream direction.

In case of yaw misalignment, the deficit is therefore mapped to downstream direction by

$deficit_{downwind} = deficit_{normal} cos \theta$

### FugaYawDeficit

The `FugaYawDeficit` model implicitly models this effect. This model requires two sets of look-up tabels, namely `UL` and `UT`. These tables describes the normalized deficit in the downwind, `U` direction caused by a unit forcing in the longitudinal, L, and transveral, T, directions.

```python
fuga = FugaYawDeficit(tfp + 'fuga/2MW/Z0=0.00001000Zi=00400Zeta0=0.00E+00/')
D = 80
R = D/2
x0,x1,y = 1,10,2 #upstream, downstream and crosswind regions to load [D]
xi0 = int(512-x0/(fuga.dx/D))
xi1 = int(512+x1/(fuga.dx/D))+1
yi = int(y/(fuga.dy/D))+1
UL, UT = [-fuga.mirror(v,anti_symmetric=a) for v, a in zip(fuga.load_luts()[:2,0,:yi, xi0:xi1], (False,True))]
```

```python
axes = plt.subplots(2,1, figsize=(15,12))[1]
X, Y = np.meshgrid(fuga.x[xi0:xi1], fuga.mirror(fuga.y[:yi], anti_symmetric=True))
wtL = np.array([[0,0],[-D/2,D/2]])
for ax, v, l in zip(axes.flatten(), [UL, UT], ['UL','UT']):
    c = ax.contourf(X, Y, v)
    plt.colorbar(c, ax=ax, label=r'Downwind deficit $\left[\frac{1}{U^2 CT}\right]$')
    s = (slice(None,None,2),slice(None,None,5))
    ax.quiver(X[s], Y[s], -v[s]*(l[0]=='U'), v[s]*(l[0]=='V'))
    ax.plot([0,0],[-D/2,D/2],'r')
    if l[1]=='L':
        ax.arrow(-10,0,-20,0,color='r', width=10,head_length=30)
        ax.arrow(60,0,-20,0,color='r', width=10,head_length=30)
    else:
        ax.arrow(-30,25,0,-20,color='r', width=10,head_length=30)
        ax.arrow(30,25,0,-20,color='r', width=10,head_length=30)

    ax.axis('equal')
    ax.set_xlim([-D,10*D])
    ax.set_ylim([-D,D])
    ax.set_title(l)
```

As Fuga is a linear model, the deficit in the downstream direction can simply be calculated by the linear superposition of `UL` and `UT`:

$deficit = (UL \cdot cos\theta + UT \cdot sin\theta) \cdot U \cdot CT$

You can move the slider below to see the deficit fields for different yaw-misalignment angles

```python
def plot(yaw):
    plt.figure(figsize=(18,5))
    theta = np.deg2rad(yaw)
    co, si = np.cos(theta), np.sin(theta)
    deficit = co**2*(UL*co + UT*si)
    c = plt.contourf(X, Y, deficit,20)#, levels=np.arange(-.4,.1,.05))
    plt.colorbar(c, label=r'Downwind deficit $\left[\frac{1}{U^2 CT}\right]$')
    s = (slice(None,None,2),slice(None,None,5))
    plt.quiver(X[s], Y[s], -deficit[s], deficit[s]*0, width=.002, scale=2)
    plt.plot([si*R, -si*R],[-co*R,co*R],'w',label='Rotor',lw=3)
    plt.arrow(-0,0,-100*co,0,color='r', width=5,head_length=10,length_includes_head=True)
    plt.arrow(-0,-0,0,-100*si,color='r', width=5,head_length=10,length_includes_head=True, zorder=32)
    plt.axis('equal')
    plt.xlim([-D,10*D])
    plt.ylim([-D/2,D/2])
    ax.set_title(l)
_ = interact(plot, yaw=IntSlider(min=-90, max=90, step=1, value=20, continuous_update=False))
```

    interactive(children=(IntSlider(value=20, continuous_update=False, description='yaw', max=90, min=-90), Output…

## Wake deflection due to traversal thrust component reaction

Wake deflection is modeled by a `DeflectionModel`, which modifies the downwind, horizontal crosswind and vertical distance between wind turbines.

### FugaDeflection

FugaDeflection is capable of modeling the wake deflection if `VL` and `VT` is known. These tables describes the normalized deficit in the crosswind, `V` direction caused by a unit forcing in the longitudinal, L, and transveral, T, directions.

```python
fugaDeflection = FugaDeflection(tfp + 'fuga/2MW/Z0=0.00001000Zi=00400Zeta0=0.00E+00/')
D = 80
R = D/2
x0,x1,y = 1,10,2 #upstream, downstream and crosswind regions to load [D]
xi0 = int(512-x0/(fugaDeflection.dx/D))
xi1 = int(512+x1/(fugaDeflection.dx/D))+1
yi = int(y/(fugaDeflection.dy/D))+1

# VL is antisymmetric, VT is symmetric
VL, VT = [fugaDeflection.mirror(v,anti_symmetric=a) for v, a in zip(fugaDeflection.load_luts()[2:4,0,:yi, xi0:xi1], (True, False))]
```

```python
axes = plt.subplots(2,1, figsize=(15,12))[1]
X, Y = np.meshgrid(fugaDeflection.x[xi0:xi1], fugaDeflection.mirror(fugaDeflection.y[:yi], anti_symmetric=True))
for ax, v, l in zip(axes.flatten(), [-VL, VT], ['VL','VT']):
    c = ax.contourf(X, Y, v)
    plt.colorbar(c, ax=ax, label=r'$WS_{Crosswind} \left[\frac{1}{U^2 CT}\right]$')
    s = (slice(None,None,10),slice(None,None,1))
    ax.quiver(X[s], Y[s], 0, v[s], scale=(.3,3)[l[1]=='T'])
    ax.plot([0,0],[-D/2,D/2],'r')
    if l[1]=='L':
        ax.arrow(-10,0,-20,0,color='r', width=10,head_length=30)
        ax.arrow(60,0,-20,0,color='r', width=10,head_length=30)
    else:
        ax.arrow(-30,25,0,-20,color='r', width=10,head_length=30)
        ax.arrow(30,25,0,-20,color='r', width=10,head_length=30)

    ax.axis('equal')
    ax.set_xlim([-D,10*D])
    ax.set_ylim([-D,D])
    ax.set_title(l)
```

As Fuga is a linear model, the effects in the crosswind direction can simply be calculated by linear superposition of `VL` and `VT`:

$WS_{crosswind} = (VL \cdot cos\theta + VT \cdot sin\theta) \cdot U \cdot CT$

You can move the slider below to see the crosswind wind speed field for different yaw-misalignment angles. Furthermore, the integrated deflection is shown with solid lines

```python
def plot(yaw):
    plt.figure(figsize=(18,5))
    theta = np.deg2rad(yaw)
    co, si = np.cos(theta), np.sin(theta)
    V = co**2*(-VL*co + VT*si)
    X, Y = np.meshgrid(fugaDeflection.x[xi0:xi1], fugaDeflection.mirror(fugaDeflection.y[:yi], anti_symmetric=True))
    c = plt.contourf(X, Y, V)#, levels=np.arange(-.09,.1,.01))
    plt.colorbar(c, label=r'$WS_{crosswind} \left[\frac{1}{U CT}\right]$')
    s = (slice(None,None,6),slice(None,None,1))
    plt.quiver(X[s], Y[s], V[s]*0, V[s], width=.002, scale=2)

    # plot deflection lines
    fL, fT = fugaDeflection.fLT.V.T[:,256-yi:255+yi, xi0:xi1]
    lambda2p = co**2 * (fL * co - fT * si)
    y = fugaDeflection.mirror(fugaDeflection.y[:yi], anti_symmetric=True)

    lambda2 = np.array([np.interp(y, y + l2p, l2p) for l2p in lambda2p.T])
    Yp = Y + lambda2.T
    plt.plot(X[yi, :], Y[yi, :], 'grey', lw=3)
    for x, y, yp in zip(X[1::4], Y[1::4], Yp[1::4]):
        plt.plot(x, y, 'grey', lw=1, zorder=-32)
        plt.plot(x, yp, 'k', lw=1)
    plt.plot(X[yi, :], Yp[yi, :], 'k', lw=3)

    plt.plot([si*R, -si*R],[-co*R,co*R],'w',label='Rotor',lw=3)
    plt.arrow(-0,0,-50*co,0,color='r', width=5,head_length=10,length_includes_head=True, zorder=32)
    plt.arrow(-0,-0,0,-50*si,color='r', width=5,head_length=10,length_includes_head=True, zorder=32)
    plt.axis('equal')
    plt.xlim([-D,10*D])
    plt.ylim([-D,D])
    ax.set_title(l)
_ = interact(plot, yaw=IntSlider(min=-90, max=90, step=1, value=20, continuous_update=False))
```

    interactive(children=(IntSlider(value=20, continuous_update=False, description='yaw', max=90, min=-90), Output…

Note, the plot above shows the wind speed in cross-wind direction (up), i.e. yellow colors corresponds to a flow towards the top of the figure, while blue corresponds to a flow towards the bottom.

Finally, all effects are put together. You can move the slider below to see the effects on the deficit behind a WT.

```python
path = tfp + 'fuga/2MW/Z0=0.00001000Zi=00400Zeta0=0.00E+00/'
fugaDeficit = FugaYawDeficit(path)
wfm = All2AllIterative(site,wt,fugaDeficit,blockage_deficitModel=fugaDeficit,
                       deflectionModel=FugaDeflection(path)
                       )
def plot(yaw):
    plt.figure(figsize=(18,5))
    fm = wfm([0],[0],ws=10,wd=270, yaw=[[[yaw]]]).flow_map(XYGrid(x=np.arange(-100,800,10), y=np.arange(-250,250,10)))
    X,Y = np.meshgrid(fm.x, fm.y)
    c1 = plt.contourf(X,Y,fm.WS_eff.squeeze(),np.arange(6.5,10.1,0.1), cmap='Blues_r')
    c2 = plt.contourf(X,Y,fm.WS_eff.squeeze(),np.arange(10,10.2,.005), cmap='Reds')
    plt.colorbar(c2, label='Wind speed (deficit regions) [m/s]')
    plt.colorbar(c1, label='Wind speed (speed-up regions) [m/s]')
    wt.plot([0],[0],wd=270,yaw=yaw)
    max_deficit_line = fm.min_WS_eff(x=np.arange(-100,800,10))
    max_deficit_line.plot(color='k', label='Max deficit line')
    plt.axhline(0, label='Center line')
    plt.legend()

_ = interact(plot, yaw=IntSlider(min=-90, max=90, step=1, value=20, continuous_update=False))
```

    interactive(children=(IntSlider(value=20, continuous_update=False, description='yaw', max=90, min=-90), Output…

Note, the scaling of wake deficit(blue) and speedup(red) does not match

```python

```


# 微分方程模型

**用sympy和scipy**

```python
# 一些参数. 来自15.2
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

x0 = np.array([41, 49, 61, 78, 96, 104], dtype=np.float)
plt.plot(np.linspace(0, 1, x0.shape[0]), x0)
plt.show()
lamb = np.array([x0[i] / x0[i+1] for i in range(x0.shape[0]-1)])
Theta = (np.exp(-2/(x0.shape[0]+1)), np.exp(2/(x0.shape[0]+2)))
OK = [i <= Theta[1] and i >= -Theta[0] for i in lamb]
x1 = np.array([x0[:i].sum() for i in range(1, x0.shape[0]+1)])
alpha1_x0 = np.array([x0[i] - x0[i-1] for i in range(1, x0.shape[0])])
z1 = np.array([(x1[i]+x1[i+1])/2 for i in range(x1.shape[0]-1)])
B = np.array(np.c_[-x0[1:], -z1, np.ones(z1.shape[0])])
Y = alpha1_x0
u = np.matmul(np.linalg.inv(np.matmul(B.T, B)), np.matmul(B.T, Y))
a1, a2, b = u[0], u[1], u[2]
x1_t = lambda k: (x0[0] - b/a) * np.exp(-a*k) + b/a
```

```python
# sympy的一个例子
# 参考: https://blog.csdn.net/and_w/article/details/80160033
import sympy

sympy.init_printing
# 定义符号常量x 与 f(x)
t = sympy.Symbol('t')
a1, a2, b = sympy.symbols('a1, a2, b')
x1_t = sympy.Function('x1_t')
# 微分方程ode
ode = sympy.Eq(x1_t(t).diff(t, t) + a1*x1_t(t).diff(t) + a2*x1_t(t), b)
init1 = sympy.Eq(x1_t(0), x1[0])
init2 = sympy.Eq(x1_t(1), x1[1])
# 调用dsolve函数,返回一个Eq对象，hint控制精度
ode_sol = sympy.dsolve((ode), x1_t(t))
print('得到带参符号解:', ode_sol)

# 初始条件字典
x1_to_value = {x1_t(0): 41, x1_t(x1.shape[0]-1): x1[-1]}
# 确定待定常数集
known_params = {a1, a2, b, t}
free_params = ode_sol.free_symbols - set(known_params)
# 将初始条件带入微分方程的后得到的多元方程
C_eqs = [sympy.Eq(ode_sol.lhs.subs(t, 0).subs(x1_to_value), ode_sol.rhs.subs(t, 0)),
         sympy.Eq(ode_sol.lhs.subs(t, x1.shape[0]).subs(x1_to_value), ode_sol.rhs.subs(t, x1.shape[0]))
        ]
print('待定常数: ', free_params)
print('关于待定常数的多元方程组:', C_eqs, sep='\n')
sol_params = sympy.solve(C_eqs, free_params)
print('解得待定常数:', sol_params)
final_sol_eq = sympy.Eq(ode_sol.lhs, ode_sol.rhs.subs(sol_params))
print('最终解:', final_sol_eq.lhs, '=', final_sol_eq.rhs)
```

    得到带参符号解: Eq(x1_t(t), C1*exp(t*(-a1 - sqrt(a1**2 - 4*a2))/2) + C2*exp(t*(-a1 + sqrt(a1**2 - 4*a2))/2) + b/a2)
    待定常数:  {C2, C1}
    关于待定常数的多元方程组:
    [Eq(41, C1 + C2 + b/a2), Eq(x1_t(6), C1*exp(-3*a1 - 3*sqrt(a1**2 - 4*a2)) + C2*exp(-3*a1 + 3*sqrt(a1**2 - 4*a2)) + b/a2)]
    解得待定常数: {C1: ((41*a2 - b)*exp(3*a1 + 6*sqrt(a1**2 - 4*a2)) - (a2*x1_t(6) - b)*exp(6*a1 + 3*sqrt(a1**2 - 4*a2)))*exp(-3*a1)/(a2*(exp(6*sqrt(a1**2 - 4*a2)) - 1)), C2: -((41*a2 - b)*exp(3*a1) - (a2*x1_t(6) - b)*exp(6*a1 + 3*sqrt(a1**2 - 4*a2)))*exp(-3*a1)/(a2*(exp(6*sqrt(a1**2 - 4*a2)) - 1))}
    最终解: x1_t(t) = b/a2 - ((41*a2 - b)*exp(3*a1) - (a2*x1_t(6) - b)*exp(6*a1 + 3*sqrt(a1**2 - 4*a2)))*exp(-3*a1)*exp(t*(-a1 + sqrt(a1**2 - 4*a2))/2)/(a2*(exp(6*sqrt(a1**2 - 4*a2)) - 1)) + ((41*a2 - b)*exp(3*a1 + 6*sqrt(a1**2 - 4*a2)) - (a2*x1_t(6) - b)*exp(6*a1 + 3*sqrt(a1**2 - 4*a2)))*exp(-3*a1)*exp(t*(-a1 - sqrt(a1**2 - 4*a2))/2)/(a2*(exp(6*sqrt(a1**2 - 4*a2)) - 1))

```python
# scipy.integrate.RK45
# 用龙格库塔法解一阶微分方程组

import numpy as np
import matplotlib.pyplot as plt
from scipy.integrate import RK45

def funcs(t, x):
    dxdt = np.zeros(2)
    a = 1e-8
    dxdt[0] = 0.05*x[0]*(1-x[0]/150000)-a*x[0]*x[1];
    dxdt[1] = 0.08*x[1]*(1-x[1]/400000)-a*x[0]*x[1];
    return dxdt

t0 = 0
y0 = [5000, 70000]
t_bound = 1000

rk45 = RK45(fun=funcs, t0=t0, y0=y0, t_bound=t_bound)

t = []
x = [[], []]

while rk45.status == 'running':
    rk45.step()
    t.append(rk45.t)
    x[0].append(rk45.y[0])
    x[1].append(rk45.y[1])
plt.subplot(1, 2, 1)
plt.plot(t, x[0], color='orange')
plt.plot(t, x[1], color='b')
plt.xlabel(u't')
plt.ylabel(u'x')
plt.subplot(1, 2, 2)
plt.scatter(x[0], x[1])
plt.xlabel(u'x_1')
plt.ylabel(u'x_2')

plt.tight_layout()
plt.show()
```


# The validatioin statistics of the posteriors
For the PIT and copPIT calculation and validation, I mainly borrow and adapt from the public code [galpo](https://github.com/smucesh/galpro/), which is based on [S. Mucesh et al. 2021](https://academic.oup.com/mnras/article/502/2/2770/6105325) and closely related to [Tilmann Gneiting et al. 2007](https://rss.onlinelibrary.wiley.com/doi/full/10.1111/j.1467-9868.2007.00587.x) and [Johanna F. Ziegel and Tilmann Gneiting. 2014](https://projecteuclid.org/journals/electronic-journal-of-statistics/volume-8/issue-2/Copula-calibration/10.1214/14-EJS964.full); For the HPD validation, I mainly refer to [Diana Harrison et al. 2015](https://academic.oup.com/mnras/article/451/3/2610/1186451).
## contents
1. The validation of the marginalized posterior for each parameter via PIT values, including the probablistic calibration and marginal calibration.
2. The validation of the multivariate posterior via the copPIT values, including the probablistic calibration and Kendall calibration.
3. The validation of the multivariate posterior via the HPD values, including the probablistic calibration.

```python
import numpy as np
from scipy.stats import norm, multivariate_normal
import matplotlib.pyplot as plt

from labellines import labelLine, labelLines
import glob
import pickle

from scipy.stats import entropy, kstest, cramervonmises, median_abs_deviation
import seaborn as sns
import statsmodels.api as sm

import warnings
warnings.filterwarnings("ignore")
```

##  PIT values

###  For the save of storage, here we use only 30 samples from HERA331 simulation to show the process of the posterior validation.

```python
f = open('data/diagnostics/hera_30.pkl', 'rb')
posterior_hera,prob_hera,true_hera,compressed_hera_true = pickle.load(f)
no_samples=len(true_hera)
```

```python
len(posterior_hera)
```

    30

```python
posterior_hera[0].shape
```

    (59136, 2)

### get metrics
A classical KS test is based on the distance measure $D$ defined by
\begin{equation}
D \equiv \max _{x}\left|S_{N}(x)-F(x)\right|,
\end{equation}
The CvM test share the similar spirit of the KS test and based on the measure $D^*$
\begin{equation}
D^* \equiv N \int_{-\infty}^{\infty}\left(S_{N}(x)-F(x)\right)^{2} \mathrm{~d} F(x).
\end{equation}

```python
#number of features
no_features=2
#number of points to generate the calibration plots
no_points = 100

path='data/'
plot_folder='diagnostics/'
```

```python
def get_pdf_metrics(pits, no_samples, no_features,  coppits=None):
    """Calculates performance metrics for PDFs."""

    pit_kst_p = np.empty(no_features)
    pit_cvm_p = np.empty(no_features)

    for feature in np.arange(no_features):
        pit_kst_p[feature] = kstest(pits[:, feature], 'uniform')[1]
        pit_cvm_p[feature] = cramervonmises(pits[:, feature], 'uniform').pvalue

    if coppits is not None:
        coppit_kst_p = kstest(coppits, 'uniform')[1]
        coppit_cvm_p = cramervonmises(coppits, 'uniform').pvalue
        return coppit_kst_p, coppit_cvm_p

    return pit_kst_p, pit_cvm_p
```

### probablistic calibration
We use the probability integral tran sform (PIT) to validate the marginal posteriors. The PIT is defined as follows:
\begin{equation}
	P I T \equiv \int_{-\infty}^{\tilde{\theta}} f(\theta) \mathrm{d} \theta,
\end{equation}
which is basically the CDF of the inferred marginal distribution $f(y)$ at the true values $\tilde{\theta}$ ($\log _ { 10 } \left( T_ { \rm vir } \right)$ or $\mathrm{log_{10}\zeta}$ in this work).
We check the uniformity of the PITs by the QQ plot.

```python
def probabilistic_calibration(posteriors,no_samples,y_test):
    """Performs probabilistic calibration"""

    pits = np.empty(( no_samples,  no_features))

    for sample in np.arange( no_samples):
        posterior =  posteriors[sample][:]
        pits[sample] = np.sum(posterior <=  y_test[sample], axis=0) / posterior.shape[0]

    print('Completed probabilistic calibration.')
    return pits
```

```python
pits_hera=probabilistic_calibration(posterior_hera,no_samples,true_hera)
```

    Completed probabilistic calibration.

```python
kst_p_hera, cvm_p_hera  = get_pdf_metrics(pits=pits_hera, no_samples=no_samples, no_features=no_features,
                                                   coppits=None)
```

```python
print (kst_p_hera, cvm_p_hera)
```

    [0.20441372 0.65692307] [0.26297231 0.43206736]

```python
target_features = ['$Tvir$', '$\zeta$']
pits=[pits_hera]
colors=['g']
label=['HERA331']
for feature in np.arange(no_features):
    fig,ax=plt.subplots(nrows=1,ncols=1)
    #for i in range(3):
    for i in range(1):
        qqplot = sm.qqplot(pits[i][:, feature], 'uniform', line='45').gca().lines
        qq_theory, qq_data = [qqplot[0].get_xdata(), qqplot[0].get_ydata()]
        plt.close()
        sns.lineplot(x=qq_theory, y=qq_data, ax=ax,color=colors[i],label=label[i])
    ax.plot([0, 1], [0, 1], color='black', linewidth=1, linestyle='--')

    ax.set_xlabel('$Q_{theory} \ | \ PIT \ ($' +  target_features[feature] + '$)$')
    ax.set_ylabel('$Q_{data}$')
    ax.set_xlim([0, 1])
    ax.set_ylim([0, 1])

    leg = plt.legend(loc='upper left')

    plt.savefig(path + plot_folder + 'var_' + str(feature) + '_pit.pdf',
                            bbox_inches='tight')

    plt.show()
    plt.close()
```

### marginal calibration
\usepackage{bbm}
The marginal calibratioin describes the comparison between the average predictive CDF
\begin{equation}
\hat{F}_{I}(\theta)\equiv\frac{1}{N} \sum_{i=1}^{N} F_{i}(\theta)\,,
\end{equation}
with the empirical CDF of the obvervations:
\begin{equation}
\tilde{G}_{I}(\theta)\equiv\frac{1}{N} \sum_{i=1}^{N} \mathbb{1}\left\{\tilde{\theta}_i \leq \theta \right\}\,,
\end{equation}
where the indicator function $\mathbb{1}\left\{\hat{\theta} \leq \theta \right\}$ equals 1 if the condition $\hat{\theta} \leq \theta$ is true, 0 otherwise.

```python
def marginal_calibration(posteriors,no_samples,y_test):
    """Performs marginal calibration"""

    points = np.linspace(np.floor(np.min( y_test, axis=0)), np.max( y_test, axis=0),  no_points)
    marginal_calibration = np.empty(( no_points,  no_features))

    for point in np.arange( no_points):
        probs = np.zeros( no_features)

        for sample in np.arange( no_samples):
            posterior =  posteriors[sample][:]
            probs += np.sum(posterior <= points[point], axis=0) / posterior.shape[0]

        pred_cdf_marg_point = probs /  no_samples
        true_cdf_marg_point = np.sum( y_test <= points[point], axis=0) /  no_samples
        marginal_calibration[point] = pred_cdf_marg_point - true_cdf_marg_point

    marginal_calibration = np.stack((points, marginal_calibration))
    print('Completed marginal calibration.')
    return marginal_calibration
```

```python
marginal_calibration_hera=marginal_calibration(posterior_hera,no_samples,true_hera)
```

    Completed marginal calibration.

```python
target_features = ['$Tvir$', '$\zeta$']
marginal_calibration=[marginal_calibration_hera]
colors=['g']
label=['HERA331']
for feature in np.arange( no_features):
    #for i in range(3):
    for i in range(1):
        sns.lineplot(x=marginal_calibration[i][0, :, feature], y=marginal_calibration[i][1, :, feature], color=colors[i],label=label[i])

    plt.axhline(0, color='black', linewidth=1, linestyle='--')
    plt.ylim([-np.max(np.abs(np.array(marginal_calibration)[:,1, :, feature])),
                  np.max(np.abs(np.array(marginal_calibration)[:,1, :, feature]))])
    plt.xlabel( target_features[feature])
    plt.ylabel(r'$\hat{F}_{I} - \tilde{G}_{I}$')
    plt.savefig(path + plot_folder + 'var_' + str(feature) +
                            '_marginal_calibration.pdf', bbox_inches='tight')
    plt.show()
    plt.close()
```

## copPIT values

### probablistic calibration
The probablistic copula calibration can be emprically accessed with the copula probability integral transformation (copPIT)
\begin{equation}
{\rm copPIT} \equiv \mathcal{K}_{H}(H(\tilde{\theta})),
\end{equation}
where $\mathcal{K}_{H}$ is the Kendall distribution function of a CDF $H(\theta)$:
\begin{equation}
\mathcal{K}_{H}(w)=\operatorname{Pr}\{H(\theta) \leq w\} \quad \text { for } \quad w \in[0,1],
\end{equation}
where the $\operatorname{pr}$ represent the probability. With this definition, the Kendall distribution function can be interpreted as the CDF of $H(\theta)$. So the coPIT is basically the Kendall distribution function evaluated at the "true" value $\tilde{\theta}$
\begin{equation}
{\rm copPIT} = \operatorname{pr}\{H(\theta) \leq H(\tilde{\theta})\}.
\end{equation}

```python
def create_templates(no_features):
    """Creates templates to perform multivariate calibration"""

    template = []
    template_same = []
    for feature in np.arange(no_features):

        if feature != (no_features - 1):
            template.append('(posterior[:,' + str(feature) + '] < posterior[pred,' + str(feature) + ']) & ')
            template_same.append('(posterior[:,' + str(feature) + '] == posterior[pred,' + str(feature) + ']) & ')
        else:
            template.append('(posterior[:,' + str(feature) + '] < posterior[pred,' + str(feature) + '])')
            template_same.append('(posterior[:,' + str(feature) + '] == posterior[pred,' + str(feature) + '])')

    template_pred = ''.join(template)
    template_same = ''.join(template_same)
    template_true = template_pred.replace('posterior[pred', 'y_test[sample')
    template_true = template_true.replace('<', '<=')

    return template_pred, template_true, template_same
```

```python
def probabilistic_copula_calibration(posteriors,no_samples,y_test):
    """Performs probabilistic copula calibration"""

    # Creating a list of list containing pred_cdf of each point in predictions
    pred_cdf_full = [[] for i in np.arange( no_samples)]
    true_cdf_full = []
    coppits = np.empty( no_samples)
    template_pred, template_true, template_same = create_templates(no_features= no_features)

    for sample in np.arange( no_samples):
        posterior =  posteriors[sample][:]
        no_preds = posterior.shape[0]

        for pred in np.arange(no_preds):
            # For point at edges, if <= used, then point counts and cdf is never 0.
            # If <= is used and the full MCMC chains are used, a large number of point will have near 0 probability, as a result, there will
            # be a peak at 0. The point in consideration does not count when determining cdf.
            # If we only use the accepted chains, this should not be the problem.
            same_preds = np.sum(eval(template_same))
            pred_cdf_full[sample].append(np.sum(eval(template_pred)) / (no_preds - same_preds))

        true_cdf_full.append(np.sum(eval(template_true)) / no_preds)
        coppits[sample] = np.sum(pred_cdf_full[sample] <= true_cdf_full[sample]) / no_preds

    print('Completed probabilistic copula calibration.')
    return coppits, pred_cdf_full, true_cdf_full
```

```python
coppits_hera, pred_cdf_full_hera, true_cdf_full_hera=probabilistic_copula_calibration(posterior_hera,no_samples,true_hera)
```

    Completed probabilistic copula calibration.

```python
kst_p_hera_coppit, cvm_p_hera_coppit  = get_pdf_metrics(pits=pits_hera, no_samples=no_samples, no_features=no_features,
                                                  coppits=coppits_hera)
```

```python
print (kst_p_hera_coppit, cvm_p_hera_coppit)
```

    0.5052413743119974 0.35191335343279806

```python
coppits=[coppits_hera]
colors=['g']
label=['HERA331']
fig,ax=plt.subplots(nrows=1,ncols=1)
#for i in range(3):
for i in range(1):
    qqplot = sm.qqplot(coppits[i], 'uniform', line='45').gca().lines
    qq_theory, qq_data = [qqplot[0].get_xdata(), qqplot[0].get_ydata()]
    plt.close()

    sns.lineplot(x=qq_theory, y=qq_data, ax=ax,color=colors[i],label=label[i])
ax.plot([0, 1], [0, 1], color='black', linewidth=1, linestyle='--')

ax.set_xlabel('$Q_{theory} \ | \ copPIT$')
ax.set_ylabel('$Q_{data}$')
ax.set_xlim([0, 1])
ax.set_ylim([0, 1])

leg = plt.legend(loc='upper left')

plt.savefig(path + plot_folder + 'coppit.pdf', bbox_inches='tight')

plt.show()
plt.close()
```

### Kendall calibration
The Kendall calibration can be accessed by comparing the average Kendall distribution function
\begin{equation}
\hat{\mathcal{K}}_{H_{I}}(w)=\frac{1}{n} \sum_{i=1}^{n} \mathcal{K}_{H_{i}}(w).
\end{equation}

with the emprical CDF of the joint CDFs evaluated on the ``true'' values $\tilde{\theta}$
\begin{equation}
\tilde{J}_{I}(w)=\frac{1}{n} \sum_{i=1}^{n} \mathbb{1}\left\{H_{i}\left(\tilde{\theta}_{i}\right) \leq w\right\},
\end{equation}

```python
def kendall_calibration(pred_cdf_full, true_cdf_full,no_samples):
    """Performs kendall calibration"""

    kendall_calibration = np.empty( no_points)
    count = 0

    for point in np.linspace(0, 1,  no_points):
        sum_ = np.zeros( no_samples)
        for sample in np.arange( no_samples):
            sum_[sample] = np.sum(pred_cdf_full[sample] <= point) / len(pred_cdf_full[sample])

        kendall_func_point = np.sum(sum_) /  no_samples
        true_cdf_point = np.sum(true_cdf_full <= point) /  no_samples
        kendall_calibration[count] = kendall_func_point - true_cdf_point
        count += 1

    print('Completed kendall calibration.')
    return kendall_calibration
```

```python
kendall_calibration_hera = kendall_calibration(pred_cdf_full_hera, true_cdf_full_hera,no_samples)
```

    Completed kendall calibration.

```python
kendall_calibration=[kendall_calibration_hera]
colors=['g']
label=['HERA331']
#for i in range(3):
for i in range(1):
    sns.lineplot(x=np.linspace(0, 1,  no_points), y=kendall_calibration[i], color=colors[i],label=label[i])

plt.axhline(0, color='black', linewidth=1, linestyle='--')
plt.xlim([0, 1])
plt.ylim([-np.max(np.abs(np.array(kendall_calibration))),
                  np.max(np.abs(np.array(kendall_calibration)))])
plt.xlabel('$w$')
plt.ylabel(r'$\mathcal{\hat{K}}_{H_{I}}  - \tilde{J}_{I}$')
plt.savefig(path + plot_folder + 'kendall_calibration.pdf', bbox_inches='tight')
plt.show()
plt.close()
```

## HPD values (hera, for other levels of signals the processes are similar)

```python
import numpy as np
import matplotlib.pyplot as plt
import pydelfi.priors as priors
import pydelfi.ndes as ndes
import pydelfi.delfi as delfi
import tensorflow as tf
```

    WARNING:tensorflow:From /home/zxs/miniconda3/envs/21cmfast/lib/python3.6/site-packages/pydelfi-0.1-py3.6.egg/pydelfi/train.py:9: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.

### probablistic calibration
The HPD is defined by
\begin{equation}
HPD(\boldsymbol{\tilde{\theta}})=\int_{f(\boldsymbol{\theta}) \geq f(\boldsymbol{\tilde{\theta}})} f(\boldsymbol{\theta}) \mathrm{d}^{n} \boldsymbol{\theta}.
\end{equation}

```python
#directory to load the trained ndes
diri = "data/diagnostics/trained_hera/"

theta_fiducial = true_hera[0]
compressed_data = compressed_hera_true[0]

nd=compressed_hera_true.shape[1]

lower = np.array([4,1])
upper = np.array([6,2.398])
prior = priors.Uniform(lower, upper)

NDEs = [ndes.ConditionalMaskedAutoregressiveFlow(n_parameters=2, n_data=nd, n_hiddens=[50,50], n_mades=5, act_fun=tf.tanh, index=0),
            ndes.ConditionalMaskedAutoregressiveFlow(n_parameters=2, n_data=nd, n_hiddens=[50,50], n_mades=5, act_fun=tf.tanh, index=1),
            ndes.ConditionalMaskedAutoregressiveFlow(n_parameters=2, n_data=nd, n_hiddens=[50,50], n_mades=5, act_fun=tf.tanh, index=2),
            ndes.ConditionalMaskedAutoregressiveFlow(n_parameters=2, n_data=nd, n_hiddens=[50,50], n_mades=5, act_fun=tf.tanh, index=3)]

DelfiEnsemble = delfi.Delfi(compressed_data, prior, NDEs,
                            Finv = None,
                            n_procs = 1,
                            theta_fiducial = theta_fiducial,
                            param_limits = [lower, upper],
                            param_names = ['\mathrm{log_{10}\,T_{vir}}', '\mathrm{log_{10}\,\zeta}'],
                            results_dir = diri,
                            progress_bar = False,
                            save = False,
                            restore = True,
                            input_normalization=None)
```

    WARNING:tensorflow:From /home/zxs/miniconda3/envs/21cmfast/lib/python3.6/site-packages/pydelfi-0.1-py3.6.egg/pydelfi/ndes.py:234: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

    WARNING:tensorflow:From /home/zxs/miniconda3/envs/21cmfast/lib/python3.6/site-packages/pydelfi-0.1-py3.6.egg/pydelfi/ndes.py:147: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead.

    WARNING:tensorflow:From /home/zxs/miniconda3/envs/21cmfast/lib/python3.6/site-packages/pydelfi-0.1-py3.6.egg/pydelfi/ndes.py:66: The name tf.losses.mean_squared_error is deprecated. Please use tf.compat.v1.losses.mean_squared_error instead.

    WARNING:tensorflow:From /home/zxs/miniconda3/envs/21cmfast/lib/python3.6/site-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where
    WARNING:tensorflow:From /home/zxs/miniconda3/envs/21cmfast/lib/python3.6/site-packages/pydelfi-0.1-py3.6.egg/pydelfi/delfi.py:74: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

    WARNING:tensorflow:From /home/zxs/miniconda3/envs/21cmfast/lib/python3.6/site-packages/pydelfi-0.1-py3.6.egg/pydelfi/delfi.py:74: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

    WARNING:tensorflow:From /home/zxs/miniconda3/envs/21cmfast/lib/python3.6/site-packages/pydelfi-0.1-py3.6.egg/pydelfi/delfi.py:176: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

    WARNING:tensorflow:From /home/zxs/miniconda3/envs/21cmfast/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use standard file APIs to check for files with this prefix.
    INFO:tensorflow:Restoring parameters from data/diagnostics/trained_hera/graph_checkpoint

```python
def probabilistic_hpd_calibration(posteriors,no_samples,y_test,compressed,prob):
    """
    Performs probabilistic calibration
    :posteriors: numpy array of a list of posteriors
    :no_samples: the number of validation samples
    :y_test: the true parameters of the validation samples
    :compressed: the mock observation data of the validation samples
    :prob: the probability of each point in the posterior chain (from MCMC sampling)
    """

    hpds = np.empty(( no_samples))
    for sample in np.arange( no_samples):
        posterior =  posteriors[sample][:]
        true_prob=DelfiEnsemble.log_posterior_stacked(y_test[sample], compressed[sample])
        hpds[sample] = np.sum(prob[sample]>=true_prob)/len(posterior)

    print('Completed probabilistic hpd calibration.')
    return hpds
```

```python
hpds_hera=probabilistic_hpd_calibration(posterior_hera,no_samples,true_hera,compressed_hera_true,prob_hera)
```

    Completed probabilistic hpd calibration.

```python
np.save('data/diagnostics/hpd_hera.npy',hpds_hera)
#hpds_hera=np.load('data/diagnostics/hpd_hera.npy')
```

```python
pit_kst_p = kstest(hpds_hera, 'uniform')[1]
pit_cvm_p = cramervonmises(hpds_hera, 'uniform').pvalue
print (pit_kst_p)
print (pit_cvm_p)
```

    0.17038226029338244
    0.19396852317676072

```python
hpds=[hpds_hera]
colors=['g']
label=['HERA331']
fig,ax=plt.subplots(nrows=1,ncols=1)
#for i in range(3):
for i in range(1):
    qqplot = sm.qqplot(hpds[i], 'uniform', line='45').gca().lines
    qq_theory, qq_data = [qqplot[0].get_xdata(), qqplot[0].get_ydata()]
    plt.close()

    sns.lineplot(x=qq_theory, y=qq_data, ax=ax,color=colors[i],label=label[i])
ax.plot([0, 1], [0, 1], color='black', linewidth=1, linestyle='--')

ax.set_xlabel('$Q_{theory} \ | \ hpd$')
ax.set_ylabel('$Q_{data}$')
ax.set_xlim([0, 1])
ax.set_ylim([0, 1])

leg = plt.legend(loc='upper left')

plt.savefig(path + plot_folder + 'hpd.pdf', bbox_inches='tight')

plt.show()
plt.close()
```

```python

```


```python
import matplotlib.pyplot as plt
import numpy as np
import scipy as sp
import scipy.linalg
import sympy as sy
sy.init_printing()
```

```python
np.set_printoptions(precision=3)
np.set_printoptions(suppress=True)
```

```python
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
```

```python
def round_expr(expr, num_digits):
    return expr.xreplace({n : round(n, num_digits) for n in expr.atoms(sy.Number)})
```

#  Computation of Determinants

For $2\times 2$ matrix $A$, the algorithm of determinant is
$$
  A=\left[\begin{matrix} a & b\cr c & d \end{matrix}\right] \qquad\text{is equivalent to}\qquad \text{det}  A=ad-bc
$$

Now we experiment with SymPy

```python
a, b, c, d, e, f, g, h, i = sy.symbols('a, b, c, d, e, f, g, h, i', real = True)
```

With defined symbols, the algorithms of $2\times 2$ and $3\times 3$ determinants are

```python
A = sy.Matrix([[a, b], [c, d]])
A.det()
```

```python
B = sy.Matrix([[a, b, c], [d, e, f], [g, h, i]])
B.det()
```

##  Cofactor Expansion

 The $(i,j)$-**cofactor** of $  A$ is denoted as $C_{ij}$ given by
 $$C_{i j}=(-1)^{i+j} \operatorname{det} A_{i j} = (-1)^{i+j} M_{ij}$$
 where $M_{ij}$ is the **minor determinant** which excludes $i$th row and $j$th column.

Consider a matrix $  A$
$$A=\left[\begin{array}{rrr}
1 & 5 & 0 \\
2 & 4 & -1 \\
0 & -2 & 0
\end{array}\right]$$
Any determinants can be expanded long its arbitrary row or column, we expand the determinant via the first row
$$\begin{aligned}
\operatorname{det} A &=1 \cdot \operatorname{det}\left[\begin{array}{rr}
4 & -1 \\
-2 & 0
\end{array}\right]-5 \cdot \operatorname{det}\left[\begin{array}{cc}
2 & -1 \\
0 & 0
\end{array}\right]+0 \cdot \operatorname{det}\left[\begin{array}{rr}
2 & 4 \\
0 & -2
\end{array}\right] \\
&=1(0-2)-5(0-0)+0(-4-0)=-2
\end{aligned}$$

The scalars, $1$, $-5$ and $0$, in front of each minor determinant are the elements of first row of $  A$.

In general, the expansions across $i$th row or $j$th column are
$$
\operatorname{det} A=a_{i 1} C_{i 1}+a_{i 2} C_{i 2}+\cdots+a_{i n} C_{i n}
$$
$$
\operatorname{det} A=a_{1 j} C_{1 j}+a_{2 j} C_{2 j}+\cdots+a_{n j} C_{n j}
$$

###  A SymPy Example of Determinant Expansion

Consider the matrix below and perform a cofactor expansion

```python
A = sy.Matrix([[49, 0, 61], [73, 22, 96], [2, 0, 32]]);A
```

$\displaystyle \left[\begin{matrix}49 & 0 & 61\\73 & 22 & 96\\2 & 0 & 32\end{matrix}\right]$

Cofactor expansion with the column which has two zero(s) involves the least computation burden:

$$\operatorname{det} A = a_{12}(-1)^{1+2}C_{12}+a_{22}(-1)^{2+2}C_{22}+a_{32}(-1)^{3+2}C_{32}$$

We can use SymPy function for calculationg minors: ```sy.matrices.matrices.MatrixDeterminant.minor(A, i, 1)```. Also we define a function for cofactor expansion:

```python
def cof_exp(matrix, c): # c is the col that we want to expand
    detA = 0
    for i in range(matrix.shape[1]): # matrix.shape[1] is the total number of columns
        detA += matrix[i, c]*(-1)**(i+c)*sy.matrices.matrices.MatrixDeterminant.minor(matrix, i, c)
    return detA
```

```python
cof_exp(A,1)
```

It's easy to verify the expansion algorithm Sympy's determinant evaluation function.

```python
A.det()
```

Actually you can experiment with any random matrices with multiple zeros, the function below has the parameter ```percent=70``` which means $70\%$ of element are non-zero.

```python
B = sy.randMatrix(r = 7, min=10, max=50, percent=70);B
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 16 & 14 & 48 & 0\\0 & 45 & 17 & 26 & 15 & 50 & 14\\0 & 31 & 31 & 0 & 0 & 33 & 43\\0 & 0 & 39 & 48 & 0 & 28 & 12\\42 & 41 & 25 & 36 & 50 & 25 & 19\\36 & 42 & 17 & 36 & 0 & 13 & 20\\0 & 11 & 0 & 16 & 27 & 36 & 19\end{matrix}\right]$

Calculate determinants with our user-defined function

```python
cof_exp(B,1)
```

Then verify the result of using determinant method  ```.det()```. We can see indeed cofactor expansion works!

```python
B.det()
```

**Minor matrices** can also be extracted by using code ```sy.matrices.matrices.MatrixDeterminant.minor_submatrix()```, for instance, the $M_{23}$ matrix of $  B$ is

```python
sy.matrices.matrices.MatrixDeterminant.minor_submatrix(B, 1, 2) # Pythonic indexing
```

$\displaystyle \left[\begin{matrix}0 & 0 & 16 & 14 & 48 & 0\\0 & 31 & 0 & 0 & 33 & 43\\0 & 0 & 48 & 0 & 28 & 12\\42 & 41 & 36 & 50 & 25 & 19\\36 & 42 & 36 & 0 & 13 & 20\\0 & 11 & 16 & 27 & 36 & 19\end{matrix}\right]$

**Cofactor matrix** is the matrix contain all cofactors of original matrix, and function ```.cofactor_matrix()``` can do the job.

$$A=\left[\begin{array}{rrr}
C_{11} & C_{12} & C_{13} \\
C_{21} & C_{22} & C_{23} \\
C_{31} & C_{32} & C_{33}
\end{array}\right]=
\left[\begin{array}{rrr}
(-1)^{1+1}M_{11} & (-1)^{1+2}M_{12} & (-1)^{1+3}M_{13} \\
(-1)^{2+1}M_{21} & (-1)^{2+2}M_{22} & (-1)^{2+3}M_{23} \\
(-1)^{3+1}M_{31} & (-1)^{3+2}M_{32} & (-1)^{3+3}M_{33}
\end{array}\right]
$$

```python
sy.matrices.matrices.MatrixDeterminant.cofactor_matrix(A)
```

$\displaystyle \left[\begin{matrix}704 & -2144 & -44\\0 & 1446 & 0\\-1342 & -251 & 1078\end{matrix}\right]$

##  Triangular Matrix

If $A$ is triangular matrix, cofactor expansion can be applied repetitively, the outcome will be a product of the elements on the principal diagonal.

$$
\operatorname{det A}_{n\times n} = \prod_{i=1}^n a_{ii}
$$

where $a_{ii}$ is the diagonal element.

Here is the proof, start with $A$

$$
A=\left[\begin{array}{cccc}
a_{11} & a_{12} & \cdots & a_{1 n} \\
& a_{22} & \cdots & a_{2 n} \\
& & \ddots & \\
& & & a_{n n}
\end{array}\right]
$$
Cofactor expanding on the first column,
$$
a_{11} \cdot \left[\begin{array}{cccc}
a_{22} & a_{22} & \cdots & a_{2 n} \\
& a_{33} & \cdots & a_{3 n} \\
& & \ddots & \\
& & & a_{n n}
\end{array}\right]
$$
Continue the cofactor expansion
$$
\operatorname{det} A=a_{11} a_{22} \cdot \left[\begin{array}{cccc}
a_{33} & a_{34} & \cdots & a_{3 n} \\
& a_{44} & \cdots & a_{4 n} \\
& & \ddots & \\
& & & a_{n n}
\end{array}\right]
$$
Iterating the expansion, eventually
$$
\text { Det } A=a_{11} \cdots a_{n-2, n-2} \cdot \left[\begin{array}{cc}
a_{n-1, n-1} & a_{n-1, n} \\
a_{n n}
\end{array}\right]=a_{11} \cdots a_{n n}
$$

Now let's verify with a numeric example, generate a random upper triangular matrix.

```python
A = np.round(np.random.rand(5,5)*100)
A_triu = np.triu(A); A_triu
```

    array([[73., 66., 34.,  9., 55.],
           [ 0.,  6., 65., 91., 54.],
           [ 0.,  0., 14., 15., 41.],
           [ 0.,  0.,  0., 85., 35.],
           [ 0.,  0.,  0.,  0., 24.]])

Compute the determinant with ```np.linalg.det```

```python
np.linalg.det(A_triu)
```

Extract the diagonal by ```np.diag()```, then calculate the product. The same results!

```python
A_diag = np.diag(A_triu)
np.prod(A_diag)
```

##  Properties of Determinants

Determinants has a long list of properties, however they are mostly derived facts from cofactor expansion. So no need to memorise any of them.

1. Let $A$ be an $n×n$ square matrix. If one row of $A$ is multiplied by $k$ to produce the matrix $B$, then: $\text{det} B = k\, \text{det} A$.
2. Let $A$ be an $n×n$ square matrix. If two rows of $A$ are interchanged to produce a matrix $B$ , then $\text{det}B=-\text{det}A$.
3. Let $A$ be an $n×n$ square matrix. If a multiple of one row of $A$ is added to another row to produce the matrix $B$, then $\text{det}A=\text{det}B$.
4. If $A$ is an $n×n$ matrix, then $\text{det}A^T = \text{det}A$
5. A square matrix $A$ is invertible if and only if $\text{det}A\neq 0$
6. If $A$ and $B$ are $n×n$ matrices, then $\text{det}AB = (\text{det}{A})(\text{det}{B})$
7. If $A$ is an $n×n$ matrix and $k$ is a scalar, then $\text{det}\,kA=k^n\,\text{det}A$.
8. If $A$ is an invertible square matrix, then $\text{det}A^{-1}=\frac{1}{\det A}$.

All of them are straightforward, the key is to demonstrate them via cofactor expansion. Here is some unrigorous proofs.

Proof of property 6:
$$\begin{aligned}
|A B| &=\left|E_{p} \cdots E_{1} B\right|=\left|E_{p}\right|\left|E_{p-1} \cdots E_{1} B\right|=\cdots \\
&=\left|E_{p}\right| \cdots\left|E_{1}\right||B|=\cdots=\left|E_{p} \cdots E_{1}\right||B| \\
&=|A||B|
\end{aligned}$$

Proof of property 7:

Because $\text{det} B = k\, \text{det} A$, one row of $A$ is multiplied by $k$ to produce $B$.Then multiply all the rows of $A$ by $k$, there will be $n$ $k$'s in front of $\text{det} A$, which is $k^n \text{det} A$

Proof of property 8:
$$\begin{aligned}
&\begin{aligned}
A A^{-1} &=I \\
\left|A A^{-1}\right| &=|I|
\end{aligned}\\
&|A|\left|A^{-1}\right|=1\\
&\left|A^{-1}\right|=\frac{1}{|A|}
\end{aligned}$$

These properties are useful in analytical derivation of some other theorems, however not efficient in numerical computation.

#  Cramer's Rule

If a linear system has $n$ equations and $n$ variables, an algorithm called **Cramer's Rule** can solve the system in terms of determinants as long as the solution is unique.
$$
A_{n\times n}\mathbf{b}_{n} = \mathbf{x}_n
$$

Some convenient notations are introduced here:

For any $A_{n\times n}$ and vector $\mathbf{b}$, denote $A_i(\mathbf{b})$ as the matrix obtained from replacing the $i$th column of $A$ by $\mathbf{b}$.

$$A_{i}(\mathbf{b})=\left[\begin{array}{lllll}
\mathbf{a}_{1} & \cdots & \mathbf{b} & \cdots & \mathbf{a}_{n}
\end{array}\right]$$

The Cramer's Rule can solve each $x_i$ without solving the whole system
$$x_{i}=\frac{\operatorname{det} A_{i}(\mathbf{b})}{\operatorname{det} A}, \quad i=1,2, \ldots, n$$

Fast Proof of Cramer's Rule：

$$\begin{aligned}
A \cdot I_{i}(\mathbf{x}) &=A\left[\mathbf{e}_{1} \quad \cdots \quad \mathbf{x} \quad \cdots \quad \mathbf{e}_{n}\right]=\left[\begin{array}{llllll}
A \mathbf{e}_{1} & \cdots & A \mathbf{x} & \cdots & A \mathbf{e}_{n}
\end{array}\right] \\
&=\left[\begin{array}{llllll}
\mathbf{a}_{1} & \cdots & \mathbf{b} & \cdots & \mathbf{a}_{n}
\end{array}\right]=A_{i}(\mathbf{b})
\end{aligned}$$

where $I_i(\mathbf{x})$ is an identity matrix whose $i$th column replaced by $\mathbf{x}$. With determinant's property,
$$(\operatorname{det} A)\left(\operatorname{det} I_{i}(\mathbf{x})\right)=\operatorname{det} A_{i}(\mathbf{b})$$

$\text{det}I_{i}(\mathbf{x})=x_i$, can be shown by cofactor expansion.

##  A NumPy Example On Cramer's Rule

Consider the system
$$\begin{aligned}
&2 x-y+3 z=-3\\
&3 x+3 y-z=10\\
&-x-y+z=-4
\end{aligned}$$

You have surely known several ways to solve it, but let's test if Cramer's rule works.

Input the matrices into NumPy arrays.

```python
A = np.array([[2, -1, 3], [3, 3, -1], [-1, -1, 1]])
b = np.array([-3, 10, -4])
```

```python
A_1b = np.copy(A) # Python variable is a reference tag
A_1b[:,0]=b

A_2b = np.copy(A)
A_2b[:,1]=b

A_3b = np.copy(A)
A_3b[:,2]=b
```

According to Cramer's rule:

```python
x1 = np.linalg.det(A_1b)/np.linalg.det(A)
x2 = np.linalg.det(A_2b)/np.linalg.det(A)
x3 = np.linalg.det(A_3b)/np.linalg.det(A)
(x1, x2, x3)
```

We can verify the results by NumPy built-in function ```np.linalg.solve```.

```python
np.linalg.solve(A, b)
```

    array([ 1.,  2., -1.])

Or in a straightforward way $A^{-1}b$

```python
np.linalg.inv(A)@b
```

    array([ 1.,  2., -1.])

All results are the same!

However, remember that Cramer's rule is rarely carried out in practice when solving system of equations, since computational cost (measured by the number of floating operations (flop), are much higher than Gaussian-Jordon elimination.

#  A Determinant Formula For $A^{-1}$

An alternative algorithm for $A^{-1}$ is
$$A^{-1}=\frac{1}{\operatorname{det} A}\left[\begin{array}{cccc}
C_{11} & C_{21} & \cdots & C_{n 1} \\
C_{12} & C_{22} & \cdots & C_{n 2} \\
\vdots & \vdots & & \vdots \\
C_{1 n} & C_{2 n} & \cdots & C_{n n}
\end{array}\right]$$

where the matrix of cofactors on RHS is the **adjugate** matrix, SymPy function is ```sy.matrices.matrices.MatrixDeterminant.adjugate```. And this is the transpose of the _cofactor matrix_ which we computed using ```sy.matrices.matrices.MatrixDeterminant.cofactor_matrix```

##  A SymPy Example

Generate a random matrix with $20\%$ of zero elements.

```python
A = sy.randMatrix(5, min=-5, max = 5, percent = 80); A
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0 & -4 & 2\\-1 & 0 & 0 & -2 & 0\\-3 & 0 & 1 & 5 & 0\\3 & 1 & 4 & 0 & -1\\1 & 1 & -2 & 1 & -4\end{matrix}\right]$

Compute the adjugate matrix

```python
A_adjugate = sy.matrices.matrices.MatrixDeterminant.adjugate(A); A_adjugate
```

$\displaystyle \left[\begin{matrix}-6 & -50 & -24 & 4 & -4\\221 & -39 & 140 & 18 & 106\\-33 & 35 & -8 & 22 & -22\\3 & -37 & 12 & -2 & 2\\71 & -49 & 36 & -6 & 6\end{matrix}\right]$

We can verify if this really the adjugate of $A$, we pick element of $(1, 3), (2, 4), (5, 4)$ of $A$ to compute the cofactors

```python
(-1)**(1+3) * sy.matrices.matrices.MatrixDeterminant.minor(A, 2, 0) # transpose the index
(-1)**(2+4) * sy.matrices.matrices.MatrixDeterminant.minor(A, 3, 1)
(-1)**(5+4) * sy.matrices.matrices.MatrixDeterminant.minor(A, 3, 4)
```

Adjugate is the transpose of cofactor matrix, thus we reverse the row and column index when referring to the elements in mind. They are correct as in adjugate matrix, so we have obtained the correct one.

To make results tidy, we need some extra steps.

The ```sy.N()``` is for converting to float approximation, i.e. if you don't like fractions.

```python
A_det = A.det()
A_inv = (1/A_det)*A_adjugate
round_expr(sy.N(A_inv), 4)
```

$\displaystyle \left[\begin{matrix}-0.0484 & -0.4032 & -0.1935 & 0.0323 & -0.0323\\1.7823 & -0.3145 & 1.129 & 0.1452 & 0.8548\\-0.2661 & 0.2823 & -0.0645 & 0.1774 & -0.1774\\0.0242 & -0.2984 & 0.0968 & -0.0161 & 0.0161\\0.5726 & -0.3952 & 0.2903 & -0.0484 & 0.0484\end{matrix}\right]$

Now again, we can verify the results with ```.inv()```

```python
round_expr(sy.N(A.inv()), 4)
```

$\displaystyle \left[\begin{matrix}-0.0484 & -0.4032 & -0.1935 & 0.0323 & -0.0323\\1.7823 & -0.3145 & 1.129 & 0.1452 & 0.8548\\-0.2661 & 0.2823 & -0.0645 & 0.1774 & -0.1774\\0.0242 & -0.2984 & 0.0968 & -0.0161 & 0.0161\\0.5726 & -0.3952 & 0.2903 & -0.0484 & 0.0484\end{matrix}\right]$

Or We can show by difference.

```python
A_inv-A.inv()
```

$\displaystyle \left[\begin{matrix}0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0\end{matrix}\right]$

So Cramer's rule indeed works perfectly.

Short Proof of $A^{-1}$ Formula With Determinants

We define $x$ the $j$th column of $A^{-1}$ which satisfies
$$
Ax= e_j
$$

and $e_j$ is the $j$th column of an identity matrix, and $j$th entry of $x$ is the $(i,j)$-entry of $A^{-1}$. By Cramer's rule,

$$\left\{(i, j) \text { -entry of } A^{-1}\right\}=x_{i}=\frac{\operatorname{det} A_{i}\left(\mathbf{e}_{j}\right)}{\operatorname{det} A}$$

The cofactor expansion along column $i$ of $A_i(e_j)$,
$$\operatorname{det} A_{i}\left(\mathbf{e}_{j}\right)=(-1)^{i+j} \operatorname{det} A_{j i}=C_{j i}$$


# Linear Cascaded Control

**NOTE** - completing this exercise without looking at the solution could take a few hours. Be prepared and good luck!

In this lesson we will be working with a simplified model of the 2D drone you worked with in the first lesson. We will be directly setting the vehicle's collective thrust through $u_1$ and the moment about the x axis, $M_x$ through $u_2$. We will ignore propeller rotation rates as well as the yaw-inducing moment $M_z$.

The state of this 2D drone can be described by the following vector:

$$X = [z , y, \phi, \dot{z}, \dot{y},\dot{\phi}]$$

```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

%load_ext autoreload
%autoreload 2

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.pylab as pylab

import trajectories
import simulate
import plotting

pylab.rcParams['figure.figsize'] = 10, 10
```

#### TODO 1 - Review Simplified `Drone2D` class
Walk through this code to refamiliarize youself with how state is represented, how controls are set, and what the underlying dynamics are.

```python
class Drone2D:
    """
    Simulates the dynamics of a drone confined to
    motion in the y-z plane.
    """
    def __init__(self,
                 I_x = 0.1, # moment of inertia around the x-axis
                 m = 0.2,   # mass of the vehicle
                ):

        self.I_x = I_x
        self.m = m

        self.u1 = 0.0 # collective thrust
        self.u2 = 0.0 # moment about the x axis
        self.g = 9.81

        # z, y, phi, z_dot, y_dot, phi_dot
        self.X = np.array([0.0,0.0,0.0,0.0,0.0,0.0])

    @property
    def y_dot_dot(self):
        phi = self.X[2]
        return self.u1 / self.m * np.sin(phi)

    @property
    def z_dot_dot(self):
        phi = self.X[2]
        return self.g - self.u1*np.cos(phi)/self.m

    @property
    def phi_dot_dot(self):
        return self.u2 / self.I_x

    def advance_state(self, dt):

        X_dot = np.array([self.X[3],
                        self.X[4],
                        self.X[5],
                        self.z_dot_dot,
                        self.y_dot_dot,
                        self.phi_dot_dot])

        # Change in state will be
        self.X = self.X + X_dot * dt
        return self.X

    def set_controls(self, u1, u2):
        self.u1 = u1
        self.u2 = u2
```

### Linear Cascading Controller

In this section, you will create the controller described in class. The schematic description of the cascading controller looks like this:

The controller architecture uses three **PD** controllers to control the drone (note that, for simplification, you will not need an I term for the controllers used in this exercise).

One PD controller is used to generate the thrust and control the altitude while two PD controllers are used to control the lateral motion. The first receives the desired lateral position information containing $y$ , $\dot{y}$ and (optionally) $\ddot{y}$. Using this information as well as the current vehicle state, it outputs the commanded $\phi$ roll angle.

The second PD controller is used to set a torque value $M_x$ to achieve the desired angle. In most cases, the inner controller responsible for roll angle operates at a higher frequency than the ones that are used to control the position.

#### TODO - Implement 2D controller with LINEAR control equations.

The linear equations of motion are shown below.

$$\begin{align}
\ddot{z} &= g - \frac{u_1}{m}
\\
\\
\ddot{y} &= g \phi
\\
\\
\ddot{\phi} &= \frac{u_2}{I_x}
\end{align}$$

These equations can be solved for $u_1$, $\phi_{\text{command}}$, and $u_2$.

$$\begin{align}
u_1 &= m(g - \bar{u_1})
\\
\\
\phi_{\text{command}} &= \frac{\ddot{y}_{\text{target}}}{g}
\\
\\
u_2 &= I_x \bar{u}_2
\end{align}$$

The first equation will be useful when implementing `altitude_controller`, the second when implementing `lateral_controller`, and the third when implementing `attitude_controller`.

Note that $\ddot{y}_{\text{target}}$ is like $\bar{u}_1$ or $\bar{u}_2$. It comes from PD control on the controller's inputs.

**Note the sequence of implementation, which should start from the INNER loop and then proceed with the OUTER loop.
In this specific case, you should start implementing the ATTITUDE controller first, then proceed with the LATERAL and ALTITUDE controller.**

```python
class LinearCascadingController:

    def __init__(self,
                 m,   # needed to convert u1_bar to u1
                 I_x, # needed to convert u2_bar to u2
                 z_k_p=1.0,
                 z_k_d=1.0,
                 y_k_p=1.0,
                 y_k_d=1.0,
                 phi_k_p=1.0,
                 phi_k_d=1.0):

        self.z_k_p = z_k_p
        self.z_k_d = z_k_d
        self.y_k_p = y_k_p
        self.y_k_d = y_k_d
        self.phi_k_p = phi_k_p
        self.phi_k_d = phi_k_d

        self.g = 9.81
        self.I_x = I_x
        self.m = m

    def altitude_controller(self,
                    z_target,
                    z_actual,
                    z_dot_target,
                    z_dot_actual,
                    z_dot_dot_target,
                    phi_actual, # unused parameter. Ignore for now.
                    ):
        """
        A PD controller which commands a thrust (u_1)
        for the vehicle.
        """

        # TODO (recommended to do AFTER attitude)
        #   Implement feedforward PD control to calculate
        #   u_1_bar and then use the linear math from above
        #   to transform u_1_bar into u_1 and then return u_1
        u_1_bar = self.z_k_p * (z_target - z_actual) + \
            self.z_k_d * (z_dot_target - z_dot_actual) + \
            z_dot_dot_target
        u_1 = self.m * (self.g - u_1_bar)
        return u_1

    def lateral_controller(self,
                        y_target,
                        y_actual,
                        y_dot_target,
                        y_dot_actual,
                        u_1=None, # unused parameter. Ignore for now.
                        y_dot_dot_ff=0.0,
                        ):
        """
        A PD controller which commands a target roll
        angle (phi_commanded).
        """

        # TODO (recommended to do AFTER attitude)
        #   Implement feedforward PD control to calculate
        #   y_dot_dot_target and then use the linear math from above
        #   to transform y_dot_dot_target into phi_commanded
        #   and then return phi_commanded
        y_dot_dot_target = self.y_k_p * (y_target - y_actual) \
            + self.y_k_d * (y_dot_target - y_dot_actual) \
            + y_dot_dot_ff
        phi_commanded = y_dot_dot_target / self.g
        return phi_commanded

    def attitude_controller(self,
                            phi_target,
                            phi_actual,
                            phi_dot_actual,
                            phi_dot_target=0.0
                           ):
        """
        A PD controller which commands a moment (u_2)
        about the x axis for the vehicle.
        """

        # TODO (recommended to do FIRST)
        #   Implement PD control to calculate u_2_bar
        #   and then use the linear math from above to
        #   transform u_2_bar into u_2 and then return u_2
        u_2 = self.I_x * (self.phi_k_p * (phi_target - phi_actual) +
            self.phi_k_d * (phi_dot_target - phi_dot_actual))
        return u_2
```

The flight path we'll use to test our controller is a figure 8 described as follows:
$$
\begin{align}
z & = a_z \sin{\omega_z t} \\
y & = a_y \cos{\omega_y t}
\end{align}
$$

where $\omega_y = \omega_z / 2$.

> NOTE - you can find the code that generates this trajectory in the file called `trajectories.py`, which you can access by clicking on the Jupyter logo in the top left corner of this notebook.

**NOTE on tuning:**

A good approach is to start with the inner loop, the ATTITUDE controller parameters (phi_k_p, phi_k_d) and the move forward with the LATERAL and ALTITUDE controller.

 1- increase phi_K_p until the drone is able to perform a shape smilar to the final path

 2- increase phi_k_d to align to the shape.

 3- Re-adjust phi_k_p until you get something similar to this:

 4- At this point you can move forward with the outer loop for LATERAL postion. Increase y_k_d in order to align the shape in the y direction. Such as:

 5- Slightly increase y_k_p.

 6- Now you can move on to the ALTITUDE controller and, as above, start increasing z_k_d and then slighlty increase z_k_p until you have a good overlap betwwen the planned and the executed path.

```python
# TESTING CELL
#
# Note - this cell will only give nice output when your code
#  is working AND you've tuned your parameters correctly.
#  you might find it helpful to come up with a strategy
#  for first testing the inner loop controller and THEN
#  testing the outer loop.
#
# Run this cell when you think your controller is ready!
#
# You'll have to tune the controller gains to get good results.

#### CONTROLLER GAINS (TUNE THESE) ######

z_k_p   = 10.0
z_k_d   = 1.0
y_k_p   = 1.3
y_k_d   = 0.1
phi_k_p = 35.0
phi_k_d = 1.5

#########################################

drone = Drone2D()

# INSTANTIATE CONTROLLER
linear_controller = LinearCascadingController(
    drone.m,
    drone.I_x,
    z_k_p=z_k_p,
    z_k_d=z_k_d,
    y_k_p=y_k_p,
    y_k_d=y_k_d,
    phi_k_p=phi_k_p,
    phi_k_d=phi_k_d
)

# TRAJECTORY PARAMETERS (you don't need to change these)
total_time = 100.0
omega_z = 1.0       # angular frequency of figure 8

# GENERATE FIGURE 8
z_traj, y_traj, t = trajectories.figure_8(omega_z, total_time, dt=0.02)
z_path, z_dot_path, z_dot_dot_path = z_traj
y_path, y_dot_path, y_dot_dot_path = y_traj

# SIMULATE MOTION
linear_history     = simulate.zy_flight(z_traj,
                                        y_traj,
                                        t,
                                        linear_controller,
                                        inner_loop_speed_up=10)
# PLOT RESULTS
plotting.plot_zy_flight_path(z_path, y_path, linear_history)
plotting.plot_z(linear_history, z_path, t)
plotting.plot_y(linear_history, y_path, t)
plotting.plot_phi(linear_history, t)
```

If everything is working correctly you should see a blue figure 8 that (nearly) overlaps with a red figure 8.

[Solution](/notebooks/Linear%20Controller%20Solution.ipynb)

```python

```