#### INFO7390 Adv Data Sci
## **Mini Project 2 - Causal Inference**

Name: Fangqing Wu
NUID: 001305642

---
**License**
This project is licensed under [MIT License](https://raw.githubusercontent.com/fwqfwq/dataset/master/MIT%20License)
\
\
**Links**
Google Colab: https://colab.research.google.com/drive/1zxApJ_wF1YlUdy08crKCe1zDxayAa2nm?usp=sharing

Dataset: Heart Failure Prediction https://www.kaggle.com/andrewmvd/heart-failure-clinical-data

 ---
### **Abstract**
'Correlations are not causations'.
Causal inference is a significant part in interpretability for machine learning. Not all high scores for correlations represent the very true causations, due to the confounders, or even, bias.

In this project the methods for causation and its sub-themes counterfactual would be applied and compared the different performance on regression models and the nerual network models.

---

### **Causal Inference**

This topic is talking after my mini1 project, interpretability.
Two parts are included as below:
* conception
* implementation on dataset

####**Conception**
'What If?'
This is the most representative expression for 'Causality", which is basically answering questions about why something happened.

**Causal Inference**, involving with cause & effect (treatment & outcomes), is an essential tool to extract information from data, analysing the response of the effect variable when the cause is changed, and to see whether the correlation is enough for inference.

Three-level hierarchy of causality[2] are:
1. Association  : Seeing
$P(y|x)$
Statistical relationship among data.

2. Intervention  : Doing
$P(y| do(x), z)$
'What if': hypothesis with a changed value and an unclear output.

3. Counterfactuals  : Imaging, Retrospecting
$P(y_{x}|x', y')$
Another 'what if', but in a retrosoecting way.

**Causal Flow Chart**
(Using causalgraphicalmodels)

```
!pip install causalgraphicalmodels
```

    Requirement already satisfied: causalgraphicalmodels in /usr/local/lib/python3.6/dist-packages (0.0.4)
    Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from causalgraphicalmodels) (1.1.4)
    Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from causalgraphicalmodels) (1.18.5)
    Requirement already satisfied: graphviz in /usr/local/lib/python3.6/dist-packages (from causalgraphicalmodels) (0.10.1)
    Requirement already satisfied: networkx in /usr/local/lib/python3.6/dist-packages (from causalgraphicalmodels) (2.5)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->causalgraphicalmodels) (2018.9)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas->causalgraphicalmodels) (2.8.1)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx->causalgraphicalmodels) (4.4.2)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.7.3->pandas->causalgraphicalmodels) (1.15.0)

```
from causalgraphicalmodels import CausalGraphicalModel
```

```
sprinkler = CausalGraphicalModel(
    nodes=["N1", "N21", "N22", "N3", "N4"],
    edges=[
        ("N1", "N21"),
        ("N1", "N22"),
        ("N21", "N3"),
        ("N22", "N3"),
        ("N3", "N4")
    ]
)

# draw return a graphviz `dot` object, which jupyter can render
sprinkler.draw()
```

***Fundamental Problem of Causal Inference***[3]

First, let's see Rubin's Model. [wiki](https://en.wikipedia.org/wiki/Rubin_causal_model)
The Rubin causal model is based on the idea of potential outcomes. For example, a person would have a particular income at age 40 if she had attended college, whereas she would have a different income at age 40 if she had not attended college. To measure the causal effect of going to college for this person, we need to compare the outcome for the same individual in both alternative futures. Since it is impossible to see both potential outcomes at once, one of the potential outcomes is always missing. This dilemma is the "fundamental problem of causal inference".

Take three elements as example X, T, Y:

X is the causation of T, T is the causation of Y.
Yet, still part of X related to Y. For example, a man with a poor health plan and a long-time smkoing habit, if he would be diagnosed with lung cancer, neither sole reason could be atrributed to arbitrarily.

The outcomes would be overlapped and not ideal for the measurement.

***Confounder***
When the aforementioned elements exist, the whole relationship among data is not 'pure', which is also called 'confounding' in statistics. Confoundings would be unmeasured when unobserved feature exists.

### **Implementation**

#### **Dataset Introduction**

The main data repository consists of:
>| Featrue | Description | | Featrue | Description | | Featrue | Description |
| ----------- | ----------- | | ----------- | ----------- | | ----------- | ----------- |
| 'age' | Age | | 'anaemia' | Decrease of red blood cells or hemoglobin (boolean) | | 'creatinine_phosphokinase' | Level of the CPK enzyme in the blood (mcg/L) |
| 'diabetes' | If the patient has diabetes (boolean) | | 'ejection_fraction' | Percentage of blood leaving the heart at each contraction (percentage) | | 'high_blood_pressure' | If the patient has hypertension (boolean)  |
| 'platelets' | Platelets in the blood (kiloplatelets/mL) | | 'serum_creatinine' | Level of serum creatinine in the blood (mg/dL) | | 'serum_sodium' | Level of serum sodium in the blood (mEq/L) |
| 'sex' | Woman or man (binary) || 'smoking' | If the patient smokes or not (boolean) |  | 'time' | Follow-up period (days) |
| 'DEATH_EVENT' | If the patient deceased during the follow-up period (boolean) |

Total with 13 features; and the target is \``DEATH_EVENT`\`.

#### **Dataset Exploration & Preprocessing**

0\. Import part

```
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import plotly.express as px
import warnings
import random
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn import metrics
import statsmodels.api as sm
import scipy.stats as stats

from sklearn.ensemble import RandomForestClassifier
from xgboost import XGBClassifier, plot_importance
from sklearn.metrics import r2_score, mean_squared_error, roc_curve, auc, roc_auc_score, classification_report, confusion_matrix
from sklearn.metrics.scorer import make_scorer

from datetime import datetime
random.seed(datetime.now())
warnings.filterwarnings('ignore')
```

    /usr/local/lib/python3.6/dist-packages/statsmodels/tools/_testing.py:19: FutureWarning:

    pandas.util.testing is deprecated. Use the functions in the public API at pandas.testing instead.

    /usr/local/lib/python3.6/dist-packages/sklearn/utils/deprecation.py:144: FutureWarning:

    The sklearn.metrics.scorer module is  deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.metrics. Anything that cannot be imported from sklearn.metrics is now part of the private API.

1\. First, a direct perception for dataset.

Load the dataset from github -- heart failure reords

```
url = 'https://raw.githubusercontent.com/fwqfwq/dataset/master/heart_failure_clinical_records_dataset.csv'
dataset = pd.read_csv(url)
print(dataset.columns.values)
```

    ['age' 'anaemia' 'creatinine_phosphokinase' 'diabetes' 'ejection_fraction'
     'high_blood_pressure' 'platelets' 'serum_creatinine' 'serum_sodium' 'sex'
     'smoking' 'time' 'DEATH_EVENT']

```
dataset.info()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 299 entries, 0 to 298
    Data columns (total 13 columns):
     #   Column                    Non-Null Count  Dtype
    ---  ------                    --------------  -----
     0   age                       299 non-null    float64
     1   anaemia                   299 non-null    int64
     2   creatinine_phosphokinase  299 non-null    int64
     3   diabetes                  299 non-null    int64
     4   ejection_fraction         299 non-null    int64
     5   high_blood_pressure       299 non-null    int64
     6   platelets                 299 non-null    float64
     7   serum_creatinine          299 non-null    float64
     8   serum_sodium              299 non-null    int64
     9   sex                       299 non-null    int64
     10  smoking                   299 non-null    int64
     11  time                      299 non-null    int64
     12  DEATH_EVENT               299 non-null    int64
    dtypes: float64(3), int64(10)
    memory usage: 30.5 KB

As shown, the records are all intact.

```
dataset.describe()
```

      age
      anaemia
      creatinine_phosphokinase
      diabetes
      ejection_fraction
      high_blood_pressure
      platelets
      serum_creatinine
      serum_sodium
      sex
      smoking
      time
      DEATH_EVENT

      count
      299.000000
      299.000000
      299.000000
      299.000000
      299.000000
      299.000000
      299.000000
      299.00000
      299.000000
      299.000000
      299.00000
      299.000000
      299.00000

      mean
      60.833893
      0.431438
      581.839465
      0.418060
      38.083612
      0.351171
      263358.029264
      1.39388
      136.625418
      0.648829
      0.32107
      130.260870
      0.32107

      std
      11.894809
      0.496107
      970.287881
      0.494067
      11.834841
      0.478136
      97804.236869
      1.03451
      4.412477
      0.478136
      0.46767
      77.614208
      0.46767

      min
      40.000000
      0.000000
      23.000000
      0.000000
      14.000000
      0.000000
      25100.000000
      0.50000
      113.000000
      0.000000
      0.00000
      4.000000
      0.00000

      25%
      51.000000
      0.000000
      116.500000
      0.000000
      30.000000
      0.000000
      212500.000000
      0.90000
      134.000000
      0.000000
      0.00000
      73.000000
      0.00000

      50%
      60.000000
      0.000000
      250.000000
      0.000000
      38.000000
      0.000000
      262000.000000
      1.10000
      137.000000
      1.000000
      0.00000
      115.000000
      0.00000

      75%
      70.000000
      1.000000
      582.000000
      1.000000
      45.000000
      1.000000
      303500.000000
      1.40000
      140.000000
      1.000000
      1.00000
      203.000000
      1.00000

      max
      95.000000
      1.000000
      7861.000000
      1.000000
      80.000000
      1.000000
      850000.000000
      9.40000
      148.000000
      1.000000
      1.00000
      285.000000
      1.00000

In this description, my concern is on the feature \``creatinine_phosphokinase`\` (CPK), with a seems-like abnormal distribution.

\

Then using the head(), to see 10examples of the dataset.

```
dataset.head(10)
```

      age
      anaemia
      creatinine_phosphokinase
      diabetes
      ejection_fraction
      high_blood_pressure
      platelets
      serum_creatinine
      serum_sodium
      sex
      smoking
      time
      DEATH_EVENT

      0
      75.0
      0
      582
      0
      20
      1
      265000.00
      1.9
      130
      1
      0
      4
      1

      1
      55.0
      0
      7861
      0
      38
      0
      263358.03
      1.1
      136
      1
      0
      6
      1

      2
      65.0
      0
      146
      0
      20
      0
      162000.00
      1.3
      129
      1
      1
      7
      1

      3
      50.0
      1
      111
      0
      20
      0
      210000.00
      1.9
      137
      1
      0
      7
      1

      4
      65.0
      1
      160
      1
      20
      0
      327000.00
      2.7
      116
      0
      0
      8
      1

      5
      90.0
      1
      47
      0
      40
      1
      204000.00
      2.1
      132
      1
      1
      8
      1

      6
      75.0
      1
      246
      0
      15
      0
      127000.00
      1.2
      137
      1
      0
      10
      1

      7
      60.0
      1
      315
      1
      60
      0
      454000.00
      1.1
      131
      1
      1
      10
      1

      8
      65.0
      0
      157
      0
      65
      0
      263358.03
      1.5
      138
      0
      0
      10
      1

      9
      80.0
      1
      123
      0
      35
      1
      388000.00
      9.4
      133
      1
      1
      10
      1

\
#### **Data Visulizaitons**
Several visualizations for disreibutions and relationship among this dataset.

1. Histograms for the sight of data distributions for all the features.
  *Using iloc function to exclude target feature \``DEATH_EVENT`\`(binary).*

```
fig = plt.figure(figsize = (10,10))
ax = fig.gca()
dataset.iloc[:,:-1].hist(ax = ax)
```

As shown,
  i. Binary features: \``anaemia`\`, \``diabetes`\`, \``high_blood_pressure`\`, \``sex`\`, \``smoking`\`
  ii. Normal Distribution features: \``platelets`\`, \``serum_sodium`\`
  iii. \``age`\` is close to the normal distribution, \``creatinine_phosphokinase`\`,\``ejection_fraction`\`, \``serum_creatinine`\` need a further exploration.

```
# Define a distribution function for features
def distributions(column, color, title):
  fig = px.histogram(
    dataset,
    x=column,
    color=color,
    title=title,
    width=400,
    height=350)
  fig.show()
```

2.  Distribution diagram for \``age`\`

```
distributions('age', 'sex', 'Age Distribution with Sex')
```

<meta charset="utf-8" />

Still distribute nomaly.

3. Distribution diagram for \``creatinine_phosphokinase`\`

```
distributions('creatinine_phosphokinase', 'DEATH_EVENT', 'CPK Distribution with Death Event')
```

<meta charset="utf-8" />

Some data is strange, outlier exists.

See in a boxplot for a clear view:

```
fig, ax = plt.subplots(figsize=(3, 5))
ax.boxplot(dataset['creatinine_phosphokinase'])
```

Create outlier feature based on boolean condition, and drop records.

```
dataset['outlier'] = np.where(dataset['creatinine_phosphokinase'] < 6000, 0, 1)
```

This dataset is rather smaill, for a better result without more risks for overfitting, set the bechmark to 6,000 rather than 4,000.

```
#Find all outlier
dataset.loc[dataset['outlier']==1]
```

      age
      anaemia
      creatinine_phosphokinase
      diabetes
      ejection_fraction
      high_blood_pressure
      platelets
      serum_creatinine
      serum_sodium
      sex
      smoking
      time
      DEATH_EVENT
      outlier

      1
      55.0
      0
      7861
      0
      38
      0
      263358.03
      1.1
      136
      1
      0
      6
      1
      1

      60
      45.0
      0
      7702
      1
      25
      1
      390000.00
      1.0
      139
      1
      0
      60
      1
      1

```
#Drop the outlier
dataset = dataset.loc[dataset['outlier']==0]
dataset.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 297 entries, 0 to 298
    Data columns (total 14 columns):
     #   Column                    Non-Null Count  Dtype
    ---  ------                    --------------  -----
     0   age                       297 non-null    float64
     1   anaemia                   297 non-null    int64
     2   creatinine_phosphokinase  297 non-null    int64
     3   diabetes                  297 non-null    int64
     4   ejection_fraction         297 non-null    int64
     5   high_blood_pressure       297 non-null    int64
     6   platelets                 297 non-null    float64
     7   serum_creatinine          297 non-null    float64
     8   serum_sodium              297 non-null    int64
     9   sex                       297 non-null    int64
     10  smoking                   297 non-null    int64
     11  time                      297 non-null    int64
     12  DEATH_EVENT               297 non-null    int64
     13  outlier                   297 non-null    int64
    dtypes: float64(3), int64(11)
    memory usage: 34.8 KB

Outliers dropped.

```
#Delete the assistance column
dataset = dataset.iloc[:, :-1]
```

4.  Distribution diagram for \``creatinine_phosphokinase`\`

```
distributions('ejection_fraction', 'DEATH_EVENT', 'Ejection Fraction Distribution with Death Event')
```

<meta charset="utf-8" />

Same as above, for not overfitting, ignore this single outlier. Plus, it's not very differently high from others.

5.  Distribution diagram for \``serum_creatinine`\`

```
distributions('serum_creatinine', 'DEATH_EVENT', 'Serum Creatinine Distribution with Death Event')
```

<meta charset="utf-8" />

Check with a boxplot too.

```
fig, ax = plt.subplots(figsize=(3, 5))
ax.boxplot(dataset['serum_creatinine'])
```

Drop the out lier.

```
dataset['outlier'] = np.where(dataset['serum_creatinine'] < 8, 0, 1)
```

```
dataset = dataset.loc[dataset['outlier']==0]
dataset.info()
```

    <class 'pandas.core.frame.DataFrame'>
    Int64Index: 295 entries, 0 to 298
    Data columns (total 14 columns):
     #   Column                    Non-Null Count  Dtype
    ---  ------                    --------------  -----
     0   age                       295 non-null    float64
     1   anaemia                   295 non-null    int64
     2   creatinine_phosphokinase  295 non-null    int64
     3   diabetes                  295 non-null    int64
     4   ejection_fraction         295 non-null    int64
     5   high_blood_pressure       295 non-null    int64
     6   platelets                 295 non-null    float64
     7   serum_creatinine          295 non-null    float64
     8   serum_sodium              295 non-null    int64
     9   sex                       295 non-null    int64
     10  smoking                   295 non-null    int64
     11  time                      295 non-null    int64
     12  DEATH_EVENT               295 non-null    int64
     13  outlier                   295 non-null    int64
    dtypes: float64(3), int64(11)
    memory usage: 34.6 KB

```
#Delete the assistance column
dataset = dataset.iloc[:, :-1]
```

6. Heatmap for correlations.

```
plt.figure(figsize=(5,5))

mask = np.zeros_like(dataset.corr())
mask[np.triu_indices_from(mask)] = True
with sns.axes_style("white"):
    ax = sns.heatmap(dataset.corr().round(2), mask=mask,
                     annot=True, cmap='rainbow', square=True)
```

As shown, the most related features with \``DEATH_EVENT`\` are \``age`\`, \``serum_creatinine`\`, \``time`\`. Then the  \``ejection_fraction`\` and  \``serum_sodium`\`.

As I believed, \``time`\` is highly related due to the factual logic, could not be considered here.

#### **Data Training & Prediction**

First, split the train and test data.

```
target = dataset.iloc[:,-1:]
data = dataset.iloc[:, :-1]

X_train, X_test, Y_train, Y_test = train_test_split(data, target, random_state=1)

print("X train shape: ", X_train.shape)
print("X test shape: ", X_test.shape)
print("Y train shape: ", Y_train.shape)
print("Y test shape: ", Y_test.shape)
```

    X train shape:  (221, 12)
    X test shape:  (74, 12)
    Y train shape:  (221, 1)
    Y test shape:  (74, 1)

In this part, two alogorithms would be appplied to train the models: one is Random Forests, and another is CNN.

1. Random Forest

Try different estimators and choose the best one.

```
scores_rf = []
models_rf = []
# Try different estimators
for i in range(50):
  rfm = RandomForestClassifier(max_depth=5, n_estimators=i+1)
  rfm.fit(X_train, Y_train)

  score = rfm.score(X_test, Y_test)
  scores_rf.append(score)
  models_rf.append(rfm)
```

Show all the scores(accuracy) for all the situations.

```
plt.scatter(range(1,51), scores_rf)
plt.title('Hyperparameters Tuning for Random Forests', weight='bold')
plt.xlabel('n_estimators')
plt.ylabel('Score')
```

Choose the best one (max score).

```
rf = models_rf[scores_rf.index(max(scores_rf))]
rf.fit(X_train, Y_train)

rf.score(X_test, Y_test)
```

    0.7972972972972973

Define the function for evaluations.

```
def eval_score(model, X_test, Y_test):

  predict = model.predict(X_test)

  # R2 Score
  r2 = r2_score(Y_test, predict)
  print('r2 score: ', r2)
  # Mean Squared Error
  mse = mean_squared_error(Y_test, predict)
  print('mse score: ', mse)

  # Confusion Matrix
  print('Confusion Matrix : \n', confusion_matrix(Y_test, predict))

  # ROC_AUC
  probility = model.predict_proba(X_test)
  # Slice out the probaility
  ras = roc_auc_score(Y_test, probility[:, 1])
  print('ROC AUC Score: ', ras)
  # ROC curve
  fpr, tpr, thresholds = roc_curve(Y_test, probility[:, 1])
  plt.plot(fpr, tpr)
  plt.plot([0,1], [0,1], 'grey', linestyle='--')
  plt.title('ROC Curve')
  plt.xlabel('False Positive Rates')
  plt.ylabel('True Positive Rates')

  # Report
  print('Report: ')
  print(classification_report(Y_test, predict))
```

Evaluatioon for the Random Forests model.

```
eval_score(rf, X_test, Y_test)
```

**Causality** part.

Install and import necessary package.

There're many differently functioned packages: causalml, CausalInference, CausalGraphicalModels, doWhy from Microsoft and so on. Many are still under developing.

1. doWhy

```
!pip install dowhy
```

    Requirement already satisfied: dowhy in /usr/local/lib/python3.6/dist-packages (0.5)
    Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from dowhy) (1.4.1)
    Requirement already satisfied: networkx>=2.0 in /usr/local/lib/python3.6/dist-packages (from dowhy) (2.5)
    Requirement already satisfied: sympy>=1.4 in /usr/local/lib/python3.6/dist-packages (from dowhy) (1.6.2)
    Requirement already satisfied: statsmodels in /usr/local/lib/python3.6/dist-packages (from dowhy) (0.10.2)
    Requirement already satisfied: pydot>=1.4 in /usr/local/lib/python3.6/dist-packages (from dowhy) (1.4.1)
    Requirement already satisfied: pandas>=0.24 in /usr/local/lib/python3.6/dist-packages (from dowhy) (1.1.4)
    Requirement already satisfied: numpy>=1.15 in /usr/local/lib/python3.6/dist-packages (from dowhy) (1.18.5)
    Requirement already satisfied: scikit-learn in /usr/local/lib/python3.6/dist-packages (from dowhy) (0.22.2.post1)
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from dowhy) (3.2.2)
    Requirement already satisfied: decorator>=4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx>=2.0->dowhy) (4.4.2)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.6/dist-packages (from sympy>=1.4->dowhy) (1.1.0)
    Requirement already satisfied: patsy>=0.4.0 in /usr/local/lib/python3.6/dist-packages (from statsmodels->dowhy) (0.5.1)
    Requirement already satisfied: pyparsing>=2.1.4 in /usr/local/lib/python3.6/dist-packages (from pydot>=1.4->dowhy) (2.4.7)
    Requirement already satisfied: python-dateutil>=2.7.3 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.24->dowhy) (2.8.1)
    Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.24->dowhy) (2018.9)
    Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn->dowhy) (0.17.0)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->dowhy) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->dowhy) (1.3.1)
    Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from patsy>=0.4.0->statsmodels->dowhy) (1.15.0)

```
from dowhy import CausalModel
```

What I meant to do here is to see the relationship between death rate and the smoking habit.
Choose 100 observations at random to see that in how many cases do the variables; ‘DEATH_EVENT’ & ‘smoking’ attain the same values. This whole process is then repeated 1000 times.

```
counts_sum=0
for i in range(1,1000):
        counts_i = 0
        rdf = dataset.sample(100)
        counts_i = rdf[rdf['DEATH_EVENT'] == rdf['smoking']].shape[0]
        counts_sum+= counts_i
counts_sum/1000
```

The expected count turns out to be 55.972% which is almost 50%.
\
Now consider the scenario when there were no changes and recalculate the expected count.

```
counts_sum=0
for i in range(1,1000):
        counts_i = 0
        rdf = dataset[dataset['anaemia'] >= 0].sample(100)
        counts_i = rdf[rdf['DEATH_EVENT']== rdf['smoking']].shape[0]
        counts_sum += counts_i
counts_sum/1000
```

The expected count turns out to be 56.175%.
\
It gives a hint that anaemia be a confounding variable.

**Create a Causal Graph**
Install the package.

```
# import sys, setuptools, tokenize
# !pip install pygraphviz
!apt-get install -y graphviz-dev
```

```
!pip install pygraphviz
```

```
import pygraphviz
causal_graph = """digraph {
DEATH_EVENT[label="Death_Event"];
smoking[label="Whether having smoking habit"];
anaemia[label="Whether having anaemia"];
diabetes[label="Whether having diabetes"];
high_blood_pressure[label="Whether having high_blood_pressure"];
time[label="Follow-up Time"];
U[label="Unobserved Confounders"];

smoking -> anaemia;
diabetes -> time;
anaemia -> time;
high_blood_pressure -> time;
time -> DEATH_EEVENT;
}"""

# sex[label="Sex"];
# creatinine_phosphokinase;
# ejection_fraction;
# platelets;
# serum_creatinine;
# serum_sodium;

```

```
import dowhy
model= dowhy.CausalModel(
        data = dataset,
        graph=causal_graph.replace("\n", " "),
        treatment='smoking',
        outcome='DEATH_EVENT')
model.view_model()
from IPython.display import Image, display
display(Image(filename="causal_model.png"))
```

### **Conclusion**

Causal Inference is a newly topic these years which is not easy to develop. The conception is not hard, but still how to define the treament for various situations is still a challenge. It still has a promising and influential in the future ML.

### **Reference**

[1] Explainable AI: Interpreting, Explaining and Visualizing Deep Learning https://link.springer.com/book/10.1007/978-3-030-28954-6
[2] The Three Layer Causal Hierarchy http://web.cs.ucla.edu/~kaoru/3-layer-causal-hierarchy.pdf
[3] Holland, Paul W. “Statistics and Causal Inference.” Journal of the American Statistical Association, vol. 81, no. 396, 1986, pp. 945–960. JSTOR, www.jstor.org/stable/2289064. Accessed 7 July 2020.
[4] doWhy https://microsoft.github.io/dowhy/index.html
[5] causalml https://causalml.readthedocs.io/en/latest/about.html


<p align="center">

</p>

## Data Analytics

### Parametric Distributions  in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Data Analytics: Parametric Distributions

Here's a demonstration of making and general use of parametric distributions in Python. This demonstration is part of the resources that I include for my courses in Spatial / Subsurface Data Analytics at the Cockrell School of Engineering at the University of Texas at Austin.

#### Parametric Distributions

We will cover the following distributions:

* Uniform
* Triangular
* Gaussian
* Log Normal

We will demonstrate:

* distribution parameters
* forward and inverse operators
* summary statistics

I have a lecture on these parametric distributions available on [YouTube](https://www.youtube.com/watch?v=U7fGsqCLPHU&t=1687s).

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - unconv_MV_v4.csv at https://git.io/fhHLT.

#### Importing Packages

We will need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
from scipy import stats                   # summary statistics
import math                               # trigonometry etc.
import scipy.signal as signal             # kernel for moving window calculation
import random                             # for randon numbers
import seaborn as sns                     # for matrix scatter plots
from scipy import linalg                  # for linear regression
from sklearn import preprocessing
import geostatspy.GSLIB as GSLIB
```

#### Set the Working Directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
#os.chdir("c:/PGE383")                     # set the working directory
```

### Uniform Distribution

Let's start with the most simple distribution.

* by default a random number is uniform distributed

* this ensures that enough random samples (Monte Carlo simulations) will reproduce the distribution

\begin{equation}
x_{\alpha}^{s} = F^{-1}_x(p_{\alpha}), \quad X^{s} \sim F_X
\end{equation}

#### Random Samples

Let's demonstrate the use of the command:

```python
uniform.rvs(size=n, loc = low, scale = interval, random_state = seed)
```

Where:

* size is the number of samples

* loc is the minimum value

* scale is the range, maximum value minus the minimum value

* random_state is the random number seed

We will observe the convergence of the samples to a uniform distribution as the number of samples becomes large.

We will make a compact set of code by looping over all the cases of number of samples

* we store the number of samples cases in the list called ns

* we store the samples as a list of lists, called X_uniform

```python
from scipy.stats import uniform

low = 0.00; interval = 0.30; ns = [1e1,1e2,1e3,1e4,1e5,1e6]; X_uniform = []

index = 0
for n in ns:
    X_uniform.append(uniform.rvs(size=int(ns[index]), loc = low, scale = interval).tolist())
    plt.subplot(2,3,index+1)
    GSLIB.hist_st(X_uniform[index],low,low+interval,log=False,cumul = False,bins=20,weights = None,xlabel='Values',title='Distribution, N = ' + str(int(ns[index])))
    index = index + 1

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=2.1, wspace=0.2, hspace=0.3)
```

We can observe that by drawing more Monte Carlo simulations, we more closely approximate the original uniform parametric distribution.

#### Forward Distribution

Let's demonstrate the forward operator. We can take any value and calculate the associated:

* density (probability density function)
* cumulative probability

The transform for the probability density function is:

\begin{equation}
p = f_x(x)
\end{equation}

where $f_x$ is the PDF and $p$ is the density for value, $x$.

and for the cumulative distribution function is:

\begin{equation}
P = F_x(x)
\end{equation}

where $F_x$ is the CDF and $P$ is the cumulative probability for value, $x$.

```python
x_values = np.linspace(0.0,0.3,100)
p_values = uniform.pdf(x_values, loc = low, scale = interval)
P_values = uniform.cdf(x_values, loc = low, scale = interval)

plt.subplot(1,2,1)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.8, label='uniform PDF'); plt.title('Uniform PDF'); plt.xlabel('Values'); plt.ylabel('Density')
plt.xlim([0.0,0.3]); plt.grid()

plt.subplot(1,2,2)
plt.plot(x_values, P_values,'r-', lw=5, alpha=0.8, label='uniform CDF'); plt.title('Uniform CDF'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')
plt.xlim([0.0,0.3]); plt.ylim([0,1]); plt.grid()

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.8, top=0.8, wspace=0.2, hspace=0.3); plt.show()
```

#### Inverse Distribution

Let's know demonstrate the reverse operator for the uniform distribution:

\begin{equation}
X = F^{-1}_X(P)
\end{equation}

```python
p = 0.90
x = uniform.ppf(p, loc = low, scale = interval)
print('The P' + str(round(p*100)) + ' of our uniform distribution is ' + str(x) + '.')
```

    The P90 of our uniform distribution is 0.27.

Let's repeat this over a range of percentile values, plot and compare to the forward operation result above.

```python
p_values = np.linspace(0.01,0.99,100)
x_values = uniform.ppf(p_values, loc = low, scale = interval)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.8, label='uniform pdf');  plt.title('Uniform CDF from Sampling the Inverse of Distribution, $F^{-1}_x(x)$'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')
plt.xlim([0.0,0.3]); plt.ylim([0,1]); plt.grid()

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.1, wspace=0.2, hspace=0.3); plt.show()
```

That work's the plots are the same.

#### Summary Statistics

We also have a couple of convience member functions to return the statistics from the parametric distribution:

* mean
* median
* mode
* variance
* standard deviation

Let's demonstrate a few of these methods.

```python
uniform.stats(loc = low, scale = interval, moments = 'mvsk')
```

returns a tuple with the mean, variance, skew and kurtosis (centered 1st, 2nd, 3rd and 4th moments)

```python
print('Stats: mean, variance, skew and kurtosis = ' + str(uniform.stats(loc = low, scale = interval, moments = 'mvsk')))
```

    Stats: mean, variance, skew and kurtosis = (array(0.15), array(0.0075), array(0.), array(-1.2))

We can confirm this by calculating the centered variance (regular variance) with this member function:

```python
uniform.var(loc = low, scale = interval)
```

```python
print('The variance is ' + str(round(uniform.var(loc = low, scale = interval),4)) + '.')
```

    The variance is 0.0075.

We can also directly calculate the:

* standard deviation - std
* mean - mean
* median - median

We can also calculate order of a non-centered moment. The moment method allows us to calculate an non-centered moment of any order. Try this out.

```python
m_order = 4
print('The ' + str(m_order) + 'th order non-centered moment is ' + str(uniform.moment(n = m_order, loc = low, scale = interval)))
```

    The 4th order non-centered moment is 0.00162

#### Symmetric Interval

We can also get the symmetric interval (e.g. prediction or confidence intervals) for any alpha level.

* Note the program mislabels the value as alpha, it is actually the significance level (1 - alpha)

```python
level = 0.95
print('The interval at alpha level ' + str(round(1-level,3)) + ' is ' + str(np.round(uniform.interval(alpha = level,loc = low,scale = interval),2)))
```

    The interval at alpha level 0.05 is [0.01 0.29]

#### Triangular Distribution

The great thing about parametric distributions is that the above member functions are the same!

* we can plug and play other parametric distributions and repeat the above.

This time we will make it much more compact!

* we will import the triangular distribution as my_dist and call the same functions as before
* we need a new parameter, the distribution mode (c parameter)

```python
from scipy.stats import triang as my_dist                                    # import traingular dist as my_dist
dist_type = 'Triangular'                                                     # give the name of the distribution for labels
low = 0.05; mode = 0.20; c = 0.10                                            # given the distribution parameters

x_values = np.linspace(0.0,0.3,100)                                          # get an array of x values
p_values = my_dist.pdf(x_values, loc = low, c = mode, scale = interval)      # calculate density for each x value
P_values = my_dist.cdf(x_values, loc = low, c = mode, scale = interval)      # calculate cumulative probablity for each x value

plt.subplot(1,3,1)                                                           # plot the resulting PDF
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.8); plt.title('Sampling the ' + str(dist_type) + ' PDF, $f_x(x)$'); plt.xlabel('Values'); plt.ylabel('Density')
plt.xlim([0.0,0.3])

plt.subplot(1,3,2)                                                           # plot the resulting CDF
plt.plot(x_values, P_values,'r-', lw=5, alpha=0.8); plt.title('Sampling the ' + str(dist_type) + ' CDF, $F_x(x)$'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')
plt.xlim([0.0,0.3])

p_values = np.linspace(0.00001,0.99999,100)                                 # get an array of p-values
x_values = my_dist.ppf(p_values, loc = low, c = mode, scale = interval)      # apply inverse to get x values from p-values
plt.subplot(1,3,3)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.8, label='uniform pdf')
plt.xlim([0.0,0.3])

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.1, wspace=0.2, hspace=0.3); plt.title('Sampling Inverse the ' + str(dist_type) + ' CDF, $F^{-1}_x(x)$'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

print('The mean is ' + str(round(my_dist.mean(loc = low,c = mode, scale = interval),4)) + '.') # calculate stats and symmetric interval
print('The variance is ' + str(round(my_dist.var(loc = low, c = mode, scale = interval),4)) + '.')
print('The interval at an alpha level of ' + str(round(1-level,3)) + ' is ' + str(np.round(my_dist.interval(alpha = level,loc = low,c = mode,scale = interval),2)))
```

The missing lower tail for the inverse is due to any value less than 0.05 having 0.0 cumulative probability.

#### Gaussian Distribution

Let's now use the Gaussian parametric distribution.

* we will need the parameters mean and the variance

We will apply the forward and reverse operations and calculate the summary statistics.

```python
from scipy.stats import norm as my_dist                                      # import traingular dist as my_dist
dist_type = 'Gaussian'                                                       # give the name of the distribution for labels
mean = 0.15; stdev = 0.05                                                    # given the distribution parameters

x_values = np.linspace(0.0,0.3,100)                                          # get an array of x values
p_values = my_dist.pdf(x_values, loc = mean, scale = stdev)                  # calculate density for each x value
P_values = my_dist.cdf(x_values, loc = mean, scale = stdev)                  # calculate cumulative probablity for each x value

plt.subplot(1,3,1)                                                           # plot the resulting PDF
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.8); plt.title('Sampling the ' + str(dist_type) + ' PDF, $f_x(x)$'); plt.xlabel('Values'); plt.ylabel('Density')
plt.xlim([0.0,0.3])

plt.subplot(1,3,2)                                                           # plot the resulting CDF
plt.plot(x_values, P_values,'r-', lw=5, alpha=0.8); plt.title('Sampling the ' + str(dist_type) + ' CDF, $F_x(x)$'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')
plt.xlim([0.0,0.3])

p_values = np.linspace(0.00001,0.99999,100)                                  # get an array of p-values
x_values = my_dist.ppf(p_values, loc = mean, scale = stdev)                  # apply inverse to get x values from p-values
plt.subplot(1,3,3)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.8, label='uniform pdf')
plt.xlim([0.0,0.3])

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.8, top=0.8, wspace=0.2, hspace=0.3); plt.title('Sampling Inverse the ' + str(dist_type) + ' CDF, $F^{-1}_x(x)$'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

print('The mean is ' + str(round(my_dist.mean(loc = mean, scale = stdev),4)) + '.') # calculate stats and symmetric interval
print('The variance is ' + str(round(my_dist.var(loc = mean, scale = stdev),4)) + '.')
print('The interval at an alpha level of ' + str(round(1-level,3)) + ' is ' + str(np.round(my_dist.interval(alpha = level,loc = mean,scale = stdev),2)))
```

#### Log Normal Distribution

Now let's check out the log normal distribution.

* We need the parameters $\mu$ and $\sigma$

```python
from scipy.stats import lognorm as my_dist                                   # import traingular dist as my_dist
dist_type = 'Log Normal'                                                     # give the name of the distribution for labels
mu = np.log(0.10); sigma = 0.2                                               # given the distribution parameters

x_values = np.linspace(0.0,0.3,100)                                          # get an array of x values
p_values = my_dist.pdf(x_values, s = sigma, scale = np.exp(mu))              # calculate density for each x value
P_values = my_dist.cdf(x_values, s = sigma, scale = np.exp(mu))              # calculate cumulative probablity for each x value

plt.subplot(1,3,1)                                                           # plot the resulting PDF
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.8); plt.title('Sampling the ' + str(dist_type) + ' PDF, $f_x(x)$'); plt.xlabel('Values'); plt.ylabel('Density')
plt.xlim([0.0,0.3])

plt.subplot(1,3,2)                                                           # plot the resulting CDF
plt.plot(x_values, P_values,'r-', lw=5, alpha=0.8); plt.title('Sampling the ' + str(dist_type) + ' CDF, $F_x(x)$'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')
plt.xlim([0.0,0.3])

p_values = np.linspace(0.00001,0.99999,100)                                        # get an array of p-values
x_values = my_dist.ppf(p_values,  s = sigma, scale = np.exp(mu))             # apply inverse to get x values from p-values
plt.subplot(1,3,3)
plt.plot(x_values, p_values,'r-', lw=5, alpha=0.8, label='uniform pdf')
plt.xlim([0.0,0.3])

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.8, top=0.8, wspace=0.2, hspace=0.3); plt.title('Sampling the Inverse the ' + str(dist_type) + ' CDF, $F^{-1}_x(x)$'); plt.xlabel('Values'); plt.ylabel('Cumulative Probability')

print('The mean is ' + str(round(my_dist.mean(s = sigma, scale = np.exp(mu)),4)) + '.') # calculate stats and symmetric interval
print('The variance is ' + str(round(my_dist.var(s = sigma, scale = np.exp(mu)),4)) + '.')
print('The interval at alpha level ' + str(round(1-level,3)) + ' is ' + str(np.round(my_dist.interval(alpha = level,s = sigma, scale = np.exp(mu)),2)))
```

There are many other parametric distributions that we could have included.  Also we could have demonstrated the distribution fitting.

#### Comments

This was a basic demonstration of working with parametric distributions.

I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at [Python Demos](https://github.com/GeostatsGuy/PythonNumericalDemos) and a Python package for data analytics and geostatistics at [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy).

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


# Computational Physics  Lectures: Ordinary differential equations

 **Morten Hjorth-Jensen**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State University

Date: **2016**

## Differential equations program
 * Ordinary differential equations, Runge-Kutta method,chapter 8

 * Ordinary differential equations with boundary conditions: one-variable equations to be solved by shooting and Green's function methods, chapter 9

 * We can solve such equations by a finite difference scheme as well, turning the equation into an eigenvalue problem. Still one variable. Done in projects 1 and 2.

 * If we have more than one variable, we need to solve  partial differential equations, see Chapter 10

The material on differential equations is covered by chapters 8, 9 and 10.
Two of  the final projects deal with ordinary differential equations.

## Differential Equations, chapter 8
The order of the ODE refers to the order of the derivative
on the left-hand side in the equation

$$
\begin{equation}
   \frac{dy}{dt}=f(t,y).
\label{_auto1} \tag{1}
\end{equation}
$$

This equation is of first order and $f$ is an arbitrary function.
A second-order equation goes typically like

$$
\begin{equation}
   \frac{d^2y}{dt^2}=f(t,\frac{dy}{dt},y).
\label{_auto2} \tag{2}
\end{equation}
$$

A well-known second-order equation is Newton's second law

$$
\begin{equation}
   m\frac{d^2x}{dt^2}=-kx,
\label{eq:newton} \tag{3}
\end{equation}
$$

where $k$ is the force constant. ODE depend only on one
variable

## Differential Equations
partial differential equations like the time-dependent Schr\"odinger
equation

$$
\begin{equation}
   i\hbar\frac{\partial \psi({\bf x},t)}{\partial t}=
   -\frac{\hbar^2}{2m}\left( \frac{\partial^2 \psi({\bf r},t)}{\partial x^2} +
                            \frac{\partial^2 \psi({\bf r},t)}{\partial y^2}+
                            \frac{\partial^2 \psi({\bf r},t)}{\partial z^2}\right) + V({\bf x})\psi({\bf x},t),
\label{_auto3} \tag{4}
\end{equation}
$$

may depend on several variables. In certain cases, like the above
equation, the wave function can be factorized in functions of the separate
variables, so that the Schroedinger equation
can be rewritten in terms of sets of ordinary differential equations.
These equations are discussed in chapter 10. Involve boundary conditions in addition to initial conditions.

## Differential Equations
We distinguish also between linear and non-linear differential
equation where for example

$$
\begin{equation}
   \frac{dy}{dt}=g^3(t)y(t),
\label{_auto4} \tag{5}
\end{equation}
$$

is an example of a linear equation, while

$$
\begin{equation}
   \frac{dy}{dt}=g^3(t)y(t)-g(t)y^2(t),
\label{_auto5} \tag{6}
\end{equation}
$$

is a non-linear ODE.

## Differential Equations
Another concept which dictates the numerical method chosen
for solving an ODE, is that of initial and boundary conditions.
To give an example, if we study white dwarf stars or neutron stars
we will need to solve two coupled first-order differential
equations, one for the total mass $m$ and one for the
pressure $P$ as functions of
$\rho$

$$
\frac{dm}{dr}=4\pi r^{2}\rho (r)/c^2,
$$

and

$$
\frac{dP}{dr}=-\frac{Gm(r)}{r^{2}}\rho (r)/c^2.
$$

where $\rho$ is the mass-energy density.
The initial conditions are dictated by the mass being
zero at the center of the star, i.e., when $r=0$,
yielding $m(r=0)=0$. The other condition is that
the pressure vanishes at the surface of the star.

In the solution of the Schroedinger equation for a particle
in a potential, we may need to apply boundary conditions as well,
such as demanding continuity of the wave function and its derivative.

## Differential Equations
In many cases it is possible to rewrite a second-order
differential equation in terms of two first-order differential
equations. Consider again the case of Newton's second law in Eq. [(3)](#eq:newton). If we define the position $x(t)=y^{(1)}(t)$
and the velocity $v(t)=y^{(2)}(t)$ as its derivative

$$
\begin{equation}
   \frac{dy^{(1)}(t)}{dt}=\frac{dx(t)}{dt}=y^{(2)}(t),
\label{_auto6} \tag{7}
\end{equation}
$$

we can rewrite Newton's second law as two coupled first-order
differential equations

$$
\begin{equation}
   m\frac{dy^{(2)}(t)}{dt}=-kx(t)=-ky^{(1)}(t),
\label{eq:n1} \tag{8}
\end{equation}
$$

and

$$
\begin{equation}
\frac{dy^{(1)}(t)}{dt}=y^{(2)}(t). \label{eq:n2} \tag{9}
\end{equation}
$$

## Differential Equations, Finite Difference
These methods fall under the general class of one-step methods.
The algoritm is rather simple.
Suppose we have an initial value for the function $y(t)$ given by

$$
\begin{equation}
  y_0=y(t=t_0).
\label{_auto7} \tag{10}
\end{equation}
$$

We are interested in solving a differential equation in a region
in space $[a,b]$. We define a step $h$ by splitting the interval
in $N$ sub intervals, so that we have

$$
\begin{equation}
  h=\frac{b-a}{N}.
\label{_auto8} \tag{11}
\end{equation}
$$

With this step and the derivative of $y$ we can construct the
next value of the function $y$ at

$$
\begin{equation}
   y_1=y(t_1=t_0+h),
\label{_auto9} \tag{12}
\end{equation}
$$

and so forth.

## Differential Equations
If the function is rather well-behaved in the domain
$[a,b]$, we can use a fixed step size. If not, adaptive steps
may be needed. Here we concentrate on fixed-step
methods only.
Let us try to generalize the above procedure by writing the
step $y_{i+1}$ in terms of the previous step $y_i$

$$
\begin{equation}
  y_{i+1}=y(t=t_i+h)=y(t_i) + h\Delta(t_i,y_i(t_i)) + O(h^{p+1}),
\label{_auto10} \tag{13}
\end{equation}
$$

where $O(h^{p+1})$ represents the truncation error. To determine
$\Delta$, we Taylor expand our function $y$

$$
\begin{equation}
     y_{i+1}=y(t=t_i+h)=y(t_i) + h(y'(t_i)+\dots +y^{(p)}(t_i)\frac{h^{p-1}}{p!}) + O(h^{p+1}), \label{eq:taylor} \tag{14}
\end{equation}
$$

where we will associate the derivatives in the parenthesis with

$$
\begin{equation}
\Delta(t_i,y_i(t_i))=(y'(t_i)+\dots +y^{(p)}(t_i)\frac{h^{p-1}}{p!}). \label{eq:delta} \tag{15}
\end{equation}
$$

## Differential Equations
We define

$$
\begin{equation}
  y'(t_i)=f(t_i,y_i)
\label{_auto11} \tag{16}
\end{equation}
$$

and if we truncate $\Delta$ at the first derivative, we have

$$
\begin{equation}
   y_{i+1}=y(t_i) + hf(t_i,y_i) + O(h^2), \label{eq:euler} \tag{17}
\end{equation}
$$

which when complemented with $t_{i+1}=t_i+h$ forms
the algorithm for the well-known Euler method.
Note that at every step we make an approximation error
of the order of $O(h^2)$, however the total error is the sum over all
steps $N=(b-a)/h$, yielding thus a global error which goes like
$NO(h^2)\approx O(h)$.

## Differential Equations
To make Euler's method more precise we can obviously
decrease $h$ (increase $N$). However, if we are computing the
derivative $f$ numerically
by for example the two-steps formula

$$
f'_{2c}(x)= \frac{f(x+h)-f(x)}{h}+O(h),
$$

we can enter into roundoff error problems when we subtract
two almost equal numbers $f(x+h)-f(x)\approx 0$.
Euler's method is not recommended for precision calculation,
although it is handy to use in order to get a first
view on how a solution may look like. As an example,
consider Newton's equation rewritten in Eqs. [(8)](#eq:n1) and [(9)](#eq:n2). We define $y_0=y^{(1)}(t=0)$
an $v_0=y^{(2)}(t=0)$. The first steps in Newton's equations
are then

$$
\begin{equation}
   y^{(1)}_1=y_0+hv_0+O(h^2)
\label{_auto12} \tag{18}
\end{equation}
$$

and

$$
\begin{equation}
      y^{(2)}_1=v_0-hy_0k/m+O(h^2).
\label{_auto13} \tag{19}
\end{equation}
$$

## Differential Equations
The Euler method is asymmetric in time, since it uses information about the derivative at the beginning
of the time interval. This means that we evaluate the position at $y^{(1)}_1$ using the velocity
at $y^{(2)}_0=v_0$. A simple variation is to determine $y^{(1)}_{n+1}$ using the velocity at
$y^{(2)}_{n+1}$, that is (in a slightly more generalized form)

$$
\begin{equation}
   y^{(1)}_{n+1}=y^{(1)}_{n}+h y^{(2)}_{n+1}+O(h^2)
\label{_auto14} \tag{20}
\end{equation}
$$

and

$$
\begin{equation}
   y^{(2)}_{n+1}=y^{(2)}_{n}+h a_{n}+O(h^2).
\label{_auto15} \tag{21}
\end{equation}
$$

The acceleration $a_n$ is a function of $a_n(y^{(1)}_{n}, y^{(2)}_{n},t)$ and needs to be evaluated
as well. This is the Euler-Cromer method.

## Differential Equations
Let us then include the second derivative in our Taylor expansion.
We have then

$$
\begin{equation}
 \Delta(t_i,y_i(t_i))=f(t_i)+\frac{h}{2}\frac{df(t_i,y_i)}{dt}+O(h^3).
\label{_auto16} \tag{22}
\end{equation}
$$

The second derivative can be rewritten as

$$
\begin{equation}
  y''=f'=\frac{df}{dt}=\frac{\partial f}{\partial t}+\frac{\partial f}{\partial y}\frac{\partial y}{\partial t}=\frac{\partial f}{\partial t}+\frac{\partial f}{\partial y}f \label{eq:derivatives} \tag{23}
\end{equation}
$$

and we can rewrite Eq.\ [(14)](#eq:taylor) as

$$
\begin{equation}
     y_{i+1}=y(t=t_i+h)=y(t_i) +hf(t_i)+
     \frac{h^2}{2}\left(\frac{\partial f}{\partial t}+\frac{\partial f}{\partial y}f\right) + O(h^{3  }),
\label{_auto17} \tag{24}
\end{equation}
$$

which has a local approximation error $O(h^{3  })$ and a global
error $O(h^{2})$.

## Differential Equations
These approximations can be generalized by using the derivative $f$ to
arbitrary order so that we have

$$
\begin{equation}
     y_{i+1}=y(t=t_i+h)=y(t_i) + h(f(t_i,y_i)+\dots f^{(p-1)}(t_i,y_i)
     \frac{h^{p-1}}{p!}) + O(h^{p+1}).
\label{_auto18} \tag{25}
\end{equation}
$$

These methods, based on higher-order derivatives, are in general not used
in numerical computation, since they rely on evaluating
derivatives several times. Unless one has analytical expressions
for these, the risk of roundoff errors is large.

## Differential Equations
The most obvious improvements to Euler's and Euler-Cromer's algorithms,
avoiding in addition the need for computing a
second derivative, is the so-called midpoint method. We have then

$$
\begin{equation}
   y^{(1)}_{n+1}=y^{(1)}_{n}+\frac{h}{2}\left(y^{(2)}_{n+1}+y^{(2)}_{n}\right)+O(h^2)
\label{_auto19} \tag{26}
\end{equation}
$$

and

$$
\begin{equation}
   y^{(2)}_{n+1}=y^{(2)}_{n}+h a_{n}+O(h^2),
\label{_auto20} \tag{27}
\end{equation}
$$

yielding

$$
\begin{equation}
   y^{(1)}_{n+1}=y^{(1)}_{n}+hy^{(2)}_{n}+\frac{h^2}{2}a_n+O(h^3)
\label{_auto21} \tag{28}
\end{equation}
$$

implying that the local truncation error in the position is now $O(h^3)$, whereas Euler's or Euler-Cromer's
methods have a local error of  $O(h^2)$.

## Differential Equations
Thus, the midpoint method yields a global error with
second-order accuracy for
the position and first-order accuracy for the velocity. However, although these methods yield exact results for
constant accelerations, the error increases in general with each time step.

One method that avoids this is the so-called half-step method. Here we define

$$
\begin{equation}
   y^{(2)}_{n+1/2}=y^{(2)}_{n-1/2}+h a_{n}+O(h^2),
\label{_auto22} \tag{29}
\end{equation}
$$

and

$$
\begin{equation}
   y^{(1)}_{n+1}=y^{(1)}_{n}+hy^{(2)}_{n+1/2} +O(h^2).
\label{_auto23} \tag{30}
\end{equation}
$$

Note that this method needs the calculation of $y^{(2)}_{1/2}$. This is done using
e.g., Euler's method

$$
\begin{equation}
   y^{(2)}_{1/2}=y^{(2)}_{0}+h a_{0}+O(h^2).
\label{_auto24} \tag{31}
\end{equation}
$$

As this method is numerically stable, it is often used instead of Euler's method.

## Differential Equations
Another method which one may encounter is the Euler-Richardson method
with

$$
\begin{equation}
   y^{(2)}_{n+1}=y^{(2)}_{n}+h a_{n+1/2}+O(h^2),
\label{eq:er1} \tag{32}
\end{equation}
$$

and

$$
\begin{equation}
\label{eq:er2} \tag{33}
   y^{(1)}_{n+1}=y^{(1)}_{n}+hy^{(2)}_{n+1/2} +O(h^2).
\end{equation}
$$

## The Verlet method
Another set of popular algorithms, which are both numerically stable and easy to implement are the
Verlet algorithms, with the velocity Verlet method as widely used in for example Molecular dynamics calculations.
Consider again a second-order differential equation  like Newton's second law, whose one-dimensional
version reads

$$
m\frac{d^2 x}{dt^2}= F(x,t),
$$

which we rewrite in terms of two coupled differential equations

$$
\frac{dx}{dt}=v(x,t) \hspace{1cm}\mathrm{and}\hspace{1cm} \frac{dv}{dt}=F(x,t)/m=a(x,t).
$$

## The Verlet method
If we now perform a Taylor expansion

$$
x(t+h) = x(t)+hx^{(1)}(t)+\frac{h^2}{2}x^{(2)}(t)+O(h^3).
$$

In our case the second derivative is known via Newton's second law, namely $x^{(2)}(t)=a(x,t)$.
If we add to the above equation the corresponding Taylor expansion for $x(t-h)$, we obtain, using the
discretized expressions

$$
x(t_i\pm h) = x_{i\pm 1} \hspace{1cm}\mathrm{and}\hspace{1cm} x_i=x(t_i),
$$

we arrive at

$$
x_{i+1} = 2x_i - x_{i-1} +h^2x^{(2)}_i+O(h^4).
$$

## The velocity Verlet method
We note that the truncation error goes like $O(h^4)$ since all the odd terms cancel when we add the two Taylor expansions.
We see also that the velocity is not directly included in the equation since the function
$x^{(2)}=a(x,t)$ is supposed to be known. If we need the velocity however, we can compute it using the well-known
formula

$$
x^{(1)}_i=\frac{x_{i+1}-x_{i-1}}{2h}+O(h^2).
$$

We note that the velocity has a truncation error which goes like $O(h^2)$. In for example so-called Molecular dynamics calculations,
since the acceleration is normally known via Newton's second law, there is seldomly a need for computing the velocity.

We note also that the algorithm for the position is not self-starting since, for $i=0$ it depends on the value of $x$
at the fictitious value $x_{-1}$.

We can amend this by introducing the velocity Verlet method.

## The velocity Verlet method
We have the Taylor expansion for the position given by

$$
x_{i+1} = x_i+hx^{(1)}_i+\frac{h^2}{2}x^{(2)}_i+O(h^3).
$$

The corresponding expansion for the velocity is

$$
v_{i+1} = v_i+hv^{(1)}_i+\frac{h^2}{2}v^{(2)}_i+O(h^3).
$$

Via Newton's second law we have normally an analytical expression for the derivative of the velocity, namely

$$
v^{(1)}_i = \frac{d^2 x}{dt^2}\vert_{i}= \frac{F(x_i,t_i)}{m},
$$

## The velocity Verlet method
If we add to this the corresponding expansion for the derivative of the velocity

$$
v^{(1)}_{i+1} = v^{(1)}_i+hv^{(2)}_i+O(h^2),
$$

and retain only terms up to the second derivative of the velocity since our error goes as $O(h^3)$, we have

$$
hv^{(2)}_i\approx v^{(1)}_{i+1}-v^{(1)}_i.
$$

We can then rewrite the Taylor expansion for the velocity as

$$
v_{i+1} = v_i+\frac{h}{2}\left( v^{(1)}_{i+1}+v^{(1)}_{i}\right)+O(h^3).
$$

## The velocity Verlet method
Our final equations for the position and the velocity become then

$$
x_{i+1} = x_i+hv_i+\frac{h^2}{2}v^{(1)}_{i}+O(h^3),
$$

and

$$
v_{i+1} = v_i+\frac{h}{2}\left( v^{(1)}_{i+1}+v^{(1)}_{i}\right)+O(h^3).
$$

Note well that the term $v^{(1)}_{i+1}$ depends on the position at $x_{i+1}$. This means that you need to calculate
the position at the updated time $t_{i+1}$ before the computing the next velocity.  Note also that the derivative of the velocity at the time
$t_i$ used in the updating of the position can be reused in the calculation of the velocity update as well.

## Differential Equations, Runge-Kutta methods
Runge-Kutta (RK) methods are based on Taylor expansion formulae, but yield
in general better algorithms for solutions of an ODE.
The basic philosophy is that it provides an intermediate step in the computation of $y_{i+1}$.

To see this, consider first the following definitions

$$
\begin{equation}
   \frac{dy}{dt}=f(t,y),
\label{_auto25} \tag{34}
\end{equation}
$$

and

$$
\begin{equation}
   y(t)=\int f(t,y) dt,
\label{_auto26} \tag{35}
\end{equation}
$$

and

$$
\begin{equation}
  y_{i+1}=y_i+ \int_{t_i}^{t_{i+1}} f(t,y) dt.
\label{_auto27} \tag{36}
\end{equation}
$$

## Differential Equations, Runge-Kutta methods
To demonstrate the philosophy behind RK methods, let us consider
the second-order RK method, RK2.
The first approximation consists in Taylor expanding $f(t,y)$
around the center of the integration interval $t_i$ to $t_{i+1}$,
that is, at $t_i+h/2$, $h$ being the step.
Using the midpoint formula for an integral,
defining $y(t_i+h/2) = y_{i+1/2}$ and
$t_i+h/2 = t_{i+1/2}$, we obtain

$$
\begin{equation}
    \int_{t_i}^{t_{i+1}} f(t,y) dt \approx hf(t_{i+1/2},y_{i+1/2}) +O(h^3).
\label{_auto28} \tag{37}
\end{equation}
$$

This means in turn that we have

$$
\begin{equation}
     y_{i+1}=y_i + hf(t_{i+1/2},y_{i+1/2}) +O(h^3).
\label{_auto29} \tag{38}
\end{equation}
$$

## Differential Equations, Runge-Kutta methods
However, we do not know the value of   $y_{i+1/2}$. Here comes thus the next approximation, namely, we use Euler's
method to approximate $y_{i+1/2}$. We have then

$$
\begin{equation}
   y_{(i+1/2)}=y_i + \frac{h}{2}\frac{dy}{dt} =
   y(t_i) + \frac{h}{2}f(t_i,y_i).
\label{_auto30} \tag{39}
\end{equation}
$$

This means that we can define the following algorithm for
the second-order Runge-Kutta method, RK2.

5
6

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\begin{equation}
  k_2=hf(t_{i+1/2},y_i+k_1/2),
\label{_auto32} \tag{41}
\end{equation}
$$

with the final value

$$
\begin{equation}
  y_{i+i}\approx y_i + k_2 +O(h^3).
\label{_auto33} \tag{42}
\end{equation}
$$

The difference between the previous one-step methods
is that we now need an intermediate step in our evaluation,
namely $t_i+h/2 = t_{(i+1/2)}$ where we evaluate the derivative $f$.
This involves more operations, but the gain is a better stability
in the solution.

## Differential Equations, Runge-Kutta methods
The fourth-order Runge-Kutta, RK4, which we will employ in the solution of various differential equations below, has the following algorithm

5
9

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
k_3=hf(t_i+h/2,y_i+k_2/2)\hspace{0.5cm}   k_4=hf(t_i+h,y_i+k_3)
$$

with the final result

$$
y_{i+1}=y_i +\frac{1}{6}\left( k_1 +2k_2+2k_3+k_4\right).
$$

Thus, the algorithm consists in first calculating $k_1$
with $t_i$, $y_1$ and $f$ as inputs. Thereafter, we increase the step
size by $h/2$ and calculate $k_2$, then $k_3$ and finally $k_4$. The global error goes as $O(h^4)$.

## Building a code for the solar system, gravitational force and constants
We start with a simpler case first, the Earth-Sun system  in two dimensions only.  The gravitational force $F_G$ is

$$
F=\frac{GM_{\odot}M_E}{r^2},
$$

where $G$ is the gravitational constant,

$$
M_E=6\times 10^{24}\mathrm{Kg},
$$

the mass of Earth,

$$
M_{\odot}=2\times 10^{30}\mathrm{Kg},
$$

the mass of the Sun and

$$
r=1.5\times 10^{11}\mathrm{m},
$$

is the distance between Earth and the Sun. The latter defines what we call an astronomical unit **AU**.
From Newton's second law we have then for the $x$ direction

$$
\frac{d^2x}{dt^2}=\frac{F_{x}}{M_E},
$$

and

$$
\frac{d^2y}{dt^2}=\frac{F_{y}}{M_E},
$$

for the $y$ direction.

## Building a code for the solar system, force equations
Introducing $x=r\cos{(\theta)}$, $y=r\sin{(\theta)}$ and

$$
r = \sqrt{x^2+y^2},
$$

we can rewrite

$$
F_{x}=-\frac{GM_{\odot}M_E}{r^2}\cos{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}x,
$$

and

$$
F_{y}=-\frac{GM_{\odot}M_E}{r^2}\sin{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}y,
$$

for the $y$ direction.

## Building a code for the solar system, coupled equations
We can rewrite these two equations

$$
F_{x}=-\frac{GM_{\odot}M_E}{r^2}\cos{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}x,
$$

and

$$
F_{y}=-\frac{GM_{\odot}M_E}{r^2}\sin{(\theta)}=-\frac{GM_{\odot}M_E}{r^3}y,
$$

as four first-order coupled differential equations

7
3

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

7
4

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

7
5

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\frac{dy}{dt}=v_y.
$$

## Building a code for the solar system, final coupled equations
The four coupled differential equations

7
7

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

7
8

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

7
9

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
\frac{dy}{dt}=v_y,
$$

can be turned into dimensionless equations (as we did in project 2) or we can introduce astronomical units with $1$ AU = $1.5\times 10^{11}$.

Using the equations from circular motion (with $r =1\mathrm{AU}$)

$$
\frac{M_E v^2}{r} = F = \frac{GM_{\odot}M_E}{r^2},
$$

we have

$$
GM_{\odot}=v^2r,
$$

and using that the velocity of Earth (assuming circular motion) is
$v = 2\pi r/\mathrm{yr}=2\pi\mathrm{AU}/\mathrm{yr}$, we have

$$
GM_{\odot}= v^2r = 4\pi^2 \frac{(\mathrm{AU})^3}{\mathrm{yr}^2}.
$$

## Building a code for the solar system, discretized equations
The four coupled differential equations can then be discretized using Euler's method as (with step length $h$)

8
4

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

8
5

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

8
6

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
y_{i+1}=y_i+hv_{y,i},
$$

## Building a code for the solar system, adding Jupiter
It is rather straightforward to add a new planet, say Jupiter.
Jupiter has mass

$$
M_J=1.9\times 10^{27}\mathrm{kg},
$$

and distance to the Sun of $5.2$ AU.
The additional gravitational force the Earth feels from Jupiter in the $x$-direction is

$$
F_{x}^{EJ}=-\frac{GM_JM_E}{r_{EJ}^3}(x_E-x_J),
$$

where $E$ stands for Earth, $J$ for Jupiter, $r_{EJ}$ is distance between Earth and Jupiter

$$
r_{EJ} = \sqrt{(x_E-x_J)^2+(y_E-y_J)^2},
$$

and $x_E$ and $y_E$ are the $x$ and $y$ coordinates of Earth, respectively, and
$x_J$ and $y_J$ are the $x$ and $y$ coordinates of Jupiter, respectively.
The $x$-component of the velocity of Earth changes thus to

$$
\frac{dv_x^E}{dt}=-\frac{GM_{\odot}}{r^3}x_E-\frac{GM_J}{r_{EJ}^3}(x_E-x_J).
$$

## Building a code for the solar system, adding Jupiter
We can rewrite

$$
\frac{dv_x^E}{dt}=-\frac{GM_{\odot}}{r^3}x_E-\frac{GM_J}{r_{EJ}^3}(x_E-x_J).
$$

to

$$
\frac{dv_x^E}{dt}=-\frac{4\pi^2}{r^3}x_E-\frac{4\pi^2M_J/M_{\odot}}{r_{EJ}^3}(x_E-x_J),
$$

where we used

$$
GM_J = GM_{\odot}\left(\frac{M_J}{M_{\odot}}\right)=4\pi^2 \frac{M_J}{M_{\odot}}.
$$

Similarly, for the velocity in $y$-direction we have

$$
\frac{dv_y^E}{dt}=-\frac{4\pi^2}{r^3}y_E-\frac{4\pi^2M_J/M_{\odot}}{r_{EJ}^3}(y_E-y_J).
$$

Similar expressions apply for Jupiter. The equations for $x$ and $y$
derivatives are unchanged. This equations are similar for all other planets and as we will see later, it will be convenient to object orient this part when we program the full solar system.

## How can I get the initial velocities and positions of the planets
[NASA](http://www.nasa.gov/index.html) has an excellent site at <http://ssd.jpl.nasa.gov/horizons.cgi#top>.
From there you can extract initial conditions in order to start your differential equation solver.
At the above website you need to change from **OBSERVER** to **VECTOR** and then write in the planet you are interested in.
NThe generated data contain the $x$, $y$ and $z$ values as well as their corresponding velocities. The velocities are in units of AU per day.
Alternatively they can be obtained in terms of km and km/s.

For the first simple system involving the Earth and the Sun, you could just initialize the position with say $x=1$ AU
and $y=0$ AU.

## Simple Example, Block tied to a Wall
Our first example is the classical case of simple harmonic oscillations, namely a block sliding on a horizontal frictionless surface. The block is tied to a wall with a spring. If the spring is not compressed or stretched too far, the force on the block at a given position $x$ is

$$
F=-kx.
$$

The negative sign means that the force acts to restore the object to an equilibrium position. Newton's equation of motion for this idealized system is then

$$
m\frac{d^2x}{dt^2}=-kx,
$$

or we could rephrase it as

$$
\frac{d^2x}{dt^2}=-\frac{k}{m}x=-\omega_0^2x,
\label{eq:newton1} \tag{43}
$$

with the angular frequency $\omega_0^2=k/m$.

The above differential equation has the advantage that it can be solved  analytically with solutions on the form

$$
x(t)=Acos(\omega_0t+\nu),
$$

where $A$ is the amplitude and $\nu$ the phase constant.   This provides in turn an important test for the numerical
solution and the development of a program for more complicated cases which cannot be solved analytically.

## Simple Example, Block tied to a Wall
With the position $x(t)$ and the velocity  $v(t)=dx/dt$ we can reformulate Newton's equation in the following way

$$
\frac{dx(t)}{dt}=v(t),
$$

and

$$
\frac{dv(t)}{dt}=-\omega_0^2x(t).
$$

We are now going to solve these equations using the Runge-Kutta method
  to fourth order discussed previously.

## Simple Example, Block tied to a Wall
Before proceeding however, it is important to note that in addition to the exact solution, we have at least two further tests which can be used to check our solution.

Since functions like $cos$ are periodic with a period $2\pi$, then the solution $x(t)$ has also to be periodic. This means that

$$
x(t+T)=x(t),
$$

with $T$ the period defined as

$$
T=\frac{2\pi}{\omega_0}=\frac{2\pi}{\sqrt{k/m}}.
$$

Observe that $T$ depends only on $k/m$ and not on the amplitude of the solution.

## Simple Example, Block tied to a Wall
In addition to the periodicity test, the total energy has also to be conserved.

Suppose we choose the initial conditions

$$
x(t=0)=1\hspace{0.1cm} \mathrm{m}\hspace{1cm} v(t=0)=0\hspace{0.1cm}\mathrm{m/s},
$$

meaning that block is at rest at $t=0$ but with a potential energy

$$
E_0=\frac{1}{2}kx(t=0)^2=\frac{1}{2}k.
$$

The total energy at any time $t$ has however to be conserved, meaning that our solution has to fulfil the condition

$$
E_0=\frac{1}{2}kx(t)^2+\frac{1}{2}mv(t)^2.
$$

## Simple Example, Block tied to a Wall
An algorithm which implements these equations is included below.
 * Choose the initial position and speed, with the most common choice $v(t=0)=0$ and some fixed value for the position.

 * Choose the method you wish to employ in solving the problem.

 * Subdivide the time interval $[t_i,t_f] $ into a grid with step size

$$
h=\frac{t_f-t_i}{N},
$$

where $N$ is the number of mesh points.
 * Calculate now the total energy given by

$$
E_0=\frac{1}{2}kx(t=0)^2=\frac{1}{2}k.
$$

* The Runge-Kutta method is used to obtain $x_{i+1}$ and $v_{i+1}$ starting from the previous values $x_i$ and $v_i$.

 * When we have computed $x(v)_{i+1}$ we upgrade  $t_{i+1}=t_i+h$.

 * This iterative  process continues till we reach the maximum time $t_f$.

 * The results are checked against the exact solution. Furthermore, one has to check the stability of the numerical solution against the chosen number of mesh points $N$.

## Simple Example, Block tied to a Wall
To run a c++ program using ipython notebook, you can use the following statements.

0

<
<
<
!
!
C
O
D
E
_
B
L
O
C
K

p
y
s
c
p
r
o

```
%%cpp
/*    This program solves Newton's equation for a block
      sliding on a horizontal frictionless surface. The block
      is tied  to a wall with a spring, and Newton's equation
      takes the form
           m d^2x/dt^2 =-kx
      with k the spring tension and m the mass of the block.
      The angular frequency is omega^2 = k/m and we set it equal
      1 in this example program.

      Newton's equation is rewritten as two coupled differential
      equations, one for the position x  and one for the velocity v
           dx/dt = v    and
           dv/dt = -x   when we set k/m=1

      We use therefore a two-dimensional array to represent x and v
      as functions of t
      y[0] == x
      y[1] == v
      dy[0]/dt = v
      dy[1]/dt = -x

      The derivatives are calculated by the user defined function
      derivatives.

      The user has to specify the initial velocity (usually v_0=0)
      the number of steps and the initial position. In the programme
      below we fix the time interval [a,b] to [0,2*pi].

*/
#include <cmath>
#include <iostream>
#include <fstream>
#include <iomanip>
//#include "lib.h"
using namespace  std;
// output file as global variable
ofstream ofile;
// function declarations
void derivatives(double, double *, double *);
void initialise ( double&, double&, int&);
void output( double, double *, double);
void runge_kutta_4(double *, double *, int, double, double,
                   double *, void (*)(double, double *, double *));

int main(int argc, char* argv[])
{
//  declarations of variables
  double *y, *dydt, *yout, t, h, tmax, E0;
  double initial_x, initial_v;
  int i, number_of_steps, n;
  char *outfilename;
  // Read in output file, abort if there are too few command-line arguments
  if( argc <= 1 ){
    cout << "Bad Usage: " << argv[0] <<
      " read also output file on same line" << endl;
    //    exit(1);
  }
  else{
    outfilename=argv[1];
  }
  ofile.open(outfilename);
  //  this is the number of differential equations
  n = 2;
  //  allocate space in memory for the arrays containing the derivatives
  dydt = new double[n];
  y = new double[n];
  yout = new double[n];
  // read in the initial position, velocity and number of steps
  initialise (initial_x, initial_v, number_of_steps);
  //  setting initial values, step size and max time tmax
  h = 4.*acos(-1.)/( (double) number_of_steps);   // the step size
  tmax = h*number_of_steps;               // the final time
  y[0] = initial_x;                       // initial position
  y[1] = initial_v;                       // initial velocity
  t=0.;                                   // initial time
  E0 = 0.5*y[0]*y[0]+0.5*y[1]*y[1];       // the initial total energy
  // now we start solving the differential equations using the RK4 method
  while (t <= tmax){
    derivatives(t, y, dydt);   // initial derivatives
    runge_kutta_4(y, dydt, n, t, h, yout, derivatives);
    for (i = 0; i < n; i++) {
      y[i] = yout[i];
    }
    t += h;
    output(t, y, E0);   // write to file
  }
  delete [] y; delete [] dydt; delete [] yout;
  ofile.close();  // close output file
  return 0;
}   //  End of main function

//     Read in from screen the number of steps,
//     initial position and initial speed
void initialise (double& initial_x, double& initial_v, int& number_of_steps)
{
 cout << "Initial position = ";
 cin >> initial_x;
 cout << "Initial speed = ";
 cin >> initial_v;
 cout << "Number of steps = ";
 cin >> number_of_steps;
}  // end of function initialise

//   this function sets up the derivatives for this special case
void derivatives(double t, double *y, double *dydt)
{
  dydt[0]=y[1];    // derivative of x
  dydt[1]=-y[0]; // derivative of v
} // end of function derivatives

//    function to write out the final results
void output(double t, double *y, double E0)
{
  ofile << setiosflags(ios::showpoint | ios::uppercase);
  Ofile << setw(15) << setprecision(8) << t;
  ofile << setw(15) << setprecision(8) << y[0];
  ofile << setw(15) << setprecision(8) << y[1];
  ofile << setw(15) << setprecision(8) << cos(t);
  ofile << setw(15) << setprecision(8) <<
    0.5*y[0]*y[0]+0.5*y[1]*y[1]-E0 << endl;
}  // end of function output

/*   This function upgrades a function y (input as a pointer)
     and returns the result yout, also as a pointer. Note that
     these variables are declared as arrays.  It also receives as
     input the starting value for the derivatives in the pointer
     dydx. It receives also the variable n which represents the
     number of differential equations, the step size h and
     the initial value of x. It receives also the name of the
     function *derivs where the given derivative is computed
*/
void runge_kutta_4(double *y, double *dydx, int n, double x, double h,
                   double *yout, void (*derivs)(double, double *, double *))
{
  int i;
  double      xh,hh,h6;
  double *dym, *dyt, *yt;
  //   allocate space for local vectors
  dym = new double [n];
  dyt =  new double [n];
  yt =  new double [n];
  hh = h*0.5;
  h6 = h/6.;
  xh = x+hh;
  for (i = 0; i < n; i++) {
    yt[i] = y[i]+hh*dydx[i];
  }
  (*derivs)(xh,yt,dyt);     // computation of k2, eq. 3.60
  for (i = 0; i < n; i++) {
    yt[i] = y[i]+hh*dyt[i];
  }
  (*derivs)(xh,yt,dym); //  computation of k3, eq. 3.61
  for (i=0; i < n; i++) {
    yt[i] = y[i]+h*dym[i];
    dym[i] += dyt[i];
  }
  (*derivs)(x+h,yt,dyt);    // computation of k4, eq. 3.62
  //      now we upgrade y in the array yout
  for (i = 0; i < n; i++){
    yout[i] = y[i]+h6*(dydx[i]+dyt[i]+2.0*dym[i]);
  }
  delete []dym;
  delete [] dyt;
  delete [] yt;
}       //  end of function Runge-kutta 4
```

## Simple Example, Block tied to a Wall, python code
The following python program performs essentially the same calculations as the previous c++ code.

```
#
# This program solves Newtons equation for a block sliding on
# an horizontal frictionless surface.
# The block is tied to the wall with a spring, so N's eq takes the form:
#
#  m d^2x/dt^2 = - kx
#
# In order to make the solution dimless, we set k/m = 1.
# This results in two coupled diff. eq's that may be written as:
#
#  dx/dt = v
#  dv/dt = -x
#
# The user has to specify the initial velocity and position,
# and the number of steps. The time interval is fixed to
# t \in [0, 4\pi) (two periods)
#
# Note that this is a highly simplifyed rk4 code, intended
# for conceptual understanding and experimentation.

import sys
import numpy, math

#Global variables
ofile = None;
E0    = 0.0

def sim(x_0, v_0, N):
    ts = 0.0
    te = 4*math.pi
    h = (te-ts)/float(N)

    t = ts;
    x = x_0
    v = v_0
    while (t < te):
        kv1 = -h*x
        kx1 = h*v

        kv2 = -h*(x+kx1/2)
        kx2 =  h*(v+kv1/2)

        kv3 = -h*(x+kx2/2)
        kx3 =  h*(v+kv2/2)

        kv4 = -h*(x+kx3/2)
        kx4 =  h*(v+kv3/2)

        #Write the old values to file
        output(t,x,v)

        #Update
        x = x + (kx1 + 2*(kx2+kx3) + kx4)/6
        v = v + (kv1 + 2*(kv2+kv3) + kv4)/6
        t = t+h

def output(t,x,v):
    de = 0.5*x**2+0.5*v**2 - E0;
    ofile.write("%15.8E %15.8E %15.8E %15.8E %15.8E\n"\
                %(t, x, v, math.cos(t),de));

#MAIN PROGRAM:

#Get input
if len(sys.argv) == 5:
    ofilename =       sys.argv[1];
    x_0       = float(sys.argv[2])
    v_0       = float(sys.argv[3])
    N         =   int(sys.argv[4])
else:
    print "Usage:", sys.argv[0], "ofilename x0 v0 N"
    sys.exit(0)

#Setup
ofile = open(ofilename, 'w')
E0    = 0.5*x_0**2+0.5*v_0**2

#Run simulation
sim(x_0,v_0,N)

#Cleanup
ofile.close()
```

## The classical pendulum
The angular equation of motion of the pendulum is given by
Newton's equation and with no external force it reads

$$
\begin{equation}
  ml\frac{d^2\theta}{dt^2}+mgsin(\theta)=0,
\label{_auto34} \tag{44}
\end{equation}
$$

with an angular velocity and acceleration given by

$$
\begin{equation}
     v=l\frac{d\theta}{dt},
\label{_auto35} \tag{45}
\end{equation}
$$

and

$$
\begin{equation}
     a=l\frac{d^2\theta}{dt^2}.
\label{_auto36} \tag{46}
\end{equation}
$$

## More on the Pendulum
We do however expect that the motion will gradually come to an end due a viscous drag torque acting on the pendulum.
In the presence of the drag, the above equation becomes

$$
\begin{equation}
   ml\frac{d^2\theta}{dt^2}+\nu\frac{d\theta}{dt}  +mgsin(\theta)=0, \label{eq:pend1} \tag{47}
\end{equation}
$$

where $\nu$ is now a positive constant parameterizing the viscosity
of the medium in question. In order to maintain the motion against
viscosity, it is necessary to add some external driving force.
We choose here a periodic driving force. The last equation becomes then

$$
\begin{equation}
   ml\frac{d^2\theta}{dt^2}+\nu\frac{d\theta}{dt}  +mgsin(\theta)=Asin(\omega t), \label{eq:pend2} \tag{48}
\end{equation}
$$

with $A$ and $\omega$ two constants representing the amplitude and
the angular frequency respectively. The latter is called the driving frequency.

## More on the Pendulum
We define

$$
\omega_0=\sqrt{g/l},
$$

the so-called natural frequency and the new dimensionless quantities

$$
\hat{t}=\omega_0t,
$$

with the dimensionless driving frequency

$$
\hat{\omega}=\frac{\omega}{\omega_0},
$$

and introducing the quantity $Q$, called the *quality factor*,

$$
Q=\frac{mg}{\omega_0\nu},
$$

and the dimensionless amplitude

$$
\hat{A}=\frac{A}{mg}
$$

## More on the Pendulum
We have

$$
\frac{d^2\theta}{d\hat{t}^2}+\frac{1}{Q}\frac{d\theta}{d\hat{t}}
     +sin(\theta)=\hat{A}cos(\hat{\omega}\hat{t}).
$$

This equation can in turn be recast in terms of two coupled first-order differential equations as follows

$$
\frac{d\theta}{d\hat{t}}=\hat{v},
$$

and

$$
\frac{d\hat{v}}{d\hat{t}}=-\frac{\hat{v}}{Q}-sin(\theta)+\hat{A}cos(\hat{\omega}\hat{t}).
$$

These are the equations to be solved.  The factor $Q$ represents the number of oscillations of the undriven system that must occur before  its energy is significantly reduced due to the viscous drag. The  amplitude $\hat{A}$ is measured in units of the maximum possible  gravitational torque while $\hat{\omega}$ is the angular frequency of the external torque measured in units of the pendulum's natural frequency.

## Adaptive methods
In case the function to integrate varies slowly or fast in different integration domains, adaptive methods are normally used. One strategy is always to decrease the step size. As we have seen earlier, this leads to more CPU cycles and may lead to loss or numerical precision. An alternative is to use higher-order RK methods for example. However, this leads again to more cycles, furthermore, there is no guarantee that higher-order leads to an improved error.

## Adaptive methods
Assume the exact result is $\tilde{x}$ and that we are using an RKM method. Suppose we run two calculations, one with $h$ (called $x_1$) and one with $h/2$ (called $x_2$). Then

$$
\tilde{x}=x_1+Ch^{M+1}+O(h^{M+2}),
$$

and

$$
\tilde{x}=x_2+2C(h/2)^{M+1}+O(h^{M+2}),
$$

with $C$ a constant. Note that we calculate two halves in the last equation. We get then

$$
|x_1-x_2| = Ch^{M+1}(1-\frac{1}{2^M}).
$$

yielding

$$
C=\frac{|x_1-x_2|}{(1-2^{-M})h^{M+1}}.
$$

We rewrite

$$
\tilde{x}=x_2+\epsilon+O((h)^{M+2}),
$$

with

$$
\epsilon = \frac{|x_1-x_2|}{2^M-1}.
$$

## Adaptive methods
With RK4 the expressions become

$$
\tilde{x}=x_2+\epsilon+O((h)^{6}),
$$

with

$$
\epsilon = \frac{|x_1-x_2|}{15}.
$$

The estimate is one order higher than the original RK4. But this method is normally rather inefficient since it requires a lot of computations. We solve typically the equation three times at each time step.
However, we can compare the estimate $\epsilon$ with some by us given accuracy $\xi$.
We can then ask the question: what is, with a given $x_j$ and $t_j$, the largest possible step size $\tilde{h}$ that leads to a truncation error below $\xi$?
We want

$$
C\tilde{h} \le \xi,
$$

which leads to

$$
\left(\frac{\tilde{h}}{h}\right)^{M+1}\frac{|x_1-x_2|}{(1-2^{-M})}\le \xi,
$$

meaning that

$$
\tilde{h}=h\left(\frac{\xi}{\epsilon}\right)^{1+1/M}.
$$

## Adaptive methods
With

$$
\tilde{h}=h\left(\frac{\xi}{\epsilon}\right)^{1+1/M}.
$$

we can design the following algorithm:
 * If the two answers are close, keep the approximation to $h$.

 * If $\epsilon > \xi$ we need to decrease the step size in the next time step.

 * If $\epsilon < \xi$ we need to increase the step size in the next time step.

A much used algorithm is the so-called RKF45 which uses a combination of fourth and fifth order RK methods.

## Adaptive methods, RKF45
At each step, two different approximations for the solution are made and compared. If the
two answers are in close agreement, the approximation is accepted. If the two answers
do not agree to a specified accuracy, the step size is reduced. If the answers agree to
more significant digits than required, the step size is increased.
Each step requires the use of the following six values:

1
3
4

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

1
3
5

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

1
3
6

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

1
3
7

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

1
3
8

<
<
<
!
!
M
A
T
H
_
B
L
O
C
K

$$
k_6 = h f (t_k + \frac{1}{2}h, y_k - \frac{8}{27}k_1 + 2k_2-\frac{3544}{2565}k_2+\frac{1859}{4104}k_4-+\frac{11}{40}k_5).
$$

## Adaptive methods, RKF45
An approximation to the solution of the ODE is made using a Runge-Kutta method of order 4:

$$
y_{k+1} = y_k  + \frac{25}{216}k_1+\frac{1408}{2565}k_3 +\frac{2197}{4101}k_4-\frac{1}{5}k_5,
$$

where the four function values $k_1$ , $k_3$ , $k_4$ , and $k_5$ are used. Notice that $k_2$ is not used  here.
A better value for the solution is determined using a Runge-Kutta
method of order 5:

$$
z_{k+1} = y_k + \frac{16}{135}k_1+\frac{6656}{12825}k_3 +\frac{28561}{56430}k_4-\frac{9}{50}k_5+\frac{2}{55}k_6.
$$

The optimal time step $\alpha h$ is then determined by

$$
\alpha = \left( \frac{\xi h}{2|z_{k+1}-y_{k+1}|}\right)^{1/4},
$$

with $\xi$ our defined tolerance.

## [Solar system code, main program](https://github.com/CompPhysics/ComputationalPhysicsMSU/tree/master/doc/Programs/OOExamples)

        #include <iostream>
        #include <solver.h>
        #include <planet.h>
        #include <cmath>
        #include <armadillo>
        using namespace arma;
        using namespace std;

        int main()
        {
        //  The solarsystem object inherits all functionality of the solver class
           solver solarsystem;
        // Here we initialize all elements in the solar system (planets)
           planet  Sun(1,0,0,0,0,0,0);
           planet  Mercury(1.2e-7, 0.39, 0, 0,0,9.96,0);
           planet  Venus(2.4e-6, 0.72, 0, 0,0,7.36,0);
           planet  Earth(1.5e-6,1,0,0, 0, 6.26, 0);
           planet  Mars(3.3e-7, 1.52, 0, 0,0,5.06,0);
           planet  Jupiter(9.5e-4, 5.20, 0,0,0,2.75,0);
           planet  Saturn(2.75e-4, 9.54, 0, 0,0,2.04,0);
           planet  Uranus(4.4e-5, 19.19, 0, 0,0,1.43,0);
           planet  Neptune(5.1e-5, 30.06, 0, 0,0,1.14,0);
           planet  Pluto(5.6e-9, 39.53, 0, 0,0,0.99,0);
        // Now we add all planets
           solarsystem.add(Sun);
           solarsystem.add(Mercury);
           solarsystem.add(Venus);
           solarsystem.add(Earth);
           solarsystem.add(Mars);
           solarsystem.add(Jupiter);
           solarsystem.add(Saturn);
           solarsystem.add(Uranus);
           solarsystem.add(Neptune);
           solarsystem.add(Pluto);

           int elements = solarsystem.number_planets;
           cout << "number of element" << elements<< endl;
           solarsystem.solverRK4(solarsystem.all_planets, 0.001, 100 );

        }

The full code, with cpp files and header files,  is at <https://github.com/CompPhysics/ComputationalPhysicsMSU/tree/master/doc/Programs/OOExamples>.

## [Solar system code, main program](https://github.com/CompPhysics/ComputationalPhysicsMSU/tree/master/doc/Programs/OOExamples), header file

        #ifndef PLANET_H
        #define PLANET_H

        class planet
        {
        public:

            double position[3];
            double velocity[3];
            double mass;
            planet(double M, double x,double y, double z, double vx, double vy, double vz);
            planet();

        };

        #endif // PLANET_H

## [Solar system code, main program](https://github.com/CompPhysics/ComputationalPhysicsMSU/tree/master/doc/Programs/OOExamples), header file, cpp file for planet class, simple example

        #include "planet.h"
        // Simple constructor
        planet::planet()
        {
        };
        // Improved constructor
        planet::planet(double M, double x,double y, double z, double vx, double vy, double vz){
        mass = M;
        position[0] = x;
        position[1] = y;
        position[2] = z;

        velocity[0] = vx;
        velocity[1] = vy;
        velocity[2] = vz;
        };


# Is Sympy really 'slow'?
I don't want to be a bad carpenter who blames his tools. Here're my attempts at understanding why my Sympy code is slow. The technical specs of my computer may have a lot to do with the long time things are taking to run, though, any techniques I discover to speed up computation will have a speed up on any system I suppose.

Ultimately my goal is to have integral and linear algebra routines that can run at sub-second speed, and preferrably a few ms.

## Potential things to try out

* ~~I may have been using the wrong Python version of Theano. The [docs](https://theano-pymc.readthedocs.io/en/latest/requirements.html) say official support for Theano is only there for Python <3.6.~~
    * ~~install Sympy, Theano nd pyMC3~~ *This doesn't work because Theano didn't seem to recognise 'external' sympy functions, eg. legendre*

* With ```lambdify``` use the 'mpmath' module as a backend. This *really* sped up computations, in comparison to the default (sympy?) backend module.

* ~~Linux specific: change the 'niceness'of the process and so the CPU spends more time on the code.~~~

## Latest realisation
* I've been using the 'normal' Hankel function instead of the spherical Hankel function. This is probably the main reason why my results are off all the time!!!!!

```python
#from joblib import Parallel, delayed
#from joblib import wrap_non_picklable_objects
from gmpy2 import *
import matplotlib.pyplot as plt
import mpmath
from mpmath import mpf
dps = 50; mpmath.mp.dps = dps
import numpy as np
#from scipy.special import jv as bessel_firstkind
from symengine import *
import sympy
from sympy import jn,yn ,symbols, legendre, sin, cos, tan, summation, I, oo, diff, pi, sqrt
from sympy import factor_terms, Matrix, besselj, bessely, Piecewise
from sympy import Abs, lambdify, integrate, expand,integrate, Integral
from sympy.printing.theanocode import theano_function
from sympy.utilities.autowrap import autowrap
import tqdm
x, alpha, gamma, index, k, m,n,p, r1, R, theta, y, z = symbols('x alpha gamma index k m n p r1 R theta,y,z')
from sympy import N, cse

from sympy.printing.theanocode import theano_function
num_sumterms = 50
```

```python
from sympy import besselj
```

```python
sph_bessel1 = besselj(n+1/2,z)*sqrt(pi/(2*z))
sph_bessel2 = bessely(n+1/2,z)*sqrt(pi/(2*z))
sph_hankel2 = sph_bessel1 - I*sph_bessel2
#sph_hankel2 = lambdify([n, p], sph_bessel1(n,p)-I*sph_bessel2(n,p), 'mpmath')
```

```python
sqrt(pi/(2*z))
```

$\displaystyle \frac{\sqrt{2} \sqrt{\pi} \sqrt{\frac{1}{z}}}{2}$

```python
r1 = (R*cos(alpha))/cos(theta)
```

```python
subs_dict = {'alpha':mpmath.pi/18, 'k':5,'R':mpf(0.1), 'm':20,'n':10}
```

```python
Lm_expr = legendre(m,cos(theta))*(r1**2/R**2)*tan(theta)
Lm = Integral(Lm_expr, (theta,0,alpha))#.doit(meijerg=True)
Lmexpr_func = lambdify([m, R, alpha,theta], Lm_expr, 'mpmath')
Lm_func = lambdify([m, R, alpha], Lm, 'mpmath') # numerical integrtion of eqn. 12.108
```

```python
# eqn 12.107
Kmn_expr = legendre(n, cos(theta))*legendre(m, cos(theta))*sin(theta) # the integrl of this expression
# has a solution given in Appendix II, eqn 70
legendre_1stderiv = diff(legendre(n,z),z)
# when m != n
num_legendre_term1 = legendre(m,cos(alpha))*legendre_1stderiv.subs({'z':cos(alpha)})
num_legendre_term2 = legendre(n,cos(alpha))*legendre_1stderiv.subs({'n':m,'z':cos(alpha)})
eqn70_mnoteqn = sin(alpha)*(num_legendre_term1-num_legendre_term2)/(m*(m+1)-n*(n+1))

# when m==n
summn_funcn = legendre(index,cos(alpha))*(legendre(index,cos(alpha))*cos(alpha)-legendre(index+1,cos(alpha)))
# substituting 'j' for 'index' because j is also used for sqrt(-1) all through the book!!

meqn_sumterm = 2*summation(summn_funcn, (index,1,m-1))
eqn70_meqn = (1+ cos(alpha)*legendre(m,cos(alpha))**2 + meqn_sumterm)/(2*m+1)
```

```python

```

```python
Kmn = Piecewise((eqn70_mnoteqn,m>n),(eqn70_mnoteqn,m<n), (eqn70_meqn,True), )
Kmn_func = lambdify([m,n,alpha],Kmn,'mpmath')
```

```python
Kmn
```

$\displaystyle \begin{cases} \frac{\left(- \frac{m \left(\cos{\left(\alpha \right)} P_{m}\left(\cos{\left(\alpha \right)}\right) - P_{m - 1}\left(\cos{\left(\alpha \right)}\right)\right) P_{n}\left(\cos{\left(\alpha \right)}\right)}{\cos^{2}{\left(\alpha \right)} - 1} + \frac{n \left(\cos{\left(\alpha \right)} P_{n}\left(\cos{\left(\alpha \right)}\right) - P_{n - 1}\left(\cos{\left(\alpha \right)}\right)\right) P_{m}\left(\cos{\left(\alpha \right)}\right)}{\cos^{2}{\left(\alpha \right)} - 1}\right) \sin{\left(\alpha \right)}}{m \left(m + 1\right) - n \left(n + 1\right)} & \text{for}\: m > n \vee m < n \\\frac{\cos{\left(\alpha \right)} \left(P_{m}\left(\cos{\left(\alpha \right)}\right)\right)^{2} + 2 \sum_{index=1}^{m - 1} \left(\cos{\left(\alpha \right)} P_{index}\left(\cos{\left(\alpha \right)}\right) - P_{index + 1}\left(\cos{\left(\alpha \right)}\right)\right) P_{index}\left(\cos{\left(\alpha \right)}\right) + 1}{2 m + 1} & \text{otherwise} \end{cases}$

```python

alternate_hankels = n*sph_hankel2.subs({'n':n-1, 'z':k*r1})-(n+1)*sph_hankel2.subs({'n':n+1, 'z':k*r1})
pt1_postterm = legendre(n,cos(theta))*cos(theta)
Imn_pt1 = alternate_hankels*pt1_postterm

pt2_preterm = n*(n+1)*sph_hankel2.subs({'z':k*r1})
alternate_legendres = (legendre(n-1,cos(theta))-legendre(n+1,cos(theta)))/(k*r1)
Imn_pt2 = pt2_preterm*alternate_legendres

whole_postterm = legendre(m,cos(theta))*(r1**2/R**2)*tan(theta)

Imn_term = (Imn_pt1 + Imn_pt2)*whole_postterm # eqn. 12.106
Imn = Integral(Imn_term,(theta,0,alpha))
Imn_term_func = lambdify([m,n,k,R,alpha,theta], Imn_term, 'mpmath')
Imn_func = lambdify([m,n,k,R,alpha],Imn,'mpmath')

```

```python

def imn_quad_value(params):
    variables = ['m','n','k','R','alpha']
    imn_onlytheta = Imn_term.subs({each: params[each] for each in variables})
    imn_onlytheta_f = lambdify([theta],imn_onlytheta,'mpmath')
    # try breaking 0-alpha into 10 parts if the error is  too high - with mpmath.linspace?
    return mpmath.quad(imn_onlytheta_f,[0,params['alpha']],error=True)
```
params['m'] = 5
params['n'] = 20
imn_value = imn_quad_value(params)
print(imn_value)out1 = Imn_func(10,10,subs_dict['k'],subs_dict['R'],subs_dict['alpha'])
out2 = Imn2_func(10,10,subs_dict['k'],subs_dict['R'],subs_dict['alpha'])print(out1.real/out2.real, out1.imag/out2.imag, abs(out1)/abs(out2))

```python
mv,nv = 2,15
npts = 100
imn_values = mpmath.matrix(1,npts)
angles = np.linspace(0,float(subs_dict['alpha']),npts)
for i,each in enumerate(angles):
    imn_values[i] = Imn_term_func(mv,nv,subs_dict['k'],subs_dict['R'],subs_dict['alpha'],each)
real_imn = [each.real for each in imn_values]
imag_imn = [each.imag for each in imn_values]
abs_imn = [abs(each) for each in imn_values]

plt.figure()
plt.subplot(131)
plt.plot(angles, real_imn,label='real');plt.legend()
plt.subplot(132)
plt.plot(angles, imag_imn,label='imag');plt.legend()
plt.xlabel('$\\theta$, radians')
plt.subplot(133)
plt.plot(angles, abs_imn,label='abs');plt.legend()

```

### Broadly speaking, the Imn function is fairly smooth, and so numerical integration shouldn't be such a problem?

```python
frequency = 50*10**3 # kHz
vsound = 330 # m/s
wavelength = vsound/frequency
alpha_value = mpmath.pi/3 # 60 degrees --> pi/3
k_value = 2*mpmath.pi/(wavelength)
ka = 10
a_value = ka/k_value
R_value = a_value/mpmath.sin(alpha_value) # m

```

```python
Nv = 12 + int(2*ka/sin(alpha_value))
M_matrix = mpmath.matrix(Nv,Nv)
b_matrix = mpmath.matrix(Nv,1)

```

```python
b = -I*Lm
b_func = lambdify([m,alpha], b,'mpmath') # eqn. 12.104
```

```python
params = {'k':k_value, 'a':a_value, 'R':R_value, 'alpha':alpha_value}
```

```python
mmn_hankels = n*sph_hankel2.subs({'n':n-1,'z':k*R})-(n+1)*sph_hankel2.subs({'n':n+1,'z':k*R})
mmn_hankels_func = lambdify([n,k,R], mmn_hankels,'mpmath')
```

```python

```

```python
all_quaderrors = []
for i in tqdm.trange(Nv):
    for j in range(Nv):
        params['m'],params['n'] = i,j
        Imn_value = Imn_func(params['m'], params['n'],params['k'],params['R'],params['alpha'])
        Kmn_value = Kmn_func(params['m'],params['n'],params['alpha'])
        numerator_hankels = mmn_hankels_func(j,params['k'],params['R'])
        numerator = Imn_value+ numerator_hankels*Kmn_value
        denom = 2*params['n']+1
        M_matrix[params['m'],params['n']] = numerator/denom

```

    100%|██████████| 35/35 [38:08<00:00, 65.37s/it]

```python
all_quaderrors
```

    []

```python
for each_m in range(Nv):
    #M_matrix[each_m, each_n] = M_mn_func(each_m, each_n, k_value, R_value, alpha_value)
    b_matrix[each_m,:] = b_func(each_m, alpha_value)
```
#a_matrix = mpmath.lu_solve(M_matrix,b_matrix) -# fast but dirty
a_matrix,res = mpmath.qr_solve(M_matrix,b_matrix) # slow but reliable

```python
a_matrix = mpmath.inverse(M_matrix)*b_matrix
```

```python
legendre_func = lambdify((m, x), legendre(m, x),'mpmath')
```

```python
#summn_term = A_n[index]*(I**index)*legendre(index,mpmath.cos(theta))
#summn_value = summation(summn_term, (index,0,N_v))

def d_theta(angle,k_v,R_v,alpha_v,An):
    num = 4
    N_v = An.rows
    denom  = (k_v**2)*(R_v**2)*mpmath.sin(alpha_v)**2
    part1 = num/denom
    jn_matrix = np.array([1j**f for f in range(N_v)])
    legendre_matrix = mpmath.matrix([legendre(n_v, mpmath.cos(angle)) for n_v in range(N_v)])
    part2_matrix = np.column_stack((An, jn_matrix, legendre_matrix))
    part2 = np.sum(np.apply_along_axis(lambda X: X[0]*X[1]*X[2], 1, part2_matrix))

    rel_level = - part1*part2
    return rel_level

def d_zero(k_v,R_v,alpha_v,An):
    num = 4
    N_v = An.rows
    denom  = (k_v**2)*(R_v**2)*mpmath.sin(alpha_v)**2
    part1 = num/denom
    jn_matrix = np.array([1j**f for f in range(N_v)])
    part2_matrix = np.column_stack((An, jn_matrix))
    part2 = np.sum(np.apply_along_axis(lambda X: X[0]*X[1], 1, part2_matrix))
    rel_level = - part1*part2
    return rel_level

def relative_directivity_db(angle,k_v,R_v,alpha_v,An):
    off_axis = d_theta(angle,k_v,R_v,alpha_v,An)
    on_axis = d_zero(k_v,R_v,alpha_v,An)
    rel_level = 20*mpmath.log10(abs(off_axis/on_axis))
    return rel_level
```

```python

angles = mpmath.linspace(0,2*mpmath.pi,200)
dirnlty = [relative_directivity_db(angle_v, k_value, R_value,alpha_value,a_matrix) for angle_v in angles]

```

```python
plt.figure()
a0 = plt.subplot(111, projection='polar')
plt.plot(angles, dirnlty)
plt.ylim(-40,0);plt.yticks([0,-10,-20,-30,-40]);
plt.xticks(np.radians(np.arange(0,360,30)));
```

```python

```