# Power calculations for a binomial proportion

## Background

In an ideal statistical world no hypothesis test would be run before a [power analysis](https://en.wikipedia.org/wiki/Power_of_a_test) has been carried out to determine a reasonable [sample size](https://en.wikipedia.org/wiki/Sample_size_determination). Most researchers carrying out statistical hypothesis testing only focus on the type-I error: the probability of erreneously rejecting a null hypothesis. Since the null hypothesis is synonymous with the "no effects" regime, this amounts to controlling the number spurious discoveries. Sophisticated techniques have also been developed to control the type-I error under the problem of [multiple comaparisons](https://en.wikipedia.org/wiki/Multiple_comparisons_problem). This focus on type-I error leads to significant problem in academic research: inflated effect size estimates. Unfortunately, advanced statistical methods give researchers a false confidence that their "significant" discoveries are "true", in the sense that they are probably not *completely* spurious.

The power of the test is probality of rejecting the null when it is false. In the "no effects" regime this is equivalent to correctly concluding there is some real effect. Power is inversely related to effect size bias because results that are statistically significant go through a filter: only realized statisticals below a certain p-value are examined. If the power of a test is large, then the descrepancy between the distribution of the statistic conditional on statistically significance will be similar to its unconditional distribution.[[^1]] Generally speaking, a test that with a large power, say 80%, will have a fairly small effect size bias.

Power analysis is by no means absent from empirical research. It is essential for grant funding applications in biomedical research and the design of clinical trials. I suspect that as statistics have moved towards "big data" and "analytics" the cultural emphasis on a well-designed statistical procedure has shifted to "mining" for hidden results. In such situations estimating the power is impossible because its not even clear what the hypothesis test is! Another reason power is less focused on is that it is harder to carry out. A researcher needs to make several assumptions that are normally not needed for calculating type-I errors including the estimated signal size and sometimes other nuissance parameters.

This post is about how to calculate the number of samples that will be needed to obtain a specified power for a test of a binomial proportion. Anytime there is a binary outcome which is being measured in aggregate (between groups, systems, cities, etc) we can ask whether this proportion is different than some amount. For example is there a difference in the rate of high-school completion between sexes or does a medical test have a better true positive rate than an existing tool? Constructing [confidence intervals](https://en.wikipedia.org/wiki/Confidence_interval) for [binomial proportions](https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval) is statistically tricky because a binomial proportion's distribution is often unknown. Asymptotically this proportion will actually have a normal distribution, and this normal approximation is often used in practice.

This post will provide analytic ways of calculating the sample size for binomial confidence intervals for both normal and more advanced methods. There are several key recommendations. First, under no circumstances should a naive version of the normal approximation method be used.

## Preliminaries

To quickly establish some notation that will be used throughout the post: we are considering the proportion of successes in a binomial trial: $p=y / n$, where $y \sim \text{Binom}(\pi, n)$. Therefore $E(p)=\pi$ and $Var(p)=\pi(1-\pi)/n = \sigma^2_\pi$. Assymptotically it is known that $p$ can be transformed so that it is normally distributed: $z=(p - \pi)/\sigma_\pi \sim N(0,1)$. Suppose we measure the proportions from two different IID groups 1 and 2: $p_1$ and $p_2$, $E(p_i)=\pi_i$, with $n_1$ and $n_2$ samples. If we want to establish whether group 1 has a higher proportion than group 2 will specify a null and alternative hypothesis as follows:

$$
\begin{align*}
H_0 &: \pi_1 \leq \pi_2 \\
H_A &: \pi_1 > \pi_2, \hspace{3mm} \pi_1 - \pi_2 = \Delta > 0 \\
z_0 &= \frac{p_1 - p_2}{\sigma_{\pi_0}} \\
\sigma_{\pi}^2 &= \frac{\pi_1(1-\pi_1)}{n_1} + \frac{\pi_2(1-\pi_2)}{n_2} \\
\sigma_{\pi_0}^2 &= \sigma_{\pi} | H_0 = \frac{\pi_1(1-\pi_1)}{n}(w_1 + w_2) \\
n &= n_1+n_2, \hspace{3mm} w_i = n / n_i
\end{align*}
$$

Note that in the null I assume $\pi_1=\pi_2$ rather than $\pi_1 < \pi_2$, both of which conditions could be true under the null. However, the former case is the most conservative assumption for our subsequent power analysis and should be used.  Also note that the symbol $\Phi$ refers to the standard normal CDF, which will be used throughout the post. The hat notation refers to a realization of the random variable so that $p_1$ is a random variable and $\hat p_1$ is a realization of the proportion for group 1. If $z_0 > t_\alpha$ we will reject the null, where $P(z_0 > t_\alpha | H_0) = \alpha$. The type-II error will be denoted as $\beta$ so that $P(z_A > t_\alpha | H_A) = 1-\beta$ is the power.

## (1) Normal approximation approach

Assume that $n$ is large enough that we can use the z-score formula seen in the preliminaries. Then it is easy enough to derive an approximation for the power calculation as follows:

$$
\begin{align*}
z_A = z | H_A &= \frac{p_1 - p_2 - \Delta}{\sigma_{\pi_A}} \sim N(0,1) \\
\sigma_{\pi_A}^2 &= \frac{\pi_1(1-\pi_1)}{n_1} + \frac{(\pi_1-\Delta)(1-(\pi_1-\Delta))}{n_2} \\
&= \sigma_{\pi_0}^2 + \epsilon, \hspace{3mm} \epsilon=\frac{\Delta(2\pi_1-1)-\Delta^2}{n_2}
\end{align*}
$$

Because $\epsilon(\Delta,\pi_1,n_2)$ is $O(n_2^{-1})$ it is often ignored in the analysis since for large enough $n_2$ it is basically zero. If one sets $\epsilon=0$, then there is a closed-form solution to solving for a specific power:

$$
\begin{align*}
P( z > t_\alpha) &= P\Bigg( \frac{p_1 - p_2}{\sigma_{\pi_0}} > t_\alpha \Bigg) \\
&= P\Bigg( \frac{p_1 - p_2}{\sigma_{\pi_0}}-\frac{\Delta}{\sigma_{\pi_0}}+\frac{\Delta}{\sigma_{\pi_0}} > t_\alpha | H_A \Bigg) \\
&= P\Bigg( z_A > t_\alpha - \frac{\Delta}{\sigma_{\pi_0}} | H_A \Bigg) \\
1-\beta &= \Phi(\Delta / \sigma_{\pi_0}-t_\alpha) \\
\frac{1}{\sigma_{\pi_0}}&= \frac{(\Phi^{-1}_{1-\beta} + t_\alpha)}{\Delta} \\
\frac{n_1n_2}{n} &= \frac{\pi_1(1-\pi_1)(\Phi^{-1}_{1-\beta} + t_\alpha)^2}{\Delta^2}
\end{align*}
$$

If the two groups have the same sample size: $n_1 = n_2$, then the exact formula is:

$$
\begin{align}
n^* &= 2\cdot \frac{\pi_1(1-\pi_1)(\Phi^{-1}_{1-\beta} + t_\alpha)^2}{\Delta^2} \label{eq:simple1}
\end{align}
$$

The terms in equation \eqref{eq:simple1} are intuitive. Higher power ($\Phi^{-1}_{1-\beta}$) or a lower type-I error rate ($t_\alpha$) require an increasing number of samples, the latter because the threshold to become significant is higher. Increases in the binomial variance ($\pi(1-\pi)$) or a smaller difference in the proportions ($\Delta$) require more samples as there is more noise and less signal, respectively.  Though equation \eqref{eq:simple1} is elegant, it has several limitations. First, its variance is off ignores $\epsilon$ term. Second it assumes an equal number of samples between groups 1 and 2. Instead it is much better to analytically solve for $n_1$,

$$
\begin{align}
\arg\min_{n_1} \hspace{3mm} \big[ 1/\sigma_{\pi_A}(n_1,n_2,\pi_1,\Delta) - (\Phi^{-1}_{1-\beta} + t_\alpha)/\Delta \big]^2 \label{eq:argmin1}
\end{align}
$$

Where $n_2$, $\pi_1$, $\Delta$, $\alpha$, and $\beta$ are treated as fixed parameters and the power can be directly targeted by $n_1$ as equation \eqref{eq:argmin1} shows. The code block below will compare approaches \eqref{eq:simple1} and \eqref{eq:argmin1} to see how well there predicted power lines up with reality.

```python
import os
import numpy as np
import pandas as pd
from scipy import stats
from scipy.stats import norm
import plotnine
from plotnine import *
seed = 1234 # Use throughout post
from scipy.optimize import minimize_scalar
from time import time

np.seterr(invalid='ignore')

def sig_n12(n1, pi1, Delta, n2=None):
    pi2 = pi1 - Delta
    assert (pi2 > 0)
    if n2 is None:
        n2 = n1
    else:
        n2 = np.log(n2)
    return np.sqrt(pi1*(1-pi1)/np.exp(n1) + pi2*(1-pi2)/np.exp(n2))

def power_eq1(pi1, Delta, alpha, power):
    return 2*pi1*(1-pi1)*(norm.ppf(power) + norm.ppf(1-alpha))**2 / Delta**2

def power_eq2(pi1, Delta, alpha, power, n2=None):
    """
    POWER FOR ONE-SIDED HYPOTHESIS TEST, DELTA>0
    Delta=E(p1)-E(p2), alpha=E(z>t|H_0), beta=E(z<t|H_0)
    """
    rhs = (norm.ppf(power) + norm.ppf(1-alpha))/Delta
    n1star = np.exp(minimize_scalar(fun=lambda x: (1/sig_n12(x, pi1=pi1, Delta=Delta, n2=n2) - rhs)**2).x)
    n2star = n2
    if n2 is None:
        n2star = n1star
    lhs = 1/sig_n12(n1=np.log(n1star), pi1=pi1, Delta=Delta, n2=n2star)
    assert np.abs(lhs - rhs) < 1e-5
    return n1star, n2star

nsim = 100000
alpha = 0.05
t_alpha = norm.ppf(1-alpha)
pi_seq = np.round(np.arange(0.25,0.51,0.05),2)
delta_seq = np.round(np.arange(0.05,0.25,0.05),2)
power_seq = np.round(np.arange(0.25,1,0.05),2)

np.random.seed(seed)
count = 0
holder = []
for pi in pi_seq:
    for delta in delta_seq:
        for power in power_seq:
            count += 1
            n_eq1 = power_eq1(pi, delta, alpha, power)
            n_eq2 = power_eq2(pi, delta, alpha, power)[0]
            # Generate data for different groups
            phat1_g1 = np.random.binomial(n_eq1, pi, size=nsim)/n_eq1
            phat1_g2 = np.random.binomial(n_eq1, pi-delta, size=nsim)/n_eq1
            phat2_g1 = np.random.binomial(n_eq2, pi, size=nsim)/n_eq2
            phat2_g2 = np.random.binomial(n_eq2, pi-delta, size=nsim)/n_eq2
            se1 = np.sqrt(phat1_g1*(1-phat1_g1)/n_eq1 + phat1_g2*(1-phat1_g2)/n_eq1)
            se2 = np.sqrt(phat2_g1*(1-phat2_g1)/n_eq2 + phat2_g2*(1-phat2_g2)/n_eq2)
            z1 = (phat1_g1-phat1_g2)/se1
            z2 = (phat2_g1-phat2_g2)/se2
            p1, p2 = np.mean(z1 > t_alpha), np.mean(z2 > t_alpha)
            tmp = pd.DataFrame({'pi':pi, 'Delta':delta,'power':power,'p1':p1, 'p2':p2, 'n1':n_eq1, 'n2':n_eq2},index=[count])
            holder.append(tmp)
sim1 = pd.concat(holder).melt(['pi','Delta','power'],None,'method')
sim1 = sim1.assign(metric=lambda x: x.method.str[0], approach=lambda x: x.method.str[1]).drop(columns = 'method')
sim1 = sim1.pivot_table('value',['pi','Delta','power','approach'],'metric').reset_index()

plotnine.options.figure_size = (8,4)
gg_sim1 = (ggplot(sim1, aes(x='power',y='p',color='Delta')) + theme_bw() +
           geom_jitter(random_state=seed, height=0,width=0.005,alpha=0.75,size=2) +
           labs(x='Expected power', y='Actual power') +
           geom_abline(slope=1, intercept=0,linetype='--',color='blue') +
           facet_wrap('~approach',labeller=labeller(approach={'1':'Eq. (1)','2':'Eq. (2)'})) +
           ggtitle('Figure 1: Comparison of normal approximation power calculations') +
           scale_color_continuous(name='  Δ'))
print(gg_sim1)

plotnine.options.figure_size = (7,3.5)
gg_power_n = (ggplot(sim1.assign(err=lambda x: x.p - x.power), aes(x='np.log(n)',y='err')) + theme_bw() +
               geom_point() +
               labs(x='log(n)', y='Actual less expected power') +
               facet_wrap('~approach',labeller=labeller(approach={'1':'Eq. (1)','2':'Eq. (2)'})) +
               ggtitle('Figure 2: Power calculation accuracy and sample size'))
print(gg_power_n)
```

Figure 1 shows that equation \eqref{eq:simple1} over predicts the number of samples that are needed (actual power exceeds predicted power), especially for large values of $\Delta$. This is to be expected since the variance formula ignores this term. In contrast \eqref{eq:argmin1} provides much tighter predictions. However both approaches suffer from inaccurate power estimates for small sample sizes, as Figure 2 shows. This is be expected because the normal approximation becomes increasingly imprecise for small sample sizes. The underlying reason for the descrepancy is that the lower-bound does not provide the actual coverage as expected:

$$
\begin{align*}
&\text{Coverage of lower bound} \\
P(\hat p_1 &> \hat p_2 - \sigma_{\hat \pi} \cdot t_\alpha | H_0 ) \neq 1-\alpha, \text{when $n$ is small}
\end{align*}
$$

The simulations below show the coverage errors as a function of sample size using equation \eqref{eq:argmin1}.

```python
nsim = 100000
pi_seq = np.round(np.arange(0.01,1,0.01),2)
n_seq = 2**np.arange(7,11,1)

holder = []
for pi in pi_seq:
    for n in n_seq:
        phat1 = np.random.binomial(n, pi, size=nsim)/n
        phat2 = np.random.binomial(n, pi, size=nsim)/n
        se = np.sqrt(phat1*(1-phat1)/n + phat2*(1-phat2)/n)
        coverage = np.mean(phat1 > phat2 - se*t_alpha)
        tmp = pd.DataFrame({'pi':pi,'n':n,'coverage':coverage},index=[0])
        holder.append(tmp)
sim_coverage = pd.concat(holder).reset_index(None,True).assign(n=lambda x: pd.Categorical(x.n, x.n.unique()))

plotnine.options.figure_size = (5, 4)
gg_coverage = (ggplot(sim_coverage, aes(x='pi',y='coverage',color='n'))+
               theme_bw() + geom_line() +
               labs(x='π',y='Coverage') +
               ggtitle('Figure 3: Coverage of Gaussian approx. lower bound\nFor the 95% level') +
               scale_color_discrete(name='Sample size') +
               theme(legend_position=(0.5,0.35)) +
               scale_y_continuous(limits=[0.85,1.0],breaks=list(np.arange(0.85,1.01,0.05))) +
               geom_hline(yintercept=0.95,linetype='--'))
gg_coverage
```

Figure 3 shows that for small sample sizes and small true proportion values ($\pi$) the lower bound for the Gaussian approximation does a poor job at actually containing the true parameter ($\pi$) at a coverage of $1-\alpha$ as would be expected if the distribution was exact.

## (2) Exact binomial distribution

```python

```

```python

```

```python

```

```python

```

## Wilson Method

If we want to either....

## Arcsin

```python
# p_seq = np.round(np.arange(0.01, 1, 0.01),2)
# n = 1000
# nsim = 2500
# np.random.seed(seed)
# df_arcsin = pd.concat([pd.DataFrame({'phat':[np.random.binomial(n,p)/n for z in range(nsim)],'p':p}) for p in p_seq])
# df_arcsin = df_arcsin.assign(ptrans= lambda x: np.arcsin(np.sqrt(x.phat)))
# df_arcsin_var = df_arcsin.groupby('p').ptrans.std().reset_index().rename(columns={'ptrans':'var_trans'})

# gg_arcsin = (ggplot(df_arcsin_var, aes(x='p',y='var_trans')) + theme_bw() +
#              geom_point() +
#              geom_hline(yintercept=np.sqrt(1/(4*n)),color='blue',linetype='--') +
#              labs(x='True proportion',y='Standard deviation of outcome') +
#              ggtitle('Variation in arcsin() transform'))
# gg_arcsin
```

```python

```

<br>
* * *

### Footnotes

[^1]: I discuss this phenomenon at length in this [post](http://www.erikdrysdale.com/winners_curse) and provide some methods to correct for lower-powered tests.


```python
%load_ext autoreload
%autoreload 2
```

```python
from torchcox import TorchCox

import numpy as np
import torch
import pandas as pd
```

We use two datasets for validation of our Cox model: one synthetic and one real

# Validation against closed-form MLE fit

A synthetic dataset for which we can fit the Cox partial likelihood Maximum Likelihood Estimate by hand, to validate numerical PyTorch implementation against.

```python
valdf = pd.DataFrame({'id':['Bob','Sally','James','Ann'], 'time':[1,3,6,10], 'status':[1,1,0,1], 'smoke':[1,0,0,1]})
valdf
```

      id
      time
      status
      smoke

      0
      Bob
      1
      1
      1

      1
      Sally
      3
      1
      0

      2
      James
      6
      0
      0

      3
      Ann
      10
      1
      1

Specify model Cox(t,d ~ smoke) for times $t$ and status indicator $d$.
$L(T, D, X) = \prod_{t_i^e} \frac{h(t_i^e,X_i)}{\sum_{j:\, t_j \geq t_i^e} h(t_j, X_j)}$

where $t_i^e$ denote event times (i.e. times where $d=1$), and $t_i$ denotes all times (whether event or censored).

In this way the ratio $\frac{h(t_i^e,X_i)}{\sum_{j:\, t_j \geq t_i} h(t_j, X_j)}$ is the ratio of the hazard for the event observed at time $t_i^e$ divided by the sum of hazards for all subjects at risk at that time (so with $t_j \geq t_i^e$, i.e. have not had an event yet, have not died, so still at risk, including the subject who had an event at that time since that person must have been at risk too, so inequality includes time $t_i^e$).

We therefore see that for each of the subjects in the data just above the hazard, $h(t, X) = h_0(t)\,\exp(X \beta)$, is:
- Bob: $h(t, X) = h_0(t)\,\exp(\beta)$
- Sally: $h(t, X) = h_0(t)\,\exp(0)$
- James: $h(t, X) = h_0(t)\,\exp(0)$
- Ann: $h(t, X) = h_0(t)\,\exp(\beta)$

The likelihood for this dataset then has three factors, one for each event time, and in each factor is the hazard of the person who had the event divided by the sum of the hazards of all the people who were still at risk then (including the person who had the event:

$
\begin{align}
L(T, D, X) = &\left[\frac{h_0(t)\exp(\beta)}{h_0(t)\exp(\beta) \,+\, h_0(t)\exp(0) \,+\, h_0(t)\exp(0) \,+\, h_0(t)\exp(\beta)} \right] \\
&\times \left[ \frac{h_0(t)\exp(0)}{h_0(t)\exp(0) \,+\, h_0(t)\exp(0) \,+\, h_0(t)\exp(\beta)} \right] \\
&\times \left[ \frac{h_0(t)\exp(\beta)}{h_0(t)\exp(\beta)} \right]
\end{align}$

Note that the baseline hazard, $h_0(t)$ will always cancel everywhere in the likelihood. We then have:

$\begin{align}
L(T, D, X) &= \left[\frac{\exp(\beta)}{2(1 + \exp(\beta))} \right] \times \left[ \frac{1}{2+ \exp(\beta)} \right] \times 1
\end{align}
$

Take the natural logarithm of this as the logarithm is a monotone transformation and will not change the position of the minimum, but will simplify the computation of the derivative and make the computation more numerically stable. Also we multipy it by $-1$ so that to maximise the likelihood we minimise the quantity below.

$-\ln[L(T,D,X)] = -\beta +\ln[2(1+\exp(\beta)] - \ln[1] + \ln[2+\exp(\beta)]$

Taking the derivative of this wrt $\beta$ gives
$\begin{align}
-\frac{d}{d\beta} \ln[L(T,D,X)] &= -1 + \frac{d}{d\beta}\ln[(1+\exp(\beta)] + \frac{d}{d\beta}\ln[2+\exp(\beta)] \\
&= -1 + \frac{1}{1+\exp(\beta)}\exp(\beta) + \frac{1}{2+\exp(\beta)}\exp(\beta)
\end{align}$

Setting this equal to zero, $-\frac{d}{d\beta} \ln[L(T,D,X)] = 0$, gives
$1 = \frac{\exp(\beta)}{1+\exp(\beta)} + \frac{\exp(\beta)}{2+\exp(\beta)}$

So that this is the equation we need to solve for $\beta$ to find the stationary point (hopefully a minimum, we haven't shown it but this score function is convex so this is indeed a minimum), and this will be the Maximum Likelihood Estimate (MLE) for $\beta$.

This gives $\beta = \ln(2)/2 \approx 0.34657$

(Note that we chose the solution in the Reals here, the solution is in fact $\beta = \ln(\sqrt{2})$ where the square root has two solutions and only the positive one gives a real value for $\beta$.)

Now we fit our Cox model code to verify that we indeed obtain the closed-form answer as expected

```python
tname = 'time'
Xnames = ['smoke']
dname = 'status'

coxmod1 = TorchCox.TorchCox(lr=1)

coxmod1.fit(valdf, Xnames=Xnames, tname=tname, dname=dname)

coxmod1.beta.detach().numpy()[0]
```

    [0.34657338]

    0.34657338

And we do! :)

```python
np.log(2)/2
```

    0.34657359027997264

# Validation against R's `survival` package

Now we will compare our result against the R package 'survival' on another dataset.

```python
df_tied = pd.read_csv('../data/ovarian_deduplicated.csv')
df_tied
```

      tyears
      d
      Karn
      Broders
      FIGO
      Ascites
      Diam
      id

      0
      0.024657
      1
      8
      2.0
      IV
      1
      1-2cm
      281

      1
      0.027398
      1
      6
      NaN
      IV
      1
      >5cm
      298

      2
      0.035617
      1
      8
      2.0
      IV
      1
      >5cm
      342

      3
      0.041094
      1
      7
      2.0
      III
      0
      <1cm
      228

      4
      0.082192
      1
      7
      3.0
      IV
      1
      <1cm
      52

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      299
      7.060274
      0
      10
      3.0
      III
      0
      >5cm
      101

      300
      7.112330
      0
      10
      1.0
      III
      1
      >5cm
      93

      301
      7.120548
      0
      9
      2.0
      III
      1
      <1cm
      40

      302
      7.290410
      0
      9
      2.0
      IV
      1
      1-2cm
      81

      303
      7.476712
      0
      10
      1.0
      III
      0
      2-5cm
      10

304 rows × 8 columns

```python
coxmod = TorchCox.TorchCox()

tname = 'tyears'
Xnames = ['Karn', 'Ascites']
dname = 'd'
```

```python
#sudo pip install rpy2
```

```python
%load_ext rpy2.ipython
```

```r
%%R
library(readr)
library(survival)
library(dplyr)
library(tidyr)

df2 = read_csv("/home/ilan/Desktop/TorchCox/data/ovarian_deduplicated.csv")

starttime = Sys.time()

rmod = coxph(Surv(tyears, d) ~ Karn + Ascites, df2, ties="breslow")
print(coef(rmod))

endtime = Sys.time()

print(endtime-starttime)
```

    R[write to console]:
    Attaching package: ‘dplyr’

    R[write to console]: The following objects are masked from ‘package:stats’:

        filter, lag

    R[write to console]: The following objects are masked from ‘package:base’:

        intersect, setdiff, setequal, union

    R[write to console]: Parsed with column specification:
    cols(
      tyears = col_double(),
      d = col_double(),
      Karn = col_double(),
      Broders = col_double(),
      FIGO = col_character(),
      Ascites = col_double(),
      Diam = col_character(),
      id = col_double()
    )

          Karn    Ascites
    -0.2364006  0.4170308
    Time difference of 0.02595592 secs

```python
%%time

coxmod.fit(df_tied, Xnames=Xnames, tname=tname, dname=dname, basehaz=False)
```

    [-0.23638311  0.4170416 ]
    CPU times: user 944 ms, sys: 9.92 ms, total: 954 ms
    Wall time: 276 ms

We indeed match that result as well!

We are about 10x slower than the R package (which runs on C code in the background). But simplicity and extensibility of our code compensates for that in my view.

(Also timings here most likely dominated by overhead of loading libraries, comparison on larger dataset required.)

To be more specific, R's survival package is a blazingly fast and mature package, however the cost of this is that the code itself is non-trivial to edit or modify, see e.g.
https://github.com/therneau/survival/blob/master/src/coxfit6.c

Contrast this with the code for this implementation which is comparably readable and can be found in `torchcox/TorchCox.py`. You can easily see how you could modify the likelihood there to insert regularisation, change the optimiser, modify the log-linear dependence on the covariates, etc.

Fit the model again, but this time computing the baseline hazard, to ensure that works as well.

```python
%%time

coxmod.fit(df_tied, Xnames=Xnames, tname=tname, dname=dname, basehaz=True)
```

    [-0.23638311  0.4170416 ]
    CPU times: user 2.2 s, sys: 95 ms, total: 2.29 s
    Wall time: 1.59 s

```python
coxmod.basehaz
```

      time
      h0
      H0

      0
      0.024657240259862743
      0.018764267
      0.018764267

      1
      0.02739761676265909
      0.018845245
      0.0376095

      2
      0.03561695837826295
      0.018976642
      0.0565862

      3
      0.04109387045962334
      0.019059468
      0.0756456

      4
      0.08219248033739741
      0.019129159
      0.0947748

      ...
      ...
      ...
      ...

      299
      7.0602742018236055
      1.4441799
      28.2556

      300
      7.112329818483311
      1.671192
      29.9268

      301
      7.120547803885543
      2.1946971
      32.1215

      302
      7.290409766965798
      3.6383288
      35.7598

      303
      7.476712205963872
      10.631603
      46.3914

304 rows × 3 columns

Predict on the training set to ensure predict_proba() method works as well.

```python
df_tied['pred'] = coxmod.predict_proba(df_tied, Xnames=Xnames, tname=tname)
df_tied
```

      tyears
      d
      Karn
      Broders
      FIGO
      Ascites
      Diam
      id
      pred

      0
      0.024657
      1
      8
      2.0
      IV
      1
      1-2cm
      281
      0.995712

      1
      0.027398
      1
      6
      NaN
      IV
      1
      >5cm
      298
      0.986277

      2
      0.035617
      1
      8
      2.0
      IV
      1
      >5cm
      342
      0.987125

      3
      0.041094
      1
      7
      2.0
      III
      0
      <1cm
      228
      0.985644

      4
      0.082192
      1
      7
      3.0
      IV
      1
      <1cm
      52
      0.972883

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      299
      7.060274
      0
      10
      3.0
      III
      0
      >5cm
      101
      0.070109

      300
      7.112330
      0
      10
      1.0
      III
      1
      >5cm
      93
      0.013961

      301
      7.120548
      0
      9
      2.0
      III
      1
      <1cm
      40
      0.003005

      302
      7.290410
      0
      9
      2.0
      IV
      1
      1-2cm
      81
      0.001557

      303
      7.476712
      0
      10
      1.0
      III
      0
      2-5cm
      10
      0.012733

304 rows × 9 columns

In summary what are some of the reasons why one might consider implementing a well-known statistical model in a differentiable programming language like PyTorch?
- Extensibility: changes to loss function or optimisation algorithm are often one-line changes
- Scalability: functionality to deploy across multiple CPUs or GPUs is often built in or easy to include
- Mobile deployment: if relevant, models can be deployed on mobile devices (Android or iOS)
- Automatic differentiation: computing confidence intervals originally involved computing second derivatives by hand and implementing the result in the code, with differentiable programming simply changing the loss is sufficient, the computation of second derivatives is automatic (provided loss is twice-differentiable, obviously)
- Ecosystem: integration with existing PyTorch libraries (see https://pytorch.org/ecosystem/) to add all sorts of functionality should be straightforward

```python

```


Exercise material of the MSc-level course **Numerical Methods in Geotechnical Engineering**.
Held at Technische Universität Bergakademie Freiberg.

Comments to:

*Prof. Dr. Thomas Nagel
Chair of Soil Mechanics and Foundation Engineering
Geotechnical Institute
Technische Universität Bergakademie Freiberg.*

https://tu-freiberg.de/en/soilmechanics

```python
import numpy as np
import matplotlib.pyplot as plt

#Some plot settings
plt.style.use('seaborn-deep')
plt.rcParams['lines.linewidth']= 2.0
plt.rcParams['lines.color']= 'black'
plt.rcParams['legend.frameon']=True
plt.rcParams['font.family'] = 'serif'
plt.rcParams['legend.fontsize']=14
plt.rcParams['font.size'] = 14
plt.rcParams['axes.spines.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.left'] = True
plt.rcParams['axes.spines.bottom'] = True
plt.rcParams['axes.axisbelow'] = True
plt.rcParams['figure.figsize'] = (12, 6)
```

# Exercise 7 - 1D Consolidation, Terzaghi

## Governing differential equation

Neglecting lateral displacements and fluid flow

$$
	\epsilon_{xx} = \epsilon_{yy} = 0 \quad \text{and} \quad q_x = q_y = 0
$$

the volume balance simplifies to

$$
	\dot{\epsilon}_{zz} = \frac{\partial q_z}{\partial z}
$$

Now we substitute the constitutive law (linear elasticity) of the solid and Darcy's law for the fluid motion into the volume balance and get

$$
    \dot{\sigma}_{zz}^\text{eff} = E_\text{S} \dot{\epsilon}_{zz} \quad \text{and} \quad \frac{\partial q_z}{\partial z} = - \frac{k}{\mu_\text{F}} \frac{\partial^2 p}{\partial z^2} \quad \text{ergibt} \quad
		 \frac{\dot{\sigma}_{zz}^\text{eff}}{E_\text{S}} = - \frac{k}{\mu_\text{F}} \frac{\partial^2 p}{\partial z^2}
$$

Assuming constant total stress (constant external loads) we find with the use of the effective stress principle

$$
	\sigma_{zz} = \sigma_{zz}^\text{eff} + p = \text{const} \quad \rightarrow \quad \dot{\sigma}_{zz} = 0 = \dot{\sigma}_{zz}^\text{eff} + \dot{p}
$$

Substitution yields the partial, linear and homogeneous differential equation

$$
	\dot{p} = \frac{E_\text{S} k}{\mu_\text{F}} p_{,zz} = \frac{E_\text{S} K_\text{D}}{\gamma_\text{F}} p_{,zz} = c_\text{v} p_{,zz} \quad \text{with } K_\text{D} = \frac{k\rho_\text{F} g}{\mu_\text{F}}
$$

$[k] = \text{m}^2$: intrinsic permeability

$[K_\text{D}] = \text{m s}^{-1}$: hydraulic conductivity

$[c_\text{v}] = \text{m}^2\text{s}^{-1}$: coefficient of consolidation

## Weak form

The pore pressure can have (the essential/Dirichlet) boundary conditions in the form:

$$
    p = \bar{p}\ \forall z \in \partial \Omega_\mathrm{D}
$$

We now introduce a test function $\eta$ which vanishes where the pore pressure is given

$$
    \eta = 0\ \forall z \in \partial \Omega_\mathrm{D}
$$

and construct the weak form (using integration by parts):

\begin{align}
   0 &= \int \limits_0^H \eta \left[\dot{p} - c_\text{v} p_{,zz} \right] \text{d}z
   \\
   &= \int \limits_0^H \left[\eta \dot{p} - \left( \eta c_\text{v} p_{,z} \right)_{,z} + \eta_{,z} c_\text{v} p_{,z} \right] \, \text{d}z
   \\
   &= \int \limits_0^H \left[\eta \dot{p} + \eta_{,z} c_\text{v} p_{,z} \right] \, \text{d}z + \left[ \eta E_\text{S} q_z \right]^H_0
\end{align}

where the natural/Neumann boundary conditions have appeared.

## Finite elements in 1D

We have a soil column of height $H$ on top of the bed rock at $z=0$.

We first create an element class. An element knows the number of nodes it has, their IDs in the global node vector, and the coordinates of its nodes. Linear elements have 2 nodes and 2 quadrature points, quadratic elements 3 nodes and 3 quadrature points. The natural coordinates of the element run from -1 to 1, and the quadrature points and weights are directly taken from Numpy.

```python
#element class
class line_element():#local coordinates go from -1 to 1
    #takes number of nodes, global nodal coordinates, global node ids
    def __init__(self, nnodes=2, ncoords=[0.,1.], nids=[0,1]):
        self.__nnodes = nnodes
        if (len(ncoords) != self.__nnodes):
            raise Exception("Number of coordinates does not match number \
                            of nodes of element (%i vs of %i)" %(self.__nnodes,len(ncoords)))
        else:
            self.__coords = np.array(ncoords)

        self.__natural_coords = (self.__coords-self.__coords[0])/(self.__coords[-1]-self.__coords[0])*2. - 1.

        if (len(nids) != self.__nnodes):
            raise Exception("Number of node IDs does not match number \
                            of nodes of element (%i vs of %i)" %(self.__nnodes,len(nids)))
        else:
            self.__global_ids = np.array(nids)
        self.__quad_degree = self.__nnodes
        self.__quad_points, self.__quad_weights = np.polynomial.legendre.leggauss(self.__quad_degree)

```

Next, we wish to generate a one-dimensional mesh by specifying the length of a line, the number of elements into which the mesh is to be split, and the number of nodes per element.

```python
def number_of_nodes(nelems,nodes_per_elem):
    return nelems*nodes_per_elem - (nelems - 1)

def generate_mesh(domain_length,nelems,nodes_per_elem):
    nn = number_of_nodes(nelems,nodes_per_elem)
    #coordinate vector of global nodes
    global_nodal_coordinates = np.linspace(0.,domain_length,nn)
    global_solution = np.array([0.]*nn)

    #generate elements
    element_vector = []
    for i in range(nelems):
        node_start = (nodes_per_elem-1)*i
        element_vector.append(
            line_element(nodes_per_elem,
                         global_nodal_coordinates[node_start:node_start+nodes_per_elem],
                         list(range(node_start,node_start+nodes_per_elem))))

    return global_nodal_coordinates, element_vector, global_solution
```

## Shape functions in 1D

As in exercise 06, we allow linear and higher-order shape functions.

```python
#N
def shape_function(element_order,xi):
    if (element_order == 2): #-1,1
            return np.array([(1.-xi)/2., (1.+xi)/2.])
    elif (element_order == 3): #-1, 0, 1
            return np.array([(xi - 1.)*xi/2., (1-xi)*(1+xi), (1+xi)*xi/2.])
#dN_dxi
def dshape_function_dxi(element_order,xi):
    if (element_order == 2): #-1,1
        return np.array([-0.5*xi/xi, 0.5*xi/xi])  #xi only later for plotting dimensions
    elif (element_order == 3):#-1,0,1
        return np.array([xi - 0.5,-2.*xi,xi + 0.5])

#dz_dxi
def element_jacobian(element,xi):
    element_order = element._line_element__nnodes
    Jacobian = 0.
    Jacobian += dshape_function_dxi(element_order,xi).dot(element._line_element__coords)
    return Jacobian

#dN_dz
def grad_shape_function(element,xi):
    element_order = element._line_element__nnodes
    Jac = element_jacobian(element,xi)
    return dshape_function_dxi(element_order,xi)/Jac

```

## Time and space discretization, Picard iterations

Using a backward Euler approach for simplicity we find the time-discrete weak form as ($p_{n+1} \equiv p$)

$$
    0 = \int \limits_0^H \left[\eta \frac{p - p_n}{\Delta t} + \eta_{,z} c_\text{v} p_{,z} \right] \, \text{d}z + \left[ \eta E_\text{S} q_z \right]^H_0
$$

Introducing standard FE approximations

$$
    p \approx N_i \hat{p}_i, \quad \eta \approx N_i \hat{\eta}_i, \quad \frac{\partial p}{\partial z} \approx \nabla  N_i \hat{p}_i, \quad \frac{\partial \eta}{\partial z} \approx \nabla N_i \hat{\eta}_i
$$

This yields

$$
\begin{align}
 0 &= \int \limits_0^H \left[ N_i \hat{\eta}_i \frac{N_k \hat{p}_k - N_k \hat{p}_{n,k}}{\Delta t} + \nabla N_i \hat{\eta}_i c_\text{v} \nabla N_k \hat{p}_k \right] \, \text{d}z + \left[ N_i \hat{\eta}_i E_\text{S} q_z \right]^H_0
 \end{align}
$$

Now we bring all quantities associated with the unknown pressure to the left-hand side (LHS) and all known quantities to the RHS:

$$
    \hat{\eta}_i \int \limits_0^H \left[ N_i \frac{1}{\Delta t} N_k + \nabla N_i c_\text{v} \nabla  N_k \right] \, \text{d}z\ \hat{p}_k =  \hat{\eta}_i \left[ N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=H} \delta_{n_\text{n}i} - N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=0} \delta_{i0} \right] +  \hat{\eta}_i \int \limits_0^H N_i \frac{1}{\Delta t} N_k \, \text{d}z\ \hat{p}_{n,k}
$$

can be simplified by realizing that the nodal test function values are arbitrary and thus

$$
    \int \limits_0^H \left[ N_i \frac{1}{\Delta t} N_k + \nabla N_i c_\text{v} \nabla N_k \right] \, \text{d}z\ \hat{p}_k = N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=H} \delta_{n_\text{n}i} - N_{n_\text{n}} E_\text{S} \bar{q}_z|_{z=0} \delta_{i0} +  \int \limits_0^H N_i \frac{1}{\Delta t} N_k \, \text{d}z\ \hat{p}_{n,k}
$$

which leaves us with $n_\text{n}$ equations for the $n_\text{n}$ unknown nodal pressures $\hat{p}_k$. If any coefficients in the above are taken as pressure-dependent, the system could be solved repeatedly usind Picard iterations. For strong non-linearities, a Newton linearization would typically be used.

### Question:

How would the equation differ for an explicit time integration scheme (forward Euler)?

What we require now is the local assembler to calculate the stiffness matrix and the local right-hand side. Local integration is performed by Gauss quadrature:

$$
    \int \limits_{-1}^1 f(\xi)\,\text{d}\xi \approx \sum \limits_{i=1}^{n_\text{gp}} f(\xi_i) w_i
$$

## Local assember

```python
def Stiffness(z):
    E0 = 5.e6 #Pa
    return E0

def Conductivity(z):
    kf = 1.e-9#m/s
    return kf

def SpecificWeight(z):
    g = 9.81 #m/s²
    rhow = 1000. #kg/m³
    return g*rhow

def ConsolidationCoeff(z):#m²/s
    return Stiffness(z)*Conductivity(z)/SpecificWeight(z)
```

```python
def local_assembler(elem,dt,prev_sol,mass_lumping=False):
    element_order = elem._line_element__nnodes
    K_loc = np.zeros((element_order,element_order))
    M_loc = np.zeros((element_order,element_order))
    b_loc = np.zeros(element_order)
    z_nodes = elem._line_element__coords
    for i in range(elem._line_element__quad_degree):
        #local integration point coordinate
        xi = elem._line_element__quad_points[i]
        #shape function
        N = shape_function(element_order,xi)
        #gradient of shape function
        dN_dX = grad_shape_function(elem,xi)
        #determinant of Jacobian
        detJ = np.abs(element_jacobian(elem,xi))
        #integration weight
        w = elem._line_element__quad_weights[i]

        #global integration point coordinate (for spatially varying properties)
        z_glob = np.dot(N,z_nodes)
        #evaluation of local material/structural properties
        E = Stiffness(z_glob)
        #evaluation of local body force
        CV = ConsolidationCoeff(z_glob)

        #assembly of local stiffness matrix
        M_loc = np.outer(N,N) / dt
        if (mass_lumping):
            M_loc = np.diag(M_loc.sum(0)) #diagonal of column sum

        K_loc += (np.outer(dN_dX,dN_dX) * CV + M_loc)* w * detJ

        #assembly of local RHS
        p_prev = np.dot(N,prev_sol)#pressure in integration point
        b_loc += N * p_prev/dt * w * detJ
    return K_loc,b_loc
```

## Global assembly

Now we can construct the global matrix system $\mathbf{K}\mathbf{u} = \mathbf{f}$ or $\mathbf{A}\mathbf{x}=\mathbf{b}$ (see lecture script).

```python
def global_assembler(nodes,elements,solution,dt,mass_lumping=False):
    K_glob = np.zeros((len(nodes),len(nodes)))
    b_glob = np.zeros(len(nodes))
    for i,elem in enumerate(elements):
        start_id = elem._line_element__global_ids[0]
        end_id = elem._line_element__global_ids[-1]
        K_i, b_i = local_assembler(elem,dt,solution[start_id:end_id+1],mass_lumping)
        K_glob[start_id:end_id+1,start_id:end_id+1] += K_i
        b_glob[start_id:end_id+1] += b_i
    return K_glob, b_glob
```

## Application of boundary conditions

First we apply flux boundary conditions

```python
def apply_Neumann_bc(b_glob,node_id,value):
    b_glob[node_id] += value
    return b_glob
```

Then we apply Dirichlet bc

```python
def apply_Dirichlet_bc(K_glob,b_glob,node_id,value):
    K_glob[node_id,:] = 0.# = K_glob[:,node_id] = 0.
    K_glob[node_id,node_id] = 1.
    b_glob[node_id] = value
    return K_glob, b_glob
```

## Application of initial conditions

Since we're dealing with a time-dependent problem (rate problem) we require initial conditions for the pore pressure, i.e. $p_0 = p(t=0)\ \forall\ z$. Due to the very specific assumptions in deriving this consolidation equation, the initial pressure is given by the (suddenly) applied load:

$$
    p_0 = \sigma_{zz} = \text{const.}
$$

```python
def apply_initial_conditions(solution,sig_v):
    solution *= 0.
    solution += sig_v
    return
```

## Time loop and problem solution

We now establish the time loop and in each time step perform the global assembly, apply a vanishing traction on the top and constrain the displacement at the bottom to zero.

```python
def time_loop(dt,nodes,elements,solution,mass_lumping=False):
    #Startwerte
    t_end = 366*24*60*60 #s
    absolute_tolerance = 1.e-6
    max_iter = 100
    iteration_counter = np.array([0])
    apply_initial_conditions(solution,200.e3)
    y = [solution] #create a list that will hold the solution vectors at all time points
    times = np.array([0.])
    #
    while times[-1]+dt < t_end: #repeat the loop as long as the final time step is below the end point
        times = np.append(times,times[-1]+dt) #here define the next time point as the previous time point plus the time increment dt
        y_old = y[-1] #Starting value for recursive update
        i = 0
        #
        while True:
            K, f = global_assembler(nodes,elements,y[-1],dt,mass_lumping)
            #f = apply_Neumann_bc(f,len(nodes)-1,0)
            K, f = apply_Dirichlet_bc(K, f, len(nodes)-1, 0.)#free draining top
            solution = np.linalg.solve(K,f)
            i += 1
            if (np.abs(np.linalg.norm(solution) - np.linalg.norm(y_old)) < absolute_tolerance or i > max_iter): #if change is below tolerance, stop iterations
                break
            y_old = solution #preparation of next recursion

        y.append(solution) #append the new found solution to the solution vector
        iteration_counter = np.append(iteration_counter,i) #store how much iterations this time step took to converge
    return times, y,iteration_counter
```

```python
#spatial discretization
H = 10.
nel = 20
n_per_el = 3
nodes,elements,solution=generate_mesh(H,nel,n_per_el)
```

```python
times, sols, iters = time_loop(24*60*60,nodes,elements,solution)
```

```python
plt.xlabel('$z$ / m')
plt.ylabel('$p$ / kPa')
plt.title('Finite element solution')
plt.plot(nodes, sols[0]/1.e3, marker='o', label='$t = %i$ d' %(times[0]/60/60))
plt.plot(nodes, sols[1]/1.e3, marker='o', label='$t = %i$ d' %(times[1]/60/60/24))
plt.plot(nodes, sols[10]/1.e3, marker='o', label='$t = %i$ d' %(times[10]/60/60/24))
plt.plot(nodes, sols[30]/1.e3, marker='o', label='$t = %i$ d' %(times[30]/60/60/24))
plt.plot(nodes, sols[200]/1.e3, marker='o', label='$t = %i$ d' %(times[200]/60/60/24))
plt.plot(nodes, sols[-1]/1.e3, marker='o', label='$t = %i$ d' %(times[-1]/60/60/24))
plt.legend();
```

```python
#from matplotlib import animation
#from IPython.display import HTML

# First set up the figure, the axis, and the plot element we want to animate
#fig, ax = plt.subplots();
#ax.set_xlim(( 0, 11));
#ax.set_ylim((0, 210));

#line, = ax.plot([], [], lw=2);

# initialization function: plot the background of each frame
#def init():
#    line.set_data(nodes, sols[0]/1e3)
#    return (line,)

# animation function. This is called sequentially
#def animate(i):
    #x = nodes
    #y = sols[i]/1e3
    #line.set_data(x, y)
    #return (line,)

# call the animator. blit=True means only re-draw the parts that have changed.
#anim = animation.FuncAnimation(fig, animate, init_func=init,
#                               frames=len(sols)-1, interval=20, blit=True);
#HTML(anim.to_html5_video())
```

### Convergence study

Let's do a simple convergence study.

```python
dts = [12.*3600,24.*3600,48.*3600]
nels = [10,20,40]
H = 10.

fig, ax = plt.subplots(nrows=3,ncols=3,figsize=(18,18))

for i,dt in enumerate(dts):
    for j,nel in enumerate(nels):
        #print("Running (n,dt) combination ", dt, nel)
        number_of_elements = nel
        nodes_per_element = 2
        nodes,elements,solution=generate_mesh(H,number_of_elements,nodes_per_element)
        times, sols, iters = time_loop(dt,nodes,elements,solution)
        nodes,elements,solution=generate_mesh(H,number_of_elements,nodes_per_element)
        times_lumped, sols_lumped, iters_lumped = time_loop(dt,nodes,elements,solution,True)
        #
        ax[i][j].plot(nodes, sols[0]/1.e3, marker='o', color='green', label='$t = %i$ d' %(times[0]/60/60/24))
        ax[i][j].plot(nodes, sols[1]/1.e3, marker='o', color='red', label='$t = %i$ d' %(times[1]/60/60/24))
        ax[i][j].plot(nodes, sols[10]/1.e3, marker='o', color='blue', label='$t = %i$ d' %(times[10]/60/60/24))
        #
        ax[i][j].plot(nodes, sols_lumped[0]/1.e3, marker='d', ls=':', color='green')
        ax[i][j].plot(nodes, sols_lumped[1]/1.e3, marker='d', ls=':', color='red')
        ax[i][j].plot(nodes, sols_lumped[10]/1.e3, marker='d', ls=':', color='blue')
        #
        ax[i][j].set_xlabel('$z$ / m')
        ax[i][j].set_ylabel('$p$ / kPa')
        ax[i][j].set_title('dt = %i h, nel = %i' %(dt/3600,nel))
        ax[i][j].legend(fontsize=10)
fig.tight_layout()
```

We observe that time step size and discretization size act together, and that unsuitable choices can lead to unphysical oscillations.


# Lagrangian mechanics

Marcos Duarte

The [Lagrangian mechanics](https://en.wikipedia.org/wiki/Lagrangian_mechanics) is a formulation of [classical mechanics](https://en.wikipedia.org/wiki/Classical_mechanics) where the [equations of motion](https://en.wikipedia.org/wiki/Equations_of_motion) are obtained from the [kinetic](https://en.wikipedia.org/wiki/Kinetic_energy) and [potential](https://en.wikipedia.org/wiki/Potential_energy) energy of the system (scalar quantities) represented in [generalized coordinates](https://en.wikipedia.org/wiki/Generalized_coordinates) instead of using [Newton's laws of motion](https://en.wikipedia.org/wiki/Newton's_laws_of_motion) to deduce the equations of motion from the [forces](https://en.wikipedia.org/wiki/Force) on the system ([vector](https://en.wikipedia.org/wiki/Euclidean_vector) quantities) represented in [Cartesian coordinates](https://en.wikipedia.org/wiki/Cartesian_coordinate_system). Lagrangian mechanics was introduced by [Joseph-Louis Lagrange](https://en.wikipedia.org/wiki/Joseph-Louis_Lagrange) in the late 18th century.

Let's deduce the Lagrange equations, but first let's review the basics of the Newtonian approach.

## Description of motion

One can describe the motion of a particle by specifying its position with respect to a frame of reference in the three-dimensional space as a function of time:

\begin{equation}
x(t),\, y(t),\, z(t) \quad \equiv \quad x_i(t) \quad i=1,\dotsc,3
\label{eq1}
\end{equation}

A system of $N$ particles will require $3N$ equations to describe their motion:

\begin{equation}
\sum\limits_{i=1}^{3N} {x_i(t)}
\label{eq2}
\end{equation}

The basic problem in classical mechanics is to find ways to determine functions such as these, also known as equations of motion, capable of specifying the position of objects over time, for any mechanical situation. Assuming as known the meaning of $x_i(t)$, one can define the components of velocity, $v_i$, and acceleration, $a_i$, at time $t$, as:

\begin{equation} \begin{array}{rcl}
v_i(t) = \frac{\mathrm d x_i(t)}{\mathrm d t} = \dot{x}_i(t) \\
a_i(t) = \frac{\mathrm d^2 x_i(t)}{\mathrm d t^2} = \dot{v}_i(t)
\label{eq3}
\end{array} \end{equation}

Where we used the Newton's notation for differentiation (also called the dot notation), a dot over the dependent variable.
Of note, [Joseph Louis Lagrange](https://pt.wikipedia.org/wiki/Joseph-Louis_Lagrange) introduced the prime mark to denote a derivative: $x'(t)$. Read more about the different notations for differentiation at [Wikipedia](https://en.wikipedia.org/wiki/Notation_for_differentiation).

## Laws of motion

The [Newton's laws of motion](https://en.wikipedia.org/wiki/Newton's_laws_of_motion) laid the foundation for classical mechanics. They describe the relationship between the motion of a body and the possible forces acting upon it. Here are these three laws in Newton's own words ([*Philosophiæ Naturalis Principia Mathematica*](http://archive.org/details/newtonspmathema00newtrich), p. 83 of Book I of the first American edition):

> LAW I
> *Every body perseveres in its state of rest, or of uniform motion in a right line, unless it is compelled to change that state by forces impressed thereon.*
> LAW II
> *The alteration of motion is ever proportional to the motive force impressed; and is made in the direction of the right line in which that force is impressed.*
> LAW III
> *To every action there is always opposed an equal reaction: or the mutual actions of two bodies upon each other are always equal, and directed to contrary parts.*

Regarding the second law, by alteration of motion, Newton was referring to the change in time of the body's linear momentum, which is defined as:

\begin{equation}
\mathbf{p} = m\mathbf{v}
\label{eq5}
\end{equation}

If the mass of the body is constant, the second law can be expressed by the well known equation:

\begin{equation}
\mathbf{F} = \frac{\mathrm d \mathbf{p}}{\mathrm d t}=\frac{\mathrm d \big(m\mathbf{v}\big)}{\mathrm d t} = m\mathbf{a}
\label{eq6}
\end{equation}

Using Newton's second law, to determine the position of the body we will have to solve the following second order ordinary differential equation:

\begin{equation}
\frac{\mathrm d^2 x_i(t)}{\mathrm d t^2} = \frac{\mathbf{F}}{m}
\label{eqa7}
\end{equation}

Which has the general solution:

\begin{equation}
\mathbf{x}(t) = \int\!\bigg(\int\frac{\mathbf{F}}{m} \mathrm{d}t\bigg)\mathrm{d}t
\label{eq8}
\end{equation}

A related physical quantity is the mechanical energy, which is the sum of kinetic and potential energies.
The kinetic energy, $T$ of a particle is given by:

\begin{equation}
T = \frac{1}{2}mv^2
\label{eq9}
\end{equation}

The kinetic energy of a particle can be expressed in terms of its linear momentum:

\begin{equation}
T = \frac{p^2}{2m}
\label{eq10}
\end{equation}

And for a given coordinate of the particle's motion, its linear momentum can be obtained from its kinetic energy by:

\begin{equation}
p_i = \frac{\partial T}{\partial v_i}
\label{eq11}
\end{equation}

The potential energy, $V$ is the stored energy of a particle and its formulation is dependent on the force acting on the particle. For example, for a conservative force dependent solely on the particle position, such as due to the gravitational field near the Earth  surface or due to a linear spring, force and potential energy are related by:

\begin{equation}
\mathbf{F} = - \frac{\partial \mathbf{V}}{\partial x}
\label{eq12}
\end{equation}

### Lagrange's equation in Cartesian Coordinates

For simplicity, let's first deduce the Lagrange's equation for a particle in Cartesian Coordinates.

Because we want to deduce the laws of motion based on the mechanical energy of the particle, one can see that the time derivative of the expression for the linear momentum as a function of the kinetic energy, cf. Eq. (\ref{eq11}), is equal to the force acting on the particle and we can substitute the force in Newton's second law by this term:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\bigg(\frac{\partial T}{\partial \dot x}\bigg) = m\ddot x
\label{eq13}
\end{equation}

We saw that a conservative force can also be expressed in terms of the potential energy of the particle, cf. Eq. (\ref{eq12}); substituting the right side of the equation above by this expression, we have:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\bigg(\frac{\partial T}{\partial \dot x}\bigg) = -\frac{\partial V}{\partial x}
\label{eq14}
\end{equation}

Using the fact that:

\begin{equation}
\frac{\partial T}{\partial x} = 0 \quad and \quad \frac{\partial V}{\partial \dot x} = 0
\label{eq15}
\end{equation}

We can write:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\bigg(\frac{\partial (T-V)}{\partial \dot x}\bigg) - \frac{\partial (T-V)}{\partial x} = 0
\label{eq16}
\end{equation}

Defining the Lagrange or Lagrangian function, $\mathcal{L}$, as the difference between the kinetic and potential energy in the system:

\begin{equation}
\mathcal{L} = T - V
\label{eq17}
\end{equation}

We have the Lagrange's equation in Cartesian Coordinates for a conservative force acting on a particle:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\bigg(\frac{\partial \mathcal{L}}{\partial \dot x}\bigg) - \frac{\partial \mathcal{L}}{\partial x} = 0
\label{eq18}
\end{equation}

Once all derivatives of the Lagrangian function are calculated, this equation will be the equation of motion for the particle. If there are $N$ independent particles in a three-dimensional space, there will be $3N$ equations for the system.

Let's see some simple examples of the Lagrange's equation in Cartesian Coordinates just to consolidate what was deduced above. The real application of Lagrangian mechanics is in generalized coordinates, what will see after these examples.

### Example 1: Particle moving under the influence of a conservative force

Let's deduce the equation of motion using the Lagrangian mechanics for a particle with mass $m$ moving in the 3D space under the influence of a [conservative force](https://en.wikipedia.org/wiki/Conservative_force).
The Lagrangian of the particle is:

\begin{equation}
\mathcal{L}(x,y,z,\dot x,\dot y,\dot z) = \frac{1}{2}m(\dot x^2 + \dot y^2 + \dot z^2) - V(x,y,z)
\end{equation}

The equations of motion for the particle are found by applying the Lagrange's equation for each coordinate.

For the x coordinate:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{x}}}
\right) = \frac{\partial \mathcal{L}}{\partial x }
\end{equation}

And the derivatives of $\mathcal{L}$ are given by:

\begin{equation}
\frac{\partial \mathcal{L}}{\partial x} = -\frac{\partial \mathbf{V}}{\partial x}, \quad \frac{\partial \mathcal{L}}{\partial \dot{x}} =m\dot{x}, \quad \frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{x}}} \right) = m\ddot{x}
\end{equation}

hence:

\begin{equation}
m\ddot{x} = -\frac{\partial \mathbf{V}}{\partial x}
\end{equation}

and similarly for the $y$ and $z$ coordinates.

For instance, if the conservative force is due to the gravitational field near Earth's surface $(\mathbf{V}=[0, mgy, 0])$, the Lagrange's equations (the equations of motion) are:

\begin{equation} \begin{array}{rcl}
m\ddot{x} &=& -\frac{\partial (0)}{\partial x} = 0 \\
m\ddot{y} &=& -\frac{\partial (mgy)}{\partial y} = -mg \\
m\ddot{z} &=& -\frac{\partial (0)}{\partial z} = 0
\end{array} \end{equation}

### Example 2: Ideal mass-spring system

Consider a system with a mass $m$ attached to an ideal spring (massless) with spring constant $k$ at the horizontal direction $x$. If the system is perturbed (a force is momentarily applied to the mass), we know the mass will oscillate around the rest position of the spring. Let's deduce the equation of motion for this system using the Lagrangian mechanics.
The Lagrangian of the system is:

\begin{equation}
\mathcal{L}(x,\dot x) = \frac{1}{2}m\dot x^2 - \frac{1}{2}kx^2
\end{equation}

The derivatives of $L$ are given by:

\begin{equation}
\frac{\partial \mathcal{L}}{\partial x} = -kx, \quad \frac{\partial \mathcal{L}}{\partial \dot{x}} =m\dot{x}, \quad \frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{x}}} \right) = m\ddot{x}
\end{equation}

And the Lagrange's equation (the equation of motion) is:

\begin{equation}
m\ddot{x} + kx = 0
\end{equation}

## Generalized coordinates

The direct application of Newton's laws to mechanical systems results in a set of equations of motion in terms of Cartesian coordinates of each of the particles that make up the system. In many cases, this is not the most convenient coordinate system to solve the problem or describe the movement of the system. For example, in problems involving many particles, it may be convenient to choose a system that includes the coordinate of the center of mass. Another example is a serial chain of rigid links, such as a member of the human body or from a robot manipulator, it may be simpler to describe the positions of each link by the angles between links.

Coordinate systems such as these are referred as [generalized coordinates](https://en.wikipedia.org/wiki/Generalized_coordinates). Generalized coordinates uniquely specify the positions of the particles in a system. Although there may be several generalized coordinates to describe a system, usually a judicious choice of generalized coordinates provides the minimum number of independent coordinates that define the configuration of a system (which is the number of degrees of freedom of the system), turning the problem simpler to solve.

In problems where it is desired to use generalized coordinates, one can write Newton's equations of motion in terms of Cartesian coordinates and then transform them into generalized coordinates. However, it would be desirable and convenient to have a general method that would directly establish the equations of motion in terms of a set of convenient generalized coordinates. In addition, general methods for writing, and perhaps solving, the equations of motion in terms of any coordinate system would also be desirable. The [Lagrangian mechanics](https://en.wikipedia.org/wiki/Lagrangian_mechanics) is such a method.

## Lagrange's equation in generalized coordinates

We have deduced the Lagrange's equation from the Newton's law just because it was a simple form of getting to the final equations. But, by no means the Lagrangian Mechanics should be viewed as derived from Newton's laws. The Lagrangian Mechanics can be deduced completely independent of Newton's law; let's deduce now the Lagrange's equation in generalized coordinates independently of Newton's law.
We will obtain exactly the same equation; you can skip this part and jump directly to the examples if you wish.

When describing a system of particles using any set of generalized coordinates, $q_1,\dotsc,q_{3N}$, these are related to the Cartesian coordinates by:

\begin{equation} \begin{array}{rcl}
q_i =q_i (x_1,\dotsc,x_{3N} ) \quad i=1,\dotsc,3N \\
x_i =x_i (q_1,\dotsc,q_{3N} ) \quad i=1,\dotsc,3N
\label{eq24}
\end{array} \end{equation}

The Cartesian components of velocity as a function of generalized coordinates are:

\begin{equation}
\dot{x}_i =\frac{\mathrm d x_i (q_1, q_2,\dotsc,q_{3N}
)}{\mathrm d t}=\sum\limits_{j=1}^{3N} {\frac{\partial x_i }{\partial q_j }}
\frac{\mathrm d q_j }{\mathrm d t}
\label{eq26}
\end{equation}

where for simplicity we omitted the explicit mention of the temporal dependence of each coordinate.

That is, any Cartesian component of the particle velocity as a function of generalized coordinates is a function of all the components of position and velocity in the generalized coordinates:

\begin{equation}
\dot{x}_i = \dot{x}_i (q_1,\dotsc,q_{3N} ,\dot{q}_1,\dotsc,\dot{q}_{3N} ) \quad i=1,\dotsc,3N
\label{eq27}
\end{equation}

Let's now deduce the Lagrange's equation in generalized coordinates. In fact, the Lagrange's equation in generalized coordinates will be similar to the equation in Cartesian Coordinates; the only difference will be that we will use $q_i$ will instead of the Cartesian coordinate.

In analogy to Newtonian mechanics, cf. Eq. (\ref{eq6}), one can think that the equations of motion can be obtained by equating the generalized force, $F_i$, to the temporal rate of change of each generalized momentum, $p_i$:

\begin{equation}
F_i =\frac{\partial p_i }{\partial t}
\label{eq28}
\end{equation}

In the formula above, let's substitute the quantity $p_i$ by its definition in terms of the kinetic energy:

\begin{equation}
\frac{\partial p_i }{\partial t} =\frac{\partial }{\partial t}\left( {\frac{\partial T}{\partial
\dot{q}_i }} \right)=\frac{\partial }{\partial t}\left(
{\sum\limits_{j=1}^{3N} {m_j \dot{x}_j \frac{\partial \dot{x}_j
}{\partial \dot{q}_i }} } \right)
\label{eq29}
\end{equation}

where we used:

\begin{equation}
\frac{\partial T}{\partial \dot{q}_i }=\sum\limits_{j=1}^{3N}
{\frac{\partial T}{\partial \dot{x}_j }\frac{\partial \dot{x}_j
}{\partial \dot{q}_i }}
\label{eq30}
\end{equation}

Using the [product rule](https://en.wikipedia.org/wiki/Product_rule), the derivative of the product in Eq. (\ref{eq29}) is:

\begin{equation}
\frac{\partial p_i }{\partial t}=\sum\limits_{j=1}^{3N} {m_j
\ddot{x}_j \frac{\partial \dot{x}_j }{\partial \dot{q}_i }}
+\sum\limits_{j=1}^{3N} {m_j \dot{x}_j \frac{\mathrm d }{\mathrm d t}\left(
{\frac{\partial \dot{x}_j }{\partial \dot{q}_i }} \right)}
\label{eq31}
\end{equation}

But:

\begin{equation}
\frac{\partial \dot{x}_i }{\partial \dot{q}_j }=\frac{\partial x_i
}{\partial q_j } \quad because \quad \frac{\partial
\dot{x}_i }{\partial \dot{q}_j }=\frac{\partial x_i }{\partial
t}\frac{\partial t}{\partial q_j }=\frac{\partial x_i }{\partial q_j}
\label{eq32}
\end{equation}

Then:

\begin{equation}
\frac{\partial p_i }{\partial t}=\sum\limits_{j=1}^{3N} {m_j
\ddot{x}_j \frac{\partial x_j }{\partial q_i }}
+\sum\limits_{j=1}^{3N} {m_j \dot{x}_j \frac{\mathrm d }{\mathrm d t}\left(
{\frac{\partial x_j }{\partial q_i }} \right)}
\label{eq33}
\end{equation}

The first term on the right side of the equation above is proportional to $m_j
\ddot{x}_j$ and we will define as the generalized force, $Q_i$. But, different from Newtonian mechanics, the temporal variation of the generalized momentum is equal to the generalized force plus another term, which will investigate now. The last part of this second term can be derived as:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial x_j }{\partial q_i }} \right) =
\sum\limits_{k=1}^{3N} {\frac{\mathrm d }{\mathrm d q_k }\left( {\frac{\partial
x_j }{\partial q_i }} \right)\frac{\mathrm d q_k }{\mathrm d t}} =\sum\limits_{k=1}^{3N}
{\frac{\partial^2 x_j }{\partial q_k \partial q_i }\dot{q}_k }
\label{eq34}
\end{equation}

where we used the [chain rule](https://en.wikipedia.org/wiki/Chain_rule) for the differentiation:
\begin{equation}
\frac{\mathrm d }{\mathrm d t}\Big( {f\big({g(t)}\big)}\Big) = \frac{\partial f}{\partial g}\frac{\partial g}{\partial t}
\label{eq35}
\end{equation}

But if we look at Eq. (\ref{eq26}) we see that the term at the right side of the Eq. (\ref{eq34}) can be obtained by:

\begin{equation}
\frac{\partial \dot{x}_j }{\partial q_i } = \frac{\partial }{\partial q_i }\left(\sum\limits_{k=1}^{3N} \frac{\partial
x_j }{\partial q_i }\dot{q}_k \right) = \sum\limits_{k=1}^{3N}
{\frac{\partial^2 x_j }{\partial q_k \partial q_i }\dot{q}_k }
\label{eq36}
\end{equation}

Comparing the Eq. (\ref{eq34}) and Eq. (\ref{eq36}) we have:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial x_j }{\partial q_i }} \right) =
\frac{\mathrm d }{\mathrm d q_i}\left( {\frac{\partial x_j }{\partial t }} \right)
\label{eq37}
\end{equation}

On the other hand, it is possible to relate the term $\partial \dot{x}_j / \partial q_i$ to the derivative of kinetic energy with respect to the coordinate $q_i$:

\begin{equation}
\frac{\partial T}{\partial q_i }=\frac{\partial }{\partial q_i }\left(
{\sum\limits_{j=1}^{3N} {\frac{1}{2}m_j \dot{x}_j^2} }
\right)=\sum\limits_{j=1}^{3N} {m_j \dot{x}_j } \frac{\partial
\dot{x}_j }{\partial q_i }
\label{eq38}
\end{equation}

where once again we used the chain rule for the differentiation.

Using Eq. (\ref{eq37}), Eq. (\ref{eq38}) becomes

\begin{equation}
\frac{\partial T}{\partial q_i }=\sum\limits_{j=1}^{3N} {m_j
\dot{x}_j } \frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial x_j }{\partial q_i }}
\right)
\label{eq39}
\end{equation}

Returning to Eq. (\ref{eq33}), it can be rewritten as:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial T}{\partial \dot{q}_i }} \right) = Q_i + \frac{\partial T}{\partial q_i }
\label{eq40}
\end{equation}

and

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial T}{\partial \dot{q}_i }} \right) - \frac{\partial T}{\partial q_i } = Q_i
\label{eq41}
\end{equation}

Now let's look at $Q_i$, the generalized force. It can be decomposed into two terms:

The first term, composed of the conservative forces, i.e. forces that can be written as potential gradients:

\begin{equation}
Q_C =-\frac{\partial V}{\partial q_i } \quad , \quad V=V\left( {q_1,\dotsc,q_{3N} } \right)
\label{eq42}
\end{equation}

An example of conservative force is the gravitational force.

And the second term, encompassing all non-conservative forces, such as the frictional force. For now, we will only consider conservative forces.

Considering only conservative forces, Eq. (\ref{eq41}) becomes

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial T}{\partial \dot{q}_i }}
\right)-\frac{\partial T}{\partial q_i }=-\frac{\partial V}{\partial q_i}
\label{eq43}
\end{equation}

Rearranging, we have:

\begin{equation} \frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \left( {T-V} \right)}{\partial
\dot{q}_i }} \right)-\frac{\partial \left( {T-V} \right)}{\partial q_i} = 0
\label{eq44}
\end{equation}

This is possible because:

\begin{equation}
\frac{\partial V}{\partial \dot{q}_i} = 0
\label{eq45}
\end{equation}

Defining:

\begin{equation}
\mathcal{L} \equiv \mathcal{L}(q_1,\dotsc,q_{3N} ,\dot{q}_1,\dotsc,\dot{q}_{3N} ) = T - V
\label{eq46}
\end{equation}

as the Lagrange or Lagrangian function, we have the Lagrange's equation:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{q}_i }}
\right)-\frac{\partial \mathcal{L}}{\partial q_i } = 0 \quad i=1,\dotsc,3N
\label{eq47}
\end{equation}

Once all derivatives of the Lagrangian function are calculated, this equation will be the equation of motion for each particle. If there are  $N$  independent particles in a three-dimensional space, there will be  $3N$  equations for the system.

### Constraints

An important class of problems in mechanics, in which the Lagrangian equations are particularly useful, are composed of constrained systems. A constraint is a restriction on the freedom of movement of a particle or a system of particles. A rigid body, or the movement of a pendulum, are examples of constrained system of particles. It can be shown, in a similar way, that the Lagrange equation, deduced here for a system of free particles, is also valid for a system of particles under the action of constraints. The Lagrange's equation, for a system of $3N$ particles and with $k$ constraints, is then defined as:

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{q}_i}} \right)-\frac{\partial \mathcal{L}}{\partial q_i } = 0 \quad i=1,\dotsc,3N-k
\label{eq48}
\end{equation}

### Example 3: Simple pendulum under the influence of gravity

Consider a pendulum with a massless rod of length $d$ and a mass $m$ at the extremity swinging in a plane forming the angle $\theta$ with vertical. Although the pendulum moves at the plane, it only has one degree of freedom, which can be described by the angle $\theta$, the generalized coordinate. It is not difficult to find the equation of motion using Newton's law, but let's find it using the Lagrangian mechanics.

The kinetic energy is:

\begin{equation} T = \frac{1}{2}md^2\dot\theta^2 \end{equation}

And the potential energy is:

\begin{equation} V = -mgd\cos\theta \end{equation}

The Lagrangian function is:

\begin{equation} \mathcal{L} = \frac{1}{2}md^2\dot\theta^2 + mgd\cos\theta \end{equation}

And the derivatives are given by:

\begin{equation} \frac{\partial \mathcal{L}}{\partial \theta} = -mgd\sin\theta, \quad \frac{\partial \mathcal{L}}{\partial \dot{\theta}} =md^2\dot{\theta}, \quad \frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{\theta}}} \right) = md^2\ddot{\theta} \end{equation}

Finally, the Lagrange's equation (the equation of motion) is:

\begin{equation} md^2\ddot\theta + mgd\sin\theta = 0 \end{equation}

### Example 4: Double pendulum under the influence of gravity

Consider a double pendulum (one pendulum attached to another) with massless rods of length $d_1$ and $d_2$ and masses $m_1$ and $m_2$ at the extremities of each rod swinging in a plane forming the angles $\theta_1$ and $\theta_2$ with vertical. This case could be solved using Newtonian mechanics, but it's not simple (e.g., see [this link](http://www.myphysicslab.com/pendulum/double-pendulum/double-pendulum-en.html)). Instead, let's use the Lagrangian mechanics to derive the equations of motion for each pendulum.

The position of $m_1$ is:
$x_1 = d_1\sin\theta_1$
$y_1 = -d_1\cos\theta_1$

And the position of $m_2$ is:
$x_2 = d_1\sin\theta_1 + d_2\sin\theta_2$
$y_2 = -d_1\cos\theta_1 - d_2\cos\theta_2$

The kinetic energy of the system is:

\begin{equation}
T = \frac{1}{2}m_1(\dot x_1^2 + \dot y_1^2) + \frac{1}{2}m_2(\dot x_2^2 + \dot y_2^2)
\end{equation}

And the potential energy of the system is:

\begin{equation}
V = m_1gy_1 + m_2gy_2
\end{equation}

However, because of the constraints introduced by the constant lengths of the rods, each mass has only one degree of freedom and the coordinates $(\theta_1,\theta_2)$ can be chosen as the generalized coordinates of the system.
The kinetic and potential energies of the system in terms of $(\theta_1,\theta_2)$ are:

\begin{equation}
T = \frac{1}{2}m_1d_1^2\dot\theta_1^2 + \frac{1}{2}m_2\big(d_1^2\dot\theta_1^2 + d_2^2\dot\theta_2^2 + 2d_1d_2\dot\theta_1\dot\theta_2\cos(\theta_1-\theta_2)\big)
\end{equation}

And

\begin{equation}
V = -m_1gd_1\cos\theta_1 - m_2g(d_1\cos\theta_1 + d_2\cos\theta_2)
\end{equation}

The Lagrangian function is:

\begin{equation}
\mathcal{L} = \frac{1}{2}(m_1+m_2)d_1^2\dot\theta_1^2 + \frac{1}{2}m_2d_2^2\dot\theta_2^2 + m_2d_1d_2\dot\theta_1\dot\theta_2\cos(\theta_1-\theta_2) + (m_1+m_2)gd_1\cos\theta_1 + m_2gd_2\cos\theta_2
\end{equation}

The derivatives w.r.t. $\theta_1$ are:

\begin{equation}
\frac{\partial \mathcal{L}}{\partial \theta_1} = -m_2d_1d_2\dot\theta_1\dot\theta_2\sin(\theta_1-\theta_2) - (m_1+m_2)gd_1\sin\theta_1
\end{equation}

\begin{equation}
\frac{\partial \mathcal{L}}{\partial \dot{\theta_1}} = (m_1+m_2)d_1^2\dot\theta_1 + m_2d_1d_2\dot\theta_2\cos(\theta_1-\theta_2)
\end{equation}

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{\theta_1}}} \right) = (m_1+m_2)d_1^2\ddot\theta_1 + m_2d_1d_2\ddot\theta_2\cos(\theta_1-\theta_2) - m_2d_1d_2\dot\theta_2\sin(\theta_1-\theta_2)(\dot\theta_1-\dot\theta_2)
\end{equation}

And the derivatives w.r.t. $\theta_2$ are:

\begin{equation}
\frac{\partial \mathcal{L}}{\partial \theta_2} = m_2d_1d_2\dot\theta_1\dot\theta_2\sin(\theta_1-\theta_2) - m_2gd_2\sin\theta_2
\end{equation}

\begin{equation}
\frac{\partial \mathcal{L}}{\partial \dot{\theta_2}} = m_2d_2^2\dot\theta_2 + m_2d_1d_2\dot\theta_1\cos(\theta_1-\theta_2)
\end{equation}

\begin{equation}
\frac{\mathrm d }{\mathrm d t}\left( {\frac{\partial \mathcal{L}}{\partial \dot{\theta_2}}} \right) = m_2d_2^2\ddot\theta_2 + m_2d_1d_2\ddot\theta_1\cos(\theta_1-\theta_2) - m_2d_1d_2\dot\theta_1\sin(\theta_1-\theta_2)(\dot\theta_1-\dot\theta_2)
\end{equation}

Finally, the Lagrange's equations (the equations of motion) are:

\begin{equation}\begin{array}{rcl}
(m_1+m_2)d_1\ddot\theta_1 + m_2d_2\ddot\theta_2\cos(\theta_1-\theta_2) + m_2d_2\dot\theta_2^2\sin(\theta_1-\theta_2) + g(m_1+m_2)\sin\theta_1 = 0 \\
m_2d_2\ddot\theta_2 + m_2d_1\ddot\theta_1\cos(\theta_1-\theta_2) - m_2d_1\dot\theta_1^2\sin(\theta_1-\theta_2) + m_2g\sin\theta_2 = 0
\end{array}\end{equation}

## Considerations on the Lagrangian mechanics

The Lagrangian mechanics does not constitute a new theory in classical mechanics; the results of a Lagrangian or Newtonian analysis must be the same for any mechanical system, only the method used to obtain the results is different.

We are accustomed to think of mechanical systems in terms of vector quantities such as force, velocity, angular momentum, torque, etc., but in the Lagrangian formalism the equations of motion are obtained entirely in terms of the kinetic and potential energies (scalar operations) in the configuration space. Another important aspect of the force vs. energy analogy is that in situations where it is not possible to make explicit all the forces acting on the body, it is still possible to obtain expressions for the kinetic and potential energies.

In fact, the concept of force does not enter into Lagrangian mechanics. This is an important property of the method. Since energy is a scalar quantity, the Lagrangian function for a system is invariant for coordinate transformations. Therefore, it is possible to move from a certain configuration space (in which the equations of motion can be somewhat complicated) to a space that can be chosen to allow maximum simplification of the problem.

## Problems

1. Derive the Lagrange's equation for a mass-spring system where the spring is attached to the ceiling and the mass in hanging in the vertical.
2. Derive the Lagrange's equation for an inverted pendulum in the vertical.
3. Derive the Lagrange's equation for a simple pendulum (massless rod of length $d$ and a mass $m$ at the extremity) attached to on a movable support with mass $M$ moving at the horizontal direction.
4. Derive the Lagrange's equation for a spring pendulum, a simple pendulum where a mass $m$ is attached to a massless spring with spring constant $k$ and length at rest $d_0$.

## References

- [Goldstein H (1980) Classical Mechanics, 3rd ed., Addison-Wesley](https://books.google.com.br/books?id=tJCuQgAACAAJ).
- [Marion JB (1970) Classical Dynamics of particles and systems, 2nd ed., Academic Press](https://books.google.com.br/books?id=Ss43BQAAQBAJ).
- [Synge JL (1949) Principles of Mechanics, 2nd ed., McGraw-hill](https://books.google.com.br/books?id=qsYfENCRG5QC).
- [Taylor J (2005) Classical Mechanics. University Science Books](https://archive.org/details/JohnTaylorClassicalMechanics).


# Week 02: Lexical Analysis & Finite Automata

## Lexical Analysis

**Token Class (or Class)**

Classify **program substrings** according to **role** (token class), **class** corresponding to **sets of strings**:

- **Identifier** : string of letters or digits, starting with a letter
- **Integer**: a non-empty string of digits
- **Keyword**: else, if, begin, ...
- **Whitespace**: a non-empty sequence of blanks, newlines, tabs

**Goal of Lexical Analysis**

- Definition
    - **lexeme**: A lexeme is a sequence of characters that matches the pattern for a token.
    - **token**: A token is a pair consisting of the token name and the value.
- Concept
    - Parttition the input string into lexemes.
    - Identity the token of each lexeme.
    - Communicate tokens to the parser.
- Input: Program Substrings
- Output: Tokens
- Sample Input: `string (foo=42)`
- Sample Output: `<class, "string">, <'('>, <ID, "foo">, <Operator, "=">, <"Int", "42>, <')'>`
- Remark
    - **Left-to-Right** scan => **lookahead** required.

### Regular Languages

**Regular Expression**

- Concept: Regular expressions (syntax) specify regular languages (set of strings).
- Two base cases
  - Single Character: $'c' = \{ "c" \}$
  - Empty String: $\epsilon = \{ "" \}$
- 3 compound expressions
  - Union: $A + B = \{a\ |\ a \in A\} \cup \{ b\ |\ b \in B \}$
  - Concatenation: $AB = \{ab\ |\ a \in A \wedge b \in B \}$
  - Iteration: $
A^* = \bigcup_{i \geq 0} A^i,
\begin{cases}
A^i = A...A \\
A^0 = \epsilon
\end{cases}
$

**Def.** The **regular expression** over $\Sigma$ are the smallest set of expressions including:
$$
\begin{align}
  R &= \epsilon \\
    &|\ 'c', c\in\Sigma\\
    &|\ R+R\\
    &|\ RR\\
    &|\ R^*
\end{align}
$$

### Formal Languages

**Def.** Let $\Sigma$ be a set of characters (an alphabet).
A **language** over $\Sigma$ is a set of strings of characters drawn from $\Sigma$.
Meaning function $L$ maps **regular expressions** to **set of strings**.

$$
\begin{align}
L(\epsilon) &= \{ "" \} \\
L('c') &= \{ "c" \} \\
L(A+B) &= L(A) \cup L(B) \\
L(AB) &= \{ab\ |\ a \in L(A) \wedge b \in L(B)\} \\
L(A^*) &= \bigcup_{i \geq 0} L(A)^i \\
\end{align}
$$

**Q**:Why use a meaning function?

- Make clear what is syntax, what is semantics.
- Allow us to consider notation as a separate issue.
- Because expression and meanings are not 1-1.
  - Meaning is many to one.

## Lexical Specifications

**Lexemes**

- **Keyword**: "if" / "else" / "then" / ...
  - $ 'if' + 'else' + 'then' + ...$
- **Integer**: a non-empty string of digits
  - $digits: '0' + '1' + '2' + ...$
  - $(digit)(digit)^* = digit^+$
- **Identifier**: strings of letters or digits, starting with a letter
  - $letter = 'a' + 'b' + 'c' + ... = [a-zA-Z]$
  - $(letter)(digit+letter)^*$
- **Whitespace**: a non-empty sequence of blanks, newlines, and tabs
  - $('\;' + '\setminus n' + '\setminus t')^+$

**More regular expressions**

- At least one: $A^+ = AA^*$
- Union: $A|B = A+B$
- Option: $A? = A+\epsilon$
- Range: $'a'+'b'+\dots+'z' = [a-z]$
- Excluded Range: $\widehat{[a-z]} = [\wedge a-z]$

**How do we check $program \in L(R)$**?

1. Write a regular expression for the lexemes of each token class
    - Number $= digit^+$
    - Keyword $= 'if'+'else'+\dots$
    - Identifiew $= letter(letter+digit)^*$
    - OpenPar $='('$
    - ...
2. Construct $R$ to match all lexemes.
    - $R = Number + Keyword + Number + ... = R_1 + R_2 + ...$
3. Let input be $x_1...x_n$. Find the longest length $i$ such that $x_1...x_i \in L(R)$.
4. Remove $x_1...x_i$. Go to step 3.
5. If $x_1...x_i \in L(R_a) \cap L(R_b)$, apply $R_{\min(a,b)}$.

```python

```


## Single Index Quantile Regression

Author: @Suoer Xu （Supervised by Prof. J. Zhang)

August 18th, 2019

This is a tutorial on how to use the Single Index Quantile Regression model package. The packages almost identically replicates the profile optimization discussed in Ma and He (2016). See the paper as
> https://pdfs.semanticscholar.org/9324/e31866435d446f147320f80acde682e8e614.pdf

Environment: Python 3

Package requirements: Numpy 1.16.3, Pandas 0.22.0, Scipy 1.3.1, Matplotlib 3.1.1.

### 1. Generate B-spline

According to Ma and He (2016, p4) and de Boor (2001), the nonparametric single index function $G_{\tau}(\cdot)$ can be approximated well
by a spline function such that $G_{\tau}(\cdot) \approx B(\cdot)^T\theta_{\tau} $, w.h. $B(\cdot)$ is the basis splines with respect to a given degree, smoothness, and domain partition.

**Part I** provides a python code to generate the B-splines for any given interval and knots.The construction of B-Splines follows the Cox-de Boor recursion formula. See the description as
> https://en.wikipedia.org/wiki/B-spline

\begin{align}
B_{i,1}(x) &= \left\{
\begin{array}{rl}
1 & \text{if } t_i \le x < t_{i+1} \\
0 & \text{if otherwise}.
\end{array} \right. \\
B_{i,k+1}(x) &= \dfrac{x-t_i}{t_{i+k}-t_i}B_{i,k}(x)+\dfrac{t_{i+k+1}-x}{t_{i+k+1}-t_{i+1}}B_{i+1,k}(x)
\end{align}

The derivatives which might be used latter in the sensitivity analysis can be easily put in

\begin{align}
B'_{i,k+1}(x) = \dfrac{1}{t_{i+k}-t_i}B_{i,k}(x)+\dfrac{x-t_i}{t_{i+k}-t_i}B'_{i,k}(x)+\dfrac{-1}{t_{i+k+1}-t_{i+1}}B_{i+1,k}(x)+\dfrac{t_{i+k+1}-x}{t_{i+k+1}-t_{i+1}}B'_{i+1,k}(x)
\end{align}

```python
import numpy as np; import pandas as pd
from pandas import DataFrame
import matplotlib.pyplot as plt
from scipy.optimize import minimize, Bounds, LinearConstraint, NonlinearConstraint, BFGS
```

```python
def indicator(x,a,b,right_close=False):
    if not right_close:
        if (x >= a) & (x < b):
            return 1
        else:
            return 0
    else:
        if (x >= a) & (x <= b):
            return 1
        else:
            return 0
def I(a,b,right_close=False):
    '''
    return indicator function for a <= x < (or <=) b
    '''
    def inter(x):
        return indicator(x,a,b,right_close)
    return inter

def add(I1,I2):
    '''
    define the addition
    '''
    def inter(x):
        return I1(x) + I2(x)
    return inter

def mult(I1,I2):
    '''
    define the multiplication
    '''
    def inter(x):
        return I1(x)*I2(x)
    return inter

def scalar(I,alpha):
    '''
    define the scalar multiplication
    '''
    def inter(x):
        return alpha*I(x)
    return inter

def f(x, t_1, t_2, x_large = True):
    if t_1 != t_2:
        if x_large:
            return (x - t_1)/(t_2 - t_1)
        else:
            return (t_2 - x)/(t_2 - t_1)
    else:
        return 0

def recur(t_1, t_2, x_large = True):
    '''
    return the recursion polynomial in the Cox-de Boor's algorithm
    '''
    def inter(x):
        return f(x, t_1, t_2, x_large)
    return inter
```

```python
def partition(a,b,N):
    '''
    interval [a,b] is evenly partitioned into a = t_0 < t_1 < ... < t_N < b = t_N+1
    return the knots [t_0, t_1, ..., t_N+1]
    '''
    h = (b - a)/(N + 1)
    return [a + i*h for i in range(0,N+2)]

def extend(t, p):
    '''
    extend the original t of length N+1 into a dictionary of length N+1+2p, convinient for de Boor algorithm
    p is the final degree of the polynomials, i.e., p = m - 1 where m is the order of B-splines
    '''
    dic = {}
    N = len(t) - 1
    for i in range(p):
        dic[i-p] = t[0]
    for i in range(N+1):
        dic[i] = t[i]
    for i in range(p):
        dic[N+i+1] = t[N]
    return dic
```

```python
def deBoor(a, b, m, N, deri = False):
    '''
    a, b : the infimum and supremum , or minimum and maximum, of the scalar product <X, \beta>
    m    : the order of B-spline (>= 2)
    N    : the number of partition, i.e., [t0(=a), t1], [t1, t2], ... , [tN, tN+1(=b)]
    deri : when True, return the derivatives. Default is False
    '''
    # the choice of N follow the implementation in Ma and He (2016, p9)
    p = m - 1
    t = partition(a,b,N)
    t = extend(t,p)

    if not deri:
        B_k_1 = {}
        for i in range(-p, N + p + 1) :
            B_k_1[i] = I(t[i],t[i+1])

        for k in range(1, p + 1):
            B_k_0 = B_k_1
            B_k_1 = {}
            for i in range(-p, N + p + 1 - k):
                recursion0 = mult( B_k_0[i] , recur(t[i], t[i+k], True) )
                recursion1 = mult( B_k_0[i+1] , recur(t[i+1], t[i+k+1], False) )
                B_k_1[i] = add(recursion0, recursion1)
        return B_k_1
    else:
        B_k_1 = {}
        b_k_1 = {}
        for i in range(-p, N + p + 1) :
            B_k_1[i] = I(t[i],t[i+1])
            b_k_1[i] = I(0.,0.)

        for k in range(1, p + 1):
            B_k_0 = B_k_1
            b_k_0 = b_k_1
            B_k_1 = {}
            b_k_1 = {}
            for i in range(-p, N + p + 1 - k):
                recursion0 = mult( B_k_0[i] , recur(t[i], t[i+k], True) )
                recursion1 = mult( B_k_0[i+1] , recur(t[i+1], t[i+k+1], False) )
                B_k_1[i] = add(recursion0, recursion1)

                deri1 = mult( b_k_0[i] , recur(t[i], t[i+k], True) )
                deri2 = mult( b_k_0[i+1] , recur(t[i+1], t[i+k+1], False) )
                deri3 = scalar( B_k_0[i] , recur(t[i], t[i+k], True)(t[i]+1) )
                deri4 = scalar( B_k_0[i+1] , recur(t[i+1], t[i+k+1], False)(t[i+k+1]+1) )
                b_k_1[i] = add( add(deri1,deri2) , add(deri3,deri4) )
        return B_k_1, b_k_1
```

```python
# an example is provided
a, b, m, N = 0, 12, 4, 3
```

```python
B_spline, b_deri = deBoor(a, b, m, N, True)
B_spline
```

    {-3: <function __main__.add.<locals>.inter(x)>,
     -2: <function __main__.add.<locals>.inter(x)>,
     -1: <function __main__.add.<locals>.inter(x)>,
     0: <function __main__.add.<locals>.inter(x)>,
     1: <function __main__.add.<locals>.inter(x)>,
     2: <function __main__.add.<locals>.inter(x)>,
     3: <function __main__.add.<locals>.inter(x)>}

```python
plt.figure(figsize=(20,16))
for i in list(B_spline.keys()):
    plt.subplot(3,3,i - list(B_spline.keys())[0] + 1)
    X = np.arange(0,12,0.05)
    Y = [B_spline[i](j) for j in X]
    l = 'B(' + str(i) + ',' + str(m) + ')'
    plt.plot(X,Y,label=l)
    plt.legend()
plt.show()
```

```python
plt.figure(figsize=(20,16))
for i in list(b_deri.keys()):
    plt.subplot(3,3,i - list(b_deri.keys())[0] + 1)
    X = np.arange(0,12,0.05)
    Y = [b_deri[i](j) for j in X]
    l = 'b(' + str(i) + ',' + str(m) + ')'
    plt.plot(X,Y,label=l)
    plt.legend()
plt.show()
```

```python
# sanity check: the sum of B-splines should be 1 over the domain

ss = lambda x : 0
for i in list(B_spline.keys()):
    ss = add(ss,B_spline[i])
x = np.arange(0,12,0.05)
y = [ss(j) for j in x]
plt.figure(figsize=(4,4))
plt.plot(x,y)
plt.show()
```

```python

```

### 2. Determine the infimum and supremum of $x^T\beta$

Y:
+ the log return of 000001.SZ

X:
+ the log return of other main commercial banks (listed before 2011)
+ 000001.SZ specific characteristics
+ macro state variables

```python
data1 = pd.read_csv('results/log_return.csv').dropna()
data2 = pd.read_csv('results/000001_specific.csv').dropna()
data3 = pd.read_csv('results/macro_state.csv').dropna()
```

```python
X = pd.concat([data1[data1.columns[2:]], data2[data2.columns[1:]], data3[data3.columns[1:]]], axis = 1)
Y = data1[data1.columns[1]]
X = np.array(X)
Y = np.array(Y)
```

```python
# sanity check
print(X.shape)
print(Y.shape)
```

```python
def u(x):
    def inter(beta):
        return np.dot(x,beta)/np.sqrt((beta**2).sum())
    return inter
def v(x):
    def inter(beta):
        return -1. * np.dot(x,beta)/np.sqrt((beta**2).sum())
    return inter

def min_max(x, min_ = True):
    d = len(x)
    beta0 = np.ones(d)

    # define the linear constraint beta_0 > 0
    ub = np.ones(d)*np.inf
    lb = - np.ones(d)*np.inf
    lb[0] = 0.
    bou = Bounds(lb, ub)

    if min_:
        res = minimize(u(x), beta0, method='L-BFGS-B',bounds = bou)
    else:
        res = minimize(v(x), beta0, method='L-BFGS-B',bounds = bou)
    return u(x)(res.x)

def inf_sup(X):
    n = X.shape[0]
    d = X.shape[1]
    inf, sup = [], []
    for i in range(n):
        inf = inf + [min_max(X[i], min_ = True)]
        sup = sup + [min_max(X[i], min_ = False)]
    return np.array(inf).min(),np.array(sup).max()
```

```python
a, b = inf_sup(X)
```

```python
print(a,b)
```

### 3. Define the loss function

```python
n = X.shape[0]
m = 4
N = round(n**(1/(2*m+1))) + 1
dB = deBoor(a, b, m, N)
B = [i for i in dB.values()]
tau = 0.95
```

```python
def linear(B,theta):
    '''
    B     : list of basis splines, dimension J = N + m
    theta : control points of basis splines, (J,) array
    '''
    J = len(theta)
    lin = scalar(B[0],theta[0])
    for i in range(1,J):
        lin = add(lin, scalar(B[i],theta[i]))
    return lin

def rho(s,tau):
    '''
    define the pinball loss
    '''
    if s >= 0:
        return tau*s
    if s < 0:
        return (tau - 1)*s

def SIQ_loss(X,Y,beta,B,theta,tau):
    '''
    X     : sample input, (n, d) array
    Y     : sample output, (n,) array
    beta  : index, (d,) array
    B     : list of basis splines, dimension J = N + m
    theta : control points of basis splines, (J,) array
    tau   : quantile to be estimated
    '''
    n = X.shape[0]

    L = 0.
    for i in range(n):
        lin = linear(B, theta)
        s = Y[i] - lin( u(X[i])(beta) )
        L += rho(s,tau)
    return L/n
```

### 4. Optimization for nonparametric function $G(\cdot)$ given index $\beta$

```python
def loss_on_theta(X,Y,beta,B,tau):
    def inter(theta):
        return SIQ_loss(X,Y,beta,B,theta,tau)
    return inter

def theta_on_others(X,Y,beta,B,tau,theta0):
    J = len(B)
    res = minimize(loss_on_theta(X,Y,beta,B,tau), theta0, method='BFGS')
    return res.x
```

### 5. Optimization for Index $\beta$

```python
def loss_on_beta_(X,Y,B,theta,tau):
    def inter(beta):
        return SIQ_loss(X,Y,beta,B,theta,tau)
    return inter

def beta_on_others_(X,Y,B,theta,tau,beta0):
    d = X.shape[1]

    # define the linear constraint beta_0 > 0
    ub = np.ones(d)*np.inf
    lb = - np.ones(d)*np.inf
    lb[0] = 0.
    bou = Bounds(lb, ub)

    res = minimize(loss_on_beta_(X,Y,B,theta,tau), beta0, method='L-BFGS-B',bounds = bou)

    return res.x
```

### 4*. Optimization for both $\beta$ and $G(\cdot)$

```python

```

### 5. Optimization for nonparametric function $G(\cdot)$

\begin{align}
BIC(N_n) = \log\{\text{Loss}\} + \dfrac{\log n}{2n}(N_n+m)
\end{align}

```python
record = {}
for N_n in range(2,2*N):
    dB_n = deBoor(a, b, m, N_n)
    B_n = [i for i in dB_n.values()]
    theta_n = theta_on_others(X,Y,beta,B_n,tau)
    BIC_n = np.log(SIQ_loss(X,Y,beta,B_n,theta_n,tau)) + np.log(n)/(2*n)*(N_n+m)
    record[N_n] = [theta, BIC_n]
```

```python
record_df = DataFrame(record, index=['theta','BIC']).T
record_df
```


```python
import sympy as sy
import numpy as np
sy.init_printing()
```

```python
K2 = sy.symbols('K2')
Ac = sy.Matrix([[-3/2, -3/4, -1/(8*K2)], [1, 0, 0], [0, K2, 0]])
```

```python
Ac.inv()
```

```python
s =sy.symbols('s', real=False)
```

```python
sy.solve(s**2*(s+2) + 1)
```

```python
len(Out[14])
```

```python
sy.__version__
```

    '1.0'

```python

```

notebooks/Apollo lunar module problem.ipynb