## Fishers F-Test

#### F-distribution

>The F-distribution is formed by the ratio of two independent chi-square variables divided by their respective degrees of freedom.
- Select a random sample of sample variance $s_{1}^2$ of size $n_{1}$ from a normal population, having a standard deviation equal to $\sigma_{1}$, $\chi_{1}^2$ is the chi-square statistic for the sample drawn from population 1.
- Select an independent random sample of sample variance $s_{2}^2$ of size $n_{2}$ from a normal population, having a standard deviation equal to $\sigma_{2}$,  $\chi_{2}^2$ is the chi-square statistic for the sample drawn from population 2 .
- If both sample comes from same population then $\mu_{1}= \mu_{2}$
- Mean of F distribution = $\mu = \frac{df(num)}{df(denom)}\, -1$

>\begin{align}F = \frac{s_{1}^2/\sigma_{1}^2}{s_{2}^2/\sigma_{2}^2} = \frac{\chi_{1}^2/df_{1}}{\chi_{2}^2/df_{2}}\end{align}

>\begin{align}\frac{df_{1}s_{1}^2}{\sigma^2} \sim \chi_{1}^2, \frac{df_{2}s_{2}^2}{\sigma^2} \sim \chi_{2}^2\end{align}

>\begin{align}df_{1} = (n_{1}-1), df_{2} = (n_{2}-1)\end{align}

**Characterisitics**
- The F-values are all non-negative
- The distribution is non-symmetric
- The mean is approximately 1
- There are two independent degrees of freedom, one for the numerator, and one for the denominator.
- Distribution is positively skewed towards right. When numerator dof and denominator dof becomes larger, the distribution becomes bell shaped and symmetric.

#### F-Test for Variances

- Null hypothesis: No difference in population variances
- Alternative hypothesis: There is a difference

The F-test is designed to test if two population variances are equal. It does this by comparing the ratio of two variances. So, if the variances are equal, the ratio of the variances will be 1.

The F-test is typically used to answer one of the following questions:
1. Do two samples come from populations with equal variances?
2. Does a new treatment or process reduce the variability of some current treatment or process?
3. Are the the means between two populations significantly different or not?

#### F-statistic

 - Is a ratio of two variances (Variance is nothing but measure of dispersion, it tells how far the data is dispersed from the mean). F statistic accounts corresponding degrees of freedom to estimate the population variance. The larger variance should always be placed in the numerator.
 - F-statistic is almost similar to t statistic. t-test states a single variable is statistically significant or not whereas F test states a group of variables are statistically significant or not.

> \begin{align} F = \frac{\text{Expained Variability}}{\text{Unexplained Variability}} = \frac{\text{Between Group Variability}}{\text{Within Group Variability}} = \frac{MS_{between}} {MS_{within}} = \frac{MS_{regression}} {MS_{error}} = \frac{\sigma_{1}^2} {\sigma_{2}^2} = \frac{s_{1}^2} {s_{2}^2}\end{align}

### F-test for linear regression model
  - To tests **any** of the independent variables (predictor) in a multiple linear regression are significant or not.
  - To compare two competing regression models (model1 with p1 predictor, model2 with p2 predictor) in their ability to **explain** the variance in the dependent/target variable
  - It also indicates a linear relationship between dependent variable and at least one of the independent variable.

>\begin{align}F = \frac{(SSE_{model1}-SSE_{model2})/(p_{2}-p_{1})}{SSE_{model2}/(n-p_{2})} \end{align}

### Assumptions

1. Normality – Samples in each group should follow normal distribution, only needed for small sample sizes, say n < 20 per group.
2. Equal Variances – The variances of the populations that the samples come from should be equal. Homogeneity is only needed if sample sizes are very unequal (Levene's Test).
3. Independence – The observations in each group are independent of each other and the observations within groups were obtained by a random sample.
4. Same Sample Size: The treatment groups should have the same sample size.

### Types of F-test

1. **Pouplation variance is known:**
F test is to find out whether the two independent estimates of population variance differ significantly.

- Null Hypothesis(H0): $\sigma_{1}^2 = \sigma_{2}^2$ (the population variances are equal)
- Alternate Hypothesis(H1): $\sigma_{1}^2 \neq \sigma_{2}^2$ (the population variances are not equal)

 In this case F ratio is
\begin{align} F = \frac{\sigma_{1}^2}{\sigma_{2}^2}, σ_{1}^2 > σ_{2}^2\end{align}
where
\begin{align} \sigma_{1}^2 = \frac{\sum (X_{1}-\mu_{1})^2}{n_{1}}, \sigma_{2}^2 = \frac{\sum (X_{2}-\mu_{2})^2}{n_{2}}\end{align}

larger estimate of variance always be in numerator and smaller estimate of variance in denominator
 - DF of larger variance (i.e numerator) $a=n_{1}-1$
 - DF of smaller variance (i.e denominator) $b=n_{2}-1$

2. **Pouplation variance is unknown:**
To find out whether the two samples drawn from the normal population having the same variance.

- Null Hypothesis(H0): $s_{1}^2 = s_{2}^2$ (the unbaised sample variances are equal)
- Alternate Hypothesis(H1): $s_{1}^2 \neq s_{2}^2$ (the unbaised sample variances are not equal)

In this case F ratio is

\begin{align}F = \frac{s_{1}^2}{s_{2}^2}, s_{1}^2 > s_{2}^2\end{align} where
\begin{align}s_{1}^2 = \frac{\sum (X_{1}-\bar{x_{1}})^2}{n_{1}-1}, s_{2}^2 = \frac{\sum (X_{2}-\bar{x_{2}})^2}{n_{2}-1}\end{align}

larger estimate of variance always be in numerator and smaller estimate of variance in denominator
 - DF of larger variance (i.e numerator) $a=n_{1}-1$
 - DF of smaller variance (i.e denominator) $b=n_{2}-1$

 F-Tests for Equality of Two population Variances

If we take a random sample from two different normally distributed population, the samples are always independent. The sampling F-statistic approximately follows an F-distribution with degrees of freedom df1=$n_{1}−1$ and df2=$n_{2}−1$.

\begin{align} F = \frac{\sigma_{1}^2} {\sigma_{2}^2} = \frac{s_{1}^2} {s_{2}^2}\end{align}
Hypotheses:

|Type of Test|Null hypothesis|Alternate hypothesis|Rejection Region|
|---|---|---|---|
|Two tailed|$\sigma_{1}^2=\sigma_{2}^2$|$\sigma_{1}^2 \neq \sigma_{1}$|$F \le F_{1-\alpha/2}$ or $F \ge F_{\alpha/2}$|
|Right tailed|$\sigma_{1}^2=\sigma_{2}^2$|$\sigma_{1}^2 \gt \sigma_{1}^2$|$F \ge F_{\alpha}$|
|Left tailed|$\sigma_{1}^2=\sigma_{2}^2$|$\sigma_{1}^2 \lt \sigma_{1}^2$|$F \le F_{1-\alpha}$|

Source: (1) saylordotorg.github.io, (2) analyticsvidhya.com

### Steps to conduct F-test
1. Choose the test: Note down the independent variables and dependent variable and also assume the samples are normally distributed
2. Calculate the F statistic, choose the highest variance in the numerator and lowest variance in the denominator with a degrees of freedom (n-1)
3. Determine the statistical hypothesis
4. State the level of significance
5. Compute the critical F value from F table. (use $\alpha/2$ for two tailed test)
6. Calculate the test statistic
7. Finally, draw the statistical conclusion. reject the null hypothesis; If the test statistic falls in the critical region.
  - p value < SL: Null hypothesis can be rejected, this means we have sufficient evidence to say that the two population variances are not equal.
  - p value > SL: Null hypothesis can not be rejected, two population variances are equal.

## Terminology

**Factors:**
The two independent variables in a two-way ANOVA are called factors, which affect the dependent variable. Each factor will have two or more levels within it, and the degrees of freedom for each factor is one less than the number of levels.

**Treatment Groups**
Treatement Groups are formed by making all possible combinations of the levels of two or more factors. So, the total number of treatment groups is the product of the number of levels for each factor. For example, if the first factor has 3 levels and the second factor has 2 levels, then there will be 3x2=6 different treatment groups.

**Completely Randomized Design**
In a completely randomized design, objects or subjects are assigned to groups completely at random. One standard method for assigning subjects to treatment groups is to label each subject, then use a table of random numbers to select from the labelled subjects. This may also be accomplished using a computer.

**Randomized Block Design**
If an experimenter is aware of specific differences among groups of subjects or objects within an experimental group, he or she may prefer a randomized block design to a completely randomized design. In a block design, experimental subjects are first divided into homogeneous blocks before they are randomly assigned to a treatment group.

**Main Effect due to Factor:**
The main effect involves the independent variables one at a time. The interaction is ignored for this part. Just the rows or just the columns are used, not mixed. This is the part which is similar to the one-way analysis of variance. Each of the variances calculated to analyze the main effects are like the between variances

**Variation due to Interaction Effect of Factors:**
The interaction effect is the effect that one factor has on the other factor. The degrees of freedom here is the product of the two degrees of freedom for each factor.

**Between Treatment Variation:**
The Between variation is the sum of squares within each treatment group. The Between variance is the Between variation divided by its degrees of freedom.

**Within Treatment Variation:**
The Within variation is the sum of squares within each treatment group. The within variance is the within variation divided by its degrees of freedom.

**F-Tests:**
There is an F-test for each of the hypotheses, and the F-test is the mean square for each main effect and the interaction effect divided by the within variance. The numerator degrees of freedom come from each effect, and the denominator degrees of freedom is the degrees of freedom for the within variance in each case.

**Replication Number:**
It is number of trials or observations performed for each treatment combination.

**Example:**

||Control|Fertilizer-1|Fertilizer-2|Fertilizer-3|
| --- | --- | --- | --- | --- |
|Plant A|21.0|32.0|22.5|28.0|
|Plant A|19.5|30.5|26.0|27.5|
|Plant B|23.7|30.1|30.6|36.1|
|Plant B|23.8|28.9|31.1|36.6|

- Main Effect A = Fertilizer, Factors/Levels of A = Type of Fertilizers used = 3
- Main Effect B = Plant, Factors/Levels of B = Type of Plants/species =  2
- Number of Treatment Groups = Factors of A * Factors of B = 3*2 = 6
- Number of Replications = R = 2

```
# F-Test in Python
# To perform Fishers-test we can use scipy.stats.f.cdf(f-dist-rv, numerator_dof, denomintor_dof)
import numpy as np
import pandas as pd
import scipy.stats
#define F-test function
def f_test(x, y):
    x = np.array(x)
    y = np.array(y)
    #By default, numpy.var calculates the population variance. To calculate the sample variance, we need to specify ddof=1
    f = np.var(x, ddof=1)/np.var(y, ddof=1) #calculate F test statistic , ratio of sample variance of two populations x, y
    dfn = x.size-1 #define degrees of freedom numerator
    dfd = y.size-1 #define degrees of freedom denominator
    p = 1-scipy.stats.f.cdf(f, dfn, dfd) #find p-value of F test statistic
    return f, p

#perform F-test
x = [18, 19, 22, 25, 27, 28, 41, 45, 51, 55]
y = [14, 15, 15, 17, 18, 22, 25, 25, 27, 34]
f_test(x, y)
```

    (4.387122002085505, 0.01912653593238578)

## ANOVA Assumptions

1. Normality: Samples in each group should follow normal distribution, only needed for small sample sizes, say n < 20 per group.
2. Equal Variances: The variances of the populations that the samples come from should be equal. Homogeneity is only needed if sample sizes are very unequal (Levene's Test).
3. Randomness & Independence: The observations in each group are independent of each other and the observations within groups were obtained by a random sample.
4. Same Sample Size: The treatment groups should have the same sample size.

## Experimental Design

#### Total variablity sum of squares

Total variablity = Between group variability + Between subjects variability + Within variability (Random variability) +  Factor Interaction variablity

>\begin{align} SST = SSB(Group) +  SSB(Subject) + SSE(Random) + SSI(Group-Subject) \end{align}

>\begin{align} SSE(Random) = SSW(Group) + SSW(Subject) \end{align}

**Completely Randomized Design (CRD):**
>In this Experimental design, obeservations are taken in random order and the envirnment in which the treatments are used is as uniform as possible.\
Experimental design follows either Fixed Effect model or Random effect model. In Random effect model, levels of all participating factors are choosen by random process while in Fixed effect model, some specific levels are chosen by the experimenter.

#### Single Factor experimental design
>There is only one factor with 2 or more levels. The levels (columns) are called groups, or condition. These levels forms treatment combination.
1. **One Way ANOVA model**
  - SSE(Random) = SSW(Groups)
  - SST = SSB(Group) + SSE(Random)
2. **Random Measures ANOVA**
  - **Randomized Complete Block Design (RBCD)**
  - SST = SSB(Group) + SSB(Subject) + SSE(Random)

#### Two Factor experimental design
>**Two Way ANOVA**: There are two factors say A & B and each factor has 2 or more levels. The levels of row arranged factor A can be called subjects or blocks or rows. The levels of column arranged factor B can be called groups or condition or columns. The pair level-i(A), level-j(B) toegether forms a treatment combination for the dependent variable. The replication count (R) is the number of samples or trails for each treatment.
  - SST = SSB(Group) + SSB(Subject) + SS(Interaction) + SSE(Random)

source: https://www.nature.com/articles/nmeth.3180
a) Two common two-factor designs with 8 measurements each. In the CR scenario, each mouse is randomly assigned a single treatment. Variability among mice can be mitigated by grouping mice by similar characteristics (e.g., litter or weight). The group becomes a block. Each block is subject to all treatments. (b) Partitioning of the total sum of squares for the CR and RCB designs in a. M represents the blocking factor. Vertical axis is relative to the SST.

## One way ANOVA (Independent ANOVA)

A one-way ANOVA compares the means of three or more independent (unrelated) groups of a factor to determine if there is a statistically significant difference between the corresponding population means.

### Hypotheses
- Null hypothesis(H0):
$\mu_{1} = \mu_{2} = \mu_{3} = … = \mu_{k}$ (all the population means are equal)
- Alternative hypothesis(H1): At least one population mean is different from the rest

**Single Factor Experiment**
A single factor (variable) experiment where factor has different levels (aslo called treatment groups), One way ANOVA compares the mean of each tretment level

> Each group / treament mean can be described in terms of overall mean of all treatment groups or levels :
\begin{align} \mu_{j} = (\mu + \tau_{j}) \end{align}
Each observation of group / treament can be described as:
\begin{align} x_{ij} = (\mu + \tau_{j}) + e_{ij} \end{align}

where
 - $\mu$ = overall mean of factor population from which these samples for each treament / group is taken
 - $e_{ij}$ = random error for component, we assume random errors $e_{j}$ are normally distributed with mean 0 and variance $\sigma^2$
 - $\tau_{j}$ = treatment or experimental effect (deviation from factor population mean), how these treatment levels chosen gives two different models namely a) FIXED effect model, b) RANDOM effect model

**FIXED effect model:**
  - Some specific B levels or treatment are choosen by experimenter from all possible levels (combination)
  - Sum of treatment/experimental effect of all groups is zero:
\begin{align} \sum_{j=1}^{B}\tau_{j} = 0 \end{align}
  - Hypotheses:
   - H0: $\tau_{1} = \tau_{1} = \tau_{1}...=\tau_{B}=0$, there is no difference in the mean of treatment groups
   - H1: $\tau_{j} \neq 0$ for at least one j-th group, there is at least one group mean different from others

**RANDOM effect model:**
  - Randomly B levels or treatment are choosen from all possible levels (combination)
  - Treatment effect $\tau_{j}$ are normally distributed with mean 0 and variance $\sigma_{\tau}^2$
  - Hypotheses:
   - H0: $\sigma_{\tau}^2 = 0$, all treatment group means are identical
   - H1: $\sigma_{\tau}^2 \neq 0$, there is variability between treatment groups

### Variability & Expectation

#### Total variability - sum of squares identity
---
As there is only one factor, there is no interaction component

\begin{align} SST = SSB(Groups) + SSE(Random) \end{align}

\begin{align} SSE(Random) = SSW(Groups) = SST - SSB(Groups) \end{align}

Partitioning SST:
>\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} (x_{ij}-\bar{x}_{grand})^2 = A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 + \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}

Partitioning SST degree of fredom:
>\begin{align} AB-1 = (B-1) + B(A-1) \end{align}
---

#### Variance Expectation
1. Expected Between Groups Variance
 - Sum of squares:
>\begin{align} SSB(Groups) = (B-1)\sigma^2+A*\sum_{j=1}^{B}\tau_{j}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Groups) = \frac{SSB(Groups)}{B-1} = \sigma^2+\frac{A}{(B-1)}\,*\sum_{j=1}^{B}\tau_{j}^2 = \sigma^2+A*\sigma_{\tau}^2\end{align}
2. Expected Random Error Variance
 - Sum of squares:
>\begin{align} SSE(Random) = B(A-1)\sigma^2\end{align}
 - Mean sum of squares:
>\begin{align} MSE(Random) = \frac{SSE(Random)}{B(A-1)} = \sigma^2\end{align}

### Sum of Squares Formula

|Source of Variation| SS Form 1 | SS Form 2 | SS Form 3 | Degree of Freedom | Notes, comments, some extra details |
|---|---|---|---|---|---|
|$SST$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} (x_{ij}-\bar{x}_{grand})^2 \end{align}|\begin{align}\sum_{i=1}^{A}\sum_{j=1}^{B}x_{ij}^2 - \frac{{GT}^2}{N} \end{align}|\begin{align} (N-1)*s_{grand}^2\end{align}|$(N-1)$|\begin{align} variance_{total} = s_{grand}^2 = \,\frac{1}{N-1}\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 \end{align}|
|$SSB(Groups)$|\begin{align} A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 \end{align}|\begin{align} \frac{1}{A}\,\sum_{j=1}^{B}(\sum_{i=1}^{A}x_{ij})^2 - \frac{GT^2}{N} \end{align}||$(B-1)$||
|$SSW(Groups) = SSE$|\begin{align} \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}||\begin{align} \sum_{j=1}^{B}(n_{j}-1)*s_{j}^2\end{align}|$B(A-1) = (N-B)$|\begin{align} variance_{group(i)} = s_{j}^2 = \, \frac{1}{A-1}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}|

### One Way ANOVA table

| Source	of Variation | Sum of Squares (SS) |	degree of freedom calculation (df)| Mean Squares (MS)	| F-Statistic or Ratio|
|---|---|---|---|---|
| Between	Groups or Regression or Model | SSR or SSB	| $df_{groups} = B-1$|	$MSR = \frac{SSR}{df_{reg}}$ | $F = \frac{MSR}{MSE}$ |
| Within Groups or Error	or Residual| SSE or SSW	| $df_{error} = N-B$ | $MSE	= \frac{SSE}{df_{error}}$| |
| Total	| SST	| $df_{total} = N-1$ | | | |

### Naming Convention

---
- Group = Column = Treatment = Level
- i = row/replication/sample number
- j = group/column number
- $A$ = number of observations in each groups or columns
- $B$ = number of groups or columns
- $n_{j}$ = $A$ = the sample size or number of observartions in j-th group/column
- $N = A*B$ = total number of observations
- $x_{ij}$ = observation at i-th subject (row) and j-th group(column)
- $\bar{x}_{j}$ = the mean of j-th group/column
- $GT = \sum_{i=1}^{A}\sum_{j=1}^{B} x_{ij}$ = the grand total of the entire dataset
- $\bar{x}_{grand}$ = the grand mean or overall mean of all observations
- $s_{grand}^2$ = the unbiased sample variance for the entire dataset
- $s_{i}^2$ = the unbiased sample variance for the i-th subject/row observations
- $s_{j}^2$ = the unbiased sample variance for the j-th group/column observations
- $df_{reg} = B-1$: regression degrees of freedom
- $df_{error} = N-B$: error degrees of freedom
- $df_{total} = N-1$: total degrees of freedom
- MSR = SSR/$df_{reg}$ = regression mean square
- MSE = SSE/$df_{error}$ = error mean square
- F test statistic (F = MSR/MSE)
---

### Steps by hand

**Example:**

| | group 1 | group 2 | group 3 |
| --- | --- | --- | --- |
|Index| j=1 | j=2 | j=3 |
| --- | --- | --- | --- |
|i=1| - | - | - |
|i=2| - | - | - |

1. **Calculate the group means and the overall mean:**

- Grand mean or overall mean
\begin{align}\bar{x}_{grand} = \frac{1}{N}\sum_{i=1}^{A}\sum_{j=1}^{B} x_{ij} = \frac{1}{N}*{GT}\end{align}

- Group mean for j-th group
\begin{align}\bar{x}_{j} = \frac{1}{n_{j}}\sum_{i=1}^{n_{j}} x_{ij} = \frac{1}{A}\sum_{i=1}^{A} x_{ij}\end{align}

2. **Total sum of squares (SST):**
- The total group variation (not variance) is comprised the sum of squares deviations of all the observations from the grand mean
- Degree of freedom is (total number of observations - 1)

Method-1
>\begin{align} SST = \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 = \sum_{i=1}^{A}\sum_{j=1}^{B}x_{ij}^2 - \frac{{GT}^2}{N} \end{align}
 - Calculate the difference between each observation and the grand mean irrespective of group(column) or subject(row)
 - Square each of these differences
 - Add the values together to get SST

Method-2
>\begin{align} SST = (N-1)*s_{grand}^2\end{align}
\begin{align} variance_{total} = s_{grand}^2 = \frac{1}{N-1}\,\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 \end{align}
 - Calculate the unbiased variance for all observations together irrespective of group(column) or subject(row)
 - Multiply it with (total number of observations - 1) to get SST

Method-3
>We will calculate the total sum of squares (SST) using the following formula:
\begin{align} SST = SSB + SSW = SSR + SSE \end{align}

3. **Between Groups/Samples Sum of Squares (SSB) or Regression sum of squares (SSR):**
- Also called SS(Model) or SS(Treatment)
- Variation between Group/column means with grand means
- It accounts the variation due to the interaction between the samples/groups. If the sample means (group means) are close to each other (and therefore the grand mean) this will be small.
- The degree of freedom is equal to (number of groups or samples - 1)

Method-1
>\begin{align} SSB = \sum_{j=1}^{B} n_{j} * (\bar{x}_{j}-\bar{x}_{grand})^2 \end{align}
 - Calculate the difference between the mean of each group and the grand mean
 - Square each of these differences
 - Multiply each result by the number of participants (sample size) within that group
 - Add the values for each group together to get SSR

Method-2
>\begin{align} SSB = \frac{1}{A}\,\sum_{j=1}^{B}(\sum_{i=1}^{A}x_{ij})^2 - \frac{GT^2}{N} \end{align}
 - Calculate the sum of observations for each group (column)
 - Square each sum (group wise sum) and add them
 - Divide it by samplesize or total number of rows, call it A
 - Sum all observations to get grand sum (grand total)
 - Square the grand total and divide it by total number of observations, call it B
 - Calculate A-B to get the SSB

4. **Within Group/Sample sum of squares (SSW) or Error sum of squares (SSE):** or SS(Residual)
- It account for the variation due to differences within individual samples, denoted as Sum of Squares Within groups. Each sample is considered independently, no interaction between samples is involved.
- The degrees of freedom is equal to the sum of the individual degrees of freedom for each sample. Since each sample has degrees of freedom equal to one less than their sample sizes, and there are k samples, the total degrees of freedom is k less than the total sample size: df = N - k.

Method-1
>\begin{align} SSW = \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}
 - Calculate the difference between each observation of the group and the group mean
 - Square each of these differences
 - Add the values for each group together and call it group error SS
 - Add the group error SS of all the groups to get the SSW

Method-2
>\begin{align} SSW = \sum_{j=1}^{B}(n_{j}-1)*s_{j}^2\end{align}
\begin{align} variance_{group(j)} = s_{j}^2 = \frac{1}{A-1}\,\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}
 - Calculate the unbiased variance for j-th group observations
 - Multiply it with (total observations in the group - 1) to get SSW

5. **ANOVA table:**
Now that we have SSR, SSE, and SST, we can fill in the ANOVA table:

6. Effect Size
>\begin{align} \eta^2 = \frac{SSB}{SST} = \frac{SSR}{SST} \end{align}

7. **Interpret the results:** To determine if this is a statistically significant result, we must compare this F-statistic to the F critical value found in the F distribution table for $F(df_{reg}, df_{error})$

  - if p-value < SL,  then we reject H0 (Null Hypothesis of all means are equal)
  - if p-value >= SL,  then we fail to reject H0 (Null Hypothesis of all means are equal)

### Post-Hoc Test
In order to determine which groups are different from one another, a post-hoc test is needed

- Tukey's Honestly Significant Differences (HSD) Test
  For each pairwise comparison for sample i and j,
  **H0 (null hypothesis):** $\mu_{i} = \mu_{j}$,
  **Ha (alternate hypothesis):** $\mu_{i} \neq \mu_{j}$

  Tukey's critical value $w = q_{\alpha}(p, df_{error}) * s_{error}$

  where
  - $q_{\alpha}$ = Obtained from a Table of Tukey values for given significance level,
  - p = the number of groups or treatment levels
  - $n_{samplesize}$ = number of replications/observations
  - $s_{error} = \sqrt{MSE/A}$ = standard error of group mean where A is the number of observations in each group

- Rank the group means in descending order, and calculate pairwise differences. For all groups i and j,
- if $mean_{i} - mean_{j} <= w$, it indicates the group means do not differ significantly
- if $mean_{i} - mean_{j} > w$, it indicates the group means differ significantly

### Example One Way ANOVA

| | group 1 | group 2 | group 3 |
| --- | --- | --- | --- |
|Index| j=1 | j=2 | j=3 |
| --- | --- | --- | --- |
|i=1| 85 | 91 | 79 |
|i=2| 86 | 92 | 78 |
|i=3| 88 | 93 | 88 |
|i=4| 75 | 85 | 94 |
|i=5| 78 | 87 | 92 |
|i=6| 94 | 84 | 85 |
|i=7| 98 | 82 | 83 |
|i=8| 79 | 88 | 85 |
|i=9| 71 | 95 | 82 |
|i=10| 80 | 96 | 81 |
|Group Mean| 83.4| 89.3 |84.7 |

1. Overall mean = 85.8, Group-1 Mean=83.4, Group-2 Mean=89.3, Group-3 Mean=84.7
2. SST = 192.2 + 1100.6 = 1292.8
3. SSR = 10(83.4-85.8)2 + 10(89.3-85.8)2 + 10(84.7-85.8)2 = 192.2
4. We calculate SSE as follows:
 - Group 1: (85-83.4)2 + (86-83.4)2 + (88-83.4)2 + (75-83.4)2 + (78-83.4)2 + (94-83.4)2 + (98-83.4)2 +  (79-83.4)2 + (71-83.4)2 + (80-83.4)2 = 640.4
 - Group 2: (91-89.3)2 + (92-89.3)2 + (93-89.3)2 + (85-89.3)2 + (87-89.3)2 + (84-89.3)2 + (82-89.3)2 +  (88-89.3)2 + (95-89.3)2 + (96-89.3)2 = 208.1
 - Group 3: (79-84.7)2 + (78-84.7)2 + (88-84.7)2 + (94-84.7)2 + (92-84.7)2 + (85-84.7)2 + (83-84.7)2 +  (85-84.7)2 + (82-84.7)2 + (81-84.7)2 = 252.1
 - SSE = 640.4 + 208.1 + 252.1 = 1100.6

 Here is how we calculated the various numbers in the table:

5. ANOVA Table
- df treatment(groups) = 3-1 = 2
- df error = 30-3 = 27
- df total = 30-1 = 29
- MS treatment group = SST / df treatment = 192.2 / 2 = 96.1
- MS error = SSE / df error = 1100.6 / 27 = 40.8
- F = MS treatment group / MS error = 96.1 / 40.8 = 2.358

6. Intrepret result
- The F test statistic for this one-way ANOVA is 2.358. To determine if this is a statistically significant result, we must compare this to the F critical value found in the F distribution table with the following values:
- $\alpha$ (significance level) = 0.05
- DF1 (numerator degrees of freedom) = df treatment = 2
- DF2 (denominator degrees of freedom) = df error = 27
We find that the F critical value is 3.3541.

Since the F test statistic in the ANOVA table is less than the F critical value in the F distribution table, we fail to reject the null hypothesis. This means we don’t have sufficient evidence to say that there is a statistically significant difference between the mean exam scores of the three groups.

```
# One way ANOVA by Statmodels Library
import numpy as np
import pandas as pd
import statsmodels.api as sm
from statsmodels.formula.api import ols
A = [12.6, 12, 11.8, 11.9, 13, 12.5, 14]
B = [10, 10.2, 10, 12, 14, 13]
C = [10.1, 13, 13.4, 12.9, 8.9, 10.7, 13.6, 12]
all_scores = A + B + C
company_names = (['A'] * len(A)) +  (['B'] * len(B)) +  (['C'] * len(C))
data = pd.DataFrame({'company': company_names, 'score': all_scores})
data.groupby('company').mean()
lm = ols('score ~ company', data = data).fit()
table = sm.stats.anova_lm(lm)
print(table)
```

                df     sum_sq   mean_sq         F    PR(>F)
    company    2.0   3.606905  1.803452  0.821297  0.455683
    Residual  18.0  39.525476  2.195860       NaN       NaN

```
# One way ANOVA by hand

# compute group means
group_means = data.groupby('company').mean()
#group_means = group_means.rename(columns = {'score': 'group_mean'})
group_means.columns = ['group_mean']
print("group_means",group_means)

# compute overall mean
overall_mean = data['score'].mean()
print("overall_mean=",overall_mean)

# compute Sum of Squares Total
data['overall_mean'] = overall_mean
ss_total = sum((data['score'] - data['overall_mean'])**2)
print("ss_total=",ss_total)

# add group means and overall mean to the original data frame
data = data.merge(group_means, left_on = 'company', right_index = True)

# compute Sum of Squares Error/Residual/Within
ss_residual = sum((data['score'] - data['group_mean'])**2)
print("ss_residual=",ss_residual)

# compute Sum of Squares Regression/Between/Explained/Model
ss_explained = sum((data['overall_mean'] - data['group_mean'])**2)
print("ss_explained=",ss_explained)

# compute Mean Square Error/Residual/Within
n_groups = len(set(data['company']))
n_obs = data.shape[0]
df_residual = n_obs - n_groups
ms_residual = ss_residual / df_residual
print("ms_residual=",ms_residual)

# compute Mean Square Regression/Between/Explained/Model
df_explained = n_groups - 1
ms_explained = ss_explained / df_explained
print("ms_explained=",ms_explained)

# compute F-Value
f = ms_explained / ms_residual
print("F-score=",f)

# compute p-value
import scipy.stats
p_value = 1 - scipy.stats.f.cdf(f, df_explained, df_residual)
print("p-value=",p_value)
```

    group_means          group_mean
    company
    A         12.542857
    B         11.533333
    C         11.825000
    overall_mean= 11.980952380952381
    ss_total= 43.132380952380956
    ss_residual= 39.52547619047619
    ss_explained= 3.6069047619047776
    ms_residual= 2.1958597883597886
    ms_explained= 1.8034523809523888
    F-score= 0.8212966923081592
    p-value= 0.4556832940515221

## Repeated Measures ANOVA

A repeated-measures (or within-participants) test is what you use when you want to compare the performance of the same group of participants in different experimental conditions.
That is, when the same participants take part in all of the conditions in your study.

The term one-way simply to refers to the number of independent variables (one independent variable) you have;
- one dependent variable
- one independent variable (with 3 or more conditions)
- each participant(subject or row) takes part in all conditions/groups/columns

### Hypotheses

A repeated measures ANOVA is used to determine whether or not there is a statistically significant difference between the means of three or more groups in which the same subjects show up in each group.

1. Measuring the mean scores of subjects(group) during three or more time points.
2. Measuring the mean scores of subjects(group) under three different conditions.

### Variability & Expectation

#### Total variability - sum of squares identity
---
As there is only one factor, there is no interaction component

\begin{align} SST = SSB(Group) + SSB(Subject) + SSE(Random) \end{align}

Partitioning SST:
>\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} (x_{ij}-\bar{x}_{grand})^2 \,= A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 +\, B*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2 \,+ \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2 \end{align}

Partitioning SST degree of fredom:
>\begin{align} AB-1 = (B-1) + (A-1) + (A-1)(B-1) \end{align}
---

#### Variance Expectation
1. Expected Between Groups Variance
 - Sum of squares:
>\begin{align} SSB(Groups) = (B-1)*\sigma^2+A*\sum_{j=1}^{B}\tau_{j}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Groups) = \frac{SSB(Groups)}{B-1} = \sigma^2+\frac{A}{(B-1)}\,*\sum_{j=1}^{B}\tau_{j}^2 = \sigma^2+A*\sigma_{\tau}^2\end{align}
2. Expected Between Subjects/Blocks Variance
 - Sum of squares:
>\begin{align} SSB(Subject) = (A-1)*\sigma^2+B*\sum_{i=1}^{A}\beta_{i}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Subject) = \frac{SSB(Subject)}{A-1} = \sigma^2+\frac{B}{(A-1)}\,*\sum_{i=1}^{A}\beta_{i}^2 = \sigma^2+B*\sigma_{\beta}^2\end{align}
3. Expected Random Error Variance
 - Sum of squares:
>\begin{align} SSE(Random) = (A-1)(B-1)*\sigma^2\end{align}
 - Mean sum of squares:
>\begin{align} MSE(Random) = \frac{SSE(Random)}{(A-1)(B-1)} = \sigma^2\end{align}

### Sum of Squares Formula

|Source of Variation| SS Form 1 | SS Form 2 | SS Form 3 | Degree of Freedom | Notes, comments, some extra details |
|---|---|---|---|---|---|
|$SST$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} (x_{ij}-\bar{x}_{grand})^2 \end{align}|\begin{align}\sum_{i=1}^{A}\sum_{j=1}^{B}x_{ij}^2 - \frac{{GT}^2}{N} \end{align}|\begin{align} (N-1)*s_{grand}^2\end{align}|$(N-1)$|\begin{align} variance_{total} = s_{grand}^2 = \,\frac{1}{N-1}\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 \end{align}|
|$SSB(Groups)$|\begin{align} A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 \end{align}|\begin{align} \frac{1}{A}\,\sum_{j=1}^{B}(\sum_{i=1}^{A}x_{ij})^2 - \frac{GT^2}{N} \end{align}||$(B-1)$||
|$SSB(Subjects)$|\begin{align} B*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2 \end{align}|\begin{align} \frac{1}{B}\,\sum_{i=1}^{A}(\sum_{j=1}^{B}x_{ij})^2 - \frac{GT^2}{N} \end{align}||$(A-1)$||
|$SSW(Groups)$|\begin{align} \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}||\begin{align} \sum_{j=1}^{B}(n_{j}-1)*s_{j}^2\end{align}|$(N-B)$|\begin{align} variance_{group(i)} = s_{j}^2 = \, \frac{1}{A-1}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}|
|$SSW(Subjects)$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i})^2 \end{align}||\begin{align} \sum_{i=1}^{A}(n_{i}-1)*s_{i}^2 \end{align}|$(N-A)$|\begin{align} variance_{subject(i)} = s_{i}^2 = \, \frac{1}{B-1}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i})^2 \end{align}|
|$SSE = SSW$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2 \end{align}|$SST-SSB(Groups)-SSB(Subjects)$||$(A-1)*(B-1)$||

### Repeated measures ANOVA table

| Source of Variation | Sum of Square (SS) | Degrees of Freedom Calculation (df) | Mean Square Calculation (MS) | F ratio |
| --- | --- | --- | --- | --- |
| Groups (Between) or Participents or Columns | SSB(Groups) | $df_{groups} = (B-1)$ | $MSB(Groups) = SSB(Groups)/df_{columns}$ | F = MSB(Groups)/MSE |
| Subject (Between) or Rows | SSB(Subjects) | $df_{subjects} = (A-1)$ | $MSB(Subjects) = SSB(Subjects)/df_{subjects}$ | F = MSB(Subjects)/MSE |
| Error (Within) or Residual | SSE | $df_{error} = (A-1)(B-1))$| $MSE = SSE/df_{error}$ | |
| Total | SST | $df_{total} = (N-1)$ |||

### Naming Convention

---
- Group = Column = Treatment = Level
- Subject = Row = Block
- i = row number/subject/Block
- j = column number/group/Level
- $A$ = total number of subjects/rows
- $B$ = total number of groups/columns
- $n_{i}$ = $B$ = total number of observations in the i-th subject/column,
- $n_{j}$ = $A$ = total number of observations in the j-th group/column,
- $N = A*B$ = the total number of observations in the entire dataset
- $x_{ij}$ = observation at i-th subject (row) and j-th group(column)
- $\bar{x}_{i} = \frac{1}{n_{i}}\sum_{j=1}^{n_{i}} x_{ij} = \frac{1}{B}\sum_{i=1}^{B} x_{ij}$ = the mean of the i-th subject/row,
- $\bar{x}_{j} = \frac{1}{n_{j}}\sum_{i=1}^{n_{j}} x_{ij} = \frac{1}{A}\sum_{i=1}^{A} x_{ij}$ = the mean of the j-th group/column,
- $GT = \sum_{i=1}^{A}\sum_{j=1}^{B} x_{ij}$ = the grand total of the entire dataset
- $\bar{x}_{grand} = \frac{1}{N}\sum_{i=1}^{A}\sum_{j=1}^{B} x_{ij}$ = the grand mean or mean of the entire dataset
- $s_{grand}^2$ = the unbiased sample variance for the entire dataset
- $s_{i}^2$ = the unbiased sample variance for the i-th subject/row observations
- $s_{j}^2$ = the unbiased sample variance for the j-th group/column observations
---

### Steps by hand

**Example:**

| Treatment ||| group-1 | group-2 | group-3 |
| --- | --- | --- | --- | --- | --- |
||Index| i=1 | i=2 | i=3 | i=4 |
|| --- | --- | --- | --- | --- |
|subject-1|j=1| - | - | - | - |
|subject-2|j=2| - | - | - | - |
|subject-3|j=3| - | - | - | - |
|subject-4|j=4| - | - | - | - |

In a typical one-way ANOVA, different subjects are used in each group.
But in repeated measures ANOVA, the same subjects show up in each group.

**H0 (null hypothesis):** µ1 = µ2 = µ3 (the population means are all equal)

**H1 (alternative hypothesis):** at least one population mean is different from the rest

### Steps

1. **Total sum of squares (SST):**

Method-1
>\begin{align} SST = \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 = \sum_{i=1}^{A}\sum_{j=1}^{B}x_{ij}^2 - \frac{{GT}^2}{N} \end{align}
 - Calculate the difference between each observation and the grand mean irrespective of group(column) or subject(row)
 - Square each of these differences
 - Add the values to get SST

Method-2
>\begin{align} SST = (N-1)*s_{grand}^2\end{align}
\begin{align} variance_{total} = s_{grand}^2 = \frac{1}{N-1}\,\sum_{i=1}^{n_{subjects}}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{grand})^2 \end{align}
 - Calculate the unbiased variance for all observations together irrespective of group(column) or subject(row)
 - Multiply it with (total number of observations - 1) to get SST

Method-3
> \begin{align} SST = SSB + SSW =  SSB + (SSS + SSE) \end{align}

 - Calculate the unbiased variance for all observations together irrespective of group(column) or subject(row)
 - Multiply it with (total number of observations - 1) to get SST

2. **Between Group sum of squares or SSB(Group):**
 - Also called SS(Group) or SS(Treatment) or SS(Model)
 - Variance of group/column mean with grand mean

Method-1
>\begin{align} SSB(Group) = \sum_{j=1}^{B}{n_{j}*(\overline{x}_{j} - \overline{x}_{grand})}^2 \end{align}
 - Calculate the difference between the mean of each group/column and the grand mean
 - Square each of these differences
 - Multiply each result by the number of observations/participants within that group
 - Add the values for each group/column together

Method-2
>\begin{align} SSB(Group) = \frac{1}{A}\,\sum_{j=1}^{B}(\sum_{i=1}^{A}x_{ij})^2 - \frac{GT^2}{N} \end{align}
 - Calculate the sum of observations for each group (column)
 - Square each sum (group wise sum) and add them
 - Divide it by total number of subjects, call it A
 - Sum all observations to get grand sum(grand total)
 - Square the grand total and divide it by total number of observations, call it B
 - Calculate A-B to get the SSB(Group)

3. **Between Subject sum of squares or SSB(Subject):**
 - Variance of subject/row mean with grand mean

Method-1
>\begin{align} SSB(Subject) = \sum_{i=1}^{A}{n_{i}*(\overline{x}_{i} - \bar{x}_{grand})}^2 \end{align}
 - Calculate the difference between the mean of each subject/row and the grand mean
 - Square each of these differences
 - Multiply each result by the number of observations within that subject/row
 - Add the values for each group together

 Method-2
>\begin{align} SSB(Subject) = \frac{1}{B}\,\sum_{i=1}^{A}(\sum_{j=1}^{B}x_{ij})^2 - \frac{GT^2}{N} \end{align}
 - Calculate the sum of observations for each subject (row)
 - Square each sum (subject wise sum) and add them
 - Divide it by total number of groups, call it A
 - Sum all observations to get grand sum(grand total)
 - Square the grand total and divide it by total number of observations, call it B
 - Calculate A-B to get the SSB(Subject)

4. **Within Group sum of squares or SSW(Groups):**
- Variance of group/column observations with group mean

Method-1
>\begin{align} SSW(Group) = \sum_{j=1}^{B}\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}
 - Calculate the difference between each observation of the group/column and the group/column mean
 - Square each of these differences
 - Add the values for each group together and call it group error SS
 - Add the group error SS of all the groups to get the SSW

Method-2
>\begin{align} SSW(Group) = \sum_{j=1}^{B}(n_{j}-1)*s_{j}^2\end{align}
\begin{align} variance_{group(i)} = s_{j}^2 = \frac{1}{A-1}\,\sum_{i=1}^{A}(x_{ij}-\bar{x}_{j})^2 \end{align}
 - Calculate the unbiased variance for j-th group/column observations
 - Multiply it with (total observations in the group/column - 1) to get SSW

5. **Within Subjects sum of squares or SSW(Subject):**
- Variance of subject/row observations with subject mean

Method-1
>\begin{align} SSW(Subject) = \sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i})^2 \end{align}
 - Calculate the difference between each observation of the subject/rows and the subject/row mean
 - Square each of these differences
 - Add the values for each subject/row together and call it subject error SS
 - Add the subject error SS of all the groups to get the SSW(Subject)

Method-2
>\begin{align} SSW(Subject) = \sum_{i=1}^{A}(n_{i}-1)*s_{i}^2 \end{align}
\begin{align} variance_{subject(i)} = s_{i}^2 = \frac{1}{B-1}\,\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i})^2 \end{align}
 - Calculate the unbiased variance for i-th subject/row observations
 - Multiply it with (total observations in the subject/row - 1) to get SSW(Subject)

6. **Error sum of squares (SSE):** Next, we will calculate the error sum of squares (SSE), which can be found using the following formula:

\begin{align} SSE = SST – SSB(Group) – SSB(Subject) \end{align}

7. **Repeated measures ANOVA table:** Now that we have SSB, SSS, and SSE, we can fill in the repeated measures ANOVA table:

8. **Interpret the results:**

### Example Repeated Measures ANOVA

|| Patient | Drug-1 | Drug-2 | Drug-3 |
| --- | --- | --- | --- | --- |
|Index| i=1 | i=2 | i=3 | i=4 |
| --- | --- | --- | --- | --- |
|j=1| Patient 1 | 30 | 28 | 16 |
|j=2| Patient 2 | 14 | 18 | 10 |
|j=3| Patient 3 | 24 | 20 | 18 |
|j=4| Patient 4 | 38 | 34 | 20 |
|j=5| Patient 5 | 26 | 28 | 14 |

1. SST = (64.2667)(15-1) = 899.7

2. SSB = (5)(26.4-22.533)2 +(5)(25.6-22.533)2 + (5)(15.6-22.533)2 = 362.1

3. SSS = ((742 + 422 + 622 + 922 + 682)/3) – (3382/(5)(3)) = 441.1

4. SSE = 899.7 – 362.1 – 441.1 = 96.5

5. ANOVA Table
- Here is how we calculated the various numbers in the table:
  - df between: #groups – 1 = 3 – 1 = 2
  - df subject: #participants – 1 = 5 – 1 = 4
  - df error: df between * df subject = 2*4 = 8
  - MS between: SSB / df between = 362.1 / 2 = 181.1
  - MS subject: SSS / df subject = 441.1 / 4 = 110.3
  - MS error: SSE / df error = 96.5 / 8 = 12.1
  - F: MS between / MS error = 181.1 / 12.1 = 15.006

6. Intrepret result
- The F test statistic for this one-way repeated measures ANOVA is 15.006. To determine if this is a statistically significant result, we must compare this to the F critical value found in the F distribution table with the following values:

  - α (significance level) = 0.05
  - DF1 (numerator degrees of freedom) = df between = 2
  - DF2 (denominator degrees of freedom) = df error = 8

We find that the F critical value is 4.459.

Since the F test statistic in the ANOVA table is greater than the F critical value in the F distribution table, we reject the null hypothesis. This means we have sufficient evidence to say that there is a statistically significant difference between the mean response times of the drugs.

## Two way ANOVA

The two-away ANOVA deals with the case where there are two variables or factors and there may be interaction effects between the two factors.

You should use a two-way ANOVA when you’d like to know how two factors affect a response variable and whether or not there is an interaction effect between the two factors on the response variable.

Source: https://www.nature.com/articles/nmeth.3180

(a) The main effect is the difference between τ values (light gray), which is the response for a given level of a factor averaged over the levels of other factors. (b) The interaction effect is the difference between effects of A at the different levels of B or vice versa (dark gray, Δ). (c) Interaction effects may mask main effects.

### Hypotheses

There are three sets of hypothesis with the two-way ANOVA.
1. Main Effect - First Factor
  - **Null hypotheses(H0):** The population means of the first factor are equal. This is like the one-way ANOVA for the row factor.
  - **Alternate hypotheses(H0):** At least one population mean of the first factor is different from others.

2. Main Effect - Second Factor
  - **Null hypotheses(H0):** The population means of the second factor are equal. This is like the one-way ANOVA for the column factor.
  - **Alternate hypotheses(H0):** At least one population mean of the second factor is different from others.

3. Interaction Effect
  - **Null hypotheses(H0):** There is no interaction between the two factors. This is similar to performing a test for independence with contingency tables.
  - **Alternate hypotheses(H0):** There is some interaction between the two factors.

**Two Factor Factorial Experiment**

Each observation of group / treament can be described as:
\begin{align} x_{ijk} = (\mu + \tau_{j} + \beta_{i}) + (\tau_{j}\beta_{i}) + e_{ijk} \end{align}

where
 - $\mu$ = overall mean of effect all samples
 - $e_{ijk}$ = random error for component, we assume random errors $e_{ijk}$ are normally distributed with mean 0 and variance $\sigma^2$
 - $\tau_{j}$ = main (treatment) effect of j-th level of factor B (column)
 - $\beta_{i}$ = main (treatment) effect of i-th level of factor A (row)
 - $(\tau_{j}\beta_{i})$ = interaction effect between the i-th level of factor A & j-th level of B

**FIXED effect model:**
  -  Specifically A levels or treatment called Subjects are choosen from all possible levels (combination) of factor A
  - Specifically B levels or treatment called Groups are choosen from all possible levels (combination) of factor B
  - Sum of main effect or interaction for each factor assumed to be zero:
\begin{align} \sum_{j=1}^{B}\tau_{j} = 0 \end{align}
\begin{align} \sum_{j=1}^{A}\beta_{i} = 0 \end{align}
\begin{align} \sum_{j=1}^{B}(\tau_{j}\beta_{i}) = 0 \end{align}
\begin{align} \sum_{j=1}^{A}(\tau_{j}\beta_{i}) = 0 \end{align}

  - Hypotheses:
   - H0: $\tau_{1} = \tau_{2} = \tau_{3}...=\tau_{B}=0$, (there is no main effect of factor B)
   - H1: $\tau_{j} \neq 0$
   - H0: $\beta_{1} = \beta{2} = \beta_{3}...=\tau_{B}=0$, (there is no main effect of factor A)
   - H1: $\beta_{i} \neq 0$
   - H0: $(\tau_{1}\beta_{1}) = (\tau_{1}\beta_{2}) = (\tau_{1}\beta_{3})...=(\tau_{B}\beta_{A})=0$, (there is no main effect of factor B)
   - H1: $(\tau_{j}\beta_{i}) \neq 0$

### Variability & Expectation

#### Total variability - sum of squares identity
---
As there is only one factor, there is no interaction component

\begin{align} SST = SSB(Group) + SSB(Subject) + SS(Interaction) + SSE(Random) \end{align}

Partitioning SST:
> \begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk}-\bar{x}_{grand})^2 =
RA*\sum_{j=1}^{B}{(\overline{x}_{j} - \bar{x}_{grand})}^2
+
RB*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2
+
R*\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ijk}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2
+
\sum_{i=1}^{A} \sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk} - \bar{x}_{ij}) \end{align}

Partitioning SST degree of fredom:
>\begin{align} ABR-1 = (B-1) + (A-1) + (A-1)(B-1) + AB(R-1) \end{align}
---

#### Variance Expectation
1. Expected Between Groups Variance
 - Sum of squares:
>\begin{align} SSB(Groups) = (B-1)\sigma^2+AR*\sum_{j=1}^{B}\tau_{j}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Groups) = \frac{SSB(Groups)}{B-1} = \sigma^2+\frac{AR}{(B-1)}\,*\sum_{j=1}^{B}\tau_{j}^2 = \sigma^2+AR*\sigma_{\tau}^2\end{align}
2. Expected Between Subjects/Blocks Variance
 - Sum of squares:
>\begin{align} SSB(Subject) = (A-1)\sigma^2+BR*\sum_{i=1}^{A}\beta_{i}^2\end{align}
 - Mean sum of squares):
>\begin{align} MSB(Subject) = \frac{SSB(Subject)}{A-1} = \sigma^2+\frac{BR}{(A-1)}\,*\sum_{i=1}^{A}\beta_{i}^2 = \sigma^2+BR*\sigma_{\beta}^2\end{align}
3. Expected Interaction Variance
 - Sum of squares:
>\begin{align} SS(Interaction) = (A-1)(B-1)\sigma^2+R*\sum_{j=1}^{B}\sum_{i=1}^{A}{(\beta_{ij}^2\tau_{ij}^2)}\end{align}
 - Mean sum of squares:
>\begin{align} MS(Interaction) = \frac{SS(Interaction)}{(A-1)(B-1)} = \sigma^2+\frac{R*\sum_{j=1}^{B}\sum_{i=1}^{A}{(\beta_{ij}^2\tau_{ij}^2)}}{(A-1)(B-1)}\end{align}
3. Expected Random Error Variance
 - Sum of squares:
>\begin{align} SSE(Random) = (AB)(R-1)\sigma^2\end{align}
 - Mean sum of squares:
>\begin{align} MSE(Random) = \frac{SSE(Random)}{(AB)(R-1)} = \sigma^2\end{align}

### Sum of Squares Formula

---
|Source of Variation| SS Form 1 | SS Form 2 | Degree of freedom | Notes, Comments and other details |
|---|---|---|---|---|
|$SST$|\begin{align} \sum_{i=1}^{A}\sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk}-\bar{x}_{grand})^2 \end{align}||$(N-1)$||
|$SSB(Groups)$|\begin{align} R*A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 \end{align}||$(B-1)$||
|$SSB(Subjects)$|\begin{align} R*B*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2 \end{align}||$(A-1)$||
|$SS(Interaction)$|\begin{align} R*\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ijk}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2 \end{align}||$(A-1)*(B-1)$||
|$SSE$|\begin{align} \sum_{i=1}^{A} \sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk} - \bar{x}_{ij})^2\end{align}||$(N-A*B)$||

### Two Way ANOVA table
| Source of Variation | Sum of Square (SS) | Degrees of Freedom Calculation (df) | Mean Square Calculation (MS) | F ratio |
| --- | --- | --- | --- | --- |
| Groups-Columns (Between) - Main Effect Factor A | SSB(Groups) | $df_{groups} = (B-1)$ | $MSB(Groups) = SSB(Groups)/df_{groups}$ | F = MSB(Groups)/MSE |
| Subject-Rows (Between) - Main Effect Factor B | SSB(Subjects) | $df_{subjects} = (A-1)$ | $MSB(Subjects) = SSB(Subjects)/df_{subjects}$ | F = MSB(Subjects)/MSE |
| Interaction Effect (Subject-Group) or (Row-Column) | SS(Interaction) | $df_{interaction} = (A-1)*(B-1)$ | $MS(Interaction) = SS(Interaction)/df_{interaction}$ | F = MS(Interaction)/MSE |
| Error (Within) or Random variation | SSE | $df_{error} = (N-A*B)$| $MSE = SSE/df_{error}$ | |
| Total | SST | $df_{total} = (N-1)$ |||

### Naming Convention

---
- Group = Column = Treatment = Level
- Subject = Row = Block
- i = row number/subject/Block
- j = column number/group/Level
- Replication = Trials/Observations  per Cell(Row- Column combination)
- i = row number/subject
- j = column number/group
- $n_{i}$ = $B$ = total number of observations in the i-th subject/column,
- $n_{j}$ = $A$ = total number of observations in the j-th group/column,
- $A$ = total number of subjects/rows
- $B$ = total number of groups/columns
- $R$ =  Number of replications/trials
- $N = A*B*R$ = the total number of observations in the entire dataset
- $x_{ijk}$ = observation at i-th subject (row) and j-th group(column)
- $GT = \sum_{k=1}^{N}{x_{k}}$ = the grand total of the entire dataset
- $\bar{x}_{i} = \frac{1}{n_{i}}\sum_{j=1}^{n_{i}} x_{ij} = \frac{1}{B}\sum_{i=1}^{B} x_{ij}$ = the mean of the i-th subject/row,
- $\bar{x}_{j} = \frac{1}{n_{j}}\sum_{i=1}^{n_{j}} x_{ij} = \frac{1}{A}\sum_{i=1}^{A} x_{ij}$ = the mean of the j-th group/column,
- $\bar{x}_{ij} = \frac{1}{R}\sum_{k=1}^{R} x_{ij} = \frac{1}{R}\sum_{k=1}^{R} x_{ij}$ = the mean of the cell (i-th subject/row, j-th group/column) = mean of observations at the i th level of subject(row) and the j th level of group(column)
- $\bar{x}_{grand} = \frac{1}{N}\sum_{ijk} x_{ijk}$ = the grand mean or mean of the entire dataset
- $s_{unbiased}^2$= the unbiased sample variance for the entire dataset
- $s_{i}^2$ = the unbiased sample variance for the i-th subject/row observations
- $s_{j}^2$ = the unbiased sample variance for the j-th group/column observations
---

### Steps by hand

**Example:**

| Treatment |||| group-1 | group-2 | group-3 |
| --- | --- | --- | --- | --- | --- | --- |
|||Index| i=1 | i=2 | i=3 | i=4 |
|Replication|| --- | --- | --- | --- | --- |
|k=1|subject-1|j=1| - | - | - | - |
|k=1|subject-2|j=2| - | - | - | - |
|k=1|subject-3|j=3| - | - | - | - |
|k=1|subject-4|j=4| - | - | - | - |
|k=2|subject-1|j=1| - | - | - | - |
|k=2|subject-2|j=2| - | - | - | - |
|k=2|subject-3|j=3| - | - | - | - |
|k=2|subject-4|j=4| - | - | - | - |

### Steps

1. **Total sum of squares (SST):**

Method-1
>\begin{align} SST = \sum_{i=1}^{A}\sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ij}-\bar{x}_{grand})^2 \end{align}
 - Calculate the difference between each observation and the grand mean irrespective of group(column) or subject(row)
 - Square each of these differences
 - Add the values together to get SST

Method-2
> \begin{align} SST = SSB(Groups) + SSB(Subjects) + SS(Interaction) + SSW(Groups) \end{align}

2. **Between Group sum of squares SSB(Groups) - Main Effect due to Factor A:**
- Also called SS(Group) or SS(Treatment) or SS(Model)
- variance of group/column mean vs grand mean

Method-1
>\begin{align} SSB(Groups) = R*A*\sum_{j=1}^{B}{(\bar{x}_{j} - \bar{x}_{grand})}^2 \end{align}
 - Calculate the difference between the mean of each group and the grand mean
 - Square each of these differences
 - Multiply each result by the number of observations/participants within that group
 - Add the values for each group together

3. **Between Subject sum of squares SSB(Subjects) - Main Effect due to Factor B:**
- variance of subject/row mean vs grand mean

Method-1
>\begin{align} SSB(Subjects) = R*B*\sum_{i=1}^{A}{(\bar{x}_{i} - \bar{x}_{grand})}^2 \end{align}
 - Calculate the difference between the mean of each subject/row and the grand mean
 - Square each of these differences
 - Multiply each result by the number of observations/participants within that subject/row
 - Add the values for each group together

4. **Interaction Effect SS(AB):**
- INteraction between two factors Subject-Group or in other word row-column factor

Method-1
>\begin{align} SS(Interaction) = R*\sum_{i=1}^{A}\sum_{j=1}^{B}(x_{ij}-\bar{x}_{i}-\bar{x}_{j}+\bar{x}_{grand})^2 \end{align}
 - Add grand mean to each observation of entire dataset
 - Subtract subject/row mean and group/column mean to these updated observations
 - Square each of these differences
 - Add the values together

5. **Error sum of squares (SSE):**
- Within samples sum of squares due to chance errors

>\begin{align} SSE = \sum_{i=1}^{A} \sum_{j=1}^{B} \sum_{k=1}^{R} (x_{ijk} - \bar{x}_{ij})^2\end{align}

6. **Two Way ANOVA table:** Now that we have SSB, SSS, and SSE, we can fill in the two way ANOVA table.

6. **Interpret the results:**

## References

- http://faculty.business.utsa.edu/manderso/STA4723/readings/Douglas-C.-Montgomery-Design-and-Analysis-of-Experiments-Wiley-2012.pdf
- https://www.statology.org/one-way-anova-by-hand/
- https://www.statology.org/repeated-measures-anova-by-hand/
- https://online.stat.psu.edu/stat200/lesson/10/10.2
- http://statweb.stanford.edu/~susan/courses/s141/hoanova.pdf
- http://statweb.stanford.edu/~susan/courses/s141/exanova.pdf
- http://web.pdx.edu/~newsomj/uvclass/ho_ANOVA.pdf
- http://web.pdx.edu/~newsomj/uvclass/ho_within.pdf
- http://web.pdx.edu/~newsomj/uvclass/ho_mixed.pdf
- https://www.spss-tutorials.com/anova-what-is-it/
- https://www.spss-tutorials.com/repeated-measures-anova/
- https://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/anova/how-to/one-way-anova/methods-and-formulas/methods-and-formulas/
- https://support.minitab.com/en-us/minitab-express/1/help-and-how-to/modeling-statistics/anova/how-to/two-way-anova/methods-and-formulas/methods-and-formulas/
- https://www.discoveringstatistics.com/statistics-hell-p/porus-comparing-means/one-way-anova/
- https://www.discoveringstatistics.com/statistics-hell-p/porus-comparing-means/repeated-measures-designs/
- https://learn.lboro.ac.uk/archive/olmp/olmp_resources/pages/workbooks_1_50_jan2008/Workbook44/44_1_one_way_anova.pdf
- https://learn.lboro.ac.uk/archive/olmp/olmp_resources/pages/workbooks_1_50_jan2008/Workbook44/44_2_two_way_anova.pdf
- https://people.richland.edu/james/lecture/m170/ch13-1wy.html
- https://people.richland.edu/james/lecture/m170/ch13-2wy.html
- http://pytolearn.csd.auth.gr/d1-hyptest/12/ptl_anova1.html
- http://pytolearn.csd.auth.gr/d1-hyptest/12/ptl_anovaR.html
- http://pytolearn.csd.auth.gr/d1-hyptest/12/ptl_anova2.html
- https://sixsigmastudyguide.com/anova-analysis-of-variation/
- https://www.alexanderdemos.org/ANOVA8.html
- https://courses.lumenlearning.com/introstats1/chapter/the-f-distribution-and-the-f-ratio/
- https://www.nature.com/articles/nmeth.3180
- http://www.stat.yale.edu/Courses/1997-98/101/expdes.htm
- https://online.stat.psu.edu/stat503


# Hydrogen calculations using `sympy`
The `sympy` library does symbolic math.  It also includes several useful things in the [`sympy.physics`](https://docs.sympy.org/latest/modules/physics/index.html) module, like hydrogen and harmonic oscillator wavefunctions.

```python
from sympy import *
from sympy.physics.hydrogen import *
```

```python
n = symbols("n")
ell = symbols("ℓ")
m = symbols("m")
```

```python
r, theta, phi = symbols("r θ ϕ", real=True)
```

```python
# Unperturbed energy of state n
E_nl(n)
```

$\displaystyle - \frac{1}{2 n^{2}}$

```python
# Wavefunciton of state |n,ℓ,m>
Psi_nlm(n, ell, m, r, phi, theta)
```

$\displaystyle \frac{\left(\frac{2 r}{n}\right)^{ℓ} \sqrt{\frac{\left(n - ℓ - 1\right)!}{n^{4} \left(n + ℓ\right)!}} \sqrt{\frac{2 ℓ \left(- m + ℓ\right)!}{\left(m + ℓ\right)!} + \frac{\left(- m + ℓ\right)!}{\left(m + ℓ\right)!}} e^{- \frac{r}{n}} e^{i m ϕ} L_{n - ℓ - 1}^{\left(2 ℓ + 1\right)}\left(\frac{2 r}{n}\right) P_{ℓ}^{\left(m\right)}\left(\cos{\left(θ \right)}\right)}{\sqrt{\pi}}$

```python
# Check a specfic state to compare
Psi_nlm(3, 1, 0, r, phi, theta)
```

$\displaystyle \frac{\sqrt{2} r \left(4 - \frac{2 r}{3}\right) e^{- \frac{r}{3}} \cos{\left(θ \right)}}{54 \sqrt{\pi}}$

```python
# define some aliases
z = r * cos(theta)
x = r * sin(theta)*cos(phi)
y = r * sin(theta)*sin(phi)
```

## Example integrals
Let's calculate
$$
\langle 2,1,0 \vert z \vert 1,0,0\rangle = \int\limits_{r=0}^{\infty} \int\limits_{\phi=0}^{2\pi} \int\limits_{\theta=0}^\pi
\phi^*_{2,1,0}\, z\, \phi_{1,0,0}\,r^2\sin\theta\,dr\,d\theta\,d\phi
$$

```python
# Set up some integrals
Integral(conjugate(Psi_nlm(2,1,0,r, phi, theta))*z*Psi_nlm(1,0,0,r,phi,theta)*r**2*sin(theta),(phi,0,2*pi),(theta,0,pi),(r,0,oo))
```

$\displaystyle \int\limits_{0}^{\infty}\int\limits_{0}^{\pi}\int\limits_{0}^{2 \pi} \frac{\sqrt{2} r^{4} e^{- \frac{3 r}{2}} \sin{\left(θ \right)} \cos^{2}{\left(θ \right)}}{8 \pi}\, dϕ\, dθ\, dr$

```python
# perform the integration
_.doit()
```

$\displaystyle \frac{128 \sqrt{2}}{243}$

```python
# approximate this in decimal form
N(_)
```

$\displaystyle 0.744935539027803$

```python
_*5.292e-11 # convert from atomic units to SI units
```

$\displaystyle 3.94219887253513 \cdot 10^{-11}$

```python
_**2
```

$\displaystyle 1.55409319506173 \cdot 10^{-21}$

```python
_21**2
```

$\displaystyle 0.554928957306644$

```python

```


```python
from scipy import optimize
import math as math
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import ipywidgets as wd
```

# A simple CGE model
In the first section of this code we construct a simple Computable General Equilibruim (CGE) model and calibrate it to an observed equilibrium. In the second section of this code we examine the properties of the model and the effect of certain shocks.

## Building a CGE model - a crash course in CGE-modelling
The building blocks of a CGE model are:
1. Model equations describing an economy
2. Behavioural parameters
3. Variable values representing an initial equilibrium

Below we will construct a simple CGE model in three steps:
1. We will define the model's equations. The model is a simple open economy with producers, consumers and a world market
2. We will discuss the behavioural parameters
3. We will calibrate the models unknown behavioural parameters such that the model can replicate the initial equilibrium

### 1. Model equations
Our model is a slightly modified version of a model from the course in applied CGE-modelling. The model consists of producers who buy labour and produce a single domestic consumption good, cosumers who sell labour to domestic producers and buy both domestic and foregin consumption goods, and a world market where producers can sell sell domestic goods and consumers can buy foreign goods.
#### Domestic producers
The domestic producers are in full competition with each other and all have the CES-production function:

$$ Y = \left[ \mu^\frac{1}{E_Y} L^\frac{E_Y-1}{E_Y} \right]^\frac{E_Y}{E_Y-1} $$

They maximize their profits:

$$ \pi = p_dY - wL $$

This implies the following labour demand and zero profit condition, i.e. the price of the domestic consumption good is equal to the avg. costs of producing it.

$$
\begin{align}
L = \mu  \left( \frac{w}{p_d} \right)^{-E_Y} Y \tag{1}
\end{align}
$$

$$
\begin{align}
p_d = \frac{w L}{Y} \tag{2}
\end{align}
$$

Below we define the two equations:

```python
## Labour demand
def E1_L_D(mu,w,p_d,E_Y,Y) :
    """ Funtion defining the producers' demand for labour. """
    return mu * (w/p_d)**(-E_Y) * Y

## Zero profit condition - expressed as a function of p_d
def E2_p_d(Y,w,L) :
    """  Function defining the zero profit condidtion. """
    return w*L / Y
```

#### Domestic consumers
The model has N domestic consumers who have a CES utility function and derive utility from the consumption of domestic and foreign goods:

$$ U(C_d,C_f)=\left[ \gamma_d^\frac{1}{E_C} C_d^\frac{E_C-1}{E_C} + \gamma_f^\frac{1}{E_C} C_f^\frac{E_C-1}{E_C} \right]^\frac{E_C}{E_C-1} $$

Their budget constaint is:

$$
\begin{align}
p_d C_d + p_f C_f = w N \tag{3}
\end{align}
$$

The consumers maximize their utility subject to their budget constraint. This implies the following demand functions:

$$
\begin{align}
C_d = \gamma_d \left(\frac{p_d}{P_C}\right)^{-E_C} \frac{wN}{P_C} \tag{4}
\end{align}
$$

$$
\begin{align}
C_f = \gamma_d \left(\frac{p_f}{P_C}\right)^{-E_C} \frac{wN}{P_C} \tag{5}
\end{align}
$$

$P_C$ is a CES-price index.

```python
## The budget of the households constraint
def E3_wN(p_d,C_d,p_f,C_f) :
    """ Budget constraint of the households. """
    return p_d*C_d+p_f*C_f

## The demand for the domestic consumption good
def E4_C_d(gamma_d,p_d,P_C,E_C,w,N) :
    """ Demand for domestic consumption."""
    return gamma_d * (p_d/P_C)**(-E_C) *w*N/P_C

## The demand for the foreign consumption good
def E5_C_f(gamma_f,p_f,P_C,E_C,w,N) :
    """ Demand for foregin consumption good."""
    return gamma_f * (p_f/P_C)**(-E_C) *w*N/P_C
```

#### The labour market
We assume that everybody works. Thus the labour market becomes:

$$
\begin{align}
L = N \tag{6}
\end{align}
$$

```python
## The labour market
def E6_L_S(N) :
    """ Labour market. """
    return N
```

#### The goods market
In equilibria the entire domestic production is either consumed by domestic consumers or exported:

$$
\begin{align}
Y = C_d + X \tag{7}
\end{align}$$

```python
## The goods market
def E7_Y(C_d,X) :
    """ The goods market. """
    return C_d + X
```

#### Foregin trade
We model the foreign trade using an Armington's approach, where the export depends on the relative price of foreign and domestic goods:

$$
\begin{align}
X = \phi \left(\frac{p_d}{p_f}\right)^{-E_X} \tag{8}
\end{align}
$$

We use the foreign price as a numéraire:

$$
\begin{align}
p_f = 1 \tag{9}
\end{align}
$$

```python
## Armington's approach to foreign trade
def E8_X(phi,p_d,p_f,E_X) :
    """ Foreign trade """
    return phi*(p_d/p_f)**(-E_X)

## Setting the foreign price as a numéraire
def E9_p_f() :
    """ The price of the foreign good """
    return 1
```

#### The model
Equation (1)-(9) represents the nine equations in our model. We treat the nine variables $Y$, $L$, $w$, $C_d$, $C_f$, $p_d$, $p_f$, $P_C$, and $X$ as endogenous.

The remaining parameters are $N$, $\mu$, $\gamma_d$, $\gamma_f$, $\phi$, $E_Y$, $E_C$ and $E_X$.

### 2. Behavioural parameters
The model has 8 unknown parameters: $N$, $\mu$, $\gamma_d$, $\gamma_f$, $\phi$, $E_Y$, $E_C$ and $E_X$. $\mu$, $\gamma_d$, $\gamma_f$, $\phi$, $E_Y$, $E_C$ and $E_X$ are the so-called behavioural parameters in the CES-functions and in Armington's approach to foreign trade. In the original assignment from the course in applied CGE-modelling, it was given that $E_Y \equiv 2.0$, $E_C \equiv 0.5$, and $E_C \equiv 5.0$.

The remaining behavioural parameters and $N$ are determined by calibrating the model to the initial equilibrium, e.g. setting $\phi$ in equation $(8)$ such that the equation is consistent with the initial values of $X$, $p_d$, and $p_f$.

```python
E_Y = 2.0
E_C = 0.5
E_X = 5.0
```

### 3. Initial dataset and calibration
To model is calibrated to an initial equilibrium described by the IO-table:

| I/O | $PS$   | $PC$  | $X$   |
| --- | --:  | --: | --: |
| $PS$  | 0    | 800 | 200 |
| $M$   | 0    | 200 |   0 |
| $w$ | 1000 | 0   |   0 |

PS is the private sector, $M$ is imports, $w$ is wages, PC is private consumption, and $X$ is exports. The rows describe input and the columns output. The private sector e.g. uses 1.000 units of wage, and outputs 800 units of goods used for domestic consumption. The table represents a simple National Account in current prices.

```python
rows    = ['PS','M' ,'w']
columns = ['PS','PC','X']

data = [(0    , 800 , 200),
        (0    , 200 ,   0) ,
        (1000 , 0   ,   0) ]

IO = pd.DataFrame(data,columns=columns,index=rows)
```

#### Initialising the variables
All prices are simply set to 1 and interpretated as price indices. The ammount of labour, the total prodcution etc. are found using the IO-table. The IO-table is in current prices so we devide by the price indices.

```python
## Endogenous variables in initial equilibrium
# Prices
p_d0 = 1
p_f0 = 1
w0   = 1
P_C0 = 1

# Amounts
L0   = IO.loc['w']['PS']/w0
Y0   = IO.loc['w']['PS']/p_d0
X0   = IO.loc['PS']['X']/p_d0
C_d0 = IO.loc['PS']['PC']/p_d0
C_f0 = IO.loc['M']['PC']/p_f0

## List of initial values
ini_list = [Y0,L0,w0,C_d0,C_f0,p_d0,p_f0,P_C0,X0]
```

We also save the initial equilibrium in a dataframe.

```python
## Saving initial equilibrium in a dictionary
ini_eq = [{"Shock" : "Initial eq.",
            "$L$"   : "%1.f"  % L0,
            "$Y$"   : "%0.2f" % Y0,
            "$w$"   : "%0.3f" % w0,
            "$C_d$" : "%0.2f" % C_d0,
            "$p_d$" : "%0.3f" % p_d0,
            "$C_f$" : "%0.2f" % C_f0,
            "$p_f$" : "%0.3f" % p_f0,
            "$P_C$" : "%0.3f" % P_C0,
            "$X$"   : "%0.2f" % X0}]

## Converting the dictionary to a data-frame
ini_eq = pd.DataFrame(ini_eq)

## Rearranging the columns of the dataframe
cols = ['Shock','$L$','$Y$','$X$','$C_d$','$C_f$','$w$','$p_d$','$p_f$','$P_C$']
ini_eq = ini_eq[cols]

## Printing the inital equilibrium
ini_eq
```

      Shock
      $L$
      $Y$
      $X$
      $C_d$
      $C_f$
      $w$
      $p_d$
      $p_f$
      $P_C$

      0
      Initial eq.
      1000
      1000.00
      200.00
      800.00
      200.00
      1.000
      1.000
      1.000
      1.000

#### Calibrating
The next step is to calibrate $N$, $\mu$, $\gamma_d$, $\gamma_f$, $\phi$, i.e. setting their values such that the model replicates the initial equilibrium. In the case of e.g. $\mu$ this is done by solving the labour demand function, equation (1), for $\mu$ with the inital values inserted:

$$
\begin{align}
L_0 = \mu  \left( \frac{w_0}{p_{d0}} \right)^{-E_Y} Y_0
\end{align}
$$

We do this numerically below.

```python
## Calibrating population size
def f_L_S(N) :
    return E6_L_S(N) - L0

solution = optimize.root(f_L_S, (0) )
N = np.asscalar(solution.x)

## Calibrating mu
def f_mu(mu) :
    return E1_L_D(mu,w0,p_d0,E_Y,Y0) - L0

solution = optimize.root(f_mu, (0) )
mu = np.asscalar(solution.x)

## Calibrating gamma_d and gamma_f
def f_gamma_d(gamma_d) :
    return E4_C_d(gamma_d,p_d0,P_C0,E_C,w0,N) -C_d0

solution = optimize.root(f_gamma_d, (0) )
gamma_d = np.asscalar(solution.x)

def f_gamma_f(gamma_f) :
    return E5_C_f(gamma_f,p_f0,P_C0,E_C,w0,N) -C_f0

solution = optimize.root(f_gamma_f, (0) )
gamma_f = np.asscalar(solution.x)

## Calibrating phi
def f_phi(phi) :
    return E8_X(phi,p_d0,p_f0,E_X) - X0

solution = optimize.root(f_phi, (0) )
phi = np.asscalar(solution.x)

## Printing the results
print("N =",N,", mu =",mu,",","gamma_d =",gamma_d,",","gamma_f =",gamma_f,",","phi =",phi)
```

    N = 1000.0 , mu = 1.0 , gamma_d = 0.8 , gamma_f = 0.2 , phi = 200.0

#### Solving the model
The last step of building our model is to solve it and check if it replicates the initial equilibrium. If it does then our model works. This is called a zero-shock in CGE-modelling.

The model is solved as a system of nine equations, equation $(1)$-$(9)$, and nine unknowns, $Y$, $L$, $w$, $C_d$, $C_f$, $p_d$, $p_f$, $P_C$, and $X$:

$$
\begin{align}
L = \mu  \left( \frac{w}{p_d} \right)^{-E_Y} Y \tag{1}
\end{align}
$$

$$
\begin{align}
p_d = \frac{w L}{Y} \tag{2}
\end{align}
$$

$$
\begin{align}
p_d C_d + p_f C_f = w N \tag{3}
\end{align}
$$

$$
\begin{align}
C_d = \gamma_d \left(\frac{p_d}{P_C}\right)^{-E_C} \frac{wN}{P_C} \tag{4}
\end{align}
$$

$$
\begin{align}
C_f = \gamma_d \left(\frac{p_f}{P_C}\right)^{-E_C} \frac{wN}{P_C} \tag{5}
\end{align}
$$

$$
\begin{align}
L = N \tag{6}
\end{align}
$$

$$
\begin{align}
Y = C_d + X \tag{7}
\end{align}
$$

$$
\begin{align}
X = \phi \left(\frac{p_d}{p_f}\right)^{-E_X} \tag{8}
\end{align}
$$

$$
\begin{align}
p_f = 1 \tag{9}
\end{align}
$$

We solve the model by defining a function called CGEsolve. The function defines the system of equations and solves it. This functionh will become handy later, when we examine the properties of the model.

```python
## Function solving the CGE model taking parameters as inputs
def CGEsolve(N,mu,gamma_d,gamma_f,phi,E_Y,E_C,E_X,ini_list,status='yes') :
    """
    This function defines and solves the CGE-model as a function of its parameters and
    a list of inital values for the solver.

    status = 'yes' prints a summary of the solver results
    """
    ## Defining the CGE model as a system of nine equations and nine unknowns
    def CGEmodel(variables) :
        ## Defining variables
        (Y,L,w,C_d,C_f,p_d,p_f,P_C,X) = variables

        ## Defining equations
        EQ_L_D = E1_L_D(mu,w,p_d,E_Y,Y) - L
        EQ_p_d = E2_p_d(Y,w,L) - p_d
        EQ_wN  = E3_wN(p_d,C_d,p_f,C_f) - w*N
        EQ_C_d = E4_C_d(gamma_d,p_d,P_C,E_C,w,N) - C_d
        EQ_C_f = E5_C_f(gamma_f,p_f,P_C,E_C,w,N) - C_f
        EQ_L_S = E6_L_S(N) - L
        EQ_Y   = E7_Y(C_d,X) - Y
        EQ_X   = E8_X(phi,p_d,p_f,E_X) - X
        EQ_p_f = E9_p_f()-p_f

        ## Returning a list of equations
        return [EQ_L_D,EQ_p_d,EQ_wN,EQ_C_d,EQ_C_f,EQ_L_S,EQ_Y,EQ_X,EQ_p_f]

    ## Solving the model  using the inital equilibrium as starting values fore the solver
    solution = optimize.root(CGEmodel,(ini_list[0],ini_list[1],ini_list[2],ini_list[3],ini_list[4],ini_list[5],ini_list[6],ini_list[7],ini_list[8]))

    ## Prints the status of the solver
    if status=='yes' :
        print(solution.message,"Success =",solution.success)

    ## Returning solution
    return solution

## Solving the model
zeroshock = CGEsolve(N,mu,gamma_d,gamma_f,phi,E_Y,E_C,E_X,ini_list,status='yes')
```

    The solution converged. Success = True

```python
## Function creating a dataframe with results
def CGEresults(solution,name,nice='yes') :
    """
    This function takes the results from CGEsolve function and stores the variable values in a data frame.

    solution = a result from the CGEsolve function
    name = name of the shock
    nice = yes if results are for printing in a table.
    """
    ## Saving the results in a dictionary - strings for presenting results
    if nice == 'yes' :
        results = [{"Shock" : name,
                "$Y$"   : "%0.2f" % solution.x[0],
                "$L$"   : "%1.f"  % solution.x[1],
                "$w$"   : "%0.3f" % solution.x[2],
                "$C_d$" : "%0.2f" % solution.x[3],
                "$p_d$" : "%0.3f" % solution.x[5],
                "$C_f$" : "%0.2f" % solution.x[4],
                "$p_f$" : "%0.3f" % solution.x[6],
                "$P_C$" : "%0.3f" % solution.x[7],
                "$X$"   : "%0.2f" % solution.x[8]}]
    ## floats for figures
    else :
        results = [{"Shock" : name,
                "$Y$"   : solution.x[0],
                "$L$"   : solution.x[1],
                "$w$"   : solution.x[2],
                "$C_d$" : solution.x[3],
                "$p_d$" : solution.x[5],
                "$C_f$" : solution.x[4],
                "$p_f$" : solution.x[6],
                "$P_C$" : solution.x[7],
                "$X$"   : solution.x[8]}]
    ## Converting the results to a dataframe
    results = pd.DataFrame(results)

    ## Rearranging the columns
    cols = ['Shock','$L$','$Y$','$X$','$C_d$','$C_f$','$w$','$p_d$','$p_f$','$P_C$']
    results = results[cols]

    ## Returning the dataframe
    return results

## Printing the results
ini_eq.append(CGEresults(zeroshock,'Zero-shock',nice='yes'))
```

      Shock
      $L$
      $Y$
      $X$
      $C_d$
      $C_f$
      $w$
      $p_d$
      $p_f$
      $P_C$

      0
      Initial eq.
      1000
      1000.00
      200.00
      800.00
      200.00
      1.000
      1.000
      1.000
      1.000

      0
      Zero-shock
      1000
      1000.00
      200.00
      800.00
      200.00
      1.000
      1.000
      1.000
      1.000

We have successfully solved the model. Our 'zero'-shock replicates the intial equilibrium. The model is correctly calibrated.

## The effects of a productivity shock and the price sensitivity of exports
In this section we examine how a 10 percent increase in productivity affects our model. The shok is implemented by increasing $\mu$ by 10 percent.

```python
prodshock = CGEsolve(N,1.1*mu,gamma_d,gamma_f,phi,E_Y,E_C,E_X,ini_list,status='yes')
```

    The solution converged. Success = True

```python
ini_eq.append(CGEresults(prodshock,'Prod-shock',nice='yes'))
```

      Shock
      $L$
      $Y$
      $X$
      $C_d$
      $C_f$
      $w$
      $p_d$
      $p_f$
      $P_C$

      0
      Initial eq.
      1000
      1000.00
      200.00
      800.00
      200.00
      1.000
      1.000
      1.000
      1.000

      0
      Prod-shock
      1000
      1100.00
      221.83
      878.17
      217.28
      1.077
      0.979
      1.000
      0.984

The shock went as expected. A 10 percent increase in the productivity of the only input in a CES-function with constant returns to scale equals a 10 percent increase in domestic production. An increase in production also increases the export, as a larger supply lowers the price of the domestic good and thus increases exports. Also, a higher productivity leads to higher wages and thus higher consumption of both domestic and foregin goods.

But how does the effect of the shock depend on $E_X$ in equation (8)? An inelastic (low values of $E_X$) foregin demand for domestic goods imply lower prices for domestic goods in order to get the market for domestic goods to clear. To see how dominant this effect is, we solve the model for a range of $E_X$'s.

```python
## Solving the models for E_X=2
shock = CGEsolve(N,1.1*mu,gamma_d,gamma_f,phi,E_Y,E_C,2,ini_list,status='yes')
results = CGEresults(shock,2,nice = 'no')
```

    The solution converged. Success = True

```python
## Solving the model for E_X=2-30 - [we don't print the status of the solver, but we have checked.
##                                   The solver behaves nicely for all values of E_X]
for i in range(21,301) :
    #print(0.1*i)
    shock = CGEsolve(N,1.1*mu,gamma_d,gamma_f,phi,E_Y,E_C,0.1*i,ini_list,status='no')
    results = results.append(CGEresults(shock,0.1*i,nice='no'))
```

We present the results in an interactive figure.

```python
varbls = ['$L$','$Y$','$X$','$C_d$','$C_f$','$w$','$p_d$','$p_f$','$P_C$']

def fig_plt(var) :
    y = results[var].values
    x = results['Shock'].values
    plt.plot(x,y)
    plt.yticks(np.arange(0.95*y.min(), 1.05*y.max(), (1.05*y.max()-0.95*y.min())/4))
    return plt.show()

var_select = wd.Dropdown(options=varbls, description='Variables')
wd.interact(fig_plt,var=var_select);
```

    interactive(children=(Dropdown(description='Variables', options=('$L$', '$Y$', '$X$', '$C_d$', '$C_f$', '$w$',…

The most interesting figures are those of $X$ and $p_d$. It shows that an inelastic foregin demand implies, that the domestic producers have to lower their prices by a lot to clear the goods market. The lower prices affects the wages and thus the consumption of both foregin and domestic goods in equilibria with different $E_X$.


```python
from sympy import init_printing; init_printing()
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all" # display all expression in one cell instead of the last one
```

## Initialization of Variables

```python
from sympy.abc import x
from sympy import integrate, Eq, symbols, Array, Function
from sinupy.algebra.tensor import cross
E_x, E_y, E_z = symbols("E_x, E_y, E_z", real=True); E = Array([E_x, E_y, E_z]) # Array([E_x, E_y, E_z]) or Array([E_x, 0, 0])
B_x, B_y, B_z = symbols("B_x, B_y, B_z", real=True); B = Array([B_x, B_y, B_z]) # Array([B_x, B_y, B_z]) or Array([0, 0, B_z])
t = symbols("t", real=True)
E; B; t
r_x, r_y, r_z = [f(t) for f in symbols("r_x, r_y, r_z", real=True, cls=Function)]; r = Array([r_x, r_y, r_z])
v_x, v_y, v_z = [f(t) for f in symbols("v_x, v_y, v_z", real=True, cls=Function)]; v = Array([v_x, v_y, v_z])
```

## Utility Functions

```python
def divide_Array_Eq(eq):
    from sympy import Eq
    assert(eq.lhs.shape == eq.rhs.shape)
    eq_shape = eq.lhs.shape
    arr_order = len(eq_shape)
    if arr_order > 1: # Not yet tested for high order tensor
        eq_list = []
        for i in range(eq_shape[0]):
            eq_list.append(
                divide_Array_Eq(Eq(eq.lhs[i], eq.rhs[i]))
            )
        return eq_list
    elif arr_order == 1:
        return [Eq(eq.lhs[i], eq.rhs[i]) for i in range(eq_shape[0])]
    else:# arr_order == 0
        return eq

def find_trig_period(expr, var):
#     from collections.abc import Iterable
#     if isinstance(var, Iterable):
#         return {find_trig_omega(expr, a_var) for a_var in var}
    from sympy.functions.elementary.trigonometric import TrigonometricFunction
    if isinstance(expr, TrigonometricFunction):
        try:
            return {expr.period(symbol=var)}
        except:
            return set()
    else:
        periods = [find_trig_period(sub_expr, var) for sub_expr in expr.args]
        return set().union(*periods)

def pick_min_period(expr, t):
    Ts = find_trig_period(expr, t)
    if len(Ts) == 0:
        raise ValueError("The expression does not contain Trigonometric functions, please decide the T parameter manually.")
    try:
        return min(Ts)
    except TypeError: # If we can not decide which period is the min one, a random one is chosen.
        return next(iter(Ts))

def time_average_of_TrigFunc(expr, t, method="Integrate_Then_Average", T=None):

    if method=="Integrate_Then_Average":
        from sympy import integrate

        if T is None:
            T = pick_min_period(expr, t)
        return integrate(expr, (t, 0, T)) / T
#     elif method=="Expand_Multiple_Angle_Then_Subs":
#         raise NotImplementedError(f"{__name__} not yet implemented for the method '{method}'.")
    else:
        raise ValueError(f"The `method` parameter can only be 'Integrate_Then_Average' right now.")

if __name__ == '__main__': # Code for test
    from sympy import sin, sinh, pi, symbols
    a = symbols("a", positive=True)
    b, t = symbols("b, t", real=True)
    find_trig_period(sin((b**2)*t)+sin(b*t), t)
```

## Set up the Newton Second Law Equation (ode system)

```python
import sympy.solvers.ode as ode
```

```python
Newton2 = Eq(E + cross(v, B), v.diff(t) ); Newton2
divide_Array_Eq(Newton2)
```

```python
from sympy import simplify, sqrt
v_sol = ode.systems.dsolve_system(
    divide_Array_Eq(Newton2))
```

```python
v_sol[0][0]; v_sol[0][1]; v_sol[0][2]
```

```python
# The substitutions are for E=[E_x, E_y, E_z], B=[B_x, B_y, B_z] configurations.
from sympy import cos
B_amp, E_amp,theta = symbols("B_{amp}, E_{amp}, theta", real=True)
v_simp_sol = [ [comp_sol\
                .subs(sqrt(B_x**2 + B_y**2 + B_z**2), B_amp)\
                .subs(B_x**2 + B_y**2 + B_z**2, B_amp**2)\
                .subs(B_x*E_x + B_y*E_y + B_z*E_z, B_amp*E_amp*cos(theta)) for comp_sol in v_sol[0]] ]
```

```python
# v_simp_sol = [ [comp_sol.simplify() for comp_sol in v_sol[0] ] ]
for i in range(3): v_simp_sol[0][i]
```

### Time Averaging

```python
from sympy import simplify, trigsimp
T = pick_min_period(sol[0][0].rhs, t)
t_avg_EqSol = lambda eq: Eq(
    time_average_of_TrigFunc(eq.lhs, t, T=T),
    time_average_of_TrigFunc(eq.rhs, t, T=T))

t_avg_EqSol(sol[0][0])
t_avg_EqSol(sol[0][1])
t_avg_EqSol(sol[0][2])
print("The above formulas look too cumbersome, let's simplify them.")
from sympy import Q, refine
refine(trigsimp(t_avg_EqSol(sol[0][0]).simplify(), method="fu"), Q.positive(B_z))
refine(trigsimp(t_avg_EqSol(sol[0][1]).simplify(), method="fu"), Q.positive(B_z))
t_avg_EqSol(sol[0][2])
```

```python

```

```python

```

```python

```

```python
sol
```

## Fast Lambdify by Cupy

```python
import cupy as cp
```

## Bugs to be reported to Sympy

```python
from sympy import sin, sinh, pi, symbols
a = symbols("a", positive=True)
b, t = symbols("b, t", real=True)
sin(a*t).period(t)
```

$\displaystyle \frac{2 \pi}{a}$

```python
sin(-a*t).period(t) # AttributeError: 'Mul' object has no attribute 'period'
sin((-1)*a*t).period(t) # AttributeError: 'Mul' object has no attribute 'period'
```

```python
sin(b*t).period(t)
```

$\displaystyle \frac{2 \pi}{\left|{b}\right|}$

```python

```