# 13 - Panel Data and Fixed Effects

## Controlling What you Cannot See

Methods like propensity score, linear regression and matching are very good at controlling for confounding in non-random data, but they rely on a key assumption: conditional unconfoundedness

$
(Y_0, Y_1) \perp T | X
$

To put it in words, they require that all the confounders are known and measured, so that we can condition on them and make the treatment as good as random. One major issue with this is that sometimes we simply can't measure a confounder. For instance, take a classical labor economics problem of figuring out the impact of marriage on men's earnings. It's a well known fact in economics that married men earn more than single men. However, it is not clear if this relationship is causal or not. It could be that more educated men are both more likely to marry and more likely to have a high earnings job, which would mean that education is a confounder of the effect of marriage on earnings. For this confounder, we could measure the education of the person in the study and run a regression controlling for that. But another confounder could be beauty. It could be that more handsome men are both more likely to get married and more likely to have a high paying job. Unfortunately, beauty is one of those characteristics like intelligence. It's something we can't measure very well.

This puts us in a difficult situation, because if we have unmeasured confounders, we have bias. One way to deal with this is with instrumental variables, as we've seen before. But coming up with good instruments it's no easy task and requires a lot of creativity. Here, let's look at an alternative that takes advantage of time or the temporal structure of data.

The idea is to use **panel data**. Panel data is when we have **observations on the same individual for multiple periods of time**. Panel data formats are very common in the industry, where they keep records of customer behavior for the same customer and for multiple time periods. The reason we can leverage panel data is because we can compare the same unit before and after the treatment and see how they behave with it. Before we dive in the math, let's see how this makes intuitive sense.

```python
import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
from matplotlib import style
from matplotlib import pyplot as plt
import statsmodels.formula.api as smf
import graphviz as gr

from linearmodels.panel import PanelOLS

%matplotlib inline
pd.set_option("display.max_columns", 6)
style.use("fivethirtyeight")
```

First, let's take a look at the causal graph that we have once we include multiple observations of the same unit across time. Suppose we have a situation where marriage at the first time causes income at the same time and subsequent marital status. This is also true for times 2 and 3. Also, suppose that beauty is the same across all time periods (a bold statement, but reasonable if time is just a few years) and it causes both marriage and income.

```python
g = gr.Digraph()
g.edge("Marriage_1", "Income_1")
g.edge("Marriage_1", "Marriage_2")
g.edge("Marriage_2", "Income_2")
g.edge("Marriage_2", "Marriage_3")
g.edge("Marriage_3", "Income_3")

g.edge("Beauty", "Marriage_1")
g.edge("Beauty", "Marriage_2")
g.edge("Beauty", "Marriage_3")

g.edge("Beauty", "Income_1")
g.edge("Beauty", "Income_2")
g.edge("Beauty", "Income_3")

g
```

Remember that we cannot control beauty, since we can't measure it. But we can still use the panel structure so it is not a problem anymore. The idea is that we can see beauty - and any other attribute that is constant across time - as the defining aspects of a person. And although we can't control them directly, we can control for the individual itself.

```python
g = gr.Digraph()
g.edge("Marriage_1", "Income_1")
g.edge("Marriage_1", "Marriage_2")
g.edge("Marriage_2", "Income_2")
g.edge("Marriage_2", "Marriage_3")
g.edge("Marriage_3", "Income_3")

g.edge("Person (Beauty, Intelligence...)", "Marriage_1")
g.edge("Person (Beauty, Intelligence...)", "Marriage_2")
g.edge("Person (Beauty, Intelligence...)", "Marriage_3")

g.edge("Person (Beauty, Intelligence...)", "Income_1")
g.edge("Person (Beauty, Intelligence...)", "Income_2")
g.edge("Person (Beauty, Intelligence...)", "Income_3")

g
```

Think about it. We can't measure attributes like beauty and intelligence, but we know that the person who has them is the same individual across time. So, we can create a dummy variable indicating that person and add that to a linear model. This is what we mean when we say we can control for the person itself: we are adding a variable (dummy in this case) that denotes that particular person. When estimating the effect of marriage on income with this person dummy in our model, regression finds the effect of marriage **while keeping the person variable fixed**. Adding this entity dummy is what we call a fixed effect model.

## Fixed Effects

To make matters more formal, let's first take a look at the data that we have. Following our example, we will try to estimate the effect of marriage on income. Our data contains those 2 variables, `married` and `lwage`, on multiple individuals (`nr`) for multiple years. Note that wage is in log form. In addition to this, we have other controls, like number of hours worked that year, years of education and so on.

```python
from linearmodels.datasets import wage_panel
data = wage_panel.load()
data.head()
```

      nr
      year
      black
      ...
      lwage
      expersq
      occupation

      0
      13
      1980
      0
      ...
      1.197540
      1
      9

      1
      13
      1981
      0
      ...
      1.853060
      4
      9

      2
      13
      1982
      0
      ...
      1.344462
      9
      9

      3
      13
      1983
      0
      ...
      1.433213
      16
      9

      4
      13
      1984
      0
      ...
      1.568125
      25
      5

5 rows × 12 columns

Generally, the fixed effect model is defined as

$
y_{it} = \beta X_{it} + \gamma U_i + e_{it}
$

where \\(y_{it}\\) is the outcome of individual \\(i\\) at time \\(t\\), \\(X_{it}\\) is the vector of variables for individual \\(i\\) at time \\(t\\). \\(U_i\\) is a set of unobservables for individual \\(i\\). Notice that those unobservables are unchanging through time, hence the lack of the time subscript. Finally,  \\(e_{it}\\) is the error term. For the education example, \\(y_{it}\\) is log wages,  \\(X_{it}\\) are the observable variables that change in time, like marriage and experience and \\(U_i\\) are the variables that are not observed but constant for each individual, like beauty and intelligence.

Now, remember how I've said that using panel data with a fixed effect model is as simple as adding a dummy for the entities. It's true, but in practice, we don't actually do it. Imagine a dataset where we have 1 million customers. If we add one dummy for each of them, we would end up with 1 million columns, which is probably not a good idea. Instead, we use the trick of partitioning the linear regression into 2 separate models. We've seen this before, but now is a good time to recap it. Suppose you have a linear regression model with a set of features \\(X_1\\) and another set of features \\(X_2\\).

$
\hat{Y} = \hat{\beta_1} X_1 + \hat{\beta_2} X_2
$

where \\(X_1\\) and \\(X_1\\) are feature matrices (one row per feature and one column per observation) and \\(\hat{\beta_1}\\) and \\(\hat{\beta_2}\\) are row vectors. You can get the exact same \\(\hat{\beta_1}\\) parameter by doing

1. regress the the outcome \\(y\\) on the second set of features \\(\hat{y^*} = \hat{\gamma_1} X_2\\)
2. regress the first set of features on the second \\(\hat{X_1} = \hat{\gamma_2} X_2\\)
3. obtain the residuals \\(\tilde{X}_1 = X_1 - \hat{X_1}\\) and \\(\tilde{y}_1 = y_1 - \hat{y^*}\\)
4. regress the residuals of the outcome on the residuals of the features \\(\hat{y} = \hat{\beta_1} \tilde{X}_1\\)

The parameter from this last regression will be exactly the same as running the regression with all the features. But how exactly does this help us? Well, we can break the estimation of the model with the entity dummies into 2. First, we use the dummies to predict the outcome and the feature. These are steps 1 and 2 above.

Now, remember how running a regression on a dummy variable is as simple as estimating the mean for that dummy? If you don't, let's use our data to show how this is true. Let's run a model where we predict wages as a function of the year dummy.

```python
mod = smf.ols("lwage ~ C(year)", data=data).fit()
mod.summary().tables[1]
```

         	            coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	           1.3935	     0.022	    63.462	  0.000	     1.350	     1.437

  C(year)[T.1981]	     0.1194	     0.031	     3.845	  0.000	     0.059	     0.180

  C(year)[T.1982]	     0.1782	     0.031	     5.738	  0.000	     0.117	     0.239

  C(year)[T.1983]	     0.2258	     0.031	     7.271	  0.000	     0.165	     0.287

  C(year)[T.1984]	     0.2968	     0.031	     9.558	  0.000	     0.236	     0.358

  C(year)[T.1985]	     0.3459	     0.031	    11.140	  0.000	     0.285	     0.407

  C(year)[T.1986]	     0.4062	     0.031	    13.082	  0.000	     0.345	     0.467

  C(year)[T.1987]	     0.4730	     0.031	    15.232	  0.000	     0.412	     0.534

Notice how this model is predicting the average income in 1980 to be 1.3935, in 1981 to be 1.5129 (1.3935+0.1194) and so on. Now, if we compute the average by year, we get the exact same result. (Remember that the base year, 1980, is the intercept. So you have to add the intercept to the parameters of the other years to get the mean `lwage` for the year).

```python
data.groupby("year")["lwage"].mean()
```

    year
    1980    1.393477
    1981    1.512867
    1982    1.571667
    1983    1.619263
    1984    1.690295
    1985    1.739410
    1986    1.799719
    1987    1.866479
    Name: lwage, dtype: float64

This means that if we get the average for every person in our panel, we are essentially regressing the individual dummy on the other variables. This motivates the following estimation procedure:

1. Create time-demeaned variables by subtracting the mean for the individual:
$\ddot{Y}_{it} = Y_{it} -  \bar{Y}_i$
$\ddot{X}_{it} = X_{it} -  \bar{X}_i$

2. Regress $\ddot{Y}_{it}$ on $\ddot{X}_{it}$

Notice that when we do so, the unobserved \\(U_i\\) vanishes. Since \\(U_i\\) is constant across time, we have that \\(\bar{U_i}=U_i\\). If we have the following system of two equations

$$
\begin{align}
Y_{it} & = \beta X_{it} + \gamma U_i + e_{it} \\
\bar{Y}_{i} & = \beta \bar{X}_{it} + \gamma \bar{U}_i + \bar{e}_{it} \\
\end{align}
$$

And we subtract one from the other, we get

$$
\begin{align}
(Y_{it} - \bar{Y}_{i}) & = (\beta X_{it} - \beta \bar{X}_{it}) + (\gamma U_i - \gamma U_i) + (e_{it}-\bar{e}_{it}) \\
(Y_{it} - \bar{Y}_{i}) & = \beta(X_{it} - \bar{X}_{it}) + (e_{it}-\bar{e}_{it}) \\
\ddot{Y}_{it} & = \beta \ddot{X}_{it} + \ddot{e}_{it} \\
\end{align}
$$

which wipes out all unobserved that are constant across time. To be honest, not only do the unobserved variables vanish. This happens to all the variables that are constant in time. For this reason, you can't include any variables that are constant across time, as they would be a linear combination of the dummy variables and the model wouldn't run.

To check which variables are those, we can group our data by individual and get the sum of the standard deviations. If it is zero, it means the variable isn't changing across time for any of the individuals.

```python
data.groupby("nr").std().sum()
```

    year            1334.971910
    black              0.000000
    exper           1334.971910
    hisp               0.000000
    hours         203098.215649
    married          140.372801
    educ               0.000000
    union            106.512445
    lwage            173.929670
    expersq        17608.242825
    occupation       739.222281
    dtype: float64

For our data, we need to remove entinicity dummies, `black` and `hisp`, since they are constant for the individual. Also, we need to remove education. We will also not use occupation, since this is probably mediating the effect of marriage on wage (it could be that single men are able to take more time demanding positions). Having selected the features we will use, it's time to estimate this model.

To run our fixed effect model, first, let's get our mean data. We can achieve this by grouping everything by individuals and taking the mean.

```python
Y = "lwage"
T = "married"
X = [T, "expersq", "union", "hours"]

mean_data = data.groupby("nr")[X+[Y]].mean()
mean_data.head()
```

      married
      expersq
      union
      hours
      lwage

      nr

      13
      0.000
      25.5
      0.125
      2807.625
      1.255652

      17
      0.000
      61.5
      0.000
      2504.125
      1.637786

      18
      1.000
      61.5
      0.000
      2350.500
      2.034387

      45
      0.125
      35.5
      0.250
      2225.875
      1.773664

      110
      0.500
      77.5
      0.125
      2108.000
      2.055129

To demean the data, we need to set the index of the original data to be the individual identifier, `nr`. Then, we can simply subtract one data frame from the mean data frame.

```python
demeaned_data = (data
               .set_index("nr") # set the index as the person indicator
               [X+[Y]]
               - mean_data) # subtract the mean data

demeaned_data.head()
```

      married
      expersq
      union
      hours
      lwage

      nr

      13
      0.0
      -24.5
      -0.125
      -135.625
      -0.058112

      13
      0.0
      -21.5
      0.875
      -487.625
      0.597408

      13
      0.0
      -16.5
      -0.125
      132.375
      0.088810

      13
      0.0
      -9.5
      -0.125
      152.375
      0.177561

      13
      0.0
      -0.5
      -0.125
      263.375
      0.312473

Finally, we can run our fixed effect model on the time-demeaned data.

```python
mod = smf.ols(f"{Y} ~ {'+'.join(X)}", data=demeaned_data).fit()
mod.summary().tables[1]
```

      	         coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	 -8.327e-17	     0.005	 -1.64e-14	  1.000	    -0.010	     0.010

  married	       0.1147	     0.017	     6.756	  0.000	     0.081	     0.148

  expersq	       0.0040	     0.000	    21.958	  0.000	     0.004	     0.004

  union	         0.0784	     0.018	     4.261	  0.000	     0.042	     0.115

  hours	      -8.46e-05	  1.25e-05	    -6.744	  0.000	    -0.000	    -6e-05

If we believe that fixed effect eliminates the all omitted variable bias, this model is telling us that marriage increases a man's wage by 11%. This result is very significant. One detail here is that for fixed effect models, the standard errors need to be clustered. So, instead of doing all our estimation by hand (which is only nice for pedagogical reasons), we can use the library `linearmodels` and set the argument `cluster_entity` to True.

```python
from linearmodels.panel import PanelOLS
mod = PanelOLS.from_formula("lwage ~ expersq+union+married+hours+EntityEffects",
                            data=data.set_index(["nr", "year"]))

result = mod.fit(cov_type='clustered', cluster_entity=True)
result.summary.tables[1]
```

Parameter Estimates

     	     Parameter	 Std. Err.	 T-stat	  P-value	 Lower CI	  Upper CI

  expersq	  0.0040	    0.0002	   16.552	  0.0000	   0.0035	    0.0044

  union	    0.0784	    0.0236	   3.3225	  0.0009	   0.0322	    0.1247

  married	  0.1147	    0.0220	   5.2213	  0.0000	   0.0716	    0.1577

  hours	   -8.46e-05	 2.22e-05	  -3.8105	 0.0001	   -0.0001	 -4.107e-05

Notice how the parameter estimates are identical to the ones we've got with time-demeaned data. The only difference is that the standard errors are a bit larger. Now, compare this to the simple OLS model that doesn't take the time structure of the data into account. For this model, we add back the variables that are constant in time.

```python
mod = smf.ols("lwage ~ expersq+union+married+hours+black+hisp+educ", data=data).fit()
mod.summary().tables[1]
```

      	         coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	     0.2654	     0.065	     4.103	  0.000	     0.139	     0.392

  expersq	       0.0032	     0.000	    15.750	  0.000	     0.003	     0.004

  union	         0.1829	     0.017	    10.598	  0.000	     0.149	     0.217

  married	       0.1410	     0.016	     8.931	  0.000	     0.110	     0.172

  hours	      -5.32e-05	  1.34e-05	    -3.978	  0.000	 -7.94e-05	  -2.7e-05

  black	        -0.1347	     0.024	    -5.679	  0.000	    -0.181	    -0.088

  hisp	          0.0132	     0.021	     0.632	  0.528	    -0.028	     0.054

  educ	          0.1057	     0.005	    22.550	  0.000	     0.097	     0.115

This model is saying that marriage increases the man's wage by 14%. A somewhat larger effect than the one we found with the fixed effect model. This suggests some omitted variable bias due to fixed individual factors, like intelligence and beauty, not being added to the model.

## Visualizing Fixed Effects

To expand our intuition about how fixed effect models work, let's diverge a little to another example. Suppose you work for a big tech company and you want to estimate the impact of a billboard marketing campaign on in-app purchase. When you look at data from the past, you see that the marketing department tends to spend more to place billboards on cities where the purchase level is lower. This makes sense right? They wouldn't need to do lots of advertisement if sales were skyrocketing. If you run a regression model on this data, it looks like higher cost in marketing leads to less in-app purchase, but only because marketing investments is biased towards low spending regions.

```python
toy_panel = pd.DataFrame({
    "mkt_costs":[5,4,3.5,3, 10,9.5,9,8, 4,3,2,1, 8,7,6,4],
    "purchase":[12,9,7.5,7, 9,7,6.5,5, 15,14.5,14,13, 11,9.5,8,5],
    "city":["C0","C0","C0","C0", "C2","C2","C2","C2", "C1","C1","C1","C1", "C3","C3","C3","C3"]
})

m = smf.ols("purchase ~ mkt_costs", data=toy_panel).fit()

plt.scatter(toy_panel.mkt_costs, toy_panel.purchase)
plt.plot(toy_panel.mkt_costs, m.fittedvalues, c="C5", label="Regression Line")
plt.xlabel("Marketing Costs (in 1000)")
plt.ylabel("In-app Purchase (in 1000)")
plt.title("Simple OLS Model")
plt.legend();
```

Knowing a lot about causal inference, you decide to run a fixed effect model, adding the city's indicator as a dummy variable to your model. The fixed effect model controls for city specific characteristics that are constant in time, so if a city is less open to your product, it will capture that. When you run that model, you can finally see that more marketing costs leads to higher in-app purchase.

```python
fe = smf.ols("purchase ~ mkt_costs + C(city)", data=toy_panel).fit()

fe_toy = toy_panel.assign(y_hat = fe.fittedvalues)

plt.scatter(toy_panel.mkt_costs, toy_panel.purchase, c=toy_panel.city)
for city in fe_toy["city"].unique():
    plot_df = fe_toy.query(f"city=='{city}'")
    plt.plot(plot_df.mkt_costs, plot_df.y_hat, c="C5")

plt.title("Fixed Effect Model")
plt.xlabel("Marketing Costs (in 1000)")
plt.ylabel("In-app Purchase (in 1000)");
```

Take a minute to appreciate what the image above is telling you about what fixed effect is doing. Notice that fixed effect is fitting **one regression line per city**. Also notice that the lines are parallel. The slope of the line is the effect of marketing costs on in-app purchase. So the **fixed effect is assuming that the causal effect is constants across all entities**, which are cities in this case. This can be a weakness or an advantage, depending on how you see it. It is a weakness if you are interested in finding the causal effect per city. Since the FE model assumes this effect is constant across entities, you won't find any difference in the causal effect. However, if you want to find the overall impact of marketing on in-app purchase, the panel structure of the data is a very useful leverage that fixed effects can explore.

## Time Effects

Just like we did a fixed effect for the individual level, we could design a fixed effect for the time level. If adding a dummy for each individual controls for fixed individual characteristics, adding a time dummy would control for variables that are fixed for each time period, but that might change across time. One example of such a variable is inflation. Prices and salary tend to go up with time, but the inflation on each time period is the same for all entities. To give a more concrete example, suppose that marriage is increasing with time. If the wage and marriage proportion also changes with time, we would have time as a confounder. Since inflation also makes salary increase with time, some of the positive association we see between marriage and wage would be simply because both are increasing with time. To correct for that, we can add a dummy variable for each time period. In `linear models`, this is as simple as adding `TimeEffects` to our formula and setting the `cluster_time` to true.

```python
mod = PanelOLS.from_formula("lwage ~ expersq+union+married+hours+EntityEffects+TimeEffects",
                            data=data.set_index(["nr", "year"]))

result = mod.fit(cov_type='clustered', cluster_entity=True, cluster_time=True)
result.summary.tables[1]
```

Parameter Estimates

     	     Parameter	 Std. Err.	 T-stat	  P-value	 Lower CI	  Upper CI

  expersq	  -0.0062	   0.0008	   -8.1479	 0.0000	   -0.0077	   -0.0047

  union	    0.0727	    0.0228	   3.1858	  0.0015	   0.0279	    0.1174

  married	  0.0476	    0.0177	   2.6906	  0.0072	   0.0129	    0.0823

  hours	    -0.0001	  3.546e-05	 -3.8258	 0.0001	   -0.0002	 -6.614e-05

In this new model, the effect of marriage on wage decreased significantly from `0.1147` to `0.0476`. Still, this result is significant at a 99% level, so man could still expect an increase in earnings from marriage.

## When Panel Data Won't Help You

Using panel data and fixed effects models is an extremely powerful tool for causal inference. When you don't have random data nor good instruments, the fixed effect is as convincing as it gets for causal inference with non experimental data. Still, it is worth mentioning that it is not a panacea. There are situations where even panel data won't help you.

The most obvious one is when you have confounders that are changing in time. Fixed effects can only eliminate bias from attributes that are constant for each individual. For instance, suppose that you can increase your intelligence level by reading books and eating lots of good fats. This causes you to get a higher paying job and a wife. Fixed effect won't be able to remove this bias due to unmeasured intelligence confounding because, in this example, intelligence is changing in time.

Another less obvious case when fixed effect fails is when you have **reversed causality**. For instance, let's say that it isn't marriage that causes you to earn more. Is earning more that increases your chances of getting married. In this case, it will appear that they have a positive correlation but earnings come first. They would change in time and in the same direction, so fixed effects wouldn't be able to control for that.

## Key Ideas

Here, we saw how to use panel data, data where we have multiple measurements of the same individuals across multiple time periods. When that is the case, we can use a fixed effect model that controls for the entity, holding all individual, time constant attributes, fixed. This is a powerful and very convincing way of controlling for confounding and it is as good as it gets with non random data.

Finally, we saw that FE is not a panacea. We saw two situations where it doesn't work: when we have reverse causality and when the unmeasured confounding is changing in time.

## References

I like to think of this entire book as a tribute to Joshua Angrist, Alberto Abadie and Christopher Walters for their amazing Econometrics class. Most of the ideas here are taken from their classes at the American Economic Association. Watching them is what is keeping me sane during this tough year of 2020.
* [Cross-Section Econometrics](https://www.aeaweb.org/conference/cont-ed/2017-webcasts)
* [Mastering Mostly Harmless Econometrics](https://www.aeaweb.org/conference/cont-ed/2020-webcasts)

I'll also like to reference the amazing books from Angrist. They have shown me that Econometrics, or 'Metrics as they call it, is not only extremely useful but also profoundly fun.

* [Mostly Harmless Econometrics](https://www.mostlyharmlesseconometrics.com/)
* [Mastering 'Metrics](https://www.masteringmetrics.com/)

Another important reference is Miguel Hernan and Jamie Robins' book. It has been my trustworthy companion in the most thorny causal questions I had to answer.

* [Causal Inference Book](https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/)

Finally, I'd also like to compliment Scott Cunningham and his brilliant work mingling Causal Inference and Rap quotes:

* [Causal Inference: The Mixtape](https://www.scunning.com/mixtape.html)

## Contribute

Causal Inference for the Brave and True is an open-source material on causal inference, the statistics of science. It uses only free software, based in Python. Its goal is to be accessible monetarily and intellectually.
If you found this book valuable and you want to support it, please go to [Patreon](https://www.patreon.com/causal_inference_for_the_brave_and_true). If you are not ready to contribute financially, you can also help by fixing typos, suggesting edits or giving feedback on passages you didn't understand. Just go to the book's repository and [open an issue](https://github.com/matheusfacure/python-causality-handbook/issues). Finally, if you liked this content, please share it with others who might find it useful and give it a [star on GitHub](https://github.com/matheusfacure/python-causality-handbook/stargazers).

```python

```


```python
import sympy as sym
import numpy as np
from exFinder.extremeValues import exFinder
x, y, z= sym.symbols('x y z', real=True)
```

# Symbolische Berechnung von Extremwerten: Sympy und Numpy
Dr. Lauer-Baré, *exfinder* funktioniert für Funktionen die von $\mathbb{R}^2$ nach $\mathbb{R}$ abbilden, die Variablen sollen $x,y$ heißen.
*exfinder* findet auf symbloische Art und Weise für eine Funktion die stationären Punkte, die Eigenwerte der entsprechenden Hessematrix und die Interpretation des stationären Punktes.

```python
f=(x-2)**4+(x-2*y)**2

stsf=exFinder(f)

#stsf[1]
```

    function:
    (x - 2)**4 + (x - 2*y)**2

    (2, 1) {10: 1, 0: 1} not classifiable

```python
f=y**2*(x-1)+x**2*(x+1)

stsf=exFinder(f)

#stsf[1]
```

    function:
    x**2*(x + 1) + y**2*(x - 1)

    (-2/3, 0) {-2: 1, -10/3: 1} max
    (0, 0) {2: 1, -2: 1} saddle

```python

```


# $$.9\overline{9} = 1?$$

# Why every proof you have seen is WRONG

Don't worry, **it is true**. Just the "proofs" you've seen are wrong.

# First, why the common "proofs" are wrong
# Then, an actual proof

# Algebraic "proofs" are ALL wrong

$$
\begin{align}
x &:= .9\overline{9} &&&1/3 &= .3\overline{3} \\
10x &= 9.9\overline{9} &\text {or}&& 3\cdot (1/3) &= 3\cdot(.3\overline{3})\\
9x &= 9 &&& 1 &= .9\overline{9}\\
x &= 1
\end{align}
$$
All algebraic proofs are wrong because there are algebras where $.9\overline{9} \neq 1$.

$$
\begin{align}
x &:= \bf\color{red}{.9\overline{9}} &&&1/3 &= \bf\color{red}{.3\overline{3}} \\
10x &= 9.9\overline{9} &\text {or}&& 3\cdot (1/3) &= 3\cdot(.3\overline{3})\\
9x &= 9 &&& 1 &= .9\overline{9}\\
x &= 1
\end{align}
$$
You have already assumed part of the conclusion, that $\color{red}{.9\overline{9}}$ or $\color{red}{.3\overline{3}}$ even *exists*.

$$
\begin{align}
x &:= \bf\color{red}{\overline{9}.} \\
10x &= \overline{9}0.  \\
10x+9 &= x\\
9x &= -9\\
x &= -1
\end{align}
$$
So do you believe that $...99999999.0 = -1$?

$$\color{green}{
\begin{align}
\color{black}{x} &\color{black}{:= .9\overline{9}} &&&1/3 &= .3\overline{3} \\
10x &= 9.9\overline{9} &\color{black}{\text {or}}&& 3\cdot (1/3) &= 3\cdot(.3\overline{3})\\
9x &= 9 &&& 1 &= .9\overline{9}\\
x &= 1
\end{align}}
$$
Okay, suppose you are willing to believe $.9\overline{9}$ is a real number, what else is wrong?

$$
\begin{align}
&&x &:= .9\overline{9} &&&&&1/3 &= .3\overline{3} \\
\implies&& 10x &= 9.9\overline{9} &\text {or}&&&\implies& 3\cdot (1/3) &= 3\cdot(.3\overline{3})\\
\implies&&9x &= 9 &&&&\implies& 1 &= .9\overline{9}\\
\implies&& x &= 1
\end{align}
$$

$$
\begin{align}
&&x &:= .9\overline{9} &&&&&1/3 &= .3\overline{3} \\
\color{red}{\implies}&& 10x &= 9.9\overline{9} &\text {or}&&&\implies& 3\cdot (1/3) &= 3\cdot(.3\overline{3})\\
\implies&&9x &= 9 &&&&\bf\color{red}{\implies}& 1 &= .9\overline{9}\\
\implies&& x &= 1
\end{align}
$$

*Why* does $10 \cdot (.9\overline{9}) = 9.9\overline{9}$?
*Why* does $3 \cdot (.3\overline{3}) = .9\overline{9}$?

$$
\begin{align}
x &:= .9\overline{9} &&&1/3 &\color{red}{=} .3\overline{3} \\
10x &= 9.9\overline{9} &\text {or}&& 3\cdot (1/3) &= 3\cdot(.3\overline{3})\\
9x &= 9 &&& 1 &= .9\overline{9}\\
x &= 1
\end{align}
$$
Isn't $1/3 \color{red}{=} .3\overline{3}$ just as mysterious? Why believe it?

# The real source of confusion: definition

What does $.9\overline{9}$ actually mean?

It is **NOT** a mystical infinite sequence of $9$'s after a decimal point.

The definition of $.9\overline{9}$:
$$
\begin{align}
x_1 &:= .9,\\
x_2 &:= .99,\\
x_3 &:= .999, \ldots \\
x_n &:= \sum_{i=1}^n 9 \cdot 10^{-i} = \underbrace{.9999}_{n}.
\end{align}
$$

Then $.9\overline{9}$ is **defined** to be the number that the sequence $$
(x_n)_n = (.9, .99, .999, \cdots)
$$ approaches, **if it exists**. That is, it is by definition the limit of the sequence.

For any repeating decimal, we use the same process to define it.

E.g. for $2.64\overline{51}$ you would use the sequence
$$
(2.645, 2.6451, 2.64515, 2.645151, 2.6451515, \cdots).
$$

# So what is $.9\overline{9} = 1$ even claiming?

It is simply claiming that $$(.9, .99, .999, \cdots)$$ does approach a real number, and that number is $1$.

```python
import numpy as np
import matplotlib.pyplot as plt

N = np.arange(1, 10)
X = np.cumsum(9 * np.power(10.,-N))

plt.axhline(1.0, color="grey", linestyle="dotted")
plt.plot(N, X, marker="x");
```

# Proof (trivial)

Compute
$$
|x_n - 1| = |  \underbrace{.9999}_{n} - 1| = 10^{-n}.
$$
For any tolerance $\epsilon>0$, for any $n > 1/\epsilon$, we have
$$
|x_n - 1| = 10^{-n} < 1/n < \epsilon.
$$
This formally shows that $x_n$ approaches 1, but you already knew that.

# Honorable mentions

# Thanks!

```python
like()
comment()
subscribe()
while True:
    try:
        tell_friends()
    except KeyError:
        tell_family()
    await asyncio.sleep(60*60*24)
```


# Artificial Neural Network (ANN) model

This is the code for the completed neural network. Throughout this document all the areas are explained in more detail.

```python
class NeuralNetwork(object):
    def __init__(self):
        # Define Hyperparameters
        self.inputLayerSize = 2
        self.hiddenLayerSize = 3
        self.outputLayerSize = 1

        # Weights (parameters)
        self.W1 = np.random.randn(self.inputLayerSize, self.hiddenLayerSize)
        self.W2 = np.random.randn(self.hiddenLayerSize, self.outputLayerSize)

    def forwardPropagation(self, X):
        # Propagate inputs through network
        self.z2 = np.dot(X, self.W1)
        self.a2 = self.sigmoid(self.z2)
        self.z3 = np.dot(self.a2, self.W2)
        yHat = self.sigmoid(self.z3)
        return yHat

    def sigmoid(self, z):
        return 1/(1+np.exp(-z))

    def sigmoidPrime(self, z):
        return np.exp(-z)/((1+np.exp(-z))**2)
```

## Problem

Suppose we want to predict our test score based on how many hours we sleep and how many hours we study the night before. In other words, we want to predict the output value $y$ which are scores for a set of input values $X$ which are hours of (sleep, study).

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
import numpy as np
```

```python
# X = (hours sleeping, hours studying), y = Score on test
X = np.array(([3,5], [5,1], [10,2]), dtype=float)
y = np.array(([75], [82], [93]), dtype=float)
```

```python
X
```

    array([[  3.,   5.],
           [  5.,   1.],
           [ 10.,   2.]])

```python
y
```

    array([[ 75.],
           [ 82.],
           [ 93.]])

This is a supervised regression problem. It's supervised because our examples have outputs($y$). It's regression because we're predicting the the test score, which is a continuous output.

We want to scale the data so the result is in the interval $[0,1]$.

```python
X = X/np.amax(X, axis=0)
y = y/100 # Max test score is 100
```

```python
X
```

    array([[ 0.3,  1. ],
           [ 0.5,  0.2],
           [ 1. ,  0.4]])

```python
y
```

    array([[ 0.75],
           [ 0.82],
           [ 0.93]])

Now we can start building the neural network. It will have 2 inputs($X$) and 1 output($y$). We call our output $\hat{y}$ because it is an estimate of $y$. We will be using a hidden later with 3 neurons. Finally, we will be using sigmoid activation functions.

## Forward Propagation

### Variables

|Code symbol|Math symbol|Definition|Dimensions|
|--|--|--|--|
|X|$X$|Input Data, each row in an example|(numExamples, inputLayerSize)|
|y|$y$|Target data|(numExamples, outputLayerSize)|
|W1|$W^{(1)}$|Layer 1 Weights|(inputLayerSize, hiddenLayerSize)|
|W2|$W^{(2)}$|Layer 2 Weights|(hiddenLayerSize, outputLayerSize)|
|z2|$z^{(2)}$|Layer 2 Activation|(numExamples, hiddenLayerSize)|
|a2|$z^{(2)}$|Layer 2 Activity|(numExamples, hiddenLayerSize)|
|z3|$z^{(2)}$|Layer 3 Activation|(numExamples, outputLayerSize)|

$$
\begin{align}
z^{(2)} &= XW^{(1)} \\
a^{(2)} &= f(z^{(2)}) \\
z^{(3)} &= a^{(3)}W^{(2)} \\
\hat{y} &= f(z^{(3)})
\end{align}
$$

Each input value in matrix $X$ should be multiplied by a corresponding weight and then added together with all the other results for each neuron.

$z^{(2)}$ is the activity of our second layer and it can be calculated as the following:

$$
z^{(2)} = XW^{(1)} = \begin{bmatrix}
        3 & 5  \\
        5 & 1  \\
        10 & 2
\end{bmatrix}
\begin{bmatrix}
        W_{11}^{(1)} & W_{12}^{(1)}  & W_{13}^{(1)}\\
        W_{21}^{(1)} & W_{22}^{(1)}  & W_{23}^{(1)}
\end{bmatrix} = \begin{bmatrix}
3 W_{11}^{(1)} + 5 W_{21}^{(1)} & 3 W_{12}^{(1)} + 5 W_{22}^{(1)} & 3 W_{13}^{(1)} + 5 W_{23}^{(1)} \\
5 W_{11}^{(1)} +  W_{21}^{(1)} & 5 W_{12}^{(1)} +  W_{22}^{(1)} & 5 W_{13}^{(1)} + W_{23}^{(1)} \\
10 W_{11}^{(1)} + 2 W_{21}^{(1)} & 10 W_{12}^{(1)} + 2 W_{22}^{(1)} & 10 W_{13}^{(1)} + 2 W_{23}^{(1)}
\end{bmatrix}
$$

Note that each entry in $z$ is a sum of weighted inputs to each hidden neuron. $z$ is a $3\times 3$ matrix, one row for each sample, and one column for each hidden unit.

### Activation function - Sigmoid

Now that we have the activities for our second layer, $z^{(2)} = XW^{(1)}$, we need to apply the activation function. We'll independently apply the sigmoid function to each entry in the matrix $z$.

```python
NN = NeuralNetwork()
testInput = np.arange(-6,6,0.01)
plot(testInput, NN.sigmoid(testInput), color='b', lineWidth=2)
grid(1)
```

Let's see how the sigmoid() takes an input and returns the result:

```python
NN.sigmoid(1)
```

    0.7310585786300049

```python
NN.sigmoid(np.array([-1,0,1]))
```

    array([ 0.26894142,  0.5       ,  0.73105858])

```python
NN.sigmoid(np.random.randn(3,3))
```

    array([[ 0.21802553,  0.76343508,  0.56604093],
           [ 0.80182811,  0.11230011,  0.32035148],
           [ 0.50937793,  0.47449956,  0.55781643]])

### Weight-matrices $W^{(1)}$ and $W^{(2)}$

These are initialized in the \__init__\() method with random numbers.

### Implementing forward propagation

Using our activation function $f$, we can write that our second layer activity $a^{(2)} = f(z^{(2)})$. The $a^{(2)}$ will be a matrix of the same size ($3 \times 3$).

To finish forward propagation we want to propagate $a^{(2)}$ all the way to the output $\hat{y}$.

All we have to do now is multiply $a^{(2)}$ by our second layer weights $W^{(2)}$ and apply on more activation function. The $W^{(2)}$ will be of size $3 \times 1$, one weight for each synapse:

$$z^{(3)}=a^{(2)}W^{(2)}$$

Multiplying $a^{(2)}$, a ($3 \times 3$ matrix), by $W^{(2)}$, a ($3 \times 1$ matrix) results in a $3 \times 1$ matrix $z^{(3)}$, the activity of our 3rd layer. The $z^{(3)}$ has three activity values, one for each sample.

Then we'll apply our activation function to $z^{(3)}$ yielding our estimate of test score, $\hat{y}$:

$$\hat{y}=f(z^{(3)})$$

### Getting an estimate of test score

Now we have a class capable of estimating our test score given how many hours we sleep and how many hours we study.

```python
X
```

    array([[ 0.3,  1. ],
           [ 0.5,  0.2],
           [ 1. ,  0.4]])

```python
NN = NeuralNetwork()
yHat = NN.forwardPropagation(X)
```

```python
yHat
```

    array([[ 0.48121278],
           [ 0.35868875],
           [ 0.28271806]])

```python
y
```

    array([[ 0.75],
           [ 0.82],
           [ 0.93]])

```python
bar([0,1,2], y, width=0.35, alpha=0.8, color='b')
bar([0.35,1.35,2.35], yHat, width=0.35, color='r', alpha=0.8)
grid(1)
legend(['y', 'yHat']);
```

We can see that our predictions $\hat{y}$ are pretty inaccurate.

## Gradient Descent

### Cost function $J$

To improve our model, we need to find a way of quantifying exactly how wrong our predictions are. One way of doing it is to use a cost function. For a given sample, a cost function tells us how costly or model is.

We'll use sum of square errors to compute an overall cost and we'll try to minimize it. Actually, training a network means minimizing a cost functions:

$$J = \sum\limits_{i=1}^N(y_i-\hat{y}_i)$$

where $N$ is the number of training samples. We can make $J$ as small as possible with a optimal combination of the weights.

### Curse of dimensionality

Suppose we want to find the optimal weight value for one weight:

```python
import time
weightToTry = np.linspace(-5,5,1000)
costs = np.zeros(1000)

startTime = time.clock()
for i in range(1000):
    NN.W1[0,0] = weightToTry[i]
    yHat = NN.forwardPropagation(X)
    costs[i] = 0.5*sum((y-yHat)**2)
endTime = time.clock()
```

```python
elapsedTime = endTime - startTime
elapsedTime
```

    0.026310512359392284

It took about 0.03 seconds to check 1000 different weight values for our neural network.

Here is the plot for the 1000 weights:

```python
plot(weightToTry, costs, color='b')
grid(1)
xlabel('Cost')
ylabel('Weight')
```

If we want to optimize 2 weights it will take 1000\*1000 iterations to check all the values. If we want to check all the weights it will take:

```python
elapsedTime*(1000**(9-1))/(3600*24*365)/1000
```

    834300873902.5966

almost 1 trillion milleniums. Needless to say, this is infeasible.

### Gradient descent method

There are two methods for the gradient descent: batch (standard) or stochastic. We're going to use the batch to train our neural network.

In batch gradient descent method sums up all the derivatives of $J$ for all samples:

$$ \sum\frac{dJ}{dW} $$

## Backpropagation of Errors

Backpropagation (backward propagation of errors) is an algorithm used to train artificial neural networks, it can update the weights very efficiently.

Basically, backpropagation is just a very computationally efficient approach to compute the derivates of a complex cost function and our goal is to use those derivaties to determine the weight coefficients for parameterizing a multi-layer neural network.

In other words, the method calculates the gradient ($\frac{dJ}{dW}$) of a cost (loss or objective) function with respect to all the weights in the network, so that the gradient is fed to the gradient descent method which in turn uses it to update the weights in order to minimize the cost function.

Since backpropagation requires a known, target data for each input value in order to calculate the cost function gradient, it is usually used in supervised networks.

This will require additional variables, so our table now becomes:

|Code symbol|Math symbol|Definition|Dimensions|
|--|--|--|--|
|X|$X$|Input Data, each row in an example|(numExamples, inputLayerSize)|
|y|$y$|Target data|(numExamples, outputLayerSize)|
|W1|$W^{(1)}$|Layer 1 Weights|(inputLayerSize, hiddenLayerSize)|
|W2|$W^{(2)}$|Layer 2 Weights|(hiddenLayerSize, outputLayerSize)|
|z2|$z^{(2)}$|Layer 2 Activation|(numExamples, hiddenLayerSize)|
|a2|$z^{(2)}$|Layer 2 Activity|(numExamples, hiddenLayerSize)|
|z3|$z^{(2)}$|Layer 3 Activation|(numExamples, outputLayerSize)|
|J|$J$|Cost|(1, outputLayerSize)|
|dJdz3|$\frac{\partial J}{\partial z^{(3)}}$|Partial derivative of cost with respect to $z^{(3)}$|(numExamples, outputLayerSize)|
|dJdW2|$\frac{\partial J}{\partial W^{(2)}}$|Partial derivative of cost with respect to $W^{(2)}$|(hiddenLayerSize, outputLayerSize)|
|dz3dz2|$\frac{\partial z^{(3)}}{\partial z^{(2)}}$|Partial derivative of $z^{(3)}$ with respect to $z^{(2)}$|(numExamples, hiddenLayerSize)|
|dJdW1|$\frac{\partial J}{\partial W^{(1)}}$|Partial derivative of cost with respect to $W^{(1)}$|(inputLayerSize, hiddenLayerSize)|
|delta2|$\delta^{(2)}$|Backpropagating Error 2|(numExamples, hiddenLayerSize)|
|delta3|$\delta^{(3)}$|Backpropagating Error 3|(numExamples, outputLayerSize)|

### Computing gradient $\frac{dJ}{dW}$

We have a hidden layer and an output layer. So, we need to compute two gradients overall: $\frac{\partial J}{\partial W^{(1)}}$, and $\frac{\partial J}{\partial W^{(2)}}$, the gradient with respect to the weight for the hidden layer, and the gradient with respect to the weight for the output later, respectively.

A way of quantifying exactly how wrong (or correct) our predictions are is by using a cost function.

We'll use sum of square errors, the difference between target (known) data and the value estimated by our network to compute an overall cost and we'll try to minimize it. the cost function is:

$$J=\frac{1}{2}\sum\limits_{i=1}^N(y_i-\hat{y}_i)^2$$

where $N$ is the number of training samples. Here, the $J$ is the error of the network for a single training iteration. Note that $\sum$ is required for our batch gradient descent algorithm.

To perform gradient descent, we need an equation and some code for our gradient $\frac{dJ}{dW}$.

We'll seperate our $\frac{dJ}{dW}$ computation by computing $\frac{\partial J}{\partial W^{(1)}}$, and $\frac{\partial J}{\partial W^{(2)}}$ independently.

Let's work on $\frac{\partial J}{\partial W^{(2)}}$ first, which is for the output layer.

The sum in our cost function adds the error from each sample to create our overall cost. We'll take advantage of the sum rule in differentiation. We can move our $\sum$ outside and worry about the derivative of the inside expression first:

$$\frac{\partial J}{\partial W^{(2)}} = \frac{\partial(\sum \frac{1}{2}(y-\hat{y})^2)}{\partial W^{(2)}} = \sum \frac{\partial(\frac{1}{2}(y-\hat{y})^2)}{\partial W^{(2)}}$$

Unlike $\hat{y}$ which depends on $W^{(2)}$, $y$ is constant. So, $\frac{\partial y}{\partial W^{(2)}}=0$, and we have the following:

$$\frac{\partial J}{\partial W^{(2)}}=-\sum(y-\hat{y})\frac{\partial\hat{y}}{\partial W^{(2)}}$$

We now need to think about the derivative of $\frac{\partial\hat{y}}{\partial W^{(2)}}$. From our earlier equation $$\hat{y}=f(z^{(3)})$$

we know that $\hat{y}$ is our activation function of $z^{(3)}$, so we may want to apply the chain rule again to break $\frac{\partial\hat{y}}{\partial W^{(2)}}$ into $\frac{\partial\hat{y}}{\partial z^{(3)}}$ times $\frac{\partial z^{(3)}}{\partial W^{(2)}}$:

$$\frac{\partial J}{\partial W^{(2)}} = -\sum (y-\hat{y}) \frac{\partial\hat{y}}{\partial z^{(3)}} \frac{\partial z^{(3)}}{\partial W^{(2)}}$$.

To keep things simple, we'll drop our summation. Once we've computed $\frac{\partial J}{\partial W}$ for a single sample, we'll add up all our individual derivative terms.

To find the rate of change of $\hat{y}$ with respect to $z^{(3)}$, we need to differentiate our sigmoid activation function with respect to $z$:

$$f(z) = \frac {1}{1+e^{-z}}$$

$$f^\prime(z) = \frac {e^{-z}} {(1+e^{-z})^2}$$

### Code for sigmoid prime $f'(z)$

The code for the sigmoid prime function will be:

```python
def sigmoidPrime(self, z):
    return np.exp(-z)/((1+np.exp(-z))**2)
```

Now the function will look like this:

```python
NN = NeuralNetwork()
```

```python
sigTestValues = np.arange(-5,5,0.1)
plot(sigTestValues, NN.sigmoid(sigTestValues), linewidth=2, color='b')
plot(sigTestValues, NN.sigmoidPrime(sigTestValues), linewidth=2, color='r')
grid(1)
legend(['f', "f'"]);
```

### Backpropagation Errors ($\delta$)

http://www.bogotobogo.com/python/scikit-learn/Artificial-Neural-Network-ANN-4-Backpropagation.php

# ???

```python

```


# 18 PDEs: waves in 2D

## 2D wave equation

The wave equation in 2D can be derived in the same fashion as for the 1D case and reads

$$
\left(\frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y^2}\right)u(x, y, t) - \frac{1}{c^2} \frac{\partial^2 u}{\partial t^2} = 0
$$

or in general

$$
\boldsymbol{\nabla}^2 u(\mathbf{x}, t) - \frac{1}{c^2} \frac{\partial^2}{\partial t^2}u(\mathbf{x}, t) = 0.
$$

For the case of a string or a membrane with areal density $\rho$ and constant tension $T$, the wave velocity is

$$
c = \sqrt{\frac{T}{\rho}}.
$$

## Numerical solution: Finite difference discretization

As before, we replace the derivatives with their finite difference approximations, hence the name *finite difference method* (FDM) for this approach.

\begin{align}
\left(\frac{\partial^2}{\partial x^2} + \frac{\partial^2}{\partial y^2}\right)u(x, y, t) &\approx
 \frac{u(x+\Delta x, y, t) - 2u(x, y, t) + u(x-\Delta x, y, t)}{\Delta x^2} \notag\\
 &+ \frac{u(x, y + \Delta y, t) - 2u(x, y, t) + u(x, y-\Delta y, t)}{\Delta y^2}
\end{align}

Setting $\Delta x = \Delta y = \delta$ (same spacing in $x$ and $y$) and $u_{ijn} := u(i\delta, j\delta, n\Delta t)$ we obtain the discretized 2D wave equation

$$
u_{i,j,n+1} = 2(1 - 2\beta^2)u_{i,j,n} - u_{i,j,n-1} + \beta^2(u_{i+1,j,n} + u_{i-1,j,n} + u_{i,j+1,n} + u_{i,j-1,n})\\
\beta := \frac{c}{\delta / \Delta t}
$$

## Implementation
Packaged as a function.

```python
import matplotlib.pyplot as plt
import numpy as np

plt.style.use('ggplot')
```

```python
def wave2D(L=0.5, N=50, Dt=1e-4, Nt=100, step=1,
           rho=1.5e-3, tension=5., aspect=1):

    Lx = L
    Ly = aspect*L
    Nx = N
    Ny = int(aspect*N)
    delta = L/N

    c = np.sqrt(tension/rho)

    beta = c*Dt/delta
    beta2 = beta**2

    print("c = {0} m/s".format(c))
    print("delta = {0} m,  Dt = {1} s, delta/Dt = {2} m/s".format(delta, Dt, delta/Dt))
    print("beta = {}".format(beta))
    print("beta * sqrt(2) = {}".format(beta * np.sqrt(2)))

    x = np.linspace(0, Lx, Nx+1)  # need N+1!
    y = np.linspace(0, Ly, Ny+1)
    # note: we use u[i*delta, j*delta] = u(x, y)
    #       (which is the transpose of the numpy row-column layout so we transpose X and Y)
    X, Y = np.meshgrid(x, y)
    X, Y = X.T, Y.T

    def gaussian(x, u0=0.05, x0=None, sigma=0.1*L):
        x0 = np.mean(x) if x0 is None else x0
        g = u0/np.sqrt(2*np.pi*sigma**2) * np.exp(-(x-x0)**2 / (2*sigma**2))
        return g

    # u[i*delta, j*delta] = u(x, y)
    # displacements at j-1, j, j+1
    u0 = np.zeros((Nx+1, Ny+1))
    u1 = np.zeros_like(u0)
    u2 = np.zeros_like(u0)

    # save array
    u_t = np.zeros((int(np.ceil(Nt/step)), Nx+1, Ny+1))

    # use a boolean mask to define regions
    inside = np.zeros_like(u0, dtype=np.bool)
    inside[1:-1, 1:-1] = True
    boundary = np.logical_not(inside)

    # boundary conditions
    def set_boundary_conditions(u):
        u[boundary] = 0

    set_boundary_conditions(u0)
    set_boundary_conditions(u1)
    set_boundary_conditions(u2)

    # initial conditions: velocity 0, i.e. no difference between y0 and y1
    u0[inside] = u1[inside] = (gaussian(X)*gaussian(Y))[inside]

    # save initial
    t_index = 0
    u_t[t_index, :, :] = u0
    if step == 1:
        t_index += 1
        u_t[t_index, :, :] = u1

    for nt in range(2, Nt):
        u2[1:-1, 1:-1] = 2*(1-2*beta2)*u1[1:-1, 1:-1] - u0[1:-1, 1:-1] + beta2*(u1[2:, 1:-1]
                                            + u1[:-2, 1:-1] + u1[1:-1, 2:] + u1[1:-1, :-2])
        u0[:, :], u1[:, :] = u1, u2

        if nt % step == 0 or nt == Nt-1:
            t_index += 1
            u_t[t_index, :, :] = u2
            print("Iteration {0:5d}".format(nt), end="\r")
    else:
        print("Completed {0:5d} iterations: t={1} s".format(nt, nt*Dt))

    return u_t, X, Y, delta, Dt, step

```

In 2D, the Courant criterion for stability is

$$
\beta = \frac{c}{\delta/\Delta t} ≤ \frac{1}{\sqrt{2}}
$$

or

$$
\sqrt{2} \beta ≤ 1.
$$

If the solution becomes unstable, check $\sqrt{2}\beta$!

```python
u_t, X, Y, delta, Dt, step = wave2D(Nt=1000)
```

    c = 57.735026918962575 m/s
    delta = 0.01 m,  Dt = 0.0001 s, delta/Dt = 100.0 m/s
    beta = 0.5773502691896257
    beta * sqrt(2) = 0.816496580927726
    Completed   999 iterations: t=0.0999 s

```python
times = Dt * np.arange(0, len(u_t))
```

Look at the minimum and maximum values for each frame:

```python
fig = plt.figure(figsize=(6,3))
ax = fig.add_subplot(111)
ax.plot(times, u_t.min(axis=(1, 2)),
         times, u_t.max(axis=(1, 2)))
ax.set_xlabel("time (s)")
fig.tight_layout()
plt.show()
```

The min/max values do not appear to grow, indicating stable integration.

#### 2D Animation
For 2D animation to work in a Jupyter notebook, use

```python
%matplotlib ipympl
```

If no animations are visible, restart kernel and execute the `%matplotlib ipympl` cell as the very first one in the notebook.

We use `matplotlib.animation` to look at movies of our solution:

```python
import matplotlib.animation as animation
```

The `update_wave()` function simply re-draws our image for every `frame`.

```python
u_limits = 0.5*u_t.min(), 0.5*u_t.max()

fig1 = plt.figure(figsize=(5,5))
ax = fig1.add_subplot(111)
ax.set_aspect(1)

def update_wave(frame, data):
    global ax, Dt, u_limits, X, Y
    ax.clear()
    ax.set_xlabel("x")
    ax.set_ylabel("y")
    ax.contour(X, Y, data[frame], 30, cmap=plt.cm.viridis, norm=plt.Normalize(*u_limits))
    ax.contourf(X, Y, data[frame], 30, cmap=plt.cm.viridis, norm=plt.Normalize(*u_limits), alpha=0.8)
    ax.text(0.1, 0.9, "t = {0:g} s".format(frame*Dt), transform=ax.transAxes)

wave_anim = animation.FuncAnimation(fig1, update_wave, len(u_t), fargs=(u_t, ),
                                    interval=5, blit=False, repeat_delay=1000)

```

If you have `ffmpeg` installed then you can export to a MP4 movie file:

```python
wave_anim.save("square_drum_contours.mp4", fps=30, dpi=300)
```

#### 3D animation

```python
from mpl_toolkits.mplot3d import Axes3D
```

```python
def plot_wireframe(X, Y, u, limits=None, ax=None):
    if ax is None:
        fig = plt.figure()
        ax = fig.add_subplot(111, projection="3d")
    ax.plot_wireframe(X, Y, u)
    ax.set_xlabel(r"position $x$ (m)")
    ax.set_ylabel(r"position $y$ (m)")
    ax.set_zlabel(r"displacement $u$ (m)")
    ax.set_zlim(limits)
    ax.figure.tight_layout()
    return ax
```

```python
plot_wireframe(X, Y, u_t[0])
```

```python
u_limits = 0.5*u_t.min(), 0.5*u_t.max()

fig2 = plt.figure(figsize=(7,7))
ax = fig2.add_subplot(111, projection="3d")
ax.set_aspect(1)

def update_wire(frame, data):
    global ax, u_limits, X, Y
    ax.clear()
    plot_wireframe(X, Y, data[frame], limits=u_limits, ax=ax)
    ax.text(0.1, 0.9, 0.9, "t = {0:3.1f} ms".format(frame*Dt*1000), transform=ax.transAxes)

wire_anim = animation.FuncAnimation(fig2, update_wire, len(u_t), fargs=(u_t,),
                                    interval=30, blit=False, repeat_delay=1000)

```

    <IPython.core.display.Javascript object>

```python
wire_anim.save("square_drum_wire.mp4", fps=30, dpi=300)
```

#### Making pretty graphics

We want to us a surface representation where the color indicates the displacement. We also want to show our animation on a [webpage](http://asu-compmethodsphysics-phy494.github.io/ASU-PHY494/2017/04/20/17_PDEs_waves/#leap-frog-algorithm-for-solving-the-wave-equation).

```python
import matplotlib.colors
```

```python
def plot_surface(X, Y, u, limits=None, ax=None, **kwargs):
    if ax is None:
        fig = plt.figure()
        ax = fig.add_subplot(111, projection="3d")
    ax.plot_surface(X, Y, u, rstride=1, cstride=1, shade=True,
                    cmap=plt.cm.coolwarm, norm=norm, **kwargs)
    ax.set_xlabel(r"$x$ (m)")
    ax.set_ylabel(r"$y$ (m)")
    ax.set_zlabel(r"$u$ (m)")

    # fix scaling see https://stackoverflow.com/a/19933125
    #scaling = np.array([getattr(ax, 'get_{}lim'.format(dim))() for dim in 'xyz'])
    scaling = np.array([X.min(), X.max(), Y.min(), Y.max(), min(limits), max(limits)])
    ax.auto_scale_xyz(*[[np.min(scaling), np.max(scaling)]]*3)

    ax.set_zlim(limits)

    ax.figure.tight_layout()
    return ax

u_limits = 0.5*u_t.min(), 0.5*u_t.max()
umax = 0.4*np.max(np.abs(u_limits))
norm = matplotlib.colors.Normalize(vmin=-umax, vmax=umax)

fig3 = plt.figure(figsize=(3.5, 3.5))
ax = fig3.add_subplot(111, projection="3d")
# ax.set_aspect(1)

def update_surface(frame, data):
    global ax, u_limits, X, Y
    ax.clear()
    plot_surface(X, Y, data[frame], limits=u_limits, ax=ax, alpha=0.9)
    ax.text(0.2, 0.9, 0.9, "$t = {0:3.1f}$ ms".format(frame*Dt*1000), transform=ax.transAxes)
    ax.figure.tight_layout()

wire_anim = animation.FuncAnimation(fig3, update_surface, len(u_t), fargs=(u_t,),
                                    interval=30, blit=False, repeat_delay=1000)

```

The whole video can be incorporated into an html page (uses the HTML5 `<video>` tag).

```python
tag = wire_anim.to_html5_video()
with open("square_drum_redblue_2d.html", "w") as html:
    html.write(tag)
```

or exported as a MPEG4 movie (uses `ffmpeg`):

```python
wire_anim.save("square_drum_surface.mp4", fps=30, dpi=300)
```

```python

```


# Jastrow Factor Derivatives

This notebook calculates the expressions for the derivatives of the following Jastrow function:

$$J(\mathbf{X}) = \prod_{i < j} \exp\left(\frac{\alpha r_{ij}}{1 + \beta r_{ij}}\right)$$

with $N$ particles in $D$ dimensions, $\mathbf{X}\in\mathbb{R}^{N\times D}$ and $r_{ij} = ||\mathbf{X_i} - \mathbf{X_j}||$.

```python
%reload_ext blackcellmagic
from sympy import *
init_printing()
```

```python
N, D, i, j, k, l, d = symbols("N D i j k l d", integer=True)
X = IndexedBase("X", shape=(N, D))

```

## Definition

```python
class r(Function):
    @classmethod
    def eval(cls, i, j):
        if i.is_Number and j.is_Number:
            return sqrt(Sum((X[i, d] - X[j, d]) ** 2, (d, 1, D)))

    def doit(self, **kwargs):
        i, j = self.args
        return sqrt(Sum((X[i, d] - X[j, d]) ** 2, (d, 1, D)))

    def _eval_derivative(self, x):
        return self.doit()._eval_derivative(x)

def J(X):
    alpha, beta = symbols(r"\alpha \beta", real=True)
    exponent = Sum(alpha * r(i, j) / (1 + beta * r(i, j)), (i, 1, N - 1), (j, i + 1, N))
    return exp(exponent), (alpha, beta)

Jastrow, (alpha, beta) = J(X)
Jastrow

```

```python
log_Jastrow = expand_log(log(Jastrow), force=True)
log_Jastrow

```

## Derivative w.r.t. $\beta$

```python
dlogJ_dBeta = diff(log_Jastrow, beta, 1)
dlogJ_dBeta

```

## Derivative w.r.t. $X_{kl}$

```python
dlogJ_dXkl = diff(log_Jastrow, X[k, l], 1)
dlogJ_dXkl

```

```python
# Manually simplify the above:
dlogJ_dXkl = Sum(
    alpha
    * beta
    * (KroneckerDelta(k, i) - KroneckerDelta(k, j))
    * (X[i, l] - X[j, l])
    / (beta * r(i, j) + 1)
    / (beta * r(i, j) * (beta * r(i, j) + 1)),
    (i, 1, N - 1),
    (j, i + 1, N),
)
dlogJ_dXkl

```

## Second derivative w.r.t. $X_{kl}$

```python
d2J_dXkl2 = dlogJ_dXkl ** 2 + diff(dlogJ_dXkl, X[k, l], 1)
d2J_dXkl2.subs(beta, 0)

```

```python
# Manually simplify the above:
d2J_dXkl2 = dlogJ_dXkl ** 2 + Sum(
    alpha
    * (KroneckerDelta(k, i) - KroneckerDelta(k, j)) ** 2
    * (X[i, l] - X[j, l]) ** 2
    / (beta * r(i, j) + 1) ** 3
    / r(i, j) ** 3
    * (
        (beta * r(i, j) + 1) * r(i, j) ** 2 / (X[i, l] - X[j, l]) ** 2
        - 3 * beta * r(i, j)
        - 1
    ),
    (i, 1, N - 1),
    (j, i + 1, N),
)
d2J_dXkl2

```

```python

```


---
## 30. Integración Numérica

Eduard Larrañaga (ealarranaga@unal.edu.co)

---

### Resumen

En este cuaderno se presentan algunas técnicas de integración numérica.

---

Una de las tareas más comunes en astrofísica es evaluar integrales como

\begin{equation}
I = \int_a^b f(x) dx ,
\end{equation}

y, en muchos casos, estas no pueden realizarse en forma analítica. El integrando en estas expresiones puede darse como una función analítica $f(x)$ o como un conjunto discreto de valores $f(x_i)$. Acontinuación describiremos algunas técnicas para realizar estas integrales numéricamente en ambos casos.

---
## Interpolación por intervalos y cuadraturas

Cualquier método de integración que utilice una suma con pesos es denominado **regla de cuadraturas**. Suponga que conocemos (o podemos evaluar) el integrando  $f(x)$ en un conjunto finito de *nodos*, $\{x_j\}$ con $j=0,\cdots,n$ en el intervalo $[a,b]$ y tal que $x_0 = a$ y $x_n = b$. Con esto se obtendrá un conjunto de $n+1$ nodos o equivalentemente $n$ intervalos. Una aproximación discreta de la integral de esta función está dada por la **regla del rectángulo**,

\begin{equation}
I = \int_a^b f(x) dx \approx  \Delta x \sum_{i=0}^{n-1} f(x_i),
\end{equation}

donde el ancho de los intervalos es $\Delta x = \frac{b-a}{n}$. A partir de la definición de una integral, es claro que esta aproximación converge al valor real de la integral cuando $n\rightarrow \infty$, i.e. cuando $\Delta x \rightarrow 0$.

A pesar de que la regla del rectangulo puede dar una buena aproximación de la integral, puede ser mejorada al utilizar una función interpolada en cada intervalo. Los métodos en los que se utiliza la interpolación de polinomios se denominan, en general, **cuadraturas de Newton-Cotes**.

---
### Regla de punto medio

La modificación más simple a la regla del rectángulo descrita arriba es utilizar el valor central de la función $f(x)$ en cada intervalo en lugar del valor en uno de los nodos. de Esta forma, si es posible evaluar el integrando en el punto medio de cada intervalo, el valor aproximado de la integral estará dado por

\begin{equation}
I = \int_{a}^{b} f(x) dx = \sum _{i=0}^{n-1} (x_{i+1} - x_i)  f(\bar{x}_i ),
\end{equation}

donde $\bar{x}_i = \frac{x_i + x_{i+1}}{2}$ es el punto medio en el intervalo $[x_i, x_{i+1}]$.

Con el fin de estimar el error asociado con este método, se utiliza una expansión en serie de Taylor del integrando en el intervalo $[x_i, x_{i+1}]$ alrededor del punto medio $\bar{x}_i$,

\begin{equation}
f(x) = f(\bar{x}_i) + f'(\bar{x}_i)(x-\bar{x}_i) + \frac{f''(\bar{x}_i)}{2}(x-\bar{x}_i)^2 + \frac{f'''(\bar{x}_i)}{6}(x-\bar{x}_i)^3 + ...
\end{equation}

Integrando esta expresión desde $x_i$ hasta $x_{i+1}$, y notando que los terminos de orden impar se anulan, se obtiene

\begin{equation}
\int_{x_i}^{x_{i+1}} f(x)dx = f(\bar{x}_i)(x_{i+1}-x_i) + \frac{f''(\bar{x}_i)}{24}(x_{i+1}-x_i)^3 + ...
\end{equation}

Esta expansión muestra que el error asociado con la aproximación en cada intervalo es de orden $\varepsilon_i =  (x_{i+1}-x_i)^3$. Ya que la integral total se obtiene como una suma de $n$ integrales similares, el error total es será de orden $\varepsilon = n \varepsilon_i $.

Cuando los nodos están igualmente espaciados, podemos escribir el tamaño de estos intervalos como  $h = \frac{b - a}{n}$ y por ello, el error asociado con cada intervalo es $\varepsilon_i =\frac{(b - a)^3}{n^3} = h^3$, mientras que el error total de la cuadratúra será de orden $\varepsilon = n \varepsilon_i = \frac{(b - a)^3}{n^2} = nh^3$.

#### Ejemplo. Integración numérica

Leeremos los datos de la función desde un archivo .txt y estimaremos numéricamente el valor de la integral de esta función utilizando la regla del punto medio. Debido a que la función esdada en forma de puntos discretos (y no en forma analítica), no es posible evaluar el valor de la función en los puntos medios por lo que utilizaremos inicialmente el valor en el primer punto de cada uno de los intervalos para calcular las sumas parciales.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Reading the data
data = np.loadtxt('data_points1.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(N-1):
    dx = x[i+1] - x[i]
    Integral = Integral + dx*f[i]

    plt.vlines([x[i], x[i+1]], 0, [f[i], f[i]], color='red')
    plt.plot([x[i], x[i+1]], [f[i], f[i]],color='red')
    plt.fill_between([x[i], x[i+1]], [f[i], f[i]],color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Integración por la regla del rectángulo')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'El resultado de la integración numérica de la función discreta')
print(f'entre x = {x[0]:.1f} y x = {x[len(x)-1]:.1f} es I = {Integral:.5e}')
```

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Reading the data
data = np.loadtxt('data_points2.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(N-1):
    dx = x[i+1] - x[i]
    Integral = Integral + dx*f[i]

    plt.vlines([x[i], x[i+1]], 0, [f[i], f[i]], color='red')
    plt.plot([x[i], x[i+1]], [f[i], f[i]],color='red')
    plt.fill_between([x[i], x[i+1]], [f[i], f[i]],color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Integración por la regla del rectángulo')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'El resultado de la integración numérica de la función discreta')
print(f'entre x = {x[0]:.1f} y x = {x[len(x)-1]:.1f} es I = {Integral:.5e}')
```

---
### Regla del Trapezoide

La siguiente generalización de la regla del rectángulo corresponde a aproximar la función $f(x)$ con un polinomio lineal en cada uno de los intervalos. Esto se conoce como la **regla del trapezoide** y la correspondiente cuadratura estará dada por

\begin{equation}
I = \int_{a}^{b} f(x) dx = \sum _{i=0}^{n-1} \frac{1}{2} (x_{i+1} - x_i)  \left[ f(x_{i+1}) + f(x_i) \right] .
\end{equation}

Contrario a lo que sucede en la regla del punto medio, este método no necesita la evaluación del integrando en el punto medio sino en los dos nodos de cada intervalo.

#### Ejemplo. Integración con la regla del trapezoide.

De nuevo se leerán los datos de la función a partir de un archivo .txt file y se integrará numéricamente utilizando la regla del trapezoide.

```python
import numpy as np
import matplotlib.pyplot as plt

# Reading the data
data = np.loadtxt('data_points1.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(N-1):
    dx = x[i+1] - x[i]
    f_mean = (f[i] + f[i+1])/2
    Integral = Integral + dx*f_mean

    plt.vlines([x[i], x[i+1]], 0, [f[i], f[i+1]], color='red')
    plt.plot([x[i], x[i+1]], [f[i], f[i+1]],color='red')
    plt.fill_between([x[i], x[i+1]], [f[i], f[i+1]],color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Integración con la regla del trapezoide')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'El resultado de la integración numérica de la función discreta')
print(f'entre x = {x[0]:.1f} y x = {x[len(x)-1]:.1f} es I = {Integral:.5e}')
```

```python
import numpy as np
import matplotlib.pyplot as plt

# Reading the data
data = np.loadtxt('data_points2.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(N-1):
    dx = x[i+1] - x[i]
    f_mean = (f[i] + f[i+1])/2
    Integral = Integral + dx*f_mean

    plt.vlines([x[i], x[i+1]], 0, [f[i], f[i+1]], color='red')
    plt.plot([x[i], x[i+1]], [f[i], f[i+1]],color='red')
    plt.fill_between([x[i], x[i+1]], [f[i], f[i+1]],color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Integración con la regla del trapezoide')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'El resultado de la integración numérica de la función discreta')
print(f'entre x = {x[0]:.1f} y x = {x[len(x)-1]:.1f} es I = {Integral:.5e}')

```

---
## Regla de Simpson

La regla de Simpson es un método en el que la integral $f(x)$ se estima aproximando el integrando por un polinomi de segundo orden en cada intervalo.

Si se conocen tres valores de la función; $f_1 =f(x_1)$, $f_2 =f(x_2)$ y $f_3 =f(x_3)$; en los puntos $x_1 < x_2 < x_3$, se puede ajustar un polinomio de segundo orden de la forma
l
\begin{equation}
p_2 (x) = A (x-x_1)^2 + B (x-x_1) + C .
\end{equation}

Al integrar este polinomio en el intervalo  $[x_1 , x_3]$, se obtiene

\begin{equation}
\int_{x_1}^{x^3} p_2 (x) dx = \frac{x_3 - x_1}{6} \left( f_1 + 4f_2 + f_3 \right) + \mathcal{O} \left( (x_3 - x_1)^5 \right)
\end{equation}

---
### Regla de Simpson con nodos igualmente espaciados

Si se tienen $N$ nodos igualmente espaciados en el intervalo de integración, o equivalentemente $n=N-1$ intervalos con un ancho constante  $h$, la integral total mediante la regla de Simpson se escribe en la forma

\begin{equation}
I = \int_a^b f(x) dx \approx \frac{\Delta x}{3} \sum_{i=0}^{\frac{n-2}{2}} \left[ f(x_{2i}) + 4f(x_{2i+1}) + f(x_{2i+2}) \right] .
\end{equation}

El error numérico en cada intervalo es de orden $\Delta x^5$ y por lo tanto, la integral total tendrá un error de orden $n \Delta x^5 = \frac{(a-b)^5}{n^4}$.

#### Ejemplo. Integración con la regla de Simpson

```python
import numpy as np
import matplotlib.pyplot as plt

def quadraticInterpolation(x1, x2, x3, f1, f2, f3, x):
    p2 = (((x-x2)*(x-x3))/((x1-x2)*(x1-x3)))*f1 + (((x-x1)*(x-x3))/((x2-x1)*(x2-x3)))*f2 +\
         (((x-x1)*(x-x2))/((x3-x1)*(x3-x2)))*f3
    return p2

# Reading the data
data = np.loadtxt('data_points1.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)
n = N-1

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(int((n-2)/2 +1)):
    dx = x[2*i+1] -x[2*i]
    Integral = Integral + dx*(f[2*i] + 4*f[2*i+1] + f[2*i+2])/3

    x_interval = np.linspace(x[2*i],x[2*i+2],6)
    y_interval = quadraticInterpolation(x[2*i], x[2*i+1], x[2*i+2], f[2*i], f[2*i+1], f[2*i+2], x_interval)
    plt.plot(x_interval, y_interval,'r')
    plt.fill_between(x_interval, y_interval, color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Integración con la regla de Simpson')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'El resultado de la integración numérica de la función discreta')
print(f'entre x = {x[0]:.1f} y x = {x[len(x)-1]:.1f} es I = {Integral:.5e}')

```

```python
import numpy as np
import matplotlib.pyplot as plt

def quadraticInterpolation(x1, x2, x3, f1, f2, f3, x):
    p2 = (((x-x2)*(x-x3))/((x1-x2)*(x1-x3)))*f1 + (((x-x1)*(x-x3))/((x2-x1)*(x2-x3)))*f2 +\
         (((x-x1)*(x-x2))/((x3-x1)*(x3-x2)))*f3
    return p2

# Reading the data
data = np.loadtxt('data_points2.txt', comments='#', delimiter=',')
x = data[:,0]
f = data[:,1]

N = len(x)
n = N-1

plt.figure(figsize=(7,5))

# Numerical integration loop
Integral = 0.
for i in range(int((n-2)/2 +1)):
    dx =  x[2*i+1] - x[2*i]
    Integral = Integral + dx*(f[2*i] + 4*f[2*i+1] + f[2*i+2])/3

    x_interval = np.linspace(x[2*i],x[2*i+2],6)
    y_interval = quadraticInterpolation(x[2*i], x[2*i+1], x[2*i+2], f[2*i], f[2*i+1], f[2*i+2], x_interval)
    plt.plot(x_interval, y_interval,'r')
    plt.fill_between(x_interval, y_interval, color='red', alpha=0.3)

plt.scatter(x, f, color='black')
plt.hlines(0, x.min(), x.max())
plt.title('Integración con la regla de Simpson')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.show()

print(f'El resultado de la integración numérica de la función discreta')
print(f'entre x = {x[0]:.1f} y x = {x[len(x)-1]:.1f} es I = {Integral:.5e}')
```

---
### Regla de Simpson para nodos no-equidistantes

Cuando los nodos de la malla de discretización de $f(x)$ no están igualmente espaciados, la regla de Simpson debe modificarse en la forma

\begin{equation}
I = \int_a^b f(x) dx \approx \sum_{i=0}^{\frac{n-2}{2}} \left[ \alpha f(x_{2i}) + \beta f(x_{2i+1}) +\gamma f(x_{2i+2}) \right]
\end{equation}

donde

\begin{align}
\alpha = &\frac{-h_{2i+1}^2 + h_{2i+1} h_{2i} + 2 h_{2i}^2}{6 h_{2i}}   \\
\beta = &\frac{ (h_{2i+1} + h_{2i})^3 }{6 h_{2i+1} h_{2i}} \\
\gamma =&  \frac{2 h_{2i+1}^2 + h_{2i+1} h_{2i} - h_{2i}^2}{6 h_{2i+1}}
\end{align}

y $h_j = x_{j+1} - x_j$.


```python

```

```python
#Import packages
from scipy import optimize,arange
from numpy import array
import numpy as np
import matplotlib.pyplot as plt
import random
import sympy as sm
from math import *
%matplotlib inline
from IPython.display import Markdown, display
import pandas as pd

```

```python
a = sm.symbols('a')
c_vec = sm.symbols('c_vec')
b = sm.symbols('b')
q_vec = sm.symbols('q_i') # q for firm i
q_minus = sm.symbols('q_{-i}') # q for the the opponents

#The profit of firm 1 is then:
Pi_i = q_vec*((a-b*(q_vec+q_minus))-c_vec)

#giving focs:
foc = sm.diff(Pi_i,q_vec)
foc

```

$\displaystyle a - b q_{i} - b \left(q_{i} + q_{-i}\right) - c_{vec}$

In order to use this in our solutionen, we rewrite $x_{i}+x_{-i} = \sum x_{i}$ using np.sum and then define a function for the foc

```python
def foc1(a,b,q_vec,c_vec):
    # Using the result from the sympy.diff
    return -b*q_vec+a-b*np.sum(q_vec)-c_vec
```

```python

```


# Curvature

Contents

1  Preliminary definitions

2  Lengths and volumes

3  Christoffel symbols (connection coefficients)

3.1  Covariant derivatives

3.2  Christoffel symbols and the metric

3.3  Example: the surface of a unit sphere in polar coordinates

4  Parallel transport of vectors

4.1  Geodesics

5  The Riemann curvature tensor

6  The Ricci and Einstein tensors

6.1  The Ricci tensor

6.2  The Ricci scalar

6.3  The Einstein tensor

6.4  Bianchi identities

## Preliminary definitions

In simple terms (to be made more precise later):

A __geodesic__ is the shortest distance between two points, e.g. a straight line in Euclidean space or a great circle on the surface of a sphere.

A __manifold__ is a smoothly curved space which is locally flat. In GR we are interested in manifolds which are (a) differentiable and (b) have a symmetric metric tensor (so $g_{\mu \nu} = g_{\nu \mu}$). This is a _Riemannian manifold_.

A __path__ is a connected series of points. A __curve__ is a parameterized path, with each point associated with a real value.

## Lengths and volumes

An $n$-dimensional Riemannian manifold has a line element given by $dl^2 = g_{ij} dx^i dx^j$.

To get the length $L$ of an arbitrary curve parameterized by $\lambda$, between $\lambda_0$ and $\lambda_1$, we use the square root of the absolute value of the line element and integrate:
$$ L = \int_{\lambda_0}^{\lambda_1} \left| g_{\alpha \beta} \frac{dx^\alpha}{d\lambda} \frac{dx^\beta}{d\lambda} \right|^{1/2} d\lambda $$

But $\frac{dx^\alpha}{d\lambda}$ are the components of the tangent vector $\vec{V}$, so
$$ L = \int_{\lambda_0}^{\lambda_1} \left| \vec{V} \cdot \vec{V} \right|^{1/2} d\lambda $$

A small 4-dimensional region in a local Lorentz frame is $dx^0 dx^1 dx^2 dx^3 \equiv d^4x$. In other frames we in principle need to use the Jacobian. In practice we can more conveniently use the determinant $g$ of the metric: $g = det(g_{\alpha \beta})$. Then the _element of proper volume_ is $d^4x' = |g|^{1/2} d^4x$.

## Christoffel symbols (connection coefficients)

Differentiation of a vector $\vec{V} = V^\alpha \vec{e}_\alpha$ gives (chain rule):
$$ \frac{\partial \vec{V}}{\partial x^\beta} = \frac{\partial V^\alpha}{\partial x^\beta} \vec{e}_\alpha + V^\alpha \frac{\partial \vec{e}_\alpha}{\partial x^\beta} $$

In flat Minkowski space the basis vectors are constant and the second term disappears, but this is not the case in the curved space of GR.

To handle derivatives of the basis vectors we introduce the Christoffel symbols $\Gamma$ (also known as connection coefficients):

$$ \frac{\partial e_\alpha}{\partial x^\beta} = \Gamma^\gamma\!_{\alpha \beta}\ e_\gamma $$

Here index $\alpha$ gives the basis being differentiated, $\beta$ the coordinate with respect to which it is diffentiated, and $\gamma$ the component of the resulting derivative vector.

Substituting back into the original derivative equation (and after juggling dummy indices a bit):

$$ \frac{\partial \vec{V}}{\partial x^\beta} = \left( \frac{\partial V^\alpha}{\partial x^\beta} +  V^\gamma \Gamma^\alpha\!_{\gamma \beta} \right) \vec{e}_\alpha = V^\alpha\!_{;\beta}\ \vec{e}_\alpha $$

where we use semicolon notation for components of the _covariant derivative_ $\nabla\vec{V}$ and comma notation for the partial derivative:

$$ V^\alpha\!_{;\beta} \equiv V^\alpha\!_{,\beta} + V^\gamma \Gamma^\alpha\!_{\gamma \beta} \qquad V^\alpha\!_{,\beta} \equiv \frac{\partial V^\alpha}{\partial x^\beta} \qquad ( \nabla \vec{V} )^\alpha\!_\beta \equiv \nabla_\beta V^\alpha \equiv  V^\alpha\!_{;\beta} $$

In 4-dimensional spacetime we expect $4^3 = 64$ Christoffel symbols, but in GR this is reduced by symmetry. In particular, $\Gamma^\gamma\!_{\alpha \beta} = \Gamma^\gamma\!_{\beta \alpha}$.

### Covariant derivatives

We saw above that the covariant derivative of a vector is $\nabla_\beta V^\alpha = V^\alpha\!_{,\beta} +  V^\gamma \Gamma^\alpha\!_{\gamma \beta}$.

Similarly, the covariant derivative of a 1-form is $\nabla_\beta V_\alpha = V_{\alpha\,,\beta} - V_\gamma \Gamma^\gamma\!_{\alpha \beta}$.

Higher rank tensors follow a regular pattern:
- take the partial derivative of the tensor
- for each upper index add a term of the form $V^\gamma \Gamma^\alpha\!_{\gamma \beta}$
- for each lower index subtract a term of the form $V_\gamma \Gamma^\gamma\!_{\alpha \beta}$

### Christoffel symbols and the metric

__Within GR__ it is always true that the covariant derivative of the metric tensor is zero (_"metric compatibility"_):
$$ \nabla_\beta g_{\mu \nu} = \nabla_\beta g^{\mu \nu} = 0 $$

Given this, it is possible (and very useful) to express the Christoffel symbols in terms of the inverse metric and partial derivatives of the metric:
$$ \Gamma^\gamma\!_{\beta \mu} = \frac{1}{2} g^{\alpha \gamma} \left( g_{\alpha \mu\,,\beta} + g_{\alpha \beta\,,\mu} + g_{\beta \mu\,,\alpha} \right) $$

### Example: the surface of a unit sphere in polar coordinates

The radius $r$ is a constant, $1$, so the coordinates are $\theta$ and $\phi$. The metric $g_{ij}$ and inverse metric $g^{ij}$ are:

$$ [g_{ij}] = \begin{bmatrix} 1 & 0 \\ 0 & \sin^2 \theta \end{bmatrix} \qquad [g^{ij}] = \begin{bmatrix} 1 & 0 \\ 0 & \frac{1}{\sin^2 \theta} \end{bmatrix}  $$

```python
from sympy.interactive.printing import init_printing
from sympy.matrices import *
from sympy import Array, sin, cos, diff, derive_by_array, simplify
from sympy.abc import phi, theta
init_printing()
```

```python
coords = (theta, phi)
g_ij = Matrix([[1, 0], [0, sin(theta)**2]])
inv_gij = g_ij.inv()
display(g_ij, inv_gij)
```

Given a (2D) matrix, SymPy can calculate all the partial derivatives at once and return them in an 3D array. Then `result[i,j,k]` = $\displaystyle \frac{\partial A_{jk}}{\partial x^i}$ (note the order of indices).

In our example, only $\displaystyle \frac{\partial g_{\phi \phi}}{\partial \theta}$ is non-zero.

```python
def getPartialDiffs(metric, coords):
    result = derive_by_array(metric, coords)
    return result
```

```python
diffs = getPartialDiffs(g_ij, coords)
display(diffs)
diffs[0,1,1]
```

Though we expect few non-zero terms in our example, the following functions calculate $\Gamma^{up}\!_{l1\,l2}$ in as general a way as possible.

Unfortunately SymPy doesn't seem to have a way to pre-allocate 3D arrays (`zeros()` only handles 1 or 2 dimensions), so there are lots of ugly list appends as a workaround.

```python
def calcOneChristoffel(invmetric, diffs, up, l1, l2):
    result = 0
    for c in range(invmetric.shape[0]): # dummy index to sum over
        result += 1/2*invmetric[up, c]*(diffs[l1, c, l2] + diffs[l2, c, l1] - diffs[c, l1, l2])
    return result

def calcAllChristoffel(metric, coords):
    diffs = getPartialDiffs(metric, coords)
    invmetric = metric.inv()
    ndims = len(coords)
    result = []
    for up in range(ndims):
        result_l1 = []
        for l1 in range(ndims):
            result_l2 = []
            for l2 in range(ndims):
                result_l2.append(calcOneChristoffel(invmetric, diffs, up, l1, l2))
            result_l1.append(result_l2)
        result.append(result_l1)
    return Array(result)
```

```python
Gamma = calcAllChristoffel(g_ij, coords)
simplify(Gamma)
```

Next show just the non-zero values. We expect $\Gamma^\theta\!_{\phi \phi} = -\sin \theta \cos \theta = -0.5 \sin(2 \theta)$ and $\Gamma^\phi\!_{\theta \phi} = \Gamma^\phi\!_{\phi \theta} = \cot \theta$. See Collier, Problem 6.3.

```python
def showNonzeroChristoffel(Gamma, coords):
    ndims = len(coords)
    for up in range(ndims):
        for l1 in range(ndims):
            for l2 in range(ndims):
                if Gamma[up, l1, l2] != 0:
                    display((coords[up], coords[l1], coords[l2]), Gamma[up, l1, l2])

showNonzeroChristoffel(Gamma, coords)
```

## Parallel transport of vectors

This is intuitive in flat Euclidean space: a vector moved around such that at each point it it parallel to the vector at the previous point will always keeping pointing in the same direction. In a curved space, such as on the surface of a sphere, the results can be different.

Consider a vector parallel-transported around the path A-N-B-A. On returning to A it is clearly ___not___ parallel to how it started.

 _(Image attribution: Fred the Oyster [CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0)])_

Consider a smooth curve parameterized by $\lambda$. The tangent vector is
$$ U^\beta = \frac{dx^\beta}{d\lambda} = \left( \frac{dx^0}{d\lambda}, \frac{dx^1}{d\lambda}, \frac{dx^2}{d\lambda}, \frac{dx^3}{d\lambda} \right) $$

We assume in GR that space is locally flat and a small enough region at a point $\mathcal{P}$ will act as a _locally intertial coordinate frame_. So at $\mathcal{P}$ on the curve, the components of a non-tangent vector $\vec{V}$ must be constant: $\frac{d\vec{V}}{d\lambda} = 0$. Also at $\mathcal{P}$, the Christoffel symbol $\Gamma^\alpha\!_{\beta \gamma} = 0$ and the covariant derivative is equal to the partial derivative.

Notation varies for how textbooks describe this, but in one version:
$$ \frac{dV^\alpha}{d\lambda} = U^\beta V^\alpha\!_{,\beta} = U^\beta V^\alpha\!_{;\beta} = 0 $$

Alternatively,
$$ \frac{d\vec{V}}{d\lambda} = \left( \frac{dV^\alpha}{d\lambda} + V^\gamma \Gamma^\alpha\!_{\gamma \beta} \frac{dx^\beta}{d\lambda} \right) \vec{e}_\alpha = \frac{DV^\alpha}{d\lambda} \vec{e}_\alpha $$
where the upper case D refers to the covariant derivative along the curve, sometimes known as the _absolute_ or _intrinsic_ derivative. For parallel transport, $\frac{DV^\alpha}{d\lambda} = 0$.

### Geodesics

These are lines that in curved space are "as straight as possible" in the sense that the tangent vector $U^\alpha = \frac{dx^\alpha}{d\lambda}$ is parallel-transported along the curve. This (eventually) leads to:
$$ \frac{DU^\alpha}{d\lambda} = \frac{d^2x^\alpha}{d\lambda^2} + \Gamma^\alpha\!_{\gamma \beta} \frac{dx^\beta}{d\lambda} \frac{dx^\gamma}{d\lambda} = 0 $$

## The Riemann curvature tensor

This rank-4 tensor is a measure of how a vector transforms a it is parallel-transported around an infinitessimal loop on the manifold. The derivation is quite long (read a book!), but the result is
$$ R^\alpha\!_{\beta \mu \nu} = \Gamma^\alpha\!_{\beta \nu,\mu} - \Gamma^\alpha\!_{\beta \mu,\nu} + \Gamma^\alpha\!_{\sigma \mu} \Gamma^\sigma\!_{\beta \nu} - \Gamma^\alpha\!_{\sigma \nu} \Gamma^\sigma\!_{\beta \mu} $$
So the first two terms are partial derivatives of the Christoffel terms and the last two are products summed over a dummy variable. In $n$ dimensions this means $n^4$ terms but this is reduced by symmetries to 1 in 2D space, 6 in 3D and 20 in 4D.

The metric can be used to lower the upstairs index: $R_{\alpha \beta \mu \nu} = g_{\alpha \lambda} R^\lambda\!_{\beta \mu \nu}$.

Both $R^\alpha\!_{\beta \mu \nu}$ and $R_{\alpha \beta \mu \nu}$ are antisymmetric on the second pair of indices and $R_{\alpha \beta \mu \nu}$ is also antisymmetric on the first pair and symmetric on exchange of the two pairs:
$$ R_{\alpha \beta \mu \nu} = -R_{\beta \alpha \mu \nu} = -R_{\alpha \beta \nu \mu} = R_{\mu \nu \alpha \beta} $$
Also:
$$ R_{\alpha \beta \mu \nu} + R_{\alpha \nu \beta \mu} +R_{\alpha \mu \nu \beta} = 0 $$

```python
def calcRiemann(Gamma, coords, up, l1, l2, l3):
    R = diff(Gamma[up, l1, l3], coords[l2]) - diff(Gamma[up, l1, l2], coords[l3])
    for c in range(len(coords)):
        R += (Gamma[up, c, l2] * Gamma[c, l1, l3] - Gamma[up, c, l3] * Gamma[c, l1, l2])
    return simplify(R)
```

Calculation for the surface of a sphere, using the Christoffel symbols we calculated above:

```python
ndims = len(coords)
for up in range(ndims):
    for l1 in range(ndims):
        for l2 in range(ndims):
            for l3 in range(ndims):
                R = calcRiemann(Gamma, coords, up, l1, l2, l3)
                if R != 0:
                    display((coords[up], coords[l1], coords[l2], coords[l3]), R)
```

As expected, we get $R^\theta\!_{\phi \theta \phi} = -R^\theta\!_{\phi \phi \theta} = \sin^2 \theta$ (see Collier, Problem 6.9 and http://einsteinrelativelyeasy.com/index.php/general-relativity/70-the-riemann-curvature-tensor-for-the-surface-of-a-sphere).

Getting $R^\phi\!_{\theta \phi \theta} = -R^\phi\!_{\theta \theta \phi} = 1$ was not expected from what I had read. Repeating this manually, I find $R^\phi\!_{\theta \phi \theta} = -\partial_\theta \Gamma^\phi\!_{\theta \phi} - \Gamma^\phi\!_{\theta \phi} \Gamma^\phi\!_{\phi \theta} = -\partial_\theta \cot \theta - \cot^2 \theta = 1$.

We know from the Riemann symmetries that $R_{\phi \theta \phi \theta} = R_{\theta \phi \theta \phi}$ and that $R_{\phi \theta \phi \theta} = g_{\phi \alpha} R^\alpha\!_{\theta \phi \theta} = g_{\phi \theta} R^\theta\!_{\theta \phi \theta} + g_{\phi \phi} R^\phi\!_{\theta \phi \theta} = 0 + \sin^2(\theta)\ R^\phi\!_{\theta \phi \theta}$. Similarly, $R_{\theta \phi \theta \phi} = g_{\theta \theta} R^\theta\!_{\phi \theta \phi} = \sin^2(\theta)$.

So $R^\phi\!_{\theta \phi \theta} = 1$ is correct, and symmetry-connected to the $\sin^2 \theta$ terms via the metric.

## The Ricci and Einstein tensors

The rank-4 Riemann tensor is clearly quite complicated, but in GR we usually work with lower-rank contractions.

### The Ricci tensor

Contracting the first and last index of the Riemann tensor gives $R^\alpha\!_{\mu \nu \alpha} = R_{\mu \nu}$, where $R_{\mu \nu}$ is the Ricci tensor. Because of the symetry of the Riemann tensor, various other contractions are possible to get $\pm R_{\mu \nu}$.

Equivalently, we can use the inverse metric with the index-lowered form of the Riemann tensor: $R_{\mu \nu} = g^{\alpha \beta} R^\alpha\!_{\alpha \mu \nu \beta}$

```python
def calcRicciTensor(Gamma, coords):
    ndims = len(coords)
    R = zeros(ndims)
    for i in range(ndims):
        for j in range(ndims):
            for c in range(len(coords)):
                R[i,j] += calcRiemann(Gamma, coords, c, i, j, c)
    return simplify(R)
```

The surface of a sphere example again:

```python
R_ij = calcRicciTensor(Gamma, coords)
R_ij
```

### The Ricci scalar

Contracting the Ricci tensor gives the Ricci scalar $R = g^{\mu \nu} R_{\mu \nu}$

In practice, we just sum the elementwise products of these matrices:

```python
def calcRicciScalar(R_ij, invmetric):
    return sum(invmetric * R_ij)
```

The surface of a sphere example yet again:

```python
R = calcRicciScalar(R_ij, inv_gij)
R
```

### The Einstein tensor

This symmetric tensor is defined from both the Ricci tensor and the Ricci scalar:
$$ G^{\alpha \beta} = G^{\beta \alpha} = R^{\alpha \beta} - \frac{1}{2} R g^{\alpha \beta} \qquad G_{\alpha \beta} = G_{\beta \alpha} = R_{\alpha \beta} - \frac{1}{2} R g_{\alpha \beta} $$

This is the left hand side of the Einstein filed equations, so extremely important.

```python
def calcEinstein(R_ij, R, g_ij):
    return R_ij - R/2*g_ij
```

The result is particularly boring on the surface of a sphere:

```python
calcEinstein(R_ij, R, g_ij)
```

### Bianchi identities

In full form, these are a relationship between covariant derivatives:
$$ R_{\alpha \beta \mu \nu ;\lambda} + R_{\alpha \beta \lambda \mu ;\nu} + R_{\alpha \beta \nu \lambda ;\mu} == 0 $$

Applying the Ricci contraction then using the symmetries eventually simplifies this to
$$ (2R^\mu\!_\lambda - \delta^\mu\!_\lambda R)_{;\mu} = 0 $$

The implication for the Einstein tensor is
$$ G^{\alpha \beta}\!_{;\beta} = 0 $$

This means that the Einstein tensor has zero divergence.

```python

```