# Regression Discontinuity Design

We don't stop to think about it much, but it is impressive how smooth nature is. You can't grow a tree without first getting a bud, you can't teleport from one place to another, a wound takes its time to heal. Even in the social realm, smoothness seems to be the norm. You can't grow a business in one day, consistency and hard work are required to build wealth and it takes years before you learn how linear regression works. Under normal circumstances, nature is very cohesive and doesn't jump around much.

> When the intelligent and animal souls are held together in one embrace, they can be kept from separating.

\- Tao Te Ching, Lao Tzu.

Which means that **when we do see jumps and spikes, they are probably artificial** and often man made situations. These events are usually accompanied by counterfactuals to the normal way of things: if a weird thing happens, this gives us some insight into what would have happened if nature was to work in a different way. Exploring these artificial jumps is at the core of Regression Discontinuity Design.

The basic setup goes like this. Imagine that you have a treatment variable \\(T\\) and potential outcomes \\(Y_0\\) and \\(Y_1\\). The treatment T is a discontinuous function of an observed running variable \\(R\\) such that

$
D_i = \mathcal{1}\{R_i>c\}
$

In other words, this is saying that treatment is zero when \\(R\\) is below a threshold \\(c\\) and one otherwise. This means that we get to observe \\(Y_1\\) when \\(R>c\\) and \\(Y_0\\) when \\(R<c\\). To wrap our head around this, think about the potential outcomes as 2 functions that we can't observe entirely. Both \\(Y_0(R)\\) and \\(Y_1(R)\\) are there, we just can't see that. The threshold acts as a switch that allows us to see one or the other of those function, but never both, much like in the image below:

The idea of regression discontinuity is to compare the outcome just above and just below the threshold to identify the treatment effect at the threshold. This is called a **sharp RD** design, since the probability of getting the treatment jumps from 0 to 1 at the threshold, but we could also think about a **fuzzy RD** design, where the probability also jumps, but is a less dramatic manner.

## Is Alcohol Killing You?

A very relevant public policy question is what should be the minimal drinking age. Most countries, Brazil included, set it to be 18 year, but in the US (most states) it is currently 21. So, is it the case that the US is being overly prudent and that they should lower their minimal drinking age? Or is it the case that other countries should make their legal drinking age higher?

One way to look at this question is from a [mortality rate perspective (Carpenter and Dobkin, 2009)](https://www.aeaweb.org/articles?id=10.1257/app.1.1.164). From the public policy standpoint, one could argue that we should lower the mortality rate as much as possible. If alcohol consumption increases the mortality rate by a lot, we should avoid lowering the minimum drinking age. This would be consistent with the objective of lowering deaths caused by alcohol consumption.

To estimate the impacts of alcohol on death, we could use the fact that legal drinking age imposes a discontinuity on nature. In the US, those just under 21 years don't drink (or drink much less) while those just older than 21 do drink. This means that the probability of drinking jumps at 21 years and that is something we can explore with an RDD.

```python
import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
from matplotlib import style
from matplotlib import pyplot as plt
import seaborn as sns
import statsmodels.formula.api as smf

%matplotlib inline

style.use("fivethirtyeight")
```

To do so we can grab some mortality data aggregated by age. Each row is the average age of a group of people and the average mortality by all causes (`all`), by moving veiche accident (`mva`) and by suicide (`suicide`).

```python
drinking = pd.read_csv("./data/drinking.csv")
drinking.head()[["agecell", "all", "mva", "suicide"]]
```

      agecell
      all
      mva
      suicide

      0
      19.068493
      92.825400
      35.829327
      11.203714

      1
      19.150684
      95.100740
      35.639256
      12.193368

      2
      19.232876
      92.144295
      34.205650
      11.715812

      3
      19.315070
      88.427760
      32.278957
      11.275010

      4
      19.397260
      88.704940
      32.650967
      10.984314

Just to aid visibility (and for another important reason we will see later) we will centralize the running variable `agecell` at the threshold 21.

```python
drinking["agecell"] -= 21
```

If we plot the multiple outcome variables (`all`, `mva`, `suicide`) with the runing variable on the x axis, we get some visual cue about some sort of jump in mortality as we cross the legal drinking age.

```python
plt.figure(figsize=(8,8))
ax = plt.subplot(3,1,1)
drinking.plot.scatter(x="agecell", y="all", ax=ax)
plt.title("Death Cause by Age (Centered at 0)")

ax = plt.subplot(3,1,2, sharex=ax)
drinking.plot.scatter(x="agecell", y="mva", ax=ax)

ax = plt.subplot(3,1,3, sharex=ax)
drinking.plot.scatter(x="agecell", y="suicide", ax=ax);

```

There are some cues, but we need more than that. What exactly is the effect of drinking on mortality at the threshold? And what is the standard error on that estimate?

## RDD Estimation

The key assumption that RDD relies on is the smoothness of the potential outcome at the threshold. Formally, the limits of the potential outcomes as the running variable approaches the threshold from the right and from the left should be the same.

$$
\lim_{r \to c^-} E[Y_{ti}|R_i=r] = \lim_{r \to c^+} E[Y_{ti}|R_i=r]
$$

If this holds true, we can find the causal effect at the threshold

\begin{align}
\lim_{r \to c^+} E[Y_{ti}|R_i=r] - \lim_{r \to c^-} E[Y_{ti}|R_i=r]=&\lim_{r \to c^+} E[Y_{1i}|R_i=r] - \lim_{r \to c^-} E[Y_{0i}|R_i=r] \\
=& E[Y_{1i}|R_i=r] - E[Y_{0i}|R_i=r] \\
=& E[Y_{1i} - Y_{0i}|R_i=r]
\end{align}

This is, in its own way, a sort of Local Average Treatment Effect (LATE), since we can only know it at the threshold. In this setting, we can think of RDD as a local randomized trial. For those at the threshold, the treatment could have gone either way and, by chance, some people fell below the threshold, and some people fell above. In our example, at the same point in time, some people are just above 21 years and some people are just below 21. What determines this is if someone was born some days later or not, which is pretty random. For this reason, RDD provides a very compelling causal story. It is not the golden standard of RCT, but it is close.

Now, to estimate the treatment effect at the threshold, all we need to do is estimate both of the limits in the formula above and compare them. The simplest way to do that is by running a linear regression

To make it work, we interact a dummy for being above the threshold with the running variable

$
y_i = \beta_0 + \beta_1 r_i + \beta_2 \mathcal{1}\{r_i>c\} + \beta_3 \mathcal{1}\{r_i>c\} r_i
$

Essentially, this is the same as fitting a linear regression above the threshold and another below it. The parameter \\(\beta_0\\) is the intercept of the regression below the threshold and \\(\beta_0+\beta_2\\) is the intercept for the regression above the threshold.

Here is where the trick of centering the running variable at the threshold comes into play. After this pre-processing step, the threshold becomes zero. This causes the intercept \\(\beta_0\\) to be the predicted value at the threshold, for the regression below it. In other words, \\(\beta_0=\lim_{r \to c^-} E[Y_{ti}|R_i=r]\\). By the same reasoning, \\(\beta_0+\beta_2\\) is the limit of the outcome from above. Wich means, that

$
\lim_{r \to c^+} E[Y_{ti}|R_i=r] - \lim_{r \to c^-} E[Y_{ti}|R_i=r]=\beta_2=E[ATE|R=c]
$

Here is what this looks like in code for the case where we want to estimate the effect of alcohol consumption on death by all causes at 21 years.

```python
rdd_df = drinking.assign(threshold=(drinking["agecell"] > 0).astype(int))

model = smf.wls("all~agecell*threshold", rdd_df).fit()

model.summary().tables[1]
```

          	             coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	            93.6184	     0.932	   100.399	  0.000	    91.739	    95.498

  agecell	               0.8270	     0.819	     1.010	  0.318	    -0.823	     2.477

  threshold	             7.6627	     1.319	     5.811	  0.000	     5.005	    10.320

  agecell:threshold	    -3.6034	     1.158	    -3.111	  0.003	    -5.937	    -1.269

This model is telling us that mortality increases by 7.6627 points with the consumption of alcohol. Another way of putting this is that alcohol increases the chance of death by all causes by 8% ((7.6627+93.6184)/93.6184). Notice that this also gives us standard errors for our causal effect estimate. In this case, the effect is statistically significant, since the p-value is below 0.01.

If we want to verify this model visually, we can show the predicted values on the data that we have. You can see that it is as though we had 2 regression models: one for those above the threshold and one for below it.

```python
ax = drinking.plot.scatter(x="agecell", y="all", color="C0")
drinking.assign(predictions=model.fittedvalues).plot(x="agecell", y="predictions", ax=ax, color="C1")
plt.title("Regression Discontinuity");
```

If we do the same for the other causes, this is what we get.

```python
plt.figure(figsize=(8,8))

for p, cause in enumerate(["all", "mva", "suicide"], 1):
    ax = plt.subplot(3,1,p)
    drinking.plot.scatter(x="agecell", y=cause, ax=ax)
    m = smf.wls(f"{cause}~agecell*threshold", rdd_df).fit()
    ate_pct = 100*((m.params["threshold"] + m.params["Intercept"])/m.params["Intercept"] - 1)
    drinking.assign(predictions=m.fittedvalues).plot(x="agecell", y="predictions", ax=ax, color="C1")
    plt.title(f"Impact of Alcohol on Death: {np.round(ate_pct, 2)}%")

plt.tight_layout()
```

RDD is telling us that alcohol increases the chance of deth by suicide and car accidents by 15%, wich is a pretty significant ammount. These results are compelling arguments to not lower the drinking age, if we want to minimize mortality rates.

### Kernel Weighting

Regression Discontinuity relies heavily on the extrapolations properties of linear regression. Since we are looking at the values at the beginning and end of 2 regression lines, we better get those limits right. What can happen is that regression might focus too much on fitting the other data points at the cost of a poor fit at the threshold. If this happens, we might get the wrong measure of the treatment effect.

One way to solve this is to give higher weights for the points that are closer to the threshold. There are many ways to do this, but a popular one is to reweight the samples with the **triangular kernel**

$
K(R, c, h) = \mathcal{1}\{|R-c| \leq h\} * \bigg(1-\frac{|R-c|}{h}\bigg)
$

The first part of this kernel is an indicator function to whether we are close to the threshold. How close? This is determined by a bandwidth parameter \\(h\\). The second part of this kernel is a weighting function. As we move away from the threshold, the weights get smaller and smaller. These weights are divided by the bandwidth. If the bandwidth is large, the weights get smaller at a slower rate. If the bandwidth is small, the weights quickly go to zero.

To make it easier to understand, here is what the weights look like for this kernel applied to our problem. I've set the bandwidth to be 1 here, meaning we will only consider data from people that are no older than 22 years and no younger than 20 years.

```python
def kernel(R, c, h):
    indicator = (np.abs(R-c) <= h).astype(float)
    return indicator * (1 - np.abs(R-c)/h)
```

```python
plt.plot(drinking["agecell"], kernel(drinking["agecell"], c=0, h=1))
plt.xlabel("agecell")
plt.ylabel("Weight")
plt.title("Kernel Weight by Age");
```

If we apply these weights to our original problem, the impact of alcohol gets bigger, at least for all causes. It jumps from 7.6627 to 9.7004. The result remains very significant. Also, notice that I'm using `wls` instead of `ols`

```python
model = smf.wls("all~agecell*threshold", rdd_df,
                weights=kernel(drinking["agecell"], c=0, h=1)).fit()

model.summary().tables[1]
```

          	             coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	            93.2002	     0.731	   127.429	  0.000	    91.726	    94.674

  agecell	               0.4109	     1.789	     0.230	  0.819	    -3.196	     4.017

  threshold	             9.7004	     1.034	     9.378	  0.000	     7.616	    11.785

  agecell:threshold	    -7.1759	     2.531	    -2.835	  0.007	   -12.276	    -2.075

```python
ax = drinking.plot.scatter(x="agecell", y="all", color="C0")
drinking.assign(predictions=model.fittedvalues).plot(x="agecell", y="predictions", ax=ax, color="C1")
plt.title("Regression Discontinuity (Local Regression)");
```

And here is what it looks like for the other causes of death. Notice how the regression on the right is more negatively sloped since it disconsiders the right most points.

```python
plt.figure(figsize=(8,8))
weights = kernel(drinking["agecell"], c=0, h=1)

for p, cause in enumerate(["all", "mva", "suicide"], 1):
    ax = plt.subplot(3,1,p)
    drinking.plot.scatter(x="agecell", y=cause, ax=ax)
    m = smf.wls(f"{cause}~agecell*threshold", rdd_df, weights=weights).fit()
    ate_pct = 100*((m.params["threshold"] + m.params["Intercept"])/m.params["Intercept"] - 1)
    drinking.assign(predictions=m.fittedvalues).plot(x="agecell", y="predictions", ax=ax, color="C1")
    plt.title(f"Impact of Alcohol on Death: {np.round(ate_pct, 2)}%")

plt.tight_layout()
```

With the exception of suicide, it looks like adding the kernel weight made the negative impact on alcohol bigger. Once again, if we want to minimize the death rate, we should NOT recommend lowering the legal drinking age, since there is a clear impact of alcohol on the death rates.

This simple case covers what happens when regression discontinuity design works perfectly. Next, we will see some diagnostics that we should run in order to check how much we can trust RDD and talk about a topic that is very dear to our heart: the effect of education on earnings.

## Sheepskin Effect and Fuzzy RDD

When it comes to the effect of education on earnings, there are two major views in economics. The first one is the widely known argument that education increases human capital, increasing productivity and thus, earnings. In this view, education actually changes you for the better. Another view is that education is simply a signaling mechanism. It just puts you through all these hard tests and academic tasks. If you can make it, it signals to the market that you are a good employee. In this way, education doesn't make you more productive. It only tells the market how productive you have always been. What matters here is the diploma. If you have it, you will be paid more. We refer to this as the **sheepskin effect**, since diplomas were printed in sheepskin in the past.

To test this hypothesis, [Clark and Martorell](https://faculty.smu.edu/millimet/classes/eco7321/papers/clark%20martorell%202014.pdf) used regression discontinuity to measure the effect of graduating 12th grade on earnings. In order to do that, they had to think about some running variable where students that fall above it graduate and those who fall below it, don't. They found such data in the Texas education system.

In order to graduate in Texas, one has to pass an exam. Testing starts at 10th grade and students can do it multiple times, but eventually, they face a last chance exam at the end of 12th grade. The idea was to get data from students who took those last chance exams and compare those that had barely failed it to those that barely passed it. These students will have very similar human capital, but different signaling credentials. Namely, those that barely passed it, will receive a diploma.

```python
sheepskin = pd.read_csv("./data/sheepskin.csv")[["avgearnings", "minscore", "receivehsd", "n"]]
sheepskin.head()
```

      avgearnings
      minscore
      receivehsd
      n

      0
      11845.086
      -30.0
      0.416667
      12

      1
      9205.679
      -29.0
      0.387097
      31

      2
      8407.745
      -28.0
      0.318182
      44

      3
      11114.087
      -27.0
      0.377778
      45

      4
      10814.624
      -26.0
      0.306667
      75

One again, this data is grouped by the running variable. It contains not only the running variable (`minscore`, already centered at zero) and the outcome (`avgearnings`), but it also has the probability of receiving a diploma in that score cell and the size of the call (`n`). So, for example, out of the 12 students in the cell -30 below the score threshold, only 5 were able to get the diploma (12 * 0,416).

This means that there is some slippage in the treatment assignment. Some students that are below the passing threshold managed to get the diploma anyway. Here, the regression discontinuity is **fuzzy**, rather than sharp. Notice how the probability of getting the diploma doesn't jump from zero to one at the threshold. But it does jump from something like 50% to 90%.

```python
sheepskin.plot.scatter(x="minscore", y="receivehsd", figsize=(10,5))
plt.xlabel("Test Scores Relative to Cut off")
plt.ylabel("Fraction Receiving Diplomas")
plt.title("Last-chance Exams");
```

We can think of fuzzy RD as a sort of non compliance. Passing the threshold should make everyone receive the diploma, but some students, the never takers, don't get it. Likewise, being below the threshold should prevent you from getting a diploma, but some students, the allways takers, manage to get it anyway.

Just like when we have the potential outcome, we have the potential treatment status in this situation. \\(T_1\\) is the treatment everyone would have received had they been above the threshold. \\(T_0\\) is the treatment everyone would have received had they been below the threshold. As you've might have noticed, we can think of the **threshold as an Instrumental Variable**. Just as in IV, if we naively estimate the treatment effect, it will be biased towards zero.

The probability of treatment being less than one, even above the threshold, makes the outcome we observe less than the true potential outcome \\(Y_1\\). By the same token, the outcome we observe below the threshold is higher than the true potential outcome \\(Y_0\\). This makes it look like the treatment effect at the threshold is smaller than it actually is and we will have to use IV techniques to correct for that.

Just like when we've assumed smoothness on the potential outcome, we now assume it for the potential treatment. Also, we need to assume monotonicity, just like in IV. In case you don't remember, it states that \\(T_{i1}>T_{i0} \ \forall i\\). This means that crossing the threshold from the left to the right only increases your chance of getting a diploma (or that there are no defiers). With these 2 assumptions, we have a Wald Estimator for LATE.

$$
\dfrac{\lim_{r \to c^+} E[Y_i|R_i=r] - \lim_{r \to c^-} E[Y_i|R_i=r]}{\lim_{r \to c^+} E[T_i|R_i=r] - \lim_{r \to c^-} E[T_i|R_i=r]} = E[Y_{1i} - Y_{0i} | T_{1i} > T_{0i}, R_i=c]
$$

Notice how this is a local estimate in two senses. First, it is local because it only gives the treatment effect at the threshold \\(c\\). This is the RD locality. Second, it is local because it only estimates the treatment effect for the compliers. This is the IV locality.

To estimate this, we will use 2 linear regression. The numerator can be estimated just like we've done before. To get the denominator, we simply replace the outcome with the treatment. But first, let's talk about a sanity check we need to run to make sure we can trust our RDD estimates.

### The McCrary Test

One thing that could break our RDD argument is if people can manipulate where they stand at the threshold. In the sheepskin example this could happen if students just below the threshold found a way around the system to increase their test score by just a bit. Another example is when you need to be below a certain income level to get a government benefit. Some families might lower their income on purpose, just to be just eligible for the program.

In these sorts of situations, we tend to see a phenomenon called bunching on the density of the running variable. This means that we will have a lot of entities just above or just below the threshold. To check for that, we can plot the density function of the running variable and see if there are any spikes around the threshold. For our case, the density is given by the `n` column in our data.

```python
plt.figure(figsize=(8,8))

ax = plt.subplot(2,1,1)
sheepskin.plot.bar(x="minscore", y="n", ax=ax)
plt.title("McCrary Test")
plt.ylabel("Smoothness at the Threshold")

ax = plt.subplot(2,1,2, sharex=ax)
sheepskin.replace({1877:1977, 1874:2277}).plot.bar(x="minscore", y="n", ax=ax)
plt.xlabel("Test Scores Relative to Cut off")
plt.ylabel("Spike at the Threshold");
```

The first plot shows how our data density looks like. As we can see, there are no spikes around the threshold, meaning there is no bunching. Students are not manipulating where they fall on the threshold. Just for illustrative purposes, the second plot shows what bunching would look like if students could manipulate where they fall on the threshold. We would see a spike in the density for the cells just above the threshold, since many students would be on that cell, barely passing the exam.

Getting this out of the way, we can go back to estimate the sheepskin effect. As I've said before, the numerator of the Wald estimator can be estimated just like we did in the Sharp RD. Here, we will use as weight the kernel with a bandwidth of 15. Since we also have the cell size, we will multiply the kernel by the sample size to get a final weight for the cell.

```python
sheepsking_rdd = sheepskin.assign(threshold=(sheepskin["minscore"]>0).astype(int))
model = smf.wls("avgearnings~minscore*threshold",
                sheepsking_rdd,
                weights=kernel(sheepsking_rdd["minscore"], c=0, h=15)*sheepsking_rdd["n"]).fit()

model.summary().tables[1]
```

           	             coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	           1.399e+04	    83.678	   167.181	  0.000	  1.38e+04	  1.42e+04

  minscore	             181.6636	    16.389	    11.084	  0.000	   148.588	   214.739

  threshold	            -97.7571	   145.723	    -0.671	  0.506	  -391.839	   196.325

  minscore:threshold	    18.1955	    30.311	     0.600	  0.552	   -42.975	    79.366

This is telling us that the effect of a diploma is -97.7571, but this is not statistically significant (P-value of 0.5). If we plot these results, we get a very continuous line at the threshold. More educated people indeed make more money, but there isn't a jump at the point where they receive the 12th grade diploma. This is an argument in favor of the view that says that education increases earnings by making people more productive, rather than being just a signal to the marker. In other words, there is no sheepskin effect.

```python
ax = sheepskin.plot.scatter(x="minscore", y="avgearnings", color="C0")
sheepskin.assign(predictions=model.fittedvalues).plot(x="minscore", y="predictions", ax=ax, color="C1", figsize=(8,5))
plt.xlabel("Test Scores Relative to Cutoff")
plt.ylabel("Average Earnings")
plt.title("Last-chance Exams");
```

However, as we know from the way non compliance bias works, this result is biased towards zero. To correct for that, we need to scale it by the first stage and get the Wald estimator. Unfortunately, there isn't a good Python implementation for this, so we will have to do it manually and use bootstrap to get the standard errors.

The code below runs the numerator of the Wald estimator just like we did before and also constructs the denominator by replacing the target variable with the treatment variable `receivehsd`. The final step just divides the numerator by the denominator.

```python
def wald_rdd(data):
    weights=kernel(data["minscore"], c=0, h=15)*data["n"]
    denominator = smf.wls("receivehsd~minscore*threshold", data, weights=weights).fit()
    numerator = smf.wls("avgearnings~minscore*threshold", data, weights=weights).fit()
    return numerator.params["threshold"]/denominator.params["threshold"]
```

```python
from joblib import Parallel, delayed

np.random.seed(45)
bootstrap_sample = 1000
ates = Parallel(n_jobs=4)(delayed(wald_rdd)(sheepsking_rdd.sample(frac=1, replace=True))
                          for _ in range(bootstrap_sample))
ates = np.array(ates)
```

With the bootstrap samples, we can plot the distribution of ATEs and see where the 95% confidence interval is.

```python
sns.distplot(ates, kde=False)
plt.vlines(np.percentile(ates, 2.5), 0, 100, linestyles="dotted")
plt.vlines(np.percentile(ates, 97.5), 0, 100, linestyles="dotted", label="95% CI")
plt.title("ATE Bootstrap Distribution")
plt.xlim([-10000, 10000])
plt.legend();
```

As you can see, even when we scale the effect by the first stage, it is still not statistically different from zero. This means that education doesn't increase earnings by a simple sheepskin effect, but rather by increasing one's productivity.

## Key Ideas

We learned how to take advantage of artificial discontinuities to estimate causal effects. The idea is that we will have some artificial threshold that makes the probability of treatment jump. One example that we saw was how age makes the probability of drinking jump at 21 years. We could use that to estimate the impact of drinking on mortality rate. We use the fact that very close to the threshold, we have something close to a randomized trial. Entities very close to the threshold could have gone either way and what determines where they've landed is essentially random. With this, we can compare those just above and just below to get the treatment effect. We saw how we could do that with weighted linear regression using a kernel and how this even gave us, for free, standard errors for our ATE.

Then, we look at what would happen in the fuzzy RD design, where we have non compliance. We saw how we could approach the situation much like we did with IV.

## References

I like to think of this entire book as a tribute to Joshua Angrist, Alberto Abadie and Christopher Walters for their amazing Econometrics class. Most of the ideas here are taken from their classes at the American Economic Association. Watching them is what is keeping me sane during this tough year of 2020.
* [Cross-Section Econometrics](https://www.aeaweb.org/conference/cont-ed/2017-webcasts)
* [Mastering Mostly Harmless Econometrics](https://www.aeaweb.org/conference/cont-ed/2020-webcasts)

I'll also like to reference the amazing books from Angrist. They have shown me that Econometrics, or 'Metrics as they call it, is not only extremely useful but also profoundly fun.

* [Mostly Harmless Econometrics](https://www.mostlyharmlesseconometrics.com/)
* [Mastering 'Metrics](https://www.masteringmetrics.com/)

Other important reference is Miguel Hernan and Jamie Robins' book. It has been my trustworthy companion in the most thorny causal questions I had to answer.

* [Causal Inference Book](https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/)

## Contribute

Causal Inference for the Brave and True is an open-source material on causal inference, the statistics of science. It uses only free software, based in Python. Its goal is to be accessible monetarily and intellectually.
If you found this book valuable and you want to support it, please go to [Patreon](https://www.patreon.com/causal_inference_for_the_brave_and_true). If you are not ready to contribute financially, you can also help by fixing typos, suggesting edits or giving feedback on passages you didn't understand. Just go to the book's repository and [open an issue](https://github.com/matheusfacure/python-causality-handbook/issues). Finally, if you liked this content, please share it with others who might find it useful and give it a [star on GitHub](https://github.com/matheusfacure/python-causality-handbook/stargazers).

```python

```


# Assignment 1
# COMP 652: Machine Learning

## Samin Yeasar Arnob
## McGill ID: 260800927

```python

```

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
```

#########################################################################################

# 1.  regression, overfitting and regularization

##########################################################################################

## Answer 1(a)

```python
df1 = pd.read_csv('hw1-q1x.csv',delim_whitespace =True,header=None,dtype=np.float)
df2 = pd.read_csv('hw1-q1y.csv',delim_whitespace =True,header=None,dtype=np.float)
Data=pd.concat([df1,df2],axis=1)
Data.columns=['X1','X2','Y']
no_data = df1.shape[0]
```

```python
#Data centering
Cen_Data = Data - Data.mean(axis=0) #axis=0 means row wise, rows are example here
```

```python
# splitting training-testing data

from sklearn.model_selection import train_test_split
train_data, test_data = train_test_split(Cen_Data, test_size=0.2,random_state=34) #randomly split train-test

feat_no = train_data.shape[1] - 1 # last column is for labels

Xm_train = train_data.iloc[:,:feat_no].values
y_train = train_data.iloc[:,feat_no:].values

Xm_test = test_data.iloc[:,:feat_no].values
y_test = test_data.iloc[:,feat_no:].values

print(Xm_train.shape)
print(Xm_test.shape)
```

    (80, 2)
    (20, 2)

## Report:

* I have centered the data which is basically remove means from input and output
$\{(x_i-\overline{x},y_i-\overline{y})\}^m_{i=1}$
* As centering makes the bias terms zero, I didn't add the bias term

## Answer 1(b)

```python
# manual code to compute W
def computeW (Xm_train,y_train,feat_no,lmbda):
    a = np.dot(Xm_train.T,Xm_train) + lmbda*np.eye(feat_no)
    a = np.linalg.pinv(a)
    W = np.dot(a, np.dot(Xm_train.T,y_train))
    return W
```

```python
# create lambda parameter vector
lmbda = []
l = 0.001
while l <= 10**1:
    l = l*10
    if l==0 :
        break
    else :
        lmbda = np.append(lmbda, l)
lmbda = lmbda.reshape(1,-1)
print('lmbda:',lmbda)
```

    lmbda: [[1.e-02 1.e-01 1.e+00 1.e+01 1.e+02]]

```python
# create lambda parameter vector
lmbda = np.array([])
l = 0.01
while l <= 10**5:
    lmbda = np.append(lmbda, l)
    l = l*10
lmbda = lmbda.reshape(1,-1)
print('lmbda:',lmbda)
```

    lmbda: [[1.e-02 1.e-01 1.e+00 1.e+01 1.e+02 1.e+03 1.e+04 1.e+05]]

```python
lmbda[0,0]
```

    0.01

```python
# lmbda has to be numpy array
def RMSEresult(Xm_train,y_train,Xm_test,y_test,lmbda,norm):
    from sklearn.metrics import mean_squared_error
    from sklearn import datasets, linear_model

    feat_no = Xm_train.shape[1]
    #W_vec = np.zeros( ( 2,len(lmbda) ) )
    W_vec = pd.DataFrame( np.zeros(( feat_no,lmbda.shape[1] )) )

    RMSE_traindata = []
    RMSE_testdata = []

    Hypothesis_train = {}
    Hypothesis_test = {}

    for i in range (lmbda.shape[1]):

    #####  manual code to compute W ########
        #W = computeW (Xm_train,y_train,feat_no,lmbda[i])
        #W_vec[i] = W
        #Hypothesis_train = np.dot(Xm_train,W)
        #Hypothesis_test = np.dot(Xm_test,W)

    ##### using built-in model ######

        if norm == 'l2':
            clf = linear_model.Ridge(alpha=lmbda[0,i])
        if norm == 'l1' :
            clf = linear_model.Lasso(alpha=lmbda[0,i])

        clf.fit(Xm_train, y_train)

        H_train = clf.predict (Xm_train)
        H_test = clf.predict (Xm_test)

        Hypothesis_train['lmbda_%d'%i] = H_train.reshape(-1,1)
        Hypothesis_test['lmbda_%d'%i] = H_test.reshape(-1,1)

        W=np.reshape(clf.coef_,(-1,1))
        W_vec[i] = W

        RMSE_traindata = np.append ( RMSE_traindata , np.sqrt( mean_squared_error(y_train, H_train) ) )
        RMSE_testdata = np.append ( RMSE_testdata , np.sqrt( mean_squared_error(y_test, H_test) ) )

    return (RMSE_traindata,RMSE_testdata, W_vec,Hypothesis_train,Hypothesis_test)
```

```python
RMSE_traindata, RMSE_testdata,W_vec,Hypothesis_train,Hypothesis_test = RMSEresult(Xm_train,y_train,Xm_test,y_test,lmbda,'l2')
```

```python
# Graph1: RMSE for training data and testing data

#plt.loglog(lmbda,RMSE_testdata,label='testing data')
#plt.loglog(lmbda,RMSE_traindata,label='training data')

plt.semilogx(RMSE_traindata,label='training data')
plt.semilogx(RMSE_testdata,label='testing data')
plt.xlabel(' lambda parameter ')
plt.ylabel('RMSE')
plt.legend()
#plt.savefig('D:\\University materials\\Winter 2018\\Applied ML\\winter 2018\\Assignments\\Kaggle\\my codes\\train_validation_H3.png')
plt.show()
```

```python
# Graph2: L2 norm of weight vectors

W_L2norm = np.sum(W_vec**2,axis=0)
plt.semilogx( W_L2norm.values )
plt.xlabel ('lambda parameter')
plt.ylabel ('L2 norm of Weight vector')
plt.show()
```

```python
#Graph3: Actual values of weight obtained

plt.semilogx( W_vec.loc[[0], : ].values[0] ,label = 'W1')
plt.semilogx( W_vec.loc[[1], : ].values[0] ,label = 'W2')
plt.xlabel ('lambda parameter')
plt.ylabel ('Weight values')
plt.legend()
plt.show()
```

## Report:
* For Lambda, $\lambda=0$ means, no regularization.
* We see that, at $\lambda=0$ we get min training error and  as we increase the value of $\lambda$ training error increases but for testing data as we increse lambda the error decreases and after $\lambda$=10 RMSE start to increase.

Thus at $\lambda$=0, when no regularization was used, training data was overfitted.

```python
print('Training data gets least RMSE for lambda:',lmbda[0,np.argwhere(RMSE_traindata == min(RMSE_traindata)) ])
print('Testing data gets least RMSE for lambda:',lmbda[0,np.argwhere(RMSE_testdata == min(RMSE_testdata)) ])
```

    Training data gets least RMSE for lambda: [[0.01]]
    Testing data gets least RMSE for lambda: [[0.01]]

## Answer 1(c)

```python
def ComputeValid_VS_lmbda(Xm_train,y_train,kfold,lmbda,norm):
    dlen = Xm_train.shape[0]
    inc = np.int(dlen/kfold)

    RMSE_traindata = np.zeros([kfold,lmbda.shape[1]])
    RMSE_validdata = np.zeros([kfold,lmbda.shape[1]])

    Hypothesis_valid = []
    Hypothesis_train = []

    valid_score = np.zeros([])

    for i in range (kfold):
        # mask the data to be used as validation
        #############################################################################
        msk = np.isin (np.linspace(1,dlen,dlen), np.linspace(1+inc*i,inc+inc*i,inc))
        #############################################################################
        train_X = Xm_train[~msk]
        train_y = y_train[~msk]

        valid_X = Xm_train[msk]
        valid_y = y_train[msk]

        feat_on = train_X.shape[1]

        RMSE_traindata[i,:], RMSE_validdata[i,:], W_vec , h_train , h_valid = RMSEresult(train_X,train_y,valid_X,valid_y,lmbda,norm)

        #Hypothesis_valid.update(h_valid)
        #Hypothesis_train.update(h_train)
       # for k in range (len(h_train)):
         #   Hypothesis_valid['lmbda_%d'%k].append (h_valid['lmbda_%d'%k])
        #    Hypothesis_train['lmbda_%d'%k].append (h_train['lmbda_%d'%k])

        Hypothesis_valid.append ( h_valid)
        Hypothesis_train.append ( h_valid)
        #Hypothesis_train = np.append ( Hypothesis_train , h_train)

        #train_score = np.append(train_score ,RMSE_traindata)
        #valid_score = valid_score + RMSE_validdata

    result = np.zeros((y_train.shape[0],lmbda.shape[1]))
    result = np.matrix(result)
    for k in range(lmbda.shape[1]):
        r = 0
        for m in range(kfold):

            result[m*inc:inc+r,k] = Hypothesis_valid[m]['lmbda_%d'%k]
            r+=inc

    #valid_score = np.mean(RMSE_validdata,axis=0)
    #print('Mean validation score for different lambda',valid_score)
    #return RMSE_traindata , RMSE_validdata , Hypothesis_train.reshape(dlen-np.int(inc),kfold), Hypothesis_valid.reshape(np.int(inc),kfold)
    return RMSE_traindata , RMSE_validdata , result

```

```python
kfold=5
RMSE_traindata, RMSE_validdata,result = ComputeValid_VS_lmbda(Xm_train,y_train,kfold,lmbda,norm='l1')
```

```python
####################################################
# plot RMSE for 5 fold validation data
####################################################

for i in range (RMSE_validdata.shape[0]):
    plt.plot(RMSE_validdata[i,:],label='valid data batch %.1f' %(i+1) )
    plt.xlabel(' lambda parameter ')
    plt.ylabel('RMSE')
    plt.legend()
plt.show()

```

```python
################################################
#plot RMSE for 5 fold training data
################################################

for i in range (RMSE_traindata.shape[0]):
    plt.plot(RMSE_traindata[i,:],label='train data batch %.1f' %(i+1) )
    plt.xlabel(' lambda parameter ')
    plt.ylabel('RMSE')
    plt.legend()
plt.show()
```

```python
valid_score = np.mean(RMSE_validdata,axis=0)
print('Average Validation RMSE Score',valid_score)
```

    Average Validation RMSE Score [16.07387259 16.07091321 16.04844751 16.35129533 32.58357559 32.58357559
     32.58357559 32.58357559]

```python
for i in range (RMSE_traindata.shape[0]):
    print('Best Value of Lambda for Training data batch %d = ' %i,  lmbda[ 0, np.argmin(RMSE_traindata[i,:]) ] )

```

    Best Value of Lambda for Training data batch 0 =  0.01
    Best Value of Lambda for Training data batch 1 =  0.01
    Best Value of Lambda for Training data batch 2 =  0.01
    Best Value of Lambda for Training data batch 3 =  0.01
    Best Value of Lambda for Training data batch 4 =  0.01

```python
for i in range (RMSE_validdata.shape[0]):
    print('Best Value of Lambda for Validation data batch %d = ' %i,  lmbda[ 0, np.argmin(RMSE_validdata[i,:]) ] )
```

    Best Value of Lambda for Validation data batch 0 =  0.01
    Best Value of Lambda for Validation data batch 1 =  10.0
    Best Value of Lambda for Validation data batch 2 =  10.0
    Best Value of Lambda for Validation data batch 3 =  10.0
    Best Value of Lambda for Validation data batch 4 =  0.01

## Report:

* best value for lambda for training data is same as 1(b) but it varies for different validation batch

## Answer 1(d)

```python
y_train_sorted = np.sort(y_train,axis=0)
msk = y_train.argsort(axis=0)
Xm_train_sorted = Xm_train[msk]                 # dimension of Xm_sort becomes (80,1,2)
Xm_train_sorted = Xm_train_sorted.reshape(80,2)       # we correct the dimension
```

```python
kfold=5
sorted_RMSE_traindata, sorted_RMSE_validdata , _ = ComputeValid_VS_lmbda(Xm_train_sorted,y_train_sorted,kfold,lmbda,'l2')
```

```python
################################################
#plot RMSE for 5 fold training data
################################################

for i in range (sorted_RMSE_traindata.shape[0]):
    plt.plot(sorted_RMSE_traindata[i,:],label='train data batch %.1f' %(i+1) )
    plt.xlabel(' lambda parameter ')
    plt.ylabel('RMSE')
    plt.legend()
plt.show()
```

```python
####################################################
# plot RMSE for 5 fold validation data
####################################################

for i in range (sorted_RMSE_validdata.shape[0]):
    plt.plot(sorted_RMSE_validdata[i,:],label='valid data batch %.1f' %(i+1) )
    plt.xlabel(' lambda parameter ')
    plt.ylabel('RMSE')
    plt.legend()
plt.show()
```

```python
for i in range (sorted_RMSE_traindata.shape[0]):
    print('Best Value of Lambda for Training data batch %d = ' %i,  lmbda[ 0, np.argmin(sorted_RMSE_traindata[i,:]) ] )

```

    Best Value of Lambda for Training data batch 0 =  0.01
    Best Value of Lambda for Training data batch 1 =  0.01
    Best Value of Lambda for Training data batch 2 =  0.01
    Best Value of Lambda for Training data batch 3 =  0.01
    Best Value of Lambda for Training data batch 4 =  0.01

```python
for i in range (sorted_RMSE_validdata.shape[0]):
    print('Best Value of Lambda for Validation data batch %d = ' %i,  lmbda[ 0, np.argmin(sorted_RMSE_validdata[i,:]) ] )
```

    Best Value of Lambda for Validation data batch 0 =  0.01
    Best Value of Lambda for Validation data batch 1 =  100.0
    Best Value of Lambda for Validation data batch 2 =  10000.0
    Best Value of Lambda for Validation data batch 3 =  100.0
    Best Value of Lambda for Validation data batch 4 =  0.01

## Report
* From traing and validation output it's clear that data, for the first batch of training batch the model was most overfitted. For batch 1 data, training give the least RMSE and validation gives the highest RMSE score across the lambda range
* So we can say, when data X have low value they are more scattered (have high variance)

## Answer 1(e)

```python
# given
mu=np.linspace(-1,1,5).reshape(1,-1)
print(mu)
print(mu.shape)
```

    [[-1.  -0.5  0.   0.5  1. ]]
    (1, 5)

```python
# given
sigma_sqr = np.array([0.1,0.5,1,5]).reshape(1,-1)
```

$ \phi_j(X) = \exp^{-(\frac{(X-\mu_j)^2}{\sigma^2})}$
* $\mu_j$ controls the position along x-axis and varies from (-1,1).
* $\sigma$ controls width/spacing

```python
np.tile(mu,Xm_train.shape[1])
```

    array([[-1. , -0.5,  0. ,  0.5,  1. , -1. , -0.5,  0. ,  0.5,  1. ]])

```python
def gaussian_basis (X,mu,sigma_sqr):
    x = X.repeat(mu.shape[1],axis=1)
    mu = np.tile(mu,X.shape[1])
    psi  = np.exp ( (-1)*( (x-mu)**2/(sigma_sqr) ) )# or X[:,np.newaxis] - w
    return psi
```

## Answer 1(f)

```python
# no regularization
lmbda  = np.array([0])
lmbda = lmbda.reshape(1,-1)
print(lmbda.shape)
```

    (1, 1)

```python

RMSE_GaussBasis_train = []
RMSE_GaussBasis_test = []
W_vec = {}
for l in range (sigma_sqr.shape[1]):
    phi_train = gaussian_basis (Xm_train,mu,sigma_sqr[0,l])
    phi_test = gaussian_basis (Xm_test,mu,sigma_sqr[0,l])

    train, test,W_vec[str(sigma_sqr[0,l])],_,_ = RMSEresult(phi_train,y_train,phi_test,y_test,lmbda,norm='l2')
    RMSE_GaussBasis_train = np.append(RMSE_GaussBasis_train,train)
    RMSE_GaussBasis_test = np.append(RMSE_GaussBasis_test,test)

RMSE_GaussBasis_train = RMSE_GaussBasis_train
RMSE_GaussBasis_test = RMSE_GaussBasis_test
RMSE_traindata, RMSE_testdata,W_vec , _ ,_ = RMSEresult(Xm_train,y_train,Xm_test,y_test,lmbda,norm='l2')
```

```python
'''
RMSE_GaussBasis_train = {}
RMSE_GaussBasis_test = {}
W_vec = {}
for l in range (sigma_sqr.shape[1]):
    phi_train = gaussian_basis (Xm_train,mu,sigma_sqr[0,l])
    phi_test = gaussian_basis (Xm_test,mu,sigma_sqr[0,l])

    RMSE_GaussBasis_train[str(sigma_sqr[0,l])], RMSE_GaussBasis_test[str(sigma_sqr[0,l])],W_vec[str(sigma_sqr[0,l])] = RMSEresult(phi_train,y_train,phi_test,y_test,lmbda)

    #RMSE_validdata[str(l)] = ComputeValid_VS_lmbda(phi_train,y_train,kfold,lmbda)
'''
```

    '\nRMSE_GaussBasis_train = {}\nRMSE_GaussBasis_test = {}\nW_vec = {}\nfor l in range (sigma_sqr.shape[1]):\n    phi_train = gaussian_basis (Xm_train,mu,sigma_sqr[0,l])\n    phi_test = gaussian_basis (Xm_test,mu,sigma_sqr[0,l])\n    \n    RMSE_GaussBasis_train[str(sigma_sqr[0,l])], RMSE_GaussBasis_test[str(sigma_sqr[0,l])],W_vec[str(sigma_sqr[0,l])] = RMSEresult(phi_train,y_train,phi_test,y_test,lmbda)\n    \n    #RMSE_validdata[str(l)] = ComputeValid_VS_lmbda(phi_train,y_train,kfold,lmbda)\n'

```python
plt.plot(RMSE_GaussBasis_train,label='Train error for gaussina basis')
plt.plot(RMSE_GaussBasis_test,label='Test error for gaussina basis')
plt.plot(RMSE_traindata[0].repeat(len(sigma_sqr[0])),label='Train error for 1(b)')
plt.plot(RMSE_testdata[0].repeat(len(sigma_sqr[0])),label='Test error for 1(b)')
plt.xlabel(' sigma^2 ')
plt.ylabel('RMSE')
plt.legend()
plt.show()
```

## Report
* The two constant lines drawn are the Train and test RMSE ($\lambda$=0, no regularization) calculated at 1(b).

* more complex (fits well = higher $\sigma^2$) == > high variance
* simple (doesn't fit well ) == > biased solution

```python
print('Higher variance as we increse sigma^2',RMSE_GaussBasis_test - RMSE_GaussBasis_train )
```

    Higher variance as we increse sigma^2 [-0.22235882 -0.75606063  0.74072019  0.46067255]

## Answer 1(g)

```python

```

In order to derive a learning algorithm that computes both the placement of the mean $\mu$
of the basis function, we need to update the weight $w$ and $\mu$ iteratively so as to performing
two gradient descent algorithms. The cost function that we need to minimize with
respect to $\mu$ is derived as follows. Considering the cost function of logistic regression,

\begin{align}
    J(W) = & - \frac{1}{m} \Bigg[  \sum_1^m y_i log(h_i) + (1-y_i) log(1-h_i)  \Bigg]
\end{align}

The derivative of the cost function J (w) with respect to  is calculated as follows.

\begin{align}
    \frac{\partial J}{\partial \mu} = & - \frac{1}{m} \Bigg[  \sum_1^m y_i w \frac{\partial \phi(x_i)}{\partial \mu} + \frac{(1-y_i)}{1-h} (-w) \frac{\partial \phi(x_i)}{\partial \mu} \Bigg]  \\
\end{align}

Now,

\begin{align}
    \frac{\partial \phi(x)}{\partial \mu}  & =  \frac{\partial  e^{-\frac{(x-\mu)^2}{2\sigma^2}} } {\partial \mu}  \nonumber \\
                                           &  =  e^{-\frac{(x-\mu)^2}{2\sigma^2}} \times 2 \times \frac{\partial -\frac{(x-\mu)}{2\sigma^2} } {\partial \mu} \nonumber \\
                                           &  =  \frac {e^{-\frac{(x-\mu)^2}{2\sigma^2}} }{\sigma^2} \nonumber \\
                                           & = \frac{\phi(x)}{\sigma^2}
\end{align}

let $L_i = \frac{\partial \phi(x_i)}{\partial \mu} $
\begin{align}
    \frac{\partial J}{\partial \mu} = & - \frac{1}{m} \Bigg[  \sum_1^m y_i w \frac{\partial \phi(x_i)}{\partial \mu} + \frac{(1-y_i)}{1-h} (-w) \frac{\partial \phi(x_i)}{\partial \mu} \Bigg] \nonumber \\
                                    = &  - \frac{1}{m} \Bigg[  \sum_1^m \frac{y_i}{h_i} w L_i - \frac{(1-y_i)}{1-h} w L_i \Bigg] \nonumber \\
                                     = &  - \frac{1}{m} \Bigg[  \sum_1^m (h_i-y_i) w L_i  \Bigg] \nonumber \\
                                      = &  - \frac{1}{m} \Bigg[  \sum_1^m (h_i-y_i) w \frac{\partial \phi(x_i)}{\partial \mu}  \Bigg]
\end{align}

$\alpha$ = stepsize to update w

$\beta$ = stepsize to update $\mu$

Repeat until convergence:

\{
\begin{align}
    w_i =: & w_i - \alpha \frac{\partial J(w)}{\partial w_i} \\
    \mu_i =: & \mu_i + \beta \frac{1}{m} \Bigg[  \sum_1^m (h_i-y_i) w \frac{\partial \phi(x_i)}{\partial \mu}  \Bigg]
\end{align}
\}

## Answer 1(h)

### Report:
The algorithm derived is iterative and updates the weight vector as well as the value of
$\mu$ simultaneously. The algorithm can converge to a local minima but not on the global
minima since we are adjusting the weight vector and the mean vector simultaneously. As
a result the it would be quite difficult to optimise the cost function for both the parameters
resulting in a convergence at the local minima but not the global minima.

## Answer 1(i)

```python
Xnew_train = Xm_train[:,1].reshape(-1,1)
Xnew_test = Xm_test[:,1].reshape(-1,1)
```

```python
# includes 1 bias for each example data
def getfeaturematrix (X,polynomial):
    Xm = []
    for i in range(0,polynomial+1):
        a = np.power(X,i)
        Xm.append(a)
    Xm = np.array(Xm)
    Xm = np.squeeze(Xm, axis=(2,)).T
    return Xm
```

## Approach 1

```python
d = np.array( [ [1,2,3,5,9]] )
valid_score = []
# set lambda to zero for "no regularization"
lmbda  = np.array([0])
lmbda = lmbda.reshape(1,-1)
kfold=5
for i in range(d.shape[1]):
    polynomial = d[0,i]
    Xp_train = getfeaturematrix (Xnew_train,polynomial)
    Xp_valid = getfeaturematrix (Xnew_test,polynomial)

    RMSE_polytraindata, RMSE_polyvaliddata, result= ComputeValid_VS_lmbda(Xp_train,y_train,kfold,lmbda,norm='l2')
    valid_score = np.append( valid_score, np.mean(RMSE_polyvaliddata,axis=0))

    plt.figure()
    plt.plot(Xnew_train,result,'*',label='prediction for d = %s'%d[0,i])
    plt.plot(Xnew_train,y_train,'o',label='y')
    plt.xlabel('X')
    plt.legend()
    plt.show()
```

### Report: the best value of degree $d \in {1,2,3,5,9}$ for non-regularized linear regression

```python
print('best value of degree,d :', d[0,valid_score.argmin()])
```

    best value of degree,d : 2

```python
# create lambda parameter vector
lmbda = []
l = 0.01
while l <= 10**1:
    if l==0 :
        break
    else :
        lmbda = np.append(lmbda, l)
    l = l*10
lmbda = lmbda.reshape(1,-1)
print('lmbda:',lmbda)
```

    lmbda: [[ 0.01  0.1   1.   10.  ]]

```python
d = np.array( [ [9]] )
d.shape
```

    (1, 1)

```python
d[0,0]
```

    9

```python
d = np.array( [ [9]] )

polynomial = d[0,0]
Xp_train = getfeaturematrix (Xnew_train,polynomial)
Xp_test = getfeaturematrix (Xnew_test,polynomial)

kfold=5

```

## Approach 2

```python
RMSE_polytraindata, RMSE_polyvaliddata, result = ComputeValid_VS_lmbda(Xp_train,y_train,kfold,lmbda,'l2')
```

### Report: best $\lambda$ value using five fold cross-validation for d=9 and L2 regulizer

```python
lmbda[0, RMSE_polytraindata.mean(0).argmin()]
```

    0.01

```python
x_train_sorted = np.sort(Xnew_train,axis=0)
msk = Xnew_train.argsort(axis=0)#.repeat(lmbda.shape[1],axis=1)
result = result[msk].reshape(80,4)              # dimension of Xm_sort becomes (80,1,2)

```

```python
plt.plot(Xnew_train,y_train,'<',label='y')
for i in range (lmbda.shape[1]):
    #plt.figure(1%1)
    plt.plot(x_train_sorted,result[:,i],label='prediction for lambda %s'%lmbda[0,i])
    plt.xlabel('X')
    #plt.plot(x_train_sorted,y_train,'<',label='y')

plt.legend()
plt.show()
```

## Approach 3

```python
RMSE_polytraindata, RMSE_polyvaliddata, result = ComputeValid_VS_lmbda(Xp_train,y_train,kfold,lmbda,'l1')
```

    /home/samin/environments/py36_env/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:491: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
      ConvergenceWarning)

### Report: best $\lambda$ value using five fold cross-validation for d=9 and L1 regulizer

```python
lmbda[0, RMSE_polytraindata.mean(0).argmin()]
```

    0.1

```python
x_train_sorted = np.sort(Xnew_train,axis=0)
msk = Xnew_train.argsort(axis=0)#.repeat(lmbda.shape[1],axis=1)
result = result[msk].reshape(80,4)              # dimension of Xm_sort becomes (80,1,2)

```

```python
plt.plot(Xnew_train,y_train,'<',label='y')
for i in range (lmbda.shape[1]):
    #plt.figure()
    plt.plot(x_train_sorted,result[:,i],label='prediction for lambda %s'%lmbda[0,i])
    plt.xlabel('X')
    #plt.plot(x_train_sorted,y_train,'<',label='y')

plt.legend()
plt.show()

```

## Test error for optimal parameters for 3 approaches

### Approach 1
* No regularization $\lambda$=0
* As we got best result for d = 2

```python
d = np.array( [ [2]] )

polynomial = d[0,0]
Xp_train = getfeaturematrix (Xnew_train,polynomial)
Xp_test = getfeaturematrix (Xnew_test,polynomial)

# set lambda to zero for "no regularization"
lmbda  = np.array([0])
lmbda = lmbda.reshape(1,-1)

```

```python
# Test error for optimal parameters
W_vec1 = []
_ , error_ap1 ,W_vec1,_,_ = RMSEresult(Xp_train,y_train,Xp_test,y_test,lmbda,'l2')
```

```python
error_ap1
```

    array([7.05318339])

### Approach 2
* Given: d=9 and L2 regulizar
* best lambda = 0.01

```python
d = np.array( [ [9]] )

polynomial = d[0,0]
Xp_train = getfeaturematrix (Xnew_train,polynomial)
Xp_test = getfeaturematrix (Xnew_test,polynomial)

```

```python
lmbda = np.array([0.01]).reshape(1,-1)
lmbda.shape
```

    (1, 1)

```python
_ , error_ap2 ,W_vec2,_,_ = RMSEresult(Xp_train,y_train,Xp_test,y_test,lmbda,'l2')
```

```python
error_ap2
```

    array([9.08962225])

### Approach 3
* Given: d=9 and L1 regulizar
* best lambda = 0.1

```python
lmbda = np.array([0.1]).reshape(1,-1)
lmbda.shape
```

    (1, 1)

```python
_ , error_ap3 ,W_vec3,_,_ = RMSEresult(Xp_train,y_train,Xp_test,y_test,lmbda,'l1')
```

    /home/samin/environments/py36_env/lib/python3.6/site-packages/sklearn/linear_model/coordinate_descent.py:491: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations. Fitting data with very small alpha may cause precision problems.
      ConvergenceWarning)

```python
error_ap3
```

    array([9.06936404])

### Report: weight associated with each polynomial for L1 and L2 regression
* As value of lambda is not mentioned we're using optimal of lambda for L1 and L2 we found on last experiment

```python
d = np.array( [ [1,2,3,5,9]] )
```

```python
lmbda = np.array([0.01]).reshape(1,-1)

for i in range(d.shape[1]):
    polynomial = d[0,i]
    Xp_train = getfeaturematrix (Xnew_train,polynomial)
    Xp_test = getfeaturematrix (Xnew_test,polynomial)

    lmbda = np.array([0.01]).reshape(1,-1)
    _ , _ ,W_vec2,_,_ = RMSEresult(Xp_train,y_train,Xp_test,y_test,lmbda,'l2')

    lmbda = np.array([0.1]).reshape(1,-1)
    _ , _ ,W_vec3,_,_ = RMSEresult(Xp_train,y_train,Xp_test,y_test,lmbda,'l1')
    #print(W_vec2)
    plt.figure()
    plt.title ('weights for polynomial d=%d'%polynomial)
    plt.ylabel('weights')
    plt.plot(W_vec2,label='L2 regression')
    plt.plot(W_vec3,label='L1 regression')

    plt.legend()
    plt.show()
```

### Report
* for X'^0 = 1 the bias term is set to zero for both L1 and L2 regression
* For lower polynomial the weights are similar but for polynomial = 9 weight differs
* At d= 9 , more L1 regression highly penalize the higher degree polynomila of data X'

### Compare the result with no regularization at optimal lambda=2

```python
polynomial = 2
Xp_train = getfeaturematrix (Xnew_train,polynomial)
Xp_test = getfeaturematrix (Xnew_test,polynomial)

lmbda = np.array([0.01]).reshape(1,-1)
_ , _ ,W_vec2,_,_ = RMSEresult(Xp_train,y_train,Xp_test,y_test,lmbda,'l2')

lmbda = np.array([0.1]).reshape(1,-1)
_ , _ ,W_vec3,_,_ = RMSEresult(Xp_train,y_train,Xp_test,y_test,lmbda,'l1')
```

```python
plt.plot(W_vec1,label='optiaml polynomial')
plt.plot(W_vec2,label = 'l2 optimal lambda')
plt.plot(W_vec3,label = 'l1 optimal lambda')
plt.legend()
plt.show()
```

### Report
* Weights are similar for optimal degree d and L1, L2 regression for optimal lambda
* So no regulizar needed for d=2

```python

```

############################################################################################

## 2. Maximum likelihood

##############################################################################################

```python
from scipy.stats import beta

#plot posterior density
a = 50
b = 50
theta = np.arange(0.01,1,0.01)
y = beta.pdf (theta,a,b)
plt.plot(theta,y)
plt.title('Beta: a=%.1f .b=%.1f' % (a,b))
plt.xlabel('$\\theta$')
plt.ylabel('Probability density')
plt.show()
```

posterior mean $\mathbb{E}[\theta] = \frac{\alpha}{\alpha+\beta}$

```python
mean = beta.stats(a,b, moments='m')
print('Posterior Mean',mean)
```

    Posterior Mean 0.5

* For infintely large data sets Bayesian and maximum likelihood  results  will agree.
* For finite data sets, posterior mean for $\theta$ always lies between the prior mean and maximum likelihood estimation of $\theta$.

Thus, Maximum likelihood estimator is a good summary of the distribution

### Derive maximum likelihood estimate of W

$y_i = h_w(x_i) +\epsilon_i$ where $\epsilon_i \sim N (0,\sigma^2_i) $

We can say $y$ is a normal distribution with mean $XW$ and variance $\sigma^2$
$$y \sim N(XW,\sigma^2)$$
\begin{align}
    L(XW,\sigma^2 |x) =& (\frac{1}{2\pi\sigma^2})^{\frac{N}{2}} e^{ - \frac{1}{2\sigma^2} \sum_{n=0}^{N-1} (y_i - x_i w)^2 }\\
                    = & (\frac{1}{2\pi\sigma^2})^{\frac{N}{2}} e^{ - \frac{1}{2\sigma^2} (Y-XW)^T(Y-XW) } \\
     ln ( L(XW,\sigma^2 |x) ) = & - \frac{N}{2} ln(2\pi) - \frac{N}{2} ln(\sigma^2) - \frac{1}{2\sigma^2} (Y-XW)^T(Y-XW) \nonumber \\
     \frac{\partial ln ( L(XW,\sigma^2 |x) ) }{\partial w} = & \frac{1}{2\sigma^2} \frac{\partial (Y-XW)^T(Y-XW)}{\partial w} \nonumber
\end{align}
Optimal value for W

\begin{align}
    \frac{\partial ln ( L(XW,\sigma^2 |x) ) }{\partial w} = & 0 \nonumber \\
    \frac{1}{2\sigma^2} \frac{\partial (Y-XW)^T(Y-XW)}{\partial w}  = & 0 \nonumber \\
    W = & \frac{(X^TY)^{-1}}{X^TX} \nonumber \\
    W = & (X^TY)^{-1} X^TX
\end{align}

```python

```

####################################################################################################

# 3. Baysian analysis and biased coin

###################################################################################

```python

```

let unknown bias  $\theta$

$P(x = H|\theta) = \theta$

thus, $P(x = T|\theta) = 1 - \theta$

Probability distribution over $x$ can be written as

$$ bern(x|\theta) = \theta^x (1-\theta)^{1-x} $$
$D = \{x_1,x_2, ... x_w\}$

We can construct a likelihood function which is a function of $\theta$

\begin{align}
    P(D|\theta) = & \prod_{n=1}^N P(x_n|\theta) = \prod_{n=1}^N \theta^{x_n} (1-\theta)^{1-x_n} \\
    ln (P(D|\theta)) = & \prod_{n=1}^N ln (P(x_n|\theta)) = \prod_{n=1}^N x_n ln(\theta) +(1-x_n) ln(1-\theta) \nonumber \\
    \frac{\partial ln (P(D|\theta))}{\partial \theta} = & \prod_{n=1}^N \frac{x_n}{\theta}  +(1-x_n) \frac{1}{1-\theta} (-1) = 0 \nonumber \\
    \theta_{ML} = & \frac{1}{N} \sum_{n=1}^N x_n
\end{align}

Now outcomes are ${ H , H , H}$ equivalent to $ x_n = {1, 1, 1}$

Thus
\begin{align}
     \theta_{ML} = & \frac{1}{3} \times [1+1+1] \nonumber \nonumber \\
                 = & 1
\end{align}

### Report:
This is not a good estimator. We don't have enough data to make good prediciton. As the dataset is very small the model overfits over samples

```python

```

#############################

# 4. Multivariate Regression

#############################

#####################

## Answer 4(a)

#####################

input-output training data $ \{(x_i,y_i) \}^m_{i=1} $ where each $x_i$ is a $d$-dimensional vector and each $y_i$ is a $p$-dimensional vector, meaning for each example of feature $x_i$ we get $p$ number of possible outputs. Thus $X$ is $m\times d $ dimensinal and $Y$ is $m \times p$ dimensional.
As we know $$Y=\phi(X) W$$
Assuming $\phi(X)$ remains same dimension as $X$, $W$ would be $d \times p$ dimensional.
For single output case, squared error loss function was
\begin{align}
    J(W) = & \sum_{i=1}^m || W^T x_i - y_i ||^2_2 \\
    J(W) = &  || X W - y ||^2_2 \\
    \bigtriangledown_W J= & \bigtriangledown_W \Big[ (XW-Y)^T (XW-Y) \Big]
\end{align}
Where W was $d \times 1$ dimensional.

But now for multiple outputs of $y_i$, assuming outputs are independet of prior,

squared error loss function for each possible $p$ output will be :
\begin{align}
    J(W_p) = &  || X W_p - y_p ||^2_2 \\
     \bigtriangledown_{W_p} J= & \bigtriangledown_{W_p} \Big[ (XW-Y)^T (XW-Y) \Big] \\
                            = & X^TX W_p -X^T y_p = 0 \\
    thus, \, W_p = & (X^TX)^{-1} X^T y_p
\end{align}

#####################

## Answer 4(b)

#####################

From the proof above we can say that if we independently solve $p$ classical linear regression an and get $W$ parameter $p$ times, we solve the multivariate regression task.

In this problem we assumed outputs are independent of prior but in baysian approximation where we have a prior on output, for multivate output we need to consider all the output prior for future $W$ parameter updates and thus can't calculate posterior output inpendently.

consider noisy conversation $y = f(x) + \epsilon = \phi(x)^T w + \epsilon$
according to bayes rules:

$ P_\phi(w|y,X) = \frac{ P_\phi(y|X,w)P(w) }{P_\phi(y|X) } $

```python

```

#####################################################

# 5. Kernels and RKHS

#####################################################

###########################

## Answer 5 (a)

###########################

```python
import numpy as np
from sklearn.metrics.pairwise import linear_kernel as lk
import matplotlib.pyplot as plt
```

```python
# defined input space
X1 = np.linspace(-1,1,10).reshape(-1,1)
```

```python
def linear_kernel (X,X1):
    lk = np.dot(X,X1)
    return lk
```

```python
k =  linear_kernel (1,X1)
k_func1 = 3 * k
k_func2 = 3 * k + 4
k_func3 = 3 * k + 4 * k
```

```python
plt.plot(X1,k_func1,label='generated func : $ g(k) = 3k $')
plt.plot(X1,k_func2,label='generated func : $ g(k) = 3k+4 $')
plt.plot(X1,k_func3,label='generated func : $ g(k) = 3k+4k $')
plt.plot(X1,k,'.',label = 'linear kernel function')
plt.legend()
plt.show()
```

Report $||f||_k$

$||f||^2_k = <f,f>_k = <k_w,k_w>_k = k(w,w)$

function 1:
   $ k_1(w,w) = 3 k(w,w) = 3 w^2  < \infty $

function 2:
   $ k_2(w,w) = 3 k(w,w)+4 = 3 w^2 +4 < \infty $

function 3:
   $ k_3(w,w) = 3 k(w,w)+4 k(w,w) = 3 w^2 + 4 w^2 < \infty$

##################################

## Answer 5 (b)

###################################

```python
def gaussian_kernel (X,X1,p):
    gk = np.exp (-1 *( (X-X1)**2 ) / (2*(p**2) ) )
    return gk
```

### For each bandwidth $\rho \in \{ 0.1, 0.5, 1 \} $ function hase been generated

```python
# given
p = np.matrix([0.1,0.5,1])

# defined input space for given limit X = [-1,1]
X1 = np.linspace(-1,1,100).reshape(-1,1)
X2 = X1

for i in range(p.shape[1]):
    k = gaussian_kernel (X1[-1,0],X2,p[0,i])
    plt.plot(X1,k , label = 'p = %s'%p[0,i])
plt.xlabel('$\mathbb{X2}$')
plt.title('kernel function k(X1,X2) plotted as a functino of X2 dor X1=1')
plt.legend()
plt.show()
```

```python

```

### We're approximating the feature mapping  $\phi(.)$ using taylor series expansion up to d=150 features

$\gamma = \frac{1}{2 \sigma^2}$
\begin{align}
    e^{- \gamma ||x_i - x_j ||} = & e^{- \gamma(x_i-x_j)^2} = e^{-\gamma x_i^2 + 2\gamma x_i x_j-\gamma x_j^2}  \\
                                = & e^{-\gamma x_i^2 \gamma x_j^2} \Bigg(1 \times 1 + \sqrt{\frac{2\gamma}{1!}}x_i \times \sqrt{\frac{2\gamma}{1!}}x_j + \sqrt{\frac{(2\gamma)^2}{2!}}x_i^2 \times \sqrt{\frac{(2\gamma)^2}{2!}}x_j^2 \nonumber \\ & + \sqrt{\frac{(2\gamma)^3}{3!}}x_i^3 \times \sqrt{\frac{(2\gamma)^3}{3!}}x_j^3 + .......  \Bigg) \\
                                = & \phi(x_i)^T\phi(x_j)
\end{align}
here $\phi(x) = e^{-\gamma x^2} \Bigg[ 1, \sqrt{\frac{(2\gamma)^1}{1!}}x^1 , \sqrt{\frac{(2\gamma)^2}{2!}}x^2 ..... \sqrt{\frac{(2\gamma)^{150} }{150!}}x^{150} \Bigg]^T$

```python

```

### ploting gaussian basis function $\phi(x_i)$, where $x_i$  = 1

```python
def gaussian_basis(x,p,d):
    gamma = 1/(2*(p**2))
    g = np.matrix(np.zeros([d+1,1]))
    for i in range(d+1):
        from scipy.special import factorial
        g[i,0] = np.sqrt(((2*gamma)**i)/factorial(i, exact=True))*(x**i)
    g_basis = np.exp(-gamma*(x)**2)*g
    return g_basis

d = 150
g_basis=gaussian_basis(X1[-1,0],p[0,1],d) #gaussian basis for Xi = 1
plt.plot(g_basis)
```

### Plotting function for fixed $\rho$=0.5

```python
d = 150
x1=X1[-1,0]
g_basis1=gaussian_basis(x1,p[0,1],d) # for fixed p=0.5, X1=1

func = np.matrix(np.zeros([X1.shape[0],1]))
for k in range(X1.shape[0]):
    x2 = X2[k,0]
    g_basis2=gaussian_basis(x2,p[0,1],d) # for fixed p
    func[k,:]=np.dot(g_basis1.T,g_basis2)
plt.plot(X1,func,label='p = %s'%p[0,1])
plt.xlabel('$\mathbb{X2}$')
plt.ylabel('k(x1,x2)')
plt.title('x1 = 1')
plt.legend()
plt.show()
```

### ploting functions for $\rho \in \{ 0.1, 0.5, 1 \} $

```python
x1=X1[-1,0]
d = 120

# evaluate kernel funciton for different bandwidth
for m in range (p.shape[1]):
    g_basis1=gaussian_basis(x1,p[0,m],d)
    func = np.matrix(np.zeros([X1.shape[0],1]))

    # get kernel function as a function of X2 (when bandwidth P and X1 fixed)
    for k in range(X1.shape[0]):
        x2 = X2[k,0]
        g_basis2=gaussian_basis(x2,p[0,m],d)
        func[k,:]=np.dot(g_basis1.T,g_basis2)
    plt.plot(X1,func,label='p = %s'%p[0,m])
    plt.xlabel('$\mathbb{X2}$')
    plt.ylabel('k(x1,x2)')
    plt.title('kernel function k(X1,X2) plotted as a functino of X2 dor X1=1')
    plt.legend()
plt.show()
```

```python

```


Model the uncertain inflow energy
==========================
In this module, we prepare the time series model for the uncertain inflow energy $X_t$.

Whole model description:
https://www2.isye.gatech.edu/people/faculty/Alex_Shapiro/Rep-Dec2011.pdf

```python
import pandas
import numpy
# read historical data for the four reserviors
hist = [
    pandas.read_csv(
        "./data/hist_{}.csv".format(i),
        sep=";",
        index_col=0,
    )
    for i in range(4)
]
# concatenate the data to fill missing value yearly
hist_all = pandas.concat(hist, axis=1, ignore_index=True)
hist_all.fillna(hist_all.mean(), inplace=True)
hist_all_log = numpy.log(hist_all)
# split the data back to individual reserviors
hist = [hist_all.iloc[:,12*i:12*(i+1)] for i in range(4)]
```

Let $Y_t = log(X_t)$. We hope to fit the following model:
\begin{equation}
    Y_t - \mu_t = \phi_t(Y_{t-1}-\mu_{t-1})+ \epsilon_t
\end{equation}
Where:
1. $\mu_t=\mu_{t+12}$ is the monthly averages of $Y_t$
2. $\epsilon_t$ independently follows multivariate normal distribution

Compute $\tilde{\mu}_t$

```python
hist_log = [numpy.log(hist[i]) for i in range(4)]
mu = numpy.array([hist_log[i].mean().values for i in range(4)]).T
exp_mu = numpy.exp(mu)
```

# Approach one: direct calibration without linearization

```python
phi = numpy.empty([12,4])
for i in range(4):
    for t in range(12):
        if t == 0:
            X = hist_log[i].iloc[:-1,11]
            Y = hist_log[i].iloc[1:,0]
        else:
            X = hist_log[i].iloc[:,t-1]
            Y = hist_log[i].iloc[:,t]
        X -= mu[t-1,i]
        Y -= mu[t,i]
        phi[t][i] = numpy.dot(X,Y)/numpy.dot(X,X)
sigma = [None] * 12
for t in range(12):
    if t == 0:
        X = pandas.DataFrame([hist_log[i].iloc[:-1,11] for i in range(4)]).T
        Y = pandas.DataFrame([hist_log[i].iloc[1:,0] for i in range(4)]).T
    else:
        X = pandas.DataFrame([hist_log[i].iloc[:,t-1] for i in range(4)]).T
        Y = pandas.DataFrame([hist_log[i].iloc[:,t] for i in range(4)]).T
    X.reset_index(drop=True,inplace=True)
    Y.reset_index(drop=True,inplace=True)
    X.columns = [0,1,2,3]
    Y.columns = [0,1,2,3]
    matrix = Y - mu[t] - phi[t] * (X - mu[t-1])
    sigma[t] = numpy.cov(matrix.T,ddof=1)
```

```python
def generator_wo_linearization(random_state, size):
    inflow = numpy.empty([size,13,4])
    inflow[:,0,:] = [[41248.715300,7386.860854,10124.561460,6123.808537]]
    for t in range(12):
        noise = random_state.multivariate_normal(
            mean=[0]*4, cov=sigma[t],size=size)
        inflow[:,t+1,:] = numpy.exp(
            mu[t] + phi[t] * (numpy.log(inflow[:,t,:]) - mu[t-1]) + noise
        )
    return inflow[:,1:,:]
```

## Approach two: approximate calibration with linearization

The above model is not linear w.r.t $X_t$. Plug in $Y_t = log(X_t)$ and linearize the model (first order Taylor expansion approximation), we get the following model

\begin{equation}
    X_t = e^{\epsilon_t} [\tilde{\mu}_t + \gamma_t \frac{\tilde{\mu}_t}{\tilde{\mu}_{t-1}} (X_{t-1}-\tilde{\mu}_{t-1})]
\end{equation}
Where:
1. $\gamma_t = \gamma_{t+12}$
2. $\tilde{\mu}_t = e^{\mu_t},\tilde{\mu}_{t+12}=\tilde{\mu}_t$
2. $\epsilon_t$ independently follows multivariate normal distribution $N(0,\Sigma_t)$, $\Sigma_{t+12}=\Sigma_{t}$

This model is called auto-regressive process with multiplicative error. We will fit this model to data.

Denote $R_t = \frac{X_t - \tilde{\mu}_t}{\tilde{\mu}_t}$ and set $\epsilon_t$ to 0, the above model can be written as
\begin{equation}
    R_t = \gamma_t R_{t-1}
\end{equation}
$\Sigma_t$ is modelled as sample covariance matrix for
\begin{equation}
    log(\frac{X_t}{(1-\gamma_t)\tilde{\mu}_t +\gamma_t \frac{\tilde{\mu}_t}{\tilde{\mu}_{t-1}} X_{t-1}})
\end{equation}

Perform least square fit to the $R_t$ sequence on a monthly basis. Beware to keep temporal order (December data should be lagged properly).

```python
gamma = numpy.empty([12,4])
for i in range(4):
    for t in range(12):
        if t == 0:
            X = hist[i].iloc[:-1,11]
            Y = hist[i].iloc[1:,0]
        else:
            X = hist[i].iloc[:,t-1]
            Y = hist[i].iloc[:,t]
        X = X/exp_mu[t-1,i] - 1
        Y = Y/exp_mu[t,i] - 1
        gamma[t][i] = numpy.dot(X,Y)/numpy.dot(X,X)
```

Compute the sample cov

```python
sigma = [None] * 12
for t in range(12):
    if t == 0:
        X = pandas.DataFrame([hist[i].iloc[:-1,11] for i in range(4)]).T
        Y = pandas.DataFrame([hist[i].iloc[1:,0] for i in range(4)]).T
    else:
        X = pandas.DataFrame([hist[i].iloc[:,t-1] for i in range(4)]).T
        Y = pandas.DataFrame([hist[i].iloc[:,t] for i in range(4)]).T
    X.reset_index(drop=True,inplace=True)
    Y.reset_index(drop=True,inplace=True)
    X.columns = [0,1,2,3]
    Y.columns = [0,1,2,3]
    matrix = numpy.log(
        Y
        / (
            (1-gamma[t]) * exp_mu[t]
            + gamma[t] * exp_mu[t]/exp_mu[t-1] * X
        )
    )
    sigma[t] = numpy.cov(matrix.T,ddof=1)
```

Now we can construct a sample path generator for the above time series model

```python
def generator_w_linearization(random_state, size):
    inflow = numpy.empty([size,13,4])
    inflow[:,0,:] = [[41248.715300,7386.860854,10124.561460,6123.808537]]
    for t in range(12):
        noise = numpy.exp(random_state.multivariate_normal(
            mean=[0]*4, cov=sigma[t],size=size))
        inflow[:,t+1,:] = noise * (
            (1-gamma[t]) * exp_mu[t]
            + gamma[t] * exp_mu[t]/exp_mu[t-1] * inflow[:,t,:]
        )
    return inflow[:,1:,:]
```

Use our sample path generator to simulate inflow data.

```python
import matplotlib.pyplot as plt
from msppy.utils.plot import fan_plot
simulation = [[] for _ in range(4)]
a_wo_linearization = generator_wo_linearization(numpy.random.RandomState(0),100)
a_w_linearization = generator_w_linearization(numpy.random.RandomState(0),100)
```

Use fan plots to compare simulated data and historical data. The black dash line represents the average value. Plots on the first column show the historical data. Plots on the second and third columns show the simulated data from model w/w.o. linearization respectively.

```python
fig, ax = plt.subplots(4,3,figsize=(15,10), sharex=True, sharey='row')
for i in range(4):
    for j in range(3):
        fan_plot(hist[i].values, ax[i][0])
        fan_plot(a_wo_linearization[...,i], ax[i][1])
        fan_plot(a_w_linearization[...,i], ax[i][2])
```


 Home Quiz 1 - Logistic Regression

$$
\text{Chatziefraimidis Lefteris 2209}\\
$$

## Problem 1: Gradient Descent

We will estimate the parameters $w_{0},w_{1},w_{2}$ using gradient descent for the following prediction model:

$$ y = w_{0} + w_{1}x_{1} + w_{2}x_{2} + w_{3}x_{1}^2 + \epsilon \space \text{  ,where }\space\epsilon\text{ ~ } N(0,\sigma^2)$$

$
\text{The error of the approximation: } \epsilon = y - y_{predicted} = y - w_{0} + w_{1}x_{1} + w_{2}x_{2} + w_{3}x_{1}^{2}
$

### $\triangleright$ Exercise (a) :

Gaussian Distribution :

$X$ is distributed according to normal (or Gaussian) distribution with mean $\mu$ and variance $\sigma^2$

$$
\begin{aligned}
X &\sim \mathcal{N}(\mu,\sigma^2) \\ \\
p(X = x|\mu,\sigma^2)&= \frac{1}{\sqrt{2\pi\sigma^2}} e^{-\frac{1}{2\sigma^2}(x-\mu)^2}
\end{aligned}
$$

So,for our model we have:

$$P(y|x_{1},x_{2}) = \frac{1}{\sqrt{2\pi\sigma^2}}e^-{\frac{(y-(w_{0} + w_{1}x_{1} + w_{2}x_{2} + w_{3}x_{1}^2))^2}{2\sigma^2}}$$

and the above expression its called likelihood function.

### $\triangleright$ Exercise (b) :
Assume you are given a set of training observations $(x_{1}^{(i)},x_{2}^{(i)},y^{(i)})$ for $i=1,....,n$

Log-likelihood of this training data:

$$LogP(y|x_{1},x_{2}) =  \sum_{i=1}^n \log \frac{1}{\sqrt{2\pi\sigma^2}} e^-{\frac{(y-(w_{0} + w_{1}x_{1}^{(i)} + w_{2}x_{2}^{(i)} + w_{3}x_{1}^{(i)2}))^2}{2\sigma^2}}$$

$$ = \sum_{i=1}^n \left[-\frac{1}{2}\log 2\pi\sigma^2 - \frac{1}{2\sigma^2}(y-(w_{0} + w_{1}x_{1}^{(i)} + w_{2}x_{2}^{(i)} + w_{3}x_{1}^{(i)2}))^2\right]$$

$$ = \sum_{i=1}^n \left[- \frac{1}{2\sigma^2}(y-(w_{0} + w_{1}x_{1}^{(i)} + w_{2}x_{2}^{(i)} + w_{3}x_{1}^{(i)2}))^2\right]$$

### $\triangleright$ Exercise (c) :
Based on your answer above, we can write a loss function $f(w_{0},w_{1},w_{2},w_{3})$ that can be minimized to find the desired parameter estimates:

$$f(w_{0},w_{1},w_{2},w_{3}) = \frac{1}{n}\sum_{i=1}^n \left[(y-(w_{0} + w_{1}x_{1}^{(i)} + w_{2}x_{2}^{(i)} + w_{3}x_{1}^{(i)2}))^2\right] , \frac{1}{n} \text{ makes the loss interpretable}$$

This particular loss function is also known as Mean squared error(MSE).We can use gradient descent to optimize it.

### $\triangleright$ Exercise (d) :
Lets calculate the gradient of $f(w)$ with respect to the parameter vector $w = [w_{0},w_{1} ,w_{2},w_{3}]^{T} :$

$$\nabla_w f(w) = \begin{bmatrix}
\frac{\partial f(w)}{\partial w_0} &\frac{\partial f(w)}{\partial w_1}  &\frac{\partial f(w)}{\partial w_2}  & \frac{\partial f(w)}{\partial w_3}
\end{bmatrix}^{T}$$

Our goal was to find $w_0,w_1,w_2,w_3$ such that:

$$\nabla_w f(w) = \begin{bmatrix}
\frac{\partial f(w)}{\partial w_0} &\frac{\partial f(w)}{\partial w_1}  &\frac{\partial f(w)}{\partial w_2}  & \frac{\partial f(w)}{\partial w_3}
\end{bmatrix}^{T} = \text{[ 0 0 0 0 ]}$$

because this guarantees that $f(w)$ is minimized.

$$
\nabla_w f(w) =
\begin{bmatrix}
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-1))
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,1})
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,2})
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,1}^{2})
\\
\\
\end{bmatrix} =
\begin{bmatrix}
0
\\
0
\\
0
\\
0
\end{bmatrix}
$$

### $\triangleright$ Exercise (e) :
Gradient descent update rule for $w$ in terms of $\nabla_w f(w)$:

$$
w^{(\textrm{iteration}+1)} = w^{(\textrm{iteration})} - \alpha\nabla f(w^{(\textrm{iteration})})
\text{ , }\alpha\text{ : learning rate}
$$

$$
w^{(\textrm{iteration}+1)} = w^{(\textrm{iteration})} - \alpha\begin{bmatrix}
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-1))
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,1})
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,2})
\\
\frac{2}{n}\sum_{i=1}^{n} [y_i-(w_{0} + w_{1}x_{i,1} + w_{2}x_{i,2} + w_{3}x_{i,1}^{2})](-x_{i,1}^{2})
\\
\\
\end{bmatrix}
$$

### $\triangleright$ Exercise (f) :

```python
from sympy import *

#Define the symbols
x1 = Symbol('x1')
x2 = Symbol('x2')

w0 = Symbol('w0')
w1 = Symbol('w1')
w2 = Symbol('w2')
w3 = Symbol('w3')
y = Symbol('y')
n = Symbol('n')
a =Symbol('a')

#Define the expression y prediction
ypred = w0 + w1*x1 + w2*x2 + w3*(x1**2)

#Define the loss function f
f = (1/n)*(y - ypred)**2

#Calculate the derivatives in order to calculate the nabla
df_dw0 = diff(f,w0)
df_dw1 = diff(f,w1)
df_dw2 = diff(f,w2)
df_dw3 = diff(f,w3)

#Nabla of loss function
nabla_f = Matrix([[df_dw0],[df_dw1],[df_dw2],[df_dw3]])

#Calculate the new weights
weights = Matrix([[w0],[w1],[w2],[w3]])
weights = weights - a*nabla_f

print("Nabla: ")
print(nabla_f)
print("Weights: ")
print(weights)
```

    Nabla:
    Matrix([[(2*w0 + 2*w1*x1 + 2*w2*x2 + 2*w3*x1**2 - 2*y)/n], [-2*x1*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n], [-2*x2*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n], [-2*x1**2*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n]])
    Weights:
    Matrix([[-a*(2*w0 + 2*w1*x1 + 2*w2*x2 + 2*w3*x1**2 - 2*y)/n + w0], [2*a*x1*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n + w1], [2*a*x2*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n + w2], [2*a*x1**2*(-w0 - w1*x1 - w2*x2 - w3*x1**2 + y)/n + w3]])

## Problem 2: Logistic Regression

### $\triangleright$ Exercise (a) :

#### Some theory will we need:
Sigmoid (Logistic) Function:
$$
\sigma(z) = \frac{1}{1 + \exp(-z)}
$$

In logistic regression we model a binary variable $y \in \{0,1\}(Bernoulli)$

$$
p(y|x,\beta_0,\beta) = \sigma(\beta_0 + x^{T}\beta) = \frac{1}{1 + e^{-(\beta_0 + x^{T}\beta)}} \\
$$

Likelihood function is:

$$
L(\beta_0,\beta|y,x) = \prod_i p(y_i | x_i,\beta_0,\beta)^{y_i}(1 - p(y_i| x_i,\beta_0,\beta))^{1-y_i}
$$

Log-likelihood function is:

$$
\log L(\beta_0,\beta|y,x) = \sum_i y_i\log  p(y_i| x_i,\beta_0,\beta) + (1 - y_i)log(1 -  p(y_i| x_i,\beta_0,\beta)) \\
=\sum_i y_i\log  p(y_i| x_i,\beta_0,\beta) + (1 - y_i)log(p(y_i| x_i,\beta_0,\beta)) \\
=\sum_i y_i\log1 - y_i\log(1 + e^{-(\beta_0 + x_i\beta)}) + \log(\frac{e^{-(\beta_0 + x_i\beta)}}{1 + e^{-(\beta_0 + x_i\beta)}}) -y_i\log(e^{-(\beta_0 + x_i\beta)}) +y_i\log(1 + e^{-(\beta_0 + x_i\beta)}) \\
=\sum_i   \log(\frac{e^{-(\beta_0 + x_i\beta)}}{1 + e^{-(\beta_0 + x_i\beta)}}) -y_i\log(e^{-(\beta_0 + x_i\beta)}) \\
=\sum_i  \log(\frac{1}{1 + e^{(\beta_0 + x_i\beta)}}) -y_i\log(e^{-(\beta_0 + x_i\beta)}) \\
=\sum_i  -\log(1 + e^{(\beta_0 + x_i\beta)}) + y_i(\beta_0 + x_i\beta)
$$

Our objective function for gradient ascent:

$$
f(w_0,w) = \sum_i  -\log(1 + e^{(w_0 + x_iw)}) + y_i(w_0 + x_iw)
$$

### $\triangleright$ Exercise (b) :
We compute the partial derivative of the objective function with respect to $w_0$ and with respect to an
arbitrary $w_j$:

$$
\frac{\partial f}{\partial w_j} = - \sum_i \frac{e^{(w_0 +x_iw)}x_i}{1 + e^{(w_0 +x_iw)}} + \sum_i y_ix_i \\
=\sum_i (y_i - p(y_i|x_i,\beta_0,\beta) ) x_{ij}
$$

##  Logistic Regression Implementation

```python
import numpy as np
import pandas as pd
import numpy.matlib
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d

############################################## Functions we will use #############################################

#Define the sigmoid function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

#Calculate the logistic regression objective value
def LR_CalcObj(XTrain,yTrain,wHat):

    #Get the dimensions
    [n,p] = XTrain.shape

    #Add one's to Xtrain
    XTrain=np.c_[np.ones((n,1)),XTrain]

    #Calculate X*w abd exp(X*w)
    Xw = np.dot(XTrain,wHat)
    eXw = np.exp(Xw)

    #Calculate objective value
    return np.sum(yTrain*Xw - np.log(1+eXw))

#Check whether the objective value has converged
def LR_CheckConvg(old,new,tol):

    #Compute difference between objectives
    diff = np.abs(old-new);

    #Compare difference to tolerance
    if(diff < tol):
        return True
    else:
        return False

#Calculate the new value of wHat using the gradient
def LR_UpdateParams(wHat,grad,eta):

    #Update value of w
    wHat = wHat + eta*grad
    return wHat

#Calculate the gradient of the logistic regression
def LR_CalcGrad(XTrain,yTrain,wHat):

    #Get the dimensions
    [n,p] = XTrain.shape

    #Add one's to Xtrain
    XTrain=np.c_[np.ones((n,1)),XTrain]

    #Calculate X*w abd exp(X*w)
    z = np.dot(XTrain,wHat)
    h = sigmoid(z)

    #Return gradient
    return np.dot(XTrain.T, (yTrain - h))

#Run the gradient ascent algorithm for logistic regression
def LR_GradientAscent(XTrain,yTrain):

    #Define step size
    eta = 0.01

    #Define the covergence tolerance
    tol = 0.001

    #Get the dimensions
    [n,p] = XTrain.shape

    #Initialize wHat
    wHat = np.zeros((p+1,1))

    #Initialize objVal
    objVals=[]
    objVals.append(LR_CalcObj(XTrain,yTrain,wHat))

    #Initialize convergence flag
    hasConverged = False

    while(not hasConverged):

        #Calculate gradient
        grad = LR_CalcGrad(XTrain,yTrain,wHat)

        #Update parameter estimate
        wHat = LR_UpdateParams(wHat,grad,eta)

        #Calculate new objective
        newObj = LR_CalcObj(XTrain,yTrain,wHat)

        #Check convergence
        hasConverged = LR_CheckConvg(objVals[-1],newObj,tol)

        #Store new objective
        objVals.append(newObj)

    return wHat,objVals

#Predict the labls for a test set using logistic regression
def LR_PredictLabels(XTest,yTest,wHat):

    #Get dimensions
    [n,p] = XTest.shape

    #Add one's to XTest
    XTest=np.c_[np.ones((n,1)),XTest]

    #Calculate X*w abd exp(X*w)
    Xw = np.dot(XTest,wHat)
    eXw = np.exp(Xw)

    #Calculate p(Y = 0)
    pY0 = 1/(1 + eXw)

    #Calculate p(Y = 1)
    pY1 = eXw/(1 + eXw)

    yHat =[]
    #Choose best propability
    for i in range(0,len(pY0)):
        if(pY1[i] > pY0[i]):
            yHat.append([1])
        else:
            yHat.append([0])

    yHat=np.array(yHat)
    #Calculate error
    numErrors = np.sum(yHat!= yTest)

    return yHat,numErrors
def PlotDB():
    #Load the data

    #Training
    XTrain = np.array(pd.read_csv('XTrain.csv',header=None))
    yTrain = np.array(pd.read_csv('yTrain.csv',header=None))

    #Testing
    XTest = np.array(pd.read_csv('XTest.csv',header=None))
    yTest = np.array(pd.read_csv('yTest.csv',header=None))

    #Train logistic regression
    [wHat,objVals] = LR_GradientAscent(XTrain,yTrain)

    ind0 = []
    ind1 = []

    for i in range(len(yTest)):

        if(yTest[i] == 0):
            ind0.append(i)
        else:
            ind1.append(i)

    #Calculate decision boundary
    dbDimJ = np.arange(np.min(XTest[:,0]),np.max(XTest[:,0]),step=.01)
    dbDimK = (-(wHat[0] + wHat[1])*dbDimJ)/wHat[2]

    plt.plot(XTest[ind0,0],XTest[ind0,1],'r.')
    plt.plot(XTest[ind1,0],XTest[ind1,1],'b.')
    plt.plot(dbDimJ,dbDimK,'k-')
    plt.xlabel('Dimension 1')
    plt.ylabel('Dimension 2')
    plt.title('Logistic Regression Decision Boundary')
    plt.show()
############################################## Start of the program #############################################

#Load the data

#Training
XTrain = np.array(pd.read_csv('XTrain.csv',header=None))
yTrain = np.array(pd.read_csv('yTrain.csv',header=None))

#Testing
XTest = np.array(pd.read_csv('XTest.csv',header=None))
yTest = np.array(pd.read_csv('yTest.csv',header=None))

#Train Phase
wHat,objVals = LR_GradientAscent(XTrain,yTrain)

#Test Phase
yHat,numErrors = LR_PredictLabels(XTest,yTest,wHat)

#Print the number of misclassified examples
print('There were %d misclassified examples in the test set\n'%(numErrors))

#Plot the objective values
plt.plot(objVals)
plt.xlabel('Gradient Ascent Iteration')
plt.ylabel('Logistic Regression Objective Value')
plt.title('Convergence of Gradient Ascent for Logistic Regression')
plt.show()

print('Gradient ascent coverges after %d iterations\n'%(len(objVals)-1))

#2D Plot
PlotDB()

#Evaluate the training set and test error as a function of training set size
n = XTrain.shape[0]
kVals =np.arange(10,n+10,step=10)
m = XTest.shape[0]

#Errors for test,train
trainingError = np.zeros((len(kVals),1));
testError = np.zeros((len(kVals),1));

for i in range(len(kVals)):

    #Set k
    k=kVals[i]

    #Generate trainingset
    subsetsInds = np.random.randint(0,n,size=k)
    XTrainSubset = XTrain[subsetsInds,:]
    yTrainSubset = yTrain[subsetsInds,:]

    #Train logistic regression
    wHat,objVals = LR_GradientAscent(XTrainSubset,yTrainSubset)

    #Test classifier on training set
    [yHatTrain,numErrorsTrain] = LR_PredictLabels(XTrainSubset,yTrainSubset,wHat)
    trainingError[i] = numErrorsTrain/k;

    #Test classifier on test set
    [yHatTest,numErrorsTest] = LR_PredictLabels(XTest,yTest,wHat)
    testError[i] = numErrorsTest/m;

#Plot the above
plt.plot(kVals,trainingError)
plt.plot(kVals,testError)
plt.xlabel('Training Set Size')
plt.ylabel('Prediction Error')
plt.title('Logistic Regression Performance by Training Set Size')
plt.legend(['Training Error','Test Error'])
plt.show()

#Perform the same experiment but average over multiple random training sets
n = XTrain.shape[0]
kVals =np.arange(10,n+10,step=10)
m = XTest.shape[0]

#Errors for test,train
trainingError = np.zeros((len(kVals),1));
testError = np.zeros((len(kVals),1));

for i in range(len(kVals)):

    #Set k
    k=kVals[i]

    for j in range(0,10):

        #Generate trainingset
        subsetsInds = np.random.randint(0,n,size=k)
        XTrainSubset = XTrain[subsetsInds,:]
        yTrainSubset = yTrain[subsetsInds,:]

         #Train logistic regression
        wHat,objVals = LR_GradientAscent(XTrainSubset,yTrainSubset)

        #Test classifier on training set
        [yHatTrain,numErrorsTrain] = LR_PredictLabels(XTrainSubset,yTrainSubset,wHat)
        trainingError[i] += numErrorsTrain/k;

        #Test classifier on test set
        [yHatTest,numErrorsTest] = LR_PredictLabels(XTest,yTest,wHat)
        testError[i] += numErrorsTest/m;

    trainingError[i]/= 10;
    testError[i] /= 10;

#Plot the above
plt.plot(kVals,trainingError)
plt.plot(kVals,testError)
plt.xlabel('Training Set Size')
plt.ylabel('Prediction Error')
plt.title('Logistic Regression Performance by Training Set Size')
plt.legend(['Training Error','Test Error'])
plt.show()

```

### $\triangleright$ Exercise (g) :

$\bullet$ As the training set size increases our model overfits.That happens because we increase the training set.
Imagine if you only had 2 points for the training set. It would be easy for almost any model to match this set exactly. However, it's very likely that the model would fail horribly on the test set. It hasn't really seen enough to learn. With small training sets, the model is likely to underfit and perform poorly on the test set. This shows a high variance in the model.

$\bullet$ The more data points you add to the training set, the more it can overcome the overfitting, and will perform better on the test set. However, it can perform worse on the training set, because the model may not be able to fit each example to the model. This can be a good thing, as noise in the training set can be ignored.

### $\triangleright$ Exercise (h) :

$\bullet$ In our classification problem with two classes, a decision boundary or decision surface is a hypersurface that partitions the underlying vector space into two sets, one for each class. The classifier will classify all the points on one side of the decision boundary as belonging to one class and all those on the other side as belonging to the other class.

$\bullet$ XTrain is an $n × p$ dimensional matrix that contains one training instance per row.So in order to classify each row you need a hypersurface to separate those data.

How to find the decision boundary:
* Assume a 2D problem with two features $x_1$ and $x_2$, then
$$
p(y=1|x,\beta_0,\beta) =
\sigma{(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}=
\frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}}
$$

The decision boundary between 0 and 1 is 0.5.So:
$$
\frac{1}{1 + e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)}} = \frac{1}{2}
$$

$$
e^{-(\beta_0 + \beta_1 x_1 + \beta_2 x_2)} = 1
$$

$$
\beta_0 + \beta_1 x_1 + \beta_2 x_2 = 0 \rightarrow  x_2 = \beta_0 - \frac{\beta_1}{\beta_2} x_1
$$

Logistic Regression - Quiz 1/Quiz_1.ipynb


# Solving the Cournot Oligopoly Model by Collocation
**DEMAPP09 Cournot Oligopolist Problem**

This example is taken from section 6.8.1, page(s) 159-162 of:

Miranda, M. J., & Fackler, P. L. (2002). Applied computational economics and finance (P. L. Fackler, ed.). Cambridge, Mass. : MIT Press.

To illustrate the implementation of the collocation method for implicit function problems, consider the case of a Cournot oligopoly. In the standard microeconomic model of the firm, the firm maximizes its profits by matching marginal revenue to marginal cost (MC). An oligopolistic firm, recognizing that its actions affect the price, knows that its marginal revenue is $p + q \frac{dp}{dq}$, where $p$ is the price, $q$ the quantity produced, and $\frac{dp}{dq}$ is the marginal impact of the product on the market price. Cournot's assumption is that the company acts as if none of its production changes would provoke a reaction from its competitors. This implies that:
\begin{equation}
    \frac{dp}{dq} = \frac{1}{D'(p)} \tag{1}
\end{equation}

where $D(p)$ is the market demand curve.

Suppose we want to derive the firm's effective supply function, which specifies the amount $q = S(p)$ that it will supply at each price. The effective supply function of the firm is characterized by the functional equation
\begin{equation}
    p + \frac{S(p)}{D'(p)} - MC(S(p)) = 0 \tag{2}
\end{equation}

for every price $p>0$. In simple cases, this function can be found explicitly. However, in more complicated cases, there is no explicit solution. Suppose for example that demand and marginal cost are given by
\begin{equation*}
D(p) = p^{-\eta}  \qquad\qquad CM(q) = \alpha\sqrt{q} + q^2
\end{equation*}

so that the functional equation to be solved for $S(p)$ is
\begin{equation} \label{eq:funcional}
\left[p - \frac{S(p)p^{\eta+1}}{\eta}\right] - \left[\alpha\sqrt{S(p)} + S(p)^2\right] = 0 \tag{3}
\end{equation}

## The collocation method
In equation (3), the unknown is the supply *function* $S(p)$, which makes (3) an infinite-dimension equation. Instead of solving the equation directly, we will approximate its solution using $n$ Chebyshev polynomials $\phi_i(x)$, which are defined recursively for $x \in [0,1]$ as:
\begin{align*}
\phi_0(x) & = 1 \\
\phi_1(x) & = x \\
\phi_{k + 1}(p_i) & = 2x \phi_k(x) - \phi_{k-1}(x), \qquad \text{for} \; k = 1,2, \dots
\end{align*}

In addition, instead of requiring that both sides of the equation be exactly equal over the entire domain of $p \in \Re^+$, we will choose $n$ Chebyshev nodes $p_i$ in the interval $[a, b]$:
\begin{equation} \label{eq:chebynodes}
p_i = \frac{a + b}{2} + \frac{ba}{2}\ cos\left(\frac{n-i + 0.5}{n}\pi\right), \qquad\text{for } i = 1,2, \dots, n  \tag{4}
\end{equation}

Thus, the supply is approximated by
\begin{equation*}
S(p_i) = \sum_{k = 0}^{n-1} c_{k}\phi_k(p_i)
\end{equation*}

Substituting this last expression in (3) for each of the placement nodes (Chebyshev in this case) results in a non-linear system of $ n $ equations (one for each node) in $ n $ unknowns $ c_k $ (one for each polynomial of Cheybshev), which in principle can be solved by Newton's method, as in the last example. Thus, in practice, the system to be solved is

\begin{equation} \label{eq:collocation}
\left[p_i - \frac{\left(\sum_{k=0}^{n-1}c_{k}\phi_k(p_i)\right)p_i^{\eta+1}}{\eta}\right] - \left[\alpha\sqrt{\sum_{k=0}^{n-1}c_{k}\phi_k(p_i)} + \left(\sum_{k=0}^{n-1}c_{k}\phi_k(p_i)\right)^2\right] = 0  \tag{5}
\end{equation}

for $i=1,2,\dots, n$ and for $k=1,2,\dots,n$.

## Solving the model withPython
To solve this model we start a new Python session:

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt

from compecon import BasisChebyshev, NLP, nodeunif
from compecon.demos import demo

```

and set the $\alpha$ and $\eta$ parameters

```python
alpha= 1.0;
eta= 1.5;
```

For convenience, we define a `lambda` function to represent the demand. **Note: A lambda function is a small anonymous function in Python that can take any number of arguments, but can have only one expression. If you are curious to learn more Google "Lambda Functions in Python".**

```python
D = lambda p: p** (-eta)
```

We will approximate the solution for prices in the $p\in [a, b]$ interval, using 25 collocation nodes. The `compecon` library provides the `BasisChebyshev` class to make computations with Chebyshev bases:

```python
n= 25;
a= 0.1;
b= 3.0
S= BasisChebyshev(n, a, b, labels= ['price'], l=['supply'])
```

Let's assume that our first guess is $S(p)=1$. To that end, we set the value of `S` to one in each of the nodes

```python
p= S.nodes
S.y= np.ones_like(p)
```

It is important to highlight that in this problem the unknowns are the $c_k$ coefficients from the Chebyshev basis; however, an object of  `BasisChebyshev` class automatically adjusts those coefficients so they are consistent with the values we set for the function at the nodes (here indicated by the `.y` property).

We are now ready to define the objective function, which we will call `resid`. This function takes as its argument a vector with the 25 Chebyshev basis coefficients and returns the left-hand side of the 25 equations defined by (5).

```python
def resid(c):
    S.c= c  # update interpolation coefficients
    q= S(p) # compute quantity supplied at price nodes
    return p- q* (p** (eta+ 1)/ eta)- alpha* np.sqrt(q)- q** 2
```

Note that the `resid` function takes a single argument (the coefficients for the Chebyshev basis). All other parameters (`Q, p, eta, alpha` must be declared in the main script, where Python will find their values.

To use Newton's method, it is necessary to compute the Jacobian matrix of the function whose roots we are looking for. In certain occasions, like in the problem we are dealing with, coding the computation of this Jacobian matrix correctly can be quite cumbersome. The `NLP` class provides, besides the Newton's method (which we used in the last example), the [Broyden's Method](http://www.matthewaaronlooney.com/Optim_Class_Fall2019/Broydens_Method.html), whose main appeal is that it does not require the coding of the Jacobian matrix (the method itself will approximate it). To learn more about Broyden's Method, click on the hyperlink above and see Quasi-Newton Methods in section 3.4, page(s) 39-42 of the text.

```python
cournot = NLP(resid)
S.c = cournot.broyden(S.c, tol=1e-12)
```

After 20 iterations, Broyden's method converges to the desired solution. We can visualize this in Figure 3, which shows the value of the function on 501 different points within the approximation interval. Notice that the residual plot crosses the horizontal axis 25 times; this occurs precisely at the collocation nodes (represented by red dots). This figure also shows the precision of the approximation: outside nodes, the function is within  $\approx 1\times10^{-17}$ units from zero.

One of the advantages of working with the `BasisChebyshev` class is that, once the collocation coefficients have been found, we can evaluate the supply function by calling the `S` object as if it were a Python function. Thus, for example, to find out the quantity supplied by the firm when the price is 1.2, we simply evaluate `print(S(1.2))`, which returns `0.3950`. We use this feature next to compute the effective supply curve when there are 5 identical firms in the market; the result is shown in Figure 2.

##### Figure 2 Supply and demand when there are 5 firms

```python
nFirms= 5;
pplot = nodeunif(501, a, b)
demo.figure('Cournot Effective Firm Supply Function',
            'Quantity', 'Price', [0, nFirms], [a, b])
plt.plot(nFirms* S(pplot), pplot, D(pplot), pplot)
plt.legend(('Supply','Demand'))
plt.show();
```

##### Figure 3: Approximation residuals for equation (5)

This block generates Figure 3.

```python
p= pplot
demo.figure('Residual Function for Cournot Problem',
            'Quantity', 'Residual')
plt.hlines(0, a, b, 'k', '--', lw= 2)
plt.plot(pplot, resid(S.c))
plt.plot(S.nodes,np.zeros_like(S.nodes),'r*');
plt.show();
```

##### Figure 4: Change in the effective supply as the number of firms increases
We now plot the effective supply for a varying number of firms; the result is shown in Figure 4.

```python
m= np.array([1, 3, 5, 10, 15, 20])
demo.figure('Supply and Demand Functions', 'Quantity', 'Price', [0, 13])
plt.plot(np.outer(S(pplot), m), pplot)
plt.plot(D(pplot), pplot, linewidth= 2, color='black')
plt.legend(['m= 1', 'm= 3', 'm= 5', 'm= 10', 'm= 15', 'm= 20', 'demand']);
plt.show();
```

In Figure 4 notice how the equilibrium price and quantity change as the number of firms increases.

##### Figure 5: Equilibrium price as a function of the number of firms

The last figure in this example (Figure 5), shows the equilibrium price as a function of the number of firms.

```python
pp= (b+ a)/ 2
dp= (b- a)/ 2
m = np.arange(1, 26)
for i in range(50):
    dp/= 2
    pp= pp- np.sign(S(pp)* m- D(pp))* dp

demo.figure('Cournot Equilibrium Price as Function of Industry Size',
            'Number of Firms', 'Price')
plt.bar(m, pp);
plt.show();
```

# References

Miranda, M. J., & Fackler, P. L. (2002). Applied computational economics and finance (P. L. Fackler, ed.). Cambridge, Mass. : MIT Press.

Miranda, M. J., & Fackler, P. L., Romero-Aguilar, R (2002, 2016). CompEcon Toolbox (MATLAB).